[==========] Running 8 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from LeaderRebalancerTest
[ RUN      ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260709 12:23:03.470775  4356 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.65.62:40331
I20260709 12:23:03.472330  4356 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260709 12:23:03.473405  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:23:03.490536  4362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:23:03.490587  4365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:23:03.492036  4356 server_base.cc:1061] running on GCE node
W20260709 12:23:03.490562  4363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:23:04.684880  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:23:04.685114  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:23:04.685329  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783599784685304 us; error 0 us; skew 500 ppm
I20260709 12:23:04.686192  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:23:04.691284  4356 webserver.cc:533] Webserver started at http://127.4.65.62:45023/ using document root <none> and password file <none>
I20260709 12:23:04.692123  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:23:04.692408  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:23:04.692883  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:23:04.698498  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/master-0-root/instance:
uuid: "ea046ee8f6f442d082016860d60f40b5"
format_stamp: "Formatted at 2026-07-09 12:23:04 on dist-test-slave-l1c9"
I20260709 12:23:04.705915  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.000s	sys 0.008s
I20260709 12:23:04.710945  4371 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:04.711962  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:23:04.712307  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/master-0-root
uuid: "ea046ee8f6f442d082016860d60f40b5"
format_stamp: "Formatted at 2026-07-09 12:23:04 on dist-test-slave-l1c9"
I20260709 12:23:04.712880  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:23:04.772732  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:23:04.774268  4356 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260709 12:23:04.774719  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:23:04.845369  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.62:40331
I20260709 12:23:04.845422  4422 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.62:40331 every 8 connection(s)
I20260709 12:23:04.853355  4423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:04.874156  4423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5: Bootstrap starting.
I20260709 12:23:04.879735  4423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:04.881548  4423 log.cc:826] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5: Log is configured to *not* fsync() on all Append() calls
I20260709 12:23:04.885766  4423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5: No bootstrap required, opened a new log
I20260709 12:23:04.906126  4423 raft_consensus.cc:359] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea046ee8f6f442d082016860d60f40b5" member_type: VOTER }
I20260709 12:23:04.906702  4423 raft_consensus.cc:385] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:04.906972  4423 raft_consensus.cc:740] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea046ee8f6f442d082016860d60f40b5, State: Initialized, Role: FOLLOWER
I20260709 12:23:04.907974  4423 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea046ee8f6f442d082016860d60f40b5" member_type: VOTER }
I20260709 12:23:04.908514  4423 raft_consensus.cc:399] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:23:04.908816  4423 raft_consensus.cc:493] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:23:04.909160  4423 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:04.913292  4423 raft_consensus.cc:515] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea046ee8f6f442d082016860d60f40b5" member_type: VOTER }
I20260709 12:23:04.914062  4423 leader_election.cc:304] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea046ee8f6f442d082016860d60f40b5; no voters: 
I20260709 12:23:04.915729  4423 leader_election.cc:290] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:23:04.916078  4426 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:04.918721  4426 raft_consensus.cc:697] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [term 1 LEADER]: Becoming Leader. State: Replica: ea046ee8f6f442d082016860d60f40b5, State: Running, Role: LEADER
I20260709 12:23:04.919677  4426 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea046ee8f6f442d082016860d60f40b5" member_type: VOTER }
I20260709 12:23:04.920696  4423 sys_catalog.cc:565] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:23:04.930328  4427 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea046ee8f6f442d082016860d60f40b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea046ee8f6f442d082016860d60f40b5" member_type: VOTER } }
I20260709 12:23:04.930292  4428 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea046ee8f6f442d082016860d60f40b5. Latest consensus state: current_term: 1 leader_uuid: "ea046ee8f6f442d082016860d60f40b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea046ee8f6f442d082016860d60f40b5" member_type: VOTER } }
I20260709 12:23:04.931068  4428 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [sys.catalog]: This master's current role is: LEADER
I20260709 12:23:04.931022  4427 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [sys.catalog]: This master's current role is: LEADER
I20260709 12:23:04.934710  4436 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:23:04.946277  4436 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:23:04.948597  4356 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:23:04.961709  4436 catalog_manager.cc:1383] Generated new cluster ID: 3893b130af75440a933ffd0760f2bafd
I20260709 12:23:04.961974  4436 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:23:05.007418  4436 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:23:05.009546  4436 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:23:05.036212  4436 catalog_manager.cc:6070] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5: Generated new TSK 0
I20260709 12:23:05.037294  4436 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:23:05.079139  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:23:05.087406  4445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:23:05.089172  4446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:23:05.092023  4356 server_base.cc:1061] running on GCE node
W20260709 12:23:05.092787  4448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:23:05.093634  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:23:05.093843  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:23:05.094058  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783599785094024 us; error 0 us; skew 500 ppm
I20260709 12:23:05.094635  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:23:05.098861  4356 webserver.cc:533] Webserver started at http://127.4.65.1:35223/ using document root <none> and password file <none>
I20260709 12:23:05.099416  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:23:05.099624  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:23:05.099937  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:23:05.101140  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-0-root/instance:
uuid: "498f39f6bf2844b3ae1bd842752f065f"
format_stamp: "Formatted at 2026-07-09 12:23:05 on dist-test-slave-l1c9"
I20260709 12:23:05.106755  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.003s
I20260709 12:23:05.111488  4453 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:05.112403  4356 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I20260709 12:23:05.112704  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-0-root
uuid: "498f39f6bf2844b3ae1bd842752f065f"
format_stamp: "Formatted at 2026-07-09 12:23:05 on dist-test-slave-l1c9"
I20260709 12:23:05.112993  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:23:05.127276  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:23:05.128572  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:23:05.131210  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:23:05.135727  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:23:05.135985  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:05.136241  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:23:05.136452  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20260709 12:23:05.241703  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.1:34043
I20260709 12:23:05.241770  4515 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.1:34043 every 8 connection(s)
I20260709 12:23:05.248631  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:23:05.258108  4520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:23:05.264412  4521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:23:05.272471  4523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:23:05.274883  4356 server_base.cc:1061] running on GCE node
I20260709 12:23:05.276087  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:23:05.276376  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:23:05.276611  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783599785276583 us; error 0 us; skew 500 ppm
I20260709 12:23:05.277230  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:23:05.278106  4516 heartbeater.cc:344] Connected to a master server at 127.4.65.62:40331
I20260709 12:23:05.278586  4516 heartbeater.cc:461] Registering TS with master...
I20260709 12:23:05.279731  4516 heartbeater.cc:507] Master 127.4.65.62:40331 requested a full tablet report, sending...
I20260709 12:23:05.282296  4356 webserver.cc:533] Webserver started at http://127.4.65.2:44571/ using document root <none> and password file <none>
I20260709 12:23:05.282861  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:23:05.283087  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:23:05.283217  4388 ts_manager.cc:194] Registered new tserver with Master: 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043)
I20260709 12:23:05.283377  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:23:05.284559  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-1-root/instance:
uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
format_stamp: "Formatted at 2026-07-09 12:23:05 on dist-test-slave-l1c9"
I20260709 12:23:05.286098  4388 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35234
I20260709 12:23:05.290378  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20260709 12:23:05.295104  4528 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:05.296020  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20260709 12:23:05.296293  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-1-root
uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
format_stamp: "Formatted at 2026-07-09 12:23:05 on dist-test-slave-l1c9"
I20260709 12:23:05.296613  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:23:05.321798  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:23:05.323021  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:23:05.325554  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:23:05.328392  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:23:05.328596  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:05.328893  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:23:05.329034  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:05.429987  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.2:38693
I20260709 12:23:05.430064  4590 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.2:38693 every 8 connection(s)
I20260709 12:23:05.438227  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:23:05.445887  4594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:23:05.448616  4595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:23:05.451088  4591 heartbeater.cc:344] Connected to a master server at 127.4.65.62:40331
I20260709 12:23:05.451606  4591 heartbeater.cc:461] Registering TS with master...
I20260709 12:23:05.452768  4591 heartbeater.cc:507] Master 127.4.65.62:40331 requested a full tablet report, sending...
I20260709 12:23:05.455787  4388 ts_manager.cc:194] Registered new tserver with Master: 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:05.458038  4388 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35246
W20260709 12:23:05.465124  4597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:23:05.465241  4356 server_base.cc:1061] running on GCE node
I20260709 12:23:05.466322  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:23:05.466507  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:23:05.466729  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783599785466684 us; error 0 us; skew 500 ppm
I20260709 12:23:05.467276  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:23:05.469883  4356 webserver.cc:533] Webserver started at http://127.4.65.3:39089/ using document root <none> and password file <none>
I20260709 12:23:05.470355  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:23:05.470539  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:23:05.470804  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:23:05.471781  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-2-root/instance:
uuid: "e3bbae884fe1434cbd0f42d01731773a"
format_stamp: "Formatted at 2026-07-09 12:23:05 on dist-test-slave-l1c9"
I20260709 12:23:05.476207  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.002s
I20260709 12:23:05.479544  4602 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:05.480336  4356 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:23:05.480703  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-2-root
uuid: "e3bbae884fe1434cbd0f42d01731773a"
format_stamp: "Formatted at 2026-07-09 12:23:05 on dist-test-slave-l1c9"
I20260709 12:23:05.481030  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599783304823-4356-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:23:05.499511  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:23:05.500778  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:23:05.503342  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:23:05.506198  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:23:05.506421  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:05.506667  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:23:05.506861  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:05.601727  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.3:35891
I20260709 12:23:05.601821  4665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.3:35891 every 8 connection(s)
I20260709 12:23:05.618844  4666 heartbeater.cc:344] Connected to a master server at 127.4.65.62:40331
I20260709 12:23:05.619207  4666 heartbeater.cc:461] Registering TS with master...
I20260709 12:23:05.619988  4666 heartbeater.cc:507] Master 127.4.65.62:40331 requested a full tablet report, sending...
I20260709 12:23:05.621942  4388 ts_manager.cc:194] Registered new tserver with Master: e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:05.622038  4356 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015213039s
I20260709 12:23:05.623426  4388 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35262
I20260709 12:23:05.683061  4388 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35276:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:23:05.686561  4388 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:23:05.795013  4629 tablet_service.cc:1511] Processing CreateTablet for tablet 27b95197718748aca01ae1e6379245ca (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:23:05.805747  4481 tablet_service.cc:1511] Processing CreateTablet for tablet 93d1398e83ae46109eb2163858c93214 (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:23:05.804296  4628 tablet_service.cc:1511] Processing CreateTablet for tablet 9bba194fd0b34047badbff2aea2f9414 (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:23:05.800860  4631 tablet_service.cc:1511] Processing CreateTablet for tablet 93d1398e83ae46109eb2163858c93214 (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:23:05.809254  4481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d1398e83ae46109eb2163858c93214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.809057  4479 tablet_service.cc:1511] Processing CreateTablet for tablet 27b95197718748aca01ae1e6379245ca (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:23:05.810084  4629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b95197718748aca01ae1e6379245ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.816857  4630 tablet_service.cc:1511] Processing CreateTablet for tablet f6c5711d5f1e451688f4fe68db57350e (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:23:05.820150  4627 tablet_service.cc:1511] Processing CreateTablet for tablet c31fd08755e44186a93833dcf59e8ff1 (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:23:05.822046  4477 tablet_service.cc:1511] Processing CreateTablet for tablet c31fd08755e44186a93833dcf59e8ff1 (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:23:05.818364  4478 tablet_service.cc:1511] Processing CreateTablet for tablet 9bba194fd0b34047badbff2aea2f9414 (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:23:05.818192  4480 tablet_service.cc:1511] Processing CreateTablet for tablet f6c5711d5f1e451688f4fe68db57350e (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:23:05.822227  4628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bba194fd0b34047badbff2aea2f9414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.827023  4479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b95197718748aca01ae1e6379245ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.827126  4476 tablet_service.cc:1511] Processing CreateTablet for tablet f31c9f2f020a45e6b1077bb51b6dafde (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:23:05.827701  4477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31fd08755e44186a93833dcf59e8ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.828545  4476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31c9f2f020a45e6b1077bb51b6dafde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.829926  4478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bba194fd0b34047badbff2aea2f9414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.831115  4627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31fd08755e44186a93833dcf59e8ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.831409  4480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6c5711d5f1e451688f4fe68db57350e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.832830  4631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d1398e83ae46109eb2163858c93214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.833896  4630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6c5711d5f1e451688f4fe68db57350e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.844573  4555 tablet_service.cc:1511] Processing CreateTablet for tablet f6c5711d5f1e451688f4fe68db57350e (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:23:05.845356  4556 tablet_service.cc:1511] Processing CreateTablet for tablet 93d1398e83ae46109eb2163858c93214 (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:23:05.848174  4554 tablet_service.cc:1511] Processing CreateTablet for tablet 27b95197718748aca01ae1e6379245ca (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:23:05.854595  4556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d1398e83ae46109eb2163858c93214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.866922  4553 tablet_service.cc:1511] Processing CreateTablet for tablet 9bba194fd0b34047badbff2aea2f9414 (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:23:05.872699  4554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b95197718748aca01ae1e6379245ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.874085  4626 tablet_service.cc:1511] Processing CreateTablet for tablet f31c9f2f020a45e6b1077bb51b6dafde (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:23:05.875727  4626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31c9f2f020a45e6b1077bb51b6dafde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.885964  4555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6c5711d5f1e451688f4fe68db57350e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.885129  4475 tablet_service.cc:1511] Processing CreateTablet for tablet de101290db964f41b3fb73588fd5964f (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:23:05.891016  4475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de101290db964f41b3fb73588fd5964f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.886813  4552 tablet_service.cc:1511] Processing CreateTablet for tablet c31fd08755e44186a93833dcf59e8ff1 (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:23:05.894857  4553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bba194fd0b34047badbff2aea2f9414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.897532  4551 tablet_service.cc:1511] Processing CreateTablet for tablet f31c9f2f020a45e6b1077bb51b6dafde (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:23:05.903666  4625 tablet_service.cc:1511] Processing CreateTablet for tablet de101290db964f41b3fb73588fd5964f (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:23:05.905282  4625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de101290db964f41b3fb73588fd5964f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.906332  4552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c31fd08755e44186a93833dcf59e8ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.909135  4550 tablet_service.cc:1511] Processing CreateTablet for tablet de101290db964f41b3fb73588fd5964f (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:23:05.910780  4551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f31c9f2f020a45e6b1077bb51b6dafde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.916678  4550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de101290db964f41b3fb73588fd5964f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.923148  4474 tablet_service.cc:1511] Processing CreateTablet for tablet a987ec62f8c04b9aad527a2b4c8c00cc (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:23:05.924458  4474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a987ec62f8c04b9aad527a2b4c8c00cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:23:05.938971  4435 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:23:05.937639  4549 tablet_service.cc:1511] Processing CreateTablet for tablet a987ec62f8c04b9aad527a2b4c8c00cc (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:23:05.950482  4549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a987ec62f8c04b9aad527a2b4c8c00cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:05.982211  4624 tablet_service.cc:1511] Processing CreateTablet for tablet a987ec62f8c04b9aad527a2b4c8c00cc (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:23:05.983990  4624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a987ec62f8c04b9aad527a2b4c8c00cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:06.014113  4473 tablet_service.cc:1511] Processing CreateTablet for tablet 7b3b696a9e3f4d479648b829ef3d167e (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:23:06.025002  4473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3b696a9e3f4d479648b829ef3d167e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:06.026453  4548 tablet_service.cc:1511] Processing CreateTablet for tablet 7b3b696a9e3f4d479648b829ef3d167e (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:23:06.032373  4548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3b696a9e3f4d479648b829ef3d167e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:06.065748  4684 tablet_bootstrap.cc:492] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f: Bootstrap starting.
I20260709 12:23:06.082666  4685 tablet_bootstrap.cc:492] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a: Bootstrap starting.
I20260709 12:23:06.103369  4626 tablet_service.cc:1511] Processing CreateTablet for tablet 7b3b696a9e3f4d479648b829ef3d167e (DEFAULT_TABLE table=test-workload [id=22d37edae2e34d8dbc0235a9046423e1]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:23:06.112247  4626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3b696a9e3f4d479648b829ef3d167e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:06.114399  4684 tablet_bootstrap.cc:654] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.126202  4686 tablet_bootstrap.cc:492] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4: Bootstrap starting.
I20260709 12:23:06.133705  4684 tablet_bootstrap.cc:492] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f: No bootstrap required, opened a new log
I20260709 12:23:06.134627  4685 tablet_bootstrap.cc:654] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.134188  4684 ts_tablet_manager.cc:1403] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f: Time spent bootstrapping tablet: real 0.084s	user 0.008s	sys 0.007s
I20260709 12:23:06.137956  4684 raft_consensus.cc:359] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.138736  4684 raft_consensus.cc:385] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.139024  4684 raft_consensus.cc:740] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.139529  4686 tablet_bootstrap.cc:654] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.139726  4684 consensus_queue.cc:260] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.148561  4685 tablet_bootstrap.cc:492] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a: No bootstrap required, opened a new log
I20260709 12:23:06.149261  4685 ts_tablet_manager.cc:1403] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a: Time spent bootstrapping tablet: real 0.083s	user 0.003s	sys 0.017s
I20260709 12:23:06.152379  4686 tablet_bootstrap.cc:492] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4: No bootstrap required, opened a new log
I20260709 12:23:06.152189  4685 raft_consensus.cc:359] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.152848  4686 ts_tablet_manager.cc:1403] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent bootstrapping tablet: real 0.060s	user 0.004s	sys 0.016s
I20260709 12:23:06.152962  4685 raft_consensus.cc:385] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.153263  4685 raft_consensus.cc:740] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.154022  4685 consensus_queue.cc:260] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.155659  4686 raft_consensus.cc:359] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.156203  4516 heartbeater.cc:499] Master 127.4.65.62:40331 was elected leader, sending a full tablet report...
I20260709 12:23:06.156366  4686 raft_consensus.cc:385] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.156679  4684 ts_tablet_manager.cc:1434] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f: Time spent starting tablet: real 0.022s	user 0.017s	sys 0.004s
I20260709 12:23:06.156664  4686 raft_consensus.cc:740] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.157764  4684 tablet_bootstrap.cc:492] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f: Bootstrap starting.
I20260709 12:23:06.157568  4686 consensus_queue.cc:260] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.164098  4684 tablet_bootstrap.cc:654] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.176542  4684 tablet_bootstrap.cc:492] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f: No bootstrap required, opened a new log
I20260709 12:23:06.176985  4684 ts_tablet_manager.cc:1403] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f: Time spent bootstrapping tablet: real 0.019s	user 0.010s	sys 0.007s
I20260709 12:23:06.178229  4666 heartbeater.cc:499] Master 127.4.65.62:40331 was elected leader, sending a full tablet report...
I20260709 12:23:06.179775  4685 ts_tablet_manager.cc:1434] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a: Time spent starting tablet: real 0.030s	user 0.005s	sys 0.002s
I20260709 12:23:06.181362  4685 tablet_bootstrap.cc:492] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a: Bootstrap starting.
I20260709 12:23:06.187870  4685 tablet_bootstrap.cc:654] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.179751  4684 raft_consensus.cc:359] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.188612  4684 raft_consensus.cc:385] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.188913  4684 raft_consensus.cc:740] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.189620  4684 consensus_queue.cc:260] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.191607  4684 ts_tablet_manager.cc:1434] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I20260709 12:23:06.192524  4684 tablet_bootstrap.cc:492] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f: Bootstrap starting.
I20260709 12:23:06.194530  4686 ts_tablet_manager.cc:1434] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent starting tablet: real 0.041s	user 0.018s	sys 0.023s
I20260709 12:23:06.194644  4591 heartbeater.cc:499] Master 127.4.65.62:40331 was elected leader, sending a full tablet report...
I20260709 12:23:06.195446  4686 tablet_bootstrap.cc:492] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4: Bootstrap starting.
I20260709 12:23:06.201519  4684 tablet_bootstrap.cc:654] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.202028  4686 tablet_bootstrap.cc:654] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.211270  4685 tablet_bootstrap.cc:492] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a: No bootstrap required, opened a new log
I20260709 12:23:06.211731  4685 ts_tablet_manager.cc:1403] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a: Time spent bootstrapping tablet: real 0.031s	user 0.004s	sys 0.010s
I20260709 12:23:06.214403  4685 raft_consensus.cc:359] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.215082  4685 raft_consensus.cc:385] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.215684  4685 raft_consensus.cc:740] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.216464  4685 consensus_queue.cc:260] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.217752  4686 tablet_bootstrap.cc:492] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4: No bootstrap required, opened a new log
I20260709 12:23:06.218195  4686 ts_tablet_manager.cc:1403] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent bootstrapping tablet: real 0.023s	user 0.008s	sys 0.007s
I20260709 12:23:06.218703  4684 tablet_bootstrap.cc:492] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f: No bootstrap required, opened a new log
I20260709 12:23:06.218753  4685 ts_tablet_manager.cc:1434] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.004s
I20260709 12:23:06.219280  4684 ts_tablet_manager.cc:1403] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f: Time spent bootstrapping tablet: real 0.027s	user 0.008s	sys 0.002s
I20260709 12:23:06.219748  4685 tablet_bootstrap.cc:492] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a: Bootstrap starting.
I20260709 12:23:06.220911  4686 raft_consensus.cc:359] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.222184  4684 raft_consensus.cc:359] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.222967  4684 raft_consensus.cc:385] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.223253  4684 raft_consensus.cc:740] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.221593  4686 raft_consensus.cc:385] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.224277  4686 raft_consensus.cc:740] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.224000  4684 consensus_queue.cc:260] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.225039  4686 consensus_queue.cc:260] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.226117  4684 ts_tablet_manager.cc:1434] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 12:23:06.226454  4685 tablet_bootstrap.cc:654] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.227061  4684 tablet_bootstrap.cc:492] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f: Bootstrap starting.
I20260709 12:23:06.234330  4684 tablet_bootstrap.cc:654] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.235743  4686 ts_tablet_manager.cc:1434] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.002s
I20260709 12:23:06.237026  4686 tablet_bootstrap.cc:492] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4: Bootstrap starting.
I20260709 12:23:06.246059  4686 tablet_bootstrap.cc:654] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4: Neither blocks nor log segments found. Creating new log.
W20260709 12:23:06.256546  4517 tablet.cc:2406] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:23:06.257423  4686 tablet_bootstrap.cc:492] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4: No bootstrap required, opened a new log
I20260709 12:23:06.257844  4686 ts_tablet_manager.cc:1403] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent bootstrapping tablet: real 0.021s	user 0.012s	sys 0.000s
I20260709 12:23:06.260637  4686 raft_consensus.cc:359] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.261323  4686 raft_consensus.cc:385] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.261607  4686 raft_consensus.cc:740] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.262274  4686 consensus_queue.cc:260] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.271297  4686 ts_tablet_manager.cc:1434] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.002s
I20260709 12:23:06.272212  4686 tablet_bootstrap.cc:492] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4: Bootstrap starting.
I20260709 12:23:06.285277  4686 tablet_bootstrap.cc:654] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.288403  4684 tablet_bootstrap.cc:492] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f: No bootstrap required, opened a new log
I20260709 12:23:06.288885  4684 ts_tablet_manager.cc:1403] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f: Time spent bootstrapping tablet: real 0.062s	user 0.025s	sys 0.034s
I20260709 12:23:06.289989  4685 tablet_bootstrap.cc:492] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a: No bootstrap required, opened a new log
I20260709 12:23:06.290432  4685 ts_tablet_manager.cc:1403] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a: Time spent bootstrapping tablet: real 0.071s	user 0.029s	sys 0.019s
I20260709 12:23:06.292130  4684 raft_consensus.cc:359] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.292897  4684 raft_consensus.cc:385] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.293184  4684 raft_consensus.cc:740] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.293231  4685 raft_consensus.cc:359] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.293933  4685 raft_consensus.cc:385] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.294333  4685 raft_consensus.cc:740] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.293944  4684 consensus_queue.cc:260] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.298698  4684 ts_tablet_manager.cc:1434] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.001s
I20260709 12:23:06.299721  4684 tablet_bootstrap.cc:492] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f: Bootstrap starting.
I20260709 12:23:06.295351  4685 consensus_queue.cc:260] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.304858  4691 raft_consensus.cc:493] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.305320  4691 raft_consensus.cc:515] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.306398  4684 tablet_bootstrap.cc:654] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.301504  4686 tablet_bootstrap.cc:492] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4: No bootstrap required, opened a new log
I20260709 12:23:06.317413  4686 ts_tablet_manager.cc:1403] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent bootstrapping tablet: real 0.045s	user 0.014s	sys 0.007s
I20260709 12:23:06.318770  4684 tablet_bootstrap.cc:492] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f: No bootstrap required, opened a new log
I20260709 12:23:06.319413  4684 ts_tablet_manager.cc:1403] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f: Time spent bootstrapping tablet: real 0.020s	user 0.007s	sys 0.009s
I20260709 12:23:06.313060  4691 leader_election.cc:290] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:06.322583  4685 ts_tablet_manager.cc:1434] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.000s
I20260709 12:23:06.323607  4685 tablet_bootstrap.cc:492] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a: Bootstrap starting.
I20260709 12:23:06.323535  4684 raft_consensus.cc:359] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.324455  4684 raft_consensus.cc:385] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.324767  4684 raft_consensus.cc:740] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.325311  4686 raft_consensus.cc:359] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.326086  4686 raft_consensus.cc:385] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.326421  4686 raft_consensus.cc:740] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.325798  4684 consensus_queue.cc:260] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.327190  4686 consensus_queue.cc:260] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.328802  4684 ts_tablet_manager.cc:1434] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.009s
I20260709 12:23:06.329869  4684 tablet_bootstrap.cc:492] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f: Bootstrap starting.
I20260709 12:23:06.334409  4685 tablet_bootstrap.cc:654] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.336862  4686 ts_tablet_manager.cc:1434] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.001s
I20260709 12:23:06.337450  4684 tablet_bootstrap.cc:654] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.338081  4686 tablet_bootstrap.cc:492] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4: Bootstrap starting.
I20260709 12:23:06.338121  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b95197718748aca01ae1e6379245ca" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f" is_pre_election: true
I20260709 12:23:06.339462  4491 raft_consensus.cc:2468] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 0.
I20260709 12:23:06.349989  4603 leader_election.cc:304] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:06.351025  4691 raft_consensus.cc:2804] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:06.351418  4691 raft_consensus.cc:493] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.352257  4691 raft_consensus.cc:3060] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.360462  4689 raft_consensus.cc:493] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.359543  4691 raft_consensus.cc:515] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.360994  4689 raft_consensus.cc:515] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.364015  4689 leader_election.cc:290] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:06.364883  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b95197718748aca01ae1e6379245ca" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:06.373519  4491 raft_consensus.cc:3060] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.373874  4691 leader_election.cc:290] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:06.386011  4686 tablet_bootstrap.cc:654] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.403400  4491 raft_consensus.cc:2468] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 1.
I20260709 12:23:06.408298  4684 tablet_bootstrap.cc:492] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f: No bootstrap required, opened a new log
I20260709 12:23:06.408802  4684 ts_tablet_manager.cc:1403] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f: Time spent bootstrapping tablet: real 0.079s	user 0.019s	sys 0.002s
I20260709 12:23:06.409869  4685 tablet_bootstrap.cc:492] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a: No bootstrap required, opened a new log
I20260709 12:23:06.410451  4685 ts_tablet_manager.cc:1403] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a: Time spent bootstrapping tablet: real 0.087s	user 0.018s	sys 0.005s
I20260709 12:23:06.420432  4689 raft_consensus.cc:493] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.421263  4689 raft_consensus.cc:515] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.423295  4689 leader_election.cc:290] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:06.424108  4684 raft_consensus.cc:359] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:06.424930  4684 raft_consensus.cc:385] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.425284  4684 raft_consensus.cc:740] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.427029  4566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b95197718748aca01ae1e6379245ca" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" is_pre_election: true
W20260709 12:23:06.432668  4605 leader_election.cc:343] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:06.432665  4566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b95197718748aca01ae1e6379245ca" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:06.432862  4684 consensus_queue.cc:260] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
W20260709 12:23:06.435314  4605 leader_election.cc:343] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:06.437070  4684 ts_tablet_manager.cc:1434] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.001s
I20260709 12:23:06.438879  4684 tablet_bootstrap.cc:492] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f: Bootstrap starting.
I20260709 12:23:06.437551  4685 raft_consensus.cc:359] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.428804  4686 tablet_bootstrap.cc:492] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4: No bootstrap required, opened a new log
I20260709 12:23:06.441953  4685 raft_consensus.cc:385] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.442600  4685 raft_consensus.cc:740] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.443364  4685 consensus_queue.cc:260] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.446089  4685 ts_tablet_manager.cc:1434] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a: Time spent starting tablet: real 0.035s	user 0.002s	sys 0.006s
I20260709 12:23:06.446193  4566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31c9f2f020a45e6b1077bb51b6dafde" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" is_pre_election: true
I20260709 12:23:06.444947  4686 ts_tablet_manager.cc:1403] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent bootstrapping tablet: real 0.107s	user 0.014s	sys 0.004s
I20260709 12:23:06.446830  4566 raft_consensus.cc:2468] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 0.
I20260709 12:23:06.450014  4692 raft_consensus.cc:493] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.450810  4692 raft_consensus.cc:515] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.449126  4565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a987ec62f8c04b9aad527a2b4c8c00cc" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" is_pre_election: true
I20260709 12:23:06.454244  4456 leader_election.cc:304] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: 
I20260709 12:23:06.455677  4689 raft_consensus.cc:2804] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:06.456125  4689 raft_consensus.cc:493] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.456559  4689 raft_consensus.cc:3060] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:23:06.464627  4456 leader_election.cc:343] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:06.466306  4603 leader_election.cc:304] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 5345e5bcd1f14d5d88bac1e08965a1c4
I20260709 12:23:06.473954  4691 raft_consensus.cc:2804] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:06.478046  4692 leader_election.cc:290] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:06.479840  4685 tablet_bootstrap.cc:492] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a: Bootstrap starting.
I20260709 12:23:06.479698  4686 raft_consensus.cc:359] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.482103  4686 raft_consensus.cc:385] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.482632  4686 raft_consensus.cc:740] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.483698  4691 raft_consensus.cc:697] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Becoming Leader. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Running, Role: LEADER
I20260709 12:23:06.470916  4689 raft_consensus.cc:515] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.484535  4691 consensus_queue.cc:237] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.487383  4689 leader_election.cc:290] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 election: Requested vote from peers 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:06.483536  4686 consensus_queue.cc:260] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.494508  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31c9f2f020a45e6b1077bb51b6dafde" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a" is_pre_election: true
I20260709 12:23:06.496378  4641 raft_consensus.cc:2468] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 0.
I20260709 12:23:06.498250  4640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a987ec62f8c04b9aad527a2b4c8c00cc" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a" is_pre_election: true
I20260709 12:23:06.501212  4640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31c9f2f020a45e6b1077bb51b6dafde" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:06.501546  4684 tablet_bootstrap.cc:654] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.501784  4640 raft_consensus.cc:3060] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.502851  4713 raft_consensus.cc:493] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:06.503700  4456 leader_election.cc:343] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:06.504142  4456 leader_election.cc:304] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f; no voters: 5345e5bcd1f14d5d88bac1e08965a1c4, e3bbae884fe1434cbd0f42d01731773a
I20260709 12:23:06.506500  4689 raft_consensus.cc:2749] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:23:06.504760  4713 raft_consensus.cc:515] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.518522  4713 leader_election.cc:290] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:06.518476  4565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31c9f2f020a45e6b1077bb51b6dafde" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:06.519524  4565 raft_consensus.cc:3060] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.520032  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f" is_pre_election: true
I20260709 12:23:06.521874  4491 raft_consensus.cc:2468] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 0.
I20260709 12:23:06.523102  4603 leader_election.cc:304] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:06.524881  4566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" is_pre_election: true
I20260709 12:23:06.524212  4685 tablet_bootstrap.cc:654] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.523825  4686 ts_tablet_manager.cc:1434] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent starting tablet: real 0.077s	user 0.008s	sys 0.000s
I20260709 12:23:06.525530  4566 raft_consensus.cc:2468] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 0.
I20260709 12:23:06.526600  4691 raft_consensus.cc:2804] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:06.529156  4691 raft_consensus.cc:493] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.531898  4691 raft_consensus.cc:3060] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.532825  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f" is_pre_election: true
I20260709 12:23:06.533226  4491 raft_consensus.cc:2468] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 0.
I20260709 12:23:06.534106  4686 tablet_bootstrap.cc:492] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4: Bootstrap starting.
I20260709 12:23:06.536186  4529 leader_election.cc:304] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: 
I20260709 12:23:06.539710  4692 raft_consensus.cc:2804] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:06.540490  4692 raft_consensus.cc:493] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.540951  4692 raft_consensus.cc:3060] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.542749  4640 raft_consensus.cc:2468] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 1.
I20260709 12:23:06.543859  4565 raft_consensus.cc:2468] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 1.
I20260709 12:23:06.543982  4456 leader_election.cc:304] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:06.549787  4689 raft_consensus.cc:2804] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:06.558350  4686 tablet_bootstrap.cc:654] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.559746  4692 raft_consensus.cc:515] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.566097  4684 tablet_bootstrap.cc:492] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f: No bootstrap required, opened a new log
I20260709 12:23:06.566578  4684 ts_tablet_manager.cc:1403] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f: Time spent bootstrapping tablet: real 0.128s	user 0.008s	sys 0.006s
I20260709 12:23:06.566463  4640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a" is_pre_election: true
I20260709 12:23:06.567492  4691 raft_consensus.cc:515] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.570096  4689 raft_consensus.cc:697] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 1 LEADER]: Becoming Leader. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:06.571570  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:06.572433  4491 raft_consensus.cc:3060] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.572871  4640 raft_consensus.cc:2393] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in current term 1: Already voted for candidate e3bbae884fe1434cbd0f42d01731773a in this term.
I20260709 12:23:06.578006  4691 leader_election.cc:290] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:06.580281  4692 leader_election.cc:290] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:06.581903  4490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:06.580560  4720 raft_consensus.cc:493] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.585162  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:06.580860  4689 consensus_queue.cc:237] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.581208  4684 raft_consensus.cc:359] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.585856  4720 raft_consensus.cc:515] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.590824  4565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:06.593632  4684 raft_consensus.cc:385] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.594215  4565 raft_consensus.cc:2393] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e3bbae884fe1434cbd0f42d01731773a in current term 1: Already voted for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in this term.
I20260709 12:23:06.593562  4692 raft_consensus.cc:493] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.594857  4684 raft_consensus.cc:740] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.596705  4692 raft_consensus.cc:515] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.598436  4605 leader_election.cc:304] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3bbae884fe1434cbd0f42d01731773a; no voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4
I20260709 12:23:06.606673  4691 raft_consensus.cc:2749] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:23:06.607714  4491 raft_consensus.cc:2468] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 1.
I20260709 12:23:06.606983  4565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de101290db964f41b3fb73588fd5964f" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" is_pre_election: true
I20260709 12:23:06.613299  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de101290db964f41b3fb73588fd5964f" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a" is_pre_election: true
I20260709 12:23:06.614013  4641 raft_consensus.cc:2468] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 0.
I20260709 12:23:06.608122  4720 leader_election.cc:290] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:06.614821  4685 tablet_bootstrap.cc:492] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a: No bootstrap required, opened a new log
I20260709 12:23:06.615316  4565 raft_consensus.cc:2468] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 0.
I20260709 12:23:06.607848  4692 leader_election.cc:290] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:06.616756  4529 leader_election.cc:304] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: e3bbae884fe1434cbd0f42d01731773a
I20260709 12:23:06.617036  4692 raft_consensus.cc:493] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.614916  4640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a" is_pre_election: true
I20260709 12:23:06.617813  4685 ts_tablet_manager.cc:1403] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a: Time spent bootstrapping tablet: real 0.138s	user 0.016s	sys 0.016s
I20260709 12:23:06.615741  4456 leader_election.cc:304] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:06.654186  4689 raft_consensus.cc:2804] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:06.654618  4689 raft_consensus.cc:493] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.654696  4720 raft_consensus.cc:493] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:06.633823  4531 leader_election.cc:343] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:06.655000  4689 raft_consensus.cc:3060] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.655190  4720 raft_consensus.cc:515] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:06.656203  4722 raft_consensus.cc:2804] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:06.656481  4725 raft_consensus.cc:493] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.598709  4684 consensus_queue.cc:260] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.658995  4684 ts_tablet_manager.cc:1434] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f: Time spent starting tablet: real 0.092s	user 0.010s	sys 0.000s
I20260709 12:23:06.659763  4684 tablet_bootstrap.cc:492] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f: Bootstrap starting.
I20260709 12:23:06.660748  4689 raft_consensus.cc:515] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.662659  4689 leader_election.cc:290] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 election: Requested vote from peers 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:06.663192  4685 raft_consensus.cc:359] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.663717  4685 raft_consensus.cc:385] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.663978  4685 raft_consensus.cc:740] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.619928  4692 raft_consensus.cc:515] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.664698  4685 consensus_queue.cc:260] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.665320  4684 tablet_bootstrap.cc:654] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.666530  4692 leader_election.cc:290] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:06.666985  4720 leader_election.cc:290] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:06.609256  4388 catalog_manager.cc:5697] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a reported cstate change: term changed from 0 to 1, leader changed from <none> to e3bbae884fe1434cbd0f42d01731773a (127.4.65.3). New cstate: current_term: 1 leader_uuid: "e3bbae884fe1434cbd0f42d01731773a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:06.668627  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f" is_pre_election: true
I20260709 12:23:06.669837  4685 ts_tablet_manager.cc:1434] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.001s
I20260709 12:23:06.672852  4685 tablet_bootstrap.cc:492] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a: Bootstrap starting.
I20260709 12:23:06.672618  4490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de101290db964f41b3fb73588fd5964f" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f" is_pre_election: true
I20260709 12:23:06.670542  4640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de101290db964f41b3fb73588fd5964f" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a" is_pre_election: true
I20260709 12:23:06.675561  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d1398e83ae46109eb2163858c93214" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a" is_pre_election: true
I20260709 12:23:06.675746  4640 raft_consensus.cc:2468] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 0.
I20260709 12:23:06.670454  4387 catalog_manager.cc:5697] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f reported cstate change: term changed from 0 to 1, leader changed from <none> to 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "498f39f6bf2844b3ae1bd842752f065f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:06.677209  4531 leader_election.cc:304] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5345e5bcd1f14d5d88bac1e08965a1c4, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:06.675297  4490 raft_consensus.cc:2393] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in current term 1: Already voted for candidate 498f39f6bf2844b3ae1bd842752f065f in this term.
I20260709 12:23:06.678354  4692 raft_consensus.cc:2804] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:06.679520  4692 raft_consensus.cc:493] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.680248  4692 raft_consensus.cc:3060] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.679286  4565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d1398e83ae46109eb2163858c93214" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" is_pre_election: true
I20260709 12:23:06.656947  4725 raft_consensus.cc:515] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.682327  4565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de101290db964f41b3fb73588fd5964f" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:06.684266  4640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de101290db964f41b3fb73588fd5964f" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:06.684810  4640 raft_consensus.cc:3060] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.685436  4686 tablet_bootstrap.cc:492] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4: No bootstrap required, opened a new log
I20260709 12:23:06.685932  4686 ts_tablet_manager.cc:1403] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent bootstrapping tablet: real 0.155s	user 0.017s	sys 0.001s
I20260709 12:23:06.693320  4640 raft_consensus.cc:2468] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 1.
I20260709 12:23:06.693617  4686 raft_consensus.cc:359] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
W20260709 12:23:06.694711  4529 leader_election.cc:343] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:06.695617  4686 raft_consensus.cc:385] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.695596  4529 leader_election.cc:304] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a
I20260709 12:23:06.696904  4686 raft_consensus.cc:740] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.699216  4686 consensus_queue.cc:260] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.706317  4722 raft_consensus.cc:697] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
W20260709 12:23:06.696540  4456 leader_election.cc:343] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:23:06.713126  4456 leader_election.cc:343] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:06.713733  4456 leader_election.cc:304] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f; no voters: 5345e5bcd1f14d5d88bac1e08965a1c4, e3bbae884fe1434cbd0f42d01731773a
I20260709 12:23:06.713207  4722 consensus_queue.cc:237] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.714401  4720 raft_consensus.cc:2749] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:23:06.706686  4726 raft_consensus.cc:2749] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:23:06.707816  4725 leader_election.cc:290] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:06.709270  4640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bba194fd0b34047badbff2aea2f9414" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a" is_pre_election: true
I20260709 12:23:06.715126  4685 tablet_bootstrap.cc:654] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.716224  4456 leader_election.cc:304] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 
W20260709 12:23:06.716521  4531 leader_election.cc:343] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:06.717175  4720 raft_consensus.cc:2804] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:06.717643  4720 raft_consensus.cc:697] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 1 LEADER]: Becoming Leader. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:06.717690  4490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bba194fd0b34047badbff2aea2f9414" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f" is_pre_election: true
I20260709 12:23:06.718379  4490 raft_consensus.cc:2468] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 0.
I20260709 12:23:06.718777  4684 tablet_bootstrap.cc:492] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f: No bootstrap required, opened a new log
I20260709 12:23:06.718508  4720 consensus_queue.cc:237] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.719189  4684 ts_tablet_manager.cc:1403] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f: Time spent bootstrapping tablet: real 0.060s	user 0.009s	sys 0.010s
I20260709 12:23:06.720129  4529 leader_election.cc:304] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: e3bbae884fe1434cbd0f42d01731773a
I20260709 12:23:06.721343  4726 raft_consensus.cc:2804] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:06.721822  4726 raft_consensus.cc:493] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.721993  4684 raft_consensus.cc:359] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.722524  4726 raft_consensus.cc:3060] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.722884  4684 raft_consensus.cc:385] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.723261  4684 raft_consensus.cc:740] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.729763  4686 ts_tablet_manager.cc:1434] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent starting tablet: real 0.044s	user 0.008s	sys 0.000s
I20260709 12:23:06.730796  4686 tablet_bootstrap.cc:492] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4: Bootstrap starting.
I20260709 12:23:06.731649  4726 raft_consensus.cc:515] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.707151  4692 raft_consensus.cc:515] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.723958  4684 consensus_queue.cc:260] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.744522  4684 ts_tablet_manager.cc:1434] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f: Time spent starting tablet: real 0.025s	user 0.009s	sys 0.001s
I20260709 12:23:06.745401  4684 tablet_bootstrap.cc:492] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f: Bootstrap starting.
I20260709 12:23:06.746205  4565 raft_consensus.cc:2393] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 498f39f6bf2844b3ae1bd842752f065f in current term 1: Already voted for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in this term.
I20260709 12:23:06.750232  4726 leader_election.cc:290] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:06.750381  4490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bba194fd0b34047badbff2aea2f9414" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:06.750952  4490 raft_consensus.cc:3060] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.751422  4640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bba194fd0b34047badbff2aea2f9414" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
W20260709 12:23:06.759191  4531 leader_election.cc:343] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:06.761261  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de101290db964f41b3fb73588fd5964f" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:06.762030  4640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de101290db964f41b3fb73588fd5964f" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:06.763478  4640 raft_consensus.cc:2393] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in current term 1: Already voted for candidate 498f39f6bf2844b3ae1bd842752f065f in this term.
I20260709 12:23:06.758741  4685 tablet_bootstrap.cc:492] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a: No bootstrap required, opened a new log
I20260709 12:23:06.761118  4692 leader_election.cc:290] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:06.765654  4529 leader_election.cc:304] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a
I20260709 12:23:06.767297  4726 raft_consensus.cc:2749] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:23:06.766815  4685 ts_tablet_manager.cc:1403] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a: Time spent bootstrapping tablet: real 0.094s	user 0.011s	sys 0.016s
I20260709 12:23:06.770021  4684 tablet_bootstrap.cc:654] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.770697  4685 raft_consensus.cc:359] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.773427  4685 raft_consensus.cc:385] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.773568  4686 tablet_bootstrap.cc:654] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.773830  4685 raft_consensus.cc:740] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.774683  4685 consensus_queue.cc:260] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.776747  4490 raft_consensus.cc:2468] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 1.
I20260709 12:23:06.781401  4529 leader_election.cc:304] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: e3bbae884fe1434cbd0f42d01731773a
I20260709 12:23:06.782558  4692 raft_consensus.cc:2804] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:06.782857  4684 tablet_bootstrap.cc:492] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f: No bootstrap required, opened a new log
I20260709 12:23:06.783391  4684 ts_tablet_manager.cc:1403] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f: Time spent bootstrapping tablet: real 0.038s	user 0.009s	sys 0.006s
I20260709 12:23:06.783347  4692 raft_consensus.cc:697] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:06.779791  4388 catalog_manager.cc:5697] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2). New cstate: current_term: 1 leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:06.780107  4685 ts_tablet_manager.cc:1434] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:23:06.789405  4685 tablet_bootstrap.cc:492] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a: Bootstrap starting.
I20260709 12:23:06.786584  4684 raft_consensus.cc:359] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.794516  4684 raft_consensus.cc:385] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.794796  4684 raft_consensus.cc:740] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.787756  4692 consensus_queue.cc:237] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.797660  4686 tablet_bootstrap.cc:492] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4: No bootstrap required, opened a new log
I20260709 12:23:06.798192  4686 ts_tablet_manager.cc:1403] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent bootstrapping tablet: real 0.068s	user 0.007s	sys 0.007s
I20260709 12:23:06.798250  4386 catalog_manager.cc:5697] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f reported cstate change: term changed from 0 to 1, leader changed from <none> to 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "498f39f6bf2844b3ae1bd842752f065f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:06.801893  4684 consensus_queue.cc:260] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.803388  4686 raft_consensus.cc:359] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.804075  4686 raft_consensus.cc:385] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.804414  4686 raft_consensus.cc:740] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.803864  4684 ts_tablet_manager.cc:1434] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f: Time spent starting tablet: real 0.020s	user 0.001s	sys 0.004s
I20260709 12:23:06.806565  4686 consensus_queue.cc:260] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.809063  4686 ts_tablet_manager.cc:1434] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.001s
I20260709 12:23:06.810025  4686 tablet_bootstrap.cc:492] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4: Bootstrap starting.
I20260709 12:23:06.813875  4685 tablet_bootstrap.cc:654] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.816637  4686 tablet_bootstrap.cc:654] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.824510  4685 tablet_bootstrap.cc:492] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a: No bootstrap required, opened a new log
I20260709 12:23:06.825057  4685 ts_tablet_manager.cc:1403] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a: Time spent bootstrapping tablet: real 0.036s	user 0.007s	sys 0.010s
I20260709 12:23:06.826856  4386 catalog_manager.cc:5697] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2). New cstate: current_term: 1 leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:06.828110  4685 raft_consensus.cc:359] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.828898  4685 raft_consensus.cc:385] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.829267  4685 raft_consensus.cc:740] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.830056  4685 consensus_queue.cc:260] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.833128  4685 ts_tablet_manager.cc:1434] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:23:06.834146  4685 tablet_bootstrap.cc:492] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a: Bootstrap starting.
I20260709 12:23:06.840651  4685 tablet_bootstrap.cc:654] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.852162  4686 tablet_bootstrap.cc:492] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4: No bootstrap required, opened a new log
I20260709 12:23:06.852592  4685 tablet_bootstrap.cc:492] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a: No bootstrap required, opened a new log
I20260709 12:23:06.852646  4686 ts_tablet_manager.cc:1403] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent bootstrapping tablet: real 0.043s	user 0.017s	sys 0.018s
I20260709 12:23:06.853152  4685 ts_tablet_manager.cc:1403] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a: Time spent bootstrapping tablet: real 0.019s	user 0.010s	sys 0.006s
I20260709 12:23:06.855400  4686 raft_consensus.cc:359] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:06.855640  4685 raft_consensus.cc:359] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:06.856086  4686 raft_consensus.cc:385] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.856310  4685 raft_consensus.cc:385] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.856439  4686 raft_consensus.cc:740] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.856693  4685 raft_consensus.cc:740] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.857156  4686 consensus_queue.cc:260] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:06.857403  4685 consensus_queue.cc:260] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:06.859045  4686 ts_tablet_manager.cc:1434] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:23:06.859304  4685 ts_tablet_manager.cc:1434] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:23:06.859897  4686 tablet_bootstrap.cc:492] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4: Bootstrap starting.
I20260709 12:23:06.860128  4685 tablet_bootstrap.cc:492] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a: Bootstrap starting.
I20260709 12:23:06.865604  4685 tablet_bootstrap.cc:654] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.867130  4686 tablet_bootstrap.cc:654] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:06.872082  4691 raft_consensus.cc:493] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.872725  4691 raft_consensus.cc:515] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.876302  4490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f" is_pre_election: true
I20260709 12:23:06.877125  4490 raft_consensus.cc:2468] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 0.
I20260709 12:23:06.879089  4565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" is_pre_election: true
I20260709 12:23:06.879882  4565 raft_consensus.cc:2468] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 0.
I20260709 12:23:06.881043  4603 leader_election.cc:304] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:06.882205  4713 raft_consensus.cc:2804] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:06.882637  4713 raft_consensus.cc:493] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.883105  4713 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.877446  4691 leader_election.cc:290] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:06.888125  4685 tablet_bootstrap.cc:492] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a: No bootstrap required, opened a new log
I20260709 12:23:06.888573  4685 ts_tablet_manager.cc:1403] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.015s
I20260709 12:23:06.891405  4685 raft_consensus.cc:359] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.892092  4685 raft_consensus.cc:385] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.892431  4685 raft_consensus.cc:740] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.893131  4685 consensus_queue.cc:260] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.894443  4713 raft_consensus.cc:515] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.895382  4685 ts_tablet_manager.cc:1434] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.002s
I20260709 12:23:06.897075  4713 leader_election.cc:290] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:06.897571  4713 raft_consensus.cc:493] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.897521  4490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:06.898185  4490 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.898092  4713 raft_consensus.cc:515] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.900040  4686 tablet_bootstrap.cc:492] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4: No bootstrap required, opened a new log
I20260709 12:23:06.900489  4686 ts_tablet_manager.cc:1403] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent bootstrapping tablet: real 0.041s	user 0.016s	sys 0.015s
I20260709 12:23:06.901499  4565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:06.902045  4565 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.902223  4713 leader_election.cc:290] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:06.903064  4686 raft_consensus.cc:359] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.903486  4566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bba194fd0b34047badbff2aea2f9414" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" is_pre_election: true
I20260709 12:23:06.903733  4686 raft_consensus.cc:385] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:06.904018  4686 raft_consensus.cc:740] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Initialized, Role: FOLLOWER
I20260709 12:23:06.904656  4686 consensus_queue.cc:260] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.906628  4686 ts_tablet_manager.cc:1434] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:23:06.906693  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bba194fd0b34047badbff2aea2f9414" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f" is_pre_election: true
I20260709 12:23:06.907488  4491 raft_consensus.cc:2393] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3bbae884fe1434cbd0f42d01731773a in current term 1: Already voted for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in this term.
I20260709 12:23:06.908635  4603 leader_election.cc:304] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3bbae884fe1434cbd0f42d01731773a; no voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4
I20260709 12:23:06.909713  4691 raft_consensus.cc:3060] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.912024  4490 raft_consensus.cc:2468] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 1.
I20260709 12:23:06.913321  4603 leader_election.cc:304] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:06.921137  4565 raft_consensus.cc:2468] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 1.
I20260709 12:23:06.925805  4713 raft_consensus.cc:2804] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:06.926276  4713 raft_consensus.cc:697] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Becoming Leader. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Running, Role: LEADER
I20260709 12:23:06.927071  4566 raft_consensus.cc:3060] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.927261  4713 consensus_queue.cc:237] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.933616  4691 raft_consensus.cc:2749] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:23:06.936192  4691 consensus_queue.cc:1048] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:06.939597  4388 catalog_manager.cc:5697] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a reported cstate change: term changed from 0 to 1, leader changed from <none> to e3bbae884fe1434cbd0f42d01731773a (127.4.65.3). New cstate: current_term: 1 leader_uuid: "e3bbae884fe1434cbd0f42d01731773a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: HEALTHY } } }
W20260709 12:23:06.943313  4435 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:23:06.949729  4691 raft_consensus.cc:493] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.950206  4691 raft_consensus.cc:515] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:06.952251  4691 leader_election.cc:290] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693), 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043)
I20260709 12:23:06.953146  4565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d1398e83ae46109eb2163858c93214" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" is_pre_election: true
I20260709 12:23:06.953739  4565 raft_consensus.cc:2468] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 0.
I20260709 12:23:06.953718  4691 consensus_queue.cc:1048] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:23:06.954584  4490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d1398e83ae46109eb2163858c93214" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f" is_pre_election: true
I20260709 12:23:06.955022  4605 leader_election.cc:304] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5345e5bcd1f14d5d88bac1e08965a1c4, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:06.955155  4490 raft_consensus.cc:2468] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 0.
I20260709 12:23:06.959270  4691 raft_consensus.cc:2804] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:06.959633  4691 raft_consensus.cc:493] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.959959  4691 raft_consensus.cc:3060] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.967147  4713 raft_consensus.cc:493] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.967607  4713 raft_consensus.cc:515] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.970752  4713 leader_election.cc:290] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:06.970662  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3b696a9e3f4d479648b829ef3d167e" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f" is_pre_election: true
I20260709 12:23:06.966477  4691 raft_consensus.cc:515] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:06.971522  4491 raft_consensus.cc:2468] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 0.
I20260709 12:23:06.970685  4565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3b696a9e3f4d479648b829ef3d167e" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" is_pre_election: true
I20260709 12:23:06.973572  4565 raft_consensus.cc:2468] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 0.
I20260709 12:23:06.973730  4564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d1398e83ae46109eb2163858c93214" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:06.974339  4564 raft_consensus.cc:3060] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.975323  4605 leader_election.cc:304] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5345e5bcd1f14d5d88bac1e08965a1c4, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:06.980432  4744 raft_consensus.cc:2804] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:06.980837  4744 raft_consensus.cc:493] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.981179  4744 raft_consensus.cc:3060] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.981803  4564 raft_consensus.cc:2468] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 1.
I20260709 12:23:06.982826  4605 leader_election.cc:304] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5345e5bcd1f14d5d88bac1e08965a1c4, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:06.983844  4691 leader_election.cc:290] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 election: Requested vote from peers 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693), 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043)
I20260709 12:23:06.984344  4691 raft_consensus.cc:2804] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:06.984781  4691 raft_consensus.cc:697] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Becoming Leader. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Running, Role: LEADER
I20260709 12:23:06.985000  4490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d1398e83ae46109eb2163858c93214" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:06.985659  4490 raft_consensus.cc:3060] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.985551  4691 consensus_queue.cc:237] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:06.987566  4744 raft_consensus.cc:515] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:06.991086  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3b696a9e3f4d479648b829ef3d167e" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:06.991650  4491 raft_consensus.cc:3060] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:06.992287  4490 raft_consensus.cc:2468] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 1.
I20260709 12:23:06.996260  4491 raft_consensus.cc:2468] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 1.
I20260709 12:23:06.997391  4603 leader_election.cc:304] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:06.998175  4713 raft_consensus.cc:2804] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:06.998374  4744 leader_election.cc:290] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 1 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:06.999255  4713 raft_consensus.cc:697] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Becoming Leader. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Running, Role: LEADER
I20260709 12:23:06.998138  4564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b3b696a9e3f4d479648b829ef3d167e" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:06.999919  4564 raft_consensus.cc:3060] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:07.000140  4713 consensus_queue.cc:237] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.009786  4564 raft_consensus.cc:2468] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 1.
I20260709 12:23:07.014403  4386 catalog_manager.cc:5697] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a reported cstate change: term changed from 0 to 1, leader changed from <none> to e3bbae884fe1434cbd0f42d01731773a (127.4.65.3). New cstate: current_term: 1 leader_uuid: "e3bbae884fe1434cbd0f42d01731773a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:07.027225  4386 catalog_manager.cc:5697] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a reported cstate change: term changed from 0 to 1, leader changed from <none> to e3bbae884fe1434cbd0f42d01731773a (127.4.65.3). New cstate: current_term: 1 leader_uuid: "e3bbae884fe1434cbd0f42d01731773a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:07.051080  4692 raft_consensus.cc:493] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.051608  4692 raft_consensus.cc:515] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.053818  4692 leader_election.cc:290] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:07.055068  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a987ec62f8c04b9aad527a2b4c8c00cc" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f" is_pre_election: true
I20260709 12:23:07.055766  4491 raft_consensus.cc:2468] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 0.
I20260709 12:23:07.055985  4640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a987ec62f8c04b9aad527a2b4c8c00cc" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a" is_pre_election: true
I20260709 12:23:07.055764  4720 consensus_queue.cc:1048] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:23:07.056913  4640 raft_consensus.cc:2468] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 0.
I20260709 12:23:07.059372  4529 leader_election.cc:304] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: 
I20260709 12:23:07.061327  4692 raft_consensus.cc:2804] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:07.069125  4689 consensus_queue.cc:1048] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.076431  4692 raft_consensus.cc:493] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.076817  4692 raft_consensus.cc:3060] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:07.100126  4692 raft_consensus.cc:515] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.102206  4692 leader_election.cc:290] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:07.103520  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a987ec62f8c04b9aad527a2b4c8c00cc" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:07.103447  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a987ec62f8c04b9aad527a2b4c8c00cc" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:07.104151  4641 raft_consensus.cc:3060] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:07.104121  4491 raft_consensus.cc:3060] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:07.110805  4491 raft_consensus.cc:2468] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 1.
I20260709 12:23:07.110752  4641 raft_consensus.cc:2468] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 1.
I20260709 12:23:07.111959  4529 leader_election.cc:304] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: 
I20260709 12:23:07.112807  4692 raft_consensus.cc:2804] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:07.113464  4692 raft_consensus.cc:697] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 LEADER]: Becoming Leader. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:07.114259  4692 consensus_queue.cc:237] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.123772  4387 catalog_manager.cc:5697] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2). New cstate: current_term: 1 leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:07.125511  4689 consensus_queue.cc:1048] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.141535  4721 consensus_queue.cc:1048] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:23:07.202363  4641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "93d1398e83ae46109eb2163858c93214"
dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
mode: GRACEFUL
new_leader_uuid: "498f39f6bf2844b3ae1bd842752f065f"
 from {username='slave'} at 127.0.0.1:55136
I20260709 12:23:07.202920  4641 raft_consensus.cc:606] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Received request to transfer leadership to TS 498f39f6bf2844b3ae1bd842752f065f
I20260709 12:23:07.208050  4641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e"
dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
mode: GRACEFUL
new_leader_uuid: "498f39f6bf2844b3ae1bd842752f065f"
 from {username='slave'} at 127.0.0.1:55136
I20260709 12:23:07.208678  4641 raft_consensus.cc:606] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Received request to transfer leadership to TS 498f39f6bf2844b3ae1bd842752f065f
I20260709 12:23:07.210143  4692 consensus_queue.cc:1048] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.214545  4641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27b95197718748aca01ae1e6379245ca"
dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
mode: GRACEFUL
new_leader_uuid: "498f39f6bf2844b3ae1bd842752f065f"
 from {username='slave'} at 127.0.0.1:55136
I20260709 12:23:07.215153  4641 raft_consensus.cc:606] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Received request to transfer leadership to TS 498f39f6bf2844b3ae1bd842752f065f
I20260709 12:23:07.223881  4729 consensus_queue.cc:1048] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.234973  4564 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9bba194fd0b34047badbff2aea2f9414"
dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
mode: GRACEFUL
new_leader_uuid: "498f39f6bf2844b3ae1bd842752f065f"
 from {username='slave'} at 127.0.0.1:44338
I20260709 12:23:07.235116  4692 consensus_queue.cc:1048] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.235590  4564 raft_consensus.cc:606] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 LEADER]: Received request to transfer leadership to TS 498f39f6bf2844b3ae1bd842752f065f
I20260709 12:23:07.254770  4722 raft_consensus.cc:993] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4: : Instructing follower 498f39f6bf2844b3ae1bd842752f065f to start an election
I20260709 12:23:07.255232  4722 raft_consensus.cc:1081] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 LEADER]: Signalling peer 498f39f6bf2844b3ae1bd842752f065f to start an election
I20260709 12:23:07.266181  4722 consensus_queue.cc:1048] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.269997  4491 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9bba194fd0b34047badbff2aea2f9414"
dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
 from {username='slave'} at 127.0.0.1:37980
I20260709 12:23:07.270682  4491 raft_consensus.cc:493] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:07.271116  4491 raft_consensus.cc:3060] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.278825  4490 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f31c9f2f020a45e6b1077bb51b6dafde"
dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
mode: GRACEFUL
new_leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
 from {username='slave'} at 127.0.0.1:37984
I20260709 12:23:07.279451  4490 raft_consensus.cc:606] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 1 LEADER]: Received request to transfer leadership to TS 5345e5bcd1f14d5d88bac1e08965a1c4
I20260709 12:23:07.278295  4491 raft_consensus.cc:515] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.283411  4564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bba194fd0b34047badbff2aea2f9414" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:07.283989  4564 raft_consensus.cc:3055] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:07.284301  4564 raft_consensus.cc:740] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:07.285101  4564 consensus_queue.cc:260] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.286051  4564 raft_consensus.cc:3060] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.288655  4490 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "de101290db964f41b3fb73588fd5964f"
dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
mode: GRACEFUL
new_leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
 from {username='slave'} at 127.0.0.1:37984
I20260709 12:23:07.289423  4490 raft_consensus.cc:606] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 1 LEADER]: Received request to transfer leadership to TS 5345e5bcd1f14d5d88bac1e08965a1c4
I20260709 12:23:07.299623  4564 raft_consensus.cc:2468] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 2.
I20260709 12:23:07.299750  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bba194fd0b34047badbff2aea2f9414" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:07.300153  4491 leader_election.cc:290] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 2 election: Requested vote from peers 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:07.300772  4641 raft_consensus.cc:3060] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.302484  4456 leader_election.cc:304] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: 
I20260709 12:23:07.303318  4721 raft_consensus.cc:2804] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:07.303757  4721 raft_consensus.cc:697] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Becoming Leader. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:07.304580  4721 consensus_queue.cc:237] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.305406  4641 raft_consensus.cc:2468] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 2.
I20260709 12:23:07.311703  4387 catalog_manager.cc:5697] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f reported cstate change: term changed from 1 to 2, leader changed from 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2) to 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1). New cstate: current_term: 2 leader_uuid: "498f39f6bf2844b3ae1bd842752f065f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:07.410202  4691 consensus_queue.cc:1048] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:23:07.415419  4713 consensus_queue.cc:1048] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.420054  4744 raft_consensus.cc:993] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a: : Instructing follower 498f39f6bf2844b3ae1bd842752f065f to start an election
I20260709 12:23:07.420594  4744 raft_consensus.cc:1081] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Signalling peer 498f39f6bf2844b3ae1bd842752f065f to start an election
I20260709 12:23:07.422719  4490 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e"
dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
 from {username='slave'} at 127.0.0.1:37972
I20260709 12:23:07.423286  4490 raft_consensus.cc:493] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:07.423678  4490 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.431097  4490 raft_consensus.cc:515] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.432682  4491 raft_consensus.cc:1240] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Rejecting Update request from peer e3bbae884fe1434cbd0f42d01731773a for earlier term 1. Current term is 2. Ops: []
I20260709 12:23:07.433568  4691 consensus_queue.cc:1048] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.437040  4565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:07.437610  4565 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.437147  4490 leader_election.cc:290] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 2 election: Requested vote from peers 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:07.440482  4744 consensus_queue.cc:1059] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.441250  4744 raft_consensus.cc:3055] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:07.441550  4744 raft_consensus.cc:740] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Running, Role: LEADER
I20260709 12:23:07.442126  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:07.442353  4744 consensus_queue.cc:260] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.443428  4744 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.444330  4565 raft_consensus.cc:2468] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 2.
I20260709 12:23:07.445873  4564 raft_consensus.cc:1240] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Rejecting Update request from peer e3bbae884fe1434cbd0f42d01731773a for earlier term 1. Current term is 2. Ops: []
I20260709 12:23:07.448822  4456 leader_election.cc:304] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: 
I20260709 12:23:07.451314  4721 raft_consensus.cc:2804] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:07.451843  4721 raft_consensus.cc:697] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Becoming Leader. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:07.452651  4721 consensus_queue.cc:237] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.462843  4691 consensus_queue.cc:1048] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.465555  4641 raft_consensus.cc:2468] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 2.
I20260709 12:23:07.469353  4387 catalog_manager.cc:5697] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f reported cstate change: term changed from 1 to 2, leader changed from e3bbae884fe1434cbd0f42d01731773a (127.4.65.3) to 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1). New cstate: current_term: 2 leader_uuid: "498f39f6bf2844b3ae1bd842752f065f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:07.477587  4691 raft_consensus.cc:993] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a: : Instructing follower 498f39f6bf2844b3ae1bd842752f065f to start an election
I20260709 12:23:07.478062  4691 raft_consensus.cc:1081] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Signalling peer 498f39f6bf2844b3ae1bd842752f065f to start an election
I20260709 12:23:07.479863  4491 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "93d1398e83ae46109eb2163858c93214"
dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
 from {username='slave'} at 127.0.0.1:37972
I20260709 12:23:07.481685  4491 raft_consensus.cc:493] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:07.482146  4491 raft_consensus.cc:3060] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.484293  4751 raft_consensus.cc:993] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a: : Instructing follower 498f39f6bf2844b3ae1bd842752f065f to start an election
I20260709 12:23:07.484752  4691 raft_consensus.cc:1081] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Signalling peer 498f39f6bf2844b3ae1bd842752f065f to start an election
I20260709 12:23:07.486845  4691 consensus_queue.cc:1048] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.488502  4490 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "27b95197718748aca01ae1e6379245ca"
dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
 from {username='slave'} at 127.0.0.1:37972
I20260709 12:23:07.489053  4490 raft_consensus.cc:493] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:07.489437  4490 raft_consensus.cc:3060] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.492317  4491 raft_consensus.cc:515] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:07.494902  4491 leader_election.cc:290] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 2 election: Requested vote from peers e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:07.497516  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d1398e83ae46109eb2163858c93214" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:07.498095  4641 raft_consensus.cc:3055] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:07.497958  4565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d1398e83ae46109eb2163858c93214" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:07.498435  4641 raft_consensus.cc:740] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Running, Role: LEADER
I20260709 12:23:07.498760  4565 raft_consensus.cc:3060] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.499251  4641 consensus_queue.cc:260] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:07.500551  4641 raft_consensus.cc:3060] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.504261  4490 raft_consensus.cc:515] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.506441  4490 leader_election.cc:290] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 2 election: Requested vote from peers 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:07.509915  4564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b95197718748aca01ae1e6379245ca" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:07.510058  4640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b95197718748aca01ae1e6379245ca" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:07.510490  4564 raft_consensus.cc:3060] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.510565  4640 raft_consensus.cc:3055] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:07.510949  4640 raft_consensus.cc:740] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Running, Role: LEADER
I20260709 12:23:07.511770  4565 raft_consensus.cc:2468] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 2.
I20260709 12:23:07.511729  4640 consensus_queue.cc:260] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.514757  4640 raft_consensus.cc:3060] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.516647  4456 leader_election.cc:304] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: 
I20260709 12:23:07.513052  4641 raft_consensus.cc:2468] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 2.
I20260709 12:23:07.517570  4721 raft_consensus.cc:2804] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:07.518098  4721 raft_consensus.cc:697] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Becoming Leader. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:07.518908  4721 consensus_queue.cc:237] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:07.521639  4640 raft_consensus.cc:2468] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 2.
I20260709 12:23:07.523289  4456 leader_election.cc:304] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:07.524135  4689 raft_consensus.cc:2804] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:07.524690  4689 raft_consensus.cc:697] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Becoming Leader. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:07.525617  4689 consensus_queue.cc:237] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.530769  4387 catalog_manager.cc:5697] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f reported cstate change: term changed from 1 to 2, leader changed from e3bbae884fe1434cbd0f42d01731773a (127.4.65.3) to 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1). New cstate: current_term: 2 leader_uuid: "498f39f6bf2844b3ae1bd842752f065f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:07.535055  4744 consensus_queue.cc:1048] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.535744  4564 raft_consensus.cc:2468] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 2.
I20260709 12:23:07.545969  4387 catalog_manager.cc:5697] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f reported cstate change: term changed from 1 to 2, leader changed from e3bbae884fe1434cbd0f42d01731773a (127.4.65.3) to 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1). New cstate: current_term: 2 leader_uuid: "498f39f6bf2844b3ae1bd842752f065f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:07.570371  4692 consensus_queue.cc:1048] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.582636  4726 consensus_queue.cc:1048] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.634205  4689 raft_consensus.cc:993] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f: : Instructing follower 5345e5bcd1f14d5d88bac1e08965a1c4 to start an election
I20260709 12:23:07.634701  4721 raft_consensus.cc:1081] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 1 LEADER]: Signalling peer 5345e5bcd1f14d5d88bac1e08965a1c4 to start an election
I20260709 12:23:07.637697  4564 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "de101290db964f41b3fb73588fd5964f"
dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
 from {username='slave'} at 127.0.0.1:44328
I20260709 12:23:07.638232  4564 raft_consensus.cc:493] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:07.638600  4564 raft_consensus.cc:3060] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.642839  4564 raft_consensus.cc:515] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.644297  4564 leader_election.cc:290] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 2 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:07.645074  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de101290db964f41b3fb73588fd5964f" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:07.645674  4491 raft_consensus.cc:3055] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:07.645709  4640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de101290db964f41b3fb73588fd5964f" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:07.646067  4491 raft_consensus.cc:740] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:07.646281  4640 raft_consensus.cc:3060] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.646878  4491 consensus_queue.cc:260] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.648032  4491 raft_consensus.cc:3060] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.650812  4640 raft_consensus.cc:2468] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 2.
I20260709 12:23:07.651810  4531 leader_election.cc:304] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5345e5bcd1f14d5d88bac1e08965a1c4, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:07.652474  4726 raft_consensus.cc:2804] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:07.652921  4726 raft_consensus.cc:697] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 LEADER]: Becoming Leader. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:07.653693  4726 consensus_queue.cc:237] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.654136  4491 raft_consensus.cc:2468] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 2.
I20260709 12:23:07.659327  4387 catalog_manager.cc:5697] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 reported cstate change: term changed from 1 to 2, leader changed from 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1) to 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:07.685046  4689 raft_consensus.cc:993] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f: : Instructing follower 5345e5bcd1f14d5d88bac1e08965a1c4 to start an election
I20260709 12:23:07.685484  4721 raft_consensus.cc:1081] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 1 LEADER]: Signalling peer 5345e5bcd1f14d5d88bac1e08965a1c4 to start an election
I20260709 12:23:07.686896  4564 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f31c9f2f020a45e6b1077bb51b6dafde"
dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
 from {username='slave'} at 127.0.0.1:44328
I20260709 12:23:07.687345  4564 raft_consensus.cc:493] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:07.687662  4564 raft_consensus.cc:3060] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.691700  4564 raft_consensus.cc:515] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.693207  4564 leader_election.cc:290] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 2 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:07.693966  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31c9f2f020a45e6b1077bb51b6dafde" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:07.694356  4640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f31c9f2f020a45e6b1077bb51b6dafde" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:07.694531  4491 raft_consensus.cc:3055] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:07.694896  4640 raft_consensus.cc:3060] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.694878  4491 raft_consensus.cc:740] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:07.695868  4491 consensus_queue.cc:260] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.697043  4491 raft_consensus.cc:3060] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:07.701375  4640 raft_consensus.cc:2468] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 2.
I20260709 12:23:07.702507  4531 leader_election.cc:304] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5345e5bcd1f14d5d88bac1e08965a1c4, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:07.703323  4726 raft_consensus.cc:2804] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:07.703436  4491 raft_consensus.cc:2468] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 2.
I20260709 12:23:07.703795  4726 raft_consensus.cc:697] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 LEADER]: Becoming Leader. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:07.704612  4726 consensus_queue.cc:237] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:07.710589  4387 catalog_manager.cc:5697] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 reported cstate change: term changed from 1 to 2, leader changed from 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1) to 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:07.888686  4564 raft_consensus.cc:1275] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Refusing update from remote peer 498f39f6bf2844b3ae1bd842752f065f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:07.889930  4721 consensus_queue.cc:1048] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:07.892828  4640 raft_consensus.cc:1275] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Refusing update from remote peer 498f39f6bf2844b3ae1bd842752f065f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:07.894140  4689 consensus_queue.cc:1048] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:07.902674  4641 raft_consensus.cc:1275] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Refusing update from remote peer 498f39f6bf2844b3ae1bd842752f065f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:07.905925  4720 consensus_queue.cc:1048] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:23:07.908851  4564 raft_consensus.cc:1275] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Refusing update from remote peer 498f39f6bf2844b3ae1bd842752f065f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:07.915520  4564 raft_consensus.cc:1275] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Refusing update from remote peer 498f39f6bf2844b3ae1bd842752f065f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:07.920439  4689 consensus_queue.cc:1048] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:07.922839  4689 consensus_queue.cc:1048] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:07.936288  4564 raft_consensus.cc:1275] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Refusing update from remote peer 498f39f6bf2844b3ae1bd842752f065f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:07.939584  4640 raft_consensus.cc:1275] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Refusing update from remote peer 498f39f6bf2844b3ae1bd842752f065f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:07.941424  4720 consensus_queue.cc:1048] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:07.942338  4689 consensus_queue.cc:1048] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:07.951304  4641 raft_consensus.cc:1275] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Refusing update from remote peer 498f39f6bf2844b3ae1bd842752f065f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:07.953331  4689 consensus_queue.cc:1048] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:08.092319  4640 raft_consensus.cc:1275] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Refusing update from remote peer 5345e5bcd1f14d5d88bac1e08965a1c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:08.093541  4726 consensus_queue.cc:1048] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:08.101063  4491 raft_consensus.cc:1275] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Refusing update from remote peer 5345e5bcd1f14d5d88bac1e08965a1c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:08.102931  4729 consensus_queue.cc:1048] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:08.177098  4640 raft_consensus.cc:1275] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Refusing update from remote peer 5345e5bcd1f14d5d88bac1e08965a1c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:08.178352  4729 consensus_queue.cc:1048] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:08.187316  4491 raft_consensus.cc:1275] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Refusing update from remote peer 5345e5bcd1f14d5d88bac1e08965a1c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:08.188602  4729 consensus_queue.cc:1048] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:10.322773  4356 auto_leader_rebalancer-test.cc:294] The leader distribution is 
I20260709 12:23:10.323550  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:10.343994  4566 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1"
dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
mode: GRACEFUL
new_leader_uuid: "e3bbae884fe1434cbd0f42d01731773a"
 from {username='slave'} at 127.0.0.1:44354
I20260709 12:23:10.344480  4566 raft_consensus.cc:606] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 LEADER]: Received request to transfer leadership to TS e3bbae884fe1434cbd0f42d01731773a
I20260709 12:23:10.353644  4490 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e"
dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
mode: GRACEFUL
new_leader_uuid: "e3bbae884fe1434cbd0f42d01731773a"
 from {username='slave'} at 127.0.0.1:37998
I20260709 12:23:10.354115  4490 raft_consensus.cc:606] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Received request to transfer leadership to TS e3bbae884fe1434cbd0f42d01731773a
I20260709 12:23:10.355049  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:23:10.355454  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:10.779764  4789 raft_consensus.cc:993] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f: : Instructing follower e3bbae884fe1434cbd0f42d01731773a to start an election
I20260709 12:23:10.780311  4721 raft_consensus.cc:1081] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Signalling peer e3bbae884fe1434cbd0f42d01731773a to start an election
I20260709 12:23:10.781831  4640 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e"
dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
 from {username='slave'} at 127.0.0.1:55114
I20260709 12:23:10.782217  4640 raft_consensus.cc:493] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:10.782509  4640 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:10.786655  4640 raft_consensus.cc:515] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:10.788128  4640 leader_election.cc:290] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 3 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:10.788834  4490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:10.789095  4566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:10.789441  4490 raft_consensus.cc:3055] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:10.789594  4566 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:10.789773  4490 raft_consensus.cc:740] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:10.790546  4490 consensus_queue.cc:260] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:10.791421  4490 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:10.794008  4566 raft_consensus.cc:2468] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 3.
I20260709 12:23:10.794909  4605 leader_election.cc:304] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5345e5bcd1f14d5d88bac1e08965a1c4, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:10.795521  4786 raft_consensus.cc:2804] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:23:10.795782  4490 raft_consensus.cc:2468] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 3.
I20260709 12:23:10.795881  4786 raft_consensus.cc:697] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 3 LEADER]: Becoming Leader. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Running, Role: LEADER
I20260709 12:23:10.796806  4786 consensus_queue.cc:237] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:10.805076  4388 catalog_manager.cc:5697] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a reported cstate change: term changed from 2 to 3, leader changed from 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1) to e3bbae884fe1434cbd0f42d01731773a (127.4.65.3). New cstate: current_term: 3 leader_uuid: "e3bbae884fe1434cbd0f42d01731773a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:10.861989  4729 raft_consensus.cc:993] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4: : Instructing follower e3bbae884fe1434cbd0f42d01731773a to start an election
I20260709 12:23:10.862416  4788 raft_consensus.cc:1081] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 LEADER]: Signalling peer e3bbae884fe1434cbd0f42d01731773a to start an election
I20260709 12:23:10.863603  4640 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1"
dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
 from {username='slave'} at 127.0.0.1:55120
I20260709 12:23:10.864055  4640 raft_consensus.cc:493] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:10.864432  4640 raft_consensus.cc:3060] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:10.868670  4640 raft_consensus.cc:515] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:10.870203  4640 leader_election.cc:290] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 2 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:10.870905  4490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:10.871145  4566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:10.871487  4490 raft_consensus.cc:3060] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:10.871675  4566 raft_consensus.cc:3055] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:10.872011  4566 raft_consensus.cc:740] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:10.872828  4566 consensus_queue.cc:260] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:10.874105  4566 raft_consensus.cc:3060] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:10.878067  4490 raft_consensus.cc:2468] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 2.
I20260709 12:23:10.879148  4603 leader_election.cc:304] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:10.879884  4786 raft_consensus.cc:2804] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:10.880276  4786 raft_consensus.cc:697] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 2 LEADER]: Becoming Leader. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Running, Role: LEADER
I20260709 12:23:10.880460  4566 raft_consensus.cc:2468] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 2.
I20260709 12:23:10.881004  4786 consensus_queue.cc:237] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:10.887153  4388 catalog_manager.cc:5697] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a reported cstate change: term changed from 1 to 2, leader changed from 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2) to e3bbae884fe1434cbd0f42d01731773a (127.4.65.3). New cstate: current_term: 2 leader_uuid: "e3bbae884fe1434cbd0f42d01731773a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:11.230212  4491 raft_consensus.cc:1275] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Refusing update from remote peer e3bbae884fe1434cbd0f42d01731773a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:11.231220  4786 consensus_queue.cc:1048] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:11.239674  4566 raft_consensus.cc:1275] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Refusing update from remote peer e3bbae884fe1434cbd0f42d01731773a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:11.241226  4799 consensus_queue.cc:1048] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:11.266144  4566 raft_consensus.cc:1275] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Refusing update from remote peer e3bbae884fe1434cbd0f42d01731773a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:11.267338  4799 consensus_queue.cc:1048] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:11.274737  4491 raft_consensus.cc:1275] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Refusing update from remote peer e3bbae884fe1434cbd0f42d01731773a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:11.276072  4786 consensus_queue.cc:1048] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:11.356453  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:11.362694  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:11.363232  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:12.364069  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:12.368748  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:12.369045  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:13.369972  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:13.374596  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:13.374902  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:14.375869  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:14.380497  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:14.380789  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:15.381673  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:15.386222  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:15.386552  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:16.387383  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:16.392282  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:16.392634  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:17.393486  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:17.398032  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:17.398494  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:18.399399  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:18.404008  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:18.404410  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:19.405303  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:19.410012  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:19.410409  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:20.442088  4356 auto_leader_rebalancer-test.cc:313] The leader distribution is 
I20260709 12:23:20.446220  4491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "93d1398e83ae46109eb2163858c93214"
dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
mode: GRACEFUL
new_leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
 from {username='slave'} at 127.0.0.1:37984
I20260709 12:23:20.446794  4491 raft_consensus.cc:606] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Received request to transfer leadership to TS 5345e5bcd1f14d5d88bac1e08965a1c4
I20260709 12:23:20.451171  4641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e"
dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
mode: GRACEFUL
new_leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
 from {username='slave'} at 127.0.0.1:55136
I20260709 12:23:20.451615  4641 raft_consensus.cc:606] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 3 LEADER]: Received request to transfer leadership to TS 5345e5bcd1f14d5d88bac1e08965a1c4
I20260709 12:23:20.455937  4491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27b95197718748aca01ae1e6379245ca"
dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
mode: GRACEFUL
new_leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
 from {username='slave'} at 127.0.0.1:37984
I20260709 12:23:20.456385  4491 raft_consensus.cc:606] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Received request to transfer leadership to TS 5345e5bcd1f14d5d88bac1e08965a1c4
I20260709 12:23:20.460464  4491 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9bba194fd0b34047badbff2aea2f9414"
dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
mode: GRACEFUL
new_leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
 from {username='slave'} at 127.0.0.1:37984
I20260709 12:23:20.460872  4491 raft_consensus.cc:606] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Received request to transfer leadership to TS 5345e5bcd1f14d5d88bac1e08965a1c4
I20260709 12:23:20.465020  4641 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1"
dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
mode: GRACEFUL
new_leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
 from {username='slave'} at 127.0.0.1:55136
I20260709 12:23:20.465445  4641 raft_consensus.cc:606] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 2 LEADER]: Received request to transfer leadership to TS 5345e5bcd1f14d5d88bac1e08965a1c4
I20260709 12:23:20.576784  4818 raft_consensus.cc:993] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f: : Instructing follower 5345e5bcd1f14d5d88bac1e08965a1c4 to start an election
I20260709 12:23:20.577423  4812 raft_consensus.cc:1081] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Signalling peer 5345e5bcd1f14d5d88bac1e08965a1c4 to start an election
I20260709 12:23:20.579233  4566 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9bba194fd0b34047badbff2aea2f9414"
dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
 from {username='slave'} at 127.0.0.1:44328
I20260709 12:23:20.579691  4566 raft_consensus.cc:493] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:20.580051  4566 raft_consensus.cc:3060] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:20.584746  4566 raft_consensus.cc:515] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:20.586205  4566 leader_election.cc:290] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 3 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:20.587013  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bba194fd0b34047badbff2aea2f9414" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:20.587296  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9bba194fd0b34047badbff2aea2f9414" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:20.587704  4491 raft_consensus.cc:3055] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:20.587824  4641 raft_consensus.cc:3060] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:20.587999  4491 raft_consensus.cc:740] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:20.588912  4491 consensus_queue.cc:260] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:20.590098  4491 raft_consensus.cc:3060] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:20.592484  4818 raft_consensus.cc:993] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f: : Instructing follower 5345e5bcd1f14d5d88bac1e08965a1c4 to start an election
I20260709 12:23:20.593060  4812 raft_consensus.cc:1081] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Signalling peer 5345e5bcd1f14d5d88bac1e08965a1c4 to start an election
I20260709 12:23:20.594558  4566 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "93d1398e83ae46109eb2163858c93214"
dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
 from {username='slave'} at 127.0.0.1:44328
I20260709 12:23:20.595022  4566 raft_consensus.cc:493] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:20.595330  4566 raft_consensus.cc:3060] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:20.595678  4641 raft_consensus.cc:2468] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 3.
I20260709 12:23:20.595821  4491 raft_consensus.cc:2468] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 3.
I20260709 12:23:20.596966  4531 leader_election.cc:304] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5345e5bcd1f14d5d88bac1e08965a1c4, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:20.597764  4815 raft_consensus.cc:2804] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:23:20.598894  4815 raft_consensus.cc:697] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Becoming Leader. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:20.599740  4815 consensus_queue.cc:237] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:20.600159  4566 raft_consensus.cc:515] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:20.601802  4566 leader_election.cc:290] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 3 election: Requested vote from peers e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891), 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043)
I20260709 12:23:20.602551  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d1398e83ae46109eb2163858c93214" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:20.602912  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d1398e83ae46109eb2163858c93214" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:20.603252  4641 raft_consensus.cc:3060] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:20.603402  4491 raft_consensus.cc:3055] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:20.604156  4491 raft_consensus.cc:740] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:20.604902  4491 consensus_queue.cc:260] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:20.605664  4491 raft_consensus.cc:3060] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:20.609433  4491 raft_consensus.cc:2468] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 3.
I20260709 12:23:20.610419  4817 raft_consensus.cc:993] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a: : Instructing follower 5345e5bcd1f14d5d88bac1e08965a1c4 to start an election
I20260709 12:23:20.610981  4806 raft_consensus.cc:1081] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 3 LEADER]: Signalling peer 5345e5bcd1f14d5d88bac1e08965a1c4 to start an election
I20260709 12:23:20.612035  4529 leader_election.cc:304] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: 
I20260709 12:23:20.612179  4641 raft_consensus.cc:2468] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 3.
I20260709 12:23:20.613716  4819 raft_consensus.cc:2804] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:23:20.614148  4566 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e"
dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
 from {username='slave'} at 127.0.0.1:44326
I20260709 12:23:20.614845  4566 raft_consensus.cc:493] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:20.615154  4566 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:20.617869  4819 raft_consensus.cc:697] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Becoming Leader. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:20.619405  4819 consensus_queue.cc:237] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:20.622936  4566 raft_consensus.cc:515] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:20.622800  4387 catalog_manager.cc:5697] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 reported cstate change: term changed from 2 to 3, leader changed from 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1) to 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2). New cstate: current_term: 3 leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:20.625521  4566 leader_election.cc:290] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 4 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:20.625754  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:20.626011  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:20.626297  4491 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:20.626610  4641 raft_consensus.cc:3055] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:23:20.626960  4641 raft_consensus.cc:740] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Running, Role: LEADER
I20260709 12:23:20.627795  4641 consensus_queue.cc:260] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:20.628875  4641 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:20.632645  4491 raft_consensus.cc:2468] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 4.
I20260709 12:23:20.633594  4529 leader_election.cc:304] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: 
I20260709 12:23:20.634250  4819 raft_consensus.cc:2804] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:23:20.634769  4819 raft_consensus.cc:697] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 LEADER]: Becoming Leader. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:20.635609  4641 raft_consensus.cc:2468] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 4.
I20260709 12:23:20.635583  4819 consensus_queue.cc:237] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:20.645915  4387 catalog_manager.cc:5697] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 reported cstate change: term changed from 2 to 3, leader changed from 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1) to 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2). New cstate: current_term: 3 leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:20.647030  4387 catalog_manager.cc:5697] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 reported cstate change: term changed from 3 to 4, leader changed from e3bbae884fe1434cbd0f42d01731773a (127.4.65.3) to 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2). New cstate: current_term: 4 leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:20.677052  4806 raft_consensus.cc:993] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a: : Instructing follower 5345e5bcd1f14d5d88bac1e08965a1c4 to start an election
I20260709 12:23:20.677479  4817 raft_consensus.cc:1081] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 2 LEADER]: Signalling peer 5345e5bcd1f14d5d88bac1e08965a1c4 to start an election
I20260709 12:23:20.678700  4566 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1"
dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
 from {username='slave'} at 127.0.0.1:44326
I20260709 12:23:20.679147  4566 raft_consensus.cc:493] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:20.679417  4566 raft_consensus.cc:3060] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:20.683360  4566 raft_consensus.cc:515] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:20.684898  4566 leader_election.cc:290] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 3 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:20.685444  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:20.685812  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:20.685940  4491 raft_consensus.cc:3060] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:20.686295  4641 raft_consensus.cc:3055] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:20.686553  4641 raft_consensus.cc:740] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Running, Role: LEADER
I20260709 12:23:20.687139  4641 consensus_queue.cc:260] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:20.688045  4641 raft_consensus.cc:3060] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:20.690842  4491 raft_consensus.cc:2468] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 3.
I20260709 12:23:20.691653  4529 leader_election.cc:304] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: 
I20260709 12:23:20.692256  4819 raft_consensus.cc:2804] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:23:20.692736  4819 raft_consensus.cc:697] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Becoming Leader. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:20.693228  4641 raft_consensus.cc:2468] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 3.
I20260709 12:23:20.693600  4819 consensus_queue.cc:237] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:20.699908  4387 catalog_manager.cc:5697] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 reported cstate change: term changed from 2 to 3, leader changed from e3bbae884fe1434cbd0f42d01731773a (127.4.65.3) to 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2). New cstate: current_term: 3 leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:20.871975  4818 raft_consensus.cc:993] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f: : Instructing follower 5345e5bcd1f14d5d88bac1e08965a1c4 to start an election
I20260709 12:23:20.872453  4812 raft_consensus.cc:1081] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Signalling peer 5345e5bcd1f14d5d88bac1e08965a1c4 to start an election
I20260709 12:23:20.873675  4566 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "27b95197718748aca01ae1e6379245ca"
dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
 from {username='slave'} at 127.0.0.1:44328
I20260709 12:23:20.874114  4566 raft_consensus.cc:493] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:20.874383  4566 raft_consensus.cc:3060] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:20.878118  4566 raft_consensus.cc:515] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:20.879437  4566 leader_election.cc:290] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 3 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:20.880092  4491 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b95197718748aca01ae1e6379245ca" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:20.880422  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b95197718748aca01ae1e6379245ca" candidate_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:20.880652  4491 raft_consensus.cc:3055] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:20.880908  4641 raft_consensus.cc:3060] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:20.880945  4491 raft_consensus.cc:740] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:20.881673  4491 consensus_queue.cc:260] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:20.882428  4491 raft_consensus.cc:3060] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:20.885624  4641 raft_consensus.cc:2468] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 3.
I20260709 12:23:20.886480  4531 leader_election.cc:304] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5345e5bcd1f14d5d88bac1e08965a1c4, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:20.887197  4491 raft_consensus.cc:2468] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5345e5bcd1f14d5d88bac1e08965a1c4 in term 3.
I20260709 12:23:20.887236  4819 raft_consensus.cc:2804] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:23:20.887791  4819 raft_consensus.cc:697] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Becoming Leader. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:20.888830  4819 consensus_queue.cc:237] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:20.894717  4387 catalog_manager.cc:5697] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 reported cstate change: term changed from 2 to 3, leader changed from 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1) to 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2). New cstate: current_term: 3 leader_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:21.098089  4491 raft_consensus.cc:1275] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Refusing update from remote peer 5345e5bcd1f14d5d88bac1e08965a1c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:23:21.099344  4819 consensus_queue.cc:1048] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:23:21.099879  4641 raft_consensus.cc:1275] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 4 FOLLOWER]: Refusing update from remote peer 5345e5bcd1f14d5d88bac1e08965a1c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:23:21.101046  4819 consensus_queue.cc:1048] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:23:21.123903  4641 raft_consensus.cc:1275] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Refusing update from remote peer 5345e5bcd1f14d5d88bac1e08965a1c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:21.125134  4834 consensus_queue.cc:1048] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:21.131894  4491 raft_consensus.cc:1275] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Refusing update from remote peer 5345e5bcd1f14d5d88bac1e08965a1c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:21.135416  4821 consensus_queue.cc:1048] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:23:21.135340  4491 raft_consensus.cc:1275] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Refusing update from remote peer 5345e5bcd1f14d5d88bac1e08965a1c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:21.137622  4821 consensus_queue.cc:1048] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:21.146737  4641 raft_consensus.cc:1275] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Refusing update from remote peer 5345e5bcd1f14d5d88bac1e08965a1c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:21.148875  4821 consensus_queue.cc:1048] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:21.166399  4641 raft_consensus.cc:1275] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Refusing update from remote peer 5345e5bcd1f14d5d88bac1e08965a1c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:21.167438  4834 consensus_queue.cc:1048] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:21.176085  4489 raft_consensus.cc:1275] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Refusing update from remote peer 5345e5bcd1f14d5d88bac1e08965a1c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:21.177433  4834 consensus_queue.cc:1048] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:21.322854  4641 raft_consensus.cc:1275] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Refusing update from remote peer 5345e5bcd1f14d5d88bac1e08965a1c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:21.324247  4834 consensus_queue.cc:1048] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:21.336205  4489 raft_consensus.cc:1275] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Refusing update from remote peer 5345e5bcd1f14d5d88bac1e08965a1c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:21.338076  4815 consensus_queue.cc:1048] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:23.618440  4356 auto_leader_rebalancer-test.cc:328] The leader distribution is 
I20260709 12:23:23.619599  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:23.627485  4566 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27b95197718748aca01ae1e6379245ca"
dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
mode: GRACEFUL
new_leader_uuid: "e3bbae884fe1434cbd0f42d01731773a"
 from {username='slave'} at 127.0.0.1:44354
I20260709 12:23:23.628264  4566 raft_consensus.cc:606] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Received request to transfer leadership to TS e3bbae884fe1434cbd0f42d01731773a
I20260709 12:23:23.631157  4566 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "93d1398e83ae46109eb2163858c93214"
dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
mode: GRACEFUL
new_leader_uuid: "498f39f6bf2844b3ae1bd842752f065f"
 from {username='slave'} at 127.0.0.1:44354
I20260709 12:23:23.631649  4566 raft_consensus.cc:606] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Received request to transfer leadership to TS 498f39f6bf2844b3ae1bd842752f065f
I20260709 12:23:23.634069  4566 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1"
dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
mode: GRACEFUL
new_leader_uuid: "e3bbae884fe1434cbd0f42d01731773a"
 from {username='slave'} at 127.0.0.1:44354
I20260709 12:23:23.634527  4566 raft_consensus.cc:606] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Received request to transfer leadership to TS e3bbae884fe1434cbd0f42d01731773a
I20260709 12:23:23.636797  4566 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "de101290db964f41b3fb73588fd5964f"
dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
mode: GRACEFUL
new_leader_uuid: "498f39f6bf2844b3ae1bd842752f065f"
 from {username='slave'} at 127.0.0.1:44354
I20260709 12:23:23.637257  4566 raft_consensus.cc:606] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 LEADER]: Received request to transfer leadership to TS 498f39f6bf2844b3ae1bd842752f065f
I20260709 12:23:23.639482  4566 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e"
dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
mode: GRACEFUL
new_leader_uuid: "498f39f6bf2844b3ae1bd842752f065f"
 from {username='slave'} at 127.0.0.1:44354
I20260709 12:23:23.640053  4566 raft_consensus.cc:606] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 LEADER]: Received request to transfer leadership to TS 498f39f6bf2844b3ae1bd842752f065f
I20260709 12:23:23.641078  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:23.641511  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:23.663015  4819 raft_consensus.cc:993] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4: : Instructing follower e3bbae884fe1434cbd0f42d01731773a to start an election
I20260709 12:23:23.663455  4819 raft_consensus.cc:1081] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Signalling peer e3bbae884fe1434cbd0f42d01731773a to start an election
I20260709 12:23:23.665069  4641 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1"
dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
 from {username='slave'} at 127.0.0.1:55120
I20260709 12:23:23.665518  4641 raft_consensus.cc:493] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:23.666067  4641 raft_consensus.cc:3060] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:23.670958  4641 raft_consensus.cc:515] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:23.672911  4641 leader_election.cc:290] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 4 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:23.673986  4489 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:23.674324  4566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c31fd08755e44186a93833dcf59e8ff1" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:23.674842  4489 raft_consensus.cc:3060] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:23.674844  4566 raft_consensus.cc:3055] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:23:23.675382  4566 raft_consensus.cc:740] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:23.676296  4566 consensus_queue.cc:260] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:23.677712  4566 raft_consensus.cc:3060] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:23.682027  4489 raft_consensus.cc:2468] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 4.
I20260709 12:23:23.683264  4603 leader_election.cc:304] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:23.684041  4843 raft_consensus.cc:2804] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:23:23.684151  4566 raft_consensus.cc:2468] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 4.
I20260709 12:23:23.685204  4843 raft_consensus.cc:697] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 4 LEADER]: Becoming Leader. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Running, Role: LEADER
I20260709 12:23:23.686259  4843 consensus_queue.cc:237] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:23.697588  4847 raft_consensus.cc:993] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4: : Instructing follower 498f39f6bf2844b3ae1bd842752f065f to start an election
I20260709 12:23:23.698276  4819 raft_consensus.cc:1081] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Signalling peer 498f39f6bf2844b3ae1bd842752f065f to start an election
I20260709 12:23:23.699987  4489 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "93d1398e83ae46109eb2163858c93214"
dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
 from {username='slave'} at 127.0.0.1:37980
I20260709 12:23:23.700588  4489 raft_consensus.cc:493] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:23.700985  4489 raft_consensus.cc:3060] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:23.700985  4388 catalog_manager.cc:5697] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a reported cstate change: term changed from 3 to 4, leader changed from 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2) to e3bbae884fe1434cbd0f42d01731773a (127.4.65.3). New cstate: current_term: 4 leader_uuid: "e3bbae884fe1434cbd0f42d01731773a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:23.708042  4489 raft_consensus.cc:515] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:23.712040  4640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d1398e83ae46109eb2163858c93214" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:23.712229  4566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d1398e83ae46109eb2163858c93214" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:23.712759  4640 raft_consensus.cc:3060] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:23.712848  4489 leader_election.cc:290] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 4 election: Requested vote from peers e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:23.712775  4566 raft_consensus.cc:3055] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:23:23.713488  4566 raft_consensus.cc:740] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:23.714483  4566 consensus_queue.cc:260] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:23.716224  4566 raft_consensus.cc:3060] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:23.720592  4640 raft_consensus.cc:2468] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 4.
I20260709 12:23:23.721093  4566 raft_consensus.cc:2468] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 4.
I20260709 12:23:23.722370  4456 leader_election.cc:304] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, 5345e5bcd1f14d5d88bac1e08965a1c4; no voters: 
I20260709 12:23:23.724318  4854 raft_consensus.cc:2804] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:23:23.725736  4854 raft_consensus.cc:697] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 4 LEADER]: Becoming Leader. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:23.726670  4854 consensus_queue.cc:237] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } }
I20260709 12:23:23.733857  4388 catalog_manager.cc:5697] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f reported cstate change: term changed from 3 to 4, leader changed from 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2) to 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1). New cstate: current_term: 4 leader_uuid: "498f39f6bf2844b3ae1bd842752f065f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:23.794502  4819 raft_consensus.cc:993] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4: : Instructing follower 498f39f6bf2844b3ae1bd842752f065f to start an election
I20260709 12:23:23.795100  4857 raft_consensus.cc:1081] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 LEADER]: Signalling peer 498f39f6bf2844b3ae1bd842752f065f to start an election
I20260709 12:23:23.796573  4857 raft_consensus.cc:993] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4: : Instructing follower e3bbae884fe1434cbd0f42d01731773a to start an election
I20260709 12:23:23.797039  4819 raft_consensus.cc:1081] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Signalling peer e3bbae884fe1434cbd0f42d01731773a to start an election
I20260709 12:23:23.797052  4489 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e"
dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
 from {username='slave'} at 127.0.0.1:37980
I20260709 12:23:23.797688  4489 raft_consensus.cc:493] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:23.798054  4489 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:23.798707  4640 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "27b95197718748aca01ae1e6379245ca"
dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
 from {username='slave'} at 127.0.0.1:55120
I20260709 12:23:23.799108  4640 raft_consensus.cc:493] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:23.799391  4640 raft_consensus.cc:3060] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:23.803079  4489 raft_consensus.cc:515] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:23.803094  4640 raft_consensus.cc:515] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:23.804934  4489 leader_election.cc:290] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 5 election: Requested vote from peers 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:23.805125  4640 leader_election.cc:290] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 4 election: Requested vote from peers 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043), 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693)
I20260709 12:23:23.805637  4490 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b95197718748aca01ae1e6379245ca" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
I20260709 12:23:23.805850  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:23.806226  4490 raft_consensus.cc:3060] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:23.806685  4641 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:23.807161  4566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27b95197718748aca01ae1e6379245ca" candidate_uuid: "e3bbae884fe1434cbd0f42d01731773a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:23.807605  4564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6c5711d5f1e451688f4fe68db57350e" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:23.807711  4566 raft_consensus.cc:3055] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:23:23.808254  4566 raft_consensus.cc:740] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:23.808285  4564 raft_consensus.cc:3055] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:23:23.809171  4564 raft_consensus.cc:740] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:23.809234  4566 consensus_queue.cc:260] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:23.810467  4566 raft_consensus.cc:3060] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:23.810105  4564 consensus_queue.cc:260] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:23.811342  4564 raft_consensus.cc:3060] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:23.812839  4490 raft_consensus.cc:2468] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 4.
I20260709 12:23:23.813458  4641 raft_consensus.cc:2468] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 5.
I20260709 12:23:23.813764  4603 leader_election.cc:304] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:23.814457  4843 raft_consensus.cc:2804] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:23:23.814888  4843 raft_consensus.cc:697] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 4 LEADER]: Becoming Leader. State: Replica: e3bbae884fe1434cbd0f42d01731773a, State: Running, Role: LEADER
I20260709 12:23:23.815861  4843 consensus_queue.cc:237] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:23.816321  4566 raft_consensus.cc:2468] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3bbae884fe1434cbd0f42d01731773a in term 4.
I20260709 12:23:23.818699  4564 raft_consensus.cc:2468] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 5.
I20260709 12:23:23.819540  4456 leader_election.cc:304] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:23.820318  4854 raft_consensus.cc:2804] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:23:23.820871  4854 raft_consensus.cc:697] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 5 LEADER]: Becoming Leader. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:23.821725  4854 consensus_queue.cc:237] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:23.824555  4388 catalog_manager.cc:5697] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a reported cstate change: term changed from 3 to 4, leader changed from 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2) to e3bbae884fe1434cbd0f42d01731773a (127.4.65.3). New cstate: current_term: 4 leader_uuid: "e3bbae884fe1434cbd0f42d01731773a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:23.829895  4387 catalog_manager.cc:5697] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f reported cstate change: term changed from 4 to 5, leader changed from 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2) to 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1). New cstate: current_term: 5 leader_uuid: "498f39f6bf2844b3ae1bd842752f065f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:23.831207  4819 raft_consensus.cc:993] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4: : Instructing follower 498f39f6bf2844b3ae1bd842752f065f to start an election
I20260709 12:23:23.831601  4847 raft_consensus.cc:1081] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 LEADER]: Signalling peer 498f39f6bf2844b3ae1bd842752f065f to start an election
I20260709 12:23:23.833139  4490 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "de101290db964f41b3fb73588fd5964f"
dest_uuid: "498f39f6bf2844b3ae1bd842752f065f"
 from {username='slave'} at 127.0.0.1:37980
I20260709 12:23:23.833564  4490 raft_consensus.cc:493] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:23.833945  4490 raft_consensus.cc:3060] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:23.838907  4490 raft_consensus.cc:515] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:23.840307  4490 leader_election.cc:290] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 3 election: Requested vote from peers 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2:38693), e3bbae884fe1434cbd0f42d01731773a (127.4.65.3:35891)
I20260709 12:23:23.840962  4564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de101290db964f41b3fb73588fd5964f" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4"
I20260709 12:23:23.841163  4641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de101290db964f41b3fb73588fd5964f" candidate_uuid: "498f39f6bf2844b3ae1bd842752f065f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e3bbae884fe1434cbd0f42d01731773a"
I20260709 12:23:23.841566  4564 raft_consensus.cc:3055] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:23.841862  4641 raft_consensus.cc:3060] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:23.841945  4564 raft_consensus.cc:740] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5345e5bcd1f14d5d88bac1e08965a1c4, State: Running, Role: LEADER
I20260709 12:23:23.842813  4564 consensus_queue.cc:260] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:23.843995  4564 raft_consensus.cc:3060] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:23.848240  4641 raft_consensus.cc:2468] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 3.
I20260709 12:23:23.849309  4456 leader_election.cc:304] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498f39f6bf2844b3ae1bd842752f065f, e3bbae884fe1434cbd0f42d01731773a; no voters: 
I20260709 12:23:23.849969  4854 raft_consensus.cc:2804] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:23:23.850113  4564 raft_consensus.cc:2468] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 498f39f6bf2844b3ae1bd842752f065f in term 3.
I20260709 12:23:23.850479  4854 raft_consensus.cc:697] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 3 LEADER]: Becoming Leader. State: Replica: 498f39f6bf2844b3ae1bd842752f065f, State: Running, Role: LEADER
I20260709 12:23:23.851435  4854 consensus_queue.cc:237] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } }
I20260709 12:23:23.858343  4387 catalog_manager.cc:5697] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f reported cstate change: term changed from 2 to 3, leader changed from 5345e5bcd1f14d5d88bac1e08965a1c4 (127.4.65.2) to 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1). New cstate: current_term: 3 leader_uuid: "498f39f6bf2844b3ae1bd842752f065f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 } health_report { overall_health: UNKNOWN } } }
W20260709 12:23:24.008442  4662 debug-util.cc:398] Leaking SignalData structure 0x7b080016d060 after lost signal to thread 4359
W20260709 12:23:24.009313  4662 debug-util.cc:398] Leaking SignalData structure 0x7b080016d740 after lost signal to thread 4422
W20260709 12:23:24.010196  4662 debug-util.cc:398] Leaking SignalData structure 0x7b08000f49a0 after lost signal to thread 4515
W20260709 12:23:24.010979  4662 debug-util.cc:398] Leaking SignalData structure 0x7b080014cc60 after lost signal to thread 4590
W20260709 12:23:24.011742  4662 debug-util.cc:398] Leaking SignalData structure 0x7b08000a1960 after lost signal to thread 4665
I20260709 12:23:24.128222  4564 raft_consensus.cc:1275] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Refusing update from remote peer e3bbae884fe1434cbd0f42d01731773a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:23:24.129601  4843 consensus_queue.cc:1048] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:23:24.152444  4490 raft_consensus.cc:1275] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Refusing update from remote peer e3bbae884fe1434cbd0f42d01731773a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:23:24.153841  4864 consensus_queue.cc:1048] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:24.206358  4564 raft_consensus.cc:1275] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Refusing update from remote peer 498f39f6bf2844b3ae1bd842752f065f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:23:24.207475  4854 consensus_queue.cc:1048] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:23:24.216687  4641 raft_consensus.cc:1275] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 4 FOLLOWER]: Refusing update from remote peer 498f39f6bf2844b3ae1bd842752f065f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:23:24.218466  4854 consensus_queue.cc:1048] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:23:24.219755  4566 raft_consensus.cc:1275] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 5 FOLLOWER]: Refusing update from remote peer 498f39f6bf2844b3ae1bd842752f065f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:23:24.221055  4854 consensus_queue.cc:1048] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:24.241868  4641 raft_consensus.cc:1275] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 5 FOLLOWER]: Refusing update from remote peer 498f39f6bf2844b3ae1bd842752f065f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:23:24.244454  4854 consensus_queue.cc:1048] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:24.271075  4490 raft_consensus.cc:1275] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Refusing update from remote peer e3bbae884fe1434cbd0f42d01731773a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:23:24.272456  4864 consensus_queue.cc:1048] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "498f39f6bf2844b3ae1bd842752f065f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 34043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:23:24.281278  4566 raft_consensus.cc:1275] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Refusing update from remote peer e3bbae884fe1434cbd0f42d01731773a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:23:24.282737  4843 consensus_queue.cc:1048] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:23:24.300608  4566 raft_consensus.cc:1275] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Refusing update from remote peer 498f39f6bf2844b3ae1bd842752f065f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:24.301851  4870 consensus_queue.cc:1048] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5345e5bcd1f14d5d88bac1e08965a1c4" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:24.311386  4640 raft_consensus.cc:1275] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Refusing update from remote peer 498f39f6bf2844b3ae1bd842752f065f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:24.312883  4855 consensus_queue.cc:1048] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3bbae884fe1434cbd0f42d01731773a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:23:24.642668  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:24.647554  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:24.647974  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:25.649125  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:25.656860  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:25.657402  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:26.658490  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:26.663650  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:26.664116  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:27.665632  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:27.671504  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:27.671955  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:28.673117  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:28.678030  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:28.678455  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:29.679476  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:29.685253  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:29.685663  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:30.686945  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:30.693203  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:30.693729  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:31.694976  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:31.700210  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:31.700685  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:32.701893  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:32.707211  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:32.707788  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:33.745893  4356 auto_leader_rebalancer-test.cc:340] The leader distribution is 
I20260709 12:23:33.746702  4356 tablet_server.cc:179] TabletServer@127.4.65.1:0 shutting down...
W20260709 12:23:33.753778  4529 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:23:33.759557  4529 consensus_peers.cc:597] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 -> Peer 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043): Couldn't send request to peer 498f39f6bf2844b3ae1bd842752f065f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:33.784260  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:23:33.785621  4356 tablet_replica.cc:333] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f: stopping tablet replica
I20260709 12:23:33.786983  4356 raft_consensus.cc:2243] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:33.787846  4356 raft_consensus.cc:2272] T 9bba194fd0b34047badbff2aea2f9414 P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.791366  4356 tablet_replica.cc:333] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f: stopping tablet replica
I20260709 12:23:33.792326  4356 raft_consensus.cc:2243] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:33.793037  4356 raft_consensus.cc:2272] T 7b3b696a9e3f4d479648b829ef3d167e P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.795293  4356 tablet_replica.cc:333] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f: stopping tablet replica
I20260709 12:23:33.795919  4356 raft_consensus.cc:2243] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:33.796466  4356 raft_consensus.cc:2272] T 27b95197718748aca01ae1e6379245ca P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.798322  4356 tablet_replica.cc:333] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f: stopping tablet replica
I20260709 12:23:33.798877  4356 raft_consensus.cc:2243] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:23:33.799702  4356 raft_consensus.cc:2272] T de101290db964f41b3fb73588fd5964f P 498f39f6bf2844b3ae1bd842752f065f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.802244  4356 tablet_replica.cc:333] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f: stopping tablet replica
I20260709 12:23:33.802825  4356 raft_consensus.cc:2243] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:33.803419  4356 raft_consensus.cc:2272] T f31c9f2f020a45e6b1077bb51b6dafde P 498f39f6bf2844b3ae1bd842752f065f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.805527  4356 tablet_replica.cc:333] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f: stopping tablet replica
I20260709 12:23:33.806028  4356 raft_consensus.cc:2243] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:23:33.806825  4356 raft_consensus.cc:2272] T f6c5711d5f1e451688f4fe68db57350e P 498f39f6bf2844b3ae1bd842752f065f [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.808668  4356 tablet_replica.cc:333] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f: stopping tablet replica
I20260709 12:23:33.809139  4356 raft_consensus.cc:2243] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:23:33.809978  4356 raft_consensus.cc:2272] T 93d1398e83ae46109eb2163858c93214 P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.811628  4356 tablet_replica.cc:333] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f: stopping tablet replica
I20260709 12:23:33.812104  4356 raft_consensus.cc:2243] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:33.812625  4356 raft_consensus.cc:2272] T a987ec62f8c04b9aad527a2b4c8c00cc P 498f39f6bf2844b3ae1bd842752f065f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.814224  4356 tablet_replica.cc:333] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f: stopping tablet replica
I20260709 12:23:33.814706  4356 raft_consensus.cc:2243] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:33.815100  4356 raft_consensus.cc:2272] T c31fd08755e44186a93833dcf59e8ff1 P 498f39f6bf2844b3ae1bd842752f065f [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.907068  4356 tablet_server.cc:196] TabletServer@127.4.65.1:0 shutdown complete.
I20260709 12:23:33.930684  4356 tablet_server.cc:179] TabletServer@127.4.65.2:0 shutting down...
I20260709 12:23:33.967738  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:23:33.968597  4356 tablet_replica.cc:333] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4: stopping tablet replica
I20260709 12:23:33.969916  4356 raft_consensus.cc:2243] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:33.970467  4356 raft_consensus.cc:2272] T 7b3b696a9e3f4d479648b829ef3d167e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.972244  4356 tablet_replica.cc:333] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4: stopping tablet replica
I20260709 12:23:33.972895  4356 raft_consensus.cc:2243] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:33.973372  4356 raft_consensus.cc:2272] T 27b95197718748aca01ae1e6379245ca P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.975098  4356 tablet_replica.cc:333] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4: stopping tablet replica
I20260709 12:23:33.975696  4356 raft_consensus.cc:2243] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:23:33.976789  4356 raft_consensus.cc:2272] T 9bba194fd0b34047badbff2aea2f9414 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.978766  4356 tablet_replica.cc:333] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4: stopping tablet replica
I20260709 12:23:33.979367  4356 raft_consensus.cc:2243] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:33.979884  4356 raft_consensus.cc:2272] T de101290db964f41b3fb73588fd5964f P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.981844  4356 tablet_replica.cc:333] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4: stopping tablet replica
W20260709 12:23:33.982301  4603 consensus_peers.cc:597] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a -> Peer 498f39f6bf2844b3ae1bd842752f065f (127.4.65.1:34043): Couldn't send request to peer 498f39f6bf2844b3ae1bd842752f065f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:34043: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:23:33.982497  4356 raft_consensus.cc:2243] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:23:33.983377  4356 raft_consensus.cc:2272] T f31c9f2f020a45e6b1077bb51b6dafde P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.985086  4356 tablet_replica.cc:333] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4: stopping tablet replica
I20260709 12:23:33.985533  4356 raft_consensus.cc:2243] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:33.985909  4356 raft_consensus.cc:2272] T 93d1398e83ae46109eb2163858c93214 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.987476  4356 tablet_replica.cc:333] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4: stopping tablet replica
I20260709 12:23:33.987895  4356 raft_consensus.cc:2243] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:23:33.988687  4356 raft_consensus.cc:2272] T a987ec62f8c04b9aad527a2b4c8c00cc P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.990159  4356 tablet_replica.cc:333] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4: stopping tablet replica
I20260709 12:23:33.990576  4356 raft_consensus.cc:2243] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:33.990994  4356 raft_consensus.cc:2272] T f6c5711d5f1e451688f4fe68db57350e P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:33.992630  4356 tablet_replica.cc:333] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4: stopping tablet replica
I20260709 12:23:33.993010  4356 raft_consensus.cc:2243] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:33.993376  4356 raft_consensus.cc:2272] T c31fd08755e44186a93833dcf59e8ff1 P 5345e5bcd1f14d5d88bac1e08965a1c4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:34.024785  4356 tablet_server.cc:196] TabletServer@127.4.65.2:0 shutdown complete.
I20260709 12:23:34.044843  4356 tablet_server.cc:179] TabletServer@127.4.65.3:0 shutting down...
I20260709 12:23:34.079535  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:23:34.080655  4356 tablet_replica.cc:333] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a: stopping tablet replica
I20260709 12:23:34.081321  4356 raft_consensus.cc:2243] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:34.081889  4356 raft_consensus.cc:2272] T 9bba194fd0b34047badbff2aea2f9414 P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:34.083999  4356 tablet_replica.cc:333] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a: stopping tablet replica
I20260709 12:23:34.084667  4356 raft_consensus.cc:2243] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:34.085168  4356 raft_consensus.cc:2272] T 93d1398e83ae46109eb2163858c93214 P e3bbae884fe1434cbd0f42d01731773a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:34.087083  4356 tablet_replica.cc:333] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a: stopping tablet replica
I20260709 12:23:34.087625  4356 raft_consensus.cc:2243] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:34.088063  4356 raft_consensus.cc:2272] T f6c5711d5f1e451688f4fe68db57350e P e3bbae884fe1434cbd0f42d01731773a [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:34.090202  4356 tablet_replica.cc:333] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a: stopping tablet replica
I20260709 12:23:34.090737  4356 raft_consensus.cc:2243] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:34.091287  4356 raft_consensus.cc:2272] T a987ec62f8c04b9aad527a2b4c8c00cc P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:34.093993  4356 tablet_replica.cc:333] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a: stopping tablet replica
I20260709 12:23:34.094715  4356 raft_consensus.cc:2243] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:23:34.095631  4356 raft_consensus.cc:2272] T 7b3b696a9e3f4d479648b829ef3d167e P e3bbae884fe1434cbd0f42d01731773a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:34.097630  4356 tablet_replica.cc:333] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a: stopping tablet replica
I20260709 12:23:34.098161  4356 raft_consensus.cc:2243] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:23:34.099035  4356 raft_consensus.cc:2272] T 27b95197718748aca01ae1e6379245ca P e3bbae884fe1434cbd0f42d01731773a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:34.100926  4356 tablet_replica.cc:333] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a: stopping tablet replica
I20260709 12:23:34.101462  4356 raft_consensus.cc:2243] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:34.101943  4356 raft_consensus.cc:2272] T f31c9f2f020a45e6b1077bb51b6dafde P e3bbae884fe1434cbd0f42d01731773a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:34.103781  4356 tablet_replica.cc:333] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a: stopping tablet replica
I20260709 12:23:34.104271  4356 raft_consensus.cc:2243] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:23:34.104809  4356 raft_consensus.cc:2272] T de101290db964f41b3fb73588fd5964f P e3bbae884fe1434cbd0f42d01731773a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:34.106832  4356 tablet_replica.cc:333] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a: stopping tablet replica
I20260709 12:23:34.107354  4356 raft_consensus.cc:2243] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:23:34.108287  4356 raft_consensus.cc:2272] T c31fd08755e44186a93833dcf59e8ff1 P e3bbae884fe1434cbd0f42d01731773a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:34.155596  4356 tablet_server.cc:196] TabletServer@127.4.65.3:0 shutdown complete.
I20260709 12:23:34.174175  4356 master.cc:562] Master@127.4.65.62:40331 shutting down...
I20260709 12:23:34.302028  4356 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:23:34.302642  4356 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:23:34.302928  4356 tablet_replica.cc:333] T 00000000000000000000000000000000 P ea046ee8f6f442d082016860d60f40b5: stopping tablet replica
I20260709 12:23:34.322145  4356 master.cc:584] Master@127.4.65.62:40331 shutdown complete.
498f39f6bf2844b3ae1bd842752f065f  4
5345e5bcd1f14d5d88bac1e08965a1c4  4
e3bbae884fe1434cbd0f42d01731773a  1
498f39f6bf2844b3ae1bd842752f065f  3
5345e5bcd1f14d5d88bac1e08965a1c4  3
e3bbae884fe1434cbd0f42d01731773a  3
498f39f6bf2844b3ae1bd842752f065f  0
5345e5bcd1f14d5d88bac1e08965a1c4  8
e3bbae884fe1434cbd0f42d01731773a  1
498f39f6bf2844b3ae1bd842752f065f  3
5345e5bcd1f14d5d88bac1e08965a1c4  3
e3bbae884fe1434cbd0f42d01731773a  3
[       OK ] LeaderRebalancerTest.FunctionalTestForDivided (30898 ms)
[ RUN      ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260709 12:23:34.366237  4356 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.65.62:43375
I20260709 12:23:34.367341  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:23:34.373698  4888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:23:34.373910  4889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:23:34.374825  4891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:23:34.376773  4356 server_base.cc:1061] running on GCE node
I20260709 12:23:34.377775  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:23:34.377975  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:23:34.378090  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783599814378080 us; error 0 us; skew 500 ppm
I20260709 12:23:34.378552  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:23:34.381745  4356 webserver.cc:533] Webserver started at http://127.4.65.62:33373/ using document root <none> and password file <none>
I20260709 12:23:34.382320  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:23:34.382520  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:23:34.382781  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:23:34.384124  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/master-0-root/instance:
uuid: "6aeedc0e365d49958b13951a04acf871"
format_stamp: "Formatted at 2026-07-09 12:23:34 on dist-test-slave-l1c9"
I20260709 12:23:34.389745  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:23:34.393726  4896 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:34.394697  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20260709 12:23:34.394963  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/master-0-root
uuid: "6aeedc0e365d49958b13951a04acf871"
format_stamp: "Formatted at 2026-07-09 12:23:34 on dist-test-slave-l1c9"
I20260709 12:23:34.395210  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:23:34.406194  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:23:34.407512  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:23:34.451930  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.62:43375
I20260709 12:23:34.452059  4947 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.62:43375 every 8 connection(s)
I20260709 12:23:34.455914  4948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:34.465407  4948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871: Bootstrap starting.
I20260709 12:23:34.470177  4948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:34.474467  4948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871: No bootstrap required, opened a new log
I20260709 12:23:34.476722  4948 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aeedc0e365d49958b13951a04acf871" member_type: VOTER }
I20260709 12:23:34.477159  4948 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:34.477443  4948 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aeedc0e365d49958b13951a04acf871, State: Initialized, Role: FOLLOWER
I20260709 12:23:34.478099  4948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aeedc0e365d49958b13951a04acf871" member_type: VOTER }
I20260709 12:23:34.478577  4948 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:23:34.478797  4948 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:23:34.479084  4948 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:34.483202  4948 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aeedc0e365d49958b13951a04acf871" member_type: VOTER }
I20260709 12:23:34.483738  4948 leader_election.cc:304] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6aeedc0e365d49958b13951a04acf871; no voters: 
I20260709 12:23:34.484934  4948 leader_election.cc:290] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:23:34.485385  4951 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:34.486719  4951 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [term 1 LEADER]: Becoming Leader. State: Replica: 6aeedc0e365d49958b13951a04acf871, State: Running, Role: LEADER
I20260709 12:23:34.487445  4951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aeedc0e365d49958b13951a04acf871" member_type: VOTER }
I20260709 12:23:34.488124  4948 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:23:34.492175  4952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6aeedc0e365d49958b13951a04acf871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aeedc0e365d49958b13951a04acf871" member_type: VOTER } }
I20260709 12:23:34.492388  4953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6aeedc0e365d49958b13951a04acf871. Latest consensus state: current_term: 1 leader_uuid: "6aeedc0e365d49958b13951a04acf871" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6aeedc0e365d49958b13951a04acf871" member_type: VOTER } }
I20260709 12:23:34.493108  4952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [sys.catalog]: This master's current role is: LEADER
I20260709 12:23:34.493280  4953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [sys.catalog]: This master's current role is: LEADER
I20260709 12:23:34.499414  4957 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:23:34.507750  4957 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:23:34.509045  4356 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:23:34.517735  4957 catalog_manager.cc:1383] Generated new cluster ID: 2c5ccae0a9354dcc84c5e98412930638
I20260709 12:23:34.518069  4957 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:23:34.533608  4957 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:23:34.534917  4957 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:23:34.547168  4957 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871: Generated new TSK 0
I20260709 12:23:34.547780  4957 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:23:34.575851  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:23:34.582178  4969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:23:34.582702  4970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:23:34.584962  4972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:23:34.585354  4356 server_base.cc:1061] running on GCE node
I20260709 12:23:34.586351  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:23:34.586526  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:23:34.586728  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783599814586699 us; error 0 us; skew 500 ppm
I20260709 12:23:34.587386  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:23:34.590009  4356 webserver.cc:533] Webserver started at http://127.4.65.1:42449/ using document root <none> and password file <none>
I20260709 12:23:34.590487  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:23:34.590667  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:23:34.590946  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:23:34.592053  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-0-root/instance:
uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
format_stamp: "Formatted at 2026-07-09 12:23:34 on dist-test-slave-l1c9"
I20260709 12:23:34.596321  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20260709 12:23:34.600513  4977 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:34.601406  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:23:34.601644  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-0-root
uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
format_stamp: "Formatted at 2026-07-09 12:23:34 on dist-test-slave-l1c9"
I20260709 12:23:34.601883  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:23:34.617818  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:23:34.618999  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:23:34.620560  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:23:34.622874  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:23:34.623045  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:34.623319  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:23:34.623490  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:34.667562  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.1:42619
I20260709 12:23:34.667753  5039 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.1:42619 every 8 connection(s)
I20260709 12:23:34.672462  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:23:34.679076  5044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:23:34.683435  5045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:23:34.684460  4356 server_base.cc:1061] running on GCE node
W20260709 12:23:34.685712  5047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:23:34.686769  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:23:34.686977  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:23:34.687183  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783599814687168 us; error 0 us; skew 500 ppm
I20260709 12:23:34.687778  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:23:34.690564  4356 webserver.cc:533] Webserver started at http://127.4.65.2:36007/ using document root <none> and password file <none>
I20260709 12:23:34.691093  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:23:34.691303  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:23:34.691604  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:23:34.693099  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-1-root/instance:
uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
format_stamp: "Formatted at 2026-07-09 12:23:34 on dist-test-slave-l1c9"
I20260709 12:23:34.693274  5040 heartbeater.cc:344] Connected to a master server at 127.4.65.62:43375
I20260709 12:23:34.693723  5040 heartbeater.cc:461] Registering TS with master...
I20260709 12:23:34.694420  5040 heartbeater.cc:507] Master 127.4.65.62:43375 requested a full tablet report, sending...
I20260709 12:23:34.696575  4913 ts_manager.cc:194] Registered new tserver with Master: 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619)
I20260709 12:23:34.698587  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:23:34.698670  4913 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47222
I20260709 12:23:34.703486  5052 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:34.704733  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20260709 12:23:34.705101  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-1-root
uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
format_stamp: "Formatted at 2026-07-09 12:23:34 on dist-test-slave-l1c9"
I20260709 12:23:34.705404  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:23:34.724594  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:23:34.725853  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:23:34.727398  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:23:34.730270  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:23:34.730486  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:34.730981  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:23:34.731137  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:34.774830  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.2:43513
I20260709 12:23:34.774936  5114 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.2:43513 every 8 connection(s)
I20260709 12:23:34.779759  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:23:34.788424  5119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:23:34.791782  5118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:23:34.793684  4356 server_base.cc:1061] running on GCE node
W20260709 12:23:34.794912  5121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:23:34.795035  5115 heartbeater.cc:344] Connected to a master server at 127.4.65.62:43375
I20260709 12:23:34.795531  5115 heartbeater.cc:461] Registering TS with master...
I20260709 12:23:34.796036  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:23:34.796262  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:23:34.796310  5115 heartbeater.cc:507] Master 127.4.65.62:43375 requested a full tablet report, sending...
I20260709 12:23:34.796561  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783599814796539 us; error 0 us; skew 500 ppm
I20260709 12:23:34.797343  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:23:34.798760  4913 ts_manager.cc:194] Registered new tserver with Master: e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:34.800062  4356 webserver.cc:533] Webserver started at http://127.4.65.3:40613/ using document root <none> and password file <none>
I20260709 12:23:34.800498  4913 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47228
I20260709 12:23:34.800622  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:23:34.800932  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:23:34.801209  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:23:34.802489  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-2-root/instance:
uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
format_stamp: "Formatted at 2026-07-09 12:23:34 on dist-test-slave-l1c9"
I20260709 12:23:34.807607  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.004s
I20260709 12:23:34.811698  5126 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:34.812762  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:23:34.813093  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-2-root
uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
format_stamp: "Formatted at 2026-07-09 12:23:34 on dist-test-slave-l1c9"
I20260709 12:23:34.813400  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599783304823-4356-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:23:34.832746  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:23:34.833971  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:23:34.835467  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:23:34.837778  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:23:34.837993  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:34.838240  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:23:34.838446  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:23:34.881531  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.3:39025
I20260709 12:23:34.881740  5188 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.3:39025 every 8 connection(s)
I20260709 12:23:34.897619  5189 heartbeater.cc:344] Connected to a master server at 127.4.65.62:43375
I20260709 12:23:34.898020  5189 heartbeater.cc:461] Registering TS with master...
I20260709 12:23:34.898677  5189 heartbeater.cc:507] Master 127.4.65.62:43375 requested a full tablet report, sending...
I20260709 12:23:34.901015  4913 ts_manager.cc:194] Registered new tserver with Master: a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:34.901482  4356 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01647657s
I20260709 12:23:34.902419  4913 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47236
I20260709 12:23:34.930397  4913 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47240:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:23:34.933597  4913 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:23:35.004590  5154 tablet_service.cc:1511] Processing CreateTablet for tablet 90480fe63f94406d93bb183ceba4ae67 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:23:35.005954  5153 tablet_service.cc:1511] Processing CreateTablet for tablet c91e288907924a9d99440800ed9f6d62 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:23:35.006414  5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90480fe63f94406d93bb183ceba4ae67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.007200  5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91e288907924a9d99440800ed9f6d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.012521  5152 tablet_service.cc:1511] Processing CreateTablet for tablet 16f1aba11d864c94a4b511ccd0be93bc (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:23:35.013904  5152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f1aba11d864c94a4b511ccd0be93bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.018715  5151 tablet_service.cc:1511] Processing CreateTablet for tablet 74c4d6a9653848a7a8951cb7807cd932 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:23:35.020313  5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c4d6a9653848a7a8951cb7807cd932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.023674  5150 tablet_service.cc:1511] Processing CreateTablet for tablet 9241c592ecc04d28b06a4bfc7acf2697 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:23:35.025265  5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9241c592ecc04d28b06a4bfc7acf2697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.027456  5005 tablet_service.cc:1511] Processing CreateTablet for tablet 90480fe63f94406d93bb183ceba4ae67 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:23:35.028858  5005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90480fe63f94406d93bb183ceba4ae67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.030303  5149 tablet_service.cc:1511] Processing CreateTablet for tablet 90561bb0627a4317adcaa4dd18688b0e (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:23:35.031998  5149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90561bb0627a4317adcaa4dd18688b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.033231  5148 tablet_service.cc:1511] Processing CreateTablet for tablet ef2cf9f651e345cc8102ae4d8235bf1c (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:23:35.034646  5148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2cf9f651e345cc8102ae4d8235bf1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.048225  5004 tablet_service.cc:1511] Processing CreateTablet for tablet c91e288907924a9d99440800ed9f6d62 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:23:35.048954  5003 tablet_service.cc:1511] Processing CreateTablet for tablet 16f1aba11d864c94a4b511ccd0be93bc (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:23:35.049832  5004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91e288907924a9d99440800ed9f6d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.050487  5003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f1aba11d864c94a4b511ccd0be93bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.052160  5147 tablet_service.cc:1511] Processing CreateTablet for tablet d59f79cb8ed5429a9ca0d1e2ebe3f935 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:23:35.053524  5002 tablet_service.cc:1511] Processing CreateTablet for tablet 74c4d6a9653848a7a8951cb7807cd932 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:23:35.053951  5147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d59f79cb8ed5429a9ca0d1e2ebe3f935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.054833  5002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c4d6a9653848a7a8951cb7807cd932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.057780  5001 tablet_service.cc:1511] Processing CreateTablet for tablet 9241c592ecc04d28b06a4bfc7acf2697 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:23:35.058257  5000 tablet_service.cc:1511] Processing CreateTablet for tablet 90561bb0627a4317adcaa4dd18688b0e (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:23:35.059150  5001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9241c592ecc04d28b06a4bfc7acf2697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.065899  5000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90561bb0627a4317adcaa4dd18688b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.064005  4998 tablet_service.cc:1511] Processing CreateTablet for tablet d59f79cb8ed5429a9ca0d1e2ebe3f935 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:23:35.067625  4998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d59f79cb8ed5429a9ca0d1e2ebe3f935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.065176  4999 tablet_service.cc:1511] Processing CreateTablet for tablet ef2cf9f651e345cc8102ae4d8235bf1c (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:23:35.069721  4999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2cf9f651e345cc8102ae4d8235bf1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.089974  5080 tablet_service.cc:1511] Processing CreateTablet for tablet 90480fe63f94406d93bb183ceba4ae67 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:23:35.091717  5080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90480fe63f94406d93bb183ceba4ae67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.093956  5079 tablet_service.cc:1511] Processing CreateTablet for tablet c91e288907924a9d99440800ed9f6d62 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:23:35.095633  5079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91e288907924a9d99440800ed9f6d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.097862  5078 tablet_service.cc:1511] Processing CreateTablet for tablet 16f1aba11d864c94a4b511ccd0be93bc (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:23:35.101424  5077 tablet_service.cc:1511] Processing CreateTablet for tablet 74c4d6a9653848a7a8951cb7807cd932 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:23:35.104272  5076 tablet_service.cc:1511] Processing CreateTablet for tablet 9241c592ecc04d28b06a4bfc7acf2697 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:23:35.106257  5078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f1aba11d864c94a4b511ccd0be93bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.109433  5005 tablet_service.cc:1511] Processing CreateTablet for tablet 588e8402cd9948778e37e9cc558923d3 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:23:35.110812  5005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588e8402cd9948778e37e9cc558923d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.114437  5146 tablet_service.cc:1511] Processing CreateTablet for tablet 588e8402cd9948778e37e9cc558923d3 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:23:35.115908  5146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588e8402cd9948778e37e9cc558923d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.115628  5074 tablet_service.cc:1511] Processing CreateTablet for tablet ef2cf9f651e345cc8102ae4d8235bf1c (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:23:35.115832  4997 tablet_service.cc:1511] Processing CreateTablet for tablet a728de06487c46648df7ae5eadcd7d9a (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:23:35.116938  5074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2cf9f651e345cc8102ae4d8235bf1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.117189  4997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a728de06487c46648df7ae5eadcd7d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.114437  5075 tablet_service.cc:1511] Processing CreateTablet for tablet 90561bb0627a4317adcaa4dd18688b0e (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:23:35.122373  5073 tablet_service.cc:1511] Processing CreateTablet for tablet d59f79cb8ed5429a9ca0d1e2ebe3f935 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:23:35.123922  5075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90561bb0627a4317adcaa4dd18688b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.125644  5072 tablet_service.cc:1511] Processing CreateTablet for tablet 588e8402cd9948778e37e9cc558923d3 (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:23:35.127099  5072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588e8402cd9948778e37e9cc558923d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.130402  5071 tablet_service.cc:1511] Processing CreateTablet for tablet a728de06487c46648df7ae5eadcd7d9a (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:23:35.131860  5071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a728de06487c46648df7ae5eadcd7d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.132795  5073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d59f79cb8ed5429a9ca0d1e2ebe3f935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.135916  5148 tablet_service.cc:1511] Processing CreateTablet for tablet a728de06487c46648df7ae5eadcd7d9a (DEFAULT_TABLE table=test-workload [id=bd7b078a504c4b4189fc1efe381f7cb1]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:23:35.136709  5077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c4d6a9653848a7a8951cb7807cd932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.148696  5076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9241c592ecc04d28b06a4bfc7acf2697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.171059  5148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a728de06487c46648df7ae5eadcd7d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:23:35.181355  5204 tablet_bootstrap.cc:492] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101: Bootstrap starting.
I20260709 12:23:35.181355  5203 tablet_bootstrap.cc:492] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b: Bootstrap starting.
I20260709 12:23:35.206559  5205 tablet_bootstrap.cc:492] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d: Bootstrap starting.
I20260709 12:23:35.220484  5203 tablet_bootstrap.cc:654] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.223225  5204 tablet_bootstrap.cc:654] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.225095  5203 tablet_bootstrap.cc:492] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b: No bootstrap required, opened a new log
I20260709 12:23:35.225622  5203 ts_tablet_manager.cc:1403] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b: Time spent bootstrapping tablet: real 0.045s	user 0.014s	sys 0.000s
I20260709 12:23:35.228091  5205 tablet_bootstrap.cc:654] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.228657  5203 raft_consensus.cc:359] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:35.229552  5203 raft_consensus.cc:385] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.230062  5203 raft_consensus.cc:740] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.231150  5203 consensus_queue.cc:260] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:35.235909  5204 tablet_bootstrap.cc:492] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101: No bootstrap required, opened a new log
I20260709 12:23:35.236327  5203 ts_tablet_manager.cc:1434] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:23:35.236446  5204 ts_tablet_manager.cc:1403] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent bootstrapping tablet: real 0.055s	user 0.013s	sys 0.008s
I20260709 12:23:35.236611  5205 tablet_bootstrap.cc:492] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d: No bootstrap required, opened a new log
I20260709 12:23:35.236910  5189 heartbeater.cc:499] Master 127.4.65.62:43375 was elected leader, sending a full tablet report...
I20260709 12:23:35.237154  5205 ts_tablet_manager.cc:1403] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent bootstrapping tablet: real 0.031s	user 0.011s	sys 0.004s
I20260709 12:23:35.237547  5203 tablet_bootstrap.cc:492] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b: Bootstrap starting.
I20260709 12:23:35.239542  5204 raft_consensus.cc:359] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.240283  5204 raft_consensus.cc:385] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.240796  5204 raft_consensus.cc:740] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.240228  5205 raft_consensus.cc:359] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.241246  5205 raft_consensus.cc:385] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.241622  5205 raft_consensus.cc:740] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.241616  5204 consensus_queue.cc:260] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.242775  5205 consensus_queue.cc:260] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.244959  5203 tablet_bootstrap.cc:654] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.250347  5205 ts_tablet_manager.cc:1434] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.004s
I20260709 12:23:35.251483  5115 heartbeater.cc:499] Master 127.4.65.62:43375 was elected leader, sending a full tablet report...
I20260709 12:23:35.261261  5205 tablet_bootstrap.cc:492] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d: Bootstrap starting.
I20260709 12:23:35.270015  5040 heartbeater.cc:499] Master 127.4.65.62:43375 was elected leader, sending a full tablet report...
I20260709 12:23:35.270398  5204 ts_tablet_manager.cc:1434] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent starting tablet: real 0.033s	user 0.025s	sys 0.007s
I20260709 12:23:35.271389  5204 tablet_bootstrap.cc:492] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101: Bootstrap starting.
I20260709 12:23:35.272802  5203 tablet_bootstrap.cc:492] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b: No bootstrap required, opened a new log
I20260709 12:23:35.273085  5205 tablet_bootstrap.cc:654] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.273311  5203 ts_tablet_manager.cc:1403] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b: Time spent bootstrapping tablet: real 0.036s	user 0.020s	sys 0.015s
I20260709 12:23:35.276813  5203 raft_consensus.cc:359] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.277889  5203 raft_consensus.cc:385] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.277904  5204 tablet_bootstrap.cc:654] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.278383  5203 raft_consensus.cc:740] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.280670  5203 consensus_queue.cc:260] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.283636  5203 ts_tablet_manager.cc:1434] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:23:35.284716  5203 tablet_bootstrap.cc:492] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b: Bootstrap starting.
I20260709 12:23:35.288487  5205 tablet_bootstrap.cc:492] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d: No bootstrap required, opened a new log
I20260709 12:23:35.289006  5205 ts_tablet_manager.cc:1403] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent bootstrapping tablet: real 0.028s	user 0.018s	sys 0.003s
I20260709 12:23:35.291213  5203 tablet_bootstrap.cc:654] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.292021  5205 raft_consensus.cc:359] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.292836  5205 raft_consensus.cc:385] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.293146  5205 raft_consensus.cc:740] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.294155  5205 consensus_queue.cc:260] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.297739  5205 ts_tablet_manager.cc:1434] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.002s
I20260709 12:23:35.299042  5205 tablet_bootstrap.cc:492] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d: Bootstrap starting.
I20260709 12:23:35.308458  5205 tablet_bootstrap.cc:654] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.343714  5204 tablet_bootstrap.cc:492] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101: No bootstrap required, opened a new log
I20260709 12:23:35.344408  5204 ts_tablet_manager.cc:1403] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent bootstrapping tablet: real 0.073s	user 0.022s	sys 0.048s
I20260709 12:23:35.346223  5203 tablet_bootstrap.cc:492] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b: No bootstrap required, opened a new log
I20260709 12:23:35.346850  5203 ts_tablet_manager.cc:1403] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b: Time spent bootstrapping tablet: real 0.062s	user 0.027s	sys 0.034s
I20260709 12:23:35.347376  5204 raft_consensus.cc:359] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:35.348138  5204 raft_consensus.cc:385] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.348456  5204 raft_consensus.cc:740] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.349447  5204 consensus_queue.cc:260] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:35.350338  5203 raft_consensus.cc:359] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.351169  5203 raft_consensus.cc:385] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.351544  5203 raft_consensus.cc:740] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.351774  5204 ts_tablet_manager.cc:1434] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:23:35.352818  5204 tablet_bootstrap.cc:492] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101: Bootstrap starting.
I20260709 12:23:35.352463  5203 consensus_queue.cc:260] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.354996  5203 ts_tablet_manager.cc:1434] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:23:35.355976  5203 tablet_bootstrap.cc:492] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b: Bootstrap starting.
I20260709 12:23:35.358230  5205 tablet_bootstrap.cc:492] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d: No bootstrap required, opened a new log
I20260709 12:23:35.358718  5205 ts_tablet_manager.cc:1403] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent bootstrapping tablet: real 0.060s	user 0.015s	sys 0.023s
I20260709 12:23:35.361562  5205 raft_consensus.cc:359] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.362324  5205 raft_consensus.cc:385] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.362620  5205 raft_consensus.cc:740] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.362965  5203 tablet_bootstrap.cc:654] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.363338  5205 consensus_queue.cc:260] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.364492  5204 tablet_bootstrap.cc:654] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.365651  5205 ts_tablet_manager.cc:1434] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:23:35.366644  5205 tablet_bootstrap.cc:492] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d: Bootstrap starting.
I20260709 12:23:35.374801  5203 tablet_bootstrap.cc:492] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b: No bootstrap required, opened a new log
I20260709 12:23:35.375345  5203 ts_tablet_manager.cc:1403] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b: Time spent bootstrapping tablet: real 0.020s	user 0.012s	sys 0.004s
I20260709 12:23:35.376505  5205 tablet_bootstrap.cc:654] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.378518  5203 raft_consensus.cc:359] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.379313  5203 raft_consensus.cc:385] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.379654  5203 raft_consensus.cc:740] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.381300  5203 consensus_queue.cc:260] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.383493  5204 tablet_bootstrap.cc:492] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101: No bootstrap required, opened a new log
I20260709 12:23:35.384013  5204 ts_tablet_manager.cc:1403] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent bootstrapping tablet: real 0.031s	user 0.008s	sys 0.014s
I20260709 12:23:35.384280  5203 ts_tablet_manager.cc:1434] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260709 12:23:35.385363  5203 tablet_bootstrap.cc:492] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b: Bootstrap starting.
I20260709 12:23:35.386915  5205 tablet_bootstrap.cc:492] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d: No bootstrap required, opened a new log
I20260709 12:23:35.387362  5205 ts_tablet_manager.cc:1403] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent bootstrapping tablet: real 0.021s	user 0.006s	sys 0.010s
I20260709 12:23:35.387197  5204 raft_consensus.cc:359] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.388043  5204 raft_consensus.cc:385] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.388444  5204 raft_consensus.cc:740] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.389460  5204 consensus_queue.cc:260] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.390298  5205 raft_consensus.cc:359] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.391039  5205 raft_consensus.cc:385] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.391335  5205 raft_consensus.cc:740] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.392249  5205 consensus_queue.cc:260] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.394030  5203 tablet_bootstrap.cc:654] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.394186  5204 ts_tablet_manager.cc:1434] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:23:35.394711  5205 ts_tablet_manager.cc:1434] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:23:35.395288  5204 tablet_bootstrap.cc:492] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101: Bootstrap starting.
I20260709 12:23:35.395707  5205 tablet_bootstrap.cc:492] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d: Bootstrap starting.
I20260709 12:23:35.396250  5211 raft_consensus.cc:493] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.396834  5211 raft_consensus.cc:515] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.402904  5204 tablet_bootstrap.cc:654] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.405856  5205 tablet_bootstrap.cc:654] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.408915  5211 leader_election.cc:290] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:35.443710  5203 tablet_bootstrap.cc:492] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b: No bootstrap required, opened a new log
I20260709 12:23:35.444514  5203 ts_tablet_manager.cc:1403] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b: Time spent bootstrapping tablet: real 0.059s	user 0.018s	sys 0.032s
I20260709 12:23:35.444682  5205 tablet_bootstrap.cc:492] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d: No bootstrap required, opened a new log
I20260709 12:23:35.445336  5205 ts_tablet_manager.cc:1403] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent bootstrapping tablet: real 0.050s	user 0.008s	sys 0.006s
I20260709 12:23:35.449684  5203 raft_consensus.cc:359] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.450729  5203 raft_consensus.cc:385] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.451169  5203 raft_consensus.cc:740] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.452297  5203 consensus_queue.cc:260] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.454130  5015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588e8402cd9948778e37e9cc558923d3" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" is_pre_election: true
W20260709 12:23:35.455804  5056 leader_election.cc:343] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:35.456238  5205 raft_consensus.cc:359] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.456956  5205 raft_consensus.cc:385] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.457238  5205 raft_consensus.cc:740] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.458077  5205 consensus_queue.cc:260] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.460532  5205 ts_tablet_manager.cc:1434] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.003s
I20260709 12:23:35.461464  5205 tablet_bootstrap.cc:492] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d: Bootstrap starting.
I20260709 12:23:35.463660  5204 tablet_bootstrap.cc:492] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101: No bootstrap required, opened a new log
I20260709 12:23:35.464125  5204 ts_tablet_manager.cc:1403] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent bootstrapping tablet: real 0.069s	user 0.033s	sys 0.029s
I20260709 12:23:35.465323  5203 ts_tablet_manager.cc:1434] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.001s
I20260709 12:23:35.466892  5204 raft_consensus.cc:359] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.468928  5204 raft_consensus.cc:385] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.469242  5204 raft_consensus.cc:740] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.469699  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588e8402cd9948778e37e9cc558923d3" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:35.470062  5204 consensus_queue.cc:260] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.467445  5205 tablet_bootstrap.cc:654] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d: Neither blocks nor log segments found. Creating new log.
W20260709 12:23:35.471482  5055 leader_election.cc:343] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:35.471980  5055 leader_election.cc:304] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b
I20260709 12:23:35.472806  5211 raft_consensus.cc:2749] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:23:35.474880  5203 tablet_bootstrap.cc:492] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b: Bootstrap starting.
I20260709 12:23:35.474980  5204 ts_tablet_manager.cc:1434] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.001s
I20260709 12:23:35.476011  5204 tablet_bootstrap.cc:492] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101: Bootstrap starting.
I20260709 12:23:35.477095  5205 tablet_bootstrap.cc:492] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d: No bootstrap required, opened a new log
I20260709 12:23:35.477610  5205 ts_tablet_manager.cc:1403] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent bootstrapping tablet: real 0.016s	user 0.003s	sys 0.007s
I20260709 12:23:35.480919  5205 raft_consensus.cc:359] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.481634  5203 tablet_bootstrap.cc:654] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.481758  5205 raft_consensus.cc:385] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.482205  5205 raft_consensus.cc:740] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.482484  5204 tablet_bootstrap.cc:654] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.483247  5205 consensus_queue.cc:260] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.486150  5205 ts_tablet_manager.cc:1434] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:23:35.487406  5205 tablet_bootstrap.cc:492] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d: Bootstrap starting.
I20260709 12:23:35.490476  5204 tablet_bootstrap.cc:492] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101: No bootstrap required, opened a new log
I20260709 12:23:35.490974  5204 ts_tablet_manager.cc:1403] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent bootstrapping tablet: real 0.015s	user 0.005s	sys 0.009s
I20260709 12:23:35.493683  5204 raft_consensus.cc:359] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.493690  5205 tablet_bootstrap.cc:654] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.494585  5204 raft_consensus.cc:385] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.494979  5204 raft_consensus.cc:740] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.495858  5204 consensus_queue.cc:260] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.497000  5203 tablet_bootstrap.cc:492] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b: No bootstrap required, opened a new log
I20260709 12:23:35.497527  5203 ts_tablet_manager.cc:1403] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b: Time spent bootstrapping tablet: real 0.023s	user 0.016s	sys 0.006s
I20260709 12:23:35.498641  5204 ts_tablet_manager.cc:1434] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:23:35.499642  5204 tablet_bootstrap.cc:492] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101: Bootstrap starting.
W20260709 12:23:35.500574  4961 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:23:35.503041  5203 raft_consensus.cc:359] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.503907  5203 raft_consensus.cc:385] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.504303  5203 raft_consensus.cc:740] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.505339  5203 consensus_queue.cc:260] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.506233  5204 tablet_bootstrap.cc:654] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.510787  5203 ts_tablet_manager.cc:1434] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.002s
I20260709 12:23:35.512261  5203 tablet_bootstrap.cc:492] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b: Bootstrap starting.
I20260709 12:23:35.519449  5205 tablet_bootstrap.cc:492] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d: No bootstrap required, opened a new log
I20260709 12:23:35.522822  5205 ts_tablet_manager.cc:1403] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent bootstrapping tablet: real 0.036s	user 0.019s	sys 0.004s
I20260709 12:23:35.522997  5203 tablet_bootstrap.cc:654] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.526445  5204 tablet_bootstrap.cc:492] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101: No bootstrap required, opened a new log
I20260709 12:23:35.526077  5205 raft_consensus.cc:359] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.527009  5204 ts_tablet_manager.cc:1403] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent bootstrapping tablet: real 0.028s	user 0.012s	sys 0.012s
I20260709 12:23:35.526983  5205 raft_consensus.cc:385] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.527580  5205 raft_consensus.cc:740] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.528314  5205 consensus_queue.cc:260] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.528795  5203 tablet_bootstrap.cc:492] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b: No bootstrap required, opened a new log
I20260709 12:23:35.529395  5203 ts_tablet_manager.cc:1403] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b: Time spent bootstrapping tablet: real 0.017s	user 0.008s	sys 0.007s
I20260709 12:23:35.530396  5205 ts_tablet_manager.cc:1434] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260709 12:23:35.531360  5205 tablet_bootstrap.cc:492] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d: Bootstrap starting.
I20260709 12:23:35.532332  5203 raft_consensus.cc:359] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.533195  5203 raft_consensus.cc:385] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.533532  5203 raft_consensus.cc:740] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.533427  5204 raft_consensus.cc:359] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.534240  5204 raft_consensus.cc:385] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.534588  5204 raft_consensus.cc:740] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.534543  5203 consensus_queue.cc:260] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.535511  5204 consensus_queue.cc:260] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.540314  5203 ts_tablet_manager.cc:1434] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:23:35.540344  5204 ts_tablet_manager.cc:1434] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 12:23:35.541661  5204 tablet_bootstrap.cc:492] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101: Bootstrap starting.
I20260709 12:23:35.541661  5203 tablet_bootstrap.cc:492] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b: Bootstrap starting.
I20260709 12:23:35.546491  5205 tablet_bootstrap.cc:654] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.546748  5209 raft_consensus.cc:493] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.547315  5209 raft_consensus.cc:515] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.552943  5203 tablet_bootstrap.cc:654] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.557164  5204 tablet_bootstrap.cc:654] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.562738  5203 tablet_bootstrap.cc:492] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b: No bootstrap required, opened a new log
I20260709 12:23:35.563282  5203 ts_tablet_manager.cc:1403] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b: Time spent bootstrapping tablet: real 0.022s	user 0.008s	sys 0.005s
I20260709 12:23:35.565861  5203 raft_consensus.cc:359] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.566567  5203 raft_consensus.cc:385] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.566844  5203 raft_consensus.cc:740] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.567687  5203 consensus_queue.cc:260] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.568044  5209 leader_election.cc:290] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:35.574640  5211 raft_consensus.cc:493] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.575253  5211 raft_consensus.cc:515] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.577865  5211 leader_election.cc:290] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:35.579629  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90561bb0627a4317adcaa4dd18688b0e" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:35.582209  5205 tablet_bootstrap.cc:492] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d: No bootstrap required, opened a new log
W20260709 12:23:35.582371  5055 leader_election.cc:343] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:35.582839  5205 ts_tablet_manager.cc:1403] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent bootstrapping tablet: real 0.052s	user 0.008s	sys 0.026s
I20260709 12:23:35.579962  5015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90561bb0627a4317adcaa4dd18688b0e" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" is_pre_election: true
I20260709 12:23:35.584282  5015 raft_consensus.cc:2468] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 0.
I20260709 12:23:35.586077  5056 leader_election.cc:304] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: a9bb0d862776474d9cbb54f9ab5c963b
I20260709 12:23:35.587458  5211 raft_consensus.cc:2804] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:35.587864  5211 raft_consensus.cc:493] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.588238  5211 raft_consensus.cc:3060] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.597926  5015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" is_pre_election: true
I20260709 12:23:35.596508  5211 raft_consensus.cc:515] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
W20260709 12:23:35.599753  5130 leader_election.cc:343] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:35.601293  5015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90561bb0627a4317adcaa4dd18688b0e" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:35.601956  5015 raft_consensus.cc:3060] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.602891  5209 raft_consensus.cc:493] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.603441  5209 raft_consensus.cc:515] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.604607  5210 raft_consensus.cc:493] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.605137  5210 raft_consensus.cc:515] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:35.605916  5209 leader_election.cc:290] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:35.608016  5210 leader_election.cc:290] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:35.608461  5014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" is_pre_election: true
I20260709 12:23:35.610757  5014 raft_consensus.cc:2468] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 0.
I20260709 12:23:35.618415  5203 ts_tablet_manager.cc:1434] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b: Time spent starting tablet: real 0.055s	user 0.022s	sys 0.018s
I20260709 12:23:35.620687  5130 leader_election.cc:304] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:35.622136  5235 raft_consensus.cc:2804] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:35.622503  5203 tablet_bootstrap.cc:492] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b: Bootstrap starting.
I20260709 12:23:35.622622  5235 raft_consensus.cc:493] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.623214  5235 raft_consensus.cc:3060] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.626061  5204 tablet_bootstrap.cc:492] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101: No bootstrap required, opened a new log
I20260709 12:23:35.626715  5204 ts_tablet_manager.cc:1403] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent bootstrapping tablet: real 0.085s	user 0.018s	sys 0.029s
I20260709 12:23:35.628685  5203 tablet_bootstrap.cc:654] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.598045  5205 raft_consensus.cc:359] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:35.629469  5205 raft_consensus.cc:385] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.633423  5205 raft_consensus.cc:740] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.635227  5205 consensus_queue.cc:260] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:35.631219  5209 raft_consensus.cc:493] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.632637  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90561bb0627a4317adcaa4dd18688b0e" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:35.634297  5211 leader_election.cc:290] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:35.630187  5015 raft_consensus.cc:2468] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 1.
I20260709 12:23:35.630523  5204 raft_consensus.cc:359] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.636289  5210 raft_consensus.cc:493] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.644186  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:35.644440  5204 raft_consensus.cc:385] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.644821  5204 raft_consensus.cc:740] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.644601  5210 raft_consensus.cc:515] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.645686  5204 consensus_queue.cc:260] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.638670  5203 tablet_bootstrap.cc:492] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b: No bootstrap required, opened a new log
I20260709 12:23:35.639346  5090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" is_pre_election: true
I20260709 12:23:35.641494  5209 raft_consensus.cc:515] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:35.647096  5210 leader_election.cc:290] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:35.648571  5163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:35.649362  5163 raft_consensus.cc:2468] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 0.
I20260709 12:23:35.649262  5209 leader_election.cc:290] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:35.642355  5211 raft_consensus.cc:493] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.641577  5088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" is_pre_election: true
I20260709 12:23:35.640926  5089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" is_pre_election: true
I20260709 12:23:35.651829  5089 raft_consensus.cc:2468] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 0.
I20260709 12:23:35.647181  5203 ts_tablet_manager.cc:1403] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b: Time spent bootstrapping tablet: real 0.025s	user 0.010s	sys 0.001s
W20260709 12:23:35.643177  5055 leader_election.cc:343] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:35.654994  5088 raft_consensus.cc:2468] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 0.
I20260709 12:23:35.655907  5015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" is_pre_election: true
I20260709 12:23:35.652801  5211 raft_consensus.cc:515] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:35.656677  5015 raft_consensus.cc:2468] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 0.
I20260709 12:23:35.657375  4980 leader_election.cc:304] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:35.658211  5211 leader_election.cc:290] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025), 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619)
I20260709 12:23:35.658382  5203 raft_consensus.cc:359] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.658998  5090 raft_consensus.cc:2468] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 0.
I20260709 12:23:35.659296  5203 raft_consensus.cc:385] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.659787  5210 raft_consensus.cc:2804] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:35.660084  5203 raft_consensus.cc:740] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.660602  5210 raft_consensus.cc:493] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.661005  5210 raft_consensus.cc:3060] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.639940  5235 raft_consensus.cc:515] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.661433  5203 consensus_queue.cc:260] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.664971  5056 leader_election.cc:304] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: a9bb0d862776474d9cbb54f9ab5c963b
I20260709 12:23:35.651307  5164 raft_consensus.cc:2468] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 0.
I20260709 12:23:35.667461  5089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" is_pre_election: true
I20260709 12:23:35.668497  5089 raft_consensus.cc:2468] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 0.
I20260709 12:23:35.668947  5210 raft_consensus.cc:515] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.670065  5130 leader_election.cc:304] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:35.671022  5130 leader_election.cc:304] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 4d3f6c31035a4dacad5e2a9d389e5101
I20260709 12:23:35.673718  5211 raft_consensus.cc:2804] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:35.677219  5210 leader_election.cc:290] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 election: Requested vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:35.677286  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:35.678020  5204 ts_tablet_manager.cc:1434] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent starting tablet: real 0.051s	user 0.004s	sys 0.021s
I20260709 12:23:35.678278  4980 leader_election.cc:304] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:35.679131  5203 ts_tablet_manager.cc:1434] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.001s
I20260709 12:23:35.661677  5088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" is_pre_election: true
I20260709 12:23:35.673197  5209 raft_consensus.cc:2804] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:35.685496  5209 raft_consensus.cc:493] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.685611  5088 raft_consensus.cc:2468] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 0.
I20260709 12:23:35.686764  5235 leader_election.cc:290] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:35.685884  5209 raft_consensus.cc:3060] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.687352  5235 raft_consensus.cc:2804] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:35.687741  5235 raft_consensus.cc:493] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.689462  5235 raft_consensus.cc:3060] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.689801  5163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:35.690418  5163 raft_consensus.cc:3060] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.691789  5203 tablet_bootstrap.cc:492] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b: Bootstrap starting.
I20260709 12:23:35.692025  5015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" is_pre_election: true
I20260709 12:23:35.693426  5015 raft_consensus.cc:2468] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 0.
I20260709 12:23:35.696283  5205 ts_tablet_manager.cc:1434] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent starting tablet: real 0.113s	user 0.008s	sys 0.012s
I20260709 12:23:35.697431  5205 tablet_bootstrap.cc:492] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d: Bootstrap starting.
I20260709 12:23:35.683424  5090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:35.700709  5090 raft_consensus.cc:3060] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.697865  5211 raft_consensus.cc:697] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 LEADER]: Becoming Leader. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:35.702945  5056 leader_election.cc:304] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:35.705237  5241 raft_consensus.cc:2804] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:35.705688  5241 raft_consensus.cc:493] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.706112  5241 raft_consensus.cc:3060] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.702818  5211 consensus_queue.cc:237] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.691222  5210 raft_consensus.cc:493] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.710443  5205 tablet_bootstrap.cc:654] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.709839  5210 raft_consensus.cc:515] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.723659  5088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f1aba11d864c94a4b511ccd0be93bc" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" is_pre_election: true
I20260709 12:23:35.724782  5204 tablet_bootstrap.cc:492] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101: Bootstrap starting.
I20260709 12:23:35.724009  5162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f1aba11d864c94a4b511ccd0be93bc" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:35.727352  5162 raft_consensus.cc:2468] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 0.
I20260709 12:23:35.728679  5244 raft_consensus.cc:493] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.729183  5244 raft_consensus.cc:515] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.731805  5203 tablet_bootstrap.cc:654] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.731128  5209 raft_consensus.cc:515] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:35.732928  4980 leader_election.cc:304] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:35.734045  5164 raft_consensus.cc:2393] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e59ac3f98bdb441ca56a0e914f7c7d1d in current term 1: Already voted for candidate a9bb0d862776474d9cbb54f9ab5c963b in this term.
W20260709 12:23:35.734450  4981 leader_election.cc:343] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:35.740327  5209 leader_election.cc:290] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:35.722671  5210 leader_election.cc:290] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:35.720276  5090 raft_consensus.cc:2468] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 1.
I20260709 12:23:35.742753  5244 leader_election.cc:290] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:35.744323  5246 raft_consensus.cc:2804] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:35.744880  5246 raft_consensus.cc:493] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.745412  5246 raft_consensus.cc:3060] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.745678  5204 tablet_bootstrap.cc:654] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.745530  5243 raft_consensus.cc:493] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.746182  5243 raft_consensus.cc:515] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.738232  5163 raft_consensus.cc:2468] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 1.
I20260709 12:23:35.749737  5015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91e288907924a9d99440800ed9f6d62" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" is_pre_election: true
I20260709 12:23:35.753156  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91e288907924a9d99440800ed9f6d62" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:35.753432  5015 raft_consensus.cc:2468] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 0.
I20260709 12:23:35.753854  5164 raft_consensus.cc:2468] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 0.
I20260709 12:23:35.742017  5242 raft_consensus.cc:2804] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:35.754675  5242 raft_consensus.cc:493] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.755329  5242 raft_consensus.cc:3060] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.756147  4980 leader_election.cc:304] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:35.721998  5235 raft_consensus.cc:515] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.760701  5055 leader_election.cc:304] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:35.749881  5243 leader_election.cc:290] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:35.763257  5235 leader_election.cc:290] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:35.765290  5243 raft_consensus.cc:2804] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:35.763908  5245 raft_consensus.cc:493] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.766311  5210 raft_consensus.cc:2804] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:35.764889  5241 raft_consensus.cc:515] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:35.768052  5243 raft_consensus.cc:493] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.773363  5243 raft_consensus.cc:3060] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.775482  5163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:35.770462  5245 raft_consensus.cc:515] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.776655  5163 raft_consensus.cc:2393] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e59ac3f98bdb441ca56a0e914f7c7d1d in current term 1: Already voted for candidate a9bb0d862776474d9cbb54f9ab5c963b in this term.
I20260709 12:23:35.763850  5246 raft_consensus.cc:515] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.784448  5245 leader_election.cc:290] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
W20260709 12:23:35.765221  5130 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:43513, user_credentials={real_user=slave}} blocked reactor thread for 94220.7us
I20260709 12:23:35.787639  5090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:35.788231  5090 raft_consensus.cc:3060] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.787417  5243 raft_consensus.cc:515] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.792550  5015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:35.793825  5015 raft_consensus.cc:3060] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.794302  5204 tablet_bootstrap.cc:492] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101: No bootstrap required, opened a new log
I20260709 12:23:35.796411  5204 ts_tablet_manager.cc:1403] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent bootstrapping tablet: real 0.086s	user 0.016s	sys 0.015s
I20260709 12:23:35.797327  5210 raft_consensus.cc:697] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:35.798141  5090 raft_consensus.cc:2468] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 1.
I20260709 12:23:35.790304  5242 raft_consensus.cc:515] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:35.799466  5014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:35.800160  5241 leader_election.cc:290] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 election: Requested vote from peers a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025), 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619)
I20260709 12:23:35.803524  5088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f1aba11d864c94a4b511ccd0be93bc" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:35.805640  5090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:35.806002  5204 raft_consensus.cc:359] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.806710  5204 raft_consensus.cc:385] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.806422  5090 raft_consensus.cc:2393] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a9bb0d862776474d9cbb54f9ab5c963b in current term 1: Already voted for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in this term.
I20260709 12:23:35.807093  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91e288907924a9d99440800ed9f6d62" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:35.807354  5014 raft_consensus.cc:2393] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e59ac3f98bdb441ca56a0e914f7c7d1d in current term 1: Already voted for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in this term.
I20260709 12:23:35.807509  5204 raft_consensus.cc:740] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.807722  5013 raft_consensus.cc:3060] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.806586  5088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91e288907924a9d99440800ed9f6d62" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" is_pre_election: true
W20260709 12:23:35.806293  4981 leader_election.cc:343] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:35.807808  5012 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91e288907924a9d99440800ed9f6d62" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" is_pre_election: true
I20260709 12:23:35.808300  5204 consensus_queue.cc:260] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.811564  5014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:35.810832  5088 raft_consensus.cc:2393] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a9bb0d862776474d9cbb54f9ab5c963b in current term 1: Already voted for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in this term.
I20260709 12:23:35.814368  5204 ts_tablet_manager.cc:1434] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.001s
I20260709 12:23:35.814340  5163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91e288907924a9d99440800ed9f6d62" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:35.812186  5056 leader_election.cc:304] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b
I20260709 12:23:35.815214  5163 raft_consensus.cc:3060] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.815990  5204 tablet_bootstrap.cc:492] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101: Bootstrap starting.
I20260709 12:23:35.805091  5248 raft_consensus.cc:493] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.816715  5243 leader_election.cc:290] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:35.816998  5246 leader_election.cc:290] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 election: Requested vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:35.812315  5014 raft_consensus.cc:2393] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a9bb0d862776474d9cbb54f9ab5c963b in current term 1: Already voted for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in this term.
I20260709 12:23:35.817399  5243 raft_consensus.cc:2749] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:23:35.822981  5205 tablet_bootstrap.cc:492] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d: No bootstrap required, opened a new log
I20260709 12:23:35.826982  5205 ts_tablet_manager.cc:1403] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent bootstrapping tablet: real 0.130s	user 0.018s	sys 0.004s
I20260709 12:23:35.819900  5203 tablet_bootstrap.cc:492] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b: No bootstrap required, opened a new log
I20260709 12:23:35.821049  5249 raft_consensus.cc:493] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.827732  5203 ts_tablet_manager.cc:1403] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b: Time spent bootstrapping tablet: real 0.136s	user 0.008s	sys 0.029s
I20260709 12:23:35.829524  5088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:35.830466  5088 raft_consensus.cc:3060] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.834354  5242 leader_election.cc:290] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 election: Requested vote from peers a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:35.834309  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f1aba11d864c94a4b511ccd0be93bc" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:35.835106  5164 raft_consensus.cc:3060] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.817052  5248 raft_consensus.cc:515] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.836869  5015 raft_consensus.cc:2468] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 1.
I20260709 12:23:35.837565  5090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:35.835701  5014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:35.838642  5162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:35.838981  5015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" is_pre_election: true
I20260709 12:23:35.839454  5162 raft_consensus.cc:2393] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d3f6c31035a4dacad5e2a9d389e5101 in current term 1: Already voted for candidate a9bb0d862776474d9cbb54f9ab5c963b in this term.
I20260709 12:23:35.831161  5249 raft_consensus.cc:515] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.838672  5090 raft_consensus.cc:2393] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d3f6c31035a4dacad5e2a9d389e5101 in current term 1: Already voted for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in this term.
I20260709 12:23:35.828449  5210 consensus_queue.cc:237] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.844177  5248 leader_election.cc:290] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:35.845997  5014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" is_pre_election: true
I20260709 12:23:35.847286  5090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" is_pre_election: true
I20260709 12:23:35.847731  5089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" is_pre_election: true
I20260709 12:23:35.847870  5090 raft_consensus.cc:2468] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 0.
I20260709 12:23:35.848212  5249 leader_election.cc:290] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:35.847750  4980 leader_election.cc:304] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101; no voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d
I20260709 12:23:35.849344  5162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:35.849421  4981 leader_election.cc:304] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:35.849972  5162 raft_consensus.cc:2468] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 0.
I20260709 12:23:35.850378  5242 raft_consensus.cc:2749] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:23:35.851114  5055 leader_election.cc:304] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 4d3f6c31035a4dacad5e2a9d389e5101
I20260709 12:23:35.840566  5015 raft_consensus.cc:2468] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 0.
I20260709 12:23:35.853675  5249 raft_consensus.cc:2804] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:35.853820  5161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:35.854144  5249 raft_consensus.cc:493] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.854537  5249 raft_consensus.cc:3060] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.854580  5161 raft_consensus.cc:2468] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 0.
I20260709 12:23:35.856412  5130 leader_election.cc:304] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:35.858245  5164 raft_consensus.cc:2468] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 1.
I20260709 12:23:35.859532  5245 raft_consensus.cc:2804] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:35.858376  5013 raft_consensus.cc:2468] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 1.
I20260709 12:23:35.860330  5245 raft_consensus.cc:697] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Becoming Leader. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Running, Role: LEADER
I20260709 12:23:35.861436  5245 consensus_queue.cc:237] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.866098  5056 leader_election.cc:304] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:35.866005  4980 leader_election.cc:304] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: e59ac3f98bdb441ca56a0e914f7c7d1d
I20260709 12:23:35.857589  5242 raft_consensus.cc:2804] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:35.870313  5249 raft_consensus.cc:515] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.871376  5243 raft_consensus.cc:2804] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:35.872015  5205 raft_consensus.cc:359] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.873719  5205 raft_consensus.cc:385] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.874239  5205 raft_consensus.cc:740] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.872601  4909 catalog_manager.cc:5697] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1). New cstate: current_term: 1 leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
W20260709 12:23:35.871474  5130 leader_election.cc:343] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:35.875613  5130 leader_election.cc:304] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:35.876752  5235 raft_consensus.cc:2804] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:35.877882  5235 raft_consensus.cc:493] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.877840  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:35.878343  5235 raft_consensus.cc:3060] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.878487  5013 raft_consensus.cc:3060] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.872608  4912 catalog_manager.cc:5697] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d reported cstate change: term changed from 0 to 1, leader changed from <none> to e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:35.886656  5235 raft_consensus.cc:515] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.873065  5243 raft_consensus.cc:697] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 LEADER]: Becoming Leader. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:35.888211  5013 raft_consensus.cc:2468] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 1.
I20260709 12:23:35.888478  5243 consensus_queue.cc:237] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.886667  5163 raft_consensus.cc:2468] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 1.
I20260709 12:23:35.891248  5204 tablet_bootstrap.cc:654] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.889060  5203 raft_consensus.cc:359] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.894111  5203 raft_consensus.cc:385] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.893157  5205 consensus_queue.cc:260] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.894866  5203 raft_consensus.cc:740] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.878666  5210 raft_consensus.cc:2804] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:35.899158  5203 consensus_queue.cc:260] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.901244  5205 ts_tablet_manager.cc:1434] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent starting tablet: real 0.074s	user 0.006s	sys 0.002s
I20260709 12:23:35.901944  5205 tablet_bootstrap.cc:492] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d: Bootstrap starting.
I20260709 12:23:35.890028  5130 leader_election.cc:304] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b; no voters: 4d3f6c31035a4dacad5e2a9d389e5101, e59ac3f98bdb441ca56a0e914f7c7d1d
I20260709 12:23:35.899639  5210 raft_consensus.cc:697] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:35.904209  5244 raft_consensus.cc:2749] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:23:35.904728  5210 consensus_queue.cc:237] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.903721  5203 ts_tablet_manager.cc:1434] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b: Time spent starting tablet: real 0.076s	user 0.007s	sys 0.000s
I20260709 12:23:35.908097  5088 raft_consensus.cc:2468] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 1.
I20260709 12:23:35.898566  5056 leader_election.cc:304] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:35.910928  5211 raft_consensus.cc:2804] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:35.911828  5211 raft_consensus.cc:697] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 LEADER]: Becoming Leader. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:35.912143  5249 leader_election.cc:290] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:35.912739  5163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:35.912691  5211 consensus_queue.cc:237] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.913553  5163 raft_consensus.cc:2393] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e59ac3f98bdb441ca56a0e914f7c7d1d in current term 1: Already voted for candidate a9bb0d862776474d9cbb54f9ab5c963b in this term.
I20260709 12:23:35.913661  5235 leader_election.cc:290] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:35.910806  5130 leader_election.cc:304] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b; no voters: 4d3f6c31035a4dacad5e2a9d389e5101, e59ac3f98bdb441ca56a0e914f7c7d1d
I20260709 12:23:35.923815  5130 leader_election.cc:304] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 4d3f6c31035a4dacad5e2a9d389e5101
I20260709 12:23:35.926748  5245 raft_consensus.cc:2804] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:35.927546  5245 raft_consensus.cc:697] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Becoming Leader. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Running, Role: LEADER
I20260709 12:23:35.924654  5235 raft_consensus.cc:2749] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:23:35.928723  5245 consensus_queue.cc:237] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.929885  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:35.931257  5204 tablet_bootstrap.cc:492] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101: No bootstrap required, opened a new log
I20260709 12:23:35.931706  5204 ts_tablet_manager.cc:1403] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent bootstrapping tablet: real 0.116s	user 0.014s	sys 0.010s
I20260709 12:23:35.935437  5205 tablet_bootstrap.cc:654] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.940140  4911 catalog_manager.cc:5697] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b reported cstate change: term changed from 0 to 1, leader changed from <none> to a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3). New cstate: current_term: 1 leader_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:35.941847  5013 raft_consensus.cc:2393] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a9bb0d862776474d9cbb54f9ab5c963b in current term 1: Already voted for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in this term.
I20260709 12:23:35.946775  5088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:35.948939  5242 raft_consensus.cc:493] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.949481  5242 raft_consensus.cc:3060] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:35.960290  5204 raft_consensus.cc:359] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.961252  5204 raft_consensus.cc:385] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:35.961596  5204 raft_consensus.cc:740] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Initialized, Role: FOLLOWER
I20260709 12:23:35.962584  5210 raft_consensus.cc:493] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.963261  5210 raft_consensus.cc:515] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.963599  5130 leader_election.cc:304] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b; no voters: 4d3f6c31035a4dacad5e2a9d389e5101, e59ac3f98bdb441ca56a0e914f7c7d1d
I20260709 12:23:35.964452  5235 raft_consensus.cc:2749] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:23:35.965754  5210 leader_election.cc:290] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:35.965866  5204 consensus_queue.cc:260] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.968875  5204 ts_tablet_manager.cc:1434] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent starting tablet: real 0.037s	user 0.003s	sys 0.007s
I20260709 12:23:35.968905  5163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a728de06487c46648df7ae5eadcd7d9a" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:35.970028  5204 tablet_bootstrap.cc:492] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101: Bootstrap starting.
I20260709 12:23:35.970420  5163 raft_consensus.cc:2468] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 0.
I20260709 12:23:35.972162  4980 leader_election.cc:304] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:35.973690  5248 raft_consensus.cc:2804] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:35.974305  5248 raft_consensus.cc:493] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.975030  5248 raft_consensus.cc:3060] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.977408  5088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a728de06487c46648df7ae5eadcd7d9a" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" is_pre_election: true
I20260709 12:23:35.977766  5235 raft_consensus.cc:493] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.978310  5235 raft_consensus.cc:515] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
W20260709 12:23:35.979647  4981 leader_election.cc:343] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:35.980855  5235 leader_election.cc:290] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:35.983393  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588e8402cd9948778e37e9cc558923d3" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" is_pre_election: true
W20260709 12:23:35.984992  5130 leader_election.cc:343] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:35.985882  5088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588e8402cd9948778e37e9cc558923d3" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" is_pre_election: true
I20260709 12:23:35.986537  5088 raft_consensus.cc:2468] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 0.
I20260709 12:23:35.986910  5204 tablet_bootstrap.cc:654] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101: Neither blocks nor log segments found. Creating new log.
I20260709 12:23:35.987720  5130 leader_election.cc:304] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 4d3f6c31035a4dacad5e2a9d389e5101
I20260709 12:23:35.988380  5242 raft_consensus.cc:515] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.989015  5235 raft_consensus.cc:2804] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:23:35.989429  5235 raft_consensus.cc:493] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.989848  5235 raft_consensus.cc:3060] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:35.990384  5248 raft_consensus.cc:515] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:35.994495  5088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a728de06487c46648df7ae5eadcd7d9a" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
W20260709 12:23:35.995754  4981 leader_election.cc:343] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:35.999495  5248 leader_election.cc:290] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 election: Requested vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:36.002337  5088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:36.009210  5242 leader_election.cc:290] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 2 election: Requested vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:36.009882  5163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a728de06487c46648df7ae5eadcd7d9a" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:36.010460  5163 raft_consensus.cc:3060] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:36.011210  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:36.011948  5164 raft_consensus.cc:3060] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:36.004467  5235 raft_consensus.cc:515] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:36.016776  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588e8402cd9948778e37e9cc558923d3" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:36.016875  5163 raft_consensus.cc:2468] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 1.
I20260709 12:23:36.018162  4980 leader_election.cc:304] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: e59ac3f98bdb441ca56a0e914f7c7d1d
I20260709 12:23:36.018846  5242 raft_consensus.cc:2804] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:23:36.019306  5130 leader_election.cc:343] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:23:36.019359  5242 raft_consensus.cc:697] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:36.020556  5242 consensus_queue.cc:237] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:36.021445  5088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588e8402cd9948778e37e9cc558923d3" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:36.022004  5088 raft_consensus.cc:3060] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:36.022027  5164 raft_consensus.cc:2468] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 2.
I20260709 12:23:36.023636  4980 leader_election.cc:304] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: e59ac3f98bdb441ca56a0e914f7c7d1d
I20260709 12:23:36.024801  5248 raft_consensus.cc:2804] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:36.025499  5248 raft_consensus.cc:697] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Becoming Leader. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:36.026363  5248 consensus_queue.cc:237] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:36.028565  5235 leader_election.cc:290] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:36.031000  5088 raft_consensus.cc:2468] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 1.
I20260709 12:23:36.032555  5130 leader_election.cc:304] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 4d3f6c31035a4dacad5e2a9d389e5101
I20260709 12:23:36.032830  5204 tablet_bootstrap.cc:492] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101: No bootstrap required, opened a new log
I20260709 12:23:36.033465  5205 tablet_bootstrap.cc:492] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d: No bootstrap required, opened a new log
I20260709 12:23:36.033457  5204 ts_tablet_manager.cc:1403] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent bootstrapping tablet: real 0.064s	user 0.011s	sys 0.018s
I20260709 12:23:36.033697  5235 raft_consensus.cc:2804] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:23:36.034068  5205 ts_tablet_manager.cc:1403] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent bootstrapping tablet: real 0.132s	user 0.008s	sys 0.007s
I20260709 12:23:36.034497  5235 raft_consensus.cc:697] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Becoming Leader. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Running, Role: LEADER
I20260709 12:23:36.035351  5235 consensus_queue.cc:237] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:36.037169  5204 raft_consensus.cc:359] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:36.038255  5204 raft_consensus.cc:385] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:36.038601  5204 raft_consensus.cc:740] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Initialized, Role: FOLLOWER
I20260709 12:23:36.039688  5204 consensus_queue.cc:260] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:36.041999  5205 raft_consensus.cc:359] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:36.042528  5204 ts_tablet_manager.cc:1434] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:23:36.044866  5205 raft_consensus.cc:385] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:23:36.053275  5205 raft_consensus.cc:740] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Initialized, Role: FOLLOWER
I20260709 12:23:36.054075  5205 consensus_queue.cc:260] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:36.058048  5205 ts_tablet_manager.cc:1434] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 12:23:36.067857  4912 catalog_manager.cc:5697] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d reported cstate change: term changed from 0 to 1, leader changed from <none> to e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:36.070394  4912 catalog_manager.cc:5697] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d reported cstate change: term changed from 0 to 1, leader changed from <none> to e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:36.071305  4911 catalog_manager.cc:5697] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b reported cstate change: term changed from 0 to 1, leader changed from <none> to a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3). New cstate: current_term: 1 leader_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:36.074002  4911 catalog_manager.cc:5697] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b reported cstate change: term changed from 0 to 1, leader changed from <none> to a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3). New cstate: current_term: 1 leader_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:36.088418  5243 raft_consensus.cc:493] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.089174  5243 raft_consensus.cc:515] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:36.092866  5243 leader_election.cc:290] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:36.092782  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f1aba11d864c94a4b511ccd0be93bc" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" is_pre_election: true
I20260709 12:23:36.093971  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f1aba11d864c94a4b511ccd0be93bc" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:36.094753  5164 raft_consensus.cc:2393] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e59ac3f98bdb441ca56a0e914f7c7d1d in current term 1: Already voted for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in this term.
I20260709 12:23:36.095613  5055 leader_election.cc:304] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b
I20260709 12:23:36.096405  5243 raft_consensus.cc:3060] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:36.101516  4909 catalog_manager.cc:5697] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1). New cstate: current_term: 1 leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:36.103689  4909 catalog_manager.cc:5697] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1). New cstate: current_term: 1 leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:36.105078  5243 raft_consensus.cc:2749] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:23:36.105650  4909 catalog_manager.cc:5697] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 reported cstate change: term changed from 1 to 2, leader changed from e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2) to 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1). New cstate: current_term: 2 leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:36.131300  5242 raft_consensus.cc:493] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.132417  5242 raft_consensus.cc:515] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:36.134330  5242 leader_election.cc:290] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:36.135533  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588e8402cd9948778e37e9cc558923d3" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:36.135533  5088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588e8402cd9948778e37e9cc558923d3" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" is_pre_election: true
I20260709 12:23:36.136588  5088 raft_consensus.cc:2393] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3f6c31035a4dacad5e2a9d389e5101 in current term 1: Already voted for candidate a9bb0d862776474d9cbb54f9ab5c963b in this term.
I20260709 12:23:36.137547  4981 leader_election.cc:304] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101; no voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d
I20260709 12:23:36.138281  5242 raft_consensus.cc:3060] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:36.143005  5242 raft_consensus.cc:2749] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:23:36.153607  5243 consensus_queue.cc:1048] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:23:36.165656  5164 raft_consensus.cc:3060] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:36.174248  5241 consensus_queue.cc:1048] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:23:36.291831  5116 tablet.cc:2406] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:23:36.311312  5013 raft_consensus.cc:1240] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Rejecting Update request from peer e59ac3f98bdb441ca56a0e914f7c7d1d for earlier term 1. Current term is 2. Ops: []
I20260709 12:23:36.312791  5241 consensus_queue.cc:1059] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:36.313437  5241 consensus_queue.cc:1243] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Peer 4d3f6c31035a4dacad5e2a9d389e5101 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260709 12:23:36.313906  5243 raft_consensus.cc:3055] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:36.314173  5243 raft_consensus.cc:740] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:36.314946  5243 consensus_queue.cc:260] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:36.315871  5243 raft_consensus.cc:3060] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:36.336200  5242 raft_consensus.cc:493] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.336797  5242 raft_consensus.cc:515] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:36.339285  5242 leader_election.cc:290] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:36.340098  5088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" is_pre_election: true
I20260709 12:23:36.340507  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:36.341002  5088 raft_consensus.cc:2393] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d3f6c31035a4dacad5e2a9d389e5101 in current term 1: Already voted for candidate a9bb0d862776474d9cbb54f9ab5c963b in this term.
I20260709 12:23:36.342095  4981 leader_election.cc:304] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101; no voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d
I20260709 12:23:36.342605  5235 consensus_queue.cc:1048] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:36.342818  5242 raft_consensus.cc:3060] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:36.349330  5242 raft_consensus.cc:2749] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:23:36.354990  5244 consensus_queue.cc:1048] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:36.355811  5248 consensus_queue.cc:1048] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:23:36.370537  5242 consensus_queue.cc:1048] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:36.378360  5243 raft_consensus.cc:493] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.378911  5243 raft_consensus.cc:515] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:36.381215  5243 leader_election.cc:290] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:36.382290  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a728de06487c46648df7ae5eadcd7d9a" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:36.383000  5164 raft_consensus.cc:2393] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e59ac3f98bdb441ca56a0e914f7c7d1d in current term 1: Already voted for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in this term.
I20260709 12:23:36.383000  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a728de06487c46648df7ae5eadcd7d9a" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" is_pre_election: true
I20260709 12:23:36.384497  5056 leader_election.cc:304] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b
I20260709 12:23:36.385237  5243 raft_consensus.cc:3060] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:23:36.390461  5243 raft_consensus.cc:2749] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:23:36.415373  5244 consensus_queue.cc:1048] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:36.422179  5243 consensus_queue.cc:1048] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:36.430132  5235 consensus_queue.cc:1048] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:23:36.431564  5244 consensus_queue.cc:1048] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:36.438697  5210 consensus_queue.cc:1048] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:36.440703  5210 consensus_queue.cc:1048] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:36.443078  5211 consensus_queue.cc:1048] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:36.452991  5242 consensus_queue.cc:1048] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:23:36.472971  5282 consensus_queue.cc:1048] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:23:36.487695  5245 consensus_queue.cc:1048] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:23:36.502911  4961 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:23:36.558947  5163 raft_consensus.cc:1275] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:23:36.560675  5248 consensus_queue.cc:1048] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:36.572002  5089 raft_consensus.cc:1275] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 12:23:36.572412  5089 pending_rounds.cc:85] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d: Aborting all ops after (but not including) 0
I20260709 12:23:36.572736  5089 pending_rounds.cc:107] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:23:36.573024  5089 raft_consensus.cc:2889] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:36.574973  5282 consensus_queue.cc:1048] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:37.425700  5241 raft_consensus.cc:493] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.426414  5241 raft_consensus.cc:515] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:37.428719  5241 leader_election.cc:290] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025), 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619)
I20260709 12:23:37.429821  5163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" is_pre_election: true
I20260709 12:23:37.430051  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" is_pre_election: true
I20260709 12:23:37.430565  5163 raft_consensus.cc:2468] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 1.
I20260709 12:23:37.430624  5013 raft_consensus.cc:2468] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 1.
I20260709 12:23:37.431959  5055 leader_election.cc:304] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:37.432730  5241 raft_consensus.cc:2804] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:23:37.433115  5241 raft_consensus.cc:493] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.433404  5241 raft_consensus.cc:3060] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.438092  5241 raft_consensus.cc:515] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:37.439872  5241 leader_election.cc:290] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 2 election: Requested vote from peers a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025), 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619)
I20260709 12:23:37.441010  5163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:37.441614  5163 raft_consensus.cc:3060] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.441661  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:37.442299  5013 raft_consensus.cc:3060] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.446460  5163 raft_consensus.cc:2468] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 2.
I20260709 12:23:37.446969  5013 raft_consensus.cc:2468] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 2.
I20260709 12:23:37.447425  5055 leader_election.cc:304] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:37.448159  5241 raft_consensus.cc:2804] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:37.448560  5241 raft_consensus.cc:697] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Becoming Leader. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:37.449141  5241 consensus_queue.cc:237] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:37.456221  4911 catalog_manager.cc:5697] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d reported cstate change: term changed from 0 to 2, leader changed from <none> to e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:37.655697  5090 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67"
dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
mode: GRACEFUL
new_leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:37708
I20260709 12:23:37.656333  5090 raft_consensus.cc:606] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Received request to transfer leadership to TS 4d3f6c31035a4dacad5e2a9d389e5101
I20260709 12:23:37.662591  5090 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c91e288907924a9d99440800ed9f6d62"
dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
mode: GRACEFUL
new_leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:37708
I20260709 12:23:37.663291  5090 raft_consensus.cc:606] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 LEADER]: Received request to transfer leadership to TS 4d3f6c31035a4dacad5e2a9d389e5101
I20260709 12:23:37.684885  5163 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697"
dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
mode: GRACEFUL
new_leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:55488
I20260709 12:23:37.685426  5163 raft_consensus.cc:606] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Received request to transfer leadership to TS 4d3f6c31035a4dacad5e2a9d389e5101
I20260709 12:23:37.694008  5163 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c"
dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
mode: GRACEFUL
new_leader_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
 from {username='slave'} at 127.0.0.1:55488
I20260709 12:23:37.694505  5163 raft_consensus.cc:606] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Received request to transfer leadership to TS e59ac3f98bdb441ca56a0e914f7c7d1d
I20260709 12:23:37.709498  5013 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
mode: GRACEFUL
new_leader_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
 from {username='slave'} at 127.0.0.1:36530
I20260709 12:23:37.710039  5013 raft_consensus.cc:606] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Received request to transfer leadership to TS e59ac3f98bdb441ca56a0e914f7c7d1d
I20260709 12:23:37.715384  5163 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "588e8402cd9948778e37e9cc558923d3"
dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
mode: GRACEFUL
new_leader_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
 from {username='slave'} at 127.0.0.1:55488
I20260709 12:23:37.715903  5163 raft_consensus.cc:606] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Received request to transfer leadership to TS e59ac3f98bdb441ca56a0e914f7c7d1d
I20260709 12:23:37.721671  5013 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a728de06487c46648df7ae5eadcd7d9a"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
mode: GRACEFUL
new_leader_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
 from {username='slave'} at 127.0.0.1:36530
I20260709 12:23:37.722196  5013 raft_consensus.cc:606] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Received request to transfer leadership to TS a9bb0d862776474d9cbb54f9ab5c963b
I20260709 12:23:37.742712  5245 raft_consensus.cc:993] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b: : Instructing follower e59ac3f98bdb441ca56a0e914f7c7d1d to start an election
I20260709 12:23:37.743120  5245 raft_consensus.cc:1081] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Signalling peer e59ac3f98bdb441ca56a0e914f7c7d1d to start an election
I20260709 12:23:37.744699  5089 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c"
dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
 from {username='slave'} at 127.0.0.1:37690
I20260709 12:23:37.745149  5089 raft_consensus.cc:493] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:37.745406  5089 raft_consensus.cc:3060] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.750197  5089 raft_consensus.cc:515] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:37.752699  5302 raft_consensus.cc:993] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101: : Instructing follower e59ac3f98bdb441ca56a0e914f7c7d1d to start an election
I20260709 12:23:37.752728  5089 leader_election.cc:290] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 2 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:37.752997  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:37.753312  5163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:37.753667  5013 raft_consensus.cc:3060] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.753901  5163 raft_consensus.cc:3055] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:37.754302  5163 raft_consensus.cc:740] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Running, Role: LEADER
I20260709 12:23:37.756690  5282 raft_consensus.cc:1081] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Signalling peer e59ac3f98bdb441ca56a0e914f7c7d1d to start an election
I20260709 12:23:37.755306  5163 consensus_queue.cc:260] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:37.762943  5089 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935"
dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
 from {username='slave'} at 127.0.0.1:37696
I20260709 12:23:37.763549  5089 raft_consensus.cc:493] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:37.763070  5163 raft_consensus.cc:3060] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.763950  5089 raft_consensus.cc:3060] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.763710  5013 raft_consensus.cc:2468] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 2.
I20260709 12:23:37.765664  5056 leader_election.cc:304] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:37.766875  5241 raft_consensus.cc:2804] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:37.767508  5241 raft_consensus.cc:697] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Becoming Leader. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:37.768744  5241 consensus_queue.cc:237] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:37.772799  5163 raft_consensus.cc:2468] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 2.
I20260709 12:23:37.772269  5089 raft_consensus.cc:515] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:37.778860  5163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:37.779214  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:37.779501  5163 raft_consensus.cc:3060] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.779515  5089 leader_election.cc:290] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 2 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:37.780084  5013 raft_consensus.cc:3055] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:37.780500  5013 raft_consensus.cc:740] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:37.780798  4911 catalog_manager.cc:5697] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d reported cstate change: term changed from 1 to 2, leader changed from a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3) to e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:37.781364  5013 consensus_queue.cc:260] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:37.782387  5013 raft_consensus.cc:3060] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.785511  5163 raft_consensus.cc:2468] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 2.
I20260709 12:23:37.786616  5055 leader_election.cc:304] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:37.787456  5241 raft_consensus.cc:2804] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:37.787920  5241 raft_consensus.cc:697] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Becoming Leader. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:37.788627  5013 raft_consensus.cc:2468] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 2.
I20260709 12:23:37.788744  5241 consensus_queue.cc:237] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:37.796568  4911 catalog_manager.cc:5697] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d reported cstate change: term changed from 1 to 2, leader changed from 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1) to e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:37.840067  5301 raft_consensus.cc:993] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b: : Instructing follower 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:37.840574  5245 raft_consensus.cc:1081] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Signalling peer 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:37.842085  5013 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:36514
I20260709 12:23:37.842577  5013 raft_consensus.cc:493] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:37.842868  5013 raft_consensus.cc:3060] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.847496  5013 raft_consensus.cc:515] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:37.849287  5013 leader_election.cc:290] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 2 election: Requested vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:37.850078  5089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:37.850694  5163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:37.850800  5089 raft_consensus.cc:3060] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.851267  5163 raft_consensus.cc:3055] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:37.851631  5163 raft_consensus.cc:740] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Running, Role: LEADER
I20260709 12:23:37.852531  5163 consensus_queue.cc:260] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:37.853464  5163 raft_consensus.cc:3060] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.856129  5089 raft_consensus.cc:2468] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 2.
I20260709 12:23:37.857439  4981 leader_election.cc:304] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:37.858242  5304 raft_consensus.cc:2804] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:37.858856  5304 raft_consensus.cc:697] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Becoming Leader. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:37.859789  5304 consensus_queue.cc:237] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:37.860598  5163 raft_consensus.cc:2468] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 2.
I20260709 12:23:37.869625  4912 catalog_manager.cc:5697] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 reported cstate change: term changed from 1 to 2, leader changed from a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3) to 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1). New cstate: current_term: 2 leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:37.902943  5163 raft_consensus.cc:1275] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Refusing update from remote peer e59ac3f98bdb441ca56a0e914f7c7d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:23:37.904569  5241 consensus_queue.cc:1048] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:37.917873  5013 raft_consensus.cc:1275] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Refusing update from remote peer e59ac3f98bdb441ca56a0e914f7c7d1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:23:37.919548  5211 consensus_queue.cc:1048] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:23:37.928992  5241 raft_consensus.cc:993] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d: : Instructing follower 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:37.929600  5310 raft_consensus.cc:1081] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 LEADER]: Signalling peer 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:37.931030  5241 raft_consensus.cc:993] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d: : Instructing follower 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:37.931329  5013 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c91e288907924a9d99440800ed9f6d62"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:36508
I20260709 12:23:37.931536  5310 raft_consensus.cc:1081] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Signalling peer 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:37.931929  5013 raft_consensus.cc:493] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:37.932328  5013 raft_consensus.cc:3060] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.933341  5015 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:36508
I20260709 12:23:37.933879  5015 raft_consensus.cc:493] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:37.934193  5015 raft_consensus.cc:3060] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:37.938941  5015 raft_consensus.cc:515] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:37.938916  5013 raft_consensus.cc:515] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:37.941386  5015 leader_election.cc:290] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 3 election: Requested vote from peers a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:37.942323  5013 leader_election.cc:290] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 2 election: Requested vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:37.942469  5089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91e288907924a9d99440800ed9f6d62" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:37.942802  5090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:37.943138  5089 raft_consensus.cc:3055] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:37.943303  5090 raft_consensus.cc:3055] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:37.943470  5089 raft_consensus.cc:740] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:37.943650  5090 raft_consensus.cc:740] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:37.944382  5089 consensus_queue.cc:260] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:37.944404  5090 consensus_queue.cc:260] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:37.945737  5090 raft_consensus.cc:3060] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:37.945751  5089 raft_consensus.cc:3060] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.946614  5163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90480fe63f94406d93bb183ceba4ae67" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:37.947193  5163 raft_consensus.cc:3060] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:37.948272  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91e288907924a9d99440800ed9f6d62" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:37.948832  5164 raft_consensus.cc:3060] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.953065  5090 raft_consensus.cc:2468] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 3.
I20260709 12:23:37.953162  5089 raft_consensus.cc:2468] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 2.
I20260709 12:23:37.954180  5163 raft_consensus.cc:2468] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 3.
I20260709 12:23:37.954614  4981 leader_election.cc:304] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:37.955351  4980 leader_election.cc:304] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:37.955545  5164 raft_consensus.cc:2468] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 2.
I20260709 12:23:37.955695  5304 raft_consensus.cc:2804] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:37.956161  5304 raft_consensus.cc:697] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Becoming Leader. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:37.956882  5302 raft_consensus.cc:2804] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:23:37.957382  5302 raft_consensus.cc:697] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 LEADER]: Becoming Leader. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:37.957013  5304 consensus_queue.cc:237] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:37.957986  5302 consensus_queue.cc:237] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } }
I20260709 12:23:37.966552  4911 catalog_manager.cc:5697] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 reported cstate change: term changed from 1 to 2, leader changed from e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2) to 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1). New cstate: current_term: 2 leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:37.967749  4911 catalog_manager.cc:5697] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 reported cstate change: term changed from 2 to 3, leader changed from e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2) to 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1). New cstate: current_term: 3 leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:37.978892  5301 raft_consensus.cc:993] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b: : Instructing follower e59ac3f98bdb441ca56a0e914f7c7d1d to start an election
I20260709 12:23:37.979358  5301 raft_consensus.cc:1081] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Signalling peer e59ac3f98bdb441ca56a0e914f7c7d1d to start an election
I20260709 12:23:37.981154  5090 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "588e8402cd9948778e37e9cc558923d3"
dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
 from {username='slave'} at 127.0.0.1:37690
I20260709 12:23:37.981750  5090 raft_consensus.cc:493] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:37.982154  5090 raft_consensus.cc:3060] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.987298  5090 raft_consensus.cc:515] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:37.989571  5090 leader_election.cc:290] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 2 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:37.990661  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588e8402cd9948778e37e9cc558923d3" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:37.991328  5013 raft_consensus.cc:3060] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.991825  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588e8402cd9948778e37e9cc558923d3" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:37.992589  5164 raft_consensus.cc:3055] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:37.992936  5164 raft_consensus.cc:740] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Running, Role: LEADER
I20260709 12:23:37.993695  5164 consensus_queue.cc:260] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:37.994899  5164 raft_consensus.cc:3060] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.994999  5282 raft_consensus.cc:993] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101: : Instructing follower a9bb0d862776474d9cbb54f9ab5c963b to start an election
I20260709 12:23:37.995522  5302 raft_consensus.cc:1081] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Signalling peer a9bb0d862776474d9cbb54f9ab5c963b to start an election
I20260709 12:23:37.997339  5163 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a728de06487c46648df7ae5eadcd7d9a"
dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
 from {username='slave'} at 127.0.0.1:55486
I20260709 12:23:37.997879  5163 raft_consensus.cc:493] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:37.998231  5163 raft_consensus.cc:3060] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:37.998194  5013 raft_consensus.cc:2468] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 2.
I20260709 12:23:37.999544  5056 leader_election.cc:304] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:38.000437  5310 raft_consensus.cc:2804] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:38.000881  5310 raft_consensus.cc:697] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Becoming Leader. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:38.001220  5164 raft_consensus.cc:2468] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 2.
I20260709 12:23:38.001708  5310 consensus_queue.cc:237] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:38.004693  5163 raft_consensus.cc:515] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:38.007859  5163 leader_election.cc:290] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 2 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:38.007874  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a728de06487c46648df7ae5eadcd7d9a" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:38.008586  5013 raft_consensus.cc:3055] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:38.008898  5013 raft_consensus.cc:740] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:38.009137  5090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a728de06487c46648df7ae5eadcd7d9a" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:38.009680  5090 raft_consensus.cc:3060] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:38.009859  5013 consensus_queue.cc:260] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:38.010084  4911 catalog_manager.cc:5697] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d reported cstate change: term changed from 1 to 2, leader changed from a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3) to e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:38.011144  5013 raft_consensus.cc:3060] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:38.017004  5090 raft_consensus.cc:2468] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 2.
I20260709 12:23:38.018447  5013 raft_consensus.cc:2468] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 2.
I20260709 12:23:38.018343  5130 leader_election.cc:304] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:38.019295  5301 raft_consensus.cc:2804] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:38.019794  5301 raft_consensus.cc:697] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [term 2 LEADER]: Becoming Leader. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Running, Role: LEADER
I20260709 12:23:38.020582  5301 consensus_queue.cc:237] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:38.029335  4911 catalog_manager.cc:5697] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b reported cstate change: term changed from 1 to 2, leader changed from 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1) to a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3). New cstate: current_term: 2 leader_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:38.286855  5163 raft_consensus.cc:1275] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Refusing update from remote peer e59ac3f98bdb441ca56a0e914f7c7d1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:38.288627  5310 consensus_queue.cc:1048] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:38.297474  5013 raft_consensus.cc:1275] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Refusing update from remote peer e59ac3f98bdb441ca56a0e914f7c7d1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:38.299708  5241 consensus_queue.cc:1048] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:38.301491  5013 raft_consensus.cc:1275] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Refusing update from remote peer e59ac3f98bdb441ca56a0e914f7c7d1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:38.303224  5163 raft_consensus.cc:1275] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Refusing update from remote peer e59ac3f98bdb441ca56a0e914f7c7d1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:38.303428  5211 consensus_queue.cc:1048] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:38.304801  5314 consensus_queue.cc:1048] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:38.384631  5015 raft_consensus.cc:1275] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Refusing update from remote peer e59ac3f98bdb441ca56a0e914f7c7d1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:38.386406  5310 consensus_queue.cc:1048] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:38.393303  5090 raft_consensus.cc:1275] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:38.395475  5302 consensus_queue.cc:1048] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:38.396842  5163 raft_consensus.cc:1275] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Refusing update from remote peer e59ac3f98bdb441ca56a0e914f7c7d1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:38.399363  5314 consensus_queue.cc:1048] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:38.408872  5163 raft_consensus.cc:1275] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:38.410887  5282 consensus_queue.cc:1048] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:38.411415  5090 raft_consensus.cc:1275] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:38.413054  5302 consensus_queue.cc:1048] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:38.424448  5164 raft_consensus.cc:1275] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:38.426502  5304 consensus_queue.cc:1048] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:38.447788  5163 raft_consensus.cc:1275] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:23:38.449038  5282 consensus_queue.cc:1048] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:38.456696  5089 raft_consensus.cc:1275] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:23:38.458105  5302 consensus_queue.cc:1048] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:38.538594  5015 raft_consensus.cc:1275] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Refusing update from remote peer a9bb0d862776474d9cbb54f9ab5c963b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:38.539928  5301 consensus_queue.cc:1048] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:38.550809  5089 raft_consensus.cc:1275] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Refusing update from remote peer a9bb0d862776474d9cbb54f9ab5c963b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:38.553526  5329 consensus_queue.cc:1048] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:40.754506  4356 auto_leader_rebalancer-test.cc:365] The leader distribution is 
I20260709 12:23:40.755182  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:40.779477  5013 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "16f1aba11d864c94a4b511ccd0be93bc"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
mode: GRACEFUL
new_leader_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
 from {username='slave'} at 127.0.0.1:36544
I20260709 12:23:40.779981  5013 raft_consensus.cc:606] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Received request to transfer leadership to TS a9bb0d862776474d9cbb54f9ab5c963b
I20260709 12:23:40.782502  5013 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
mode: GRACEFUL
new_leader_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
 from {username='slave'} at 127.0.0.1:36544
I20260709 12:23:40.782944  5013 raft_consensus.cc:606] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Received request to transfer leadership to TS a9bb0d862776474d9cbb54f9ab5c963b
I20260709 12:23:40.784039  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:23:40.784708  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:41.131033  5302 raft_consensus.cc:993] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101: : Instructing follower a9bb0d862776474d9cbb54f9ab5c963b to start an election
I20260709 12:23:41.131507  5302 raft_consensus.cc:1081] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Signalling peer a9bb0d862776474d9cbb54f9ab5c963b to start an election
I20260709 12:23:41.133123  5164 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "16f1aba11d864c94a4b511ccd0be93bc"
dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
 from {username='slave'} at 127.0.0.1:55486
I20260709 12:23:41.133641  5164 raft_consensus.cc:493] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:41.133913  5164 raft_consensus.cc:3060] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:41.140750  5164 raft_consensus.cc:515] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:41.142657  5164 leader_election.cc:290] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 2 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:41.143496  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f1aba11d864c94a4b511ccd0be93bc" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:41.144104  5013 raft_consensus.cc:3055] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:41.144032  5089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f1aba11d864c94a4b511ccd0be93bc" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:41.144632  5013 raft_consensus.cc:740] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:41.144985  5089 raft_consensus.cc:3060] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:41.145583  5013 consensus_queue.cc:260] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:41.146479  5013 raft_consensus.cc:3060] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:41.150866  5013 raft_consensus.cc:2468] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 2.
I20260709 12:23:41.150911  5089 raft_consensus.cc:2468] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 2.
I20260709 12:23:41.152060  5130 leader_election.cc:304] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:41.152848  5329 raft_consensus.cc:2804] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:41.153782  5329 raft_consensus.cc:697] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 2 LEADER]: Becoming Leader. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Running, Role: LEADER
I20260709 12:23:41.154429  5329 consensus_queue.cc:237] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:41.161059  4909 catalog_manager.cc:5697] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b reported cstate change: term changed from 1 to 2, leader changed from 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1) to a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3). New cstate: current_term: 2 leader_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:41.227596  5335 raft_consensus.cc:993] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101: : Instructing follower a9bb0d862776474d9cbb54f9ab5c963b to start an election
I20260709 12:23:41.228082  5302 raft_consensus.cc:1081] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Signalling peer a9bb0d862776474d9cbb54f9ab5c963b to start an election
I20260709 12:23:41.230026  5164 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697"
dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
 from {username='slave'} at 127.0.0.1:55486
I20260709 12:23:41.230485  5164 raft_consensus.cc:493] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:41.230770  5164 raft_consensus.cc:3060] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:41.235617  5164 raft_consensus.cc:515] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:41.237177  5164 leader_election.cc:290] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 3 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:41.238286  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:41.238535  5089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:41.238777  5013 raft_consensus.cc:3055] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:41.239063  5089 raft_consensus.cc:3060] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:41.239096  5013 raft_consensus.cc:740] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:41.239886  5013 consensus_queue.cc:260] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:41.240756  5013 raft_consensus.cc:3060] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:41.243873  5089 raft_consensus.cc:2468] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 3.
I20260709 12:23:41.244621  5013 raft_consensus.cc:2468] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 3.
I20260709 12:23:41.245268  5130 leader_election.cc:304] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:41.246490  5329 raft_consensus.cc:2804] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:23:41.247094  5329 raft_consensus.cc:697] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 LEADER]: Becoming Leader. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Running, Role: LEADER
I20260709 12:23:41.248029  5329 consensus_queue.cc:237] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:41.255522  4909 catalog_manager.cc:5697] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b reported cstate change: term changed from 2 to 3, leader changed from 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1) to a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3). New cstate: current_term: 3 leader_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:41.668243  5013 raft_consensus.cc:1275] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Refusing update from remote peer a9bb0d862776474d9cbb54f9ab5c963b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:41.669736  5329 consensus_queue.cc:1048] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:41.681974  5089 raft_consensus.cc:1275] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Refusing update from remote peer a9bb0d862776474d9cbb54f9ab5c963b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:41.684629  5345 consensus_queue.cc:1048] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:41.730490  5089 raft_consensus.cc:1275] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Refusing update from remote peer a9bb0d862776474d9cbb54f9ab5c963b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:41.732249  5329 consensus_queue.cc:1048] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:41.741295  5013 raft_consensus.cc:1275] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Refusing update from remote peer a9bb0d862776474d9cbb54f9ab5c963b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:41.742597  5350 consensus_queue.cc:1048] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:41.785846  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:41.791482  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:41.791842  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:42.792814  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:42.798285  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:42.798666  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:43.799944  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:43.808037  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:43.808485  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:44.809597  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:44.816336  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:44.816735  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:45.817606  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:45.824061  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:45.824509  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:46.825605  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:46.832387  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:46.833029  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:47.833998  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:47.839452  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:47.839906  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:48.841116  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:48.849021  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:48.849691  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:49.851037  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:49.857748  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:49.858289  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:50.899633  4356 auto_leader_rebalancer-test.cc:384] The leader distribution is 
I20260709 12:23:50.912091  5161 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "16f1aba11d864c94a4b511ccd0be93bc"
dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
mode: GRACEFUL
new_leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:55488
I20260709 12:23:50.912793  5161 raft_consensus.cc:606] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 2 LEADER]: Received request to transfer leadership to TS 4d3f6c31035a4dacad5e2a9d389e5101
I20260709 12:23:50.923295  5161 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697"
dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
mode: GRACEFUL
new_leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:55488
I20260709 12:23:50.923866  5161 raft_consensus.cc:606] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 LEADER]: Received request to transfer leadership to TS 4d3f6c31035a4dacad5e2a9d389e5101
I20260709 12:23:50.930009  5090 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90561bb0627a4317adcaa4dd18688b0e"
dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
mode: GRACEFUL
new_leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:37708
I20260709 12:23:50.930581  5090 raft_consensus.cc:606] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 LEADER]: Received request to transfer leadership to TS 4d3f6c31035a4dacad5e2a9d389e5101
I20260709 12:23:50.936040  5090 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c"
dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
mode: GRACEFUL
new_leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:37708
I20260709 12:23:50.936568  5090 raft_consensus.cc:606] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Received request to transfer leadership to TS 4d3f6c31035a4dacad5e2a9d389e5101
I20260709 12:23:50.942138  5090 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935"
dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
mode: GRACEFUL
new_leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:37708
I20260709 12:23:50.942704  5090 raft_consensus.cc:606] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Received request to transfer leadership to TS 4d3f6c31035a4dacad5e2a9d389e5101
I20260709 12:23:50.980789  5363 raft_consensus.cc:993] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d: : Instructing follower 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:50.981426  5360 raft_consensus.cc:1081] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 LEADER]: Signalling peer 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:50.983194  5015 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "90561bb0627a4317adcaa4dd18688b0e"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:36508
I20260709 12:23:50.983758  5015 raft_consensus.cc:493] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:50.984087  5015 raft_consensus.cc:3060] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:50.989477  5015 raft_consensus.cc:515] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:50.991786  5015 leader_election.cc:290] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 2 election: Requested vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:50.992995  5161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90561bb0627a4317adcaa4dd18688b0e" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:50.992916  5090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90561bb0627a4317adcaa4dd18688b0e" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:50.993674  5161 raft_consensus.cc:3060] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:50.994032  5090 raft_consensus.cc:3055] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:23:50.994465  5090 raft_consensus.cc:740] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:50.995539  5090 consensus_queue.cc:260] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:50.997215  5090 raft_consensus.cc:3060] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:23:50.999477  5161 raft_consensus.cc:2468] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 2.
I20260709 12:23:51.000983  4980 leader_election.cc:304] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:51.002286  5362 raft_consensus.cc:2804] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:23:51.002873  5362 raft_consensus.cc:697] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Becoming Leader. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:51.003827  5090 raft_consensus.cc:2468] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 2.
I20260709 12:23:51.004154  5362 consensus_queue.cc:237] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:51.012125  4909 catalog_manager.cc:5697] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 reported cstate change: term changed from 1 to 2, leader changed from e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2) to 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1). New cstate: current_term: 2 leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:51.101591  5363 raft_consensus.cc:993] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d: : Instructing follower 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:51.102075  5363 raft_consensus.cc:1081] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Signalling peer 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:51.103577  5015 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:36508
I20260709 12:23:51.104055  5015 raft_consensus.cc:493] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:51.104300  5015 raft_consensus.cc:3060] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:51.109071  5015 raft_consensus.cc:515] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:51.110476  5015 leader_election.cc:290] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 3 election: Requested vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:51.111315  5090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:51.111606  5161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:51.111901  5090 raft_consensus.cc:3055] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:51.112248  5090 raft_consensus.cc:740] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:51.112282  5161 raft_consensus.cc:3060] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:51.113171  5090 consensus_queue.cc:260] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:51.114094  5090 raft_consensus.cc:3060] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:51.118157  5161 raft_consensus.cc:2468] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 3.
I20260709 12:23:51.118919  5090 raft_consensus.cc:2468] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 3.
I20260709 12:23:51.119282  4980 leader_election.cc:304] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:51.120092  5362 raft_consensus.cc:2804] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:23:51.120633  5362 raft_consensus.cc:697] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 LEADER]: Becoming Leader. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:51.121776  5362 consensus_queue.cc:237] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:51.129374  4909 catalog_manager.cc:5697] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 reported cstate change: term changed from 2 to 3, leader changed from e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2) to 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1). New cstate: current_term: 3 leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:51.386961  5360 raft_consensus.cc:993] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d: : Instructing follower 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:51.387425  5363 raft_consensus.cc:1081] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Signalling peer 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:51.388944  5015 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:36508
I20260709 12:23:51.389386  5015 raft_consensus.cc:493] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:51.389714  5015 raft_consensus.cc:3060] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:51.394028  5015 raft_consensus.cc:515] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:51.395366  5015 leader_election.cc:290] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 3 election: Requested vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:51.396504  5161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:51.396569  5090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:51.396932  5161 raft_consensus.cc:3060] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:51.397298  5090 raft_consensus.cc:3055] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:51.397605  5090 raft_consensus.cc:740] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:51.398624  5090 consensus_queue.cc:260] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:51.399935  5090 raft_consensus.cc:3060] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:51.401279  5161 raft_consensus.cc:2468] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 3.
I20260709 12:23:51.402364  4980 leader_election.cc:304] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:51.403254  5362 raft_consensus.cc:2804] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:23:51.403734  5362 raft_consensus.cc:697] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 LEADER]: Becoming Leader. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:51.404752  5362 consensus_queue.cc:237] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:51.406621  5090 raft_consensus.cc:2468] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 3.
I20260709 12:23:51.412227  4909 catalog_manager.cc:5697] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 reported cstate change: term changed from 2 to 3, leader changed from e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2) to 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1). New cstate: current_term: 3 leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:51.461201  5372 raft_consensus.cc:993] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b: : Instructing follower 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:51.461795  5364 raft_consensus.cc:1081] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 LEADER]: Signalling peer 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:51.463346  5015 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:36514
I20260709 12:23:51.463780  5015 raft_consensus.cc:493] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:51.464059  5015 raft_consensus.cc:3060] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:51.469866  5364 raft_consensus.cc:993] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b: : Instructing follower 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:51.469851  5015 raft_consensus.cc:515] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:51.470460  5372 raft_consensus.cc:1081] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 2 LEADER]: Signalling peer 4d3f6c31035a4dacad5e2a9d389e5101 to start an election
I20260709 12:23:51.471889  5015 leader_election.cc:290] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 4 election: Requested vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:51.472460  5013 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "16f1aba11d864c94a4b511ccd0be93bc"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
 from {username='slave'} at 127.0.0.1:36514
I20260709 12:23:51.472517  5090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:51.473083  5161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9241c592ecc04d28b06a4bfc7acf2697" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:51.473157  5090 raft_consensus.cc:3060] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:51.473161  5013 raft_consensus.cc:493] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:51.473615  5161 raft_consensus.cc:3055] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:23:51.473721  5013 raft_consensus.cc:3060] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:51.473934  5161 raft_consensus.cc:740] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Running, Role: LEADER
I20260709 12:23:51.474718  5161 consensus_queue.cc:260] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:51.475905  5161 raft_consensus.cc:3060] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:51.479424  5013 raft_consensus.cc:515] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:51.480106  5090 raft_consensus.cc:2468] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 4.
I20260709 12:23:51.481088  5013 leader_election.cc:290] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 3 election: Requested vote from peers e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:51.481719  4981 leader_election.cc:304] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:51.482074  5090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f1aba11d864c94a4b511ccd0be93bc" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:51.482116  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f1aba11d864c94a4b511ccd0be93bc" candidate_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:51.482609  5362 raft_consensus.cc:2804] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:23:51.483131  5090 raft_consensus.cc:3060] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:51.483192  5164 raft_consensus.cc:3055] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:51.483664  5164 raft_consensus.cc:740] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Running, Role: LEADER
I20260709 12:23:51.483522  5362 raft_consensus.cc:697] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 4 LEADER]: Becoming Leader. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:51.484510  5164 consensus_queue.cc:260] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:51.485013  5362 consensus_queue.cc:237] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:51.485731  5164 raft_consensus.cc:3060] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:51.492566  5164 raft_consensus.cc:2468] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 3.
I20260709 12:23:51.493129  5090 raft_consensus.cc:2468] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 3.
I20260709 12:23:51.483151  5161 raft_consensus.cc:2468] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3f6c31035a4dacad5e2a9d389e5101 in term 4.
I20260709 12:23:51.493129  4909 catalog_manager.cc:5697] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 reported cstate change: term changed from 3 to 4, leader changed from a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3) to 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1). New cstate: current_term: 4 leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:51.493975  4980 leader_election.cc:304] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d3f6c31035a4dacad5e2a9d389e5101, a9bb0d862776474d9cbb54f9ab5c963b; no voters: 
I20260709 12:23:51.495115  5361 raft_consensus.cc:2804] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:23:51.495474  5361 raft_consensus.cc:697] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 LEADER]: Becoming Leader. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:51.496395  5361 consensus_queue.cc:237] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:51.506325  4909 catalog_manager.cc:5697] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 reported cstate change: term changed from 2 to 3, leader changed from a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3) to 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1). New cstate: current_term: 3 leader_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:51.514755  5090 raft_consensus.cc:1275] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:51.516580  5362 consensus_queue.cc:1048] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:51.527156  5164 raft_consensus.cc:1275] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:23:51.528311  5362 consensus_queue.cc:1048] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:51.537714  5090 raft_consensus.cc:1275] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:51.539120  5361 consensus_queue.cc:1048] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:51.548651  5164 raft_consensus.cc:1275] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:51.550357  5362 consensus_queue.cc:1048] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:51.834794  5090 raft_consensus.cc:1275] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:51.836010  5362 consensus_queue.cc:1048] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:51.846876  5164 raft_consensus.cc:1275] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:51.848558  5362 consensus_queue.cc:1048] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:51.869741  5090 raft_consensus.cc:1275] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 4 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:23:51.871238  5362 consensus_queue.cc:1048] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:23:51.881229  5164 raft_consensus.cc:1275] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 4 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:23:51.882618  5374 consensus_queue.cc:1048] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:52.040786  5164 raft_consensus.cc:1275] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:52.041819  5090 raft_consensus.cc:1275] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Refusing update from remote peer 4d3f6c31035a4dacad5e2a9d389e5101: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:52.042033  5362 consensus_queue.cc:1048] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:52.043077  5374 consensus_queue.cc:1048] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:54.054965  4356 auto_leader_rebalancer-test.cc:399] The leader distribution is 
I20260709 12:23:54.055842  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:54.064203  5013 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
mode: GRACEFUL
new_leader_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
 from {username='slave'} at 127.0.0.1:36544
I20260709 12:23:54.064997  5013 raft_consensus.cc:606] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Received request to transfer leadership to TS a9bb0d862776474d9cbb54f9ab5c963b
I20260709 12:23:54.068792  5013 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c91e288907924a9d99440800ed9f6d62"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
mode: GRACEFUL
new_leader_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
 from {username='slave'} at 127.0.0.1:36544
I20260709 12:23:54.069526  5013 raft_consensus.cc:606] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Received request to transfer leadership to TS e59ac3f98bdb441ca56a0e914f7c7d1d
I20260709 12:23:54.072504  5013 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
mode: GRACEFUL
new_leader_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
 from {username='slave'} at 127.0.0.1:36544
I20260709 12:23:54.073136  5013 raft_consensus.cc:606] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 LEADER]: Received request to transfer leadership to TS e59ac3f98bdb441ca56a0e914f7c7d1d
I20260709 12:23:54.075559  5013 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c"
dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
mode: GRACEFUL
new_leader_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
 from {username='slave'} at 127.0.0.1:36544
I20260709 12:23:54.076036  5013 raft_consensus.cc:606] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 LEADER]: Received request to transfer leadership to TS a9bb0d862776474d9cbb54f9ab5c963b
I20260709 12:23:54.076993  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:23:54.077574  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:54.098325  5394 raft_consensus.cc:993] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101: : Instructing follower a9bb0d862776474d9cbb54f9ab5c963b to start an election
I20260709 12:23:54.098938  5361 raft_consensus.cc:1081] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 LEADER]: Signalling peer a9bb0d862776474d9cbb54f9ab5c963b to start an election
I20260709 12:23:54.101066  5164 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c"
dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
 from {username='slave'} at 127.0.0.1:55486
I20260709 12:23:54.101629  5164 raft_consensus.cc:493] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:54.101924  5164 raft_consensus.cc:3060] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:54.108965  5394 raft_consensus.cc:993] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101: : Instructing follower a9bb0d862776474d9cbb54f9ab5c963b to start an election
I20260709 12:23:54.109472  5394 raft_consensus.cc:1081] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Signalling peer a9bb0d862776474d9cbb54f9ab5c963b to start an election
I20260709 12:23:54.109447  5164 raft_consensus.cc:515] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:54.111497  5161 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932"
dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
 from {username='slave'} at 127.0.0.1:55486
I20260709 12:23:54.111974  5161 raft_consensus.cc:493] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:54.112174  5164 leader_election.cc:290] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 4 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:54.112341  5161 raft_consensus.cc:3060] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:54.113291  5089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:54.112974  5013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef2cf9f651e345cc8102ae4d8235bf1c" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:54.113767  5089 raft_consensus.cc:3060] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:54.114271  5013 raft_consensus.cc:3055] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:23:54.114714  5013 raft_consensus.cc:740] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:54.115723  5013 consensus_queue.cc:260] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:54.117214  5013 raft_consensus.cc:3060] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:54.119329  5089 raft_consensus.cc:2468] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 4.
I20260709 12:23:54.120497  5130 leader_election.cc:304] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:54.121250  5372 raft_consensus.cc:2804] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:23:54.120961  5161 raft_consensus.cc:515] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:54.122989  5161 leader_election.cc:290] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 3 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2:43513)
I20260709 12:23:54.124217  5013 raft_consensus.cc:2468] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 4.
I20260709 12:23:54.124279  5089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
I20260709 12:23:54.125054  5089 raft_consensus.cc:3060] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:54.125712  5015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74c4d6a9653848a7a8951cb7807cd932" candidate_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:54.126376  5015 raft_consensus.cc:3055] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:54.125669  5372 raft_consensus.cc:697] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 4 LEADER]: Becoming Leader. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Running, Role: LEADER
I20260709 12:23:54.126667  5015 raft_consensus.cc:740] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:54.127313  5372 consensus_queue.cc:237] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:54.127476  5015 consensus_queue.cc:260] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:54.128938  5015 raft_consensus.cc:3060] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:54.130544  5089 raft_consensus.cc:2468] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 3.
I20260709 12:23:54.131597  5130 leader_election.cc:304] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:54.133349  5395 raft_consensus.cc:2804] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:23:54.138113  5015 raft_consensus.cc:2468] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9bb0d862776474d9cbb54f9ab5c963b in term 3.
I20260709 12:23:54.140138  5395 raft_consensus.cc:697] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 LEADER]: Becoming Leader. State: Replica: a9bb0d862776474d9cbb54f9ab5c963b, State: Running, Role: LEADER
I20260709 12:23:54.141196  5395 consensus_queue.cc:237] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:54.148039  4909 catalog_manager.cc:5697] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b reported cstate change: term changed from 3 to 4, leader changed from 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1) to a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3). New cstate: current_term: 4 leader_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.170028  4909 catalog_manager.cc:5697] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b reported cstate change: term changed from 2 to 3, leader changed from 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1) to a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3). New cstate: current_term: 3 leader_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.181340  5361 raft_consensus.cc:993] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101: : Instructing follower e59ac3f98bdb441ca56a0e914f7c7d1d to start an election
I20260709 12:23:54.181834  5361 raft_consensus.cc:1081] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Signalling peer e59ac3f98bdb441ca56a0e914f7c7d1d to start an election
I20260709 12:23:54.183485  5089 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c91e288907924a9d99440800ed9f6d62"
dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
 from {username='slave'} at 127.0.0.1:37696
I20260709 12:23:54.184075  5089 raft_consensus.cc:493] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:54.184459  5089 raft_consensus.cc:3060] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:54.190442  5089 raft_consensus.cc:515] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:54.192942  5089 leader_election.cc:290] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 3 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:54.193809  5015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91e288907924a9d99440800ed9f6d62" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:54.193923  5161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91e288907924a9d99440800ed9f6d62" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:54.194505  5015 raft_consensus.cc:3055] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:54.194553  5161 raft_consensus.cc:3060] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:54.194900  5015 raft_consensus.cc:740] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:54.195712  5015 consensus_queue.cc:260] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:54.196712  5015 raft_consensus.cc:3060] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:54.200878  5161 raft_consensus.cc:2468] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 3.
I20260709 12:23:54.200896  5015 raft_consensus.cc:2468] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 3.
I20260709 12:23:54.202266  5055 leader_election.cc:304] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:54.203217  5363 raft_consensus.cc:2804] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:23:54.204682  5363 raft_consensus.cc:697] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 LEADER]: Becoming Leader. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:54.205665  5363 consensus_queue.cc:237] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:54.212909  4909 catalog_manager.cc:5697] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d reported cstate change: term changed from 2 to 3, leader changed from 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1) to e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2). New cstate: current_term: 3 leader_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
W20260709 12:23:54.340620  5111 debug-util.cc:398] Leaking SignalData structure 0x7b0800115320 after lost signal to thread 4359
W20260709 12:23:54.341621  5111 debug-util.cc:398] Leaking SignalData structure 0x7b0800115a00 after lost signal to thread 4947
W20260709 12:23:54.342617  5111 debug-util.cc:398] Leaking SignalData structure 0x7b08001b2f80 after lost signal to thread 5039
W20260709 12:23:54.343519  5111 debug-util.cc:398] Leaking SignalData structure 0x7b0800096700 after lost signal to thread 5114
W20260709 12:23:54.344393  5111 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea440 after lost signal to thread 5188
I20260709 12:23:54.510393  5394 raft_consensus.cc:993] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101: : Instructing follower e59ac3f98bdb441ca56a0e914f7c7d1d to start an election
I20260709 12:23:54.511003  5361 raft_consensus.cc:1081] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 LEADER]: Signalling peer e59ac3f98bdb441ca56a0e914f7c7d1d to start an election
I20260709 12:23:54.512784  5089 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935"
dest_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d"
 from {username='slave'} at 127.0.0.1:37696
I20260709 12:23:54.513322  5089 raft_consensus.cc:493] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:54.513698  5089 raft_consensus.cc:3060] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:54.521129  5089 raft_consensus.cc:515] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:54.524024  5089 leader_election.cc:290] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 4 election: Requested vote from peers 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619), a9bb0d862776474d9cbb54f9ab5c963b (127.4.65.3:39025)
I20260709 12:23:54.524516  5164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a9bb0d862776474d9cbb54f9ab5c963b"
I20260709 12:23:54.525069  5164 raft_consensus.cc:3060] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:54.526135  5015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59f79cb8ed5429a9ca0d1e2ebe3f935" candidate_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4d3f6c31035a4dacad5e2a9d389e5101"
I20260709 12:23:54.526706  5015 raft_consensus.cc:3055] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:23:54.526995  5015 raft_consensus.cc:740] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4d3f6c31035a4dacad5e2a9d389e5101, State: Running, Role: LEADER
I20260709 12:23:54.527796  5015 consensus_queue.cc:260] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:54.528920  5015 raft_consensus.cc:3060] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:54.531392  5164 raft_consensus.cc:2468] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 4.
I20260709 12:23:54.532462  5055 leader_election.cc:304] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9bb0d862776474d9cbb54f9ab5c963b, e59ac3f98bdb441ca56a0e914f7c7d1d; no voters: 
I20260709 12:23:54.533286  5363 raft_consensus.cc:2804] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:23:54.533831  5363 raft_consensus.cc:697] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 4 LEADER]: Becoming Leader. State: Replica: e59ac3f98bdb441ca56a0e914f7c7d1d, State: Running, Role: LEADER
I20260709 12:23:54.534680  5363 consensus_queue.cc:237] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } }
I20260709 12:23:54.536115  5015 raft_consensus.cc:2468] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59ac3f98bdb441ca56a0e914f7c7d1d in term 4.
I20260709 12:23:54.543556  4909 catalog_manager.cc:5697] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d reported cstate change: term changed from 3 to 4, leader changed from 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1) to e59ac3f98bdb441ca56a0e914f7c7d1d (127.4.65.2). New cstate: current_term: 4 leader_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:54.550836  5013 raft_consensus.cc:1275] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Refusing update from remote peer a9bb0d862776474d9cbb54f9ab5c963b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:23:54.552650  5395 consensus_queue.cc:1048] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:54.554006  5089 raft_consensus.cc:1275] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 4 FOLLOWER]: Refusing update from remote peer a9bb0d862776474d9cbb54f9ab5c963b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:23:54.557349  5397 consensus_queue.cc:1048] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:54.596920  5089 raft_consensus.cc:1275] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Refusing update from remote peer a9bb0d862776474d9cbb54f9ab5c963b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:23:54.598713  5395 consensus_queue.cc:1048] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59ac3f98bdb441ca56a0e914f7c7d1d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 43513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:54.602455  5164 raft_consensus.cc:1275] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Refusing update from remote peer e59ac3f98bdb441ca56a0e914f7c7d1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:54.603775  5363 consensus_queue.cc:1048] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:23:54.625015  5013 raft_consensus.cc:1275] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 4 FOLLOWER]: Refusing update from remote peer a9bb0d862776474d9cbb54f9ab5c963b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:23:54.627825  5372 consensus_queue.cc:1048] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:23:54.647917  5013 raft_consensus.cc:1275] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Refusing update from remote peer e59ac3f98bdb441ca56a0e914f7c7d1d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:23:54.649461  5363 consensus_queue.cc:1048] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:55.078666  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:55.084911  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:55.085390  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:55.114540  5164 raft_consensus.cc:1275] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 4 FOLLOWER]: Refusing update from remote peer e59ac3f98bdb441ca56a0e914f7c7d1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:23:55.116113  5363 consensus_queue.cc:1048] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9bb0d862776474d9cbb54f9ab5c963b" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:23:55.126219  5013 raft_consensus.cc:1275] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 4 FOLLOWER]: Refusing update from remote peer e59ac3f98bdb441ca56a0e914f7c7d1d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:23:55.128202  5415 consensus_queue.cc:1048] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f6c31035a4dacad5e2a9d389e5101" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:56.086381  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:56.091820  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:56.092304  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:57.093385  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:57.098809  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:57.099298  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:58.100191  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:58.105631  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:58.106096  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:59.107156  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:59.112752  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:23:59.113201  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:00.114200  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:00.119729  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:24:00.120232  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:01.121335  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:01.128398  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:24:01.128950  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:02.130376  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:02.136394  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:24:02.136926  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:03.137928  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:03.144104  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:24:03.144644  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:04.184371  4356 auto_leader_rebalancer-test.cc:410] The leader distribution is 
I20260709 12:24:04.185410  4356 tablet_server.cc:179] TabletServer@127.4.65.1:0 shutting down...
W20260709 12:24:04.193645  5056 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
W20260709 12:24:04.197767  5056 consensus_peers.cc:597] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d -> Peer 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619): Couldn't send request to peer 4d3f6c31035a4dacad5e2a9d389e5101. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:04.217607  5056 consensus_peers.cc:597] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d -> Peer 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619): Couldn't send request to peer 4d3f6c31035a4dacad5e2a9d389e5101. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:04.226410  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:24:04.227355  4356 tablet_replica.cc:333] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101: stopping tablet replica
I20260709 12:24:04.229118  4356 raft_consensus.cc:2243] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.230342  4356 raft_consensus.cc:2272] T 588e8402cd9948778e37e9cc558923d3 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.234269  4356 tablet_replica.cc:333] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101: stopping tablet replica
W20260709 12:24:04.234418  5056 consensus_peers.cc:597] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d -> Peer 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619): Couldn't send request to peer 4d3f6c31035a4dacad5e2a9d389e5101. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:04.235224  4356 raft_consensus.cc:2243] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.235895  4356 raft_consensus.cc:2272] T a728de06487c46648df7ae5eadcd7d9a P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.238368  4356 tablet_replica.cc:333] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101: stopping tablet replica
I20260709 12:24:04.239244  4356 raft_consensus.cc:2243] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.239809  4356 raft_consensus.cc:2272] T ef2cf9f651e345cc8102ae4d8235bf1c P 4d3f6c31035a4dacad5e2a9d389e5101 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.242324  4356 tablet_replica.cc:333] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101: stopping tablet replica
I20260709 12:24:04.243363  4356 raft_consensus.cc:2243] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.244264  4356 raft_consensus.cc:2272] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.247047  4356 tablet_replica.cc:333] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101: stopping tablet replica
I20260709 12:24:04.247543  4356 raft_consensus.cc:2243] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:24:04.248322  4356 raft_consensus.cc:2272] T 9241c592ecc04d28b06a4bfc7acf2697 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.250511  4356 tablet_replica.cc:333] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101: stopping tablet replica
I20260709 12:24:04.251049  4356 raft_consensus.cc:2243] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.251564  4356 raft_consensus.cc:2272] T 74c4d6a9653848a7a8951cb7807cd932 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.253475  4356 tablet_replica.cc:333] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101: stopping tablet replica
I20260709 12:24:04.254102  4356 raft_consensus.cc:2243] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.254534  4356 raft_consensus.cc:2272] T c91e288907924a9d99440800ed9f6d62 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.256533  4356 tablet_replica.cc:333] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101: stopping tablet replica
I20260709 12:24:04.256973  4356 raft_consensus.cc:2243] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:24:04.257768  4356 raft_consensus.cc:2272] T 90480fe63f94406d93bb183ceba4ae67 P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.259487  4356 tablet_replica.cc:333] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101: stopping tablet replica
I20260709 12:24:04.259956  4356 raft_consensus.cc:2243] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:24:04.260911  4356 raft_consensus.cc:2272] T 16f1aba11d864c94a4b511ccd0be93bc P 4d3f6c31035a4dacad5e2a9d389e5101 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.262611  4356 tablet_replica.cc:333] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101: stopping tablet replica
I20260709 12:24:04.263124  4356 raft_consensus.cc:2243] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:24:04.263829  4356 raft_consensus.cc:2272] T 90561bb0627a4317adcaa4dd18688b0e P 4d3f6c31035a4dacad5e2a9d389e5101 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:24:04.291711  5130 consensus_peers.cc:597] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b -> Peer 4d3f6c31035a4dacad5e2a9d389e5101 (127.4.65.1:42619): Couldn't send request to peer 4d3f6c31035a4dacad5e2a9d389e5101. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42619: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:04.295442  4356 tablet_server.cc:196] TabletServer@127.4.65.1:0 shutdown complete.
I20260709 12:24:04.321158  4356 tablet_server.cc:179] TabletServer@127.4.65.2:0 shutting down...
I20260709 12:24:04.349852  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:24:04.350827  4356 tablet_replica.cc:333] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d: stopping tablet replica
I20260709 12:24:04.351511  4356 raft_consensus.cc:2243] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.352016  4356 raft_consensus.cc:2272] T a728de06487c46648df7ae5eadcd7d9a P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.353986  4356 tablet_replica.cc:333] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d: stopping tablet replica
I20260709 12:24:04.354552  4356 raft_consensus.cc:2243] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.355019  4356 raft_consensus.cc:2272] T 16f1aba11d864c94a4b511ccd0be93bc P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.356986  4356 tablet_replica.cc:333] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d: stopping tablet replica
I20260709 12:24:04.357538  4356 raft_consensus.cc:2243] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.357988  4356 raft_consensus.cc:2272] T 90561bb0627a4317adcaa4dd18688b0e P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.359737  4356 tablet_replica.cc:333] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d: stopping tablet replica
I20260709 12:24:04.360258  4356 raft_consensus.cc:2243] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.360764  4356 raft_consensus.cc:2272] T 90480fe63f94406d93bb183ceba4ae67 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.362354  4356 tablet_replica.cc:333] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d: stopping tablet replica
I20260709 12:24:04.363042  4356 raft_consensus.cc:2243] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:24:04.363853  4356 raft_consensus.cc:2272] T c91e288907924a9d99440800ed9f6d62 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.365656  4356 tablet_replica.cc:333] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d: stopping tablet replica
I20260709 12:24:04.366158  4356 raft_consensus.cc:2243] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:24:04.367040  4356 raft_consensus.cc:2272] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.368733  4356 tablet_replica.cc:333] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d: stopping tablet replica
I20260709 12:24:04.369228  4356 raft_consensus.cc:2243] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.369709  4356 raft_consensus.cc:2272] T 9241c592ecc04d28b06a4bfc7acf2697 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.371415  4356 tablet_replica.cc:333] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d: stopping tablet replica
I20260709 12:24:04.371907  4356 raft_consensus.cc:2243] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.372342  4356 raft_consensus.cc:2272] T 74c4d6a9653848a7a8951cb7807cd932 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.374025  4356 tablet_replica.cc:333] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d: stopping tablet replica
I20260709 12:24:04.374495  4356 raft_consensus.cc:2243] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:24:04.375249  4356 raft_consensus.cc:2272] T 588e8402cd9948778e37e9cc558923d3 P e59ac3f98bdb441ca56a0e914f7c7d1d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.376930  4356 tablet_replica.cc:333] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d: stopping tablet replica
I20260709 12:24:04.377367  4356 raft_consensus.cc:2243] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.377777  4356 raft_consensus.cc:2272] T ef2cf9f651e345cc8102ae4d8235bf1c P e59ac3f98bdb441ca56a0e914f7c7d1d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.426220  4356 tablet_server.cc:196] TabletServer@127.4.65.2:0 shutdown complete.
I20260709 12:24:04.447104  4356 tablet_server.cc:179] TabletServer@127.4.65.3:0 shutting down...
I20260709 12:24:04.478498  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:24:04.479333  4356 tablet_replica.cc:333] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b: stopping tablet replica
I20260709 12:24:04.480022  4356 raft_consensus.cc:2243] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:24:04.481208  4356 raft_consensus.cc:2272] T a728de06487c46648df7ae5eadcd7d9a P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.483268  4356 tablet_replica.cc:333] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b: stopping tablet replica
I20260709 12:24:04.483844  4356 raft_consensus.cc:2243] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.484335  4356 raft_consensus.cc:2272] T d59f79cb8ed5429a9ca0d1e2ebe3f935 P a9bb0d862776474d9cbb54f9ab5c963b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.486155  4356 tablet_replica.cc:333] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b: stopping tablet replica
I20260709 12:24:04.486699  4356 raft_consensus.cc:2243] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.487121  4356 raft_consensus.cc:2272] T 588e8402cd9948778e37e9cc558923d3 P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.488917  4356 tablet_replica.cc:333] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b: stopping tablet replica
I20260709 12:24:04.489446  4356 raft_consensus.cc:2243] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.489907  4356 raft_consensus.cc:2272] T 9241c592ecc04d28b06a4bfc7acf2697 P a9bb0d862776474d9cbb54f9ab5c963b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.491636  4356 tablet_replica.cc:333] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b: stopping tablet replica
I20260709 12:24:04.492161  4356 raft_consensus.cc:2243] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:24:04.493072  4356 raft_consensus.cc:2272] T 74c4d6a9653848a7a8951cb7807cd932 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.494786  4356 tablet_replica.cc:333] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b: stopping tablet replica
I20260709 12:24:04.495273  4356 raft_consensus.cc:2243] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:24:04.496630  4356 raft_consensus.cc:2272] T ef2cf9f651e345cc8102ae4d8235bf1c P a9bb0d862776474d9cbb54f9ab5c963b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.498347  4356 tablet_replica.cc:333] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b: stopping tablet replica
I20260709 12:24:04.499027  4356 raft_consensus.cc:2243] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.499475  4356 raft_consensus.cc:2272] T 90561bb0627a4317adcaa4dd18688b0e P a9bb0d862776474d9cbb54f9ab5c963b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.501272  4356 tablet_replica.cc:333] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b: stopping tablet replica
I20260709 12:24:04.501714  4356 raft_consensus.cc:2243] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.502079  4356 raft_consensus.cc:2272] T 16f1aba11d864c94a4b511ccd0be93bc P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.503620  4356 tablet_replica.cc:333] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b: stopping tablet replica
I20260709 12:24:04.504021  4356 raft_consensus.cc:2243] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.504479  4356 raft_consensus.cc:2272] T c91e288907924a9d99440800ed9f6d62 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.506091  4356 tablet_replica.cc:333] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b: stopping tablet replica
I20260709 12:24:04.506475  4356 raft_consensus.cc:2243] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:04.506835  4356 raft_consensus.cc:2272] T 90480fe63f94406d93bb183ceba4ae67 P a9bb0d862776474d9cbb54f9ab5c963b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.537297  4356 tablet_server.cc:196] TabletServer@127.4.65.3:0 shutdown complete.
I20260709 12:24:04.561679  4356 master.cc:562] Master@127.4.65.62:43375 shutting down...
I20260709 12:24:04.587561  4356 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:24:04.588191  4356 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:04.588590  4356 tablet_replica.cc:333] T 00000000000000000000000000000000 P 6aeedc0e365d49958b13951a04acf871: stopping tablet replica
I20260709 12:24:04.610621  4356 master.cc:584] Master@127.4.65.62:43375 shutdown complete.
4d3f6c31035a4dacad5e2a9d389e5101  5
a9bb0d862776474d9cbb54f9ab5c963b  1
e59ac3f98bdb441ca56a0e914f7c7d1d  4
4d3f6c31035a4dacad5e2a9d389e5101  3
a9bb0d862776474d9cbb54f9ab5c963b  3
e59ac3f98bdb441ca56a0e914f7c7d1d  4
4d3f6c31035a4dacad5e2a9d389e5101  8
a9bb0d862776474d9cbb54f9ab5c963b  1
e59ac3f98bdb441ca56a0e914f7c7d1d  1
4d3f6c31035a4dacad5e2a9d389e5101  4
a9bb0d862776474d9cbb54f9ab5c963b  3
e59ac3f98bdb441ca56a0e914f7c7d1d  3
[       OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (30290 ms)
[ RUN      ] LeaderRebalancerTest.AddTserver
I20260709 12:24:04.657208  4356 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.65.62:38921
I20260709 12:24:04.658592  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:24:04.665305  5434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:24:04.665503  5435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:24:04.666251  5437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:24:04.667984  4356 server_base.cc:1061] running on GCE node
I20260709 12:24:04.669453  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:24:04.669732  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:24:04.669910  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783599844669891 us; error 0 us; skew 500 ppm
I20260709 12:24:04.670447  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:24:04.673137  4356 webserver.cc:533] Webserver started at http://127.4.65.62:43401/ using document root <none> and password file <none>
I20260709 12:24:04.673674  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:24:04.673882  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:24:04.674178  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:24:04.675555  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/master-0-root/instance:
uuid: "b3df7d767dc641b7bb01c3435d842dd5"
format_stamp: "Formatted at 2026-07-09 12:24:04 on dist-test-slave-l1c9"
I20260709 12:24:04.680266  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.007s	sys 0.000s
I20260709 12:24:04.684432  5442 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:24:04.685511  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:24:04.685870  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/master-0-root
uuid: "b3df7d767dc641b7bb01c3435d842dd5"
format_stamp: "Formatted at 2026-07-09 12:24:04 on dist-test-slave-l1c9"
I20260709 12:24:04.686246  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:24:04.711181  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:24:04.712486  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:24:04.756651  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.62:38921
I20260709 12:24:04.756769  5493 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.62:38921 every 8 connection(s)
I20260709 12:24:04.760610  5494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:04.770987  5494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Bootstrap starting.
I20260709 12:24:04.775714  5494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:04.779984  5494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: No bootstrap required, opened a new log
I20260709 12:24:04.782332  5494 raft_consensus.cc:359] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3df7d767dc641b7bb01c3435d842dd5" member_type: VOTER }
I20260709 12:24:04.782768  5494 raft_consensus.cc:385] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:04.783033  5494 raft_consensus.cc:740] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3df7d767dc641b7bb01c3435d842dd5, State: Initialized, Role: FOLLOWER
I20260709 12:24:04.783619  5494 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3df7d767dc641b7bb01c3435d842dd5" member_type: VOTER }
I20260709 12:24:04.784126  5494 raft_consensus.cc:399] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:24:04.784379  5494 raft_consensus.cc:493] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:24:04.784646  5494 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:04.789021  5494 raft_consensus.cc:515] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3df7d767dc641b7bb01c3435d842dd5" member_type: VOTER }
I20260709 12:24:04.789775  5494 leader_election.cc:304] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3df7d767dc641b7bb01c3435d842dd5; no voters: 
I20260709 12:24:04.791481  5494 leader_election.cc:290] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:24:04.791869  5497 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:04.793488  5497 raft_consensus.cc:697] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [term 1 LEADER]: Becoming Leader. State: Replica: b3df7d767dc641b7bb01c3435d842dd5, State: Running, Role: LEADER
I20260709 12:24:04.794232  5497 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3df7d767dc641b7bb01c3435d842dd5" member_type: VOTER }
I20260709 12:24:04.795154  5494 sys_catalog.cc:565] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:24:04.799502  5499 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3df7d767dc641b7bb01c3435d842dd5. Latest consensus state: current_term: 1 leader_uuid: "b3df7d767dc641b7bb01c3435d842dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3df7d767dc641b7bb01c3435d842dd5" member_type: VOTER } }
I20260709 12:24:04.800148  5498 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3df7d767dc641b7bb01c3435d842dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3df7d767dc641b7bb01c3435d842dd5" member_type: VOTER } }
I20260709 12:24:04.800465  5499 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [sys.catalog]: This master's current role is: LEADER
I20260709 12:24:04.800905  5498 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [sys.catalog]: This master's current role is: LEADER
I20260709 12:24:04.802644  5504 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:24:04.807562  5504 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:24:04.811132  4356 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:24:04.817061  5504 catalog_manager.cc:1383] Generated new cluster ID: 66164a65f6e744c6942a93c69318b458
I20260709 12:24:04.817355  5504 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:24:04.844481  5504 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:24:04.845736  5504 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:24:04.865599  5504 catalog_manager.cc:6070] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Generated new TSK 0
I20260709 12:24:04.866219  5504 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:24:04.878139  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:24:04.884968  5515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:24:04.885703  5516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:24:04.887512  5518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:24:04.890261  4356 server_base.cc:1061] running on GCE node
I20260709 12:24:04.891381  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:24:04.891597  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:24:04.891810  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783599844891775 us; error 0 us; skew 500 ppm
I20260709 12:24:04.892491  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:24:04.895099  4356 webserver.cc:533] Webserver started at http://127.4.65.1:36573/ using document root <none> and password file <none>
I20260709 12:24:04.895658  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:24:04.895846  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:24:04.896220  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:24:04.897483  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/instance:
uuid: "cea66aa15be3423880843974acdbf50f"
format_stamp: "Formatted at 2026-07-09 12:24:04 on dist-test-slave-l1c9"
I20260709 12:24:04.902256  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:24:04.906260  5523 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:24:04.907068  4356 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260709 12:24:04.907382  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root
uuid: "cea66aa15be3423880843974acdbf50f"
format_stamp: "Formatted at 2026-07-09 12:24:04 on dist-test-slave-l1c9"
I20260709 12:24:04.907686  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:24:04.923054  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:24:04.924317  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:24:04.925984  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:24:04.928756  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:24:04.929037  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:24:04.929422  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:24:04.929637  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:24:04.982728  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.1:43285
I20260709 12:24:04.982841  5585 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.1:43285 every 8 connection(s)
I20260709 12:24:04.988381  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:24:04.997541  5590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:24:04.999374  5591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:24:05.001103  5593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:24:05.001683  4356 server_base.cc:1061] running on GCE node
I20260709 12:24:05.003221  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:24:05.003496  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:24:05.003683  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783599845003662 us; error 0 us; skew 500 ppm
I20260709 12:24:05.004438  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:24:05.008056  4356 webserver.cc:533] Webserver started at http://127.4.65.2:45115/ using document root <none> and password file <none>
I20260709 12:24:05.008188  5586 heartbeater.cc:344] Connected to a master server at 127.4.65.62:38921
I20260709 12:24:05.008718  5586 heartbeater.cc:461] Registering TS with master...
I20260709 12:24:05.008821  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:24:05.009167  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:24:05.009646  5586 heartbeater.cc:507] Master 127.4.65.62:38921 requested a full tablet report, sending...
I20260709 12:24:05.009792  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:24:05.011365  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/instance:
uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
format_stamp: "Formatted at 2026-07-09 12:24:05 on dist-test-slave-l1c9"
I20260709 12:24:05.011965  5459 ts_manager.cc:194] Registered new tserver with Master: cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:05.013859  5459 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35586
I20260709 12:24:05.017130  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.005s
I20260709 12:24:05.021131  5598 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:24:05.022114  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:24:05.022553  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root
uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
format_stamp: "Formatted at 2026-07-09 12:24:05 on dist-test-slave-l1c9"
I20260709 12:24:05.022955  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:24:05.038429  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:24:05.039701  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:24:05.041558  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:24:05.043900  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:24:05.044142  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:24:05.044441  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:24:05.044620  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:24:05.086863  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.2:38587
I20260709 12:24:05.086957  5660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.2:38587 every 8 connection(s)
I20260709 12:24:05.091980  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:24:05.099332  5664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:24:05.101492  5665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:24:05.103255  5667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:24:05.103591  4356 server_base.cc:1061] running on GCE node
I20260709 12:24:05.104831  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:24:05.105134  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:24:05.105388  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783599845105368 us; error 0 us; skew 500 ppm
I20260709 12:24:05.105542  5661 heartbeater.cc:344] Connected to a master server at 127.4.65.62:38921
I20260709 12:24:05.106065  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:24:05.106076  5661 heartbeater.cc:461] Registering TS with master...
I20260709 12:24:05.107352  5661 heartbeater.cc:507] Master 127.4.65.62:38921 requested a full tablet report, sending...
I20260709 12:24:05.109519  4356 webserver.cc:533] Webserver started at http://127.4.65.3:43297/ using document root <none> and password file <none>
I20260709 12:24:05.110208  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:24:05.110246  5459 ts_manager.cc:194] Registered new tserver with Master: 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:05.110585  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:24:05.110921  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:24:05.112191  5459 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35600
I20260709 12:24:05.112534  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root/instance:
uuid: "f613ba929b1046d3b5aed909414a95bf"
format_stamp: "Formatted at 2026-07-09 12:24:05 on dist-test-slave-l1c9"
I20260709 12:24:05.117412  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:24:05.120980  5672 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:24:05.121809  4356 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.002s
I20260709 12:24:05.122141  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root
uuid: "f613ba929b1046d3b5aed909414a95bf"
format_stamp: "Formatted at 2026-07-09 12:24:05 on dist-test-slave-l1c9"
I20260709 12:24:05.122462  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:24:05.142802  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:24:05.144035  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:24:05.145524  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:24:05.147686  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:24:05.147909  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:24:05.148159  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:24:05.148330  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:24:05.191488  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.3:45979
I20260709 12:24:05.191576  5734 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.3:45979 every 8 connection(s)
I20260709 12:24:05.205896  5735 heartbeater.cc:344] Connected to a master server at 127.4.65.62:38921
I20260709 12:24:05.206297  5735 heartbeater.cc:461] Registering TS with master...
I20260709 12:24:05.206950  5735 heartbeater.cc:507] Master 127.4.65.62:38921 requested a full tablet report, sending...
I20260709 12:24:05.208855  5459 ts_manager.cc:194] Registered new tserver with Master: f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:05.209856  4356 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015279467s
I20260709 12:24:05.210310  5459 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35602
I20260709 12:24:05.239303  5459 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35616:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:24:05.245496  5459 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:24:05.458741  5700 tablet_service.cc:1511] Processing CreateTablet for tablet 0b93fd2210654c988a29dff61aabacb6 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:24:05.461606  5699 tablet_service.cc:1511] Processing CreateTablet for tablet b0a3a361d19b43b9a0a562f49f9b719e (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:24:05.466506  5551 tablet_service.cc:1511] Processing CreateTablet for tablet 0b93fd2210654c988a29dff61aabacb6 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:24:05.466569  5697 tablet_service.cc:1511] Processing CreateTablet for tablet feb9fb12cc5a41e6958575c17961252f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:24:05.468050  5551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b93fd2210654c988a29dff61aabacb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.467749  5548 tablet_service.cc:1511] Processing CreateTablet for tablet feb9fb12cc5a41e6958575c17961252f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:24:05.469102  5548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feb9fb12cc5a41e6958575c17961252f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.472071  5696 tablet_service.cc:1511] Processing CreateTablet for tablet 1af549c25a724a2f9d4f65cc1c59f22a (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:24:05.473038  5550 tablet_service.cc:1511] Processing CreateTablet for tablet b0a3a361d19b43b9a0a562f49f9b719e (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:24:05.474363  5550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a3a361d19b43b9a0a562f49f9b719e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.474490  5695 tablet_service.cc:1511] Processing CreateTablet for tablet 378854044c0e42f4b0a737cdbdbcc464 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:24:05.460230  5700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b93fd2210654c988a29dff61aabacb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.466596  5549 tablet_service.cc:1511] Processing CreateTablet for tablet e69273530292453d87614627debcc13e (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:24:05.479370  5549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e69273530292453d87614627debcc13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.469120  5698 tablet_service.cc:1511] Processing CreateTablet for tablet e69273530292453d87614627debcc13e (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:24:05.481252  5547 tablet_service.cc:1511] Processing CreateTablet for tablet 1af549c25a724a2f9d4f65cc1c59f22a (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:24:05.481527  5543 tablet_service.cc:1511] Processing CreateTablet for tablet 10927a6912ad4d40b0d6c0d9e9cd2a1e (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:24:05.482599  5547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1af549c25a724a2f9d4f65cc1c59f22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.482951  5544 tablet_service.cc:1511] Processing CreateTablet for tablet 4ad0bd0e70f549b7bdcfb140817535f7 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:24:05.484251  5544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ad0bd0e70f549b7bdcfb140817535f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.485576  5546 tablet_service.cc:1511] Processing CreateTablet for tablet 378854044c0e42f4b0a737cdbdbcc464 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:24:05.486924  5546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 378854044c0e42f4b0a737cdbdbcc464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.487346  5694 tablet_service.cc:1511] Processing CreateTablet for tablet addbcf2919194d2e866b5e8826422851 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:24:05.488729  5543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10927a6912ad4d40b0d6c0d9e9cd2a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.481931  5698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e69273530292453d87614627debcc13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.490208  5693 tablet_service.cc:1511] Processing CreateTablet for tablet 4ad0bd0e70f549b7bdcfb140817535f7 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:24:05.491920  5693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ad0bd0e70f549b7bdcfb140817535f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.493829  5692 tablet_service.cc:1511] Processing CreateTablet for tablet 10927a6912ad4d40b0d6c0d9e9cd2a1e (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:24:05.495272  5692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10927a6912ad4d40b0d6c0d9e9cd2a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.499867  5542 tablet_service.cc:1511] Processing CreateTablet for tablet 1b585af92ddc4f4fa5bada0f9b198495 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:24:05.501292  5542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b585af92ddc4f4fa5bada0f9b198495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.501930  5695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 378854044c0e42f4b0a737cdbdbcc464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.503953  5699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a3a361d19b43b9a0a562f49f9b719e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.490198  5545 tablet_service.cc:1511] Processing CreateTablet for tablet addbcf2919194d2e866b5e8826422851 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:24:05.505139  5545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addbcf2919194d2e866b5e8826422851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.506973  5696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1af549c25a724a2f9d4f65cc1c59f22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.509652  5691 tablet_service.cc:1511] Processing CreateTablet for tablet 1b585af92ddc4f4fa5bada0f9b198495 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:24:05.511117  5697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feb9fb12cc5a41e6958575c17961252f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.512547  5694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addbcf2919194d2e866b5e8826422851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.514271  5691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b585af92ddc4f4fa5bada0f9b198495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.555797  5626 tablet_service.cc:1511] Processing CreateTablet for tablet 0b93fd2210654c988a29dff61aabacb6 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:24:05.557592  5626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b93fd2210654c988a29dff61aabacb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.561705  5625 tablet_service.cc:1511] Processing CreateTablet for tablet b0a3a361d19b43b9a0a562f49f9b719e (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:24:05.563288  5625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a3a361d19b43b9a0a562f49f9b719e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.565737  5624 tablet_service.cc:1511] Processing CreateTablet for tablet e69273530292453d87614627debcc13e (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:24:05.568369  5623 tablet_service.cc:1511] Processing CreateTablet for tablet feb9fb12cc5a41e6958575c17961252f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:24:05.570979  5622 tablet_service.cc:1511] Processing CreateTablet for tablet 1af549c25a724a2f9d4f65cc1c59f22a (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:24:05.573424  5621 tablet_service.cc:1511] Processing CreateTablet for tablet 378854044c0e42f4b0a737cdbdbcc464 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:24:05.576930  5621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 378854044c0e42f4b0a737cdbdbcc464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.578357  5623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet feb9fb12cc5a41e6958575c17961252f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.579620  5622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1af549c25a724a2f9d4f65cc1c59f22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.581121  5624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e69273530292453d87614627debcc13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.597076  5548 tablet_service.cc:1511] Processing CreateTablet for tablet 44c764c1f49048f28e1caefabe1cc376 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:24:05.598717  5548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c764c1f49048f28e1caefabe1cc376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.599457  5620 tablet_service.cc:1511] Processing CreateTablet for tablet addbcf2919194d2e866b5e8826422851 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:24:05.601078  5620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addbcf2919194d2e866b5e8826422851. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.608134  5619 tablet_service.cc:1511] Processing CreateTablet for tablet 4ad0bd0e70f549b7bdcfb140817535f7 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:24:05.610498  5619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ad0bd0e70f549b7bdcfb140817535f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.618808  5618 tablet_service.cc:1511] Processing CreateTablet for tablet 10927a6912ad4d40b0d6c0d9e9cd2a1e (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:24:05.621306  5618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10927a6912ad4d40b0d6c0d9e9cd2a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.708441  5617 tablet_service.cc:1511] Processing CreateTablet for tablet 1b585af92ddc4f4fa5bada0f9b198495 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:24:05.710106  5617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b585af92ddc4f4fa5bada0f9b198495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.712913  5551 tablet_service.cc:1511] Processing CreateTablet for tablet 9ff38a877a89473392a3e27657516383 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:24:05.722860  5550 tablet_service.cc:1511] Processing CreateTablet for tablet 6f938b4d80d34f89a34d90b3c31fa84a (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:24:05.724344  5550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f938b4d80d34f89a34d90b3c31fa84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.742396  5551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ff38a877a89473392a3e27657516383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.747346  5549 tablet_service.cc:1511] Processing CreateTablet for tablet 423d6357e8364cb3b7725b1735b560ac (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:24:05.750226  5549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423d6357e8364cb3b7725b1735b560ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.753209  5698 tablet_service.cc:1511] Processing CreateTablet for tablet 44c764c1f49048f28e1caefabe1cc376 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:24:05.754577  5698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c764c1f49048f28e1caefabe1cc376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.756170  5548 tablet_service.cc:1511] Processing CreateTablet for tablet 607035a460944cc2b065c4d4cc59f813 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:24:05.757489  5548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607035a460944cc2b065c4d4cc59f813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.769480  5695 tablet_service.cc:1511] Processing CreateTablet for tablet 9ff38a877a89473392a3e27657516383 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:24:05.771138  5695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ff38a877a89473392a3e27657516383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.771600  5546 tablet_service.cc:1511] Processing CreateTablet for tablet 2acd6f5df0204b2295547e021cd8cb9f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:24:05.772585  5545 tablet_service.cc:1511] Processing CreateTablet for tablet 4d439f3f9c7c40288efc981cec05e7ab (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:24:05.777712  5626 tablet_service.cc:1511] Processing CreateTablet for tablet 44c764c1f49048f28e1caefabe1cc376 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:24:05.779000  5626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c764c1f49048f28e1caefabe1cc376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.772979  5546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2acd6f5df0204b2295547e021cd8cb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.794873  5545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d439f3f9c7c40288efc981cec05e7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.806167  5752 tablet_bootstrap.cc:492] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
W20260709 12:24:05.811079  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:24:05.814504  5700 tablet_service.cc:1511] Processing CreateTablet for tablet 6f938b4d80d34f89a34d90b3c31fa84a (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:24:05.815172  5621 tablet_service.cc:1511] Processing CreateTablet for tablet 9ff38a877a89473392a3e27657516383 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:24:05.816078  5700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f938b4d80d34f89a34d90b3c31fa84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.816406  5621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ff38a877a89473392a3e27657516383. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.816754  5752 tablet_bootstrap.cc:654] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:05.818192  5547 tablet_service.cc:1511] Processing CreateTablet for tablet 90c64867bae6493c846d941815aa1074 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:24:05.819422  5547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c64867bae6493c846d941815aa1074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.821709  5691 tablet_service.cc:1511] Processing CreateTablet for tablet 423d6357e8364cb3b7725b1735b560ac (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:24:05.822932  5691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423d6357e8364cb3b7725b1735b560ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.824440  5750 tablet_bootstrap.cc:492] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:05.831333  5696 tablet_service.cc:1511] Processing CreateTablet for tablet 4d439f3f9c7c40288efc981cec05e7ab (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:24:05.825572  5699 tablet_service.cc:1511] Processing CreateTablet for tablet 2acd6f5df0204b2295547e021cd8cb9f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:24:05.832729  5696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d439f3f9c7c40288efc981cec05e7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.833489  5699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2acd6f5df0204b2295547e021cd8cb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.833701  5551 tablet_service.cc:1511] Processing CreateTablet for tablet 34a1556938b2414c9e883f1659d8fa0d (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:24:05.834925  5551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a1556938b2414c9e883f1659d8fa0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.801857  5549 tablet_service.cc:1511] Processing CreateTablet for tablet 637a73babedf4b1aa47b3b50937db9ef (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:24:05.845510  5549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637a73babedf4b1aa47b3b50937db9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.845424  5542 tablet_service.cc:1511] Processing CreateTablet for tablet 11fc89db1c9243b2af65c665c04ba84e (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:24:05.846791  5542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11fc89db1c9243b2af65c665c04ba84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.850615  5625 tablet_service.cc:1511] Processing CreateTablet for tablet 6f938b4d80d34f89a34d90b3c31fa84a (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:24:05.851929  5625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f938b4d80d34f89a34d90b3c31fa84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.855587  5543 tablet_service.cc:1511] Processing CreateTablet for tablet df7bbda98d6948f9a851d59f4954b6bb (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:24:05.857347  5750 tablet_bootstrap.cc:654] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:05.857506  5544 tablet_service.cc:1511] Processing CreateTablet for tablet fa294357a99443e8bd25587ba629d3de (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:24:05.858807  5544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa294357a99443e8bd25587ba629d3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.861617  5620 tablet_service.cc:1511] Processing CreateTablet for tablet 607035a460944cc2b065c4d4cc59f813 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:24:05.862989  5620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607035a460944cc2b065c4d4cc59f813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.863539  5751 tablet_bootstrap.cc:492] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:05.882265  5693 tablet_service.cc:1511] Processing CreateTablet for tablet 637a73babedf4b1aa47b3b50937db9ef (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:24:05.883597  5693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637a73babedf4b1aa47b3b50937db9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.893335  5622 tablet_service.cc:1511] Processing CreateTablet for tablet 2acd6f5df0204b2295547e021cd8cb9f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:24:05.897119  5543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df7bbda98d6948f9a851d59f4954b6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.902762  5697 tablet_service.cc:1511] Processing CreateTablet for tablet 90c64867bae6493c846d941815aa1074 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:24:05.904134  5697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c64867bae6493c846d941815aa1074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.922904  5619 tablet_service.cc:1511] Processing CreateTablet for tablet 423d6357e8364cb3b7725b1735b560ac (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:24:05.928925  5695 tablet_service.cc:1511] Processing CreateTablet for tablet 11fc89db1c9243b2af65c665c04ba84e (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:24:05.930595  5695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11fc89db1c9243b2af65c665c04ba84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.934660  5699 tablet_service.cc:1511] Processing CreateTablet for tablet df7bbda98d6948f9a851d59f4954b6bb (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:24:05.937983  5694 tablet_service.cc:1511] Processing CreateTablet for tablet fa294357a99443e8bd25587ba629d3de (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:24:05.939363  5694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa294357a99443e8bd25587ba629d3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.940820  5624 tablet_service.cc:1511] Processing CreateTablet for tablet ccec0cc7269743d6ad581695ea601d1f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:24:05.924016  5692 tablet_service.cc:1511] Processing CreateTablet for tablet 607035a460944cc2b065c4d4cc59f813 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:24:05.945822  5692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607035a460944cc2b065c4d4cc59f813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.950358  5691 tablet_service.cc:1511] Processing CreateTablet for tablet 786a49bad9be400ea9ce584030aad446 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:24:05.951684  5699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df7bbda98d6948f9a851d59f4954b6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.954423  5752 tablet_bootstrap.cc:492] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:05.963959  5752 ts_tablet_manager.cc:1403] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.158s	user 0.027s	sys 0.019s
I20260709 12:24:05.956766  5622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2acd6f5df0204b2295547e021cd8cb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.968490  5691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 786a49bad9be400ea9ce584030aad446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.955578  5623 tablet_service.cc:1511] Processing CreateTablet for tablet 4d439f3f9c7c40288efc981cec05e7ab (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:24:05.968742  5619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423d6357e8364cb3b7725b1735b560ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.975989  5623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d439f3f9c7c40288efc981cec05e7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.976186  5618 tablet_service.cc:1511] Processing CreateTablet for tablet 90c64867bae6493c846d941815aa1074 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:24:05.967350  5752 raft_consensus.cc:359] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:05.981043  5624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccec0cc7269743d6ad581695ea601d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.993320  5618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c64867bae6493c846d941815aa1074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.994020  5752 raft_consensus.cc:385] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:05.995257  5752 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:05.995335  5548 tablet_service.cc:1511] Processing CreateTablet for tablet ac4fdba265724dea9f440080f8d803e2 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:24:05.997859  5751 tablet_bootstrap.cc:654] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:05.998129  5752 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:05.989042  5626 tablet_service.cc:1511] Processing CreateTablet for tablet 637a73babedf4b1aa47b3b50937db9ef (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:24:06.003230  5626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637a73babedf4b1aa47b3b50937db9ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:05.990657  5700 tablet_service.cc:1511] Processing CreateTablet for tablet 34a1556938b2414c9e883f1659d8fa0d (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:24:05.958650  5547 tablet_service.cc:1511] Processing CreateTablet for tablet f0f7b2d2a7e34d629491c96dbfbc7fdc (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:24:06.015487  5750 tablet_bootstrap.cc:492] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:06.016018  5750 ts_tablet_manager.cc:1403] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.196s	user 0.011s	sys 0.006s
I20260709 12:24:06.019129  5621 tablet_service.cc:1511] Processing CreateTablet for tablet df7bbda98d6948f9a851d59f4954b6bb (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:24:06.019364  5549 tablet_service.cc:1511] Processing CreateTablet for tablet af208afb558f47dab74acae8d2b8da60 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:24:06.020824  5550 tablet_service.cc:1511] Processing CreateTablet for tablet ccec0cc7269743d6ad581695ea601d1f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:24:06.023478  5750 raft_consensus.cc:359] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.025765  5750 raft_consensus.cc:385] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:06.026081  5750 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:06.026860  5750 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.013631  5548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4fdba265724dea9f440080f8d803e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.036706  5586 heartbeater.cc:499] Master 127.4.65.62:38921 was elected leader, sending a full tablet report...
I20260709 12:24:06.041174  5546 tablet_service.cc:1511] Processing CreateTablet for tablet abd24f2e500a44729c9111fae83c25ed (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:24:06.041335  5617 tablet_service.cc:1511] Processing CreateTablet for tablet 34a1556938b2414c9e883f1659d8fa0d (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:24:06.042419  5547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0f7b2d2a7e34d629491c96dbfbc7fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.024566  5545 tablet_service.cc:1511] Processing CreateTablet for tablet 786a49bad9be400ea9ce584030aad446 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:24:06.025075  5700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a1556938b2414c9e883f1659d8fa0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.025426  5621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df7bbda98d6948f9a851d59f4954b6bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.053844  5623 tablet_service.cc:1511] Processing CreateTablet for tablet fa294357a99443e8bd25587ba629d3de (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:24:06.055608  5623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa294357a99443e8bd25587ba629d3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.061246  5620 tablet_service.cc:1511] Processing CreateTablet for tablet 11fc89db1c9243b2af65c665c04ba84e (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:24:06.063104  5620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11fc89db1c9243b2af65c665c04ba84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.066792  5695 tablet_service.cc:1511] Processing CreateTablet for tablet ccec0cc7269743d6ad581695ea601d1f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:24:06.068101  5695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccec0cc7269743d6ad581695ea601d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.074644  5698 tablet_service.cc:1511] Processing CreateTablet for tablet f0f7b2d2a7e34d629491c96dbfbc7fdc (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:24:06.076028  5698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0f7b2d2a7e34d629491c96dbfbc7fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.080178  5693 tablet_service.cc:1511] Processing CreateTablet for tablet abd24f2e500a44729c9111fae83c25ed (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:24:06.081871  5693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd24f2e500a44729c9111fae83c25ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.092978  5545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 786a49bad9be400ea9ce584030aad446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.079777  5619 tablet_service.cc:1511] Processing CreateTablet for tablet f0f7b2d2a7e34d629491c96dbfbc7fdc (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:24:06.039917  5543 tablet_service.cc:1511] Processing CreateTablet for tablet 52ccafecca54436198d9d455d076ef8d (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:24:06.119998  5546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd24f2e500a44729c9111fae83c25ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.121346  5617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34a1556938b2414c9e883f1659d8fa0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.119757  5661 heartbeater.cc:499] Master 127.4.65.62:38921 was elected leader, sending a full tablet report...
I20260709 12:24:06.133855  5752 ts_tablet_manager.cc:1434] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.169s	user 0.019s	sys 0.046s
I20260709 12:24:06.135142  5752 tablet_bootstrap.cc:492] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:06.141251  5622 tablet_service.cc:1511] Processing CreateTablet for tablet 786a49bad9be400ea9ce584030aad446 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:24:06.150134  5619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0f7b2d2a7e34d629491c96dbfbc7fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.153507  5542 tablet_service.cc:1511] Processing CreateTablet for tablet 9efccdcce1164ab5b57aa40caea54717 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:24:06.155079  5549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af208afb558f47dab74acae8d2b8da60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.156672  5550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccec0cc7269743d6ad581695ea601d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.157639  5622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 786a49bad9be400ea9ce584030aad446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.158315  5543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ccafecca54436198d9d455d076ef8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.169986  5542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9efccdcce1164ab5b57aa40caea54717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.166116  5618 tablet_service.cc:1511] Processing CreateTablet for tablet af208afb558f47dab74acae8d2b8da60 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:24:06.172869  5618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af208afb558f47dab74acae8d2b8da60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.180933  5751 tablet_bootstrap.cc:492] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:06.175797  5548 tablet_service.cc:1511] Processing CreateTablet for tablet 8b21feec8630479b94ed9384256483fc (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:24:06.181476  5751 ts_tablet_manager.cc:1403] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.318s	user 0.016s	sys 0.007s
I20260709 12:24:06.188294  5548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b21feec8630479b94ed9384256483fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.200551  5624 tablet_service.cc:1511] Processing CreateTablet for tablet 9efccdcce1164ab5b57aa40caea54717 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:24:06.202277  5624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9efccdcce1164ab5b57aa40caea54717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.202958  5692 tablet_service.cc:1511] Processing CreateTablet for tablet ac4fdba265724dea9f440080f8d803e2 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:24:06.204197  5692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4fdba265724dea9f440080f8d803e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.206175  5750 ts_tablet_manager.cc:1434] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.190s	user 0.011s	sys 0.010s
I20260709 12:24:06.207196  5750 tablet_bootstrap.cc:492] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:06.184286  5751 raft_consensus.cc:359] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.209741  5751 raft_consensus.cc:385] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:06.210129  5751 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:06.211128  5751 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.222754  5735 heartbeater.cc:499] Master 127.4.65.62:38921 was elected leader, sending a full tablet report...
I20260709 12:24:06.229321  5757 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.229810  5757 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.233537  5698 tablet_service.cc:1511] Processing CreateTablet for tablet 52ccafecca54436198d9d455d076ef8d (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:24:06.234871  5698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ccafecca54436198d9d455d076ef8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.188182  5620 tablet_service.cc:1511] Processing CreateTablet for tablet abd24f2e500a44729c9111fae83c25ed (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:24:06.260071  5694 tablet_service.cc:1511] Processing CreateTablet for tablet 9efccdcce1164ab5b57aa40caea54717 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:24:06.261520  5694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9efccdcce1164ab5b57aa40caea54717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.148344  5699 tablet_service.cc:1511] Processing CreateTablet for tablet af208afb558f47dab74acae8d2b8da60 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:24:06.265913  5699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af208afb558f47dab74acae8d2b8da60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.270107  5625 tablet_service.cc:1511] Processing CreateTablet for tablet ac4fdba265724dea9f440080f8d803e2 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:24:06.274813  5752 tablet_bootstrap.cc:654] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:06.276907  5620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abd24f2e500a44729c9111fae83c25ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.285597  5545 tablet_service.cc:1511] Processing CreateTablet for tablet 52e5c61368994c6bbeee20e7007995e2 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:24:06.286494  5544 tablet_service.cc:1511] Processing CreateTablet for tablet 3ca6496163ec40978a2dfd744edddb18 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:24:06.288148  5697 tablet_service.cc:1511] Processing CreateTablet for tablet 8b21feec8630479b94ed9384256483fc (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:24:06.289489  5697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b21feec8630479b94ed9384256483fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.281101  5750 tablet_bootstrap.cc:654] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:06.286926  5545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52e5c61368994c6bbeee20e7007995e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.238297  5757 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:06.318092  5623 tablet_service.cc:1511] Processing CreateTablet for tablet 52ccafecca54436198d9d455d076ef8d (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:24:06.322374  5621 tablet_service.cc:1511] Processing CreateTablet for tablet 8b21feec8630479b94ed9384256483fc (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:24:06.320200  5623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52ccafecca54436198d9d455d076ef8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.328727  5625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac4fdba265724dea9f440080f8d803e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.331542  5544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca6496163ec40978a2dfd744edddb18. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:24:06.343641  5662 tablet.cc:2406] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:24:06.344595  5622 tablet_service.cc:1511] Processing CreateTablet for tablet 52e5c61368994c6bbeee20e7007995e2 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:24:06.357754  5621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b21feec8630479b94ed9384256483fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.365777  5547 tablet_service.cc:1511] Processing CreateTablet for tablet a3cceeae7fc046fb99b738e957155815 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:24:06.367587  5622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52e5c61368994c6bbeee20e7007995e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.373680  5751 ts_tablet_manager.cc:1434] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.192s	user 0.013s	sys 0.012s
I20260709 12:24:06.393755  5751 tablet_bootstrap.cc:492] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:06.395558  5696 tablet_service.cc:1511] Processing CreateTablet for tablet 52e5c61368994c6bbeee20e7007995e2 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:24:06.396648  5763 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.397265  5763 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.399678  5691 tablet_service.cc:1511] Processing CreateTablet for tablet 3ca6496163ec40978a2dfd744edddb18 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:24:06.400043  5763 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:06.379349  5549 tablet_service.cc:1511] Processing CreateTablet for tablet b3353dcd7bce4f9bbf059a7205c5bf3d (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
W20260709 12:24:06.384776  5445 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 63403.1us
W20260709 12:24:06.385986  5443 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47749.5us
I20260709 12:24:06.422436  5692 tablet_service.cc:1511] Processing CreateTablet for tablet a3cceeae7fc046fb99b738e957155815 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:24:06.422436  5550 tablet_service.cc:1511] Processing CreateTablet for tablet 6adc8ae91dfe46039ab5bd2bc4c69708 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:24:06.429322  5752 tablet_bootstrap.cc:492] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:06.429845  5752 ts_tablet_manager.cc:1403] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.295s	user 0.013s	sys 0.000s
I20260709 12:24:06.411823  5750 tablet_bootstrap.cc:492] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:06.448542  5619 tablet_service.cc:1511] Processing CreateTablet for tablet 3ca6496163ec40978a2dfd744edddb18 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:24:06.450399  5619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca6496163ec40978a2dfd744edddb18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.439791  5696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52e5c61368994c6bbeee20e7007995e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.453315  5547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cceeae7fc046fb99b738e957155815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.453022  5750 ts_tablet_manager.cc:1403] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.246s	user 0.012s	sys 0.007s
I20260709 12:24:06.470541  5691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca6496163ec40978a2dfd744edddb18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.408550  5551 tablet_service.cc:1511] Processing CreateTablet for tablet bf758b480ecf41619471236195bf1838 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:24:06.484764  5755 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.486567  5755 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.493602  5693 tablet_service.cc:1511] Processing CreateTablet for tablet bf758b480ecf41619471236195bf1838 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:24:06.495113  5693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf758b480ecf41619471236195bf1838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.497273  5617 tablet_service.cc:1511] Processing CreateTablet for tablet b3353dcd7bce4f9bbf059a7205c5bf3d (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:24:06.498654  5617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3353dcd7bce4f9bbf059a7205c5bf3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.489413  5698 tablet_service.cc:1511] Processing CreateTablet for tablet b3353dcd7bce4f9bbf059a7205c5bf3d (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:24:06.505651  5698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3353dcd7bce4f9bbf059a7205c5bf3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.511592  5751 tablet_bootstrap.cc:654] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:06.511238  5626 tablet_service.cc:1511] Processing CreateTablet for tablet a3cceeae7fc046fb99b738e957155815 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:24:06.513202  5626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cceeae7fc046fb99b738e957155815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.488942  5549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3353dcd7bce4f9bbf059a7205c5bf3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.518713  5750 raft_consensus.cc:359] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.519543  5750 raft_consensus.cc:385] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:06.519879  5750 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:06.532120  5755 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:06.528223  5750 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.534819  5750 ts_tablet_manager.cc:1434] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.069s	user 0.006s	sys 0.000s
I20260709 12:24:06.532094  5622 tablet_service.cc:1511] Processing CreateTablet for tablet 6adc8ae91dfe46039ab5bd2bc4c69708 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:24:06.536442  5757 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.537391  5622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6adc8ae91dfe46039ab5bd2bc4c69708. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.537060  5757 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.537822  5550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6adc8ae91dfe46039ab5bd2bc4c69708. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.542028  5692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3cceeae7fc046fb99b738e957155815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.556602  5750 tablet_bootstrap.cc:492] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:06.560966  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:06.528888  5618 tablet_service.cc:1511] Processing CreateTablet for tablet bf758b480ecf41619471236195bf1838 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:24:06.563933  5618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf758b480ecf41619471236195bf1838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.563982  5752 raft_consensus.cc:359] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:06.564859  5752 raft_consensus.cc:385] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:06.565164  5752 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:06.565872  5752 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:06.573470  5757 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:06.574532  5446 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56000.3us
I20260709 12:24:06.575735  5551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf758b480ecf41619471236195bf1838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.578637  5699 tablet_service.cc:1511] Processing CreateTablet for tablet 6adc8ae91dfe46039ab5bd2bc4c69708 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:24:06.579896  5699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6adc8ae91dfe46039ab5bd2bc4c69708. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.585948  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:06.587785  5676 leader_election.cc:343] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:24:06.588742  5445 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44120.4us
I20260709 12:24:06.599148  5750 tablet_bootstrap.cc:654] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:06.609488  5751 tablet_bootstrap.cc:492] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:06.609973  5751 ts_tablet_manager.cc:1403] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.216s	user 0.014s	sys 0.000s
I20260709 12:24:06.612195  5751 raft_consensus.cc:359] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.612869  5751 raft_consensus.cc:385] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:06.613117  5751 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:06.613794  5751 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.615808  5751 ts_tablet_manager.cc:1434] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:24:06.630798  5545 tablet_service.cc:1511] Processing CreateTablet for tablet 832269e5980743c48521740b22eefa9f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:24:06.632802  5545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 832269e5980743c48521740b22eefa9f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:24:06.632951  5524 leader_election.cc:343] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:06.633296  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:06.635421  5694 tablet_service.cc:1511] Processing CreateTablet for tablet 832269e5980743c48521740b22eefa9f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:24:06.636572  5700 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:38456 (request call id 19) took 1092 ms. Trace:
I20260709 12:24:06.637018  5700 rpcz_store.cc:276] 0709 12:24:05.544051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.912984 (+368933us) service_pool.cc:224] Handling call
0709 12:24:06.009335 (+ 96351us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:24:06.024591 (+ 15256us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:24:06.390249 (+365658us) tablet_metadata.cc:665] Metadata flushed
0709 12:24:06.599486 (+209237us) tablet_replica.cc:206] Creating consensus instance
0709 12:24:06.636509 (+ 37023us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:06.641139  5524 leader_election.cc:343] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:06.647614  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:06.648392  5636 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:06.656909  5751 tablet_bootstrap.cc:492] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:06.668774  5697 tablet_service.cc:1511] Processing CreateTablet for tablet e20e319f46504838a20c558a66692da6 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:24:06.672652  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:06.673903  5763 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:06.674248  5763 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.674539  5763 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:06.678710  5694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 832269e5980743c48521740b22eefa9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.679863  5550 tablet_service.cc:1511] Processing CreateTablet for tablet e20e319f46504838a20c558a66692da6 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:24:06.681777  5550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20e319f46504838a20c558a66692da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.693389  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:06.697779  5697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20e319f46504838a20c558a66692da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.698385  5621 tablet_service.cc:1511] Processing CreateTablet for tablet 35344119b8ff4cdebc4810661ff94f79 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:24:06.698801  5698 tablet_service.cc:1511] Processing CreateTablet for tablet 35344119b8ff4cdebc4810661ff94f79 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:24:06.699656  5621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35344119b8ff4cdebc4810661ff94f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.701081  5763 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.702282  5710 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 1: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:06.702448  5698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35344119b8ff4cdebc4810661ff94f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.705195  5622 tablet_service.cc:1511] Processing CreateTablet for tablet e20e319f46504838a20c558a66692da6 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:24:06.707007  5622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20e319f46504838a20c558a66692da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.710597  5625 tablet_service.cc:1511] Processing CreateTablet for tablet 832269e5980743c48521740b22eefa9f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:24:06.712126  5625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 832269e5980743c48521740b22eefa9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.712778  5551 tablet_service.cc:1511] Processing CreateTablet for tablet 785e79f6fbd0463aafaca5498918a142 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:24:06.724540  5551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785e79f6fbd0463aafaca5498918a142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.724308  5623 tablet_service.cc:1511] Processing CreateTablet for tablet 785e79f6fbd0463aafaca5498918a142 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:24:06.725488  5623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785e79f6fbd0463aafaca5498918a142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.727135  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:06.719558  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:06.721990  5549 tablet_service.cc:1511] Processing CreateTablet for tablet 52cee1d09af749489de4689a67047497 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:24:06.729296  5549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52cee1d09af749489de4689a67047497. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.729592  5751 tablet_bootstrap.cc:654] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:06.731385  5752 ts_tablet_manager.cc:1434] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.301s	user 0.010s	sys 0.000s
I20260709 12:24:06.731702  5696 tablet_service.cc:1511] Processing CreateTablet for tablet 52cee1d09af749489de4689a67047497 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:24:06.722241  5617 tablet_service.cc:1511] Processing CreateTablet for tablet 52cee1d09af749489de4689a67047497 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:24:06.720969  5692 tablet_service.cc:1511] Processing CreateTablet for tablet 785e79f6fbd0463aafaca5498918a142 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:24:06.722903  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:06.734678  5617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52cee1d09af749489de4689a67047497. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:24:06.739681  5602 leader_election.cc:343] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:06.739910  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:06.740226  5602 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:06.741341  5755 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:06.741886  5696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52cee1d09af749489de4689a67047497. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.743784  5692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785e79f6fbd0463aafaca5498918a142. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:24:06.743587  5526 leader_election.cc:343] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:24:06.744278  5676 leader_election.cc:343] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:06.744287  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:06.745523  5757 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:06.695016  5545 tablet_service.cc:1511] Processing CreateTablet for tablet 35344119b8ff4cdebc4810661ff94f79 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
W20260709 12:24:06.746515  5526 leader_election.cc:343] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:06.746876  5545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35344119b8ff4cdebc4810661ff94f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.746940  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:06.748019  5757 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:06.750571  5750 tablet_bootstrap.cc:492] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:06.751197  5750 ts_tablet_manager.cc:1403] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.216s	user 0.030s	sys 0.030s
I20260709 12:24:06.739735  5752 tablet_bootstrap.cc:492] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:06.764775  5544 tablet_service.cc:1511] Processing CreateTablet for tablet 64af413ffb404a58b5adea5cb742bf01 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:24:06.766398  5544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64af413ffb404a58b5adea5cb742bf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.777042  5763 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:06.778262  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:06.783329  5755 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:06.785218  5750 raft_consensus.cc:359] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.786032  5750 raft_consensus.cc:385] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:06.786339  5750 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:06.792531  5619 tablet_service.cc:1511] Processing CreateTablet for tablet 64af413ffb404a58b5adea5cb742bf01 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:24:06.792532  5697 tablet_service.cc:1511] Processing CreateTablet for tablet 64af413ffb404a58b5adea5cb742bf01 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:24:06.793857  5547 tablet_service.cc:1511] Processing CreateTablet for tablet 14483990837441a99fa83b4a981b4162 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:24:06.795357  5547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14483990837441a99fa83b4a981b4162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.797152  5697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64af413ffb404a58b5adea5cb742bf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.794272  5619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64af413ffb404a58b5adea5cb742bf01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.808854  5751 tablet_bootstrap.cc:492] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:06.809398  5751 ts_tablet_manager.cc:1403] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.188s	user 0.012s	sys 0.001s
I20260709 12:24:06.812644  5635 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
W20260709 12:24:06.813614  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:24:06.814432  5752 tablet_bootstrap.cc:654] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:06.814780  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:06.816035  5763 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:06.787181  5750 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.818171  5750 ts_tablet_manager.cc:1434] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.067s	user 0.008s	sys 0.000s
I20260709 12:24:06.826232  5751 raft_consensus.cc:359] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.826999  5751 raft_consensus.cc:385] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:06.827288  5751 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:06.832489  5750 tablet_bootstrap.cc:492] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:06.836397  5763 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:06.842697  5698 tablet_service.cc:1511] Processing CreateTablet for tablet 14483990837441a99fa83b4a981b4162 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:24:06.844125  5698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14483990837441a99fa83b4a981b4162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.844276  5621 tablet_service.cc:1511] Processing CreateTablet for tablet 26edf338d41e42f1a5ce0d4416ec67e1 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:24:06.845556  5621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26edf338d41e42f1a5ce0d4416ec67e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.840456  5751 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.850756  5751 ts_tablet_manager.cc:1434] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.041s	user 0.007s	sys 0.000s
I20260709 12:24:06.851799  5751 tablet_bootstrap.cc:492] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:06.838500  5763 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.855402  5618 tablet_service.cc:1511] Processing CreateTablet for tablet 14483990837441a99fa83b4a981b4162 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:24:06.857733  5618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14483990837441a99fa83b4a981b4162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.856477  5549 tablet_service.cc:1511] Processing CreateTablet for tablet 26edf338d41e42f1a5ce0d4416ec67e1 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:24:06.860760  5549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26edf338d41e42f1a5ce0d4416ec67e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.870877  5544 tablet_service.cc:1511] Processing CreateTablet for tablet 9db6962eb8f9432289d80decd309ff40 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:24:06.862584  5750 tablet_bootstrap.cc:654] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:06.872236  5544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db6962eb8f9432289d80decd309ff40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.863304  5692 tablet_service.cc:1511] Processing CreateTablet for tablet 26edf338d41e42f1a5ce0d4416ec67e1 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:24:06.873425  5692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26edf338d41e42f1a5ce0d4416ec67e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.878418  5752 tablet_bootstrap.cc:492] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:06.878945  5752 ts_tablet_manager.cc:1403] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.139s	user 0.012s	sys 0.002s
I20260709 12:24:06.882490  5622 tablet_service.cc:1511] Processing CreateTablet for tablet 9db6962eb8f9432289d80decd309ff40 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:24:06.890472  5752 raft_consensus.cc:359] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.891283  5752 raft_consensus.cc:385] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:06.891597  5752 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:06.892441  5622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db6962eb8f9432289d80decd309ff40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.894660  5751 tablet_bootstrap.cc:654] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:06.895661  5697 tablet_service.cc:1511] Processing CreateTablet for tablet 9db6962eb8f9432289d80decd309ff40 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:24:06.897676  5546 tablet_service.cc:1511] Processing CreateTablet for tablet fa0101c69f7746009c5d77c9616f3360 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:24:06.899492  5697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db6962eb8f9432289d80decd309ff40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.899663  5546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa0101c69f7746009c5d77c9616f3360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.908715  5752 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.918125  5752 ts_tablet_manager.cc:1434] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.039s	user 0.006s	sys 0.002s
I20260709 12:24:06.923125  5457 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:06.925904  5750 tablet_bootstrap.cc:492] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:06.926447  5750 ts_tablet_manager.cc:1403] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.099s	user 0.004s	sys 0.012s
I20260709 12:24:06.927630  5618 tablet_service.cc:1511] Processing CreateTablet for tablet fa0101c69f7746009c5d77c9616f3360 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:24:06.928970  5618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa0101c69f7746009c5d77c9616f3360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.930585  5694 tablet_service.cc:1511] Processing CreateTablet for tablet fa0101c69f7746009c5d77c9616f3360 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:24:06.931810  5694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa0101c69f7746009c5d77c9616f3360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.931666  5623 tablet_service.cc:1511] Processing CreateTablet for tablet 5d78f6c4aaa14250ba8528ff2fe6af34 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:24:06.932861  5750 raft_consensus.cc:359] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:06.934298  5623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d78f6c4aaa14250ba8528ff2fe6af34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.937350  5696 tablet_service.cc:1511] Processing CreateTablet for tablet 5d78f6c4aaa14250ba8528ff2fe6af34 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:24:06.938551  5696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d78f6c4aaa14250ba8528ff2fe6af34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.939499  5547 tablet_service.cc:1511] Processing CreateTablet for tablet 458143f1c77141febe7d9725625617c2 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:24:06.940737  5547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 458143f1c77141febe7d9725625617c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.944470  5750 raft_consensus.cc:385] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:06.945972  5621 tablet_service.cc:1511] Processing CreateTablet for tablet 458143f1c77141febe7d9725625617c2 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:24:06.949652  5698 tablet_service.cc:1511] Processing CreateTablet for tablet 458143f1c77141febe7d9725625617c2 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:24:06.950817  5621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 458143f1c77141febe7d9725625617c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.946393  5752 tablet_bootstrap.cc:492] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:06.954361  5545 tablet_service.cc:1511] Processing CreateTablet for tablet 9f7bab74248a4aeda157f7a50fa82233 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:24:06.955260  5750 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:06.955886  5545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f7bab74248a4aeda157f7a50fa82233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.956321  5750 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:06.950870  5698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 458143f1c77141febe7d9725625617c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.965224  5544 tablet_service.cc:1511] Processing CreateTablet for tablet 5d78f6c4aaa14250ba8528ff2fe6af34 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:24:06.967099  5544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d78f6c4aaa14250ba8528ff2fe6af34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.962672  5751 tablet_bootstrap.cc:492] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:06.968662  5751 ts_tablet_manager.cc:1403] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.117s	user 0.012s	sys 0.000s
I20260709 12:24:06.966468  5622 tablet_service.cc:1511] Processing CreateTablet for tablet 9f7bab74248a4aeda157f7a50fa82233 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:24:06.977346  5622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f7bab74248a4aeda157f7a50fa82233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:06.994024  5751 raft_consensus.cc:359] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:06.994818  5751 raft_consensus.cc:385] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:06.995113  5751 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:06.995572  5750 ts_tablet_manager.cc:1434] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.069s	user 0.010s	sys 0.003s
I20260709 12:24:06.997676  5752 tablet_bootstrap.cc:654] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.000437  5697 tablet_service.cc:1511] Processing CreateTablet for tablet 9f7bab74248a4aeda157f7a50fa82233 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:24:07.001874  5697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f7bab74248a4aeda157f7a50fa82233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.002017  5750 tablet_bootstrap.cc:492] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:07.004670  5549 tablet_service.cc:1511] Processing CreateTablet for tablet 2578c6c739884fad8fa75e16cfa14fec (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:24:07.013340  5619 tablet_service.cc:1511] Processing CreateTablet for tablet 2578c6c739884fad8fa75e16cfa14fec (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:24:07.014500  5619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2578c6c739884fad8fa75e16cfa14fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.016932  5751 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.019352  5751 ts_tablet_manager.cc:1434] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.050s	user 0.008s	sys 0.000s
I20260709 12:24:07.024802  5751 tablet_bootstrap.cc:492] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:07.027413  5763 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.027945  5763 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.030050  5763 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:07.031872  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:07.033494  5549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2578c6c739884fad8fa75e16cfa14fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.033638  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:07.035315  5676 leader_election.cc:343] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:24:07.036803  5675 leader_election.cc:343] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.037235  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:07.038293  5763 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:07.038967  5751 tablet_bootstrap.cc:654] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.043377  5692 tablet_service.cc:1511] Processing CreateTablet for tablet 2578c6c739884fad8fa75e16cfa14fec (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:24:07.045042  5546 tablet_service.cc:1511] Processing CreateTablet for tablet fdcede9bc9104440a7ce1e3a4eb3a86f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:24:07.046713  5546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdcede9bc9104440a7ce1e3a4eb3a86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.046460  5622 tablet_service.cc:1511] Processing CreateTablet for tablet fdcede9bc9104440a7ce1e3a4eb3a86f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:24:07.053673  5622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdcede9bc9104440a7ce1e3a4eb3a86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.058617  5692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2578c6c739884fad8fa75e16cfa14fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.061535  5757 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.062045  5757 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.062227  5551 tablet_service.cc:1511] Processing CreateTablet for tablet 8c6474db7a70405694509f6976d850ce (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:24:07.063498  5551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c6474db7a70405694509f6976d850ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.063167  5691 tablet_service.cc:1511] Processing CreateTablet for tablet fdcede9bc9104440a7ce1e3a4eb3a86f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:24:07.067500  5691 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdcede9bc9104440a7ce1e3a4eb3a86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.064153  5757 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.067859  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:07.071133  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:07.072546  5526 leader_election.cc:343] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:24:07.073913  5524 leader_election.cc:343] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.074329  5524 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:07.075029  5698 tablet_service.cc:1511] Processing CreateTablet for tablet 8c6474db7a70405694509f6976d850ce (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:24:07.075418  5757 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:07.076277  5698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c6474db7a70405694509f6976d850ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.082276  5755 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.082088  5623 tablet_service.cc:1511] Processing CreateTablet for tablet edd23660363f48bf95cee8a70c5a2f26 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:24:07.082808  5755 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:07.083964  5623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edd23660363f48bf95cee8a70c5a2f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.084949  5755 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:07.086951  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:07.087641  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:07.088382  5560 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
W20260709 12:24:07.089102  5599 leader_election.cc:343] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.089798  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:07.090974  5750 tablet_bootstrap.cc:654] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.090945  5755 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:07.092535  5755 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.095479  5755 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.097437  5693 tablet_service.cc:1511] Processing CreateTablet for tablet edd23660363f48bf95cee8a70c5a2f26 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:24:07.099198  5693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edd23660363f48bf95cee8a70c5a2f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.091087  5547 tablet_service.cc:1511] Processing CreateTablet for tablet edd23660363f48bf95cee8a70c5a2f26 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:24:07.101969  5547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edd23660363f48bf95cee8a70c5a2f26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.112191  5550 tablet_service.cc:1511] Processing CreateTablet for tablet f4f86de9f88949219896e4c020485a11 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:24:07.113597  5550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f86de9f88949219896e4c020485a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.134174  5618 tablet_service.cc:1511] Processing CreateTablet for tablet 8c6474db7a70405694509f6976d850ce (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:24:07.135932  5618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c6474db7a70405694509f6976d850ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.173116  5622 tablet_service.cc:1511] Processing CreateTablet for tablet f4f86de9f88949219896e4c020485a11 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:24:07.174362  5622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f86de9f88949219896e4c020485a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.179265  5752 tablet_bootstrap.cc:492] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:07.179771  5752 ts_tablet_manager.cc:1403] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.234s	user 0.012s	sys 0.026s
I20260709 12:24:07.181942  5751 tablet_bootstrap.cc:492] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:07.182431  5751 ts_tablet_manager.cc:1403] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.158s	user 0.011s	sys 0.005s
I20260709 12:24:07.182649  5752 raft_consensus.cc:359] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.183389  5752 raft_consensus.cc:385] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.183696  5752 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.184494  5752 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.230321  5696 tablet_service.cc:1511] Processing CreateTablet for tablet f4f86de9f88949219896e4c020485a11 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:24:07.231763  5696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f86de9f88949219896e4c020485a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.223198  5763 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.234732  5763 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.235312  5622 tablet_service.cc:1511] Processing CreateTablet for tablet 733a5b54e4dd4f86999821705033529f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:24:07.236788  5622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 733a5b54e4dd4f86999821705033529f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.241232  5549 tablet_service.cc:1511] Processing CreateTablet for tablet 733a5b54e4dd4f86999821705033529f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:24:07.242462  5549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 733a5b54e4dd4f86999821705033529f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.213188  5755 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:07.255668  5752 ts_tablet_manager.cc:1434] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.076s	user 0.028s	sys 0.015s
I20260709 12:24:07.257025  5752 tablet_bootstrap.cc:492] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:07.258718  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:07.258728  5545 tablet_service.cc:1511] Processing CreateTablet for tablet d816d235ce5a41c796c8949c48c0a53c (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:24:07.260748  5619 tablet_service.cc:1511] Processing CreateTablet for tablet d816d235ce5a41c796c8949c48c0a53c (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:24:07.261667  5545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d816d235ce5a41c796c8949c48c0a53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.261968  5619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d816d235ce5a41c796c8949c48c0a53c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:24:07.263948  5676 leader_election.cc:343] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.264402  5763 leader_election.cc:290] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:07.265650  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:07.267267  5675 leader_election.cc:343] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.267728  5675 leader_election.cc:304] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:07.268973  5763 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:07.270627  5752 tablet_bootstrap.cc:654] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.279181  5751 raft_consensus.cc:359] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.280294  5751 raft_consensus.cc:385] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.280714  5751 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.281977  5751 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.292022  5694 tablet_service.cc:1511] Processing CreateTablet for tablet d816d235ce5a41c796c8949c48c0a53c (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:24:07.293736  5694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d816d235ce5a41c796c8949c48c0a53c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.297662  5755 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:07.298766  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:07.299360  5560 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.299896  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:24:07.301848  5599 leader_election.cc:343] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.305363  5755 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.305851  5755 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.312906  5697 tablet_service.cc:1511] Processing CreateTablet for tablet 733a5b54e4dd4f86999821705033529f (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:24:07.313657  5622 tablet_service.cc:1511] Processing CreateTablet for tablet 021ceb67cc2f4876b5dbdf1ca21c5f07 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:24:07.314817  5622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 021ceb67cc2f4876b5dbdf1ca21c5f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.314515  5697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 733a5b54e4dd4f86999821705033529f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.321770  5751 ts_tablet_manager.cc:1434] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.139s	user 0.004s	sys 0.007s
I20260709 12:24:07.324390  5755 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.325006  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:07.327340  5602 leader_election.cc:343] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.327360  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:07.328159  5710 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:07.329337  5751 tablet_bootstrap.cc:492] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:07.329967  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:07.331103  5755 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:07.331470  5755 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.331823  5755 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.334334  5560 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:07.337481  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:07.338490  5788 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:07.339454  5763 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:07.340507  5751 tablet_bootstrap.cc:654] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.347710  5549 tablet_service.cc:1511] Processing CreateTablet for tablet 021ceb67cc2f4876b5dbdf1ca21c5f07 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:24:07.349607  5549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 021ceb67cc2f4876b5dbdf1ca21c5f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.351150  5788 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:07.353155  5788 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:07.353570  5693 tablet_service.cc:1511] Processing CreateTablet for tablet 021ceb67cc2f4876b5dbdf1ca21c5f07 (DEFAULT_TABLE table=test-workload [id=53989103653948c0b4757ec009e63460]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:24:07.355273  5693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 021ceb67cc2f4876b5dbdf1ca21c5f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:07.363903  5750 tablet_bootstrap.cc:492] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:07.364981  5790 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.365688  5790 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.371016  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:07.373683  5676 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:07.371752  5790 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.375875  5752 tablet_bootstrap.cc:492] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:07.376468  5752 ts_tablet_manager.cc:1403] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.120s	user 0.029s	sys 0.017s
I20260709 12:24:07.379391  5752 raft_consensus.cc:359] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.380105  5752 raft_consensus.cc:385] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.380507  5750 ts_tablet_manager.cc:1403] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.379s	user 0.026s	sys 0.056s
I20260709 12:24:07.381551  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:07.381378  5599 leader_election.cc:343] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:24:07.389967  5602 leader_election.cc:343] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.390578  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:07.391812  5791 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:07.388453  5752 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.392917  5751 tablet_bootstrap.cc:492] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:07.393772  5751 ts_tablet_manager.cc:1403] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.065s	user 0.016s	sys 0.021s
I20260709 12:24:07.393517  5752 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.395460  5755 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.398553  5752 ts_tablet_manager.cc:1434] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.001s
I20260709 12:24:07.399272  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:07.399564  5752 tablet_bootstrap.cc:492] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:07.400084  5751 raft_consensus.cc:359] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:24:07.400617  5602 leader_election.cc:343] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.400983  5751 raft_consensus.cc:385] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.401429  5751 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.402819  5755 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.402477  5751 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:07.403812  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:07.404398  5710 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.417354  5750 raft_consensus.cc:359] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.418453  5750 raft_consensus.cc:385] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.418799  5750 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.412170  5710 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:07.424253  5457 catalog_manager.cc:5697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:07.425985  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:07.427124  5788 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:07.427587  5788 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:07.427743  5750 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.428534  5788 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.435467  5750 ts_tablet_manager.cc:1434] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.050s	user 0.007s	sys 0.000s
I20260709 12:24:07.436820  5750 tablet_bootstrap.cc:492] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:07.449662  5752 tablet_bootstrap.cc:654] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.472234  5750 tablet_bootstrap.cc:654] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.479214  5459 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:07.493984  5750 tablet_bootstrap.cc:492] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:07.494591  5750 ts_tablet_manager.cc:1403] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.058s	user 0.014s	sys 0.004s
I20260709 12:24:07.503345  5751 ts_tablet_manager.cc:1434] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.109s	user 0.049s	sys 0.045s
I20260709 12:24:07.504396  5751 tablet_bootstrap.cc:492] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:07.510931  5751 tablet_bootstrap.cc:654] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.510895  5752 tablet_bootstrap.cc:492] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:07.510861  5750 raft_consensus.cc:359] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.511673  5752 ts_tablet_manager.cc:1403] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.112s	user 0.021s	sys 0.005s
I20260709 12:24:07.511822  5750 raft_consensus.cc:385] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.512187  5750 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.512966  5750 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.514376  5752 raft_consensus.cc:359] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.515030  5752 raft_consensus.cc:385] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.515311  5752 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.515864  5750 ts_tablet_manager.cc:1434] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.021s	user 0.009s	sys 0.000s
I20260709 12:24:07.516115  5752 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.517182  5750 tablet_bootstrap.cc:492] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:07.518299  5752 ts_tablet_manager.cc:1434] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.003s
I20260709 12:24:07.528164  5750 tablet_bootstrap.cc:654] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.528929  5751 tablet_bootstrap.cc:492] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:07.529445  5751 ts_tablet_manager.cc:1403] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.025s	user 0.015s	sys 0.009s
I20260709 12:24:07.532832  5752 tablet_bootstrap.cc:492] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:07.532486  5751 raft_consensus.cc:359] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.533381  5751 raft_consensus.cc:385] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.533723  5751 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.533838  5796 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.534121  5750 tablet_bootstrap.cc:492] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:07.534327  5796 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.534574  5750 ts_tablet_manager.cc:1403] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.018s	user 0.013s	sys 0.000s
I20260709 12:24:07.534593  5751 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.536554  5796 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:07.537273  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:07.537900  5560 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:07.539100  5752 tablet_bootstrap.cc:654] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.539341  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:07.539590  5751 ts_tablet_manager.cc:1434] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.003s
I20260709 12:24:07.540040  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:07.540196  5796 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:07.540612  5751 tablet_bootstrap.cc:492] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:07.540642  5796 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:07.541374  5675 leader_election.cc:343] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.541110  5796 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.547291  5751 tablet_bootstrap.cc:654] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.553750  5752 tablet_bootstrap.cc:492] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:07.554239  5752 ts_tablet_manager.cc:1403] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.022s	user 0.014s	sys 0.006s
I20260709 12:24:07.556149  5750 raft_consensus.cc:359] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.556951  5750 raft_consensus.cc:385] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.557322  5750 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.557029  5752 raft_consensus.cc:359] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.557785  5752 raft_consensus.cc:385] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.553426  5796 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.558076  5752 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.558666  5751 tablet_bootstrap.cc:492] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:07.559088  5751 ts_tablet_manager.cc:1403] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.011s
I20260709 12:24:07.558094  5750 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.559700  5796 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:07.559880  5752 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.560899  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:07.561640  5751 raft_consensus.cc:359] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.562077  5752 ts_tablet_manager.cc:1434] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.001s
I20260709 12:24:07.562289  5751 raft_consensus.cc:385] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.562382  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:07.562682  5751 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.562964  5560 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.563043  5752 tablet_bootstrap.cc:492] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:07.563433  5751 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.565409  5751 ts_tablet_manager.cc:1434] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.001s
I20260709 12:24:07.566224  5751 tablet_bootstrap.cc:492] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:07.566594  5750 ts_tablet_manager.cc:1434] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.031s	user 0.005s	sys 0.001s
I20260709 12:24:07.567584  5750 tablet_bootstrap.cc:492] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:07.571024  5752 tablet_bootstrap.cc:654] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.572031  5751 tablet_bootstrap.cc:654] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
W20260709 12:24:07.573004  5675 leader_election.cc:343] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.573896  5750 tablet_bootstrap.cc:654] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.583637  5752 tablet_bootstrap.cc:492] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:07.584273  5752 ts_tablet_manager.cc:1403] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.009s
I20260709 12:24:07.587675  5752 raft_consensus.cc:359] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.588559  5752 raft_consensus.cc:385] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.588923  5752 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.591092  5752 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.596972  5752 ts_tablet_manager.cc:1434] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 12:24:07.598062  5752 tablet_bootstrap.cc:492] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:07.602442  5560 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:07.603775  5752 tablet_bootstrap.cc:654] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.604252  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:07.605718  5796 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:07.606345  5796 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:07.607426  5796 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.614037  5750 tablet_bootstrap.cc:492] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:07.614523  5750 ts_tablet_manager.cc:1403] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.047s	user 0.020s	sys 0.004s
I20260709 12:24:07.615376  5752 tablet_bootstrap.cc:492] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:07.615806  5752 ts_tablet_manager.cc:1403] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.018s	user 0.009s	sys 0.007s
I20260709 12:24:07.617146  5750 raft_consensus.cc:359] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.617830  5750 raft_consensus.cc:385] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.618105  5750 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.618789  5750 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.620779  5750 ts_tablet_manager.cc:1434] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 12:24:07.621912  5750 tablet_bootstrap.cc:492] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:07.625633  5757 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.626142  5757 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.628041  5750 tablet_bootstrap.cc:654] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.628301  5757 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.630738  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:07.631397  5635 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:07.631597  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:07.632730  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:24:07.633411  5524 leader_election.cc:343] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.633744  5757 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:07.634294  5757 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.634713  5757 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.635429  5751 tablet_bootstrap.cc:492] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:07.635818  5751 ts_tablet_manager.cc:1403] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.070s	user 0.025s	sys 0.040s
I20260709 12:24:07.625455  5752 raft_consensus.cc:359] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.637077  5752 raft_consensus.cc:385] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.637406  5752 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.638089  5752 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.638309  5751 raft_consensus.cc:359] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.638916  5751 raft_consensus.cc:385] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.639189  5751 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.639835  5751 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.640419  5752 ts_tablet_manager.cc:1434] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.001s
I20260709 12:24:07.641925  5752 tablet_bootstrap.cc:492] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:07.641731  5456 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:07.643105  5751 ts_tablet_manager.cc:1434] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:24:07.643998  5751 tablet_bootstrap.cc:492] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:07.648087  5752 tablet_bootstrap.cc:654] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.649236  5751 tablet_bootstrap.cc:654] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.653877  5788 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.654383  5788 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.658221  5788 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.659204  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:07.660058  5560 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:07.660223  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:07.661927  5599 leader_election.cc:343] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.662976  5602 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:07.664036  5788 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:07.664395  5788 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.664995  5788 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.679958  5751 tablet_bootstrap.cc:492] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:07.680853  5751 ts_tablet_manager.cc:1403] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.037s	user 0.017s	sys 0.019s
I20260709 12:24:07.684331  5751 raft_consensus.cc:359] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.685236  5751 raft_consensus.cc:385] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.685577  5751 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.686609  5751 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.686806  5757 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.679636  5788 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.690510  5755 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.690989  5755 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.692180  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:07.692819  5560 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.693059  5755 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.693347  5751 ts_tablet_manager.cc:1434] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20260709 12:24:07.694602  5751 tablet_bootstrap.cc:492] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:07.695518  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:07.696154  5710 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:07.697463  5599 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:07.698763  5755 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:07.699168  5755 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.699503  5755 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.702061  5751 tablet_bootstrap.cc:654] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.703706  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:07.704372  5561 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:07.709200  5750 tablet_bootstrap.cc:492] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:07.709856  5750 ts_tablet_manager.cc:1403] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.088s	user 0.008s	sys 0.014s
I20260709 12:24:07.713379  5560 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:07.713183  5750 raft_consensus.cc:359] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.713881  5750 raft_consensus.cc:385] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.714182  5750 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.714615  5602 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:07.714790  5751 tablet_bootstrap.cc:492] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:07.714857  5750 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.715600  5791 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:07.716284  5791 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:07.717478  5791 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.715581  5751 ts_tablet_manager.cc:1403] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.021s	user 0.014s	sys 0.004s
I20260709 12:24:07.722460  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:07.723213  5635 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.724970  5750 ts_tablet_manager.cc:1434] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.015s	user 0.012s	sys 0.003s
I20260709 12:24:07.726473  5750 tablet_bootstrap.cc:492] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:07.726236  5751 raft_consensus.cc:359] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.727224  5751 raft_consensus.cc:385] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.732996  5790 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.734035  5750 tablet_bootstrap.cc:654] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.734174  5790 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.736675  5790 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.737901  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:07.739307  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:07.739858  5602 leader_election.cc:343] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.740151  5710 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:07.742074  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:07.743621  5790 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:07.744113  5790 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.744583  5790 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.745707  5750 tablet_bootstrap.cc:492] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:07.746246  5796 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.746750  5796 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:07.748899  5796 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:07.749595  5816 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.750090  5816 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.752518  5751 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.754859  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:07.755374  5816 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.755601  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:07.756403  5710 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 1: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:24:07.756214  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:07.757360  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:07.757778  5561 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 1: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:24:07.758656  5524 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:07.759562  5788 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.760160  5816 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.760599  5757 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.756389  5635 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:07.762380  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:07.763298  5757 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:07.763198  5755 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.764214  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:07.765056  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:07.765765  5755 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.765625  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:07.766228  5796 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.767107  5752 tablet_bootstrap.cc:492] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:07.767699  5752 ts_tablet_manager.cc:1403] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.126s	user 0.054s	sys 0.051s
I20260709 12:24:07.768893  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:07.769420  5458 catalog_manager.cc:5697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:07.770905  5709 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:24:07.773726  5524 leader_election.cc:343] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.774159  5796 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:07.774772  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:07.775309  5560 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:24:07.767606  5599 leader_election.cc:343] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.777101  5750 ts_tablet_manager.cc:1403] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.051s	user 0.015s	sys 0.003s
I20260709 12:24:07.776755  5752 raft_consensus.cc:359] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.777478  5752 raft_consensus.cc:385] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.777815  5752 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.781548  5757 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:07.782431  5560 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:07.782583  5757 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.783641  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:07.784373  5755 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:07.784911  5755 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:07.785782  5755 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.786988  5752 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.787880  5709 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:07.788851  5788 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.789363  5752 ts_tablet_manager.cc:1434] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.000s
I20260709 12:24:07.789238  5750 raft_consensus.cc:359] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.789984  5750 raft_consensus.cc:385] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.789383  5788 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.790303  5750 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.790560  5752 tablet_bootstrap.cc:492] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:07.791061  5750 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.793583  5788 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.793955  5816 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:07.795253  5751 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.796694  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:07.797307  5560 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:07.797900  5751 ts_tablet_manager.cc:1434] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.075s	user 0.009s	sys 0.000s
I20260709 12:24:07.798866  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:07.799160  5790 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.799738  5788 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:07.800112  5788 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.800501  5788 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.801411  5790 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.801802  5752 tablet_bootstrap.cc:654] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.802011  5816 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.801918  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:07.802692  5561 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.805342  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:07.805945  5710 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.814409  5751 tablet_bootstrap.cc:492] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:07.814014  5710 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:07.817895  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:24:07.819155  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:24:07.820653  5791 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:07.821264  5791 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:07.822322  5791 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.823156  5788 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.830706  5751 tablet_bootstrap.cc:654] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.830570  5755 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:07.835614  5752 tablet_bootstrap.cc:492] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:07.834550  5458 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:07.836282  5752 ts_tablet_manager.cc:1403] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.046s	user 0.009s	sys 0.008s
I20260709 12:24:07.837960  5816 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.839480  5560 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.841001  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:07.842805  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:07.813848  5750 ts_tablet_manager.cc:1434] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.000s
I20260709 12:24:07.844401  5751 tablet_bootstrap.cc:492] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:07.844827  5750 tablet_bootstrap.cc:492] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:07.844899  5751 ts_tablet_manager.cc:1403] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.031s	user 0.013s	sys 0.004s
I20260709 12:24:07.845258  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:07.848107  5751 raft_consensus.cc:359] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.839397  5752 raft_consensus.cc:359] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.850179  5561 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:07.866068  5752 raft_consensus.cc:385] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.866425  5752 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.840153  5816 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.857401  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:07.867151  5752 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.864684  5791 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:07.864743  5458 catalog_manager.cc:5697] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:07.868484  5457 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:07.867538  5709 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:07.869724  5752 ts_tablet_manager.cc:1434] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.000s
I20260709 12:24:07.863188  5750 tablet_bootstrap.cc:654] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.870783  5752 tablet_bootstrap.cc:492] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:07.872432  5751 raft_consensus.cc:385] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.872936  5751 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.873706  5751 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.875432  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:07.876099  5710 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.876286  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:07.877599  5561 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.881911  5788 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:07.876657  5751 ts_tablet_manager.cc:1434] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.000s
I20260709 12:24:07.883816  5751 tablet_bootstrap.cc:492] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:07.892850  5710 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:07.894102  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:07.895692  5788 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:07.895666  5752 tablet_bootstrap.cc:654] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.897375  5751 tablet_bootstrap.cc:654] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.899117  5788 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:07.900034  5788 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.915490  5752 tablet_bootstrap.cc:492] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:07.916167  5752 ts_tablet_manager.cc:1403] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.046s	user 0.009s	sys 0.007s
I20260709 12:24:07.922748  5751 tablet_bootstrap.cc:492] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:07.923084  5750 tablet_bootstrap.cc:492] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:07.923377  5751 ts_tablet_manager.cc:1403] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.040s	user 0.019s	sys 0.002s
I20260709 12:24:07.923512  5750 ts_tablet_manager.cc:1403] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.079s	user 0.032s	sys 0.012s
I20260709 12:24:07.924847  5796 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.925971  5752 raft_consensus.cc:359] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.926306  5751 raft_consensus.cc:359] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.925559  5796 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.926726  5752 raft_consensus.cc:385] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.926970  5751 raft_consensus.cc:385] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.927182  5752 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.927320  5751 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.928028  5752 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.929049  5750 raft_consensus.cc:359] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.929725  5750 raft_consensus.cc:385] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.928028  5751 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.930008  5750 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.930356  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:07.930694  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:07.931703  5799 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.930860  5750 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.929141  5796 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:07.932588  5558 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:07.932545  5799 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.932631  5752 ts_tablet_manager.cc:1434] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.004s
I20260709 12:24:07.933588  5751 ts_tablet_manager.cc:1434] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.002s
I20260709 12:24:07.934103  5676 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:07.934419  5751 tablet_bootstrap.cc:492] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:07.935928  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:07.936141  5796 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:07.936610  5796 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.936776  5750 ts_tablet_manager.cc:1434] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 12:24:07.937091  5796 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:24:07.937368  5675 leader_election.cc:343] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.937687  5750 tablet_bootstrap.cc:492] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:07.935928  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:07.939079  5799 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:24:07.939785  5676 leader_election.cc:343] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:07.931145  5635 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:07.941013  5751 tablet_bootstrap.cc:654] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.940233  5676 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:07.943291  5799 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:07.943885  5750 tablet_bootstrap.cc:654] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.934419  5752 tablet_bootstrap.cc:492] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:07.950289  5819 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:07.952469  5752 tablet_bootstrap.cc:654] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.961956  5750 tablet_bootstrap.cc:492] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:07.962582  5750 ts_tablet_manager.cc:1403] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.025s	user 0.012s	sys 0.000s
I20260709 12:24:07.964021  5796 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.966986  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:07.967638  5559 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.966683  5457 catalog_manager.cc:5697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:07.969303  5752 tablet_bootstrap.cc:492] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:07.969880  5752 ts_tablet_manager.cc:1403] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.036s	user 0.012s	sys 0.004s
I20260709 12:24:07.973377  5752 raft_consensus.cc:359] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.974026  5752 raft_consensus.cc:385] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.974292  5752 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.974756  5559 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:07.975056  5752 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.976130  5676 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:07.977101  5799 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:07.977322  5752 ts_tablet_manager.cc:1434] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:24:07.978550  5752 tablet_bootstrap.cc:492] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:07.978981  5799 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:07.979784  5799 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.969340  5561 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:07.966562  5750 raft_consensus.cc:359] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.982980  5750 raft_consensus.cc:385] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.983374  5750 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.987368  5796 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:07.988242  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:07.988844  5635 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:07.984192  5750 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.990406  5751 tablet_bootstrap.cc:492] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:07.990834  5751 ts_tablet_manager.cc:1403] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.057s	user 0.017s	sys 0.017s
I20260709 12:24:07.993166  5752 tablet_bootstrap.cc:654] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:07.992884  5458 catalog_manager.cc:5697] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:07.993886  5751 raft_consensus.cc:359] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.994511  5751 raft_consensus.cc:385] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:07.994786  5751 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:07.995460  5751 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:07.996383  5635 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:07.997262  5750 ts_tablet_manager.cc:1434] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.034s	user 0.009s	sys 0.001s
I20260709 12:24:07.997684  5751 ts_tablet_manager.cc:1434] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:24:07.998116  5750 tablet_bootstrap.cc:492] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:07.997970  5788 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:07.998566  5751 tablet_bootstrap.cc:492] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:08.004993  5751 tablet_bootstrap.cc:654] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.005782  5750 tablet_bootstrap.cc:654] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.010471  5819 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.011039  5819 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.013216  5819 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.013134  5796 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:08.015161  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:08.015868  5709 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:08.017278  5752 tablet_bootstrap.cc:492] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:08.017727  5752 ts_tablet_manager.cc:1403] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.039s	user 0.012s	sys 0.001s
I20260709 12:24:08.018982  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:08.022737  5816 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:08.023147  5816 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.023501  5816 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.031735  5788 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:08.031972  5752 raft_consensus.cc:359] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.032681  5752 raft_consensus.cc:385] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.032967  5752 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.035883  5752 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.036815  5816 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.038208  5752 ts_tablet_manager.cc:1434] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260709 12:24:08.039105  5752 tablet_bootstrap.cc:492] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:08.039510  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:08.040062  5636 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.046182  5796 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.046813  5796 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.056831  5752 tablet_bootstrap.cc:654] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.058393  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:08.065470  5819 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.066114  5819 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.067201  5751 tablet_bootstrap.cc:492] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:08.067807  5751 ts_tablet_manager.cc:1403] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.069s	user 0.014s	sys 0.009s
I20260709 12:24:08.070004  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:08.071580  5751 raft_consensus.cc:359] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:08.072198  5526 leader_election.cc:343] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.072537  5751 raft_consensus.cc:385] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.072913  5751 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.073954  5751 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.078773  5819 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.080386  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:08.081071  5709 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:08.082646  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:08.090526  5751 ts_tablet_manager.cc:1434] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.022s	user 0.013s	sys 0.004s
I20260709 12:24:08.091741  5751 tablet_bootstrap.cc:492] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:08.097023  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:08.098150  5634 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 1: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:24:08.099198  5819 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:08.099711  5819 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.100782  5751 tablet_bootstrap.cc:654] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.102548  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:08.103745  5847 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.104071  5634 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.111023  5847 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:08.112059  5796 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:08.100137  5819 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.123095  5636 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:08.124248  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:08.124491  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:08.124905  5635 raft_consensus.cc:2376] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1. Re-sending same reply.
I20260709 12:24:08.125463  5757 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:08.129992  5788 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.130468  5788 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.135735  5751 tablet_bootstrap.cc:492] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:08.136181  5751 ts_tablet_manager.cc:1403] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.045s	user 0.011s	sys 0.008s
I20260709 12:24:08.136361  5788 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.138768  5751 raft_consensus.cc:359] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.131999  5819 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:08.139683  5599 consensus_peers.cc:597] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:08.140468  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:08.141786  5599 leader_election.cc:343] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.142437  5819 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.142765  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:08.139421  5751 raft_consensus.cc:385] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.143270  5751 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.143333  5708 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.142455  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:08.148259  5708 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:08.149432  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:08.149549  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:08.150250  5526 leader_election.cc:343] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:24:08.150766  5602 leader_election.cc:343] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.151144  5837 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:08.150272  5819 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:08.152091  5819 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:08.151264  5602 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:08.152952  5819 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.154141  5788 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:08.144032  5751 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.159693  5816 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.163179  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:08.164290  5710 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.165714  5751 ts_tablet_manager.cc:1434] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.000s
I20260709 12:24:08.169188  5751 tablet_bootstrap.cc:492] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:08.187479  5757 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:08.189106  5757 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.191067  5750 tablet_bootstrap.cc:492] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:08.191531  5750 ts_tablet_manager.cc:1403] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.194s	user 0.063s	sys 0.080s
I20260709 12:24:08.194186  5750 raft_consensus.cc:359] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.194878  5750 raft_consensus.cc:385] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.195210  5750 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.195752  5849 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:08.199749  5710 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:08.195919  5750 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.199766  5751 tablet_bootstrap.cc:654] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.202355  5750 ts_tablet_manager.cc:1434] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 12:24:08.203884  5750 tablet_bootstrap.cc:492] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:08.202505  5819 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:24:08.216017  5790 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:08.230281  5837 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.231520  5837 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.251952  5846 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.252830  5846 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.257731  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:08.259828  5676 leader_election.cc:343] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.262439  5788 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.263154  5788 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.270583  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:08.290230  5846 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:08.295540  5837 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:08.296011  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:08.297320  5558 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 1: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:24:08.314038  5755 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:08.326288  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:08.327028  5707 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:08.328433  5751 tablet_bootstrap.cc:492] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:08.328372  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:08.329329  5755 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:08.329734  5755 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.330073  5755 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.330602  5788 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.327946  5458 catalog_manager.cc:5697] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:08.328981  5751 ts_tablet_manager.cc:1403] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.160s	user 0.013s	sys 0.031s
I20260709 12:24:08.288718  5561 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:08.344303  5752 tablet_bootstrap.cc:492] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:08.345062  5752 ts_tablet_manager.cc:1403] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.306s	user 0.045s	sys 0.039s
I20260709 12:24:08.346334  5750 tablet_bootstrap.cc:654] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.350123  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:08.355265  5675 leader_election.cc:343] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.355751  5675 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:08.356689  5837 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:08.364593  5790 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:08.369637  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:08.377256  5752 raft_consensus.cc:359] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.382128  5752 raft_consensus.cc:385] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.382450  5752 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.385535  5752 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.394486  5755 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.402153  5755 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.421155  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:08.422468  5837 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.426443  5751 raft_consensus.cc:359] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.426857  5752 ts_tablet_manager.cc:1434] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.081s	user 0.011s	sys 0.010s
I20260709 12:24:08.427841  5751 raft_consensus.cc:385] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.428403  5751 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.430104  5751 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.433147  5751 ts_tablet_manager.cc:1434] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.101s	user 0.008s	sys 0.000s
I20260709 12:24:08.433826  5752 tablet_bootstrap.cc:492] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:08.434149  5751 tablet_bootstrap.cc:492] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:08.441946  5750 tablet_bootstrap.cc:492] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:08.442453  5750 ts_tablet_manager.cc:1403] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.239s	user 0.008s	sys 0.012s
I20260709 12:24:08.445319  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:08.445961  5709 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.445598  5750 raft_consensus.cc:359] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.446485  5750 raft_consensus.cc:385] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.446808  5750 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.448207  5837 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:08.451915  5752 tablet_bootstrap.cc:654] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.448158  5750 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.453150  5751 tablet_bootstrap.cc:654] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.454480  5750 ts_tablet_manager.cc:1434] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.001s
I20260709 12:24:08.454360  5709 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:08.455752  5750 tablet_bootstrap.cc:492] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:08.456007  5860 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:08.459836  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:08.463850  5790 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:08.449031  5456 catalog_manager.cc:5697] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:08.464401  5790 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:08.465356  5790 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.469241  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:08.469776  5560 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.476974  5837 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:08.478371  5755 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.478852  5755 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.479266  5788 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:08.481516  5791 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:08.480867  5560 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:08.486078  5755 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.488698  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:08.490155  5837 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.490641  5837 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:08.492715  5599 leader_election.cc:343] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.493489  5837 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:08.493626  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:08.494916  5750 tablet_bootstrap.cc:654] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
W20260709 12:24:08.495407  5676 leader_election.cc:343] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.502240  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:08.509366  5752 tablet_bootstrap.cc:492] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:08.509869  5752 ts_tablet_manager.cc:1403] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.076s	user 0.011s	sys 0.004s
I20260709 12:24:08.520592  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:08.522740  5675 leader_election.cc:343] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.523276  5675 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:08.523885  5837 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:24:08.525418  5602 leader_election.cc:343] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.525586  5752 raft_consensus.cc:359] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.526345  5752 raft_consensus.cc:385] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.526659  5752 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.527845  5837 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.528319  5837 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.528275  5752 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.528945  5750 tablet_bootstrap.cc:492] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:08.530782  5752 ts_tablet_manager.cc:1434] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.000s
I20260709 12:24:08.530882  5837 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:08.531287  5750 ts_tablet_manager.cc:1403] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.076s	user 0.025s	sys 0.008s
I20260709 12:24:08.531737  5752 tablet_bootstrap.cc:492] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:08.531963  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:08.534173  5750 raft_consensus.cc:359] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.534947  5750 raft_consensus.cc:385] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.535272  5750 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
W20260709 12:24:08.534212  5676 leader_election.cc:343] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.536248  5750 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.537436  5847 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:08.525877  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:08.539009  5750 ts_tablet_manager.cc:1434] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:24:08.539703  5755 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:08.540099  5750 tablet_bootstrap.cc:492] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:08.541498  5752 tablet_bootstrap.cc:654] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.546506  5752 tablet_bootstrap.cc:492] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:08.546969  5752 ts_tablet_manager.cc:1403] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.015s	user 0.009s	sys 0.001s
I20260709 12:24:08.547655  5750 tablet_bootstrap.cc:654] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.549585  5752 raft_consensus.cc:359] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.550252  5752 raft_consensus.cc:385] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.550499  5752 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.551235  5752 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.554248  5752 ts_tablet_manager.cc:1434] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:24:08.559276  5752 tablet_bootstrap.cc:492] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:08.565698  5752 tablet_bootstrap.cc:654] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.566246  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:08.570910  5456 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:08.571787  5751 tablet_bootstrap.cc:492] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:08.572911  5751 ts_tablet_manager.cc:1403] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.139s	user 0.025s	sys 0.007s
W20260709 12:24:08.575623  5675 leader_election.cc:343] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.576200  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:08.576596  5750 tablet_bootstrap.cc:492] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:08.577144  5750 ts_tablet_manager.cc:1403] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.037s	user 0.012s	sys 0.000s
I20260709 12:24:08.577289  5837 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:08.578902  5752 tablet_bootstrap.cc:492] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:08.579355  5752 ts_tablet_manager.cc:1403] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.004s
I20260709 12:24:08.581882  5752 raft_consensus.cc:359] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.582548  5752 raft_consensus.cc:385] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.582828  5752 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.582866  5849 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:08.580646  5750 raft_consensus.cc:359] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.583766  5752 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.585098  5750 raft_consensus.cc:385] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.585456  5750 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.586639  5752 ts_tablet_manager.cc:1434] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:24:08.586280  5750 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.587277  5755 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:08.588939  5750 ts_tablet_manager.cc:1434] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:24:08.589016  5752 tablet_bootstrap.cc:492] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:08.593510  5751 raft_consensus.cc:359] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.594187  5751 raft_consensus.cc:385] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.594919  5750 tablet_bootstrap.cc:492] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:08.594497  5751 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.598088  5751 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.599311  5752 tablet_bootstrap.cc:654] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.600207  5751 ts_tablet_manager.cc:1434] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.027s	user 0.005s	sys 0.000s
I20260709 12:24:08.600169  5635 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.601230  5751 tablet_bootstrap.cc:492] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:08.606662  5750 tablet_bootstrap.cc:654] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.607636  5751 tablet_bootstrap.cc:654] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.610605  5816 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:08.620217  5750 tablet_bootstrap.cc:492] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:08.620564  5752 tablet_bootstrap.cc:492] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:08.620859  5750 ts_tablet_manager.cc:1403] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.026s	user 0.012s	sys 0.000s
I20260709 12:24:08.620972  5752 ts_tablet_manager.cc:1403] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.032s	user 0.007s	sys 0.005s
I20260709 12:24:08.624208  5750 raft_consensus.cc:359] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.624970  5750 raft_consensus.cc:385] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.625258  5750 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.625797  5750 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.627375  5750 ts_tablet_manager.cc:1434] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:24:08.628073  5750 tablet_bootstrap.cc:492] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:08.624286  5752 raft_consensus.cc:359] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.628566  5752 raft_consensus.cc:385] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.628894  5752 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.629657  5752 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.631748  5752 ts_tablet_manager.cc:1434] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:24:08.632743  5752 tablet_bootstrap.cc:492] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:08.633849  5750 tablet_bootstrap.cc:654] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.639256  5752 tablet_bootstrap.cc:654] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.641237  5755 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.641726  5755 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.643801  5755 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.644743  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:08.645383  5561 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:08.646746  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:08.647462  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:08.648013  5755 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:08.648430  5755 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.648806  5755 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:24:08.649227  5599 leader_election.cc:343] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.651690  5816 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.652169  5816 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.654259  5816 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.654922  5755 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.655861  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:08.657356  5526 leader_election.cc:343] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.658954  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:08.659611  5709 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:08.660931  5524 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:08.662330  5816 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:08.662719  5816 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.663055  5816 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.663527  5755 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.666617  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:08.667219  5561 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.669942  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:24:08.672144  5599 leader_election.cc:343] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.697431  5561 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:08.699095  5788 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.699748  5788 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.702454  5788 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.703444  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:08.704376  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:08.704954  5709 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
W20260709 12:24:08.705031  5602 leader_election.cc:343] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.706492  5599 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:08.707511  5788 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:08.707863  5788 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.708202  5788 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.711663  5816 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.713146  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:08.713927  5755 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:08.714380  5755 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:08.715173  5755 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.716398  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:24:08.717744  5526 leader_election.cc:343] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.720078  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:08.720682  5709 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.720613  5788 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.723402  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:24:08.724836  5602 leader_election.cc:343] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.727560  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:08.728154  5708 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.728282  5709 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:08.729494  5524 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:08.730348  5757 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:08.730804  5757 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:08.731612  5757 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.735005  5708 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:08.735862  5816 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.736171  5599 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:08.737111  5791 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:08.737553  5791 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:08.738339  5791 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.741387  5849 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:08.743314  5788 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.752174  5752 tablet_bootstrap.cc:492] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:08.755635  5752 ts_tablet_manager.cc:1403] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.123s	user 0.027s	sys 0.072s
I20260709 12:24:08.755972  5757 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:08.756731  5750 tablet_bootstrap.cc:492] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:08.757174  5750 ts_tablet_manager.cc:1403] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.129s	user 0.052s	sys 0.071s
I20260709 12:24:08.759531  5750 raft_consensus.cc:359] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.760246  5750 raft_consensus.cc:385] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.760548  5750 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.758539  5752 raft_consensus.cc:359] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.761379  5752 raft_consensus.cc:385] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.761713  5752 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.762523  5752 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.764004  5751 tablet_bootstrap.cc:492] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:08.764494  5751 ts_tablet_manager.cc:1403] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.163s	user 0.021s	sys 0.017s
I20260709 12:24:08.764149  5750 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.765350  5752 ts_tablet_manager.cc:1434] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.004s
I20260709 12:24:08.766176  5750 ts_tablet_manager.cc:1434] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:24:08.766316  5752 tablet_bootstrap.cc:492] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:08.767028  5750 tablet_bootstrap.cc:492] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:08.767678  5458 catalog_manager.cc:5697] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:08.769821  5751 raft_consensus.cc:359] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.770458  5751 raft_consensus.cc:385] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.770740  5751 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.774111  5752 tablet_bootstrap.cc:654] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.773658  5751 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.776473  5750 tablet_bootstrap.cc:654] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.789271  5750 tablet_bootstrap.cc:492] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:08.789906  5750 ts_tablet_manager.cc:1403] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.005s
I20260709 12:24:08.793473  5750 raft_consensus.cc:359] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.794288  5750 raft_consensus.cc:385] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.794585  5750 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.795333  5750 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.801558  5456 catalog_manager.cc:5697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:08.802462  5849 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.802942  5849 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.805202  5849 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.805719  5750 ts_tablet_manager.cc:1434] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.000s
I20260709 12:24:08.806520  5751 ts_tablet_manager.cc:1434] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.042s	user 0.004s	sys 0.001s
I20260709 12:24:08.809283  5757 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.812028  5757 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.812605  5751 tablet_bootstrap.cc:492] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:08.809468  5752 tablet_bootstrap.cc:492] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:08.813742  5752 ts_tablet_manager.cc:1403] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.048s	user 0.017s	sys 0.018s
I20260709 12:24:08.814129  5757 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.815404  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:08.815727  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:08.816378  5635 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:08.816519  5752 raft_consensus.cc:359] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.817323  5752 raft_consensus.cc:385] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.817595  5752 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.818212  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:08.818506  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:08.818374  5752 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.819123  5757 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:08.819293  5708 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 1: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:24:08.819514  5757 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.820282  5757 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.820292  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:08.821858  5750 tablet_bootstrap.cc:492] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:08.822165  5524 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:08.823948  5849 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.820871  5752 ts_tablet_manager.cc:1434] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
W20260709 12:24:08.823371  5524 leader_election.cc:343] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.825428  5752 tablet_bootstrap.cc:492] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:08.847211  5757 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.849831  5757 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.850528  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:08.850821  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:08.851169  5634 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:24:08.852046  5524 leader_election.cc:343] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.852681  5751 tablet_bootstrap.cc:654] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.856299  5459 catalog_manager.cc:5697] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:08.866009  5752 tablet_bootstrap.cc:654] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.872717  5634 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:08.877846  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:08.878998  5757 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:08.880242  5757 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:08.881104  5757 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.879035  5849 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:08.891305  5755 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.891815  5755 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.892642  5752 tablet_bootstrap.cc:492] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:08.893146  5752 ts_tablet_manager.cc:1403] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.068s	user 0.009s	sys 0.003s
I20260709 12:24:08.894160  5755 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.895856  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:08.896239  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:24:08.898135  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:08.898453  5602 leader_election.cc:343] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.896039  5752 raft_consensus.cc:359] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.896888  5750 tablet_bootstrap.cc:654] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.899178  5752 raft_consensus.cc:385] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.899471  5752 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.900135  5752 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.898742  5709 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:08.902066  5752 ts_tablet_manager.cc:1434] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:24:08.902516  5599 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:08.902962  5752 tablet_bootstrap.cc:492] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:08.903571  5755 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:08.903926  5755 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.904250  5755 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.909240  5757 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.908784  5457 catalog_manager.cc:5697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:08.910009  5757 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.912221  5757 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.912949  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:08.913597  5635 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:08.914638  5752 tablet_bootstrap.cc:654] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.914842  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:08.915509  5709 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:08.916484  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:08.917683  5791 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:08.917717  5757 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:08.919981  5757 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.920531  5757 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.921908  5752 tablet_bootstrap.cc:492] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:08.926828  5752 ts_tablet_manager.cc:1403] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.003s
I20260709 12:24:08.928731  5750 tablet_bootstrap.cc:492] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:08.929203  5750 ts_tablet_manager.cc:1403] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.108s	user 0.011s	sys 0.011s
I20260709 12:24:08.929498  5752 raft_consensus.cc:359] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.929533  5757 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.930316  5752 raft_consensus.cc:385] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.930619  5752 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.931937  5757 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.931474  5752 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.932631  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:08.932979  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:08.933202  5635 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.933507  5709 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.935117  5751 tablet_bootstrap.cc:492] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:08.935523  5751 ts_tablet_manager.cc:1403] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.123s	user 0.006s	sys 0.022s
I20260709 12:24:08.938865  5752 ts_tablet_manager.cc:1434] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:24:08.939774  5752 tablet_bootstrap.cc:492] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:08.939842  5709 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:08.941151  5524 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:08.941205  5750 raft_consensus.cc:359] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.941908  5750 raft_consensus.cc:385] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.942106  5757 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:08.942211  5750 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.942838  5757 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:08.943210  5750 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.939746  5635 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:08.945497  5751 raft_consensus.cc:359] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.946004  5751 raft_consensus.cc:385] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:08.946231  5751 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:08.946754  5751 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.948516  5752 tablet_bootstrap.cc:654] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:08.948894  5751 ts_tablet_manager.cc:1434] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:24:08.951889  5751 tablet_bootstrap.cc:492] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:08.951956  5791 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.950729  5757 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.952642  5791 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.952687  5755 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:08.954726  5791 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.955083  5755 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:08.960995  5750 ts_tablet_manager.cc:1434] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.031s	user 0.005s	sys 0.002s
I20260709 12:24:08.962419  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:08.963982  5750 tablet_bootstrap.cc:492] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:08.963596  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:24:08.969430  5599 leader_election.cc:343] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.970593  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:08.976758  5710 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:08.980696  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:08.984783  5788 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.985299  5788 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:08.987494  5602 leader_election.cc:343] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:08.987982  5602 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:08.988762  5791 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:08.988238  5459 catalog_manager.cc:5697] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:08.993908  5710 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:08.995373  5599 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:08.996950  5790 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:08.997524  5790 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:08.998574  5790 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.004055  5750 tablet_bootstrap.cc:654] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
W20260709 12:24:09.008057  5602 leader_election.cc:343] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.009363  5788 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.010114  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:09.010907  5710 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 1: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:24:09.012473  5788 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:09.012035  5459 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:09.015848  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:09.017293  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:09.017978  5751 tablet_bootstrap.cc:654] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.020859  5788 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.025991  5791 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.028713  5752 tablet_bootstrap.cc:492] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:09.029181  5752 ts_tablet_manager.cc:1403] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.090s	user 0.009s	sys 0.004s
I20260709 12:24:09.026820  5791 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.031888  5752 raft_consensus.cc:359] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.032598  5752 raft_consensus.cc:385] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.032907  5752 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.033063  5788 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:09.033649  5752 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.035567  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:09.035926  5752 ts_tablet_manager.cc:1434] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:24:09.036603  5751 tablet_bootstrap.cc:492] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:09.037038  5751 ts_tablet_manager.cc:1403] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.085s	user 0.012s	sys 0.000s
W20260709 12:24:09.036829  5602 leader_election.cc:343] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.039377  5752 tablet_bootstrap.cc:492] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:09.039629  5751 raft_consensus.cc:359] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.040268  5751 raft_consensus.cc:385] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.040103  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:09.040599  5751 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.041350  5751 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:09.042084  5599 leader_election.cc:343] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.042686  5599 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:09.043666  5751 ts_tablet_manager.cc:1434] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:24:09.044591  5751 tablet_bootstrap.cc:492] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:09.044539  5755 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:09.045955  5755 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.045063  5791 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.046396  5755 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.052794  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:09.054033  5602 leader_election.cc:343] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.055130  5757 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.055724  5757 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.058045  5849 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.058518  5849 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.062821  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:09.064481  5526 leader_election.cc:343] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.075750  5846 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.076478  5846 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.104897  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:09.105412  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:09.105604  5635 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
W20260709 12:24:09.106793  5676 leader_election.cc:343] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.107316  5755 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.108548  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:09.110002  5599 leader_election.cc:343] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.110466  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:09.112663  5751 tablet_bootstrap.cc:654] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.112113  5846 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:09.113149  5837 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.113960  5837 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.116372  5755 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:09.116587  5837 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:09.117367  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:09.118001  5556 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:09.119350  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:09.120203  5847 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:09.120571  5847 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.120899  5847 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.129043  5846 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.129557  5846 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.131820  5846 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:09.133059  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:09.139092  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:09.140014  5634 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:09.141705  5755 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:09.141942  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:09.142889  5634 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:09.149385  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:09.151770  5849 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.151855  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:09.152463  5849 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:09.152899  5849 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.152830  5710 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:09.153437  5849 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.153869  5757 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.157207  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:09.158687  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:09.160887  5816 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:09.161414  5816 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.161842  5816 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.166977  5849 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.172602  5757 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:09.183964  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:09.186168  5849 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.199895  5752 tablet_bootstrap.cc:700] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent opening tablet: real 0.156s	user 0.010s	sys 0.008s
I20260709 12:24:09.200894  5752 tablet_bootstrap.cc:654] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.206081  5634 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:09.147964  5847 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.215423  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:09.220542  5903 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:09.221131  5903 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.221582  5903 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.222302  5708 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 1: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:09.229874  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:09.231709  5676 leader_election.cc:343] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.235478  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:09.236454  5558 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 1: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:24:09.236750  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 103776us
I20260709 12:24:09.238974  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:09.240463  5750 tablet_bootstrap.cc:492] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:09.240919  5750 ts_tablet_manager.cc:1403] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.277s	user 0.022s	sys 0.027s
I20260709 12:24:09.235877  5903 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.243495  5750 raft_consensus.cc:359] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.244522  5846 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:09.244925  5846 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.245270  5846 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.245347  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:09.245910  5847 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:09.248903  5788 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:24:09.250028  5676 leader_election.cc:343] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.244180  5750 raft_consensus.cc:385] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.252612  5750 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.253307  5750 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.238327  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:09.256201  5634 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.256616  5816 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.259793  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:09.260428  5708 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.263230  5634 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:09.257381  5903 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:09.264394  5846 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.266842  5751 tablet_bootstrap.cc:492] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:09.267395  5751 ts_tablet_manager.cc:1403] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.223s	user 0.072s	sys 0.049s
I20260709 12:24:09.267585  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:24:09.269004  5676 leader_election.cc:343] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.269338  5752 tablet_bootstrap.cc:492] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:09.269771  5752 ts_tablet_manager.cc:1403] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.231s	user 0.016s	sys 0.012s
I20260709 12:24:09.270409  5751 raft_consensus.cc:359] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.271225  5751 raft_consensus.cc:385] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.271189  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:09.271741  5751 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.272225  5635 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 1: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:24:09.272289  5752 raft_consensus.cc:359] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.273097  5752 raft_consensus.cc:385] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.272771  5751 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.273367  5752 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.273551  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:09.274125  5636 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.274040  5752 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.275609  5750 ts_tablet_manager.cc:1434] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.034s	user 0.006s	sys 0.000s
I20260709 12:24:09.275658  5751 ts_tablet_manager.cc:1434] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.003s
I20260709 12:24:09.276058  5752 ts_tablet_manager.cc:1434] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.004s
I20260709 12:24:09.276685  5846 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:09.278661  5752 tablet_bootstrap.cc:492] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:09.278818  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:09.277434  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:09.278661  5751 tablet_bootstrap.cc:492] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:09.280640  5750 tablet_bootstrap.cc:492] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:09.281780  5904 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:09.282249  5904 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:24:09.282866  5526 leader_election.cc:343] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.283061  5904 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.285830  5907 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:24:09.293025  5636 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:09.283289  5816 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.297571  5752 tablet_bootstrap.cc:654] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.299741  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:09.301157  5903 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:24:09.301478  5708 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:09.304013  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:09.304986  5634 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.305751  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:09.306954  5903 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:09.307401  5903 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:09.308151  5903 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.326355  5750 tablet_bootstrap.cc:654] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.328688  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:09.331794  5751 tablet_bootstrap.cc:654] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.334036  5457 catalog_manager.cc:5697] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:09.338012  5459 catalog_manager.cc:5697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:09.345469  5708 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.351141  5788 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:09.354274  5849 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:24:09.346680  5524 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:09.357637  5788 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:09.359661  5634 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:09.369601  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:09.372737  5903 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:09.373406  5903 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:09.374224  5903 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.378162  5752 tablet_bootstrap.cc:492] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:09.380141  5752 ts_tablet_manager.cc:1403] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.102s	user 0.010s	sys 0.014s
I20260709 12:24:09.393987  5910 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:24:09.397605  5912 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:24:09.401994  5910 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:09.402908  5910 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.415927  5751 tablet_bootstrap.cc:492] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:09.415575  5816 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:09.416414  5751 ts_tablet_manager.cc:1403] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.138s	user 0.008s	sys 0.006s
I20260709 12:24:09.419261  5751 raft_consensus.cc:359] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.420060  5751 raft_consensus.cc:385] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.404239  5752 raft_consensus.cc:359] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.435803  5752 raft_consensus.cc:385] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.436110  5752 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.438956  5750 tablet_bootstrap.cc:492] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:09.438736  5752 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.439421  5750 ts_tablet_manager.cc:1403] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.159s	user 0.016s	sys 0.008s
I20260709 12:24:09.442361  5750 raft_consensus.cc:359] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.443005  5750 raft_consensus.cc:385] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.443292  5750 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.445359  5750 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.447707  5750 ts_tablet_manager.cc:1434] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 12:24:09.448657  5750 tablet_bootstrap.cc:492] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:09.452055  5751 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.453176  5752 ts_tablet_manager.cc:1434] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.073s	user 0.006s	sys 0.000s
I20260709 12:24:09.454421  5752 tablet_bootstrap.cc:492] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:09.455968  5750 tablet_bootstrap.cc:654] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.456700  5751 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.459796  5751 ts_tablet_manager.cc:1434] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.043s	user 0.007s	sys 0.000s
I20260709 12:24:09.460734  5751 tablet_bootstrap.cc:492] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:09.470721  5752 tablet_bootstrap.cc:654] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.473821  5459 catalog_manager.cc:5697] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:09.476684  5910 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.477285  5910 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.480554  5910 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.480732  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:09.482112  5526 leader_election.cc:343] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.483232  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:09.487929  5524 leader_election.cc:343] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.488713  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:09.491389  5561 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.494630  5752 tablet_bootstrap.cc:492] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:09.495460  5752 ts_tablet_manager.cc:1403] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.041s	user 0.012s	sys 0.007s
I20260709 12:24:09.497803  5904 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:09.499722  5752 raft_consensus.cc:359] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.500509  5752 raft_consensus.cc:385] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.500844  5752 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.501667  5752 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.502442  5910 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:09.503523  5752 ts_tablet_manager.cc:1434] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:24:09.504541  5752 tablet_bootstrap.cc:492] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:09.504930  5788 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:09.512780  5788 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.513278  5788 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.515269  5788 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.517122  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:09.518699  5602 leader_election.cc:343] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.519433  5752 tablet_bootstrap.cc:654] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.523191  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:09.525045  5599 leader_election.cc:343] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.525663  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:09.527073  5788 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:09.527508  5791 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:09.529251  5456 catalog_manager.cc:5697] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:09.532178  5751 tablet_bootstrap.cc:654] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.542040  5752 tablet_bootstrap.cc:492] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:09.542527  5752 ts_tablet_manager.cc:1403] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.038s	user 0.010s	sys 0.000s
I20260709 12:24:09.551118  5750 tablet_bootstrap.cc:492] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:09.550843  5752 raft_consensus.cc:359] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.551635  5750 ts_tablet_manager.cc:1403] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.103s	user 0.020s	sys 0.038s
I20260709 12:24:09.551791  5752 raft_consensus.cc:385] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.552244  5752 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.554589  5752 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.556656  5750 raft_consensus.cc:359] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.557116  5752 ts_tablet_manager.cc:1434] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.000s
I20260709 12:24:09.557412  5750 raft_consensus.cc:385] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.557739  5750 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.557898  5751 tablet_bootstrap.cc:492] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:09.558152  5752 tablet_bootstrap.cc:492] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:09.558827  5751 ts_tablet_manager.cc:1403] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.098s	user 0.010s	sys 0.000s
I20260709 12:24:09.558522  5750 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.560714  5750 ts_tablet_manager.cc:1434] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.005s
I20260709 12:24:09.561584  5750 tablet_bootstrap.cc:492] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:09.562465  5751 raft_consensus.cc:359] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.563172  5751 raft_consensus.cc:385] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.563424  5751 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.564265  5751 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.566730  5751 ts_tablet_manager.cc:1434] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:24:09.567642  5751 tablet_bootstrap.cc:492] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:09.575554  5896 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.588826  5750 tablet_bootstrap.cc:654] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.588865  5752 tablet_bootstrap.cc:654] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.588856  5751 tablet_bootstrap.cc:654] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.588791  5896 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.591739  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:09.591913  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:09.593317  5599 leader_election.cc:343] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:24:09.594005  5602 leader_election.cc:343] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.595003  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:09.596163  5788 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:09.593473  5896 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.600852  5752 tablet_bootstrap.cc:492] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:09.601341  5751 tablet_bootstrap.cc:492] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:09.601450  5752 ts_tablet_manager.cc:1403] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.043s	user 0.013s	sys 0.003s
I20260709 12:24:09.601819  5751 ts_tablet_manager.cc:1403] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.034s	user 0.020s	sys 0.006s
I20260709 12:24:09.602248  5750 tablet_bootstrap.cc:492] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:09.602715  5750 ts_tablet_manager.cc:1403] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.041s	user 0.026s	sys 0.009s
I20260709 12:24:09.604322  5752 raft_consensus.cc:359] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.605079  5752 raft_consensus.cc:385] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.604802  5751 raft_consensus.cc:359] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.605367  5752 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.605533  5751 raft_consensus.cc:385] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.605273  5750 raft_consensus.cc:359] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.605899  5751 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.606164  5752 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.606774  5751 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.608188  5752 ts_tablet_manager.cc:1434] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 12:24:09.609210  5752 tablet_bootstrap.cc:492] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:09.606261  5750 raft_consensus.cc:385] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.610006  5750 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.610661  5750 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.614205  5816 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.614930  5750 ts_tablet_manager.cc:1434] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.002s
I20260709 12:24:09.614707  5816 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.615474  5752 tablet_bootstrap.cc:654] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.615859  5750 tablet_bootstrap.cc:492] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:09.622090  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:09.622927  5634 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 1: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:09.624001  5750 tablet_bootstrap.cc:654] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.624909  5751 ts_tablet_manager.cc:1434] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 12:24:09.636969  5816 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.639307  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:09.641070  5524 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:09.642051  5752 tablet_bootstrap.cc:492] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:09.642196  5751 tablet_bootstrap.cc:492] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:09.642649  5752 ts_tablet_manager.cc:1403] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.034s	user 0.016s	sys 0.013s
I20260709 12:24:09.645641  5752 raft_consensus.cc:359] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.646407  5752 raft_consensus.cc:385] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.646762  5752 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.642234  5816 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.647706  5752 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.650159  5752 ts_tablet_manager.cc:1434] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:24:09.651265  5752 tablet_bootstrap.cc:492] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:09.652487  5751 tablet_bootstrap.cc:654] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.654338  5750 tablet_bootstrap.cc:492] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:09.654842  5750 ts_tablet_manager.cc:1403] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.039s	user 0.018s	sys 0.021s
I20260709 12:24:09.657634  5752 tablet_bootstrap.cc:654] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.683353  5816 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:09.658489  5750 raft_consensus.cc:359] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.691020  5750 raft_consensus.cc:385] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.691831  5750 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.693017  5750 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.695433  5750 ts_tablet_manager.cc:1434] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.040s	user 0.004s	sys 0.004s
I20260709 12:24:09.696024  5751 tablet_bootstrap.cc:492] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:09.696599  5751 ts_tablet_manager.cc:1403] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.055s	user 0.029s	sys 0.022s
I20260709 12:24:09.697322  5752 tablet_bootstrap.cc:492] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:09.697770  5752 ts_tablet_manager.cc:1403] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.047s	user 0.017s	sys 0.022s
I20260709 12:24:09.698952  5751 raft_consensus.cc:359] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.699656  5751 raft_consensus.cc:385] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.700017  5751 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.700205  5752 raft_consensus.cc:359] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.700896  5752 raft_consensus.cc:385] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.701181  5752 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.700909  5751 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.701931  5752 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.704195  5752 ts_tablet_manager.cc:1434] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:24:09.705171  5752 tablet_bootstrap.cc:492] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:09.711040  5752 tablet_bootstrap.cc:654] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.711203  5751 ts_tablet_manager.cc:1434] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:24:09.712563  5751 tablet_bootstrap.cc:492] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:09.723639  5751 tablet_bootstrap.cc:654] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.740187  5750 tablet_bootstrap.cc:492] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:09.749455  5750 tablet_bootstrap.cc:654] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.765823  5750 tablet_bootstrap.cc:492] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:09.766295  5750 ts_tablet_manager.cc:1403] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.027s	user 0.016s	sys 0.004s
I20260709 12:24:09.769040  5903 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.769085  5750 raft_consensus.cc:359] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.770179  5903 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.774996  5816 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.777987  5791 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.778679  5791 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.781514  5846 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.782202  5846 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.775724  5816 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.786420  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:09.787287  5634 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:09.787734  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:09.788568  5636 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 1: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:09.788734  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:09.789606  5941 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.790073  5941 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.791617  5846 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:09.792138  5941 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:09.792393  5846 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:09.792932  5846 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.793421  5846 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.796240  5903 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:09.797784  5752 tablet_bootstrap.cc:492] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:09.807063  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:09.770181  5750 raft_consensus.cc:385] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.807322  5752 ts_tablet_manager.cc:1403] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.102s	user 0.027s	sys 0.048s
I20260709 12:24:09.808205  5750 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.798795  5751 tablet_bootstrap.cc:492] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:09.808826  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:09.809098  5751 ts_tablet_manager.cc:1403] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.097s	user 0.036s	sys 0.030s
I20260709 12:24:09.809175  5750 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:09.810460  5676 leader_election.cc:343] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.810544  5752 raft_consensus.cc:359] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.805801  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:09.811236  5752 raft_consensus.cc:385] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.811513  5752 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.811753  5751 raft_consensus.cc:359] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.812093  5752 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.800098  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:09.812896  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:09.814365  5599 leader_election.cc:343] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.803385  5846 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.815724  5710 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.816290  5846 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:09.817235  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:09.817812  5557 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.819491  5636 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.799307  5791 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.821416  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:09.804664  5816 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.824119  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:09.802654  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:09.825034  5675 leader_election.cc:343] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.825490  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:09.825598  5816 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:09.826664  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:09.826834  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:09.827278  5846 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:09.827359  5634 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.828604  5752 ts_tablet_manager.cc:1434] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260709 12:24:09.828325  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:09.829528  5752 tablet_bootstrap.cc:492] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:09.829689  5849 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.831768  5557 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
W20260709 12:24:09.833606  5602 leader_election.cc:343] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.834091  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:09.812541  5751 raft_consensus.cc:385] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.834326  5676 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:09.834966  5751 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.835323  5903 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:09.835884  5903 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:09.836159  5751 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.839133  5791 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:09.836880  5903 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.841974  5751 ts_tablet_manager.cc:1434] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.001s
I20260709 12:24:09.836018  5752 tablet_bootstrap.cc:654] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.843802  5751 tablet_bootstrap.cc:492] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:09.845362  5634 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:09.848312  5750 ts_tablet_manager.cc:1434] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.082s	user 0.008s	sys 0.000s
I20260709 12:24:09.854338  5752 tablet_bootstrap.cc:492] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:09.854818  5752 ts_tablet_manager.cc:1403] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.025s	user 0.004s	sys 0.007s
I20260709 12:24:09.854347  5791 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.855304  5791 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.857379  5791 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.857537  5752 raft_consensus.cc:359] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.858196  5752 raft_consensus.cc:385] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.858455  5752 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.859373  5750 tablet_bootstrap.cc:492] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:09.859066  5752 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.859193  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:09.860044  5708 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:09.861102  5751 tablet_bootstrap.cc:654] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.861735  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:09.865154  5791 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:09.865518  5791 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.865859  5791 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.865378  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:09.872195  5791 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.876080  5942 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:09.877429  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:09.874185  5791 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.877640  5752 ts_tablet_manager.cc:1434] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.000s
I20260709 12:24:09.878022  5708 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:24:09.879307  5602 leader_election.cc:343] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.880414  5846 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:09.883826  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:09.878871  5752 tablet_bootstrap.cc:492] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:09.886590  5816 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:09.889133  5849 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:09.886754  5708 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:09.892026  5750 tablet_bootstrap.cc:654] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.896716  5788 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.897348  5788 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.897756  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:09.898617  5790 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:09.899104  5790 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:09.899636  5788 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:09.899914  5790 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.900902  5903 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:09.902333  5849 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:09.903192  5757 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:09.905140  5751 tablet_bootstrap.cc:492] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
W20260709 12:24:09.904955  5602 leader_election.cc:343] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.905697  5942 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.907294  5751 ts_tablet_manager.cc:1403] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.064s	user 0.013s	sys 0.000s
I20260709 12:24:09.907588  5942 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.910140  5942 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:09.911173  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:09.912401  5599 leader_election.cc:343] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.912671  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:09.915594  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:09.916245  5559 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:09.917900  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:09.918015  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:24:09.919360  5602 leader_election.cc:343] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.919780  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:09.920805  5896 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:09.920939  5751 raft_consensus.cc:359] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.922049  5751 raft_consensus.cc:385] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.922358  5751 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.922998  5751 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.925035  5751 ts_tablet_manager.cc:1434] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.000s
I20260709 12:24:09.925994  5751 tablet_bootstrap.cc:492] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
W20260709 12:24:09.926043  5675 leader_election.cc:343] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.929134  5752 tablet_bootstrap.cc:654] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.932808  5751 tablet_bootstrap.cc:654] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.932686  5903 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:09.933287  5903 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.933648  5903 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.939054  5456 catalog_manager.cc:5697] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:09.949466  5941 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:09.960438  5903 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.962711  5903 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:09.964226  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:09.964972  5557 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:09.971009  5750 tablet_bootstrap.cc:492] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:09.971465  5750 ts_tablet_manager.cc:1403] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.112s	user 0.011s	sys 0.001s
I20260709 12:24:09.972649  5459 catalog_manager.cc:5697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:09.973901  5750 raft_consensus.cc:359] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.974534  5750 raft_consensus.cc:385] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.974777  5750 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.975395  5750 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.981896  5750 ts_tablet_manager.cc:1434] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:24:09.982800  5750 tablet_bootstrap.cc:492] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:09.983490  5752 tablet_bootstrap.cc:492] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:09.983920  5752 ts_tablet_manager.cc:1403] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.105s	user 0.009s	sys 0.002s
I20260709 12:24:09.986435  5752 raft_consensus.cc:359] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.987082  5752 raft_consensus.cc:385] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:09.987386  5752 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:09.988262  5750 tablet_bootstrap.cc:654] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:09.988155  5557 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:09.989337  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:09.990061  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:09.990972  5903 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:09.992674  5903 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:09.993295  5752 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:09.994946  5750 tablet_bootstrap.cc:492] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:09.995381  5752 ts_tablet_manager.cc:1434] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.001s
I20260709 12:24:09.996224  5752 tablet_bootstrap.cc:492] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:09.995381  5750 ts_tablet_manager.cc:1403] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.013s	user 0.009s	sys 0.000s
W20260709 12:24:09.997444  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:24:10.000519  5750 raft_consensus.cc:359] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.001174  5750 raft_consensus.cc:385] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.001448  5750 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.004279  5752 tablet_bootstrap.cc:654] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.005110  5846 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.005620  5846 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.007930  5846 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:10.012194  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:10.013655  5558 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:10.014825  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:10.015695  5846 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:10.016057  5846 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.016417  5846 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.017369  5752 tablet_bootstrap.cc:492] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:10.020812  5752 ts_tablet_manager.cc:1403] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.025s	user 0.007s	sys 0.010s
I20260709 12:24:10.022957  5846 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.024202  5896 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.024694  5896 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:10.025029  5675 leader_election.cc:343] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:09.995877  5903 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.026902  5942 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:10.033624  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:10.034543  5558 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:10.036017  5846 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:10.037074  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:10.037503  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:10.037781  5750 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.038156  5559 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.038645  5896 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:10.036042  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:24:10.040527  5599 leader_election.cc:343] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:10.043076  5790 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:10.043488  5790 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.043816  5790 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.044100  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:10.044757  5635 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:10.045069  5559 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:10.047035  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:10.023952  5752 raft_consensus.cc:359] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.048552  5752 raft_consensus.cc:385] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.048848  5752 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.050963  5751 tablet_bootstrap.cc:492] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:10.051889  5790 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.053975  5942 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:10.054628  5942 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:10.055658  5942 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.066151  5751 ts_tablet_manager.cc:1403] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.140s	user 0.013s	sys 0.000s
I20260709 12:24:10.066629  5790 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:10.066844  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:10.068023  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:10.068650  5558 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.073534  5750 ts_tablet_manager.cc:1434] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.076s	user 0.005s	sys 0.000s
I20260709 12:24:10.074496  5750 tablet_bootstrap.cc:492] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
W20260709 12:24:10.075263  5599 leader_election.cc:343] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:10.077854  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:10.078677  5635 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.078971  5751 raft_consensus.cc:359] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.079633  5751 raft_consensus.cc:385] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.079895  5751 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.080797  5752 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.082953  5752 ts_tablet_manager.cc:1434] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.062s	user 0.004s	sys 0.001s
I20260709 12:24:10.084061  5752 tablet_bootstrap.cc:492] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:10.080277  5558 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:10.091547  5635 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:10.095325  5751 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.103246  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:10.105963  5751 ts_tablet_manager.cc:1434] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.000s
I20260709 12:24:10.110428  5751 tablet_bootstrap.cc:492] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:10.109553  5791 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:10.114274  5791 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:10.115223  5791 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.121595  5896 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.122154  5896 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.123792  5752 tablet_bootstrap.cc:654] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.124274  5896 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:10.123980  5459 catalog_manager.cc:5697] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:10.126076  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:10.126734  5708 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:10.127220  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:10.127868  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:24:10.129026  5602 leader_election.cc:343] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:10.130681  5790 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:10.131110  5790 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.131615  5790 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.139721  5751 tablet_bootstrap.cc:654] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.161757  5750 tablet_bootstrap.cc:654] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.179628  5752 tablet_bootstrap.cc:492] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:10.180145  5752 ts_tablet_manager.cc:1403] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.096s	user 0.014s	sys 0.021s
I20260709 12:24:10.185124  5752 raft_consensus.cc:359] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.189949  5752 raft_consensus.cc:385] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.190276  5752 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.190966  5752 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.192950  5790 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.196807  5752 ts_tablet_manager.cc:1434] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.001s
I20260709 12:24:10.198000  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:10.198642  5708 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.198912  5752 tablet_bootstrap.cc:492] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:10.206583  5459 catalog_manager.cc:5697] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:10.206003  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:10.205396  5790 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:10.218469  5602 leader_election.cc:343] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:10.219056  5708 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:10.221313  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:10.221972  5750 tablet_bootstrap.cc:492] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:10.222494  5750 ts_tablet_manager.cc:1403] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.148s	user 0.015s	sys 0.026s
I20260709 12:24:10.223698  5966 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:10.225246  5966 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:10.226182  5966 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.227396  5750 raft_consensus.cc:359] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.228228  5750 raft_consensus.cc:385] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.228600  5750 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.229131  5791 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.229743  5791 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.229791  5750 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.232620  5750 ts_tablet_manager.cc:1434] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.002s
I20260709 12:24:10.232702  5751 tablet_bootstrap.cc:492] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:10.233409  5751 ts_tablet_manager.cc:1403] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.126s	user 0.017s	sys 0.007s
I20260709 12:24:10.234004  5750 tablet_bootstrap.cc:492] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:10.235848  5965 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.236408  5965 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.239130  5965 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:10.241068  5750 tablet_bootstrap.cc:654] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.241168  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:10.232113  5791 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:10.241927  5751 raft_consensus.cc:359] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.243754  5751 raft_consensus.cc:385] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.244093  5751 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.242599  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:10.241883  5557 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:10.244004  5459 catalog_manager.cc:5697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:10.245143  5751 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:10.246426  5602 leader_election.cc:343] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:10.247136  5752 tablet_bootstrap.cc:654] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.247107  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:10.247771  5751 ts_tablet_manager.cc:1434] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:24:10.248016  5791 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:10.249145  5751 tablet_bootstrap.cc:492] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:10.248456  5791 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.258441  5791 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.267699  5791 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.273419  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:10.274714  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:10.275164  5599 leader_election.cc:343] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:10.275625  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:24:10.276841  5599 leader_election.cc:343] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:10.277523  5790 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:10.286728  5965 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:10.287026  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:10.293133  5559 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.294458  5791 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:10.299371  5751 tablet_bootstrap.cc:654] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.300519  5750 tablet_bootstrap.cc:492] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:10.301000  5750 ts_tablet_manager.cc:1403] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.067s	user 0.015s	sys 0.009s
I20260709 12:24:10.311941  5750 raft_consensus.cc:359] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.312700  5750 raft_consensus.cc:385] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.312973  5750 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.313757  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:10.313659  5750 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.315848  5750 ts_tablet_manager.cc:1434] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:24:10.316258  5559 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:10.317430  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:10.318471  5965 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:10.318960  5965 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:24:10.320631  5599 leader_election.cc:343] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:10.322952  5752 tablet_bootstrap.cc:492] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:10.324767  5752 ts_tablet_manager.cc:1403] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.126s	user 0.016s	sys 0.006s
I20260709 12:24:10.326391  5965 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.317094  5750 tablet_bootstrap.cc:492] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
W20260709 12:24:10.336274  5602 consensus_peers.cc:597] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:10.338162  5459 catalog_manager.cc:5697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:10.345211  5751 tablet_bootstrap.cc:492] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:10.345711  5750 tablet_bootstrap.cc:654] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.345711  5751 ts_tablet_manager.cc:1403] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.097s	user 0.022s	sys 0.010s
I20260709 12:24:10.347623  5752 raft_consensus.cc:359] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.348321  5752 raft_consensus.cc:385] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.348600  5751 raft_consensus.cc:359] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.349370  5751 raft_consensus.cc:385] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.349658  5751 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.350346  5751 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.358621  5751 ts_tablet_manager.cc:1434] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260709 12:24:10.359840  5751 tablet_bootstrap.cc:492] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:10.363785  5457 catalog_manager.cc:5697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:10.348636  5752 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.368433  5751 tablet_bootstrap.cc:654] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.375805  5752 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.381515  5752 ts_tablet_manager.cc:1434] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.045s	user 0.003s	sys 0.002s
I20260709 12:24:10.383404  5752 tablet_bootstrap.cc:492] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:10.384703  5751 tablet_bootstrap.cc:492] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:10.385174  5751 ts_tablet_manager.cc:1403] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.026s	user 0.007s	sys 0.011s
I20260709 12:24:10.386773  5750 tablet_bootstrap.cc:492] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:10.387538  5750 ts_tablet_manager.cc:1403] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.071s	user 0.023s	sys 0.020s
I20260709 12:24:10.388263  5751 raft_consensus.cc:359] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.388890  5751 raft_consensus.cc:385] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.389129  5751 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.389745  5751 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.390938  5750 raft_consensus.cc:359] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.391729  5751 ts_tablet_manager.cc:1434] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:24:10.391739  5750 raft_consensus.cc:385] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.392257  5750 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.392819  5751 tablet_bootstrap.cc:492] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:10.393211  5750 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.394773  5752 tablet_bootstrap.cc:654] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.395100  5942 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.395717  5750 ts_tablet_manager.cc:1434] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.008s	user 0.000s	sys 0.007s
I20260709 12:24:10.395561  5942 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.398885  5751 tablet_bootstrap.cc:654] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.402170  5750 tablet_bootstrap.cc:492] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:10.404909  5903 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:10.410192  5751 tablet_bootstrap.cc:492] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:10.411213  5751 ts_tablet_manager.cc:1403] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.018s	user 0.010s	sys 0.005s
I20260709 12:24:10.413667  5942 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:10.414088  5751 raft_consensus.cc:359] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.414783  5751 raft_consensus.cc:385] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.414263  5942 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:10.415062  5751 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.416903  5751 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.417325  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:10.417484  5750 tablet_bootstrap.cc:654] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.417936  5557 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:10.419867  5751 ts_tablet_manager.cc:1434] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 12:24:10.421099  5751 tablet_bootstrap.cc:492] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:10.422122  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:10.422742  5635 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:10.426270  5676 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:10.427104  5942 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:10.427465  5942 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.427654  5751 tablet_bootstrap.cc:654] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.427878  5942 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.430316  5750 tablet_bootstrap.cc:492] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:10.430930  5846 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.431489  5750 ts_tablet_manager.cc:1403] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.004s
I20260709 12:24:10.431394  5846 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.436937  5750 raft_consensus.cc:359] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.437907  5750 raft_consensus.cc:385] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.438246  5750 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.439002  5750 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.433804  5846 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:10.441789  5750 ts_tablet_manager.cc:1434] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.001s
I20260709 12:24:10.443269  5750 tablet_bootstrap.cc:492] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:10.441278  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:10.448426  5557 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 1: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:24:10.437400  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:10.452821  5751 tablet_bootstrap.cc:492] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:10.453653  5751 ts_tablet_manager.cc:1403] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.033s	user 0.006s	sys 0.011s
I20260709 12:24:10.453634  5750 tablet_bootstrap.cc:654] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.456744  5751 raft_consensus.cc:359] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.457508  5751 raft_consensus.cc:385] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.457829  5751 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.459659  5751 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.462533  5751 ts_tablet_manager.cc:1434] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:24:10.463515  5751 tablet_bootstrap.cc:492] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:10.473759  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:10.474555  5941 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.476835  5751 tablet_bootstrap.cc:654] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.482832  5965 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.483537  5965 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.487352  5965 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:10.489295  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:10.490362  5557 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 1: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:10.491871  5791 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.492532  5791 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.495110  5791 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:10.495383  5750 tablet_bootstrap.cc:492] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:10.496409  5750 ts_tablet_manager.cc:1403] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.053s	user 0.012s	sys 0.020s
I20260709 12:24:10.497653  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:10.501423  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:10.502028  5709 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:10.502593  5750 raft_consensus.cc:359] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.503342  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:10.503571  5750 raft_consensus.cc:385] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.504135  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:10.504388  5849 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.505268  5791 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:10.505059  5849 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:10.506507  5602 leader_election.cc:343] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:10.507910  5849 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:10.505415  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:10.505707  5791 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.509225  5966 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.509529  5791 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.503986  5750 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.518563  5750 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.527340  5791 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.527747  5846 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:10.530315  5816 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.530990  5816 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.533674  5816 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:10.534339  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:10.534976  5634 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:10.536176  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:10.539000  5816 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:10.539377  5816 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.539758  5816 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.546437  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:10.550158  5750 ts_tablet_manager.cc:1434] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.053s	user 0.008s	sys 0.000s
I20260709 12:24:10.551399  5750 tablet_bootstrap.cc:492] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:10.562278  5757 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.564822  5757 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.566774  5757 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:10.569520  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:10.572396  5966 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:10.573375  5791 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:10.574640  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:10.575436  5710 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.583561  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:24:10.593560  5602 leader_election.cc:343] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:10.637553  5710 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:10.638293  5966 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:10.638677  5752 tablet_bootstrap.cc:492] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:10.638537  5751 tablet_bootstrap.cc:492] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:10.639161  5752 ts_tablet_manager.cc:1403] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.256s	user 0.085s	sys 0.136s
I20260709 12:24:10.639272  5751 ts_tablet_manager.cc:1403] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.176s	user 0.073s	sys 0.064s
I20260709 12:24:10.640144  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:10.641744  5896 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:10.641849  5752 raft_consensus.cc:359] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.642642  5752 raft_consensus.cc:385] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.642478  5896 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:10.642958  5752 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.643721  5896 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.648277  5752 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.649859  5752 ts_tablet_manager.cc:1434] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.010s	user 0.011s	sys 0.000s
I20260709 12:24:10.650753  5752 tablet_bootstrap.cc:492] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:10.652323  5751 raft_consensus.cc:359] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.655027  5751 raft_consensus.cc:385] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:10.655894  5751 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:10.656836  5752 tablet_bootstrap.cc:654] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.656976  5751 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.668154  5941 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:10.670329  5750 tablet_bootstrap.cc:654] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.692934  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:10.693938  5708 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:10.695973  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:10.697335  5710 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 1: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:24:10.698948  5791 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:10.679052  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:10.657765  5942 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.719480  5849 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.733933  5849 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.738833  5942 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:10.713514  5816 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.773710  5751 ts_tablet_manager.cc:1434] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.134s	user 0.028s	sys 0.015s
I20260709 12:24:10.775197  5751 tablet_bootstrap.cc:492] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:10.770504  5849 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:10.771147  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 88557.9us
I20260709 12:24:10.783901  5941 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 12:24:10.806170  5558 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.790208  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:10.782871  5964 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.808275  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:10.819998  5635 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.813192  5557 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.806051  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:10.798024  5816 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:10.795449  5989 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.824790  5990 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260709 12:24:10.825395  5989 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.826196  5633 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:10.827841  5989 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:10.830054  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:10.830483  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:10.831017  5849 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.831348  5816 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:10.831727  5816 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.832050  5816 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.837106  5750 tablet_bootstrap.cc:492] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:10.837739  5750 ts_tablet_manager.cc:1403] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.287s	user 0.015s	sys 0.009s
I20260709 12:24:10.845526  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:10.847569  5459 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:10.843685  5941 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.849138  5941 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:10.851758  5941 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:10.817282  5964 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:10.836321  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47303.3us
I20260709 12:24:10.842680  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:10.864001  5752 tablet_bootstrap.cc:492] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:10.864745  5752 ts_tablet_manager.cc:1403] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.214s	user 0.011s	sys 0.004s
I20260709 12:24:10.865587  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:10.866250  5634 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:10.867143  5751 tablet_bootstrap.cc:654] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:10.871668  5633 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.871981  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:10.870261  5990 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:24:10.900452  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:10.893368  5706 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:10.914911  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:10.917459  5964 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:11.005218  5941 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.973803  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:11.058987  5750 raft_consensus.cc:359] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.098605  5750 raft_consensus.cc:385] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:11.099017  5750 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:11.101423  5750 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.106220  5752 raft_consensus.cc:359] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.107100  5752 raft_consensus.cc:385] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:11.107419  5752 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:11.108213  5752 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.080139  5849 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:11.123574  5750 ts_tablet_manager.cc:1434] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.285s	user 0.022s	sys 0.025s
I20260709 12:24:11.123642  5752 ts_tablet_manager.cc:1434] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.259s	user 0.009s	sys 0.023s
I20260709 12:24:11.054499  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:11.088109  5964 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
W20260709 12:24:11.129581  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 135805us
I20260709 12:24:11.057219  5941 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.140750  5989 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:24:11.136982  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 212097us
I20260709 12:24:11.155413  5635 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:11.166314  5557 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:11.167950  5750 tablet_bootstrap.cc:492] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:11.152644  5989 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.201463  5751 tablet_bootstrap.cc:492] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:11.202466  5751 ts_tablet_manager.cc:1403] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.428s	user 0.022s	sys 0.031s
I20260709 12:24:11.183790  5989 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:11.181097  5633 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:11.217163  5706 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:11.212968  5816 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.239966  6002 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:11.242470  5816 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:11.228000  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:11.244105  5560 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 1: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:24:11.244059  5751 raft_consensus.cc:359] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.244918  5751 raft_consensus.cc:385] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:11.245244  5751 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:11.245903  5751 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.248152  5998 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.248664  5998 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:11.254576  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 335044us
I20260709 12:24:11.258031  5751 ts_tablet_manager.cc:1434] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.055s	user 0.007s	sys 0.000s
I20260709 12:24:11.259068  5751 tablet_bootstrap.cc:492] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:11.259897  5904 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.261176  5998 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:11.255750  5524 leader_election.cc:343] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:11.262091  5904 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.263209  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:11.268374  6005 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
W20260709 12:24:11.279086  5676 leader_election.cc:343] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:11.279209  5989 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.285954  5705 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:11.290755  5752 tablet_bootstrap.cc:492] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:11.291867  5904 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:11.294966  5676 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:11.300254  5990 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:11.301962  5990 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:24:11.301676  5599 leader_election.cc:343] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:11.305791  5941 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:11.308418  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:11.317049  6002 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:11.303016  5990 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.321317  6002 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:11.315910  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:11.332094  5989 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:11.338923  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:11.348068  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46874.2us
I20260709 12:24:11.353664  5791 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.355489  5630 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:11.357995  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:11.365667  5791 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.355696  6002 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.372606  5629 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:11.368498  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:11.397526  5752 tablet_bootstrap.cc:654] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:11.403791  5750 tablet_bootstrap.cc:654] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:11.402757  5628 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:11.409209  5791 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:11.419768  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:11.420478  5628 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:11.428896  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:11.436419  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:11.436990  5629 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:11.437170  5555 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:11.442210  5751 tablet_bootstrap.cc:654] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:11.455242  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:11.448042  5816 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:11.457504  5816 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.457894  5816 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:11.465711  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:11.466480  5702 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:11.469827  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:11.474718  5997 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:11.481024  6003 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:11.481660  6003 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:11.482504  6003 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.497704  5750 tablet_bootstrap.cc:492] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:11.498261  5750 ts_tablet_manager.cc:1403] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.331s	user 0.024s	sys 0.012s
W20260709 12:24:11.507800  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:24:11.510360  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:11.513321  5999 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260709 12:24:11.498116  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:11.515733  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:11.524941  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:11.525301  5964 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:11.525724  5964 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.526062  5964 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:11.528754  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:11.533830  5459 catalog_manager.cc:5697] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:11.539356  6018 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:11.539794  6018 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.540150  6018 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:11.540484  5999 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20260709 12:24:11.542846  5676 leader_election.cc:343] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:11.530268  5628 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:11.555493  5816 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.558111  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:11.563993  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:11.567827  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:11.570425  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:11.571831  6016 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:11.602140  5751 tablet_bootstrap.cc:492] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:11.603034  5751 ts_tablet_manager.cc:1403] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.344s	user 0.001s	sys 0.040s
I20260709 12:24:11.604074  6016 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:11.604034  5750 raft_consensus.cc:359] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.605437  5750 raft_consensus.cc:385] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:11.605686  5750 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:11.606428  5750 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.609987  5750 ts_tablet_manager.cc:1434] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.111s	user 0.008s	sys 0.000s
I20260709 12:24:11.611070  5750 tablet_bootstrap.cc:492] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:11.608222  5816 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:11.617601  5752 tablet_bootstrap.cc:492] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
W20260709 12:24:11.609987  5524 leader_election.cc:343] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:11.618613  5752 ts_tablet_manager.cc:1403] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.328s	user 0.013s	sys 0.012s
W20260709 12:24:11.622886  5602 leader_election.cc:343] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:11.623288  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:11.606804  6016 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.625623  5896 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:24:11.627839  5602 consensus_peers.cc:597] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:11.622344  5599 leader_election.cc:343] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:11.618804  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:11.621855  6018 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:11.633051  5602 leader_election.cc:343] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:24:11.636188  5602 leader_election.cc:343] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:11.636698  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:11.636301  5752 raft_consensus.cc:359] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.638444  5752 raft_consensus.cc:385] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:11.638761  5752 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:11.640106  5966 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:11.641888  6006 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 12:24:11.646735  5706 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 1: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:11.640668  5631 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:11.640547  5752 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.635212  6009 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:11.667124  5752 ts_tablet_manager.cc:1434] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.048s	user 0.008s	sys 0.000s
I20260709 12:24:11.668442  5752 tablet_bootstrap.cc:492] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:11.670274  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:11.671087  5710 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 1: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:11.677174  6009 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.684808  6009 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.691926  5964 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.703889  6009 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:11.707546  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:11.708470  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:11.709091  5602 leader_election.cc:343] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:11.705808  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:24:11.710135  5602 leader_election.cc:343] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:11.710263  5555 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 1: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:24:11.710507  5750 tablet_bootstrap.cc:654] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:11.711901  5964 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:11.719377  5751 raft_consensus.cc:359] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.720093  5751 raft_consensus.cc:385] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:11.720382  5751 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:11.720898  5631 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:11.722083  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:11.722913  5751 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.725015  5816 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:11.726390  5816 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:11.726585  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:11.727206  5816 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.727711  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:11.728240  5710 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:24:11.730077  5599 leader_election.cc:343] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:11.735208  5751 ts_tablet_manager.cc:1434] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.131s	user 0.005s	sys 0.003s
I20260709 12:24:11.738389  5751 tablet_bootstrap.cc:492] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:11.730607  5599 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:11.736652  6018 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:11.741927  5998 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:11.750489  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:11.751714  5750 tablet_bootstrap.cc:492] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:11.764333  5631 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 1: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:24:11.766373  5750 ts_tablet_manager.cc:1403] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.155s	user 0.012s	sys 0.009s
I20260709 12:24:11.767668  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:11.770258  5751 tablet_bootstrap.cc:654] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:11.771250  5752 tablet_bootstrap.cc:654] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:11.773511  6018 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:24:11.775699  5710 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:11.788740  5458 catalog_manager.cc:5697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:11.794049  5750 raft_consensus.cc:359] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.794657  5750 raft_consensus.cc:385] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:11.794910  5750 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:11.794164  5458 catalog_manager.cc:5697] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:11.795601  5750 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.812325  5750 ts_tablet_manager.cc:1434] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.046s	user 0.006s	sys 0.001s
I20260709 12:24:11.813467  5750 tablet_bootstrap.cc:492] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:11.817404  6016 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:24:11.820905  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:11.821877  5999 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:11.822342  5999 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:11.826908  5849 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.828258  5849 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.831775  5849 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:11.833262  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:11.833981  5628 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:11.836997  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:11.838299  5904 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:11.838748  5904 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.839123  5904 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:11.830828  5999 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:11.847021  6038 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.078s	user 0.000s	sys 0.002s
I20260709 12:24:11.859165  5751 tablet_bootstrap.cc:492] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:11.860611  5751 ts_tablet_manager.cc:1403] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.124s	user 0.017s	sys 0.001s
I20260709 12:24:11.867260  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:11.865446  5459 catalog_manager.cc:5697] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:11.885962  5751 raft_consensus.cc:359] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.889979  5751 raft_consensus.cc:385] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:11.890249  5751 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:11.891094  5751 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.892030  5709 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:11.897486  6016 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:11.897923  5751 ts_tablet_manager.cc:1434] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.037s	user 0.004s	sys 0.002s
I20260709 12:24:11.898923  5751 tablet_bootstrap.cc:492] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:11.902209  5904 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.909140  5750 tablet_bootstrap.cc:654] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:11.913173  5752 tablet_bootstrap.cc:492] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:11.913796  5752 ts_tablet_manager.cc:1403] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.246s	user 0.030s	sys 0.012s
I20260709 12:24:11.918340  5457 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:11.922040  5904 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:11.922034  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:11.923563  5636 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:11.924329  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:11.924847  5709 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:11.934614  5709 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:11.937551  5524 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:11.940789  5904 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:11.951673  5904 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:11.952968  5904 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.954151  5752 raft_consensus.cc:359] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.954651  5816 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:11.954862  5752 raft_consensus.cc:385] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:11.955332  5752 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:11.956333  5752 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.957140  5636 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:11.958873  5752 ts_tablet_manager.cc:1434] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.045s	user 0.007s	sys 0.000s
I20260709 12:24:11.960289  5752 tablet_bootstrap.cc:492] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:11.974380  5750 tablet_bootstrap.cc:492] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:11.975019  5750 ts_tablet_manager.cc:1403] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.162s	user 0.016s	sys 0.013s
I20260709 12:24:11.979216  5751 tablet_bootstrap.cc:654] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:11.983693  5941 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.984267  5941 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.986655  5941 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:11.988549  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:11.988901  5750 raft_consensus.cc:359] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.989640  5750 raft_consensus.cc:385] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:11.989923  5750 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:11.990664  5750 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:11.993350  5750 ts_tablet_manager.cc:1434] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260709 12:24:11.982543  5457 catalog_manager.cc:5697] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:11.992851  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:11.994429  5750 tablet_bootstrap.cc:492] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:11.997087  5631 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 1: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:24:12.014492  5752 tablet_bootstrap.cc:654] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.021095  5675 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:12.024927  6016 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.025064  6008 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:24:12.045670  5459 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:12.051252  5751 tablet_bootstrap.cc:492] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:12.051827  5751 ts_tablet_manager.cc:1403] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.153s	user 0.014s	sys 0.013s
I20260709 12:24:12.053074  5750 tablet_bootstrap.cc:654] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.054183  6008 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:12.056609  5751 raft_consensus.cc:359] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.058744  5751 raft_consensus.cc:385] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.059408  5751 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.060882  5751 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.065760  5751 ts_tablet_manager.cc:1434] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 12:24:12.068475  5751 tablet_bootstrap.cc:492] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:12.079165  6008 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.079762  6008 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.084957  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:12.085929  5628 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 1: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:12.086426  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:12.084968  6008 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.089866  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:12.092689  6003 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.099685  6016 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:12.116058  5757 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:24:12.123054  5752 tablet_bootstrap.cc:492] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:12.123549  5752 ts_tablet_manager.cc:1403] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.164s	user 0.016s	sys 0.010s
I20260709 12:24:12.128420  5560 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.130957  5751 tablet_bootstrap.cc:654] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.134862  5752 raft_consensus.cc:359] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.135527  5752 raft_consensus.cc:385] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.135788  5752 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.137426  6003 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:12.138008  5752 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.141703  5752 ts_tablet_manager.cc:1434] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260709 12:24:12.143998  5752 tablet_bootstrap.cc:492] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:12.150559  5966 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.153746  5966 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.156880  5966 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.158151  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:12.158749  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:12.159833  5599 leader_election.cc:343] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:24:12.160315  5602 leader_election.cc:343] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.160809  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:12.166929  5966 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:12.174212  5941 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:12.188669  5750 tablet_bootstrap.cc:492] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:12.189167  5750 ts_tablet_manager.cc:1403] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.195s	user 0.010s	sys 0.011s
I20260709 12:24:12.193131  5750 raft_consensus.cc:359] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.193737  5750 raft_consensus.cc:385] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.193996  5750 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.194587  5750 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.196587  5750 ts_tablet_manager.cc:1434] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:24:12.197429  5750 tablet_bootstrap.cc:492] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:12.203176  5752 tablet_bootstrap.cc:654] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.210920  5750 tablet_bootstrap.cc:654] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.211053  5757 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:12.214476  5752 tablet_bootstrap.cc:492] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:12.215029  5752 ts_tablet_manager.cc:1403] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.071s	user 0.011s	sys 0.000s
I20260709 12:24:12.217942  6016 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:12.218187  5752 raft_consensus.cc:359] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.218958  5752 raft_consensus.cc:385] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.219230  5752 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.220206  5751 tablet_bootstrap.cc:492] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:12.219937  5752 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.220773  5751 ts_tablet_manager.cc:1403] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.153s	user 0.015s	sys 0.000s
I20260709 12:24:12.222167  5752 ts_tablet_manager.cc:1434] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:24:12.223165  5752 tablet_bootstrap.cc:492] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:12.224027  5751 raft_consensus.cc:359] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.224917  5751 raft_consensus.cc:385] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.225346  5751 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.226383  5751 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.229149  5751 ts_tablet_manager.cc:1434] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:24:12.234591  5751 tablet_bootstrap.cc:492] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:12.238711  5752 tablet_bootstrap.cc:654] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.248164  5751 tablet_bootstrap.cc:654] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.259433  5999 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.259248  5816 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:12.259907  5999 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.270143  5750 tablet_bootstrap.cc:492] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:12.271497  5750 ts_tablet_manager.cc:1403] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.074s	user 0.013s	sys 0.009s
I20260709 12:24:12.273321  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:12.275333  5602 leader_election.cc:343] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.274660  5750 raft_consensus.cc:359] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.277181  5750 raft_consensus.cc:385] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.281940  5750 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.277467  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:12.282711  5750 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:12.291379  5599 leader_election.cc:343] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.291968  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:12.293272  5966 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:12.277976  5999 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.310315  5752 tablet_bootstrap.cc:492] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:12.312636  5750 ts_tablet_manager.cc:1434] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.041s	user 0.006s	sys 0.001s
I20260709 12:24:12.313834  5750 tablet_bootstrap.cc:492] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:12.316018  5752 ts_tablet_manager.cc:1403] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.093s	user 0.014s	sys 0.007s
I20260709 12:24:12.328820  5816 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.329972  5816 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.338155  5816 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.338914  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:12.342139  5752 raft_consensus.cc:359] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.342633  5752 raft_consensus.cc:385] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.342796  5752 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.344197  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:12.344811  5706 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:12.345948  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:12.347030  6003 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:12.347115  5634 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:12.347499  6003 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.347898  6003 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.348918  5752 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.352728  5752 ts_tablet_manager.cc:1434] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.036s	user 0.006s	sys 0.000s
I20260709 12:24:12.360929  5751 tablet_bootstrap.cc:492] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:12.361503  5751 ts_tablet_manager.cc:1403] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.127s	user 0.029s	sys 0.022s
I20260709 12:24:12.365202  5751 raft_consensus.cc:359] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.365875  5751 raft_consensus.cc:385] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.366139  5751 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.367703  5750 tablet_bootstrap.cc:654] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.369910  6003 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.370747  5752 tablet_bootstrap.cc:492] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:12.370487  5751 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.373014  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:12.374198  5634 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.379009  5751 ts_tablet_manager.cc:1434] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.000s
I20260709 12:24:12.380470  5751 tablet_bootstrap.cc:492] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:12.422622  5634 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:12.422821  5751 tablet_bootstrap.cc:654] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.423906  6003 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.424125  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:12.425027  5752 tablet_bootstrap.cc:654] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.427656  6003 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:12.430171  5750 tablet_bootstrap.cc:492] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:12.430174  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:12.430846  5750 ts_tablet_manager.cc:1403] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.117s	user 0.017s	sys 0.015s
I20260709 12:24:12.430965  5706 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.435369  5750 raft_consensus.cc:359] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.436097  5750 raft_consensus.cc:385] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.428177  6003 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:12.436388  5750 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.437104  5750 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.437162  6003 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.438501  5706 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:12.438930  5750 ts_tablet_manager.cc:1434] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.002s
I20260709 12:24:12.440544  5750 tablet_bootstrap.cc:492] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:12.441838  5999 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.442322  5999 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.444479  5999 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.445922  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:12.446939  5709 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:12.448565  5599 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:12.449883  5790 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:12.450264  5790 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.450619  5790 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.446944  5750 tablet_bootstrap.cc:654] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.455497  5816 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:12.458781  5790 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.460136  6009 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:12.460711  5752 tablet_bootstrap.cc:492] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:12.461400  5752 ts_tablet_manager.cc:1403] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.091s	user 0.010s	sys 0.003s
I20260709 12:24:12.464761  5752 raft_consensus.cc:359] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.468632  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:12.479468  5752 raft_consensus.cc:385] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.479862  5752 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.469813  5558 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.485551  5458 catalog_manager.cc:5697] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:12.472254  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:12.474977  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:24:12.479984  5602 leader_election.cc:343] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.492763  5709 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:24:12.494318  5602 leader_election.cc:343] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.469656  5790 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.501596  5752 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.505497  6003 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.506024  6003 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.510093  5791 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.510630  5791 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.511250  6003 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.512713  5791 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.513130  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:12.513767  5636 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:12.514254  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:12.518246  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:12.521106  5752 ts_tablet_manager.cc:1434] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.059s	user 0.006s	sys 0.000s
I20260709 12:24:12.525172  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:12.526237  5752 tablet_bootstrap.cc:492] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
W20260709 12:24:12.524026  5599 leader_election.cc:343] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.524502  5904 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:12.525048  5709 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:12.521903  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:24:12.529162  5602 leader_election.cc:343] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.537611  5599 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:12.538188  5602 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:24:12.538956  5524 leader_election.cc:343] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.540480  5999 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:12.538630  6009 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:12.541291  5904 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:12.541833  5904 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.542225  5904 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.543185  5966 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:12.541361  6009 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:12.549182  6009 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.566092  5751 tablet_bootstrap.cc:492] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:12.566648  5751 ts_tablet_manager.cc:1403] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.186s	user 0.011s	sys 0.015s
I20260709 12:24:12.566704  5752 tablet_bootstrap.cc:654] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.569111  5750 tablet_bootstrap.cc:492] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:12.569600  5750 ts_tablet_manager.cc:1403] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.130s	user 0.023s	sys 0.004s
I20260709 12:24:12.592072  5751 raft_consensus.cc:359] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.592864  5751 raft_consensus.cc:385] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.593209  5751 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.593832  5751 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.598110  5751 ts_tablet_manager.cc:1434] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.031s	user 0.004s	sys 0.002s
I20260709 12:24:12.605067  5904 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.609539  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:12.610140  5636 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.611796  5458 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:12.612581  5750 raft_consensus.cc:359] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.613281  5750 raft_consensus.cc:385] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.613554  5750 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.614344  5750 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.616552  5904 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.616801  5750 ts_tablet_manager.cc:1434] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.047s	user 0.004s	sys 0.004s
I20260709 12:24:12.618278  5750 tablet_bootstrap.cc:492] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:12.633749  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:12.634349  5709 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.624753  5751 tablet_bootstrap.cc:492] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:12.637065  5750 tablet_bootstrap.cc:654] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.640285  5636 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:12.641477  5752 tablet_bootstrap.cc:492] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:12.642218  5752 ts_tablet_manager.cc:1403] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.116s	user 0.015s	sys 0.004s
I20260709 12:24:12.641971  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:12.643203  5904 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:12.643815  5904 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:12.645053  5904 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.653676  5709 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:12.653899  5751 tablet_bootstrap.cc:654] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.665283  5750 tablet_bootstrap.cc:492] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:12.657645  5752 raft_consensus.cc:359] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.666474  5752 raft_consensus.cc:385] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.666764  5752 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.672456  5750 ts_tablet_manager.cc:1403] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.054s	user 0.008s	sys 0.012s
I20260709 12:24:12.675427  5750 raft_consensus.cc:359] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.676187  5750 raft_consensus.cc:385] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.676486  5750 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.676946  5752 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.677181  5750 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.679492  5752 ts_tablet_manager.cc:1434] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.037s	user 0.008s	sys 0.004s
I20260709 12:24:12.684675  5457 catalog_manager.cc:5697] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:12.685592  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:24:12.689559  5750 ts_tablet_manager.cc:1434] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.000s
I20260709 12:24:12.690629  5750 tablet_bootstrap.cc:492] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:12.691324  5751 tablet_bootstrap.cc:492] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:12.692288  5752 tablet_bootstrap.cc:492] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:12.698165  5751 ts_tablet_manager.cc:1403] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.074s	user 0.014s	sys 0.000s
I20260709 12:24:12.700949  5751 raft_consensus.cc:359] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.701634  5751 raft_consensus.cc:385] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.701913  5751 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.703413  5750 tablet_bootstrap.cc:654] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.702567  5751 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.713800  5752 tablet_bootstrap.cc:654] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.715906  5903 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.716461  5903 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.718639  5903 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:12.721326  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:12.723598  5676 leader_election.cc:343] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.724633  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:12.725358  5636 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:12.727458  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:12.728119  5752 tablet_bootstrap.cc:492] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:12.728608  6018 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:12.728639  5752 ts_tablet_manager.cc:1403] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.037s	user 0.008s	sys 0.005s
I20260709 12:24:12.728855  5750 tablet_bootstrap.cc:492] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:12.729270  5751 ts_tablet_manager.cc:1434] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.031s	user 0.006s	sys 0.000s
I20260709 12:24:12.729416  5750 ts_tablet_manager.cc:1403] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.039s	user 0.013s	sys 0.001s
I20260709 12:24:12.730248  5751 tablet_bootstrap.cc:492] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:12.732535  5750 raft_consensus.cc:359] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.733174  5750 raft_consensus.cc:385] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.733422  5750 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.734064  5750 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.736310  5751 tablet_bootstrap.cc:654] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.735976  5752 raft_consensus.cc:359] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.736771  5752 raft_consensus.cc:385] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.737105  5752 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.737768  5752 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.743239  5750 ts_tablet_manager.cc:1434] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20260709 12:24:12.744073  5750 tablet_bootstrap.cc:492] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:12.745534  6003 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.746515  6003 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.749893  5750 tablet_bootstrap.cc:654] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.729067  6018 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.752439  5752 ts_tablet_manager.cc:1434] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.023s	user 0.005s	sys 0.001s
I20260709 12:24:12.752566  6018 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.753844  5752 tablet_bootstrap.cc:492] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:12.762264  5750 tablet_bootstrap.cc:492] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:12.762778  5750 ts_tablet_manager.cc:1403] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.019s	user 0.012s	sys 0.003s
I20260709 12:24:12.765867  5750 raft_consensus.cc:359] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.766530  5750 raft_consensus.cc:385] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.766829  5750 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.767081  5752 tablet_bootstrap.cc:654] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.769085  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:12.770597  5628 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:12.771276  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:12.771960  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:12.772948  5816 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:12.773267  5816 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.773550  5816 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.768847  6003 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:12.776857  5524 leader_election.cc:343] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.780433  5751 tablet_bootstrap.cc:492] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:12.780995  5751 ts_tablet_manager.cc:1403] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.051s	user 0.020s	sys 0.019s
I20260709 12:24:12.767690  5750 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.783509  5751 raft_consensus.cc:359] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.784158  5751 raft_consensus.cc:385] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.784425  5751 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.785370  5751 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.788132  5751 ts_tablet_manager.cc:1434] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:24:12.790541  5816 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.794847  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:12.795406  5636 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.798110  5751 tablet_bootstrap.cc:492] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:12.800096  5903 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.800592  5903 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.802599  5903 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:12.804039  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:12.804554  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:12.804709  5558 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:12.805176  5634 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:12.805960  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:12.808663  6016 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:12.809343  6016 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.809747  6016 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.812868  5752 tablet_bootstrap.cc:492] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:12.792472  5750 ts_tablet_manager.cc:1434] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.001s
I20260709 12:24:12.813968  5752 ts_tablet_manager.cc:1403] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.060s	user 0.008s	sys 0.006s
I20260709 12:24:12.815093  5816 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.817034  5750 tablet_bootstrap.cc:492] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:12.825774  5751 tablet_bootstrap.cc:654] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.828799  5752 raft_consensus.cc:359] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.831642  5752 raft_consensus.cc:385] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:12.832017  5752 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:12.832800  5752 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.837159  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:24:12.839584  5524 leader_election.cc:343] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.840788  5752 ts_tablet_manager.cc:1434] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.000s
I20260709 12:24:12.844619  5752 tablet_bootstrap.cc:492] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:12.847991  5636 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:12.854369  5904 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 12:24:12.857991  5966 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.831552  6018 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.866670  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:12.868547  6083 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:12.869047  6083 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:12.869896  6083 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.869042  5966 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.876204  5966 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.869231  5750 tablet_bootstrap.cc:654] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.884496  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:12.885950  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:12.886317  5602 leader_election.cc:343] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:24:12.888294  5599 leader_election.cc:343] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.888720  5599 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:12.890642  5791 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:12.907073  5816 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.907586  5816 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.911641  5816 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.915308  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:12.916150  5709 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:12.917825  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:12.918593  6084 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:12.919719  6084 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.920439  6084 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.924857  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:24:12.927034  5676 leader_election.cc:343] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.933985  5966 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.934863  5966 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.939318  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:12.942284  5599 leader_election.cc:343] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.949091  6084 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.952749  6084 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.957661  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:12.958168  5709 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.955986  5458 catalog_manager.cc:5697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:12.942229  5966 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:12.960669  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:12.961969  5752 tablet_bootstrap.cc:654] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:12.941421  6016 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:12.958848  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:12.959440  6018 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:12.965111  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:12.970050  5904 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:12.970082  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:12.970842  5751 tablet_bootstrap.cc:492] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
W20260709 12:24:12.973179  5602 leader_election.cc:343] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:12.974462  5602 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:12.974704  5751 ts_tablet_manager.cc:1403] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.177s	user 0.023s	sys 0.033s
I20260709 12:24:12.979847  5709 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:12.975666  5634 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.982409  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:12.983021  5558 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:12.976670  6009 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:12.965307  5636 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:12.991621  6016 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:12.992451  5750 tablet_bootstrap.cc:492] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:12.991637  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:12.993778  5904 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:12.994609  5750 ts_tablet_manager.cc:1403] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.178s	user 0.023s	sys 0.007s
I20260709 12:24:12.995069  5904 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:12.995996  5904 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.032676  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:13.033475  5635 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.034114  5751 raft_consensus.cc:359] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.030076  5558 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:13.035074  5751 raft_consensus.cc:385] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.042709  5751 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.043413  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:13.043521  5751 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.042742  5752 tablet_bootstrap.cc:492] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:13.044628  5752 ts_tablet_manager.cc:1403] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.200s	user 0.021s	sys 0.001s
I20260709 12:24:13.045171  6016 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:13.045724  6016 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:13.046572  6016 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.048276  5752 raft_consensus.cc:359] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.049084  5752 raft_consensus.cc:385] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.049341  5752 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.045876  5634 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:13.052525  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:13.053491  6018 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:13.054792  6018 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:13.055909  6018 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.051714  5752 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.058785  5751 ts_tablet_manager.cc:1434] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.083s	user 0.008s	sys 0.000s
I20260709 12:24:13.059181  5752 ts_tablet_manager.cc:1434] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 12:24:13.060912  5752 tablet_bootstrap.cc:492] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:13.064338  5751 tablet_bootstrap.cc:492] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:13.070859  5752 tablet_bootstrap.cc:654] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.073127  5750 raft_consensus.cc:359] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.073879  5750 raft_consensus.cc:385] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.074167  5750 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.074849  5750 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.080320  5558 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.105162  6084 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.105862  6084 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.108574  5751 tablet_bootstrap.cc:654] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.108412  6009 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:13.118316  5635 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:13.111301  5456 catalog_manager.cc:5697] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:13.121568  5456 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:13.123589  5816 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:13.127761  5458 catalog_manager.cc:5697] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:13.134776  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:13.134979  6084 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:13.119596  5750 ts_tablet_manager.cc:1434] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.124s	user 0.030s	sys 0.010s
I20260709 12:24:13.136611  5750 tablet_bootstrap.cc:492] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:13.139199  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:13.139884  5706 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:13.143361  5524 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:13.149829  6083 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:13.165364  5966 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:13.166958  5752 tablet_bootstrap.cc:492] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:13.167514  5752 ts_tablet_manager.cc:1403] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.107s	user 0.008s	sys 0.001s
I20260709 12:24:13.170099  5904 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:13.175618  5904 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:13.176141  5904 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:13.203812  5904 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.218957  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:13.221735  5904 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:13.239579  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:13.247279  5752 raft_consensus.cc:359] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.250600  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:13.258484  5904 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:13.259289  5752 raft_consensus.cc:385] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.259640  5752 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.282644  5752 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.286093  5750 tablet_bootstrap.cc:654] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.287592  5751 tablet_bootstrap.cc:492] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:13.288156  5751 ts_tablet_manager.cc:1403] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.224s	user 0.035s	sys 0.026s
I20260709 12:24:13.290729  5751 raft_consensus.cc:359] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.291613  5751 raft_consensus.cc:385] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.291906  5751 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.292603  5752 ts_tablet_manager.cc:1434] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.118s	user 0.009s	sys 0.000s
I20260709 12:24:13.292790  5751 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.310801  6104 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.311411  6104 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.314050  6104 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:13.315272  6104 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.315814  6104 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.316454  5750 tablet_bootstrap.cc:492] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:13.315871  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:13.316989  5750 ts_tablet_manager.cc:1403] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.181s	user 0.010s	sys 0.020s
W20260709 12:24:13.318322  5599 leader_election.cc:343] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:13.319595  6104 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:13.319973  5750 raft_consensus.cc:359] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.320700  5750 raft_consensus.cc:385] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.320959  5750 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.321703  5750 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.323192  5751 ts_tablet_manager.cc:1434] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.035s	user 0.014s	sys 0.008s
I20260709 12:24:13.323829  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:13.324427  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:13.325666  5751 tablet_bootstrap.cc:492] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:13.326406  5750 ts_tablet_manager.cc:1434] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:24:13.327535  5750 tablet_bootstrap.cc:492] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:13.327900  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:13.327329  5602 leader_election.cc:343] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:13.328740  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:24:13.329651  5599 leader_election.cc:343] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:13.335088  5966 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:13.334440  5750 tablet_bootstrap.cc:654] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
W20260709 12:24:13.336284  5602 leader_election.cc:343] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:13.336818  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:13.338394  5752 tablet_bootstrap.cc:492] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:13.338640  6104 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:13.341518  5751 tablet_bootstrap.cc:654] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.344475  5752 tablet_bootstrap.cc:654] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.346604  6018 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.347084  6018 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.349097  6018 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:13.350664  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:13.351257  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:13.351352  5559 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:13.351912  5631 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:13.352602  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:13.353417  6016 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:13.353775  6016 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.354095  6016 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.356760  5750 tablet_bootstrap.cc:492] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:13.357800  5750 ts_tablet_manager.cc:1403] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.031s	user 0.013s	sys 0.003s
I20260709 12:24:13.360522  5750 raft_consensus.cc:359] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.361333  5750 raft_consensus.cc:385] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.361675  5750 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.362474  5750 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.361872  6016 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.367234  6016 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:13.369421  5750 ts_tablet_manager.cc:1434] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:24:13.369493  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:13.370133  5751 tablet_bootstrap.cc:492] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:13.370291  5559 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.371086  5750 tablet_bootstrap.cc:492] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:13.371225  5751 ts_tablet_manager.cc:1403] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.046s	user 0.015s	sys 0.009s
I20260709 12:24:13.374629  5751 raft_consensus.cc:359] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.375303  5751 raft_consensus.cc:385] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.375582  5751 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.376163  5751 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.378180  5751 ts_tablet_manager.cc:1434] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:24:13.378532  5750 tablet_bootstrap.cc:654] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.379853  5752 tablet_bootstrap.cc:492] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:13.380373  5752 ts_tablet_manager.cc:1403] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.042s	user 0.011s	sys 0.001s
I20260709 12:24:13.379096  5751 tablet_bootstrap.cc:492] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:13.387074  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:13.387665  5751 tablet_bootstrap.cc:654] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.387686  5631 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.399272  5751 tablet_bootstrap.cc:492] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:13.399840  5751 ts_tablet_manager.cc:1403] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.003s
I20260709 12:24:13.402565  5751 raft_consensus.cc:359] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.403291  5751 raft_consensus.cc:385] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.403609  5751 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.404497  5706 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.404314  5751 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.406497  5751 ts_tablet_manager.cc:1434] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.003s
I20260709 12:24:13.407526  5751 tablet_bootstrap.cc:492] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:13.414412  5751 tablet_bootstrap.cc:654] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.424155  6084 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:13.427630  5559 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:13.454166  5904 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:13.455955  6084 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.456629  6084 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.459187  6084 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:13.462044  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:13.462961  5635 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 1: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:13.467589  6084 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.467579  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:13.469684  6018 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:13.470212  6018 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:13.470850  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:13.471159  5904 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:13.471083  6018 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.473958  5631 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:13.411551  5752 raft_consensus.cc:359] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.480605  5752 raft_consensus.cc:385] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.480973  5752 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.481885  5752 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.483503  5752 ts_tablet_manager.cc:1434] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.103s	user 0.005s	sys 0.001s
I20260709 12:24:13.475342  5751 tablet_bootstrap.cc:492] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:13.484139  5751 ts_tablet_manager.cc:1403] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.077s	user 0.027s	sys 0.036s
I20260709 12:24:13.488265  5752 tablet_bootstrap.cc:492] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:13.487282  5751 raft_consensus.cc:359] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.468400  6084 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.490589  5751 raft_consensus.cc:385] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.491783  5751 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.492790  6084 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:13.493537  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:13.495107  5628 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:13.494359  5751 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.496205  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:13.501003  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:13.501763  5904 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:13.502394  5904 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.502831  5904 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.503793  5849 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.503705  5751 ts_tablet_manager.cc:1434] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260709 12:24:13.506268  5751 tablet_bootstrap.cc:492] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:13.506131  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:13.507864  5524 leader_election.cc:343] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:13.508911  5750 tablet_bootstrap.cc:492] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:13.509531  5750 ts_tablet_manager.cc:1403] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.139s	user 0.032s	sys 0.043s
I20260709 12:24:13.512082  5750 raft_consensus.cc:359] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.512871  5750 raft_consensus.cc:385] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.513171  5750 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.515587  5849 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:13.513887  5750 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.519343  5750 ts_tablet_manager.cc:1434] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:24:13.522378  5752 tablet_bootstrap.cc:654] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.523140  5628 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.527688  5750 tablet_bootstrap.cc:492] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:13.539186  5904 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.542145  5989 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:13.543568  5752 tablet_bootstrap.cc:492] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:13.554054  5966 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.555178  5966 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.558753  5750 tablet_bootstrap.cc:654] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.562889  5904 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:13.573280  5966 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:13.574180  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:13.585944  5989 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:13.575500  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:13.580749  6018 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:13.586262  5635 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:24:13.587810  5599 leader_election.cc:343] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:13.591121  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:13.593005  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:13.594241  5458 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:13.595208  6016 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:13.574757  5752 ts_tablet_manager.cc:1403] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.087s	user 0.016s	sys 0.016s
I20260709 12:24:13.599584  5751 tablet_bootstrap.cc:654] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.599797  5752 raft_consensus.cc:359] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.605595  5752 raft_consensus.cc:385] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:24:13.605810  5524 leader_election.cc:343] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:13.606421  5752 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.604174  5635 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:13.608332  5752 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.609045  5849 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:13.614235  5752 ts_tablet_manager.cc:1434] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.000s
I20260709 12:24:13.611624  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:13.615141  5752 tablet_bootstrap.cc:492] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:13.616026  5849 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:13.616726  5849 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:13.617539  5849 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.621495  5752 tablet_bootstrap.cc:654] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.617619  5750 tablet_bootstrap.cc:492] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:13.632424  5750 ts_tablet_manager.cc:1403] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.105s	user 0.014s	sys 0.000s
W20260709 12:24:13.605073  5602 leader_election.cc:343] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:13.634191  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:13.635682  5791 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:13.636519  6018 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.637243  6018 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.641983  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:13.644150  5676 leader_election.cc:343] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:13.645480  5750 raft_consensus.cc:359] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.646497  5750 raft_consensus.cc:385] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.646914  5750 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.653412  5750 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.661266  5750 ts_tablet_manager.cc:1434] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.000s
I20260709 12:24:13.662295  5750 tablet_bootstrap.cc:492] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:13.666350  5752 tablet_bootstrap.cc:492] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:13.667027  5752 ts_tablet_manager.cc:1403] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.052s	user 0.020s	sys 0.024s
I20260709 12:24:13.670018  5752 raft_consensus.cc:359] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.670734  5752 raft_consensus.cc:385] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.671027  5752 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.671700  5752 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.673370  5750 tablet_bootstrap.cc:654] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.674106  5751 tablet_bootstrap.cc:492] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:13.677793  5751 ts_tablet_manager.cc:1403] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.172s	user 0.014s	sys 0.000s
I20260709 12:24:13.678896  5989 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.679399  5989 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.680505  5751 raft_consensus.cc:359] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.681229  5751 raft_consensus.cc:385] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.681483  5751 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.682194  5751 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.683928  5989 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:13.686203  5791 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.686337  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:13.686736  5791 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.687119  5555 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:13.689213  5791 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:13.690492  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:13.691570  5752 ts_tablet_manager.cc:1434] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 12:24:13.684181  5751 ts_tablet_manager.cc:1434] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
W20260709 12:24:13.692862  5599 leader_election.cc:343] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:13.697449  5676 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:13.698550  5752 tablet_bootstrap.cc:492] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:13.699333  6016 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:13.699716  6016 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.700042  6016 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.706847  5750 tablet_bootstrap.cc:492] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:13.707346  5750 ts_tablet_manager.cc:1403] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.045s	user 0.010s	sys 0.000s
I20260709 12:24:13.712810  5750 raft_consensus.cc:359] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.713706  5750 raft_consensus.cc:385] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.714043  5750 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.645815  6018 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:13.714829  5750 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.717068  5752 tablet_bootstrap.cc:654] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.720870  5750 ts_tablet_manager.cc:1434] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.002s
I20260709 12:24:13.721874  5750 tablet_bootstrap.cc:492] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:13.736928  5751 tablet_bootstrap.cc:492] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:13.732627  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:13.744563  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:13.745263  5636 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:13.747241  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:13.747802  5636 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:13.748234  5751 tablet_bootstrap.cc:654] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.758495  5458 catalog_manager.cc:5697] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:13.762431  5750 tablet_bootstrap.cc:654] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.777675  5750 tablet_bootstrap.cc:492] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:13.778136  5750 ts_tablet_manager.cc:1403] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.056s	user 0.009s	sys 0.002s
I20260709 12:24:13.786404  5750 raft_consensus.cc:359] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.787528  5750 raft_consensus.cc:385] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.787923  5750 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.788841  5750 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:13.792426  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:24:13.803100  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:13.804699  6018 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:13.805186  6018 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.805581  6018 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.816200  5750 ts_tablet_manager.cc:1434] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.038s	user 0.009s	sys 0.000s
I20260709 12:24:13.819571  6018 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.820432  6016 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.823343  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:13.823920  5555 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.830341  5750 tablet_bootstrap.cc:492] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:13.839974  6016 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:13.844484  5560 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:13.847080  6018 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:13.848317  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:13.848932  5558 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:24:13.855836  5602 leader_election.cc:343] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:13.856285  5602 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:13.860507  5791 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:13.861683  5558 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:13.862597  5849 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.862744  5752 tablet_bootstrap.cc:492] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:13.862825  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:13.863335  5752 ts_tablet_manager.cc:1403] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.167s	user 0.033s	sys 0.018s
I20260709 12:24:13.863617  6117 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:13.864113  6117 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:13.865060  5791 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:13.865447  5555 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:13.865484  6117 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.866173  6104 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:13.863201  5849 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.866549  6104 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:13.866763  5676 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:13.867444  5989 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:13.867369  6104 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.868157  5989 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:24:13.873108  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.874581  5849 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:13.875602  6104 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:13.877465  5750 tablet_bootstrap.cc:654] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:13.878093  5751 tablet_bootstrap.cc:492] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:13.878582  5751 ts_tablet_manager.cc:1403] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.142s	user 0.031s	sys 0.016s
I20260709 12:24:13.879040  5989 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.875304  5752 raft_consensus.cc:359] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.882807  5752 raft_consensus.cc:385] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.883147  5752 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.888907  5752 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.889096  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:13.892807  5631 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:13.895414  5750 tablet_bootstrap.cc:492] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:13.895912  5750 ts_tablet_manager.cc:1403] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.066s	user 0.015s	sys 0.000s
I20260709 12:24:13.896919  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:13.897502  5631 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.897753  5751 raft_consensus.cc:359] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.891328  5752 ts_tablet_manager.cc:1434] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.001s
I20260709 12:24:13.898447  5751 raft_consensus.cc:385] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:13.899489  5751 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.899691  5752 tablet_bootstrap.cc:492] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:13.898808  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:13.912089  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:13.918032  6084 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:13.919965  6084 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.925179  6084 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.889412  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:13.917055  6134 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.927383  6134 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:13.932399  5524 leader_election.cc:343] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:13.928918  5751 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:13.918393  5628 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:13.946142  5751 ts_tablet_manager.cc:1434] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.067s	user 0.009s	sys 0.007s
I20260709 12:24:13.948513  6134 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:13.991962  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:13.947237  5750 raft_consensus.cc:359] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:14.020767  5676 leader_election.cc:343] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:14.021665  5750 raft_consensus.cc:385] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:14.022197  5750 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:13.947265  5751 tablet_bootstrap.cc:492] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:14.023654  5750 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.007912  5628 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:14.038218  5752 tablet_bootstrap.cc:654] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:14.069906  5904 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.081419  6018 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.081779  5904 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.082095  6018 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.084694  5904 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:14.086838  5989 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:14.088722  5750 ts_tablet_manager.cc:1434] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.192s	user 0.023s	sys 0.018s
I20260709 12:24:14.095155  6018 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:14.066936  6084 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.106408  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:14.104636  5751 tablet_bootstrap.cc:654] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:14.106639  6084 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:14.110788  5706 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:24:14.113677  5524 leader_election.cc:343] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:14.118152  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:14.119942  5628 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:14.117223  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:14.126586  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:14.127368  5636 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:24:14.129127  5524 leader_election.cc:343] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:14.129215  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:14.130297  5816 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:14.130704  5816 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.131039  5816 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:14.133314  6003 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:14.139608  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:14.140722  5634 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:14.147716  6018 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.148300  6018 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.150678  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:14.152503  5676 leader_election.cc:343] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:14.155742  6018 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:14.156960  5631 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:14.158910  5750 tablet_bootstrap.cc:492] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:14.165870  5816 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.182341  5457 catalog_manager.cc:5697] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:14.197001  5816 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:14.184158  5636 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:14.231312  5457 catalog_manager.cc:5697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:14.470917  6083 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:24:14.470983  6016 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:14.478340  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:14.479130  5636 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:14.481950  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:14.484632  6134 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:14.485049  6134 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.485466  6134 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:14.485087  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:14.498958  5904 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.503043  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:14.504693  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:14.506485  5751 tablet_bootstrap.cc:492] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:14.506668  6117 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:14.507041  5751 ts_tablet_manager.cc:1403] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.560s	user 0.022s	sys 0.020s
I20260709 12:24:14.508800  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:14.511451  5751 raft_consensus.cc:359] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.512228  5751 raft_consensus.cc:385] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:14.512609  5751 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:14.513829  5751 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.514237  5636 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:14.514833  6083 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:14.515403  6083 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:14.516336  5751 ts_tablet_manager.cc:1434] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:24:14.516503  6083 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.519366  5751 tablet_bootstrap.cc:492] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:14.504297  5904 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.531564  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:14.534039  5750 tablet_bootstrap.cc:654] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:14.534197  6134 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.544487  6140 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.545759  6140 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.546587  6134 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:14.549870  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:14.550428  6140 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:14.556675  5966 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.558146  5966 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.562815  5751 tablet_bootstrap.cc:654] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:14.562711  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:14.564442  6138 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:14.565014  6138 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:14.565954  6138 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:14.577988  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 72289.4us
I20260709 12:24:14.590085  5904 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:14.604862  5966 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:14.605461  5599 leader_election.cc:343] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:14.614948  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:24:14.618009  5676 leader_election.cc:343] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:14.674566  5752 tablet_bootstrap.cc:492] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:14.691998  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:14.689973  5752 ts_tablet_manager.cc:1403] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.791s	user 0.020s	sys 0.014s
W20260709 12:24:14.710089  5675 leader_election.cc:343] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:14.710697  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:14.697924  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:14.712330  6134 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:14.712633  5631 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:14.713146  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:14.713835  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:14.713935  5675 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:14.714807  6134 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:14.715910  5456 catalog_manager.cc:5697] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:14.716718  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:14.717278  5554 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
I20260709 12:24:14.720956  6145 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:14.722443  6145 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:24:14.727923  5524 leader_election.cc:343] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:14.737757  5752 raft_consensus.cc:359] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.740703  5752 raft_consensus.cc:385] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:14.751891  6018 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:14.755399  5752 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:14.754503  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:14.756100  5752 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.758170  5752 ts_tablet_manager.cc:1434] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.063s	user 0.006s	sys 0.000s
I20260709 12:24:14.758315  5904 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:14.758690  5904 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.759025  5904 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:14.759924  5752 tablet_bootstrap.cc:492] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:14.778036  5904 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:14.779502  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60462.2us
W20260709 12:24:14.782823  5524 leader_election.cc:343] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:14.832664  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:14.833838  6157 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:24:14.835399  5599 leader_election.cc:343] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:14.834514  6157 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:14.845600  5524 leader_election.cc:343] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:14.847154  5750 tablet_bootstrap.cc:492] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:14.848090  5750 ts_tablet_manager.cc:1403] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.689s	user 0.017s	sys 0.024s
I20260709 12:24:14.847977  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:24:14.847569  5602 leader_election.cc:343] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:14.849258  5602 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:14.862864  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:14.848668  5628 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:14.872130  6161 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:14.890283  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:14.891392  5557 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 2: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:24:14.895247  6157 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:14.896528  5904 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:14.894032  5750 raft_consensus.cc:359] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.899214  5750 raft_consensus.cc:385] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:14.899513  5750 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:14.881852  5752 tablet_bootstrap.cc:654] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:14.901952  6016 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.902498  6016 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.905368  6016 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:14.906581  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:14.908264  5751 tablet_bootstrap.cc:492] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:14.908226  5750 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.908869  5751 ts_tablet_manager.cc:1403] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.390s	user 0.034s	sys 0.002s
I20260709 12:24:14.909143  5904 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.909752  5904 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.915848  5628 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:14.916648  5750 ts_tablet_manager.cc:1434] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.068s	user 0.006s	sys 0.002s
I20260709 12:24:14.916697  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:14.926440  6139 raft_consensus.cc:2764] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:24:14.927790  5631 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:14.931077  5751 raft_consensus.cc:359] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.932013  5751 raft_consensus.cc:385] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:14.932292  5751 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:14.932444  5750 tablet_bootstrap.cc:492] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:14.933425  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:14.934851  6084 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:14.935377  6084 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:14.936868  6084 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.937603  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:14.938369  5629 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:14.951138  5751 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:14.951356  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:14.965370  5751 ts_tablet_manager.cc:1434] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.056s	user 0.008s	sys 0.002s
I20260709 12:24:14.966509  5751 tablet_bootstrap.cc:492] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:14.966594  5629 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 1: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:24:14.986586  5675 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:14.993217  6154 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:14.993707  6154 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:14.994078  6154 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:15.010242  5675 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:15.022750  5750 tablet_bootstrap.cc:654] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.013586  6157 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:24:15.026551  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:24:15.035472  5456 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:15.046962  5631 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:15.054456  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:15.052778  6157 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:15.055733  5904 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.058131  5751 tablet_bootstrap.cc:654] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.062990  5752 tablet_bootstrap.cc:492] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:15.063531  5752 ts_tablet_manager.cc:1403] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.304s	user 0.025s	sys 0.005s
I20260709 12:24:15.063783  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:15.072587  5526 leader_election.cc:343] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:15.069190  6169 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:15.073496  6138 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:15.078284  6138 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:15.080659  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:15.069478  6157 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.081950  6169 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:15.082254  6157 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.082116  6138 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:15.083315  5709 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:24:15.087257  5524 leader_election.cc:343] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:15.088591  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:15.089993  6003 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:15.091979  5752 raft_consensus.cc:359] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.092950  5752 raft_consensus.cc:385] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.101166  5752 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.102010  5752 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.095659  6169 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.085923  6154 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.122068  6003 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:24:15.137272  5752 ts_tablet_manager.cc:1434] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.073s	user 0.008s	sys 0.000s
I20260709 12:24:15.146075  5752 tablet_bootstrap.cc:492] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:15.239483  5752 tablet_bootstrap.cc:654] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.243021  5750 tablet_bootstrap.cc:492] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:15.243498  5750 ts_tablet_manager.cc:1403] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.312s	user 0.013s	sys 0.004s
I20260709 12:24:15.244793  5752 tablet_bootstrap.cc:492] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:15.245272  5752 ts_tablet_manager.cc:1403] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.100s	user 0.017s	sys 0.006s
I20260709 12:24:15.245826  6157 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:15.247233  6154 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:15.255682  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:15.263343  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:15.264492  5554 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 2: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:24:15.273520  5752 raft_consensus.cc:359] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.274245  5752 raft_consensus.cc:385] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.274511  5752 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.276391  5750 raft_consensus.cc:359] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.279007  5750 raft_consensus.cc:385] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.279354  5750 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.276989  5751 tablet_bootstrap.cc:492] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:15.280478  5751 ts_tablet_manager.cc:1403] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.314s	user 0.019s	sys 0.004s
I20260709 12:24:15.280164  5750 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.282434  5750 ts_tablet_manager.cc:1434] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.039s	user 0.007s	sys 0.000s
I20260709 12:24:15.282341  6140 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.282964  6140 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.284744  6140 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.277380  5752 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.286013  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:15.287189  5752 ts_tablet_manager.cc:1434] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.042s	user 0.007s	sys 0.000s
I20260709 12:24:15.287678  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:15.288566  5752 tablet_bootstrap.cc:492] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
W20260709 12:24:15.289850  5602 leader_election.cc:343] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:24:15.291357  5599 leader_election.cc:343] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:15.291759  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:15.292843  5966 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:15.293416  5751 raft_consensus.cc:359] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.294142  5751 raft_consensus.cc:385] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.294375  5751 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.294876  5751 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.296838  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:15.300597  5752 tablet_bootstrap.cc:654] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.297046  5751 ts_tablet_manager.cc:1434] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260709 12:24:15.303995  5751 tablet_bootstrap.cc:492] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:15.306466  5750 tablet_bootstrap.cc:492] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:15.307562  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:15.312755  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:15.335002  6018 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:15.342595  5750 tablet_bootstrap.cc:654] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.348804  5631 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:15.350975  5675 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:15.351846  6018 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:15.352439  6018 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:15.353401  6018 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.355337  5752 tablet_bootstrap.cc:492] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:15.355715  5750 tablet_bootstrap.cc:492] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:15.358184  5750 ts_tablet_manager.cc:1403] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.052s	user 0.007s	sys 0.004s
I20260709 12:24:15.358356  5751 tablet_bootstrap.cc:654] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.355787  5752 ts_tablet_manager.cc:1403] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.068s	user 0.023s	sys 0.010s
I20260709 12:24:15.364300  5750 raft_consensus.cc:359] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.373116  5750 raft_consensus.cc:385] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.373443  5750 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.374085  5750 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.374882  6018 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.357731  5456 catalog_manager.cc:5697] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:15.375419  6018 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.366250  5752 raft_consensus.cc:359] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.376587  5752 raft_consensus.cc:385] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.376852  5752 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.377461  5752 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.377987  6018 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:15.375676  5750 ts_tablet_manager.cc:1434] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.000s
I20260709 12:24:15.379776  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:15.380271  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:15.380432  5631 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:15.380806  5750 tablet_bootstrap.cc:492] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:15.381001  5555 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:15.382761  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:15.379776  5752 ts_tablet_manager.cc:1434] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.001s
I20260709 12:24:15.384313  5752 tablet_bootstrap.cc:492] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Bootstrap starting.
I20260709 12:24:15.385063  6169 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:15.385468  6169 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.385823  6169 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.387583  5750 tablet_bootstrap.cc:654] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.390781  6140 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.391230  6140 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.394168  6140 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.396833  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:15.396937  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:15.397888  5709 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 0.
W20260709 12:24:15.398720  5602 leader_election.cc:343] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:15.399121  5750 tablet_bootstrap.cc:492] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:15.399235  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:15.399577  5750 ts_tablet_manager.cc:1403] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.019s	user 0.010s	sys 0.006s
I20260709 12:24:15.400581  6140 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:15.400969  6140 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.401192  5751 tablet_bootstrap.cc:492] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:15.401386  6140 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.401680  5751 ts_tablet_manager.cc:1403] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.098s	user 0.011s	sys 0.003s
I20260709 12:24:15.402248  5750 raft_consensus.cc:359] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.402870  5750 raft_consensus.cc:385] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.403162  5750 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.403761  5750 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.410817  5750 ts_tablet_manager.cc:1434] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.003s
I20260709 12:24:15.410817  5751 raft_consensus.cc:359] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.412170  5751 raft_consensus.cc:385] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.404405  6169 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.412549  5751 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.413374  5751 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.414938  6169 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:15.415735  5751 ts_tablet_manager.cc:1434] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.014s	user 0.001s	sys 0.005s
I20260709 12:24:15.416320  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:15.416831  5751 tablet_bootstrap.cc:492] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:15.416999  5631 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.419454  6140 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.420614  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:15.421232  5555 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.421561  6140 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.422062  5750 tablet_bootstrap.cc:492] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:15.422180  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:24:15.423903  5602 leader_election.cc:343] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:15.426232  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:15.426846  5709 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.430197  5458 catalog_manager.cc:5697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:15.434755  5750 tablet_bootstrap.cc:654] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.435501  5555 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:15.435405  5631 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:15.437080  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:15.438436  6018 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:15.438942  6018 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:15.439806  5709 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:15.439924  6018 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.440945  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:15.441285  5751 tablet_bootstrap.cc:654] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.441957  6140 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:15.442175  5752 tablet_bootstrap.cc:654] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.444379  6140 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:15.445377  6148 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.445267  6140 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.445865  6148 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.453001  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:15.454586  5599 leader_election.cc:343] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:15.455595  6148 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.467046  5709 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.470538  5849 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.471182  5849 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.473801  5849 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.474710  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:15.476018  5631 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 1: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:15.480486  5849 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.481243  5849 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.484244  5849 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.484654  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:15.485590  5631 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:15.465406  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:15.489503  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:15.495564  5849 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:15.557650  5752 tablet_bootstrap.cc:492] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: No bootstrap required, opened a new log
I20260709 12:24:15.588232  5849 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.558120  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:15.562568  5751 tablet_bootstrap.cc:492] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:15.588665  5752 ts_tablet_manager.cc:1403] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent bootstrapping tablet: real 0.205s	user 0.010s	sys 0.020s
I20260709 12:24:15.565841  5750 tablet_bootstrap.cc:492] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:15.568969  6138 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.589426  5750 ts_tablet_manager.cc:1403] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.168s	user 0.016s	sys 0.024s
I20260709 12:24:15.589474  6138 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.591645  6138 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.574072  6169 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.574865  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:15.592549  6169 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.592682  5750 raft_consensus.cc:359] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.593338  5750 raft_consensus.cc:385] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.593603  5750 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.594306  6169 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:15.594195  5750 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.588675  5849 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:15.591693  5752 raft_consensus.cc:359] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.595467  5752 raft_consensus.cc:385] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.595844  5752 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.596340  5751 ts_tablet_manager.cc:1403] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.180s	user 0.024s	sys 0.006s
I20260709 12:24:15.596532  5752 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.598091  5752 ts_tablet_manager.cc:1434] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.003s
I20260709 12:24:15.598982  5751 raft_consensus.cc:359] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.599630  5751 raft_consensus.cc:385] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.599875  5751 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.600543  5751 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.608626  6138 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:15.609998  5456 catalog_manager.cc:5697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:15.613168  5751 ts_tablet_manager.cc:1434] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.000s
I20260709 12:24:15.614209  5751 tablet_bootstrap.cc:492] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:15.620452  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:15.634878  5849 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.635653  5635 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:15.621487  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:15.620757  5750 ts_tablet_manager.cc:1434] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.031s	user 0.006s	sys 0.001s
I20260709 12:24:15.634167  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:15.637380  5849 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.637727  5750 tablet_bootstrap.cc:492] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
W20260709 12:24:15.621806  5602 leader_election.cc:343] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:15.638500  5602 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:15.620013  5751 tablet_bootstrap.cc:654] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.651180  5458 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 1 to 2, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 2 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:15.645251  5750 tablet_bootstrap.cc:654] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.655979  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:15.657126  6138 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.656013  5458 catalog_manager.cc:5697] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:15.662257  5751 tablet_bootstrap.cc:492] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:15.662757  5751 ts_tablet_manager.cc:1403] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.049s	user 0.005s	sys 0.006s
I20260709 12:24:15.662899  6083 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:15.665488  5751 raft_consensus.cc:359] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.666504  6138 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:15.666793  5751 raft_consensus.cc:385] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.667196  5751 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.657390  5849 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.667899  5849 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.669742  5849 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.670471  5750 tablet_bootstrap.cc:492] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:15.670940  5750 ts_tablet_manager.cc:1403] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.034s	user 0.007s	sys 0.006s
I20260709 12:24:15.671635  6140 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:15.667966  5751 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:15.672613  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 50840.5us
I20260709 12:24:15.676306  5751 ts_tablet_manager.cc:1434] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:24:15.677385  5751 tablet_bootstrap.cc:492] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:15.678303  6018 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.678761  6018 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.685381  6018 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:15.685674  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:15.686309  5631 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:15.686790  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:15.673507  5524 leader_election.cc:343] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:15.673995  5750 raft_consensus.cc:359] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.687583  5750 raft_consensus.cc:385] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.687873  5750 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
W20260709 12:24:15.688036  5676 leader_election.cc:343] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:24:15.688411  5750 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.689949  5751 tablet_bootstrap.cc:654] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.690107  5750 ts_tablet_manager.cc:1434] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.019s	user 0.004s	sys 0.000s
I20260709 12:24:15.691022  5750 tablet_bootstrap.cc:492] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Bootstrap starting.
I20260709 12:24:15.696441  5750 tablet_bootstrap.cc:654] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.700136  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:15.700935  5751 tablet_bootstrap.cc:492] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:15.701365  5751 ts_tablet_manager.cc:1403] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.024s	user 0.014s	sys 0.004s
I20260709 12:24:15.702309  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:15.703580  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:15.704412  5708 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:15.704538  5751 raft_consensus.cc:359] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.705271  5751 raft_consensus.cc:385] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.705535  5751 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.706166  5751 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.705489  6084 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:15.708000  6084 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.708384  6084 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.708936  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:15.709434  5635 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:15.712009  6117 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:15.716735  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:15.717599  5630 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:15.720219  5635 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:24:15.724865  5751 ts_tablet_manager.cc:1434] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.000s
I20260709 12:24:15.725811  5751 tablet_bootstrap.cc:492] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Bootstrap starting.
I20260709 12:24:15.728204  5750 tablet_bootstrap.cc:492] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: No bootstrap required, opened a new log
I20260709 12:24:15.723311  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:15.728847  5750 ts_tablet_manager.cc:1403] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Time spent bootstrapping tablet: real 0.038s	user 0.014s	sys 0.009s
I20260709 12:24:15.730388  5849 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:15.730572  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:15.730786  5849 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.731103  5849 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.731894  5751 tablet_bootstrap.cc:654] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Neither blocks nor log segments found. Creating new log.
I20260709 12:24:15.731755  5750 raft_consensus.cc:359] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.732641  5750 raft_consensus.cc:385] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.733011  5750 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.733752  5816 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:15.733724  5750 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.734290  5816 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:15.735915  5816 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.738606  5750 ts_tablet_manager.cc:1434] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:24:15.742267  6169 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.742923  6169 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.744462  5751 tablet_bootstrap.cc:492] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: No bootstrap required, opened a new log
I20260709 12:24:15.744902  5751 ts_tablet_manager.cc:1403] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent bootstrapping tablet: real 0.019s	user 0.010s	sys 0.007s
I20260709 12:24:15.745379  6169 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:15.746250  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:15.747352  5751 raft_consensus.cc:359] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.747862  5751 raft_consensus.cc:385] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:24:15.748111  5751 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Initialized, Role: FOLLOWER
I20260709 12:24:15.748740  5751 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.750463  5751 ts_tablet_manager.cc:1434] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:24:15.750919  5849 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.703595  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:15.757253  5631 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 0.
I20260709 12:24:15.758378  6084 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.759739  5557 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 1: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:24:15.766372  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:15.767096  5705 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.767081  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:15.767889  5632 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.766321  6084 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.774843  5849 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.777531  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:15.779615  5708 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.788277  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:15.786755  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:15.789160  6018 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:15.789615  6018 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:15.789945  6018 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:15.791230  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:15.793220  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:15.793838  5628 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.794427  6117 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:15.794828  6117 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.795156  6117 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.796602  6018 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.799248  6018 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:15.804517  5705 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:15.805783  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:15.810702  6154 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:15.811198  5632 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:15.812049  6117 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.812340  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:15.813124  6198 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:15.813814  6198 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:15.814736  6198 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.814097  6117 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:15.796207  5708 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:15.817324  5628 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:15.820454  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:15.820266  5458 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 2 to 3, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 3 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:15.821992  6083 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:15.822482  6083 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:15.823282  6083 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.823724  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:15.818424  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:15.826998  5553 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.826180  6154 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:15.835405  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:15.836103  5631 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.836804  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:15.847381  6083 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.848371  5631 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:15.848480  5553 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:15.848140  6083 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.850332  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:15.851969  6154 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:15.852533  6154 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:15.852803  6083 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.855033  6154 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.858036  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:15.859293  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:15.859843  6201 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:15.860020  5631 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:15.865274  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:15.866164  6117 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:15.866671  5816 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:24:15.867029  5816 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.867324  5816 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.869743  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:15.872910  6117 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:15.876750  5708 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 0.
I20260709 12:24:15.880568  6083 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.881624  6083 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.884701  5816 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.885658  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:15.887499  5816 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.889019  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:15.889597  5636 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.898106  5458 catalog_manager.cc:5697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:15.899742  5458 catalog_manager.cc:5697] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:15.900794  5555 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.905839  6083 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:15.913010  5636 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:15.914803  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:15.930253  5636 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:15.927388  6198 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:24:15.933854  6198 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:15.934826  6198 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.936828  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:15.937743  5708 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 1: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:24:15.941409  5524 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:15.905876  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:15.946236  5705 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:24:15.945762  6169 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260709 12:24:15.958847  6169 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:15.959280  6169 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:15.960060  6169 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:15.973330  5816 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:24:15.976907  6169 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:15.978467  5457 catalog_manager.cc:5697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 0 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:15.983592  5966 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:15.990710  5705 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:16.007223  5456 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 0 to 1, leader changed from <none> to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:16.022202  5966 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:16.117213  5631 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:24:16.118468  6084 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:16.135960  5705 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:24:16.136410  5705 pending_rounds.cc:85] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 1
I20260709 12:24:16.136648  5705 pending_rounds.cc:107] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:24:16.136917  5705 raft_consensus.cc:2889] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:16.138753  6138 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:16.214246  5705 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:16.216104  5816 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:16.217154  6138 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:16.217540  6138 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:16.218253  6138 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:16.219313  6138 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:16.229851  6138 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:16.253829  5816 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:16.268280  6169 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:16.283219  6117 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:16.309401  5816 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:16.320202  6083 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:24:16.402559  6198 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:16.423550  5849 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:24:17.257761  6148 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:17.258466  6148 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:17.261420  6148 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:17.262287  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:17.262964  5555 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:17.265312  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:17.265528  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:17.266162  5708 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 2: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:17.266389  6148 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:17.266770  6148 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:17.267164  6148 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:17.274387  6148 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:17.276871  6148 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:17.277348  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:17.278167  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:17.278892  5708 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 2: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:17.284077  5555 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:17.285600  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:17.286418  6148 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:17.287179  6148 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:17.287981  6148 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:17.298941  5458 catalog_manager.cc:5697] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:17.695247  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:24:17.718039  6225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:24:17.718235  6226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:24:17.721369  6228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:24:17.722817  4356 server_base.cc:1061] running on GCE node
I20260709 12:24:17.724298  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:24:17.724668  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:24:17.724939  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783599857724918 us; error 0 us; skew 500 ppm
I20260709 12:24:17.725630  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:24:17.731235  4356 webserver.cc:533] Webserver started at http://127.4.65.4:36807/ using document root <none> and password file <none>
I20260709 12:24:17.732093  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:24:17.732507  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:24:17.733131  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:24:17.734663  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-3-root/instance:
uuid: "38736cc545eb4389b220a3bd21d3b466"
format_stamp: "Formatted at 2026-07-09 12:24:17 on dist-test-slave-l1c9"
I20260709 12:24:17.741825  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.002s	sys 0.006s
I20260709 12:24:17.747733  6233 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:24:17.748850  4356 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.001s	sys 0.001s
I20260709 12:24:17.749198  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-3-root
uuid: "38736cc545eb4389b220a3bd21d3b466"
format_stamp: "Formatted at 2026-07-09 12:24:17 on dist-test-slave-l1c9"
I20260709 12:24:17.749567  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:24:17.761600  5561 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:17.762915  6139 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:17.781759  5708 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:17.783252  6139 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:17.788653  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:24:17.792661  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:24:17.850986  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:24:17.857851  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:24:17.858141  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:24:17.858485  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:24:17.858723  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:24:17.930629  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.4:46059
I20260709 12:24:17.931074  6304 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.4:46059 every 8 connection(s)
I20260709 12:24:17.948949  4356 auto_leader_rebalancer-test.cc:439] add a tserver: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:17.960552  6305 heartbeater.cc:344] Connected to a master server at 127.4.65.62:38921
I20260709 12:24:17.961120  6305 heartbeater.cc:461] Registering TS with master...
I20260709 12:24:17.962347  6305 heartbeater.cc:507] Master 127.4.65.62:38921 requested a full tablet report, sending...
I20260709 12:24:17.964480  5458 ts_manager.cc:194] Registered new tserver with Master: 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:17.966019  5458 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38466
I20260709 12:24:18.223170  5636 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:18.231068  5705 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:18.232689  6139 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:18.238287  5555 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:18.241931  6148 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:18.252343  6315 raft_consensus.cc:2955] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:18.255290  5708 raft_consensus.cc:2955] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
W20260709 12:24:18.258406  5601 consensus_peers.cc:597] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 378854044c0e42f4b0a737cdbdbcc464. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:18.271240  5561 raft_consensus.cc:2955] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:18.292200  5458 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:18.313730  5555 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:18.326454  5708 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:18.329120  5628 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:18.329005  6245 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:18.330636  5849 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:18.338709  5849 raft_consensus.cc:2955] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:18.340588  5628 raft_consensus.cc:2955] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:18.346568  5708 raft_consensus.cc:2955] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
W20260709 12:24:18.369436  5526 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d439f3f9c7c40288efc981cec05e7ab. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:18.381883  5628 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:18.407575  5456 catalog_manager.cc:5697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:18.418216  5706 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:24:18.418612  5601 consensus_peers.cc:597] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11fc89db1c9243b2af65c665c04ba84e. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:18.420058  5555 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:18.420567  6139 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:18.423044  6139 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:18.431423  6321 raft_consensus.cc:2955] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:18.433321  5555 raft_consensus.cc:2955] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:18.450091  5708 raft_consensus.cc:2955] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:18.458443  5457 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:18.475188  5708 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:18.488867  5631 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:18.490304  6169 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:18.491215  5555 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:18.493718  6317 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:24:18.503328  5675 consensus_peers.cc:597] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b585af92ddc4f4fa5bada0f9b198495. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:18.506582  6317 raft_consensus.cc:2955] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:18.508580  5561 raft_consensus.cc:2955] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:18.509301  5631 raft_consensus.cc:2955] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:18.528790  5458 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:18.786530  6337 ts_tablet_manager.cc:933] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:18.789525  6337 tablet_copy_client.cc:323] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.2:38587
I20260709 12:24:18.790421  6339 ts_tablet_manager.cc:933] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:18.797206  6339 tablet_copy_client.cc:323] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.1:43285
I20260709 12:24:18.810370  5646 tablet_copy_service.cc:140] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Received BeginTabletCopySession request for tablet 378854044c0e42f4b0a737cdbdbcc464 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:40390)
I20260709 12:24:18.811038  5646 tablet_copy_service.cc:161] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Beginning new tablet copy session on tablet 378854044c0e42f4b0a737cdbdbcc464 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:40390: session id = 38736cc545eb4389b220a3bd21d3b466-378854044c0e42f4b0a737cdbdbcc464
I20260709 12:24:18.811093  5571 tablet_copy_service.cc:140] P cea66aa15be3423880843974acdbf50f: Received BeginTabletCopySession request for tablet 4d439f3f9c7c40288efc981cec05e7ab from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:58854)
I20260709 12:24:18.811823  5571 tablet_copy_service.cc:161] P cea66aa15be3423880843974acdbf50f: Beginning new tablet copy session on tablet 4d439f3f9c7c40288efc981cec05e7ab from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:58854: session id = 38736cc545eb4389b220a3bd21d3b466-4d439f3f9c7c40288efc981cec05e7ab
I20260709 12:24:18.819612  5646 tablet_copy_source_session.cc:215] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:18.819851  5571 tablet_copy_source_session.cc:215] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:18.823863  6337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 378854044c0e42f4b0a737cdbdbcc464. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:18.825539  6339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d439f3f9c7c40288efc981cec05e7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:18.854489  6339 tablet_copy_client.cc:806] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:18.855159  6339 tablet_copy_client.cc:670] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:18.859690  6339 tablet_copy_client.cc:538] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:18.862846  6337 tablet_copy_client.cc:806] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:18.863411  6337 tablet_copy_client.cc:670] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:18.868039  6337 tablet_copy_client.cc:538] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:18.871165  6339 tablet_bootstrap.cc:492] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:18.881210  6337 tablet_bootstrap.cc:492] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:18.902077  6339 tablet_bootstrap.cc:492] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:18.903198  6339 tablet_bootstrap.cc:492] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:18.903832  6339 ts_tablet_manager.cc:1403] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.033s	user 0.019s	sys 0.013s
I20260709 12:24:18.907052  6339 raft_consensus.cc:359] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:18.907708  6339 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:18.908274  6339 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:18.911949  6337 tablet_bootstrap.cc:492] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:18.912837  6337 tablet_bootstrap.cc:492] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:18.913456  6337 ts_tablet_manager.cc:1403] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.032s	user 0.019s	sys 0.008s
I20260709 12:24:18.916628  6339 ts_tablet_manager.cc:1434] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.004s
I20260709 12:24:18.918645  6305 heartbeater.cc:499] Master 127.4.65.62:38921 was elected leader, sending a full tablet report...
I20260709 12:24:18.921753  6337 raft_consensus.cc:359] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:18.922878  6337 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:18.923301  5571 tablet_copy_service.cc:342] P cea66aa15be3423880843974acdbf50f: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-4d439f3f9c7c40288efc981cec05e7ab received from {username='slave'} at 127.0.0.1:58854
I20260709 12:24:18.923802  5571 tablet_copy_service.cc:434] P cea66aa15be3423880843974acdbf50f: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-4d439f3f9c7c40288efc981cec05e7ab on tablet 4d439f3f9c7c40288efc981cec05e7ab with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:18.923666  6337 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:18.932118  6337 ts_tablet_manager.cc:1434] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.001s
I20260709 12:24:18.935434  5646 tablet_copy_service.cc:342] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-378854044c0e42f4b0a737cdbdbcc464 received from {username='slave'} at 127.0.0.1:40390
I20260709 12:24:18.935890  5646 tablet_copy_service.cc:434] P 4a1c5e6dd8e249bf82d38f90fa37b43d: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-378854044c0e42f4b0a737cdbdbcc464 on tablet 378854044c0e42f4b0a737cdbdbcc464 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:18.937788  6339 ts_tablet_manager.cc:933] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:18.951596  6339 tablet_copy_client.cc:323] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.2:38587
I20260709 12:24:18.956921  5646 tablet_copy_service.cc:140] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Received BeginTabletCopySession request for tablet 11fc89db1c9243b2af65c665c04ba84e from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:40390)
I20260709 12:24:18.957475  5646 tablet_copy_service.cc:161] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Beginning new tablet copy session on tablet 11fc89db1c9243b2af65c665c04ba84e from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:40390: session id = 38736cc545eb4389b220a3bd21d3b466-11fc89db1c9243b2af65c665c04ba84e
I20260709 12:24:18.963182  5646 tablet_copy_source_session.cc:215] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:18.965727  6339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11fc89db1c9243b2af65c665c04ba84e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:18.977205  6339 tablet_copy_client.cc:806] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:18.977710  6339 tablet_copy_client.cc:670] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:18.981389  6339 tablet_copy_client.cc:538] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:18.990504  6339 tablet_bootstrap.cc:492] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:19.011931  6339 tablet_bootstrap.cc:492] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:19.012908  6339 tablet_bootstrap.cc:492] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:19.013562  6339 ts_tablet_manager.cc:1403] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.023s	user 0.019s	sys 0.004s
I20260709 12:24:19.016433  6339 raft_consensus.cc:359] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:19.017226  6339 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:19.017859  6339 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:19.020113  6339 ts_tablet_manager.cc:1434] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:24:19.022107  5646 tablet_copy_service.cc:342] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-11fc89db1c9243b2af65c665c04ba84e received from {username='slave'} at 127.0.0.1:40390
I20260709 12:24:19.022512  5646 tablet_copy_service.cc:434] P 4a1c5e6dd8e249bf82d38f90fa37b43d: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-11fc89db1c9243b2af65c665c04ba84e on tablet 11fc89db1c9243b2af65c665c04ba84e with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:19.099272  6339 ts_tablet_manager.cc:933] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:19.101109  6339 tablet_copy_client.cc:323] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.3:45979
I20260709 12:24:19.116883  5720 tablet_copy_service.cc:140] P f613ba929b1046d3b5aed909414a95bf: Received BeginTabletCopySession request for tablet 1b585af92ddc4f4fa5bada0f9b198495 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:42568)
I20260709 12:24:19.117498  5720 tablet_copy_service.cc:161] P f613ba929b1046d3b5aed909414a95bf: Beginning new tablet copy session on tablet 1b585af92ddc4f4fa5bada0f9b198495 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:42568: session id = 38736cc545eb4389b220a3bd21d3b466-1b585af92ddc4f4fa5bada0f9b198495
I20260709 12:24:19.124095  5720 tablet_copy_source_session.cc:215] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:19.130546  6339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b585af92ddc4f4fa5bada0f9b198495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:19.148554  6339 tablet_copy_client.cc:806] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:19.149204  6339 tablet_copy_client.cc:670] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:19.153878  6339 tablet_copy_client.cc:538] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:19.162814  6339 tablet_bootstrap.cc:492] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:19.175925  6339 tablet_bootstrap.cc:492] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:19.176862  6339 tablet_bootstrap.cc:492] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:19.177508  6339 ts_tablet_manager.cc:1403] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.015s	user 0.011s	sys 0.005s
I20260709 12:24:19.180099  6339 raft_consensus.cc:359] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:19.180881  6339 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:19.181480  6339 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:19.183808  6339 ts_tablet_manager.cc:1434] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:24:19.188413  5720 tablet_copy_service.cc:342] P f613ba929b1046d3b5aed909414a95bf: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-1b585af92ddc4f4fa5bada0f9b198495 received from {username='slave'} at 127.0.0.1:42568
I20260709 12:24:19.188896  5720 tablet_copy_service.cc:434] P f613ba929b1046d3b5aed909414a95bf: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-1b585af92ddc4f4fa5bada0f9b198495 on tablet 1b585af92ddc4f4fa5bada0f9b198495 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:19.278313  6280 raft_consensus.cc:1217] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:19.321538  6280 raft_consensus.cc:1217] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:19.443819  6280 raft_consensus.cc:1217] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:19.571424  6280 raft_consensus.cc:1217] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:19.763301  6315 raft_consensus.cc:1064] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:19.765511  6315 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:19.776432  5705 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:19.777058  6280 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:19.778587  5557 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:19.778820  6321 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:19.780403  6315 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:19.781867  6354 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:19.805514  6315 raft_consensus.cc:2955] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:19.818192  5561 raft_consensus.cc:2955] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:19.818840  5705 raft_consensus.cc:2955] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:19.821228  6280 raft_consensus.cc:2955] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:19.836082  5457 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:19.862778  5628 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:19.868425  5705 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:19.868474  6280 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:19.871268  6354 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:19.872283  6321 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:19.885505  6353 raft_consensus.cc:1064] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:19.888031  6353 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:19.887359  6354 raft_consensus.cc:2955] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:19.896077  6280 raft_consensus.cc:2955] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:19.890744  5708 raft_consensus.cc:2955] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:19.902043  5445 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 378854044c0e42f4b0a737cdbdbcc464 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:19.923699  5458 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:19.943879  6315 raft_consensus.cc:1064] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:19.946970  6315 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:19.948947  5550 tablet_service.cc:1558] Processing DeleteTablet for tablet 378854044c0e42f4b0a737cdbdbcc464 with delete_type TABLET_DATA_TOMBSTONED (TS cea66aa15be3423880843974acdbf50f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49014
I20260709 12:24:19.952955  5708 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:19.954666  6245 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:19.948163  6280 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:19.964900  6245 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:19.972791  5706 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:19.974574  6366 tablet_replica.cc:333] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:24:19.977142  6366 raft_consensus.cc:2243] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:19.974292  6354 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:19.977842  6366 raft_consensus.cc:2272] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:19.981607  6366 ts_tablet_manager.cc:1916] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:19.987053  6279 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:19.989416  6321 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:19.991032  5561 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:19.993582  6321 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:20.006469  5628 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:20.011792  6245 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:20.025910  6366 ts_tablet_manager.cc:1929] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:20.026518  6366 log.cc:1199] T 378854044c0e42f4b0a737cdbdbcc464 P cea66aa15be3423880843974acdbf50f: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/wals/378854044c0e42f4b0a737cdbdbcc464
I20260709 12:24:20.073520  6315 raft_consensus.cc:2955] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:20.075429  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 48196.2us
I20260709 12:24:20.077418  6279 raft_consensus.cc:2955] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:20.083676  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56798.4us
I20260709 12:24:20.087579  5446 catalog_manager.cc:5028] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): tablet 378854044c0e42f4b0a737cdbdbcc464 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:20.093842  5459 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.105518  5849 raft_consensus.cc:2955] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.117324  6169 raft_consensus.cc:1064] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:20.119664  6169 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:20.128896  5708 raft_consensus.cc:2955] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.134290  6280 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:20.131968  5632 raft_consensus.cc:2955] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.135830  5561 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:20.135622  6317 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:20.137943  6317 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:20.151350  5628 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:20.161664  6279 raft_consensus.cc:2955] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.166997  5705 raft_consensus.cc:2955] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.174997  5561 raft_consensus.cc:2955] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.177515  6169 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:20.186859  5458 catalog_manager.cc:5697] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:20.218969  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49627.2us
I20260709 12:24:20.229661  5632 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:20.238008  5561 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:20.238091  6280 raft_consensus.cc:2955] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.246039  6315 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:24:20.252769  5458 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.256943  6363 raft_consensus.cc:2955] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.264410  5561 raft_consensus.cc:2955] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.270957  5557 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:20.231029  6378 raft_consensus.cc:2955] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.277930  6280 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:20.280249  6353 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:20.280898  6279 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:20.291955  5708 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:20.294961  5849 raft_consensus.cc:2955] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.297814  6279 raft_consensus.cc:2955] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.293421  5557 raft_consensus.cc:2955] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.305948  5708 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:20.307993  5459 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.319154  6245 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:20.319478  6279 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260709 12:24:20.322762  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44010.6us
I20260709 12:24:20.326160  5445 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 11fc89db1c9243b2af65c665c04ba84e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:20.332841  5693 tablet_service.cc:1558] Processing DeleteTablet for tablet 11fc89db1c9243b2af65c665c04ba84e with delete_type TABLET_DATA_TOMBSTONED (TS f613ba929b1046d3b5aed909414a95bf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38456
I20260709 12:24:20.332305  5456 catalog_manager.cc:5697] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.350337  5446 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4d439f3f9c7c40288efc981cec05e7ab with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:20.363181  5631 raft_consensus.cc:2955] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.379369  6385 tablet_replica.cc:333] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:24:20.386780  6354 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:20.391356  6378 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:20.392457  5619 tablet_service.cc:1558] Processing DeleteTablet for tablet 4d439f3f9c7c40288efc981cec05e7ab with delete_type TABLET_DATA_TOMBSTONED (TS 4a1c5e6dd8e249bf82d38f90fa37b43d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35612
I20260709 12:24:20.395613  5706 raft_consensus.cc:2955] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.380851  6385 raft_consensus.cc:2243] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:20.405643  6385 raft_consensus.cc:2272] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:20.409664  6279 raft_consensus.cc:2955] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.424952  5561 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:20.427316  6381 raft_consensus.cc:2955] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.428889  6317 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:20.434386  5561 raft_consensus.cc:2955] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.447675  5443 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1b585af92ddc4f4fa5bada0f9b198495 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:20.445237  6280 raft_consensus.cc:2955] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:20.450314  6386 tablet_replica.cc:333] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:24:20.451267  6386 raft_consensus.cc:2243] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:20.452415  6386 raft_consensus.cc:2272] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:20.457895  6386 ts_tablet_manager.cc:1916] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:20.478273  6385 ts_tablet_manager.cc:1916] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:20.478421  5459 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:20.495299  5619 tablet_service.cc:1558] Processing DeleteTablet for tablet 1b585af92ddc4f4fa5bada0f9b198495 with delete_type TABLET_DATA_TOMBSTONED (TS 4a1c5e6dd8e249bf82d38f90fa37b43d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35612
I20260709 12:24:20.501317  6386 ts_tablet_manager.cc:1929] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:20.501904  6386 log.cc:1199] T 4d439f3f9c7c40288efc981cec05e7ab P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/wals/4d439f3f9c7c40288efc981cec05e7ab
I20260709 12:24:20.503911  5445 catalog_manager.cc:5028] TS 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): tablet 4d439f3f9c7c40288efc981cec05e7ab (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:20.505116  6386 tablet_replica.cc:333] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:24:20.506203  6386 raft_consensus.cc:2243] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:20.506881  6386 raft_consensus.cc:2272] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:20.519484  6386 ts_tablet_manager.cc:1916] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:20.537604  6385 ts_tablet_manager.cc:1929] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:20.538077  6385 log.cc:1199] T 11fc89db1c9243b2af65c665c04ba84e P f613ba929b1046d3b5aed909414a95bf: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root/wals/11fc89db1c9243b2af65c665c04ba84e
I20260709 12:24:20.543001  5443 catalog_manager.cc:5028] TS f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): tablet 11fc89db1c9243b2af65c665c04ba84e (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:20.546674  6386 ts_tablet_manager.cc:1929] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:20.547228  6386 log.cc:1199] T 1b585af92ddc4f4fa5bada0f9b198495 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/wals/1b585af92ddc4f4fa5bada0f9b198495
I20260709 12:24:20.549285  5445 catalog_manager.cc:5028] TS 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): tablet 1b585af92ddc4f4fa5bada0f9b198495 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:21.570755  5706 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:21.578118  5632 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:21.578847  5561 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:24:21.579103  5675 consensus_peers.cc:597] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 786a49bad9be400ea9ce584030aad446. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:21.580144  6390 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:21.581704  6317 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:21.589944  6390 raft_consensus.cc:2955] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:21.591814  5561 raft_consensus.cc:2955] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:21.593441  5632 raft_consensus.cc:2955] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:21.606000  5561 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:21.608088  5458 catalog_manager.cc:5697] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:21.614857  5526 consensus_peers.cc:597] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 607035a460944cc2b065c4d4cc59f813. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:21.617585  5628 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:21.619220  5706 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:21.619314  6245 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:21.626256  5849 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:21.640792  5849 raft_consensus.cc:2955] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:21.641911  5706 raft_consensus.cc:2955] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:21.661940  5706 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:21.682736  5459 catalog_manager.cc:5697] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:21.693929  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47087.4us
I20260709 12:24:21.701289  5628 raft_consensus.cc:2955] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
W20260709 12:24:21.716756  5675 consensus_peers.cc:597] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4f86de9f88949219896e4c020485a11. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:21.718832  5628 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:21.720021  5561 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:21.720918  6390 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:21.721869  6169 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:21.729990  6390 raft_consensus.cc:2955] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:21.732548  5561 raft_consensus.cc:2955] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:21.740240  5632 raft_consensus.cc:2955] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:21.747341  5706 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:21.752170  5458 catalog_manager.cc:5697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:21.772050  5628 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:21.772233  5554 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:24:21.785077  5675 consensus_peers.cc:597] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa294357a99443e8bd25587ba629d3de. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:21.791895  6169 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:21.793007  6317 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:21.806914  6317 raft_consensus.cc:2955] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:21.812263  5561 raft_consensus.cc:2955] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:21.812500  5628 raft_consensus.cc:2955] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:21.825868  5458 catalog_manager.cc:5697] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:21.993454  6410 ts_tablet_manager.cc:933] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:21.996129  6410 tablet_copy_client.cc:323] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.1:43285
I20260709 12:24:21.997836  5571 tablet_copy_service.cc:140] P cea66aa15be3423880843974acdbf50f: Received BeginTabletCopySession request for tablet 607035a460944cc2b065c4d4cc59f813 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:58854)
I20260709 12:24:21.998317  5571 tablet_copy_service.cc:161] P cea66aa15be3423880843974acdbf50f: Beginning new tablet copy session on tablet 607035a460944cc2b065c4d4cc59f813 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:58854: session id = 38736cc545eb4389b220a3bd21d3b466-607035a460944cc2b065c4d4cc59f813
I20260709 12:24:22.005100  5571 tablet_copy_source_session.cc:215] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:22.008621  6410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607035a460944cc2b065c4d4cc59f813. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:22.025456  6410 tablet_copy_client.cc:806] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:22.026062  6410 tablet_copy_client.cc:670] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:22.030011  6410 tablet_copy_client.cc:538] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:22.040190  6410 tablet_bootstrap.cc:492] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:22.085475  6413 ts_tablet_manager.cc:933] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:22.091768  6410 tablet_bootstrap.cc:492] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:22.092640  6410 tablet_bootstrap.cc:492] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:22.092442  6413 tablet_copy_client.cc:323] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.3:45979
I20260709 12:24:22.093413  6410 ts_tablet_manager.cc:1403] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.053s	user 0.025s	sys 0.019s
I20260709 12:24:22.094619  5720 tablet_copy_service.cc:140] P f613ba929b1046d3b5aed909414a95bf: Received BeginTabletCopySession request for tablet 786a49bad9be400ea9ce584030aad446 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:42568)
I20260709 12:24:22.095234  5720 tablet_copy_service.cc:161] P f613ba929b1046d3b5aed909414a95bf: Beginning new tablet copy session on tablet 786a49bad9be400ea9ce584030aad446 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:42568: session id = 38736cc545eb4389b220a3bd21d3b466-786a49bad9be400ea9ce584030aad446
I20260709 12:24:22.096098  6410 raft_consensus.cc:359] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:22.096990  6410 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:22.097671  6410 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:22.101253  6410 ts_tablet_manager.cc:1434] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.005s
I20260709 12:24:22.103648  5720 tablet_copy_source_session.cc:215] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:22.104494  5571 tablet_copy_service.cc:342] P cea66aa15be3423880843974acdbf50f: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-607035a460944cc2b065c4d4cc59f813 received from {username='slave'} at 127.0.0.1:58854
I20260709 12:24:22.104960  5571 tablet_copy_service.cc:434] P cea66aa15be3423880843974acdbf50f: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-607035a460944cc2b065c4d4cc59f813 on tablet 607035a460944cc2b065c4d4cc59f813 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:22.106917  6413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 786a49bad9be400ea9ce584030aad446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:22.118922  6413 tablet_copy_client.cc:806] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:22.119467  6413 tablet_copy_client.cc:670] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:22.123616  6413 tablet_copy_client.cc:538] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:22.133468  6413 tablet_bootstrap.cc:492] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:22.154909  6413 tablet_bootstrap.cc:492] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:22.155663  6413 tablet_bootstrap.cc:492] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:22.156244  6413 ts_tablet_manager.cc:1403] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.023s	user 0.016s	sys 0.000s
I20260709 12:24:22.158593  6413 raft_consensus.cc:359] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:22.159159  6413 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:22.159758  6413 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:22.161880  6413 ts_tablet_manager.cc:1434] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.005s	user 0.000s	sys 0.004s
I20260709 12:24:22.164695  5720 tablet_copy_service.cc:342] P f613ba929b1046d3b5aed909414a95bf: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-786a49bad9be400ea9ce584030aad446 received from {username='slave'} at 127.0.0.1:42568
I20260709 12:24:22.165133  5720 tablet_copy_service.cc:434] P f613ba929b1046d3b5aed909414a95bf: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-786a49bad9be400ea9ce584030aad446 on tablet 786a49bad9be400ea9ce584030aad446 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:22.253840  6413 ts_tablet_manager.cc:933] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:22.256417  6413 tablet_copy_client.cc:323] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.3:45979
I20260709 12:24:22.258399  5720 tablet_copy_service.cc:140] P f613ba929b1046d3b5aed909414a95bf: Received BeginTabletCopySession request for tablet fa294357a99443e8bd25587ba629d3de from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:42568)
I20260709 12:24:22.258937  5720 tablet_copy_service.cc:161] P f613ba929b1046d3b5aed909414a95bf: Beginning new tablet copy session on tablet fa294357a99443e8bd25587ba629d3de from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:42568: session id = 38736cc545eb4389b220a3bd21d3b466-fa294357a99443e8bd25587ba629d3de
I20260709 12:24:22.264974  5720 tablet_copy_source_session.cc:215] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:22.267998  6413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa294357a99443e8bd25587ba629d3de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:22.279491  6413 tablet_copy_client.cc:806] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:22.280057  6413 tablet_copy_client.cc:670] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:22.280658  6410 ts_tablet_manager.cc:933] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:22.283838  6410 tablet_copy_client.cc:323] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.3:45979
I20260709 12:24:22.286257  6413 tablet_copy_client.cc:538] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:22.286720  5720 tablet_copy_service.cc:140] P f613ba929b1046d3b5aed909414a95bf: Received BeginTabletCopySession request for tablet f4f86de9f88949219896e4c020485a11 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:42568)
I20260709 12:24:22.287279  5720 tablet_copy_service.cc:161] P f613ba929b1046d3b5aed909414a95bf: Beginning new tablet copy session on tablet f4f86de9f88949219896e4c020485a11 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:42568: session id = 38736cc545eb4389b220a3bd21d3b466-f4f86de9f88949219896e4c020485a11
I20260709 12:24:22.300606  6413 tablet_bootstrap.cc:492] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:22.309028  5720 tablet_copy_source_session.cc:215] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:22.313295  6410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f86de9f88949219896e4c020485a11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:22.361024  6413 tablet_bootstrap.cc:492] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:22.362119  6413 tablet_bootstrap.cc:492] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:22.362835  6413 ts_tablet_manager.cc:1403] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.062s	user 0.029s	sys 0.008s
I20260709 12:24:22.364857  6410 tablet_copy_client.cc:806] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:22.365439  6410 tablet_copy_client.cc:670] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:22.366778  6413 raft_consensus.cc:359] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:22.367760  6413 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:22.368505  6413 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:22.371065  6413 ts_tablet_manager.cc:1434] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:24:22.377578  5720 tablet_copy_service.cc:342] P f613ba929b1046d3b5aed909414a95bf: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-fa294357a99443e8bd25587ba629d3de received from {username='slave'} at 127.0.0.1:42568
I20260709 12:24:22.378029  5720 tablet_copy_service.cc:434] P f613ba929b1046d3b5aed909414a95bf: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-fa294357a99443e8bd25587ba629d3de on tablet fa294357a99443e8bd25587ba629d3de with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:22.380959  6410 tablet_copy_client.cc:538] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:22.389081  6410 tablet_bootstrap.cc:492] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:22.411900  6410 tablet_bootstrap.cc:492] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:22.413014  6410 tablet_bootstrap.cc:492] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:22.413786  6410 ts_tablet_manager.cc:1403] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.025s	user 0.018s	sys 0.004s
I20260709 12:24:22.417476  6410 raft_consensus.cc:359] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:22.418409  6410 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:22.419118  6410 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:22.421792  6410 ts_tablet_manager.cc:1434] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:24:22.423509  5720 tablet_copy_service.cc:342] P f613ba929b1046d3b5aed909414a95bf: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-f4f86de9f88949219896e4c020485a11 received from {username='slave'} at 127.0.0.1:42568
I20260709 12:24:22.423964  5720 tablet_copy_service.cc:434] P f613ba929b1046d3b5aed909414a95bf: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-f4f86de9f88949219896e4c020485a11 on tablet f4f86de9f88949219896e4c020485a11 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:22.461273  6280 raft_consensus.cc:1217] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:22.494876  6280 raft_consensus.cc:1217] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:22.725685  6280 raft_consensus.cc:1217] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:22.837738  6280 raft_consensus.cc:1217] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:22.864882  6421 raft_consensus.cc:1064] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:22.866827  6421 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:22.880379  6280 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:22.882831  5632 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:22.882644  6421 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:22.884959  5708 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:22.886894  6399 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:22.900820  6421 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:22.914124  6245 raft_consensus.cc:2955] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:22.916545  5632 raft_consensus.cc:2955] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:22.937637  5459 catalog_manager.cc:5697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:22.940608  6280 raft_consensus.cc:2955] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:22.951036  5708 raft_consensus.cc:2955] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:22.999895  6317 raft_consensus.cc:1064] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:23.004405  6317 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:23.039460  5632 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:23.041687  6280 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:23.044464  6169 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:23.048684  6390 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:23.051290  5554 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:23.066983  5561 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:23.068487  6390 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:23.070434  5632 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:23.070411  6280 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:23.072371  6422 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:23.073856  6434 raft_consensus.cc:2955] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.075455  5628 raft_consensus.cc:2955] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.080729  6422 raft_consensus.cc:2955] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.093920  5458 catalog_manager.cc:5697] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:23.094530  6280 raft_consensus.cc:2955] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.096455  6429 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:24:23.100026  5631 raft_consensus.cc:2955] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.106237  6279 raft_consensus.cc:2955] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.111913  5708 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:23.118145  5561 raft_consensus.cc:2955] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.129004  5636 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:23.134977  5456 catalog_manager.cc:5697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.144991  6435 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:23.152058  6169 raft_consensus.cc:2955] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.153484  5446 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 607035a460944cc2b065c4d4cc59f813 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:23.156075  5628 raft_consensus.cc:2955] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.161895  5443 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 786a49bad9be400ea9ce584030aad446 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:23.164788  5693 tablet_service.cc:1558] Processing DeleteTablet for tablet 607035a460944cc2b065c4d4cc59f813 with delete_type TABLET_DATA_TOMBSTONED (TS f613ba929b1046d3b5aed909414a95bf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38456
I20260709 12:24:23.184957  6278 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:23.190577  6434 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:23.190758  5457 catalog_manager.cc:5697] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:23.198798  6278 raft_consensus.cc:2955] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.323108  6439 tablet_replica.cc:333] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:24:23.324416  6439 raft_consensus.cc:2243] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:23.342378  5550 tablet_service.cc:1558] Processing DeleteTablet for tablet 786a49bad9be400ea9ce584030aad446 with delete_type TABLET_DATA_TOMBSTONED (TS cea66aa15be3423880843974acdbf50f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49014
I20260709 12:24:23.353848  6439 raft_consensus.cc:2272] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:23.358075  6439 ts_tablet_manager.cc:1916] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:23.388051  6446 tablet_replica.cc:333] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:24:23.389256  6446 raft_consensus.cc:2243] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:23.390133  6446 raft_consensus.cc:2272] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:23.409112  6439 ts_tablet_manager.cc:1929] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:23.411782  6439 log.cc:1199] T 607035a460944cc2b065c4d4cc59f813 P f613ba929b1046d3b5aed909414a95bf: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root/wals/607035a460944cc2b065c4d4cc59f813
I20260709 12:24:23.415798  6446 ts_tablet_manager.cc:1916] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:23.416743  5443 catalog_manager.cc:5028] TS f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): tablet 607035a460944cc2b065c4d4cc59f813 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:23.430402  6446 ts_tablet_manager.cc:1929] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:23.430966  6446 log.cc:1199] T 786a49bad9be400ea9ce584030aad446 P cea66aa15be3423880843974acdbf50f: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/wals/786a49bad9be400ea9ce584030aad446
I20260709 12:24:23.432757  5446 catalog_manager.cc:5028] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): tablet 786a49bad9be400ea9ce584030aad446 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:23.446326  6443 raft_consensus.cc:1064] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:23.449263  6443 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:23.451292  6445 raft_consensus.cc:1064] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:23.456866  6445 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:23.482285  6278 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:23.483757  5554 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:23.483668  6444 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:23.487490  5636 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:23.482282  5628 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:23.488613  6434 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:23.490270  5554 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:23.490526  6278 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:23.492143  6447 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:23.494215  6434 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:23.496061  6447 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:23.504717  6444 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:23.522745  6390 raft_consensus.cc:2955] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.532887  5557 raft_consensus.cc:2955] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.537387  6278 raft_consensus.cc:2955] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.556461  5457 catalog_manager.cc:5697] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.558277  5636 raft_consensus.cc:2955] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.560294  6443 raft_consensus.cc:2955] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.565793  6278 raft_consensus.cc:2955] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.582392  5557 raft_consensus.cc:2955] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.584123  5455 catalog_manager.cc:5697] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.579941  5636 raft_consensus.cc:2955] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.597569  5702 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:23.624261  6278 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:23.626579  6443 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:24:23.631395  5557 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:23.633845  5702 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:23.634377  6390 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:23.638159  6445 raft_consensus.cc:2955] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.647974  5554 raft_consensus.cc:2955] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.649372  6278 raft_consensus.cc:2955] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.654212  5443 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet f4f86de9f88949219896e4c020485a11 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:23.672413  5459 catalog_manager.cc:5697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:23.680393  5628 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:23.680935  6278 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:23.681890  6390 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:23.682791  6445 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:23.687803  5619 tablet_service.cc:1558] Processing DeleteTablet for tablet f4f86de9f88949219896e4c020485a11 with delete_type TABLET_DATA_TOMBSTONED (TS 4a1c5e6dd8e249bf82d38f90fa37b43d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35612
I20260709 12:24:23.695542  6456 tablet_replica.cc:333] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:24:23.696880  6456 raft_consensus.cc:2243] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:23.698105  6456 raft_consensus.cc:2272] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:23.703255  6278 raft_consensus.cc:2955] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.704185  5628 raft_consensus.cc:2955] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.701315  6443 raft_consensus.cc:2955] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.724020  5455 catalog_manager.cc:5697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:23.724975  6456 ts_tablet_manager.cc:1916] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:23.740978  5550 tablet_service.cc:1558] Processing DeleteTablet for tablet fa294357a99443e8bd25587ba629d3de with delete_type TABLET_DATA_TOMBSTONED (TS cea66aa15be3423880843974acdbf50f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49014
I20260709 12:24:23.742070  6446 tablet_replica.cc:333] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:24:23.743435  6446 raft_consensus.cc:2243] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:23.748243  5443 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet fa294357a99443e8bd25587ba629d3de with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:23.773909  6446 raft_consensus.cc:2272] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:23.780454  6456 ts_tablet_manager.cc:1929] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:23.780833  6446 ts_tablet_manager.cc:1916] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:23.781056  6456 log.cc:1199] T f4f86de9f88949219896e4c020485a11 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/wals/f4f86de9f88949219896e4c020485a11
I20260709 12:24:23.782912  5445 catalog_manager.cc:5028] TS 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): tablet f4f86de9f88949219896e4c020485a11 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:23.798827  6446 ts_tablet_manager.cc:1929] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:23.799261  6446 log.cc:1199] T fa294357a99443e8bd25587ba629d3de P cea66aa15be3423880843974acdbf50f: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/wals/fa294357a99443e8bd25587ba629d3de
I20260709 12:24:23.800576  5446 catalog_manager.cc:5028] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): tablet fa294357a99443e8bd25587ba629d3de (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:24.854401  5561 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
W20260709 12:24:24.861277  5526 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e20e319f46504838a20c558a66692da6. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:24.861107  5708 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:24:24.861107  5631 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:24:24.863016  6459 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:24.863791  6458 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:24.873694  6458 raft_consensus.cc:2955] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:24.875581  5631 raft_consensus.cc:2955] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:24.876143  5708 raft_consensus.cc:2955] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:24.895170  5561 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:24.896019  5458 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index -1 to 3, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 3 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
W20260709 12:24:24.909461  5526 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f938b4d80d34f89a34d90b3c31fa84a. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:24.909893  5709 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:24.917629  6422 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:24.925828  5628 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:24.930709  6458 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:24.941215  6459 raft_consensus.cc:2955] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:24.942978  5709 raft_consensus.cc:2955] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:24.959415  5628 raft_consensus.cc:2955] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:24.969602  5561 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
W20260709 12:24:24.980667  5526 consensus_peers.cc:597] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 021ceb67cc2f4876b5dbdf1ca21c5f07. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:24.985518  5708 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:24.986594  5458 catalog_manager.cc:5697] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:24.985769  5628 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:24.988999  6465 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:24.990962  6422 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:25.019373  6458 raft_consensus.cc:2955] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:25.020658  5708 raft_consensus.cc:2955] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:25.024149  5628 raft_consensus.cc:2955] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:25.035063  5631 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:25.040906  5455 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:25.042723  5708 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:25.045679  5561 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:24:25.046382  5601 consensus_peers.cc:597] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c6474db7a70405694509f6976d850ce. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:25.047289  6388 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:25.048321  6423 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:25.063515  6388 raft_consensus.cc:2955] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:25.064615  5708 raft_consensus.cc:2955] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:25.065430  5554 raft_consensus.cc:2955] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:25.081741  5459 catalog_manager.cc:5697] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:25.279767  6479 ts_tablet_manager.cc:933] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:25.284041  6479 tablet_copy_client.cc:323] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.1:43285
I20260709 12:24:25.286780  5571 tablet_copy_service.cc:140] P cea66aa15be3423880843974acdbf50f: Received BeginTabletCopySession request for tablet e20e319f46504838a20c558a66692da6 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:58854)
I20260709 12:24:25.287449  5571 tablet_copy_service.cc:161] P cea66aa15be3423880843974acdbf50f: Beginning new tablet copy session on tablet e20e319f46504838a20c558a66692da6 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:58854: session id = 38736cc545eb4389b220a3bd21d3b466-e20e319f46504838a20c558a66692da6
I20260709 12:24:25.295945  5571 tablet_copy_source_session.cc:215] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:25.306818  6479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e20e319f46504838a20c558a66692da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:25.322640  6479 tablet_copy_client.cc:806] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:25.323369  6479 tablet_copy_client.cc:670] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:25.328768  6479 tablet_copy_client.cc:538] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:25.341480  6479 tablet_bootstrap.cc:492] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:25.369091  6479 tablet_bootstrap.cc:492] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:25.369915  6479 tablet_bootstrap.cc:492] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:25.370642  6479 ts_tablet_manager.cc:1403] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.029s	user 0.025s	sys 0.000s
I20260709 12:24:25.373353  6479 raft_consensus.cc:359] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:25.374178  6479 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:25.374814  6479 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:25.380765  6479 ts_tablet_manager.cc:1434] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.010s	user 0.012s	sys 0.000s
I20260709 12:24:25.385459  5571 tablet_copy_service.cc:342] P cea66aa15be3423880843974acdbf50f: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-e20e319f46504838a20c558a66692da6 received from {username='slave'} at 127.0.0.1:58854
I20260709 12:24:25.386014  5571 tablet_copy_service.cc:434] P cea66aa15be3423880843974acdbf50f: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-e20e319f46504838a20c558a66692da6 on tablet e20e319f46504838a20c558a66692da6 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:25.407202  6479 ts_tablet_manager.cc:933] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:25.409647  6479 tablet_copy_client.cc:323] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.1:43285
I20260709 12:24:25.411623  5571 tablet_copy_service.cc:140] P cea66aa15be3423880843974acdbf50f: Received BeginTabletCopySession request for tablet 021ceb67cc2f4876b5dbdf1ca21c5f07 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:58854)
I20260709 12:24:25.412271  5571 tablet_copy_service.cc:161] P cea66aa15be3423880843974acdbf50f: Beginning new tablet copy session on tablet 021ceb67cc2f4876b5dbdf1ca21c5f07 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:58854: session id = 38736cc545eb4389b220a3bd21d3b466-021ceb67cc2f4876b5dbdf1ca21c5f07
I20260709 12:24:25.419369  5571 tablet_copy_source_session.cc:215] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:25.422814  6479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 021ceb67cc2f4876b5dbdf1ca21c5f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:25.441042  6479 tablet_copy_client.cc:806] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:25.441715  6479 tablet_copy_client.cc:670] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:25.445114  6479 tablet_copy_client.cc:538] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:25.454697  6479 tablet_bootstrap.cc:492] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:25.473050  6484 ts_tablet_manager.cc:933] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:25.475431  6484 tablet_copy_client.cc:323] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.1:43285
I20260709 12:24:25.477411  5571 tablet_copy_service.cc:140] P cea66aa15be3423880843974acdbf50f: Received BeginTabletCopySession request for tablet 6f938b4d80d34f89a34d90b3c31fa84a from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:58854)
I20260709 12:24:25.477919  5571 tablet_copy_service.cc:161] P cea66aa15be3423880843974acdbf50f: Beginning new tablet copy session on tablet 6f938b4d80d34f89a34d90b3c31fa84a from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:58854: session id = 38736cc545eb4389b220a3bd21d3b466-6f938b4d80d34f89a34d90b3c31fa84a
I20260709 12:24:25.481472  6479 tablet_bootstrap.cc:492] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:25.482496  6479 tablet_bootstrap.cc:492] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:25.483214  6479 ts_tablet_manager.cc:1403] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.029s	user 0.016s	sys 0.011s
I20260709 12:24:25.485085  5571 tablet_copy_source_session.cc:215] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:25.486244  6479 raft_consensus.cc:359] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:25.487138  6479 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:25.488783  6479 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:25.491473  6479 ts_tablet_manager.cc:1434] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:24:25.493283  5571 tablet_copy_service.cc:342] P cea66aa15be3423880843974acdbf50f: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-021ceb67cc2f4876b5dbdf1ca21c5f07 received from {username='slave'} at 127.0.0.1:58854
I20260709 12:24:25.493685  5571 tablet_copy_service.cc:434] P cea66aa15be3423880843974acdbf50f: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-021ceb67cc2f4876b5dbdf1ca21c5f07 on tablet 021ceb67cc2f4876b5dbdf1ca21c5f07 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:25.495208  6484 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f938b4d80d34f89a34d90b3c31fa84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:25.510010  6484 tablet_copy_client.cc:806] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:25.510691  6484 tablet_copy_client.cc:670] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:25.514667  6484 tablet_copy_client.cc:538] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:25.523864  6484 tablet_bootstrap.cc:492] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:25.542670  6479 ts_tablet_manager.cc:933] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:25.546980  6484 tablet_bootstrap.cc:492] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:25.548007  6484 tablet_bootstrap.cc:492] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:25.548744  6484 ts_tablet_manager.cc:1403] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.025s	user 0.008s	sys 0.015s
I20260709 12:24:25.553685  6484 raft_consensus.cc:359] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:25.554555  6484 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:25.555208  6484 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:25.562335  6484 ts_tablet_manager.cc:1434] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.013s	user 0.000s	sys 0.006s
I20260709 12:24:25.564074  5571 tablet_copy_service.cc:342] P cea66aa15be3423880843974acdbf50f: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-6f938b4d80d34f89a34d90b3c31fa84a received from {username='slave'} at 127.0.0.1:58854
I20260709 12:24:25.564519  5571 tablet_copy_service.cc:434] P cea66aa15be3423880843974acdbf50f: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-6f938b4d80d34f89a34d90b3c31fa84a on tablet 6f938b4d80d34f89a34d90b3c31fa84a with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:25.571923  6479 tablet_copy_client.cc:323] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.2:38587
I20260709 12:24:25.576649  5646 tablet_copy_service.cc:140] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Received BeginTabletCopySession request for tablet 8c6474db7a70405694509f6976d850ce from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:40390)
I20260709 12:24:25.577334  5646 tablet_copy_service.cc:161] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Beginning new tablet copy session on tablet 8c6474db7a70405694509f6976d850ce from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:40390: session id = 38736cc545eb4389b220a3bd21d3b466-8c6474db7a70405694509f6976d850ce
I20260709 12:24:25.584568  5646 tablet_copy_source_session.cc:215] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:25.588157  6479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c6474db7a70405694509f6976d850ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:25.605569  6479 tablet_copy_client.cc:806] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:25.606217  6479 tablet_copy_client.cc:670] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:25.614125  6479 tablet_copy_client.cc:538] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:25.624199  6479 tablet_bootstrap.cc:492] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:25.650070  6479 tablet_bootstrap.cc:492] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:25.650928  6479 tablet_bootstrap.cc:492] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:25.651602  6479 ts_tablet_manager.cc:1403] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.028s	user 0.027s	sys 0.000s
I20260709 12:24:25.653951  6479 raft_consensus.cc:359] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:25.654493  6479 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:25.654983  6479 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:25.656649  6479 ts_tablet_manager.cc:1434] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 12:24:25.661605  5646 tablet_copy_service.cc:342] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-8c6474db7a70405694509f6976d850ce received from {username='slave'} at 127.0.0.1:40390
I20260709 12:24:25.662063  5646 tablet_copy_service.cc:434] P 4a1c5e6dd8e249bf82d38f90fa37b43d: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-8c6474db7a70405694509f6976d850ce on tablet 8c6474db7a70405694509f6976d850ce with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:25.699728  6280 raft_consensus.cc:1217] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3]   Dedup: 3.3->[]
I20260709 12:24:25.793291  6280 raft_consensus.cc:1217] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:26.072014  6280 raft_consensus.cc:1217] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:26.109988  6280 raft_consensus.cc:1217] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:26.233485  6422 raft_consensus.cc:1064] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:26.235358  6422 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:26.241257  5708 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:24:26.242218  6280 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEARNER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:24:26.243425  6422 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:26.242951  5628 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:24:26.245220  6465 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:26.246768  6458 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:26.266204  6458 raft_consensus.cc:2955] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.268973  5708 raft_consensus.cc:2955] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.268275  5632 raft_consensus.cc:2955] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.278035  6280 raft_consensus.cc:2955] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.291916  5455 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 3 to 4, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.332231  5554 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:26.340907  6465 raft_consensus.cc:1064] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:26.342267  6280 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:24:26.343429  6465 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:26.345139  5632 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:24:26.346930  6422 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:26.347726  6458 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:24:26.355226  6278 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:26.355538  5631 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:26.357160  5708 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:26.361256  6465 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:26.363133  6458 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:26.367101  6458 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:26.370237  6458 raft_consensus.cc:2955] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.397444  5446 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e20e319f46504838a20c558a66692da6 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:26.398543  5631 raft_consensus.cc:2955] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.401360  6280 raft_consensus.cc:2955] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.412873  6465 raft_consensus.cc:2955] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.416519  5456 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 4 to 5, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New cstate: current_term: 3 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:26.423715  5702 raft_consensus.cc:2955] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.429461  5693 tablet_service.cc:1558] Processing DeleteTablet for tablet e20e319f46504838a20c558a66692da6 with delete_type TABLET_DATA_TOMBSTONED (TS f613ba929b1046d3b5aed909414a95bf not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38456
I20260709 12:24:26.438344  5632 raft_consensus.cc:2955] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.447113  6505 tablet_replica.cc:333] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:24:26.448310  6505 raft_consensus.cc:2243] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:26.449188  6505 raft_consensus.cc:2272] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:26.451773  6280 raft_consensus.cc:2955] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.462126  6422 raft_consensus.cc:1064] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:26.465303  5458 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.469866  6422 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:26.480373  6280 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:26.481778  5702 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:26.489202  6505 ts_tablet_manager.cc:1916] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:26.489758  6465 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:26.493350  6501 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:26.491024  5632 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:26.496752  6501 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:26.501811  5561 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:26.507926  6278 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:26.509953  5709 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:26.514487  6501 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:26.516376  6422 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:26.522357  6505 ts_tablet_manager.cc:1929] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20260709 12:24:26.524220  6505 log.cc:1199] T e20e319f46504838a20c558a66692da6 P f613ba929b1046d3b5aed909414a95bf: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root/wals/e20e319f46504838a20c558a66692da6
I20260709 12:24:26.524080  6502 raft_consensus.cc:2955] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.529552  5443 catalog_manager.cc:5028] TS f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): tablet e20e319f46504838a20c558a66692da6 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:26.534247  5702 raft_consensus.cc:2955] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.543772  5446 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 021ceb67cc2f4876b5dbdf1ca21c5f07 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:26.543069  6278 raft_consensus.cc:2955] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.552604  5459 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:26.568029  6279 raft_consensus.cc:2955] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.564009  6458 raft_consensus.cc:2955] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.587438  6388 raft_consensus.cc:1064] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:26.590374  6388 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:26.596796  5455 catalog_manager.cc:5697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.620474  5632 raft_consensus.cc:2955] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.652076  5619 tablet_service.cc:1558] Processing DeleteTablet for tablet 021ceb67cc2f4876b5dbdf1ca21c5f07 with delete_type TABLET_DATA_TOMBSTONED (TS 4a1c5e6dd8e249bf82d38f90fa37b43d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35612
I20260709 12:24:26.655030  6513 tablet_replica.cc:333] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
W20260709 12:24:26.665210  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58733.4us
I20260709 12:24:26.668013  6279 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:26.670307  6516 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:26.675216  5554 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:26.682596  6475 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:26.696760  5702 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:26.708173  6475 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:26.720077  5554 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:26.721928  5708 raft_consensus.cc:2955] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.726752  6513 raft_consensus.cc:2243] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:26.727499  6513 raft_consensus.cc:2272] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:26.731326  6513 ts_tablet_manager.cc:1916] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:26.735558  6388 raft_consensus.cc:2955] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.737373  6279 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:26.737373  5628 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:26.743669  6465 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:26.744491  6458 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:26.748550  5554 raft_consensus.cc:2955] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.756981  6465 raft_consensus.cc:2955] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.760278  5628 raft_consensus.cc:2955] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.764688  6279 raft_consensus.cc:2955] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.791226  5446 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 6f938b4d80d34f89a34d90b3c31fa84a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:26.801985  5456 catalog_manager.cc:5697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.792943  5459 catalog_manager.cc:5697] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.812896  6279 raft_consensus.cc:2955] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.831707  5709 raft_consensus.cc:2955] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.841173  5693 tablet_service.cc:1558] Processing DeleteTablet for tablet 6f938b4d80d34f89a34d90b3c31fa84a with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:38456
I20260709 12:24:26.842555  6505 tablet_replica.cc:333] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:24:26.858314  6513 ts_tablet_manager.cc:1929] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:26.859110  6513 log.cc:1199] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/wals/021ceb67cc2f4876b5dbdf1ca21c5f07
I20260709 12:24:26.861763  5445 catalog_manager.cc:5028] TS 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): tablet 021ceb67cc2f4876b5dbdf1ca21c5f07 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:26.877499  6505 raft_consensus.cc:2243] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:26.878413  6505 raft_consensus.cc:2272] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:26.882640  6505 ts_tablet_manager.cc:1916] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:24:26.908893  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 87952.7us
I20260709 12:24:26.933034  6505 ts_tablet_manager.cc:1929] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:26.933574  6505 log.cc:1199] T 6f938b4d80d34f89a34d90b3c31fa84a P f613ba929b1046d3b5aed909414a95bf: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root/wals/6f938b4d80d34f89a34d90b3c31fa84a
I20260709 12:24:26.935490  5443 catalog_manager.cc:5028] TS f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): tablet 6f938b4d80d34f89a34d90b3c31fa84a (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:26.946184  5632 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:26.951694  5709 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:26.953286  6522 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:24:26.953903  6279 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:26.955925  6475 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:26.961774  6388 raft_consensus.cc:2955] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.963599  5709 raft_consensus.cc:2955] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.966174  6278 raft_consensus.cc:2955] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.981950  5459 catalog_manager.cc:5697] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:26.985708  5445 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8c6474db7a70405694509f6976d850ce with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:27.003901  5550 tablet_service.cc:1558] Processing DeleteTablet for tablet 8c6474db7a70405694509f6976d850ce with delete_type TABLET_DATA_TOMBSTONED (TS cea66aa15be3423880843974acdbf50f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49014
I20260709 12:24:27.005887  6524 tablet_replica.cc:333] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:24:27.010134  6524 raft_consensus.cc:2243] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:27.010766  6524 raft_consensus.cc:2272] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:27.013656  6524 ts_tablet_manager.cc:1916] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:27.029639  6524 ts_tablet_manager.cc:1929] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:27.030058  6524 log.cc:1199] T 8c6474db7a70405694509f6976d850ce P cea66aa15be3423880843974acdbf50f: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/wals/8c6474db7a70405694509f6976d850ce
I20260709 12:24:27.031602  5446 catalog_manager.cc:5028] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): tablet 8c6474db7a70405694509f6976d850ce (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:28.080210  5554 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:28.088168  5702 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:28.089356  5628 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:24:28.088284  5526 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f7bab74248a4aeda157f7a50fa82233. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:28.090631  6458 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:28.091324  6521 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:28.101429  6521 raft_consensus.cc:2955] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:28.103562  5628 raft_consensus.cc:2955] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:28.107101  5702 raft_consensus.cc:2955] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:28.118172  5708 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:28.120422  5458 catalog_manager.cc:5697] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:28.145900  5561 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:28.148144  6511 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:28.152089  5632 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:28.156337  6534 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:24:28.164450  5675 consensus_peers.cc:597] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9db6962eb8f9432289d80decd309ff40. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:28.172179  6534 raft_consensus.cc:2955] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:28.174031  5561 raft_consensus.cc:2955] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:28.175416  5632 raft_consensus.cc:2955] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:28.191072  5628 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:28.203379  5459 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:28.222790  5561 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:24:28.232050  5601 consensus_peers.cc:597] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44c764c1f49048f28e1caefabe1cc376. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:28.239033  5702 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:28.243443  6522 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:28.250262  6516 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:24:28.278705  5702 raft_consensus.cc:2955] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:28.278301  6522 raft_consensus.cc:2955] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:28.308465  5561 raft_consensus.cc:2955] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
W20260709 12:24:28.316815  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 96894.5us
I20260709 12:24:28.334604  5628 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:28.355369  5455 catalog_manager.cc:5697] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:28.371389  5561 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:24:28.371726  5601 consensus_peers.cc:597] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ad0bd0e70f549b7bdcfb140817535f7. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:28.372951  5702 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:28.373380  6546 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:28.374431  6541 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:28.384663  6546 raft_consensus.cc:2955] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:28.386415  5561 raft_consensus.cc:2955] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:28.390308  5702 raft_consensus.cc:2955] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:28.408638  5458 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:28.676045  6551 ts_tablet_manager.cc:933] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:28.678473  6551 tablet_copy_client.cc:323] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.1:43285
I20260709 12:24:28.680203  5571 tablet_copy_service.cc:140] P cea66aa15be3423880843974acdbf50f: Received BeginTabletCopySession request for tablet 9f7bab74248a4aeda157f7a50fa82233 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:58854)
I20260709 12:24:28.680930  5571 tablet_copy_service.cc:161] P cea66aa15be3423880843974acdbf50f: Beginning new tablet copy session on tablet 9f7bab74248a4aeda157f7a50fa82233 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:58854: session id = 38736cc545eb4389b220a3bd21d3b466-9f7bab74248a4aeda157f7a50fa82233
I20260709 12:24:28.688777  5571 tablet_copy_source_session.cc:215] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:28.692687  6551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f7bab74248a4aeda157f7a50fa82233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:28.704272  6553 ts_tablet_manager.cc:933] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:28.706753  6553 tablet_copy_client.cc:323] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.2:38587
I20260709 12:24:28.708534  5646 tablet_copy_service.cc:140] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Received BeginTabletCopySession request for tablet 44c764c1f49048f28e1caefabe1cc376 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:40390)
I20260709 12:24:28.709046  5646 tablet_copy_service.cc:161] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Beginning new tablet copy session on tablet 44c764c1f49048f28e1caefabe1cc376 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:40390: session id = 38736cc545eb4389b220a3bd21d3b466-44c764c1f49048f28e1caefabe1cc376
I20260709 12:24:28.709754  6551 tablet_copy_client.cc:806] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:28.710376  6551 tablet_copy_client.cc:670] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:28.714995  6551 tablet_copy_client.cc:538] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:28.727792  6551 tablet_bootstrap.cc:492] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:28.737632  5646 tablet_copy_source_session.cc:215] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:28.742492  6553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c764c1f49048f28e1caefabe1cc376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:28.745206  6555 ts_tablet_manager.cc:933] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:28.760376  6555 tablet_copy_client.cc:323] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.3:45979
I20260709 12:24:28.763746  5720 tablet_copy_service.cc:140] P f613ba929b1046d3b5aed909414a95bf: Received BeginTabletCopySession request for tablet 9db6962eb8f9432289d80decd309ff40 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:42568)
I20260709 12:24:28.764608  5720 tablet_copy_service.cc:161] P f613ba929b1046d3b5aed909414a95bf: Beginning new tablet copy session on tablet 9db6962eb8f9432289d80decd309ff40 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:42568: session id = 38736cc545eb4389b220a3bd21d3b466-9db6962eb8f9432289d80decd309ff40
I20260709 12:24:28.775236  6551 tablet_bootstrap.cc:492] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:28.776506  6551 tablet_bootstrap.cc:492] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:28.777283  6551 ts_tablet_manager.cc:1403] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.050s	user 0.023s	sys 0.007s
I20260709 12:24:28.781189  6551 raft_consensus.cc:359] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:28.782160  6551 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:28.782930  6551 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:28.797853  6553 tablet_copy_client.cc:806] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:28.798606  6553 tablet_copy_client.cc:670] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:28.820492  5720 tablet_copy_source_session.cc:215] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:28.825542  6551 ts_tablet_manager.cc:1434] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.048s	user 0.021s	sys 0.024s
I20260709 12:24:28.827955  5571 tablet_copy_service.cc:342] P cea66aa15be3423880843974acdbf50f: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-9f7bab74248a4aeda157f7a50fa82233 received from {username='slave'} at 127.0.0.1:58854
I20260709 12:24:28.828426  5571 tablet_copy_service.cc:434] P cea66aa15be3423880843974acdbf50f: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-9f7bab74248a4aeda157f7a50fa82233 on tablet 9f7bab74248a4aeda157f7a50fa82233 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:28.831190  6555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db6962eb8f9432289d80decd309ff40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:28.837075  6553 tablet_copy_client.cc:538] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:28.849766  6553 tablet_bootstrap.cc:492] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:28.866340  6555 tablet_copy_client.cc:806] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:28.867017  6555 tablet_copy_client.cc:670] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:28.871114  6555 tablet_copy_client.cc:538] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:28.874465  6553 tablet_bootstrap.cc:492] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:28.875270  6553 tablet_bootstrap.cc:492] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:28.875937  6553 ts_tablet_manager.cc:1403] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.026s	user 0.009s	sys 0.010s
I20260709 12:24:28.878531  6553 raft_consensus.cc:359] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:28.879397  6553 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:28.880071  6553 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:28.881421  6555 tablet_bootstrap.cc:492] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:28.886330  6553 ts_tablet_manager.cc:1434] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.004s
I20260709 12:24:28.888129  5646 tablet_copy_service.cc:342] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-44c764c1f49048f28e1caefabe1cc376 received from {username='slave'} at 127.0.0.1:40390
I20260709 12:24:28.888705  5646 tablet_copy_service.cc:434] P 4a1c5e6dd8e249bf82d38f90fa37b43d: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-44c764c1f49048f28e1caefabe1cc376 on tablet 44c764c1f49048f28e1caefabe1cc376 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:28.904654  6553 ts_tablet_manager.cc:933] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:28.908226  6553 tablet_copy_client.cc:323] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.2:38587
I20260709 12:24:28.910293  5646 tablet_copy_service.cc:140] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Received BeginTabletCopySession request for tablet 4ad0bd0e70f549b7bdcfb140817535f7 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:40390)
I20260709 12:24:28.910841  5646 tablet_copy_service.cc:161] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Beginning new tablet copy session on tablet 4ad0bd0e70f549b7bdcfb140817535f7 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:40390: session id = 38736cc545eb4389b220a3bd21d3b466-4ad0bd0e70f549b7bdcfb140817535f7
I20260709 12:24:28.916005  6555 tablet_bootstrap.cc:492] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:28.917184  6555 tablet_bootstrap.cc:492] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:28.917987  6555 ts_tablet_manager.cc:1403] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.037s	user 0.019s	sys 0.005s
I20260709 12:24:28.919128  5646 tablet_copy_source_session.cc:215] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:28.922642  6553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ad0bd0e70f549b7bdcfb140817535f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:28.922566  6555 raft_consensus.cc:359] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:28.923609  6555 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:28.924309  6555 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:28.936239  6555 ts_tablet_manager.cc:1434] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.000s
I20260709 12:24:28.953168  5720 tablet_copy_service.cc:342] P f613ba929b1046d3b5aed909414a95bf: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-9db6962eb8f9432289d80decd309ff40 received from {username='slave'} at 127.0.0.1:42568
I20260709 12:24:28.953876  5720 tablet_copy_service.cc:434] P f613ba929b1046d3b5aed909414a95bf: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-9db6962eb8f9432289d80decd309ff40 on tablet 9db6962eb8f9432289d80decd309ff40 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:28.960407  6553 tablet_copy_client.cc:806] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:28.961076  6553 tablet_copy_client.cc:670] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:28.967166  6553 tablet_copy_client.cc:538] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:28.979121  6553 tablet_bootstrap.cc:492] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:29.001318  6553 tablet_bootstrap.cc:492] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:29.002254  6553 tablet_bootstrap.cc:492] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:29.002941  6553 ts_tablet_manager.cc:1403] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.024s	user 0.015s	sys 0.005s
I20260709 12:24:29.005805  6553 raft_consensus.cc:359] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:29.006624  6553 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:29.007241  6553 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:29.009718  6553 ts_tablet_manager.cc:1434] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:24:29.011307  5646 tablet_copy_service.cc:342] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-4ad0bd0e70f549b7bdcfb140817535f7 received from {username='slave'} at 127.0.0.1:40390
I20260709 12:24:29.011669  5646 tablet_copy_service.cc:434] P 4a1c5e6dd8e249bf82d38f90fa37b43d: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-4ad0bd0e70f549b7bdcfb140817535f7 on tablet 4ad0bd0e70f549b7bdcfb140817535f7 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:29.091140  6279 raft_consensus.cc:1217] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:29.221740  6279 raft_consensus.cc:1217] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:29.232834  6279 raft_consensus.cc:1217] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:29.412858  6279 raft_consensus.cc:1217] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:29.537286  6458 raft_consensus.cc:1064] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:29.539091  6458 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:29.545445  5709 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:29.545639  6279 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:29.547365  6458 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:29.548918  6521 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:29.558022  5630 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:29.565534  6458 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:29.585458  6559 raft_consensus.cc:2955] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.591688  5709 raft_consensus.cc:2955] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.597066  5630 raft_consensus.cc:2955] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.617323  5455 catalog_manager.cc:5697] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:29.619073  6279 raft_consensus.cc:2955] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.660207  5561 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:29.664619  6279 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:29.664816  5630 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:29.666549  6559 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:29.667798  6521 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:29.673297  6559 raft_consensus.cc:2955] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.676563  6279 raft_consensus.cc:2955] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.679235  5630 raft_consensus.cc:2955] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.688854  5446 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9f7bab74248a4aeda157f7a50fa82233 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:29.699960  6511 raft_consensus.cc:1064] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:29.702987  6511 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:29.710253  5632 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:29.712270  6563 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:29.719414  5456 catalog_manager.cc:5697] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.722787  6279 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:29.728618  6566 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:29.734278  5561 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:29.736054  6566 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:29.739100  5693 tablet_service.cc:1558] Processing DeleteTablet for tablet 9f7bab74248a4aeda157f7a50fa82233 with delete_type TABLET_DATA_TOMBSTONED (TS f613ba929b1046d3b5aed909414a95bf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38456
I20260709 12:24:29.775869  6577 tablet_replica.cc:333] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:24:29.778488  6577 raft_consensus.cc:2243] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:29.781221  6577 raft_consensus.cc:2272] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:29.786499  6577 ts_tablet_manager.cc:1916] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:29.799204  6541 raft_consensus.cc:1064] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:29.802227  6541 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:29.803776  6577 ts_tablet_manager.cc:1929] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:29.804337  6577 log.cc:1199] T 9f7bab74248a4aeda157f7a50fa82233 P f613ba929b1046d3b5aed909414a95bf: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root/wals/9f7bab74248a4aeda157f7a50fa82233
I20260709 12:24:29.806550  5443 catalog_manager.cc:5028] TS f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): tablet 9f7bab74248a4aeda157f7a50fa82233 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:29.814432  6566 raft_consensus.cc:2955] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.817610  5561 raft_consensus.cc:2955] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.819999  5630 raft_consensus.cc:2955] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.831327  5702 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:29.833055  6279 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:29.834756  6516 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:29.837036  5458 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:29.838127  5557 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:29.859351  6547 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:29.863238  6278 raft_consensus.cc:2955] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.881839  6547 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:29.889559  5708 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:29.904734  6280 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:29.907778  6547 raft_consensus.cc:2955] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.915391  5557 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:29.909263  6541 raft_consensus.cc:1064] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:29.918222  6511 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:29.918843  6541 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:29.932572  6511 raft_consensus.cc:2955] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.935019  6564 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:29.941722  5554 raft_consensus.cc:2955] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.942727  5702 raft_consensus.cc:2955] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.944662  6279 raft_consensus.cc:2955] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.965077  6280 raft_consensus.cc:2955] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:29.969338  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45041.1us
I20260709 12:24:29.971486  5456 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.974783  5458 catalog_manager.cc:5697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:29.986850  5443 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9db6962eb8f9432289d80decd309ff40 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:29.994431  5708 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:29.996977  5619 tablet_service.cc:1558] Processing DeleteTablet for tablet 9db6962eb8f9432289d80decd309ff40 with delete_type TABLET_DATA_TOMBSTONED (TS 4a1c5e6dd8e249bf82d38f90fa37b43d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35612
I20260709 12:24:29.997452  6516 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:30.008755  6280 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:30.010216  6516 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:30.018101  5557 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:30.021405  6588 tablet_replica.cc:333] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:24:30.023118  6588 raft_consensus.cc:2243] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:30.023910  6588 raft_consensus.cc:2272] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:30.035530  6280 raft_consensus.cc:2955] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:30.037807  6541 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260709 12:24:30.046597  6588 ts_tablet_manager.cc:1916] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:30.033592  6516 raft_consensus.cc:2955] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:30.033779  5557 raft_consensus.cc:2955] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:30.056736  5630 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:30.062116  5456 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:30.065225  5709 raft_consensus.cc:2955] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:30.073586  6280 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:30.074997  6547 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:30.076231  5702 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:30.078528  6583 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:30.087670  6279 raft_consensus.cc:2955] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:30.088744  6541 raft_consensus.cc:2955] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:30.093556  5632 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:30.099203  6280 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:30.102383  5708 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:30.105641  6547 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:30.111373  6516 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260709 12:24:30.112422  5458 catalog_manager.cc:5697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:30.125295  6583 raft_consensus.cc:2955] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:30.130270  5550 tablet_service.cc:1558] Processing DeleteTablet for tablet 44c764c1f49048f28e1caefabe1cc376 with delete_type TABLET_DATA_TOMBSTONED (TS cea66aa15be3423880843974acdbf50f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49014
I20260709 12:24:30.124799  5554 raft_consensus.cc:2955] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:30.127110  6279 raft_consensus.cc:2955] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:30.145835  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47021us
I20260709 12:24:30.161934  5445 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 44c764c1f49048f28e1caefabe1cc376 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:24:30.167150  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63403.1us
I20260709 12:24:30.172012  6595 tablet_replica.cc:333] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:24:30.173671  5445 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4ad0bd0e70f549b7bdcfb140817535f7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:30.156112  6588 ts_tablet_manager.cc:1929] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:30.174547  6595 raft_consensus.cc:2243] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:30.175381  6595 raft_consensus.cc:2272] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:30.179034  6595 ts_tablet_manager.cc:1916] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:30.174994  6588 log.cc:1199] T 9db6962eb8f9432289d80decd309ff40 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/wals/9db6962eb8f9432289d80decd309ff40
I20260709 12:24:30.186213  5445 catalog_manager.cc:5028] TS 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): tablet 9db6962eb8f9432289d80decd309ff40 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:30.189889  5458 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:30.198489  6595 ts_tablet_manager.cc:1929] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:30.199028  6595 log.cc:1199] T 44c764c1f49048f28e1caefabe1cc376 P cea66aa15be3423880843974acdbf50f: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/wals/44c764c1f49048f28e1caefabe1cc376
I20260709 12:24:30.202267  5446 catalog_manager.cc:5028] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): tablet 44c764c1f49048f28e1caefabe1cc376 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:30.205955  5550 tablet_service.cc:1558] Processing DeleteTablet for tablet 4ad0bd0e70f549b7bdcfb140817535f7 with delete_type TABLET_DATA_TOMBSTONED (TS cea66aa15be3423880843974acdbf50f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49014
I20260709 12:24:30.212191  5709 raft_consensus.cc:2955] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:30.225822  5708 raft_consensus.cc:2955] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:30.226961  6595 tablet_replica.cc:333] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:24:30.227959  6595 raft_consensus.cc:2243] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:30.228644  6595 raft_consensus.cc:2272] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:30.247457  6595 ts_tablet_manager.cc:1916] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:30.267872  6595 ts_tablet_manager.cc:1929] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:30.268229  6595 log.cc:1199] T 4ad0bd0e70f549b7bdcfb140817535f7 P cea66aa15be3423880843974acdbf50f: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/wals/4ad0bd0e70f549b7bdcfb140817535f7
I20260709 12:24:30.269538  5446 catalog_manager.cc:5028] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): tablet 4ad0bd0e70f549b7bdcfb140817535f7 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:31.292043  5709 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
W20260709 12:24:31.301858  5675 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0a3a361d19b43b9a0a562f49f9b719e. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:31.302237  5630 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:31.303526  5561 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:31.305580  6563 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:31.306550  6566 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:31.386876  6599 raft_consensus.cc:2955] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:31.416700  5630 raft_consensus.cc:2955] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:31.450095  5554 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:31.456269  5458 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:31.466722  5526 consensus_peers.cc:597] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52e5c61368994c6bbeee20e7007995e2. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:31.468282  5630 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:31.470777  5557 raft_consensus.cc:2955] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:31.473100  6559 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:31.484980  5702 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:31.490981  6559 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:31.497143  6596 raft_consensus.cc:2955] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:31.499672  5630 raft_consensus.cc:2955] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:31.503870  5702 raft_consensus.cc:2955] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:31.517585  5630 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:31.540737  5456 catalog_manager.cc:5697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:31.547595  5702 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:24:31.549217  5601 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e69273530292453d87614627debcc13e. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:31.549868  6516 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:31.551040  5557 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:31.552384  6516 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:31.561765  6594 raft_consensus.cc:2955] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:31.565376  5557 raft_consensus.cc:2955] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:31.573401  5702 raft_consensus.cc:2955] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:31.581779  5709 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:31.586987  5456 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:31.597167  5630 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:31.602818  6608 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:24:31.614012  5675 consensus_peers.cc:597] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35344119b8ff4cdebc4810661ff94f79. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:31.617509  5557 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:31.617724  6604 raft_consensus.cc:2955] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:31.618990  6607 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:31.619297  5630 raft_consensus.cc:2955] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:31.624749  5554 raft_consensus.cc:2955] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:31.640971  5458 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:31.687093  6623 ts_tablet_manager.cc:933] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:31.690644  6623 tablet_copy_client.cc:323] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.3:45979
I20260709 12:24:31.692680  5720 tablet_copy_service.cc:140] P f613ba929b1046d3b5aed909414a95bf: Received BeginTabletCopySession request for tablet b0a3a361d19b43b9a0a562f49f9b719e from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:42568)
I20260709 12:24:31.693220  5720 tablet_copy_service.cc:161] P f613ba929b1046d3b5aed909414a95bf: Beginning new tablet copy session on tablet b0a3a361d19b43b9a0a562f49f9b719e from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:42568: session id = 38736cc545eb4389b220a3bd21d3b466-b0a3a361d19b43b9a0a562f49f9b719e
I20260709 12:24:31.700953  5720 tablet_copy_source_session.cc:215] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:31.704308  6623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0a3a361d19b43b9a0a562f49f9b719e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:31.728555  6623 tablet_copy_client.cc:806] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:31.729454  6623 tablet_copy_client.cc:670] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:31.733460  6623 tablet_copy_client.cc:538] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:31.741598  6623 tablet_bootstrap.cc:492] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:31.761786  6623 tablet_bootstrap.cc:492] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:31.762712  6623 tablet_bootstrap.cc:492] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:31.763386  6623 ts_tablet_manager.cc:1403] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.022s	user 0.014s	sys 0.005s
I20260709 12:24:31.766287  6623 raft_consensus.cc:359] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:31.767139  6623 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:31.767830  6623 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:31.785916  6623 ts_tablet_manager.cc:1434] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.022s	user 0.013s	sys 0.009s
I20260709 12:24:31.787917  5720 tablet_copy_service.cc:342] P f613ba929b1046d3b5aed909414a95bf: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-b0a3a361d19b43b9a0a562f49f9b719e received from {username='slave'} at 127.0.0.1:42568
I20260709 12:24:31.788412  5720 tablet_copy_service.cc:434] P f613ba929b1046d3b5aed909414a95bf: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-b0a3a361d19b43b9a0a562f49f9b719e on tablet b0a3a361d19b43b9a0a562f49f9b719e with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:32.025537  6623 ts_tablet_manager.cc:933] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:32.027155  6623 tablet_copy_client.cc:323] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.1:43285
I20260709 12:24:32.028894  5571 tablet_copy_service.cc:140] P cea66aa15be3423880843974acdbf50f: Received BeginTabletCopySession request for tablet 52e5c61368994c6bbeee20e7007995e2 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:58854)
I20260709 12:24:32.029464  5571 tablet_copy_service.cc:161] P cea66aa15be3423880843974acdbf50f: Beginning new tablet copy session on tablet 52e5c61368994c6bbeee20e7007995e2 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:58854: session id = 38736cc545eb4389b220a3bd21d3b466-52e5c61368994c6bbeee20e7007995e2
I20260709 12:24:32.037410  5571 tablet_copy_source_session.cc:215] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:32.040627  6623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52e5c61368994c6bbeee20e7007995e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:32.058259  6623 tablet_copy_client.cc:806] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:32.058933  6623 tablet_copy_client.cc:670] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:32.075599  6623 tablet_copy_client.cc:538] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:32.088282  6628 ts_tablet_manager.cc:933] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:32.089383  6623 tablet_bootstrap.cc:492] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:32.099138  6629 ts_tablet_manager.cc:933] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:32.112272  6628 tablet_copy_client.cc:323] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.2:38587
I20260709 12:24:32.115214  5646 tablet_copy_service.cc:140] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Received BeginTabletCopySession request for tablet e69273530292453d87614627debcc13e from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:40390)
I20260709 12:24:32.115751  5646 tablet_copy_service.cc:161] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Beginning new tablet copy session on tablet e69273530292453d87614627debcc13e from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:40390: session id = 38736cc545eb4389b220a3bd21d3b466-e69273530292453d87614627debcc13e
I20260709 12:24:32.120947  6629 tablet_copy_client.cc:323] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.3:45979
I20260709 12:24:32.123421  5720 tablet_copy_service.cc:140] P f613ba929b1046d3b5aed909414a95bf: Received BeginTabletCopySession request for tablet 35344119b8ff4cdebc4810661ff94f79 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:42568)
I20260709 12:24:32.123966  5720 tablet_copy_service.cc:161] P f613ba929b1046d3b5aed909414a95bf: Beginning new tablet copy session on tablet 35344119b8ff4cdebc4810661ff94f79 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:42568: session id = 38736cc545eb4389b220a3bd21d3b466-35344119b8ff4cdebc4810661ff94f79
I20260709 12:24:32.127379  5646 tablet_copy_source_session.cc:215] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:32.136773  6628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e69273530292453d87614627debcc13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:32.137694  5720 tablet_copy_source_session.cc:215] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:32.141052  6629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35344119b8ff4cdebc4810661ff94f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:32.143033  6623 tablet_bootstrap.cc:492] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:32.144074  6623 tablet_bootstrap.cc:492] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:32.145064  6623 ts_tablet_manager.cc:1403] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.059s	user 0.025s	sys 0.011s
I20260709 12:24:32.149721  6623 raft_consensus.cc:359] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:32.150858  6623 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:32.154423  6623 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:32.156215  6628 tablet_copy_client.cc:806] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:32.156903  6628 tablet_copy_client.cc:670] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:32.160984  6628 tablet_copy_client.cc:538] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:32.161862  6623 ts_tablet_manager.cc:1434] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260709 12:24:32.173921  6280 raft_consensus.cc:1217] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:32.190235  5571 tablet_copy_service.cc:342] P cea66aa15be3423880843974acdbf50f: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-52e5c61368994c6bbeee20e7007995e2 received from {username='slave'} at 127.0.0.1:58854
I20260709 12:24:32.190672  5571 tablet_copy_service.cc:434] P cea66aa15be3423880843974acdbf50f: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-52e5c61368994c6bbeee20e7007995e2 on tablet 52e5c61368994c6bbeee20e7007995e2 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:32.204146  6629 tablet_copy_client.cc:806] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:32.204869  6629 tablet_copy_client.cc:670] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:32.205961  6628 tablet_bootstrap.cc:492] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:32.209324  6629 tablet_copy_client.cc:538] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:32.221171  6629 tablet_bootstrap.cc:492] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:32.256227  6628 tablet_bootstrap.cc:492] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:32.257519  6628 tablet_bootstrap.cc:492] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:32.256227  6629 tablet_bootstrap.cc:492] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:32.259482  6629 tablet_bootstrap.cc:492] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:32.260221  6629 ts_tablet_manager.cc:1403] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.039s	user 0.022s	sys 0.005s
I20260709 12:24:32.258303  6628 ts_tablet_manager.cc:1403] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.053s	user 0.017s	sys 0.021s
I20260709 12:24:32.263372  6629 raft_consensus.cc:359] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:32.264223  6629 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:32.264498  6628 raft_consensus.cc:359] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:32.265208  6628 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:32.264904  6629 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:32.265820  6628 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:32.268460  6629 ts_tablet_manager.cc:1434] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:24:32.268059  6628 ts_tablet_manager.cc:1434] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.003s
I20260709 12:24:32.270817  5720 tablet_copy_service.cc:342] P f613ba929b1046d3b5aed909414a95bf: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-35344119b8ff4cdebc4810661ff94f79 received from {username='slave'} at 127.0.0.1:42568
I20260709 12:24:32.271215  5720 tablet_copy_service.cc:434] P f613ba929b1046d3b5aed909414a95bf: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-35344119b8ff4cdebc4810661ff94f79 on tablet 35344119b8ff4cdebc4810661ff94f79 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:32.272598  5646 tablet_copy_service.cc:342] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-e69273530292453d87614627debcc13e received from {username='slave'} at 127.0.0.1:40390
I20260709 12:24:32.273027  5646 tablet_copy_service.cc:434] P 4a1c5e6dd8e249bf82d38f90fa37b43d: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-e69273530292453d87614627debcc13e on tablet e69273530292453d87614627debcc13e with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:32.573988  6607 raft_consensus.cc:1064] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:32.576455  6607 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:32.583062  5632 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:32.584545  6633 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:32.590790  6280 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:32.592775  5557 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:32.594784  6609 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:32.615890  6638 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:32.623128  6279 raft_consensus.cc:1217] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:32.622457  6633 raft_consensus.cc:2955] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:32.631655  6280 raft_consensus.cc:1217] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:32.618196  5632 raft_consensus.cc:2955] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:32.683787  6280 raft_consensus.cc:2955] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:32.709405  5554 raft_consensus.cc:2955] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:32.712224  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 81565.4us
I20260709 12:24:32.724933  5459 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:32.749194  6280 raft_consensus.cc:1217] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:33.110059  6647 raft_consensus.cc:1064] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:33.112288  6647 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:33.127197  6280 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:33.127197  5702 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:33.127190  5554 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:33.128762  6647 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:33.130715  6641 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:33.137312  6516 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:33.158439  6516 raft_consensus.cc:2955] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.165462  6633 raft_consensus.cc:1064] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:33.168140  6633 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:33.167022  5557 raft_consensus.cc:2955] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.176533  5702 raft_consensus.cc:2955] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.196211  6280 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:33.195644  5455 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.197629  6638 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:33.200112  5630 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:33.204185  6638 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:33.200245  6280 raft_consensus.cc:2955] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.208760  5557 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:33.216921  6638 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:33.248004  5628 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:33.272683  5554 raft_consensus.cc:2955] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.278168  5557 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:33.279053  5702 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:33.283540  6647 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:33.284875  6279 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:33.289806  6641 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:33.296963  5557 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:33.299552  6609 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:33.310748  6609 raft_consensus.cc:2955] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.313174  5561 raft_consensus.cc:2955] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.297348  6641 raft_consensus.cc:2955] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.317621  5557 raft_consensus.cc:2955] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.320036  6643 raft_consensus.cc:1064] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:33.331101  6643 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:33.348321  6633 raft_consensus.cc:2955] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.350883  6279 raft_consensus.cc:2955] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:33.380674  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 119432us
I20260709 12:24:33.384099  6280 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:33.388088  6278 raft_consensus.cc:2955] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.390602  6609 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:33.390244  5628 raft_consensus.cc:2955] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.402534  6277 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:33.399474  5458 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.405714  5630 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:33.405799  5443 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b0a3a361d19b43b9a0a562f49f9b719e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:33.407317  5702 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:33.407559  6596 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:33.409547  6596 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:33.421485  6644 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:33.425374  5455 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.400156  6280 raft_consensus.cc:2955] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.415405  5445 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e69273530292453d87614627debcc13e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:33.455739  5693 tablet_service.cc:1558] Processing DeleteTablet for tablet e69273530292453d87614627debcc13e with delete_type TABLET_DATA_TOMBSTONED (TS f613ba929b1046d3b5aed909414a95bf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38456
I20260709 12:24:33.456094  5709 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:33.458122  6644 raft_consensus.cc:2955] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.460379  5458 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.479101  6278 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:33.481591  5554 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:33.483238  6666 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:33.484915  6667 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:33.496269  6663 raft_consensus.cc:2955] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.498672  5619 tablet_service.cc:1558] Processing DeleteTablet for tablet b0a3a361d19b43b9a0a562f49f9b719e with delete_type TABLET_DATA_TOMBSTONED (TS 4a1c5e6dd8e249bf82d38f90fa37b43d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35612
I20260709 12:24:33.502095  5557 raft_consensus.cc:2955] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.507213  6672 tablet_replica.cc:333] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:24:33.474805  6280 raft_consensus.cc:2955] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.502157  6279 raft_consensus.cc:2955] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.539410  6672 raft_consensus.cc:2243] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:33.539469  6671 tablet_replica.cc:333] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:24:33.545184  6672 raft_consensus.cc:2272] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:33.570384  5456 catalog_manager.cc:5697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.573994  5443 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 35344119b8ff4cdebc4810661ff94f79 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:33.574416  6672 ts_tablet_manager.cc:1916] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:33.554387  5708 raft_consensus.cc:2955] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.586810  5628 raft_consensus.cc:2955] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.598106  5458 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.616994  5619 tablet_service.cc:1558] Processing DeleteTablet for tablet 35344119b8ff4cdebc4810661ff94f79 with delete_type TABLET_DATA_TOMBSTONED (TS 4a1c5e6dd8e249bf82d38f90fa37b43d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35612
I20260709 12:24:33.628105  5557 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:33.629776  6671 raft_consensus.cc:2243] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:33.632246  6671 raft_consensus.cc:2272] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:33.638036  6671 ts_tablet_manager.cc:1916] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:33.637336  6672 ts_tablet_manager.cc:1929] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:33.643457  6672 log.cc:1199] T b0a3a361d19b43b9a0a562f49f9b719e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/wals/b0a3a361d19b43b9a0a562f49f9b719e
I20260709 12:24:33.643527  6279 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:33.643474  5628 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:33.645793  5445 catalog_manager.cc:5028] TS 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): tablet b0a3a361d19b43b9a0a562f49f9b719e (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:33.647029  6672 tablet_replica.cc:333] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:24:33.648898  6674 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:24:33.649513  6672 raft_consensus.cc:2243] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:33.650272  6672 raft_consensus.cc:2272] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:33.650417  6645 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:24:33.659883  6643 raft_consensus.cc:2955] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.666529  6672 ts_tablet_manager.cc:1916] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:33.670845  6671 ts_tablet_manager.cc:1929] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:33.672587  6671 log.cc:1199] T e69273530292453d87614627debcc13e P f613ba929b1046d3b5aed909414a95bf: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root/wals/e69273530292453d87614627debcc13e
I20260709 12:24:33.674777  5443 catalog_manager.cc:5028] TS f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): tablet e69273530292453d87614627debcc13e (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:33.675273  5628 raft_consensus.cc:2955] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.687510  6672 ts_tablet_manager.cc:1929] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:33.693101  6672 log.cc:1199] T 35344119b8ff4cdebc4810661ff94f79 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/wals/35344119b8ff4cdebc4810661ff94f79
I20260709 12:24:33.693928  5455 catalog_manager.cc:5697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.692766  6279 raft_consensus.cc:2955] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:33.697780  5445 catalog_manager.cc:5028] TS 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): tablet 35344119b8ff4cdebc4810661ff94f79 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:33.697007  5446 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 52e5c61368994c6bbeee20e7007995e2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:33.710990  5693 tablet_service.cc:1558] Processing DeleteTablet for tablet 52e5c61368994c6bbeee20e7007995e2 with delete_type TABLET_DATA_TOMBSTONED (TS f613ba929b1046d3b5aed909414a95bf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38456
I20260709 12:24:33.712061  6671 tablet_replica.cc:333] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:24:33.713213  6671 raft_consensus.cc:2243] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:33.713913  6671 raft_consensus.cc:2272] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:33.717191  6671 ts_tablet_manager.cc:1916] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:33.736646  6671 ts_tablet_manager.cc:1929] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:33.737058  6671 log.cc:1199] T 52e5c61368994c6bbeee20e7007995e2 P f613ba929b1046d3b5aed909414a95bf: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root/wals/52e5c61368994c6bbeee20e7007995e2
I20260709 12:24:33.738903  5443 catalog_manager.cc:5028] TS f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): tablet 52e5c61368994c6bbeee20e7007995e2 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:34.800213  5704 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:34.809901  5553 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:34.810432  5630 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:24:34.810702  5675 consensus_peers.cc:597] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9efccdcce1164ab5b57aa40caea54717. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:34.812067  6677 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:34.813310  6633 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:34.822731  6677 raft_consensus.cc:2955] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:34.826654  5630 raft_consensus.cc:2955] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:34.841684  5553 raft_consensus.cc:2955] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:34.850114  5630 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:34.850448  5456 catalog_manager.cc:5697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:34.862972  5708 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:24:34.864499  5601 consensus_peers.cc:597] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf758b480ecf41619471236195bf1838. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:34.866139  5554 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:34.868058  6678 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:34.870669  6675 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:34.889411  6594 raft_consensus.cc:2955] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:34.890944  5708 raft_consensus.cc:2955] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:34.907092  5557 raft_consensus.cc:2955] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:34.921362  5553 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:34.927502  5455 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:34.946287  5526 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2578c6c739884fad8fa75e16cfa14fec. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:34.950260  5630 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:34.950397  5708 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:34.951784  6643 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:34.953034  6679 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:34.966306  6673 raft_consensus.cc:2955] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:34.968060  5708 raft_consensus.cc:2955] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:34.968031  5630 raft_consensus.cc:2955] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:34.983521  5553 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:34.991055  5458 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
W20260709 12:24:34.993465  5526 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa0101c69f7746009c5d77c9616f3360. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:34.992774  5704 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:34.994992  6679 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:34.996622  5630 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:34.997857  6679 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:35.029341  6679 raft_consensus.cc:2955] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:35.034991  5704 raft_consensus.cc:2955] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:35.031154  5630 raft_consensus.cc:2955] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:35.055261  5458 catalog_manager.cc:5697] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:35.287575  6696 ts_tablet_manager.cc:933] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:35.290019  6696 tablet_copy_client.cc:323] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.3:45979
I20260709 12:24:35.292848  5720 tablet_copy_service.cc:140] P f613ba929b1046d3b5aed909414a95bf: Received BeginTabletCopySession request for tablet 9efccdcce1164ab5b57aa40caea54717 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:42568)
I20260709 12:24:35.293538  5720 tablet_copy_service.cc:161] P f613ba929b1046d3b5aed909414a95bf: Beginning new tablet copy session on tablet 9efccdcce1164ab5b57aa40caea54717 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:42568: session id = 38736cc545eb4389b220a3bd21d3b466-9efccdcce1164ab5b57aa40caea54717
I20260709 12:24:35.302277  5720 tablet_copy_source_session.cc:215] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:35.306061  6696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9efccdcce1164ab5b57aa40caea54717. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:35.322258  6696 tablet_copy_client.cc:806] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:35.323266  6696 tablet_copy_client.cc:670] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:35.329082  6696 tablet_copy_client.cc:538] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:35.336499  6696 tablet_bootstrap.cc:492] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:35.377490  6696 tablet_bootstrap.cc:492] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:35.378777  6696 tablet_bootstrap.cc:492] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:35.379534  6696 ts_tablet_manager.cc:1403] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.043s	user 0.029s	sys 0.012s
I20260709 12:24:35.382889  6696 raft_consensus.cc:359] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:35.383823  6696 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:35.384575  6696 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:35.388077  6696 ts_tablet_manager.cc:1434] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.008s	user 0.000s	sys 0.007s
I20260709 12:24:35.389899  5720 tablet_copy_service.cc:342] P f613ba929b1046d3b5aed909414a95bf: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-9efccdcce1164ab5b57aa40caea54717 received from {username='slave'} at 127.0.0.1:42568
I20260709 12:24:35.390377  5720 tablet_copy_service.cc:434] P f613ba929b1046d3b5aed909414a95bf: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-9efccdcce1164ab5b57aa40caea54717 on tablet 9efccdcce1164ab5b57aa40caea54717 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:35.429683  6696 ts_tablet_manager.cc:933] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:35.431670  6696 tablet_copy_client.cc:323] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.2:38587
I20260709 12:24:35.433884  5646 tablet_copy_service.cc:140] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Received BeginTabletCopySession request for tablet bf758b480ecf41619471236195bf1838 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:40390)
I20260709 12:24:35.434482  5646 tablet_copy_service.cc:161] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Beginning new tablet copy session on tablet bf758b480ecf41619471236195bf1838 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:40390: session id = 38736cc545eb4389b220a3bd21d3b466-bf758b480ecf41619471236195bf1838
I20260709 12:24:35.440565  5646 tablet_copy_source_session.cc:215] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:35.444532  6696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf758b480ecf41619471236195bf1838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:35.467442  6696 tablet_copy_client.cc:806] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:35.468379  6696 tablet_copy_client.cc:670] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:35.469944  6702 ts_tablet_manager.cc:933] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:35.472853  6696 tablet_copy_client.cc:538] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:35.476850  6702 tablet_copy_client.cc:323] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.1:43285
I20260709 12:24:35.479015  5571 tablet_copy_service.cc:140] P cea66aa15be3423880843974acdbf50f: Received BeginTabletCopySession request for tablet 2578c6c739884fad8fa75e16cfa14fec from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:58854)
I20260709 12:24:35.479902  5571 tablet_copy_service.cc:161] P cea66aa15be3423880843974acdbf50f: Beginning new tablet copy session on tablet 2578c6c739884fad8fa75e16cfa14fec from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:58854: session id = 38736cc545eb4389b220a3bd21d3b466-2578c6c739884fad8fa75e16cfa14fec
I20260709 12:24:35.496927  6696 tablet_bootstrap.cc:492] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:35.497119  5571 tablet_copy_source_session.cc:215] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:35.501978  6702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2578c6c739884fad8fa75e16cfa14fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:35.534198  6696 tablet_bootstrap.cc:492] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:35.535887  6696 tablet_bootstrap.cc:492] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:35.536954  6696 ts_tablet_manager.cc:1403] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.054s	user 0.025s	sys 0.011s
I20260709 12:24:35.540699  6702 tablet_copy_client.cc:806] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:35.541455  6702 tablet_copy_client.cc:670] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:35.543331  6696 raft_consensus.cc:359] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:35.544500  6696 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:35.545289  6696 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:35.554442  6696 ts_tablet_manager.cc:1434] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.000s
I20260709 12:24:35.559470  5646 tablet_copy_service.cc:342] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-bf758b480ecf41619471236195bf1838 received from {username='slave'} at 127.0.0.1:40390
I20260709 12:24:35.560171  5646 tablet_copy_service.cc:434] P 4a1c5e6dd8e249bf82d38f90fa37b43d: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-bf758b480ecf41619471236195bf1838 on tablet bf758b480ecf41619471236195bf1838 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:35.571672  6696 ts_tablet_manager.cc:933] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:35.582376  6702 tablet_copy_client.cc:538] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:35.585216  6696 tablet_copy_client.cc:323] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.1:43285
I20260709 12:24:35.587007  5571 tablet_copy_service.cc:140] P cea66aa15be3423880843974acdbf50f: Received BeginTabletCopySession request for tablet fa0101c69f7746009c5d77c9616f3360 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:58854)
I20260709 12:24:35.587570  5571 tablet_copy_service.cc:161] P cea66aa15be3423880843974acdbf50f: Beginning new tablet copy session on tablet fa0101c69f7746009c5d77c9616f3360 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:58854: session id = 38736cc545eb4389b220a3bd21d3b466-fa0101c69f7746009c5d77c9616f3360
I20260709 12:24:35.592926  6702 tablet_bootstrap.cc:492] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:35.597167  5571 tablet_copy_source_session.cc:215] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:35.601120  6696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa0101c69f7746009c5d77c9616f3360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:35.617570  6696 tablet_copy_client.cc:806] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:35.618111  6696 tablet_copy_client.cc:670] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:35.627743  6702 tablet_bootstrap.cc:492] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:35.629230  6702 tablet_bootstrap.cc:492] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:35.630136  6702 ts_tablet_manager.cc:1403] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.038s	user 0.033s	sys 0.001s
I20260709 12:24:35.636071  6702 raft_consensus.cc:359] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:35.639062  6702 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:35.639887  6702 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:35.645890  6702 ts_tablet_manager.cc:1434] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:24:35.646737  6696 tablet_copy_client.cc:538] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:35.647964  5571 tablet_copy_service.cc:342] P cea66aa15be3423880843974acdbf50f: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-2578c6c739884fad8fa75e16cfa14fec received from {username='slave'} at 127.0.0.1:58854
I20260709 12:24:35.648594  5571 tablet_copy_service.cc:434] P cea66aa15be3423880843974acdbf50f: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-2578c6c739884fad8fa75e16cfa14fec on tablet 2578c6c739884fad8fa75e16cfa14fec with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:35.657300  6696 tablet_bootstrap.cc:492] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:35.676262  6696 tablet_bootstrap.cc:492] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:35.677213  6696 tablet_bootstrap.cc:492] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:35.677937  6696 ts_tablet_manager.cc:1403] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.021s	user 0.016s	sys 0.004s
I20260709 12:24:35.680735  6696 raft_consensus.cc:359] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:35.681599  6696 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:35.682260  6696 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:35.684825  6696 ts_tablet_manager.cc:1434] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:24:35.686947  5571 tablet_copy_service.cc:342] P cea66aa15be3423880843974acdbf50f: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-fa0101c69f7746009c5d77c9616f3360 received from {username='slave'} at 127.0.0.1:58854
I20260709 12:24:35.687413  5571 tablet_copy_service.cc:434] P cea66aa15be3423880843974acdbf50f: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-fa0101c69f7746009c5d77c9616f3360 on tablet fa0101c69f7746009c5d77c9616f3360 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:35.732612  6278 raft_consensus.cc:1217] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:35.910631  6278 raft_consensus.cc:1217] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:35.929600  6278 raft_consensus.cc:1217] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:36.176537  6277 raft_consensus.cc:1217] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260709 12:24:36.310402  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 123922us
W20260709 12:24:36.310467  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 99862.8us
I20260709 12:24:36.340168  6633 raft_consensus.cc:1064] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:36.350521  6712 raft_consensus.cc:1064] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:36.353492  6712 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:36.354671  6633 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:36.391764  6277 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:36.394037  6633 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:36.420426  5554 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:36.443070  5630 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:36.447434  6633 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:36.465703  6714 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:36.514623  6277 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:36.514631  5703 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:36.518287  5557 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:36.519501  6678 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:36.520627  6717 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:36.521677  6712 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:36.529646  6714 raft_consensus.cc:2955] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.551877  5630 raft_consensus.cc:2955] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.571736  6712 raft_consensus.cc:2955] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.579457  5554 raft_consensus.cc:2955] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.585525  6673 raft_consensus.cc:1064] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:36.589733  6644 raft_consensus.cc:1064] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:36.590005  6277 raft_consensus.cc:2955] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.601908  5553 raft_consensus.cc:2955] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.610090  5703 raft_consensus.cc:2955] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.618468  5455 catalog_manager.cc:5697] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.627687  6277 raft_consensus.cc:2955] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.641706  6644 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:36.645874  6673 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:36.662148  5703 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:36.662798  6277 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:36.664739  6713 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:36.667124  6710 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:36.678476  5636 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:36.691531  5705 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:36.695181  6713 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:36.696201  6278 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:36.696741  5628 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:36.697254  6713 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:36.701472  6710 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:36.702446  6643 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:36.714970  6713 raft_consensus.cc:2955] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.718505  5634 raft_consensus.cc:2955] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.757725  6673 raft_consensus.cc:2955] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.761354  5458 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:36.763018  6277 raft_consensus.cc:2955] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.764472  5703 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:36.770493  5705 raft_consensus.cc:2955] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.786497  6278 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:36.789172  6633 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:36.797444  6704 raft_consensus.cc:2955] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.795022  6277 raft_consensus.cc:2955] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.802752  6278 raft_consensus.cc:2955] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.812498  5459 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.817323  5630 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:36.831027  5443 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9efccdcce1164ab5b57aa40caea54717 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:36.834450  5631 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:36.835223  5455 catalog_manager.cc:5697] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.838696  6633 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:36.850484  6278 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:36.852903  6594 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:36.857124  5631 raft_consensus.cc:2955] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.860523  5705 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:36.861991  6594 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:24:36.873201  6678 raft_consensus.cc:2955] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.878551  5702 raft_consensus.cc:2955] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.888234  5630 raft_consensus.cc:2955] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.889040  5459 catalog_manager.cc:5697] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:36.901363  5550 tablet_service.cc:1558] Processing DeleteTablet for tablet 9efccdcce1164ab5b57aa40caea54717 with delete_type TABLET_DATA_TOMBSTONED (TS cea66aa15be3423880843974acdbf50f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49014
I20260709 12:24:36.900163  6278 raft_consensus.cc:2955] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.940032  5445 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet bf758b480ecf41619471236195bf1838 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:36.942531  5702 raft_consensus.cc:2955] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.963677  6742 tablet_replica.cc:333] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:24:36.968758  6742 raft_consensus.cc:2243] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:36.969920  6742 raft_consensus.cc:2272] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:36.975447  5459 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:36.983052  6742 ts_tablet_manager.cc:1916] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:36.984190  5557 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:36.991824  6278 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:36.995128  5550 tablet_service.cc:1558] Processing DeleteTablet for tablet bf758b480ecf41619471236195bf1838 with delete_type TABLET_DATA_TOMBSTONED (TS cea66aa15be3423880843974acdbf50f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49014
I20260709 12:24:36.995201  6732 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:24:36.991509  5554 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:37.005204  6710 raft_consensus.cc:2955] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:37.014060  5705 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:37.014036  6280 raft_consensus.cc:2955] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:37.016567  6644 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:37.020654  5446 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet fa0101c69f7746009c5d77c9616f3360 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:37.021165  6742 ts_tablet_manager.cc:1929] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:37.021660  6742 log.cc:1199] T 9efccdcce1164ab5b57aa40caea54717 P cea66aa15be3423880843974acdbf50f: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/wals/9efccdcce1164ab5b57aa40caea54717
I20260709 12:24:37.023370  6742 tablet_replica.cc:333] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:24:37.023588  6278 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:37.024521  6742 raft_consensus.cc:2243] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:37.023306  5446 catalog_manager.cc:5028] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): tablet 9efccdcce1164ab5b57aa40caea54717 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:37.026696  5631 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:37.026823  6713 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:37.031677  6732 raft_consensus.cc:2955] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:37.034343  5458 catalog_manager.cc:5697] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:37.039176  5705 raft_consensus.cc:2955] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:37.042375  6278 raft_consensus.cc:2955] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:37.044476  5446 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2578c6c739884fad8fa75e16cfa14fec with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:37.027501  6742 raft_consensus.cc:2272] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:37.029538  6673 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:37.052103  6742 ts_tablet_manager.cc:1916] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:37.052901  5631 raft_consensus.cc:2955] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:37.062330  5693 tablet_service.cc:1558] Processing DeleteTablet for tablet fa0101c69f7746009c5d77c9616f3360 with delete_type TABLET_DATA_TOMBSTONED (TS f613ba929b1046d3b5aed909414a95bf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38456
I20260709 12:24:37.074012  5458 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:37.080504  6742 ts_tablet_manager.cc:1929] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:37.081048  6742 log.cc:1199] T bf758b480ecf41619471236195bf1838 P cea66aa15be3423880843974acdbf50f: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/wals/bf758b480ecf41619471236195bf1838
I20260709 12:24:37.082820  5446 catalog_manager.cc:5028] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): tablet bf758b480ecf41619471236195bf1838 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:37.084616  6743 tablet_replica.cc:333] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:24:37.087283  6743 raft_consensus.cc:2243] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:37.088069  6743 raft_consensus.cc:2272] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:37.093761  6743 ts_tablet_manager.cc:1916] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:37.095278  5619 tablet_service.cc:1558] Processing DeleteTablet for tablet 2578c6c739884fad8fa75e16cfa14fec with delete_type TABLET_DATA_TOMBSTONED (TS 4a1c5e6dd8e249bf82d38f90fa37b43d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35612
I20260709 12:24:37.103204  6744 tablet_replica.cc:333] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:24:37.104410  6744 raft_consensus.cc:2243] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:37.105209  6744 raft_consensus.cc:2272] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:37.108659  6744 ts_tablet_manager.cc:1916] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:37.116009  6743 ts_tablet_manager.cc:1929] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:37.116472  6743 log.cc:1199] T fa0101c69f7746009c5d77c9616f3360 P f613ba929b1046d3b5aed909414a95bf: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root/wals/fa0101c69f7746009c5d77c9616f3360
I20260709 12:24:37.118275  5443 catalog_manager.cc:5028] TS f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): tablet fa0101c69f7746009c5d77c9616f3360 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:37.122259  6744 ts_tablet_manager.cc:1929] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:37.122746  6744 log.cc:1199] T 2578c6c739884fad8fa75e16cfa14fec P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/wals/2578c6c739884fad8fa75e16cfa14fec
I20260709 12:24:37.124246  5445 catalog_manager.cc:5028] TS 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): tablet 2578c6c739884fad8fa75e16cfa14fec (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
W20260709 12:24:37.278012  5582 debug-util.cc:398] Leaking SignalData structure 0x7b08000e1aa0 after lost signal to thread 4359
W20260709 12:24:37.279081  5582 debug-util.cc:398] Leaking SignalData structure 0x7b08000e1440 after lost signal to thread 5493
W20260709 12:24:37.280015  5582 debug-util.cc:398] Leaking SignalData structure 0x7b08003211a0 after lost signal to thread 5585
W20260709 12:24:37.280848  5582 debug-util.cc:398] Leaking SignalData structure 0x7b0800301d00 after lost signal to thread 5660
W20260709 12:24:37.281646  5582 debug-util.cc:398] Leaking SignalData structure 0x7b080031b6a0 after lost signal to thread 5734
W20260709 12:24:37.282444  5582 debug-util.cc:398] Leaking SignalData structure 0x7b0800311a80 after lost signal to thread 6304
I20260709 12:24:38.006125  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:38.085788  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:24:38.086648  5630 raft_consensus.cc:606] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:38.089877  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:24:38.090293  5630 raft_consensus.cc:606] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Received request to transfer leadership to TS 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:38.128219  5702 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:24:38.128822  5702 raft_consensus.cc:606] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:38.179755  5554 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:47734
I20260709 12:24:38.180337  5554 raft_consensus.cc:606] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:38.183200  5554 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e20e319f46504838a20c558a66692da6"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:47734
I20260709 12:24:38.183712  5554 raft_consensus.cc:606] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Received request to transfer leadership to TS 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:38.186641  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e69273530292453d87614627debcc13e"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:24:38.187244  5630 raft_consensus.cc:606] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Received request to transfer leadership to TS 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:38.188670  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:24:38.190531  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:38.191669  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:38.227759  6745 raft_consensus.cc:993] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: : Instructing follower 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:38.228611  6713 raft_consensus.cc:1081] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Signalling peer 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:38.230434  6280 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e20e319f46504838a20c558a66692da6"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:47932
I20260709 12:24:38.231050  6280 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:38.231396  6280 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:38.238065  5554 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:38.247222  6280 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:38.242947  6745 raft_consensus.cc:993] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:38.248622  6673 raft_consensus.cc:1081] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:38.250825  5630 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:35614
I20260709 12:24:38.250939  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:38.251477  5630 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:38.251610  5557 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:24:38.251909  5630 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:38.251968  5557 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:38.255070  5557 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:38.256210  5557 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:38.278045  5630 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:38.280171  5630 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:38.281631  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:38.282266  5702 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:38.299788  5702 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:38.303697  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:38.304873  6678 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:38.305387  6678 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:38.306284  6678 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:38.317988  5705 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:38.328253  6752 raft_consensus.cc:993] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:24:38.332257  6713 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:24:38.336619  5526 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ca6496163ec40978a2dfd744edddb18. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:38.338608  5557 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:24:38.340131  6237 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:38.346659  6717 raft_consensus.cc:1081] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Signalling peer f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:24:38.347806  6751 raft_consensus.cc:993] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:38.349037  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:38.349030  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:38.349788  5554 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:38.349968  5628 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:38.350194  5554 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:38.350602  5630 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:38.348032  6280 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:38.348973  5705 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:38468
I20260709 12:24:38.351831  5705 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:38.352207  5705 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:38.353432  6673 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:38.359555  5554 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:38.361034  5554 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:38.375627  5554 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:38.382017  5459 catalog_manager.cc:5697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:38.349166  6751 raft_consensus.cc:1081] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:38.405272  5631 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:35630
I20260709 12:24:38.406023  5631 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:38.406396  5631 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:38.410277  6755 raft_consensus.cc:993] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:38.410846  6755 raft_consensus.cc:1081] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Signalling peer 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:38.412629  6280 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:47920
I20260709 12:24:38.413185  6280 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:38.413537  6280 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:38.414090  5631 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:38.421736  5631 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:38.422554  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:38.424810  5553 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:38.430935  6280 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:38.396028  5628 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:24:38.387672  5705 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:38.452131  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:38.452992  5557 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:38.472169  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:38.497258  5703 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:38.497605  5703 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:38.498510  5703 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:38.499742  5703 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:38.481727  6757 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:24:38.510530  5703 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:38.483598  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:38.512552  5628 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:38.512909  5628 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:38.513715  5628 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:38.515133  5628 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:24:38.517474  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47444.7us
I20260709 12:24:38.484452  5705 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:38.487094  6280 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:38.491472  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:38.523226  5636 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:38.523519  5636 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:24:38.492511  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 152483us
I20260709 12:24:38.524554  5636 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:38.525859  5636 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:38.535352  6745 raft_consensus.cc:2955] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:38.488406  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:38.546504  5553 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:38.551134  5636 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:38.551689  5561 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:38.552597  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:38.546403  5702 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:38.565004  6766 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:38.565640  6766 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:38.566593  6766 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:38.460306  5557 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:38.552721  6757 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:38.578655  6757 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:38.580518  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:24:38.548542  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 56919.1us
I20260709 12:24:38.581789  6717 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:38.583293  6717 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:38.584213  6717 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:38.587064  6678 raft_consensus.cc:993] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:38.587580  6678 raft_consensus.cc:1081] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Signalling peer 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:38.580892  5703 raft_consensus.cc:2955] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:38.589727  6278 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e69273530292453d87614627debcc13e"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:47920
I20260709 12:24:38.590386  6278 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:38.590799  6278 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:38.601799  5702 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:38.606841  5628 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:38.613047  6278 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:38.617566  5459 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 1 to 2, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 2 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:38.625029  6278 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:38.633459  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:38.634425  6768 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:38.634970  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:38.635589  5557 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:38.649435  6768 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:38.650453  5459 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:38.652704  6768 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:38.656834  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:38.657771  5628 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:38.658216  5628 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:38.659277  5628 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:38.662778  5636 raft_consensus.cc:2955] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:38.671886  5458 catalog_manager.cc:5697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 2, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:38.691433  5628 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:38.703946  5557 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:38.705960  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:38.706979  6773 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:38.707677  6773 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:38.708600  6773 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:38.771471  5628 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:38.773090  5636 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:38.780807  5459 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 3 to 4, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 4 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:38.782733  5459 catalog_manager.cc:5697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 2, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 2 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:24:38.785972  5601 consensus_peers.cc:597] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6adc8ae91dfe46039ab5bd2bc4c69708. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:38.787038  5702 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:38.788417  6678 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:38.789956  5553 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:38.792670  6678 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:38.808977  6678 raft_consensus.cc:2955] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:38.808095  5456 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 2, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 2 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:38.811954  5702 raft_consensus.cc:2955] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:38.814940  5561 raft_consensus.cc:2955] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:38.831635  5456 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:38.836473  5561 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:38.837478  5553 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:38.839327  6754 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:38.845569  6781 ts_tablet_manager.cc:933] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:38.883256  5702 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:38.885224  6754 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:24:38.999274  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 43825us
I20260709 12:24:39.004374  5557 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:39.018270  5628 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:24:39.019802  6768 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:24:39.026978  6751 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:39.126762  5554 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:24:39.128400  6768 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:24:39.135243  5636 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:39.150516  5704 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:39.163508  5558 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:39.150487  6768 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 12:24:39.169982  6752 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260709 12:24:39.211199  6753 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:39.257072  5704 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:39.259214  6757 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:24:39.267918  5526 consensus_peers.cc:597] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14483990837441a99fa83b4a981b4162. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:39.275247  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 173983us
I20260709 12:24:39.279654  5557 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:24:39.281005  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 115427us
I20260709 12:24:39.289585  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:39.313922  6796 ts_tablet_manager.cc:933] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:39.308387  6757 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 12:24:39.375511  5705 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:24:39.375536  6781 thread.cc:641] tablet-download-pool [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.027s	sys 0.055s
W20260709 12:24:39.410271  6781 thread.cc:608] tablet-download-pool [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.028s	sys 0.056s
I20260709 12:24:39.414188  6781 tablet_copy_client.cc:323] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.1:43285
W20260709 12:24:39.424665  6784 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 12:24:39.435501  5571 tablet_copy_service.cc:140] P cea66aa15be3423880843974acdbf50f: Received BeginTabletCopySession request for tablet 3ca6496163ec40978a2dfd744edddb18 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:58854)
I20260709 12:24:39.436097  5571 tablet_copy_service.cc:161] P cea66aa15be3423880843974acdbf50f: Beginning new tablet copy session on tablet 3ca6496163ec40978a2dfd744edddb18 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:58854: session id = 38736cc545eb4389b220a3bd21d3b466-3ca6496163ec40978a2dfd744edddb18
I20260709 12:24:39.444558  6713 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:39.449311  5628 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:39.457809  5571 tablet_copy_source_session.cc:215] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:39.467903  6781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca6496163ec40978a2dfd744edddb18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:39.493067  5630 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:39.521916  6781 tablet_copy_client.cc:806] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:39.526047  6781 tablet_copy_client.cc:670] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:39.529608  5634 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:39.530642  6775 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:39.536779  6768 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:39.547194  6796 tablet_copy_client.cc:323] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.2:38587
I20260709 12:24:39.549165  5646 tablet_copy_service.cc:140] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Received BeginTabletCopySession request for tablet 6adc8ae91dfe46039ab5bd2bc4c69708 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:40390)
I20260709 12:24:39.549824  5646 tablet_copy_service.cc:161] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Beginning new tablet copy session on tablet 6adc8ae91dfe46039ab5bd2bc4c69708 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:40390: session id = 38736cc545eb4389b220a3bd21d3b466-6adc8ae91dfe46039ab5bd2bc4c69708
I20260709 12:24:39.553545  6781 tablet_copy_client.cc:538] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:39.560547  5646 tablet_copy_source_session.cc:215] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:39.569536  6781 tablet_bootstrap.cc:492] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:39.579716  6796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6adc8ae91dfe46039ab5bd2bc4c69708. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:39.589918  6704 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:39.758144  5632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:24:39.759019  5632 raft_consensus.cc:606] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Received request to transfer leadership to TS 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:39.763741  6752 raft_consensus.cc:993] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:39.768195  5554 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:47734
I20260709 12:24:39.768882  5554 raft_consensus.cc:606] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:39.764231  6752 raft_consensus.cc:1081] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Signalling peer 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:39.778644  6279 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:47920
I20260709 12:24:39.779214  6279 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:39.779564  6279 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:39.785916  6775 raft_consensus.cc:2955] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:39.793159  5632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:24:39.793977  5632 raft_consensus.cc:606] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Received request to transfer leadership to TS 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:39.801976  6796 tablet_copy_client.cc:806] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:39.803085  6796 tablet_copy_client.cc:670] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:39.794186  6279 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:39.811359  6279 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:39.811453  6781 tablet_bootstrap.cc:492] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:39.812083  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:39.816090  5561 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:39.811769  5554 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:47734
I20260709 12:24:39.817447  5554 raft_consensus.cc:606] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Received request to transfer leadership to TS 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:39.820886  5630 raft_consensus.cc:2955] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:39.815162  5708 raft_consensus.cc:2955] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:39.829967  5703 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:24:39.833204  5703 raft_consensus.cc:606] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Received request to transfer leadership to TS 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:39.837095  6781 tablet_bootstrap.cc:492] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:39.839570  6751 raft_consensus.cc:993] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: : Instructing follower 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:39.840185  6751 raft_consensus.cc:1081] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Signalling peer 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:39.838344  6781 ts_tablet_manager.cc:1403] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.269s	user 0.017s	sys 0.018s
I20260709 12:24:39.845746  5561 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:39.844861  6812 ts_tablet_manager.cc:933] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:39.854672  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:39.857990  6773 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:39.864576  6773 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:39.865527  6773 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:39.888331  5458 catalog_manager.cc:5697] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:39.892673  5561 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:47734
I20260709 12:24:39.893379  5561 raft_consensus.cc:606] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:39.894843  6280 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:47934
I20260709 12:24:39.897154  6280 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:39.897717  6280 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:39.898229  6781 raft_consensus.cc:359] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:39.895691  6775 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:58838 (request call id 10) took 1061 ms. Trace:
I20260709 12:24:39.899087  6781 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
W20260709 12:24:39.863158  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 127395us
I20260709 12:24:39.899158  6775 rpcz_store.cc:276] 0709 12:24:38.833180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.833303 (+   123us) service_pool.cc:224] Handling call
0709 12:24:38.846426 (+ 13123us) log.cc:844] Serialized 485 byte log entry
0709 12:24:39.894858 (+1048432us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3663,"raft.run_cpu_time_us":1546,"raft.run_wall_time_us":1543,"thread_start_us":410308,"threads_started":1,"wal-append.queue_time_us":404812}
I20260709 12:24:39.909963  6280 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:39.920490  5708 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:39.899873  6781 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:39.927910  6781 ts_tablet_manager.cc:1434] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.080s	user 0.007s	sys 0.000s
I20260709 12:24:39.931357  5571 tablet_copy_service.cc:342] P cea66aa15be3423880843974acdbf50f: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-3ca6496163ec40978a2dfd744edddb18 received from {username='slave'} at 127.0.0.1:58854
I20260709 12:24:39.933248  5571 tablet_copy_service.cc:434] P cea66aa15be3423880843974acdbf50f: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-3ca6496163ec40978a2dfd744edddb18 on tablet 3ca6496163ec40978a2dfd744edddb18 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:39.933553  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:24:39.932013  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:39.934528  5630 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:39.934823  5630 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:39.935231  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:39.935504  5630 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:39.938647  6796 tablet_copy_client.cc:538] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:39.938971  5630 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:39.944737  6280 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:39.946873  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:39.947570  5703 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:39.947877  5703 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:39.948719  5703 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:39.949864  5703 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:39.958273  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:39.959007  5561 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:39.961544  6775 raft_consensus.cc:993] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:39.962114  6775 raft_consensus.cc:1081] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:39.966185  5634 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:35614
I20260709 12:24:39.966747  5634 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:39.967072  5634 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:39.967552  6796 tablet_bootstrap.cc:492] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:39.972122  5455 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 2, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 2 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:39.990787  5634 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:40.001202  5553 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:40.001202  5632 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:39.998690  5703 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:40.044473  6791 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:24:40.000619  5675 consensus_peers.cc:597] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1af549c25a724a2f9d4f65cc1c59f22a. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:40.045591  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45118.5us
I20260709 12:24:40.045624  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:40.046537  6773 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:40.047199  6773 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:40.022080  6793 raft_consensus.cc:993] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:40.050714  6793 raft_consensus.cc:1081] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Signalling peer 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:40.048015  6773 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:40.049851  6751 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 12:24:40.055083  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:40.055655  5708 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:40.055987  6279 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:47920
I20260709 12:24:40.056533  6279 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:40.056645  5634 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:40.056959  6279 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:40.063678  6279 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:39.993556  5561 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:40.042409  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:40.081111  5554 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:40.081763  5554 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:40.080912  5634 raft_consensus.cc:2955] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:40.082746  5554 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:40.006660  5630 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:40.085866  5557 raft_consensus.cc:2955] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:40.093830  6775 raft_consensus.cc:993] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: : Instructing follower 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:40.094362  6713 raft_consensus.cc:1081] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Signalling peer 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:40.097294  6280 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:47932
I20260709 12:24:40.097842  6280 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:40.098177  6280 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:40.084174  5554 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:40.009580  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:39.996718  6812 tablet_copy_client.cc:323] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.1:43285
I20260709 12:24:40.111635  5571 tablet_copy_service.cc:140] P cea66aa15be3423880843974acdbf50f: Received BeginTabletCopySession request for tablet 14483990837441a99fa83b4a981b4162 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:58854)
I20260709 12:24:40.112208  5571 tablet_copy_service.cc:161] P cea66aa15be3423880843974acdbf50f: Beginning new tablet copy session on tablet 14483990837441a99fa83b4a981b4162 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:58854: session id = 38736cc545eb4389b220a3bd21d3b466-14483990837441a99fa83b4a981b4162
I20260709 12:24:40.113346  6280 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:40.117733  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:40.118304  5634 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:40.120882  5571 tablet_copy_source_session.cc:215] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:40.079015  6766 raft_consensus.cc:2955] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:40.141885  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:40.144418  5703 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:40.129102  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:40.147100  5630 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:40.147420  5630 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:40.118886  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:40.126540  5634 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:40.148391  5561 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:40.148175  5630 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:40.148738  5561 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:40.070508  5708 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:40.149348  5630 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:40.173586  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:40.174587  6792 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:40.175067  6792 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:40.157238  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:24:40.175932  6792 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:40.177497  6773 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:40.178023  6773 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:40.179176  6773 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:40.149557  5561 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:40.189363  5561 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:40.194715  6812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14483990837441a99fa83b4a981b4162. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:40.153950  5554 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:40.198796  5459 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 2, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 2 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:40.203053  5455 catalog_manager.cc:5697] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:40.154650  6280 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:40.211370  5561 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:40.142513  6279 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:40.254269  5630 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:40.258399  5703 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:40.259608  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:40.260424  6811 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:40.260918  6811 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:40.261894  6811 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:40.274181  6812 tablet_copy_client.cc:806] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:40.275007  6812 tablet_copy_client.cc:670] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:40.276046  6796 tablet_bootstrap.cc:492] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:40.277298  6796 tablet_bootstrap.cc:492] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:40.278061  6796 ts_tablet_manager.cc:1403] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.311s	user 0.018s	sys 0.010s
I20260709 12:24:40.282181  6796 raft_consensus.cc:359] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:40.283069  6796 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:40.284689  6812 tablet_copy_client.cc:538] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:40.283954  6796 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:40.297291  6796 ts_tablet_manager.cc:1434] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.002s
I20260709 12:24:40.299458  5646 tablet_copy_service.cc:342] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-6adc8ae91dfe46039ab5bd2bc4c69708 received from {username='slave'} at 127.0.0.1:40390
I20260709 12:24:40.299898  5646 tablet_copy_service.cc:434] P 4a1c5e6dd8e249bf82d38f90fa37b43d: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-6adc8ae91dfe46039ab5bd2bc4c69708 on tablet 6adc8ae91dfe46039ab5bd2bc4c69708 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:40.304337  6812 tablet_bootstrap.cc:492] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:40.328537  5456 catalog_manager.cc:5697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:40.332178  5632 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:40.340098  6773 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:40.342463  5455 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 2, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 2 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:40.349345  5455 catalog_manager.cc:5697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 2 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:40.355723  6812 tablet_bootstrap.cc:492] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:40.358695  5557 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:40.359241  6812 tablet_bootstrap.cc:492] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:40.361234  6812 ts_tablet_manager.cc:1403] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.057s	user 0.020s	sys 0.008s
I20260709 12:24:40.364444  6812 raft_consensus.cc:359] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:40.365403  6812 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:40.366010  6812 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:40.360217  6773 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:40.386682  6812 ts_tablet_manager.cc:1434] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.001s
I20260709 12:24:40.388453  5571 tablet_copy_service.cc:342] P cea66aa15be3423880843974acdbf50f: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-14483990837441a99fa83b4a981b4162 received from {username='slave'} at 127.0.0.1:58854
I20260709 12:24:40.388890  5571 tablet_copy_service.cc:434] P cea66aa15be3423880843974acdbf50f: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-14483990837441a99fa83b4a981b4162 on tablet 14483990837441a99fa83b4a981b4162 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:40.402303  6775 raft_consensus.cc:993] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:40.402899  6775 raft_consensus.cc:1081] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:40.406095  5630 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:35614
I20260709 12:24:40.406759  5630 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:40.407212  5630 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:40.413822  5630 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:40.416420  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:40.416985  5557 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:40.417260  5557 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:40.418008  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:40.417948  5557 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:40.418707  5703 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:40.419027  5557 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:40.426997  5557 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:40.427459  5630 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:40.428170  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:40.429309  5703 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:40.429823  6754 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:40.430294  6754 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:40.431092  6754 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:40.442433  6280 raft_consensus.cc:1217] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:40.505826  5455 catalog_manager.cc:5697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:40.524219  6812 ts_tablet_manager.cc:933] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:40.537770  6812 tablet_copy_client.cc:323] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.3:45979
I20260709 12:24:40.539662  5720 tablet_copy_service.cc:140] P f613ba929b1046d3b5aed909414a95bf: Received BeginTabletCopySession request for tablet 1af549c25a724a2f9d4f65cc1c59f22a from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:42568)
I20260709 12:24:40.540194  5720 tablet_copy_service.cc:161] P f613ba929b1046d3b5aed909414a95bf: Beginning new tablet copy session on tablet 1af549c25a724a2f9d4f65cc1c59f22a from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:42568: session id = 38736cc545eb4389b220a3bd21d3b466-1af549c25a724a2f9d4f65cc1c59f22a
I20260709 12:24:40.548030  5720 tablet_copy_source_session.cc:215] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:40.551936  6812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1af549c25a724a2f9d4f65cc1c59f22a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:40.588690  5708 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:40.595708  6811 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:24:40.599354  6280 raft_consensus.cc:1217] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:40.620169  5561 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:40.622627  6810 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:40.628166  6812 tablet_copy_client.cc:806] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:40.628957  6812 tablet_copy_client.cc:670] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:40.632594  5632 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:40.634307  6810 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:40.638427  6812 tablet_copy_client.cc:538] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:40.644265  5557 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:40.652750  6810 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:40.665123  6812 tablet_bootstrap.cc:492] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:40.689004  5709 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:40.698163  6810 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:24:40.713588  6280 raft_consensus.cc:1217] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:40.766119  5632 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:40.759557  6812 tablet_bootstrap.cc:492] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:40.767498  6812 tablet_bootstrap.cc:492] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:40.767840  6773 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:40.768280  6812 ts_tablet_manager.cc:1403] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.103s	user 0.024s	sys 0.001s
I20260709 12:24:40.773447  5553 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:40.780396  6835 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:40.783341  6812 raft_consensus.cc:359] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:40.792654  6812 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:40.800658  6812 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:40.804450  6812 ts_tablet_manager.cc:1434] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.027s	user 0.009s	sys 0.000s
I20260709 12:24:40.805708  5703 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:40.806558  5720 tablet_copy_service.cc:342] P f613ba929b1046d3b5aed909414a95bf: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-1af549c25a724a2f9d4f65cc1c59f22a received from {username='slave'} at 127.0.0.1:42568
I20260709 12:24:40.806975  5720 tablet_copy_service.cc:434] P f613ba929b1046d3b5aed909414a95bf: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-1af549c25a724a2f9d4f65cc1c59f22a on tablet 1af549c25a724a2f9d4f65cc1c59f22a with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:40.807688  6835 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:40.874760  5708 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:40.876319  6835 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:40.919898  5557 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:40.922976  6754 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:40.967850  6713 raft_consensus.cc:1064] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:40.970228  6713 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:41.025645  6280 raft_consensus.cc:1217] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:41.076859  5708 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:41.077472  5636 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:41.078917  6280 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:41.080693  6765 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:41.081822  6775 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:41.084754  6775 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:41.087778  6793 raft_consensus.cc:1064] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:41.090520  6793 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:41.115763  5703 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:41.118274  6713 raft_consensus.cc:2955] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.119750  6280 raft_consensus.cc:2955] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.123055  6852 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:41.140965  5708 raft_consensus.cc:2955] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.146399  5557 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:41.148964  6279 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:41.150269  6836 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:41.153671  5630 raft_consensus.cc:2955] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.161072  6754 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:41.154870  5457 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:41.210172  6852 raft_consensus.cc:2955] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.217996  6775 raft_consensus.cc:1064] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:41.220817  6775 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:41.242583  6280 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:41.242282  5703 raft_consensus.cc:2955] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.244988  5708 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:41.253516  6713 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:41.252898  6279 raft_consensus.cc:2955] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.261412  6713 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:41.262423  5636 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:41.218567  5561 raft_consensus.cc:2955] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.277886  6713 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:41.356341  6775 raft_consensus.cc:2955] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.366302  5630 raft_consensus.cc:2955] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.371580  5561 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:41.387396  5636 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:41.389638  6277 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:41.392096  5455 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.395668  6745 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:24:41.365568  6280 raft_consensus.cc:2955] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.401682  6765 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:24:41.420727  6765 raft_consensus.cc:2955] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.426847  5705 raft_consensus.cc:2955] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.431887  5632 raft_consensus.cc:2955] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.433686  6280 raft_consensus.cc:2955] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.444236  5446 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 3ca6496163ec40978a2dfd744edddb18 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:41.458767  5630 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:41.463032  5456 catalog_manager.cc:5697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:41.480441  5456 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 3 to 4, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:41.486270  6277 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:41.488873  6793 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:24:41.508020  6754 raft_consensus.cc:2955] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.516830  6277 raft_consensus.cc:2955] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.527014  6851 raft_consensus.cc:1064] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:41.533466  6851 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:41.561355  5693 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ca6496163ec40978a2dfd744edddb18 with delete_type TABLET_DATA_TOMBSTONED (TS f613ba929b1046d3b5aed909414a95bf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38456
I20260709 12:24:41.571491  5557 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:41.572420  5705 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:41.574613  6852 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:41.581971  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:41.583429  5705 raft_consensus.cc:2955] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.560269  5458 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.600689  6280 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:41.610773  5703 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:41.612433  6713 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:24:41.622445  5550 tablet_service.cc:1558] Processing DeleteTablet for tablet 6adc8ae91dfe46039ab5bd2bc4c69708 with delete_type TABLET_DATA_TOMBSTONED (TS cea66aa15be3423880843974acdbf50f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49014
W20260709 12:24:41.678133  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44263.5us
I20260709 12:24:41.688612  6713 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:41.696426  6870 tablet_replica.cc:333] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
W20260709 12:24:41.715610  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 71655.5us
I20260709 12:24:41.719264  6870 raft_consensus.cc:2243] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:41.720184  6870 raft_consensus.cc:2272] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:41.734897  6870 ts_tablet_manager.cc:1916] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:41.721973  6869 tablet_replica.cc:333] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:24:41.690836  6745 raft_consensus.cc:2955] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.740998  6869 raft_consensus.cc:2243] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:41.828280  5705 raft_consensus.cc:2955] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:41.781484  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 76602.1us
I20260709 12:24:41.841096  5630 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 12:24:41.842257  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 119271us
W20260709 12:24:41.840679  6868 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 12:24:41.873823  6280 raft_consensus.cc:2955] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:41.887701  6869 raft_consensus.cc:2272] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:41.962459  5445 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 6adc8ae91dfe46039ab5bd2bc4c69708 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:24:41.984826  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 112566us
W20260709 12:24:42.010931  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49655us
I20260709 12:24:42.026921  6869 ts_tablet_manager.cc:1916] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:41.997469  6873 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20260709 12:24:42.057879  6276 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:42.062107  5446 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 14483990837441a99fa83b4a981b4162 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:42.065284  5553 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:42.094769  6886 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 12:24:42.096889  6873 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:24:42.146788  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 272930us
I20260709 12:24:42.147588  5705 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:24:42.148254  5705 raft_consensus.cc:606] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:24:42.152833  5561 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:47734
I20260709 12:24:42.153443  5561 raft_consensus.cc:606] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Received request to transfer leadership to TS 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:42.158162  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:24:42.158737  5630 raft_consensus.cc:606] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Received request to transfer leadership to TS 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:42.183840  6885 raft_consensus.cc:993] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: : Instructing follower cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:24:42.184500  6885 raft_consensus.cc:1081] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Signalling peer cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:24:42.187916  5554 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:49030
I20260709 12:24:42.188578  5554 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:42.188966  5554 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.200383  6875 raft_consensus.cc:993] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:42.205570  6875 raft_consensus.cc:1081] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Signalling peer 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:42.218709  6869 ts_tablet_manager.cc:1929] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:42.219347  6869 log.cc:1199] T 3ca6496163ec40978a2dfd744edddb18 P f613ba929b1046d3b5aed909414a95bf: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root/wals/3ca6496163ec40978a2dfd744edddb18
I20260709 12:24:42.222563  6870 ts_tablet_manager.cc:1929] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:42.222576  5443 catalog_manager.cc:5028] TS f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): tablet 3ca6496163ec40978a2dfd744edddb18 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:42.224663  6276 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:47920
I20260709 12:24:42.203994  5554 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.225286  6276 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:42.225653  6276 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.226722  5554 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:42.228427  5455 catalog_manager.cc:5697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New cstate: current_term: 1 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:42.229262  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:42.229817  6278 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.236549  6278 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:42.238541  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:42.239624  6745 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:42.240669  6745 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:42.241649  6745 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.223215  6870 log.cc:1199] T 6adc8ae91dfe46039ab5bd2bc4c69708 P cea66aa15be3423880843974acdbf50f: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/wals/6adc8ae91dfe46039ab5bd2bc4c69708
I20260709 12:24:42.245489  6765 raft_consensus.cc:993] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: : Instructing follower 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:42.250252  6713 raft_consensus.cc:1081] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Signalling peer 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:42.251101  6751 raft_consensus.cc:2955] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:42.258349  5557 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "832269e5980743c48521740b22eefa9f"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:47734
I20260709 12:24:42.258961  5557 raft_consensus.cc:606] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:42.259914  6278 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:47932
I20260709 12:24:42.260716  6278 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:42.261240  6278 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.275774  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:42.276542  5703 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:42.276868  5703 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:42.277652  5703 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.280925  5703 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.287746  5703 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:42.326890  5446 catalog_manager.cc:5028] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): tablet 6adc8ae91dfe46039ab5bd2bc4c69708 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:42.333216  6278 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.335405  6278 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:42.339083  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:42.340874  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:42.341775  5703 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.342957  5553 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:42.343375  5553 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:42.344251  5553 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.346705  5628 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8c6474db7a70405694509f6976d850ce"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:24:42.347314  5628 raft_consensus.cc:606] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Received request to transfer leadership to TS 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:42.351169  6713 raft_consensus.cc:993] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:42.352062  6765 raft_consensus.cc:1081] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:42.338646  6276 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.355361  5553 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.357091  6276 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:42.366321  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:42.367049  5705 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.378167  5703 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:42.379798  5630 raft_consensus.cc:2955] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:42.383234  5705 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:42.379829  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:42.387830  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:42.388751  6811 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:42.391685  6891 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:42.394335  5554 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:47734
I20260709 12:24:42.395121  6891 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:42.395216  5554 raft_consensus.cc:606] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:42.399477  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:24:42.402570  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:42.389271  6811 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:42.405227  6811 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.409303  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:42.409942  5628 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:42.410241  5628 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:42.409376  5632 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "832269e5980743c48521740b22eefa9f"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:35614
I20260709 12:24:42.413134  5632 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:42.413713  5632 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.414675  5628 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.396590  6891 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.456648  6892 raft_consensus.cc:993] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:42.457417  6754 raft_consensus.cc:1081] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Signalling peer 38736cc545eb4389b220a3bd21d3b466 to start an election
I20260709 12:24:42.465561  6278 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8c6474db7a70405694509f6976d850ce"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
 from {username='slave'} at 127.0.0.1:47920
I20260709 12:24:42.466292  6278 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:42.466652  6278 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.499754  6880 raft_consensus.cc:993] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:42.500543  6745 raft_consensus.cc:1081] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:42.558609  5634 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:35614
I20260709 12:24:42.559273  5634 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:42.559821  5634 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.567261  5634 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.571497  6276 raft_consensus.cc:2955] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:42.583416  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:42.589026  5619 tablet_service.cc:1558] Processing DeleteTablet for tablet 14483990837441a99fa83b4a981b4162 with delete_type TABLET_DATA_TOMBSTONED (TS 4a1c5e6dd8e249bf82d38f90fa37b43d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35612
I20260709 12:24:42.589112  5628 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.583384  5632 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:42.627549  5628 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:42.627770  6278 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.555866  5553 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:42.629335  5634 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:42.630520  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:42.631335  6277 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.636541  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:42.637770  5554 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:42.638833  5554 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:42.640267  5554 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.642635  5554 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.648375  5632 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:42.648919  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:42.648548  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:42.650085  5709 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.654098  6277 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:42.659456  6278 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:42.663743  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:42.664577  5703 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.657562  5601 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:24:42.662081  6905 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.072s	user 0.004s	sys 0.000s
I20260709 12:24:42.672649  5554 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:42.669296  6881 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:42.678431  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:42.681556  5634 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:42.681943  5634 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:42.683596  5703 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:42.684973  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:24:42.691565  6903 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.150s	user 0.002s	sys 0.000s
I20260709 12:24:42.691881  6881 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:42.694684  5553 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:42.695785  5553 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:42.699687  5634 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.701959  5709 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:42.700166  5455 catalog_manager.cc:5697] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:42.703248  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:42.709556  5634 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.711390  6754 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:42.703559  6876 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:42.722385  5456 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 1 to 2, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 2 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:42.738121  5553 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:42.741628  5553 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:42.746927  6876 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:42.762467  6876 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.745261  6906 tablet_replica.cc:333] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:24:42.767360  5634 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:42.793047  6906 raft_consensus.cc:2243] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:42.793920  6906 raft_consensus.cc:2272] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:42.767225  6881 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:42.833163  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 127322us
I20260709 12:24:42.841750  5557 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:42.847669  6906 ts_tablet_manager.cc:1916] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:24:42.849145  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 75321.6us
I20260709 12:24:42.857156  6883 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:42.859766  5703 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:42.867193  6278 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:42.870399  6871 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:42.881225  5553 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
W20260709 12:24:42.883312  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 109346us
I20260709 12:24:42.889233  5557 raft_consensus.cc:2955] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:42.896055  6278 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:42.899075  5630 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:42.882495  6754 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:42.902014  6754 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:42.903999  5455 catalog_manager.cc:5697] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 2 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:42.908416  5455 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 2, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 2 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:42.922637  5705 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:42.942092  5628 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:42.944125  6811 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:42.953739  6878 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:42.961609  5703 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:42.963209  6811 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:24:42.984207  5632 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:42.989392  6906 ts_tablet_manager.cc:1929] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:42.989897  6906 log.cc:1199] T 14483990837441a99fa83b4a981b4162 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/wals/14483990837441a99fa83b4a981b4162
I20260709 12:24:42.991575  5445 catalog_manager.cc:5028] TS 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): tablet 14483990837441a99fa83b4a981b4162 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:42.999373  5457 catalog_manager.cc:5697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:43.001471  6745 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:43.001191  5457 catalog_manager.cc:5697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:43.003963  6912 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:43.030112  6912 raft_consensus.cc:2955] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:43.029732  5455 catalog_manager.cc:5697] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 2, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 2 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:43.033996  5630 raft_consensus.cc:2955] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:43.050850  6276 raft_consensus.cc:2955] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:43.054607  5443 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1af549c25a724a2f9d4f65cc1c59f22a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:43.096995  5456 catalog_manager.cc:5697] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 3 to 4, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:43.119228  5550 tablet_service.cc:1558] Processing DeleteTablet for tablet 1af549c25a724a2f9d4f65cc1c59f22a with delete_type TABLET_DATA_TOMBSTONED (TS cea66aa15be3423880843974acdbf50f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49014
I20260709 12:24:43.142256  6928 tablet_replica.cc:333] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:24:43.143488  6928 raft_consensus.cc:2243] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:43.144317  6928 raft_consensus.cc:2272] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:43.150991  6928 ts_tablet_manager.cc:1916] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:43.173867  6928 ts_tablet_manager.cc:1929] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:43.174396  6928 log.cc:1199] T 1af549c25a724a2f9d4f65cc1c59f22a P cea66aa15be3423880843974acdbf50f: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/wals/1af549c25a724a2f9d4f65cc1c59f22a
I20260709 12:24:43.176431  5446 catalog_manager.cc:5028] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): tablet 1af549c25a724a2f9d4f65cc1c59f22a (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:43.228628  6276 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:43.230203  6881 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:43.234059  5628 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:43.235708  6883 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:24:43.269568  5703 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:43.272817  6908 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:43.313947  5561 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:43.320189  6881 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:24:43.391458  5561 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:43.393060  6881 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:43.404996  5703 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:43.407171  6875 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:43.707576  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:43.770257  5554 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:47734
I20260709 12:24:43.770882  5554 raft_consensus.cc:606] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:43.773765  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:24:43.774317  5630 raft_consensus.cc:606] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:43.777056  5703 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:24:43.777776  5703 raft_consensus.cc:606] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:43.779290  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:24:43.780911  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:43.781685  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:43.900139  6912 raft_consensus.cc:993] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:43.900772  6912 raft_consensus.cc:1081] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:43.904896  5628 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:35630
I20260709 12:24:43.905676  5628 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:43.906157  5628 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:43.914126  5628 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:43.919932  5628 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:43.920578  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:43.920869  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:43.921211  5705 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:43.921461  6276 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:43.921571  5705 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:43.922763  5705 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:43.924129  5705 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:43.930378  6276 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:43.931740  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:24:43.932673  6875 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:43.933164  6875 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:43.934062  6875 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:43.947124  5705 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:43.953423  5457 catalog_manager.cc:5697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 2, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:43.994925  6907 raft_consensus.cc:993] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:24:43.995502  6875 raft_consensus.cc:1081] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Signalling peer f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:24:43.997257  5705 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:38468
I20260709 12:24:43.997723  5705 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:43.998010  5705 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:44.006471  5705 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:44.008050  5705 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:44.008860  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:44.009459  5553 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:44.010030  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:44.010655  5628 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:44.010977  5628 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:44.011770  5628 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:44.013092  5628 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:44.015017  5553 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:24:44.016690  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:44.017446  6912 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:44.018158  6912 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:44.018965  6912 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:44.024782  5628 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:24:44.035936  5457 catalog_manager.cc:5697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 2 to 3, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 3 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:44.085219  6871 raft_consensus.cc:993] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:44.085807  6871 raft_consensus.cc:1081] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:44.087611  5628 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:35614
I20260709 12:24:44.088217  5628 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:44.088629  5628 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:44.097443  5628 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:44.099560  5628 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:44.100392  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:44.100682  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:44.101016  5553 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:44.101209  6276 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:44.101368  5553 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:44.102401  5553 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:44.106325  5553 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:44.108486  6276 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:44.112092  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:24:44.112936  6907 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:44.113407  6907 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:44.114393  6907 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:44.114779  5553 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:44.132598  5457 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:44.342542  5703 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
W20260709 12:24:44.385598  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 73395.2us
I20260709 12:24:44.400626  5628 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:44.403086  6912 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 12:24:44.422314  5553 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:44.423914  6912 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:24:44.425985  5675 consensus_peers.cc:597] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccec0cc7269743d6ad581695ea601d1f. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:44.469544  6935 raft_consensus.cc:2955] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:44.490147  5708 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:44.494125  6881 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:24:44.505512  5630 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:24:44.508948  6911 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:44.509514  5553 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:44.508698  5630 raft_consensus.cc:2955] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:44.513291  6881 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 12:24:44.543807  6277 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:44.556713  6277 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:44.562613  6907 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:24:44.588970  6942 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:24:44.674705  5561 raft_consensus.cc:2955] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:44.684237  5630 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:44.724601  5457 catalog_manager.cc:5697] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:44.788472  5561 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:24:44.807824  6958 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:44.824699  5703 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:44.826712  6968 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:44.834371  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:24:44.845149  5601 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2acd6f5df0204b2295547e021cd8cb9f. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:24:44.877316  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 68251.3us
I20260709 12:24:44.882076  5553 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:44.915046  6942 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:44.922251  6875 raft_consensus.cc:2955] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:44.949831  5703 raft_consensus.cc:2955] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:44.974218  6979 ts_tablet_manager.cc:933] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:44.973299  5557 raft_consensus.cc:2955] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:44.991093  5704 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:45.014237  5458 catalog_manager.cc:5697] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:45.024606  6979 tablet_copy_client.cc:323] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.3:45979
I20260709 12:24:45.025530  5630 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:45.026639  5628 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:24:45.026988  5720 tablet_copy_service.cc:140] P f613ba929b1046d3b5aed909414a95bf: Received BeginTabletCopySession request for tablet ccec0cc7269743d6ad581695ea601d1f from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:42568)
I20260709 12:24:45.027376  5628 raft_consensus.cc:606] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:24:45.027690  5720 tablet_copy_service.cc:161] P f613ba929b1046d3b5aed909414a95bf: Beginning new tablet copy session on tablet ccec0cc7269743d6ad581695ea601d1f from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:42568: session id = 38736cc545eb4389b220a3bd21d3b466-ccec0cc7269743d6ad581695ea601d1f
W20260709 12:24:45.034875  5675 consensus_peers.cc:597] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0f7b2d2a7e34d629491c96dbfbc7fdc. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:45.035660  6958 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:45.037559  5628 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:24:45.038149  5628 raft_consensus.cc:606] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:24:45.041584  5708 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:24:45.042145  5708 raft_consensus.cc:606] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:45.047483  5628 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:24:45.048167  5628 raft_consensus.cc:606] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:45.049970  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:24:45.051771  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:45.052901  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:45.055154  5720 tablet_copy_source_session.cc:215] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:45.061635  5553 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:45.064404  6911 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:45.069643  6979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccec0cc7269743d6ad581695ea601d1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:45.085469  6958 raft_consensus.cc:2955] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:45.093891  5630 raft_consensus.cc:2955] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:45.095223  6979 tablet_copy_client.cc:806] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:45.096058  6979 tablet_copy_client.cc:670] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:45.117167  5704 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:45.125455  6979 tablet_copy_client.cc:538] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:45.127049  5456 catalog_manager.cc:5697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:45.149505  5628 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:45.149722  6979 tablet_bootstrap.cc:492] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:45.151491  5553 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:24:45.154773  6911 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:45.158638  6911 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:24:45.166033  5675 consensus_peers.cc:597] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26edf338d41e42f1a5ce0d4416ec67e1. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:24:45.185038  6967 raft_consensus.cc:2955] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:45.187516  5553 raft_consensus.cc:2955] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:45.201458  5557 raft_consensus.cc:2955] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:45.205183  6979 tablet_bootstrap.cc:492] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:45.206269  6979 tablet_bootstrap.cc:492] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:45.208244  6979 ts_tablet_manager.cc:1403] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.059s	user 0.024s	sys 0.021s
I20260709 12:24:45.211979  5630 raft_consensus.cc:2955] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:45.227975  6979 raft_consensus.cc:359] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:45.228931  6979 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:45.235082  6979 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:45.237377  5456 catalog_manager.cc:5697] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index -1 to 2, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:24:45.239289  6979 ts_tablet_manager.cc:1434] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.030s	user 0.005s	sys 0.001s
I20260709 12:24:45.241144  5720 tablet_copy_service.cc:342] P f613ba929b1046d3b5aed909414a95bf: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-ccec0cc7269743d6ad581695ea601d1f received from {username='slave'} at 127.0.0.1:42568
I20260709 12:24:45.241658  5720 tablet_copy_service.cc:434] P f613ba929b1046d3b5aed909414a95bf: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-ccec0cc7269743d6ad581695ea601d1f on tablet ccec0cc7269743d6ad581695ea601d1f with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:45.253218  6942 raft_consensus.cc:993] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:24:45.253788  6942 raft_consensus.cc:1081] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Signalling peer cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:24:45.256496  5557 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:49044
I20260709 12:24:45.257334  5557 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:45.259205  5557 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:45.287179  6875 raft_consensus.cc:993] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:24:45.287781  6875 raft_consensus.cc:1081] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Signalling peer f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:24:45.290337  5704 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:38468
I20260709 12:24:45.290881  5704 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:45.291246  5704 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:45.299311  5704 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:45.309011  6967 raft_consensus.cc:993] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:45.309664  6911 raft_consensus.cc:1081] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:45.305701  6979 ts_tablet_manager.cc:933] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:45.303210  5557 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:45.310144  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:45.310411  5704 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:45.310845  5553 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:45.311919  5557 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:45.313539  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:45.314523  5630 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:35630
I20260709 12:24:45.314702  6277 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:45.314760  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:45.315089  5630 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:45.315397  5634 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:45.315495  5630 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:45.315737  5634 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:45.316602  5634 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:45.317685  5634 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:45.322441  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:45.323086  5628 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:45.323386  5628 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:45.324244  5553 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:24:45.324705  6277 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:24:45.331298  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:45.332199  6967 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:45.333030  6967 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:45.333928  6967 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:45.326803  5628 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:45.331185  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:45.343211  5628 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:45.343923  6889 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:45.344527  6889 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:45.345490  6889 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:45.329092  5634 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:24:45.354773  5458 catalog_manager.cc:5697] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 2 to 3, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 3 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:45.363672  5628 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:24:45.367537  5630 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:45.370930  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:45.371536  5553 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:45.363693  5456 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 2 to 3, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 3 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:45.378162  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:45.378733  5704 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:45.379019  5704 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:45.380018  5704 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:45.381676  5704 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:45.389561  6979 tablet_copy_client.cc:323] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.2:38587
I20260709 12:24:45.391458  5646 tablet_copy_service.cc:140] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Received BeginTabletCopySession request for tablet 2acd6f5df0204b2295547e021cd8cb9f from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:40390)
I20260709 12:24:45.392014  5646 tablet_copy_service.cc:161] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Beginning new tablet copy session on tablet 2acd6f5df0204b2295547e021cd8cb9f from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:40390: session id = 38736cc545eb4389b220a3bd21d3b466-2acd6f5df0204b2295547e021cd8cb9f
I20260709 12:24:45.397497  5553 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:45.398335  5630 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:45.402199  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:45.402951  5646 tablet_copy_source_session.cc:215] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:45.403700  6875 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:45.404219  6875 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:45.405102  6875 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:45.413702  6979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2acd6f5df0204b2295547e021cd8cb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:45.417891  6277 raft_consensus.cc:1217] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:45.426775  5704 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:45.445976  5458 catalog_manager.cc:5697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 2, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:45.446710  6979 tablet_copy_client.cc:806] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:45.450029  6979 tablet_copy_client.cc:670] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:45.467676  6979 tablet_copy_client.cc:538] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:45.484582  6979 tablet_bootstrap.cc:492] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:45.506909  6979 tablet_bootstrap.cc:492] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:45.507757  6979 tablet_bootstrap.cc:492] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:45.508327  6979 ts_tablet_manager.cc:1403] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.024s	user 0.018s	sys 0.005s
I20260709 12:24:45.510522  6979 raft_consensus.cc:359] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:45.511173  6979 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:45.511835  6979 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:45.515311  6979 ts_tablet_manager.cc:1434] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.003s
I20260709 12:24:45.517731  5646 tablet_copy_service.cc:342] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-2acd6f5df0204b2295547e021cd8cb9f received from {username='slave'} at 127.0.0.1:40390
I20260709 12:24:45.518187  5646 tablet_copy_service.cc:434] P 4a1c5e6dd8e249bf82d38f90fa37b43d: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-2acd6f5df0204b2295547e021cd8cb9f on tablet 2acd6f5df0204b2295547e021cd8cb9f with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:45.559978  6942 raft_consensus.cc:993] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:24:45.560546  6942 raft_consensus.cc:1081] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Signalling peer cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:24:45.574663  5557 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:49044
I20260709 12:24:45.575233  5557 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:45.575634  5557 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:45.594527  5557 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:45.605834  5557 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:45.606783  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:45.608592  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:45.610446  5704 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:45.610689  5630 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:45.611001  5630 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:45.611747  5630 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:45.614032  5630 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:45.620057  5704 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:45.621485  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:45.622440  6991 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:45.622915  6991 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:45.623864  6991 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:45.634135  5630 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:45.651093  6979 ts_tablet_manager.cc:933] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:45.657043  5458 catalog_manager.cc:5697] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 1 to 2, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 2 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:45.673684  6979 tablet_copy_client.cc:323] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.3:45979
I20260709 12:24:45.675650  5720 tablet_copy_service.cc:140] P f613ba929b1046d3b5aed909414a95bf: Received BeginTabletCopySession request for tablet f0f7b2d2a7e34d629491c96dbfbc7fdc from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:42568)
I20260709 12:24:45.676211  5720 tablet_copy_service.cc:161] P f613ba929b1046d3b5aed909414a95bf: Beginning new tablet copy session on tablet f0f7b2d2a7e34d629491c96dbfbc7fdc from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:42568: session id = 38736cc545eb4389b220a3bd21d3b466-f0f7b2d2a7e34d629491c96dbfbc7fdc
I20260709 12:24:45.686035  5720 tablet_copy_source_session.cc:215] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:45.690470  6979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0f7b2d2a7e34d629491c96dbfbc7fdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:45.712661  6979 tablet_copy_client.cc:806] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:45.713346  6979 tablet_copy_client.cc:670] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:45.719695  6277 raft_consensus.cc:1217] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:45.771229  6979 tablet_copy_client.cc:538] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:45.785643  5557 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:24:45.788782  6979 tablet_bootstrap.cc:492] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:45.790613  6958 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:45.800410  6277 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:24:45.806133  6871 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:24:45.832197  5708 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:45.835005  5630 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:24:45.834817  6968 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:45.837060  6966 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:45.889218  6997 ts_tablet_manager.cc:933] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:45.903894  5628 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:24:45.915741  6997 tablet_copy_client.cc:323] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.3:45979
I20260709 12:24:45.920615  5720 tablet_copy_service.cc:140] P f613ba929b1046d3b5aed909414a95bf: Received BeginTabletCopySession request for tablet 26edf338d41e42f1a5ce0d4416ec67e1 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:42568)
I20260709 12:24:45.921202  5720 tablet_copy_service.cc:161] P f613ba929b1046d3b5aed909414a95bf: Beginning new tablet copy session on tablet 26edf338d41e42f1a5ce0d4416ec67e1 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:42568: session id = 38736cc545eb4389b220a3bd21d3b466-26edf338d41e42f1a5ce0d4416ec67e1
I20260709 12:24:45.930773  5720 tablet_copy_source_session.cc:215] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:24:45.934476  6936 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:24:45.940162  6958 raft_consensus.cc:1064] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:45.945467  6958 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:45.955552  6979 tablet_bootstrap.cc:492] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:45.960311  6997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26edf338d41e42f1a5ce0d4416ec67e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:24:45.963464  5630 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:45.964603  6279 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:45.967124  6979 tablet_bootstrap.cc:492] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:45.964865  5557 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:45.967860  6979 ts_tablet_manager.cc:1403] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.179s	user 0.025s	sys 0.026s
I20260709 12:24:45.969014  6958 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:45.974434  6966 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 12:24:45.994231  6967 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:46.042331  6979 raft_consensus.cc:359] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:46.043309  6979 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:46.047068  6979 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
W20260709 12:24:46.136564  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 75309.2us
I20260709 12:24:46.144277  5557 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:24:46.165697  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 119652us
I20260709 12:24:46.170208  6979 ts_tablet_manager.cc:1434] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.202s	user 0.011s	sys 0.005s
I20260709 12:24:46.170344  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:24:46.190708  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 141072us
W20260709 12:24:46.216434  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45907.8us
I20260709 12:24:46.228699  6911 raft_consensus.cc:2955] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:46.246717  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 114476us
I20260709 12:24:46.251091  5704 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:46.261337  7020 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
W20260709 12:24:46.265007  7010 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
W20260709 12:24:46.343627  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 163550us
I20260709 12:24:46.357309  6279 raft_consensus.cc:1217] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260709 12:24:46.383219  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 148360us
I20260709 12:24:46.378626  6997 tablet_copy_client.cc:806] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:24:46.390676  5634 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:46.401666  6997 tablet_copy_client.cc:670] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:24:46.406929  6992 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:46.421207  5720 tablet_copy_service.cc:342] P f613ba929b1046d3b5aed909414a95bf: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-f0f7b2d2a7e34d629491c96dbfbc7fdc received from {username='slave'} at 127.0.0.1:42568
I20260709 12:24:46.421888  5720 tablet_copy_service.cc:434] P f613ba929b1046d3b5aed909414a95bf: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-f0f7b2d2a7e34d629491c96dbfbc7fdc on tablet f0f7b2d2a7e34d629491c96dbfbc7fdc with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:46.471674  5553 raft_consensus.cc:2955] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:46.545439  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 156883us
I20260709 12:24:46.554410  5628 raft_consensus.cc:2955] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:46.568251  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48034.7us
I20260709 12:24:46.550765  5457 catalog_manager.cc:5697] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:46.580651  6277 raft_consensus.cc:2955] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:46.582204  6997 tablet_copy_client.cc:538] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:24:46.629447  6992 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:46.633733  6973 raft_consensus.cc:1064] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:46.634936  6997 tablet_bootstrap.cc:492] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:24:46.747735  5553 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:47734
I20260709 12:24:46.769562  6973 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:46.795815  5553 raft_consensus.cc:606] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:24:46.804512  7046 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
W20260709 12:24:46.806824  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 139019us
W20260709 12:24:46.856595  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 98149.4us
I20260709 12:24:46.874035  7031 raft_consensus.cc:1064] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
W20260709 12:24:46.939306  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54509.9us
I20260709 12:24:46.936848  7031 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:46.950075  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61005.7us
W20260709 12:24:47.064517  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 182423us
W20260709 12:24:47.123502  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44196.7us
I20260709 12:24:47.143432  5561 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:47.147192  6276 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:47.149096  7043 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:24:47.150256  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 389254us
I20260709 12:24:47.153132  5631 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:47.254423  7043 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:47.256099  7031 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:47.371078  6997 tablet_bootstrap.cc:492] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:24:47.372280  6997 tablet_bootstrap.cc:492] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:24:47.379419  6997 ts_tablet_manager.cc:1403] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 0.751s	user 0.021s	sys 0.017s
W20260709 12:24:47.414394  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 190585us
I20260709 12:24:47.419474  7070 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:47.420044  7070 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:47.493546  5628 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:24:47.514941  5628 raft_consensus.cc:606] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:47.517109  6276 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 12:24:47.574791  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 92382.9us
I20260709 12:24:47.578420  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:47.578559  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:47.508901  7070 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:47.606694  5704 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:47.517326  6997 raft_consensus.cc:359] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:47.617561  5560 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:47.622138  7078 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260709 12:24:47.615715  6997 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:24:47.641793  6997 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:47.668298  5704 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:24:47.668988  5704 raft_consensus.cc:606] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:24:47.673820  7078 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:47.683738  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:47.686230  7070 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:47.688462  7014 raft_consensus.cc:2955] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:47.694186  6871 raft_consensus.cc:993] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:47.696453  7036 raft_consensus.cc:1081] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:47.712112  5560 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "addbcf2919194d2e866b5e8826422851"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:47734
I20260709 12:24:47.722834  5631 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:35614
I20260709 12:24:47.724455  5631 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:47.726508  5631 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:47.731200  5561 raft_consensus.cc:2955] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:47.721737  5458 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 169) took 1013 ms. Trace:
I20260709 12:24:47.745239  5458 rpcz_store.cc:276] 0709 12:24:46.695404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.695874 (+   470us) service_pool.cc:224] Handling call
0709 12:24:47.708685 (+1012811us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":136714,"reported_tablets":1,"spinlock_wait_cycles":20736}
I20260709 12:24:47.747696  5560 raft_consensus.cc:606] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:47.774297  5457 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 268) took 1293 ms. Trace:
I20260709 12:24:47.775446  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:24:47.777263  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:47.778779  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:47.760737  6278 raft_consensus.cc:2955] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:47.786402  7017 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:47.847692  5631 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:47.865658  5708 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:47.899683  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:47.900416  5704 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:47.774730  5457 rpcz_store.cc:276] 0709 12:24:46.471344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.531070 (+ 59726us) service_pool.cc:224] Handling call
0709 12:24:47.764224 (+1233154us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:47.083442 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:47.084380 (+   938us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:47.084416 (+    36us) write_op.cc:435] Acquired schema lock
0709 12:24:47.084434 (+    18us) tablet.cc:662] Decoding operations
0709 12:24:47.084761 (+   327us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:47.084831 (+    70us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:47.084868 (+    37us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:24:47.085057 (+   189us) tablet.cc:701] Row locks acquired
0709 12:24:47.085075 (+    18us) write_op.cc:260] PREPARE: finished
0709 12:24:47.085204 (+   129us) write_op.cc:270] Start()
0709 12:24:47.085319 (+   115us) write_op.cc:276] Timestamp: P: 1783599887085164 usec, L: 0
0709 12:24:47.085342 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:24:47.085969 (+   627us) log.cc:844] Serialized 587 byte log entry
0709 12:24:47.405812 (+319843us) op_driver.cc:464] REPLICATION: finished
0709 12:24:47.682473 (+276661us) write_op.cc:301] APPLY: starting
0709 12:24:47.682549 (+    76us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:47.682650 (+   101us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:47.682763 (+   113us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:47.682974 (+   211us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:47.683126 (+   152us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:24:47.683169 (+    43us) write_op.cc:312] APPLY: finished
0709 12:24:47.683374 (+   205us) log.cc:844] Serialized 23 byte log entry
0709 12:24:47.684207 (+   833us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:47.684286 (+    79us) write_op.cc:454] Released schema lock
0709 12:24:47.684322 (+    36us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:47.685729 (+  1407us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:47.685762 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":20970,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":276405,"apply.run_cpu_time_us":1542,"apply.run_wall_time_us":3256,"num_ops":1,"prepare.queue_time_us":466519,"prepare.run_cpu_time_us":34504,"prepare.run_wall_time_us":46199,"raft.queue_time_us":224490,"raft.run_cpu_time_us":28219,"raft.run_wall_time_us":218545,"replication_time_us":320312,"thread_start_us":810249,"threads_started":4,"wal-append.queue_time_us":89827}]]}
W20260709 12:24:47.909313  7080 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:24:47.924050  6958 raft_consensus.cc:993] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: : Instructing follower cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:24:47.924743  6997 ts_tablet_manager.cc:1434] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.540s	user 0.010s	sys 0.017s
W20260709 12:24:47.925268  6997 ts_tablet_manager.cc:1467] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Tablet startup took 540ms
I20260709 12:24:47.924780  7070 raft_consensus.cc:1081] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Signalling peer cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:24:47.987516  5555 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:49030
I20260709 12:24:47.988216  5555 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:47.988593  5555 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:24:47.976576  7084 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.061s	user 0.002s	sys 0.000s
I20260709 12:24:48.002957  5555 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:48.026176  5555 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:48.040546  5631 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:48.044921  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:48.045609  5561 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:48.045926  5561 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:48.046728  5561 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:48.048278  5561 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:48.072273  5560 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:24:48.076119  7073 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:24:48.078127  6958 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:24:48.078636  6958 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:48.081333  6958 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:48.112006  6958 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:24:47.925704  6997 ts_tablet_manager.cc:1469] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Trace:
0709 12:24:45.770196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.770587 (+   391us) service_pool.cc:224] Handling call
0709 12:24:45.889635 (+119048us) ts_tablet_manager.cc:934] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
0709 12:24:45.958784 (+ 69149us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:24:46.033641 (+ 74857us) tablet_metadata.cc:665] Metadata flushed
0709 12:24:46.049246 (+ 15605us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:24:46.283539 (+234293us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:24:46.283924 (+   385us) tablet_replica.cc:206] Creating consensus instance
0709 12:24:46.354511 (+ 70587us) ts_tablet_manager.cc:989] Replica registered.
0709 12:24:46.360898 (+  6387us) inbound_call.cc:177] Queueing success response
0709 12:24:46.627982 (+267084us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:24:47.384653 (+756671us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:24:47.386078 (+  1425us) tablet_replica.cc:261] Starting instrumentation
0709 12:24:47.386871 (+   793us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":271,"raft.queue_time_us":1042,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"tablet-copy.queue_time_us":118007,"tablet-download-pool.queue_time_us":893,"tablet-download-pool.run_cpu_time_us":1177,"tablet-download-pool.run_wall_time_us":177403,"thread_start_us":265933,"threads_started":3}
I20260709 12:24:48.088943  5635 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:48.090664  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:48.118036  5634 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:48.099689  6871 raft_consensus.cc:993] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:48.098624  7013 raft_consensus.cc:2955] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:48.103506  6276 raft_consensus.cc:2955] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:48.109417  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 81638.9us
I20260709 12:24:48.131804  5634 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:24:48.135888  5704 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:24:48.153623  5561 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:24:48.155012  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:48.169845  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:48.170463  5705 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:48.176416  5629 raft_consensus.cc:2955] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:48.180186  5720 tablet_copy_service.cc:342] P f613ba929b1046d3b5aed909414a95bf: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-26edf338d41e42f1a5ce0d4416ec67e1 received from {username='slave'} at 127.0.0.1:42568
I20260709 12:24:48.180716  5720 tablet_copy_service.cc:434] P f613ba929b1046d3b5aed909414a95bf: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-26edf338d41e42f1a5ce0d4416ec67e1 on tablet 26edf338d41e42f1a5ce0d4416ec67e1 with peer 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:48.155628  6871 raft_consensus.cc:1081] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:48.197590  7087 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:24:48.198405  7087 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:48.205675  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:48.218483  5634 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "addbcf2919194d2e866b5e8826422851"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:35614
I20260709 12:24:48.219331  5634 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:48.225270  5634 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:24:48.238868  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55435.2us
I20260709 12:24:48.199580  7087 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:48.257084  5634 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:48.297362  5634 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:48.298435  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 58899.5us
I20260709 12:24:48.300856  7088 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:48.301429  7088 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:48.302299  7088 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:48.313560  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:48.314400  5706 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:48.326516  5704 raft_consensus.cc:2955] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:48.442201  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 147048us
W20260709 12:24:48.492710  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.510131  5706 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:48.527057  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:48.530922  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:48.534793  5559 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:48.535115  5559 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:48.535632  6942 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:48.536166  6942 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:48.535919  5559 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:48.537034  6942 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:48.545150  7041 raft_consensus.cc:993] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower f613ba929b1046d3b5aed909414a95bf to start an election
W20260709 12:24:48.550726  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 126968us
I20260709 12:24:48.564077  7041 raft_consensus.cc:1081] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Signalling peer f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:24:48.573478  5703 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
W20260709 12:24:50.010306  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800038620 after lost signal to thread 4359
I20260709 12:24:49.200918  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:50.055018  5457 catalog_manager.cc:5697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:50.107429  5455 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35600 (request call id 272) took 2119 ms. Trace:
I20260709 12:24:50.108049  5455 rpcz_store.cc:276] 0709 12:24:47.987370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.987574 (+   204us) service_pool.cc:224] Handling call
0709 12:24:50.107170 (+2119596us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":1}
I20260709 12:24:50.218600  7023 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:50.219216  7023 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.228009  7023 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.228757  7023 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:50.229342  7023 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.231549  7023 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.232110  7023 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:50.232823  7023 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.234977  7102 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:50.235338  7023 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.238957  7023 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:50.239507  7023 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.239522  7102 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.242511  7102 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.243839  7023 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.243253  7102 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.245795  7023 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.246007  7102 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.246423  7023 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.248761  7102 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.249359  7102 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.249851  7023 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.250396  7076 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.249873  7102 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.250543  7023 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:50.250944  7076 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.251128  7023 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.253084  7023 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.236994  7103 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.253325  7076 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.253562  7102 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.253791  7103 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.256711  7103 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.273082  7104 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.274351  7104 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.278762  7104 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.290087  6991 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:50.290643  6991 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.292974  6991 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.295429  6991 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:50.296000  6991 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.296952  7105 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.297554  7105 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.298785  6991 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.300634  7038 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.301198  7038 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.299855  7105 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.313755  7110 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.315718  7110 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.314342  7105 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.304231  7038 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.319192  7105 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.319854  7038 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:50.321360  7105 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.321951  7105 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.322196  7038 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.323249  7105 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:50.327030  7105 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.324988  7110 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.324745  7038 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.332922  7110 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:50.331060  7106 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.333509  7110 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.333031  7038 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.301174  6991 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.334998  6991 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.334878  7038 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.337129  6991 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.333597  7106 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.338723  7110 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.337126  7038 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.342825  7031 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:50.347560  7031 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.351285  7031 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.344632  7115 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:50.352111  7031 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:50.352325  7115 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.352934  7031 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.356861  7115 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.343900  7102 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:50.358367  7102 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.344312  7023 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.360662  7023 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.343349  7103 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.331888  7112 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:50.344332  7034 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:50.344514  7076 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:24:50.345232  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800034cc0 after lost signal to thread 5493
I20260709 12:24:50.346686  7089 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.357060  7031 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.340164  7106 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.373647  7102 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.379298  7114 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.371170  7087 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.371659  7076 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.384922  7110 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.372951  7038 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:50.388303  7038 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.390774  7038 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.391407  7038 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.386178  7087 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.369148  7103 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.433660  7023 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.435370  7103 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.437631  7105 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.438366  7105 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.437661  7106 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.441186  7106 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.387369  7110 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.380251  7089 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.449391  7089 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.449923  7087 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.449414  7110 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.391981  7038 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.371495  7112 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.401185  7114 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.457243  6936 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:50.461081  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.462505  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.463737  5561 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:50.463826  5557 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:50.452883  6991 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.465724  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.466915  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.467088  7116 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:24:50.468454  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 1.89924e+06us
I20260709 12:24:50.469544  5557 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:50.470304  7107 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.470456  7106 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.460594  7076 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.461962  7105 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.461061  7110 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.459918  6936 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.463440  7087 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.481842  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.374890  7034 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.484517  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.470927  7107 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.485718  7112 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.480370  7087 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.479722  7110 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.486167  6945 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:50.488652  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.487529  6991 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.489895  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.493003  6991 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.467772  7116 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.516899  7106 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:50.519495  6936 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.533661  7117 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:24:50.516877  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.535955  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.492044  7110 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.535878  7114 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.539937  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.541213  5558 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:50.542192  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.542549  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.543092  5554 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:50.543848  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.544898  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.545286  5553 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:50.545807  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.546149  5553 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:50.546653  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.546543  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:50.547502  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.547659  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:50.548071  5553 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:50.538093  7034 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.547392  7023 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:50.548956  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.550007  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.543980  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.540040  5557 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:50.551461  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.551854  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:50.552026  5558 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:50.552894  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.553833  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.552907  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:50.490734  7038 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.550899  7076 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:50.518029  7087 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.490500  6945 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.520193  7107 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.564528  7116 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.564854  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:50.559286  7115 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:50.550938  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.556906  7117 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.557706  7031 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:50.560213  7076 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.570377  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:50.572768  7115 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.572876  7034 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:50.573026  5552 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:50.555913  7106 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.569378  7066 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:50.591229  7066 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.591701  7066 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:50.568949  7038 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:24:50.574071  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.948s (SENT) [suppressed 3 similar messages]
I20260709 12:24:50.590356  7114 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:50.583876  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:50.579949  7076 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:50.596558  6958 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:50.580914  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:24:50.590837  7099 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.430s	user 0.007s	sys 0.005s
I20260709 12:24:50.580888  7115 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:50.578179  7031 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.579761  7102 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:50.579643  6945 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.582588  7117 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.596988  7112 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:50.597013  6958 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.593381  7114 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.581120  7034 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.615170  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.619072  7111 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:50.619866  7111 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.621378  7034 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:50.605284  7031 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:50.613807  7102 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.626377  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.616688  7023 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.616905  6958 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:50.618613  7114 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:50.629752  7116 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.635890  7110 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:50.616304  7073 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.622825  7111 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:50.597399  7038 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.645555  7106 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.636202  7116 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.636698  7023 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:50.657397  5703 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1475) took 2491 ms (client timeout 1948 ms). Trace:
I20260709 12:24:50.620745  7112 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:50.657887  5703 rpcz_store.cc:269] 0709 12:24:48.166336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.169627 (+  3291us) service_pool.cc:224] Handling call
0709 12:24:50.657192 (+2487565us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b5400323f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:50.657295 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":874716928}
I20260709 12:24:50.646456  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1471) took 2608 ms. Trace:
I20260709 12:24:50.646522  7038 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.647174  7120 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.640861  7110 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.648036  7073 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.652949  7116 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.689941  5705 rpcz_store.cc:276] 0709 12:24:48.038094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.038260 (+   166us) service_pool.cc:224] Handling call
0709 12:24:48.148584 (+110324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:48.170856 (+ 22272us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.646242 (+2475386us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54000e01d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:50.646375 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":252937088}
I20260709 12:24:50.682575  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.710053  5552 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:50.708542  7119 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:24:50.712082  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 138036us
I20260709 12:24:50.711265  7119 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.714900  7023 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.719497  7115 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.659384  7112 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.726249  7023 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.657378  7102 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.688161  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:50.737497  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 139583us
W20260709 12:24:50.746448  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.747583  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.726459  7110 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.688635  7122 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.745711  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.720093  7066 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.715754  7120 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.727815  7023 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:50.765219  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.766098  5703 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:50.763881  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.768735  6958 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.777158  7119 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.779458  6958 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.751222  7122 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.780408  7115 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.782850  7031 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.784207  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.784687  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.785283  5706 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:50.785708  7031 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.786568  7034 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.788321  7114 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:50.788502  7100 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.009s	sys 0.008s
I20260709 12:24:50.789438  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:50.790273  7120 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.790499  7023 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.799973  7034 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:50.791932  7100 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.010s	sys 0.008s
I20260709 12:24:50.802166  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.805397  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.805438  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.806365  5705 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:50.806084  5708 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:50.810640  7087 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.810832  7107 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.811198  7087 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.812310  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.811429  7107 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:50.812882  7066 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.813236  7118 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.815544  5708 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:50.815148  7104 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.815382  6936 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.816540  7104 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.816857  6936 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.818663  7104 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.818985  7087 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.819423  6936 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.813736  7118 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.809271  7076 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.822484  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:50.824168  5558 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:50.814042  7073 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.829305  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.829329  7122 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:50.830927  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.831506  5708 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:50.824592  7114 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.825639  7107 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.832741  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.834126  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.834688  5708 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:50.827812  7076 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.825251  7118 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:50.835850  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.837719  5708 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:50.838953  5558 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:50.840134  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:50.842509  7083 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:50.835417  7111 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:50.849085  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800325920 after lost signal to thread 5585
W20260709 12:24:50.851508  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800303d20 after lost signal to thread 5586
W20260709 12:24:50.852656  5490 debug-util.cc:398] Leaking SignalData structure 0x7b080027b020 after lost signal to thread 5660
W20260709 12:24:50.852943  5490 debug-util.cc:398] Leaking SignalData structure 0x7b080028f0c0 after lost signal to thread 5661
W20260709 12:24:50.851533  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 2.2975e+06us
I20260709 12:24:50.854421  6271 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:50.855786  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.857385  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.858438  5703 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:38468
I20260709 12:24:50.860083  5703 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:50.860528  5703 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:50.869210  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:50.877514  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:50.879081  7111 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:50.856376  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800342600 after lost signal to thread 5734
W20260709 12:24:50.881654  5490 debug-util.cc:398] Leaking SignalData structure 0x7b08001695a0 after lost signal to thread 6304
W20260709 12:24:50.882053  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800169400 after lost signal to thread 6305
I20260709 12:24:50.883529  5709 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:50.884418  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:50.885290  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:50.885833  5555 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:50.887336  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:50.888955  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:50.890341  5703 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:50.891252  5559 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:50.892086  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.892903  6272 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:50.893456  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.893932  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:50.894546  5561 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:50.894651  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:50.895799  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.894008  5553 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:50.899453  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:50.900024  5558 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:24:50.900089  5703 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.855301  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.855477  7083 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:50.903149  5561 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:50.904923  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:50.905463  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:50.906474  5556 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:50.907743  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:50.909628  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:50.910177  6280 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:50.910228  7023 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:24:50.911446  7023 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:50.911594  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:50.911829  7023 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:50.911605  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.910276  7070 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:50.919669  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.918826  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.922315  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.923264  7127 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:50.928013  6277 raft_consensus.cc:1217] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:24:50.924508  7083 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:50.924957  7076 raft_consensus.cc:2955] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:50.918136  7070 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.932173  5558 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:50.933554  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.934743  7070 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:50.939989  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.939823  7023 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:50.940660  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.940686  6276 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:50.942358  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.942528  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.943188  6276 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:50.944753  6273 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:50.946463  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.947009  6277 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:50.949846  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:50.950459  5560 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:50.947790  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.927932  7127 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:50.945374  7023 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:50.953504  7066 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:24:50.957185  5490 debug-util.cc:398] Leaking SignalData structure 0x7b080013e500 after lost signal to thread 6992
W20260709 12:24:50.957546  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800169580 after lost signal to thread 6997
W20260709 12:24:50.958017  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800243f00 after lost signal to thread 7043
W20260709 12:24:50.958305  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800243f80 after lost signal to thread 7046
W20260709 12:24:50.958770  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800243de0 after lost signal to thread 7077
I20260709 12:24:50.958881  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:24:50.959133  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800234fc0 after lost signal to thread 7088
I20260709 12:24:50.960525  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:50.961222  7118 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:50.961609  7118 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.961915  7034 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:50.962551  7034 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.962980  7034 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:50.963259  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.964062  6279 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:50.965652  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:50.969362  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:50.970285  7096 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:50.972100  7114 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:50.972995  5457 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 171) took 3007 ms. Trace:
I20260709 12:24:50.980055  5555 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:50.973510  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:50.988953  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:50.978854  5457 rpcz_store.cc:276] 0709 12:24:47.965932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.966083 (+   151us) service_pool.cc:224] Handling call
0709 12:24:50.972919 (+3006836us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:50.180151 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:50.180785 (+   634us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:50.180823 (+    38us) write_op.cc:435] Acquired schema lock
0709 12:24:50.180843 (+    20us) tablet.cc:662] Decoding operations
0709 12:24:50.181229 (+   386us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:50.181306 (+    77us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:50.181343 (+    37us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:24:50.181534 (+   191us) tablet.cc:701] Row locks acquired
0709 12:24:50.181553 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:24:50.181719 (+   166us) write_op.cc:270] Start()
0709 12:24:50.181835 (+   116us) write_op.cc:276] Timestamp: P: 1783599890181668 usec, L: 0
0709 12:24:50.181860 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 12:24:50.182799 (+   939us) log.cc:844] Serialized 587 byte log entry
0709 12:24:50.713084 (+530285us) spinlock_profiling.cc:243] Waited 62 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ff75680ac 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b
0709 12:24:50.880495 (+167411us) op_driver.cc:464] REPLICATION: finished
0709 12:24:50.934662 (+ 54167us) write_op.cc:301] APPLY: starting
0709 12:24:50.934740 (+    78us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:50.934834 (+    94us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:50.934855 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:50.935075 (+   220us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:50.935224 (+   149us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:24:50.935264 (+    40us) write_op.cc:312] APPLY: finished
0709 12:24:50.935472 (+   208us) log.cc:844] Serialized 23 byte log entry
0709 12:24:50.937099 (+  1627us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:50.937189 (+    90us) write_op.cc:454] Released schema lock
0709 12:24:50.937225 (+    36us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:50.952438 (+ 15213us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:50.955375 (+  2937us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1558201,"reported_tablets":1,"spinlock_wait_cycles":55552,"child_traces":[["op",{"apply.queue_time_us":45328,"apply.run_cpu_time_us":1484,"apply.run_wall_time_us":14024,"num_ops":1,"prepare.queue_time_us":119315,"prepare.run_cpu_time_us":20811,"prepare.run_wall_time_us":629259,"raft.queue_time_us":43476,"raft.run_cpu_time_us":19163,"raft.run_wall_time_us":28485,"replication_time_us":698474,"spinlock_wait_cycles":142508416,"thread_start_us":810028,"threads_started":4,"wal-append.queue_time_us":607787}]]}
I20260709 12:24:50.989795  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:50.991416  7072 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:50.961954  7118 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:50.997589  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:50.975135  5558 raft_consensus.cc:2955] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:51.012164  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.976758  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:50.994896  7014 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.991945  7072 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.042289  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:51.013305  7119 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:51.051021  7119 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.051172  7116 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:51.051478  7119 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:51.051596  7116 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.052877  5555 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:47734
I20260709 12:24:51.053481  5555 raft_consensus.cc:606] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:50.979926  7114 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.043381  7019 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:51.046126  7122 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:24:51.048650  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0920137110001633s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1515s for a response
W20260709 12:24:51.081920  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0668137110001226s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1543s for a response
I20260709 12:24:51.031121  7066 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:51.049144  7023 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.049482  7092 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:51.049800  7103 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.049949  7082 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:51.050040  7102 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:50.998522  7120 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:51.005668  7089 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:51.053223  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:24:51.082298  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0632137110001167s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1547s for a response
W20260709 12:24:51.103477  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0623137110001153s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15480000000000002s for a response
I20260709 12:24:51.040493  6958 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:24:51.104017  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0596137110001109s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
I20260709 12:24:51.005558  7073 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:24:51.103533  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 50408.9us
I20260709 12:24:51.043702  7014 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.052225  7116 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:51.083050  7114 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:51.098177  7023 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.096526  7122 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:24:51.106731  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0425137110000833s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15700000000000003s for a response
I20260709 12:24:51.108281  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:24:51.112666  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0425137110000833s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15700000000000003s for a response
I20260709 12:24:51.114005  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:50.996116  7127 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.104492  7066 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:51.098899  7019 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.103176  7089 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.118363  5458 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 271) took 2618 ms. Trace:
I20260709 12:24:51.107245  7111 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:51.108775  7082 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:51.124930  5458 rpcz_store.cc:276] 0709 12:24:48.499841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.500096 (+   255us) service_pool.cc:224] Handling call
0709 12:24:51.118313 (+2618217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1584072,"reported_tablets":3}
I20260709 12:24:51.109004  7092 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.116235  7038 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:51.104192  7103 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.117501  7110 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:51.005927  7096 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:51.136708  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:51.105981  7102 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.110003  6958 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.143465  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:51.125634  7089 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:51.110631  7073 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.107708  7120 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.129781  7082 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:51.128580  7111 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:51.151736  5559 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:51.157467  7132 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:51.142438  6942 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:24:51.161962  5559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1298) took 2630 ms (client timeout 1573 ms). Trace:
W20260709 12:24:51.162535  5559 rpcz_store.cc:269] 0709 12:24:48.527742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.527881 (+   139us) service_pool.cc:224] Handling call
0709 12:24:50.452920 (+1925039us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b180007b2a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6ad646 00007f4ffa5e758e 00007f4ffa668f9e 00007f4ffa66a7de 00007f4ffa67627c 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c
0709 12:24:50.891049 (+438129us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b5800074fd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5ce0aa 00007f4ffa650447 00007f4ffa668fe7 00007f4ffa66a7de 00007f4ffa67627c 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c
0709 12:24:51.158453 (+267404us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5254548608}
I20260709 12:24:51.147964  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:51.160060  7111 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:51.117374  7101 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260709 12:24:51.170950  7134 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:51.167919  7088 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:51.186175  7132 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.193379  7019 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:51.193922  7019 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:51.194262  7019 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:51.194511  7132 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.192701  7014 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:51.195078  7133 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.195256  7019 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.199000  7023 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:51.200960  7072 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.201392  7078 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.202054  7078 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.202507  7072 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.203088  7072 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.181699  7053 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.179602  7115 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:51.166355  6942 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:51.214303  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.246235  5560 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:24:51.160954  7122 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.204540  7130 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.174719  7066 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:51.262351  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 114211us
I20260709 12:24:51.253641  7013 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:51.251089  7041 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:51.267060  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:51.273041  6279 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:51.268618  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.269549  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.271384  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.275504  5560 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:51.221760  7053 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.199060  7133 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.278898  7053 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.279649  7133 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.211725  7134 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.195282  7138 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.119s
I20260709 12:24:51.209457  7088 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.290531  6942 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.291152  7102 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:51.291569  7136 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:51.281777  7013 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.254703  7092 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.298305  7137 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.296665  7132 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:51.299257  7034 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.286509  7122 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:51.327198  7096 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:51.321872  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 178539us
I20260709 12:24:51.324862  7071 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:51.325908  7075 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.345794  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:51.303414  5455 catalog_manager.cc:5697] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:51.272954  7078 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.353147  5675 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:51.339689  7072 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.354843  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:51.285014  7130 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.339857  6958 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:51.320972  7070 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.347750  7030 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:51.349210  6942 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:51.351509  7120 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:51.343832  7137 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.349499  7073 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:51.355212  7138 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:51.337392  7132 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.358541  7023 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:51.387701  7023 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:51.388213  7023 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:51.340458  7136 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.376616  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:51.383414  5703 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:51.286451  7041 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.366809  7096 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.372009  5629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:24:51.388615  6275 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:51.404172  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:51.413057  5709 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:51.415613  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:51.371245  7030 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.415881  7145 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.419667  5703 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:51.373574  6942 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.425743  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.427449  5634 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:51.358661  7075 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.397213  7143 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:51.409955  7127 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:51.433406  5708 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
W20260709 12:24:51.421088  5556 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.514s	user 0.006s	sys 0.006s
I20260709 12:24:51.435523  7034 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.434301  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.440732  7097 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 12:24:51.441207  7070 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:51.426793  6958 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:51.399459  7144 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:51.359241  7071 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:51.444875  7139 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:24:51.476153  7030 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.430259  7073 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:51.436270  7013 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:24:51.475189  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 215362us
I20260709 12:24:51.416464  7145 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.429816  7134 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.405970  7096 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.438442  7078 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.421842  5629 raft_consensus.cc:606] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:51.442907  7127 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:51.447435  5556 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:24:51.489926  7137 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.469216  7131 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:51.498100  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.494879  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.445242  7141 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.076s
I20260709 12:24:51.498799  7088 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.501773  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.495107  7127 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:24:51.517575  7140 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 12:24:51.518749  7096 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.518929  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.474603  7144 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:51.520452  7144 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:51.521200  6942 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.521529  7116 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.478695  7072 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:51.502544  7145 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:51.528651  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.530710  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.539798  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.540761  5710 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:51.531394  5635 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:51.549777  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.479090  7147 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:51.487130  7078 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.553930  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.437186  7082 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.554828  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.554948  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.558406  7082 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.559825  5705 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:51.541370  7053 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:24:51.564721  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.442771  7143 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.567023  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.567206  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.549443  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.529831  7072 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.549672  7076 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:38456 (request call id 77) took 3681 ms. Trace:
I20260709 12:24:51.572573  5710 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:24:51.573753  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.577389  5709 raft_consensus.cc:2368] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4.
I20260709 12:24:51.578236  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.578852  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.582034  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.580314  7136 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.585568  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:51.582289  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.588498  5558 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:51.589278  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.589977  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.590440  6271 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:51.573836  7132 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.567189  7130 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.591596  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.593879  5561 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:51.596517  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.570811  7053 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.563963  7131 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.562579  7147 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.598034  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.582684  5556 raft_consensus.cc:2955] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:51.577693  7076 rpcz_store.cc:276] 0709 12:24:47.849668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.850141 (+   473us) service_pool.cc:224] Handling call
0709 12:24:47.926693 (+ 76552us) spinlock_profiling.cc:243] Waited 4.51 ms on lock 0x7b4c00359640. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff893ea20 00007f4ff894fa29 00007f4ff8915644 00007f4ff893cf2d 00007f4ff893cb93 00007f4ffa5cb1af 00007f4ffa5caf3e 00007f4ffa650888 00007f4ffa66d75c 00007f4ffa66e76a 00007f4ffa66d91b
0709 12:24:47.975371 (+ 48678us) log.cc:844] Serialized 480 byte log entry
0709 12:24:48.542019 (+566648us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b1800152f20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5f2b26 00007f4ffa5ec49a 00007f4ffa5eb714
0709 12:24:48.542747 (+   728us) spinlock_profiling.cc:243] Waited 7.36 ms on lock 0x7b540003ffa0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5f2b26 00007f4ffa5ec49a 00007f4ffa5eb714 00007f4ffa5cbeaa 00007f4ffa5d2d53 00007f4ffa5d2cd9 00007f4ffa5d2c69 00007f4ffa5d2c31 00007f4ffa5d1dfd 0000000000388634
0709 12:24:50.727419 (+2184672us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b540003ffa0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5eb142 00007f4ffa5cbeaa 00007f4ffa5d2d53 00007f4ffa5d2cd9 00007f4ffa5d2c69 00007f4ffa5d2c31 00007f4ffa5d1dfd 0000000000388634 0000000000388479 00007f4ff7585475
0709 12:24:51.530644 (+803225us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2974,"raft.run_cpu_time_us":30479,"raft.run_wall_time_us":2499978,"spinlock_wait_cycles":77752576,"thread_start_us":404691,"threads_started":1,"wal-append.queue_time_us":485837,"wal-append.run_cpu_time_us":2147,"wal-append.run_wall_time_us":1002188}
I20260709 12:24:51.666318  7103 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:51.667164  7072 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.668200  7131 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.669183  5629 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:51.670125  7053 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.623795  7013 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:51.670892  7013 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:51.670989  7019 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:51.635655  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:51.672430  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.598130  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.675422  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.677192  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:51.679637  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.680720  5558 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:51.684152  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.684201  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.684641  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.685501  5558 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:51.685895  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.688876  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.691006  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:51.691017  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.693768  5631 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:24:51.694723  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.695258  5631 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:51.695760  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.688737  7144 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.697337  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.697630  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:51.698236  5703 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:51.698763  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.699836  5634 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:51.700320  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.615784  7116 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.717564  7144 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.719166  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.728442  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.707659  7041 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.743603  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.699069  7103 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:24:51.671717  7013 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:51.765156  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:51.765802  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.775415  5632 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:51.697386  5633 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:51.774155  5701 raft_consensus.cc:2376] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4. Re-sending same reply.
I20260709 12:24:51.778079  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.785256  7143 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.786969  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.786839  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.788735  5628 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:51.777534  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.727591  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.792514  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.796816  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:24:51.769002  7089 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.613s	user 0.012s	sys 0.001s
I20260709 12:24:51.800791  5557 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:24:51.801602  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.802181  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.802561  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.804775  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:51.727206  7071 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.805078  7147 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.799741  6271 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:51.805569  7078 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:51.814031  7119 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.762s	user 0.011s	sys 0.007s
I20260709 12:24:51.814512  7119 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.799242  7103 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:51.816895  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.817778  7103 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:51.828053  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.812176  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:51.809546  7114 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.692s	user 0.014s	sys 0.000s
I20260709 12:24:51.830255  7114 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.831204  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:51.818837  7119 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.833732  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:51.822496  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.823220  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:51.825260  7149 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:24:51.815924  7127 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.837766  5708 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:51.838001  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.838388  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.810002  7089 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:51.826138  7111 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.637s	user 0.005s	sys 0.004s
W20260709 12:24:51.779755  7118 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.766s	user 0.012s	sys 0.001s
I20260709 12:24:51.845434  7075 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.806187  5707 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:24:51.844274  7111 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.847373  7127 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.848796  7111 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.848905  5628 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:51.850302  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.850384  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:51.832455  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.825562  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.808110  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:51.845122  7118 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.854997  7089 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.856387  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.840559  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.820107  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:51.856828  7118 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:51.850570  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.860056  5704 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:51.866559  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:51.869343  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.871865  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:51.872228  5628 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:24:51.873342  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 393) took 1021 ms. Trace:
I20260709 12:24:51.873744  6275 rpcz_store.cc:276] 0709 12:24:50.851629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.855099 (+  3470us) service_pool.cc:224] Handling call
0709 12:24:50.909938 (+ 54839us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:51.431436 (+521498us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.873068 (+441632us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b54001e87d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:51.873237 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":663554944}
I20260709 12:24:51.839143  7023 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.877728  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:51.877866  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.878896  7115 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:51.881142  7115 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:51.881228  5636 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:51.882903  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:51.885881  5702 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 2: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:51.931747  7115 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:51.932830  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.931897  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:51.930150  7114 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:51.945184  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.950167  7115 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:51.964490  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:51.965502  6280 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:51.967430  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.974323  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:51.975298  6273 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:51.976118  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.981386  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:51.984166  7013 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:52.001278  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:52.002792  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:52.003687  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:52.005854  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:52.008889  7124 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:52.009290  7124 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:52.009640  7124 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:52.012461  7073 raft_consensus.cc:2764] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260709 12:24:52.012511  7127 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:52.013247  7103 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:52.013393  7127 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:52.013597  7023 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:52.013839  7127 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:52.013636  7103 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:52.014825  7103 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:52.017015  7119 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:24:52.032155  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.032766  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:52.035465  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:52.035787  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.036669  5627 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:24:52.040510  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:52.041129  5705 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:52.041827  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:52.043358  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.043700  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.044292  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:52.044627  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.045035  5704 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:52.046811  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.047421  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.043375  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:52.048242  6280 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:52.048135  5632 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:52.049333  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:52.050843  5704 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:52.053011  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.054242  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:52.054500  7103 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:52.055244  7135 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:24:52.055680  5710 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 4. Ops: []
I20260709 12:24:52.057292  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.058807  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:52.058830  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.059561  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.059770  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:52.059954  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.064327  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.065158  6275 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:52.065315  7111 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:52.066218  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.066804  7041 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:52.068104  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:52.060982  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:52.061352  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:52.069871  6276 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:52.071499  6276 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:52.062722  6279 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:52.073036  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.073174  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.075670  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.067411  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.076689  5628 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:52.077813  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.078480  5636 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:52.079164  7124 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:52.080216  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.080664  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.081483  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.082154  5703 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:52.082912  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.079537  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.075799  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.085899  5632 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:52.083328  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.081779  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.063235  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.055749  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:52.089340  7124 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:52.092096  7037 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:24:52.093727  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.095260  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:52.105791  5705 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
W20260709 12:24:52.107194  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.109407  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.089107  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.111310  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.089452  7118 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:52.117425  6279 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:52.115885  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.121052  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.122957  5703 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:52.110102  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.125586  5631 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:24:52.122376  5702 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:24:52.132535  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.132748  6280 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:24:52.138700  6275 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:52.131664  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.138240  7093 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:52.107335  7132 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:24:52.143550  5708 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:52.145691  6279 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:52.147558  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.148119  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.151001  5708 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:52.138377  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:52.121786  7118 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:52.141883  7093 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:52.172842  5443 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet ccec0cc7269743d6ad581695ea601d1f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:24:52.167817  7135 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
W20260709 12:24:52.162774  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 67504.2us
I20260709 12:24:52.184073  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.185453  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.200287  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.208508  5630 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
W20260709 12:24:52.212193  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.216988  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:52.182569  7041 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260709 12:24:52.228804  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.207468  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.231037  5702 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 2: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:52.232084  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:52.234853  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:52.239133  5560 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:52.235608  5709 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:24:52.241349  5709 raft_consensus.cc:606] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:52.245200  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.246960  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.249514  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:52.186050  7071 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
W20260709 12:24:52.252701  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45361.3us
I20260709 12:24:52.209528  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:52.260070  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.261548  5636 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:52.262964  7017 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:24:52.256179  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:52.261866  5636 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:52.267412  5560 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:24:52.268529  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.268950  7158 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:52.269443  7158 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:52.269816  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.269876  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:52.271432  7158 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:52.271950  7114 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:52.228827  7132 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260709 12:24:52.272660  7114 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:52.272830  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:52.273463  5556 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:52.276774  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.255587  7119 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:52.277108  7075 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:24:52.286821  5636 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:52.315600  5636 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 4
W20260709 12:24:52.319636  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.256338  7147 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:52.277901  7161 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:52.331404  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.281443  7150 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:24:52.330873  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 134454us
W20260709 12:24:52.333102  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 86258.5us
I20260709 12:24:52.335559  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:52.341259  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.338183  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:52.345072  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.351660  7119 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:52.353417  7119 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:24:52.355607  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 88127us
I20260709 12:24:52.272142  7127 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:52.358346  5636 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:24:52.364784  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.355811  5458 catalog_manager.cc:5697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:52.376930  5458 catalog_manager.cc:5697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 2 to 3, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 3 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:52.332005  7150 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:52.385301  7089 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:24:52.389700  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.389673  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.391299  7119 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:52.393379  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.399899  5455 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 269) took 1518 ms. Trace:
I20260709 12:24:52.402232  5599 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.406524  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.406898  7071 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:52.401813  7078 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:52.408984  7118 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:52.409229  5710 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:52.268157  6271 raft_consensus.cc:2955] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:52.266342  7093 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:52.402279  5709 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:52.403621  7150 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:52.405006  7158 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:52.408048  7071 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:52.365773  7037 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:52.408157  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.343914  7161 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:52.375556  7017 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260709 12:24:52.418593  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:52.412326  7127 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:52.437497  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:52.409005  7078 consensus_queue.cc:1243] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20260709 12:24:52.511840  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 107929us
I20260709 12:24:52.512957  5561 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:24:52.515535  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 79533.4us
I20260709 12:24:52.515872  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.517325  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.561203  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:24:52.522284  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 103765us
I20260709 12:24:52.417110  7037 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:24:52.442540  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 171454us
I20260709 12:24:52.576637  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.577045  6236 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.523900  5553 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:52.423480  7132 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:52.421599  7158 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:52.524099  5675 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:52.525583  7014 raft_consensus.cc:1064] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:24:52.530577  5556 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
W20260709 12:24:52.534714  7159 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.202s	user 0.001s	sys 0.000s
I20260709 12:24:52.449183  7106 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:52.600695  7106 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:52.602015  7106 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:52.559978  7093 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:52.590186  7127 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:52.591804  7124 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:52.622982  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.421248  6273 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:52.562602  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:24:52.586032  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 62032.7us
W20260709 12:24:52.700884  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 138342us
W20260709 12:24:52.589354  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72107.8us
I20260709 12:24:52.703769  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.550617  7078 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:52.550099  7131 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:52.551353  7069 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:52.593504  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.414484  5455 rpcz_store.cc:276] 0709 12:24:50.881082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.881257 (+   175us) service_pool.cc:224] Handling call
0709 12:24:50.932123 (+ 50866us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5dc090 00007f4ffa67af3a 00007f5000430e9c 00007f500052f04e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc 00007f4ffe9ef392 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:52.399817 (+1467694us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:51.490424 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:51.522876 (+ 32452us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:51.522916 (+    40us) write_op.cc:435] Acquired schema lock
0709 12:24:51.522938 (+    22us) tablet.cc:662] Decoding operations
0709 12:24:51.545092 (+ 22154us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:51.549657 (+  4565us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:51.554923 (+  5266us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:24:51.574304 (+ 19381us) tablet.cc:701] Row locks acquired
0709 12:24:51.574326 (+    22us) write_op.cc:260] PREPARE: finished
0709 12:24:51.574518 (+   192us) write_op.cc:270] Start()
0709 12:24:51.574659 (+   141us) write_op.cc:276] Timestamp: P: 1783599891574453 usec, L: 0
0709 12:24:51.574691 (+    32us) op_driver.cc:348] REPLICATION: starting
0709 12:24:51.604451 (+ 29760us) log.cc:844] Serialized 587 byte log entry
0709 12:24:51.821281 (+216830us) op_driver.cc:464] REPLICATION: finished
0709 12:24:51.970245 (+148964us) write_op.cc:301] APPLY: starting
0709 12:24:51.970334 (+    89us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:51.970440 (+   106us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:51.970563 (+   123us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:51.982980 (+ 12417us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:51.983165 (+   185us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:24:51.983213 (+    48us) write_op.cc:312] APPLY: finished
0709 12:24:51.983410 (+   197us) log.cc:844] Serialized 23 byte log entry
0709 12:24:51.983583 (+   173us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:51.983675 (+    92us) write_op.cc:454] Released schema lock
0709 12:24:51.983726 (+    51us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:51.984288 (+   562us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:51.984329 (+    41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":26903,"reported_tablets":7,"spinlock_wait_cycles":59419392,"child_traces":[["op",{"apply.queue_time_us":148681,"apply.run_cpu_time_us":2084,"apply.run_wall_time_us":14013,"num_ops":1,"prepare.queue_time_us":129770,"prepare.run_cpu_time_us":4260,"prepare.run_wall_time_us":135698,"replication_time_us":246404,"spinlock_wait_cycles":62848,"thread_start_us":129523,"threads_started":2}]]}
I20260709 12:24:52.447535  7150 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:52.660137  7160 raft_consensus.cc:993] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:52.585304  7037 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:52.598388  7114 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:52.625100  7161 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:52.695487  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:52.668781  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:24:52.667474  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 330106us
I20260709 12:24:52.686195  5560 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:52.687467  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.687019  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.752691  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:52.717914  7078 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:52.702845  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:24:52.758040  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55285.7us
W20260709 12:24:52.758178  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 181241us
I20260709 12:24:52.758821  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:24:52.759518  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.630970  5629 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:52.622319  7053 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:24:52.605474  7147 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:52.621812  7133 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:52.669174  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.644881  7013 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:52.604342  7106 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:52.599925  5561 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:52.781394  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:52.636207  7127 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:52.715832  7041 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:24:52.744055  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48710.6us
I20260709 12:24:52.787856  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.762887  7078 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:24:52.744341  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 218731us
I20260709 12:24:52.655030  7103 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:52.722620  7069 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:52.748904  5556 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:24:52.720285  7031 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260709 12:24:52.716877  7014 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:52.718173  7171 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:52.755800  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.700892  5457 catalog_manager.cc:5697] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f reported cstate change: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:52.780730  7013 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:52.718456  7131 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:52.597725  7158 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:52.763576  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.816829  5457 catalog_manager.cc:5697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 3 to 4, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 4 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:52.821722  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.822949  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.823933  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.825233  7131 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:52.826506  6273 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:52.827625  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.765668  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:24:52.828451  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62861.5us
I20260709 12:24:52.828711  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.829337  5632 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:24:52.830114  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.831465  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:52.832056  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.766480  7135 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:52.832939  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.771405  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.764220  7132 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:52.833860  7132 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:24:52.834208  7132 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:52.834571  7132 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:24:52.834393  7075 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:24:52.835379  7075 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:52.836311  7075 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:52.836499  7041 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:52.836714  7075 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:52.836915  7041 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:52.770313  7017 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260709 12:24:52.837863  7017 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:52.837687  7041 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:52.838302  7017 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:52.839615  5704 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:24:52.840651  5709 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:52.757540  7160 raft_consensus.cc:1081] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:52.755887  7124 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:52.765327  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:24:52.662196  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.842517  5707 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:52.766461  7164 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:52.777011  7133 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:24:52.846385  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 81114us
I20260709 12:24:52.847498  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:52.848685  5636 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:24:52.804852  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.756021  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.801576  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:52.803224  7019 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260709 12:24:52.811220  7069 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:52.791258  7169 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:24:52.814389  6277 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:52.814569  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.815681  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.791133  7127 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:52.860074  6236 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:52.832851  7135 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:24:52.766417  7053 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:52.837637  7075 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:52.839170  7017 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:52.842401  7124 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:52.867849  7148 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:52.875135  7017 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:52.796979  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:52.768808  7147 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:24:52.776382  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.845160  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.875423  7013 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:52.882448  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.882750  7013 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:52.883551  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.852445  5634 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:52.846443  7164 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:52.846540  7041 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:52.846767  7111 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:52.855041  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.855155  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:52.857239  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55779.3us
I20260709 12:24:52.889201  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:24:52.886334  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:52.859355  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54575.6us
I20260709 12:24:52.892119  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.826588  7131 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:24:52.896163  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.862051  7167 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:52.862121  6236 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.868680  7165 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:24:52.861069  7169 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:52.844477  7114 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:52.875362  7124 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:52.880755  7147 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:52.884721  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 87821.9us
I20260709 12:24:52.880712  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:52.894963  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.865262  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.870870  7163 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:52.868851  7158 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:52.899000  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.870591  7023 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20260709 12:24:52.899824  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:24:52.960325  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60630.6us
I20260709 12:24:52.945865  7177 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.860837  7150 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 12:24:52.889626  7073 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:52.900040  7118 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:24:52.963994  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 65110.7us
I20260709 12:24:52.985512  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:52.944487  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:52.886752  7164 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:52.902690  7089 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:24:52.950392  7159 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 12:24:52.983714  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.901149  7160 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:52.973109  7075 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:52.999254  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:53.017722  5628 raft_consensus.cc:2368] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4.
I20260709 12:24:52.992772  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:52.952301  7169 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:24:53.007467  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:52.973435  7158 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:24:52.983255  7147 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:52.953924  7163 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:52.954110  7174 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260709 12:24:52.984697  7150 consensus_queue.cc:1243] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:24:53.000638  7073 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:52.984925  7118 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:52.990247  7031 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:53.020161  7175 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:53.000066  7076 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:52.980033  7066 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260709 12:24:52.997972  7089 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:53.063390  7089 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:53.005961  7167 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:53.036192  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:53.027395  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 66228.9us
W20260709 12:24:53.066299  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 73656.7us
I20260709 12:24:53.066776  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:24:53.066989  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 101172us
I20260709 12:24:53.043154  7114 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.047909  5708 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:53.048775  7069 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:53.015529  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:53.023488  7078 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:53.025216  7103 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.983618  7177 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.031245  7163 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:53.069024  7076 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:53.068519  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 124231us
I20260709 12:24:53.069068  5702 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:53.081122  5524 leader_election.cc:400] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4.
I20260709 12:24:53.094458  5524 leader_election.cc:403] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:53.058763  7118 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:53.060240  7170 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:53.084059  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.086615  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 415) took 1122 ms. Trace:
I20260709 12:24:53.107726  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.108536  5630 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:53.108922  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.107867  6271 rpcz_store.cc:276] 0709 12:24:51.955476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.958651 (+  3175us) service_pool.cc:224] Handling call
0709 12:24:52.025343 (+ 66692us) spinlock_profiling.cc:243] Waited 5.83 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:24:52.065743 (+ 40400us) spinlock_profiling.cc:243] Waited 8.07 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754de9c 00007f4ff754da5b 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:24:52.074339 (+  8596us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:52.091810 (+ 17471us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.091843 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:52.110753 (+ 18910us) log.cc:844] Serialized 480 byte log entry
0709 12:24:52.257519 (+146766us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:24:52.597356 (+339837us) log.cc:844] Serialized 14 byte log entry
0709 12:24:52.817560 (+220204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.846197 (+ 28637us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:52.846249 (+    52us) raft_consensus.cc:1685] finished
0709 12:24:52.846263 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:53.059767 (+213504us) spinlock_profiling.cc:243] Waited 6.09 ms on lock 0x7b5400771d50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:53.069667 (+  9900us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1921,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"spinlock_wait_cycles":45962112,"thread_start_us":308712,"threads_started":2,"wal-append.queue_time_us":175748}
I20260709 12:24:53.115293  5635 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:24:53.110584  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.121380  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:53.122848  5636 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:53.061003  7175 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:24:53.123222  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55210.1us
I20260709 12:24:53.069736  7124 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:53.076741  7019 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:53.125476  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:24:53.128322  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.130285  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:53.062705  7031 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.083918  7186 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.115835  7170 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:53.129249  7161 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:53.129233  7158 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:24:53.141997  5627 raft_consensus.cc:2368] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4.
W20260709 12:24:53.148056  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 139573us
I20260709 12:24:53.084777  7103 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.128137  7175 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:24:53.147562  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 76528.3us
I20260709 12:24:53.131162  7188 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:53.135871  7119 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.139308  7189 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:24:53.149183  7161 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:53.170426  6273 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 1 LEARNER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:53.135560  7106 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 12:24:53.139780  7019 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.141705  7174 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 12:24:53.180480  7186 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:53.147578  7170 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:53.174186  7158 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:53.192058  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:53.142903  7124 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.177798  5633 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:53.198942  5558 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:53.203037  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:53.187372  7161 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:53.239494  7106 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:53.241022  7106 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:24:53.237295  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 99575.4us
I20260709 12:24:53.228965  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:53.243598  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.188159  7160 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:53.170471  7073 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:53.230297  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:53.249058  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:53.191952  7194 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.256608  5554 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:53.199827  7158 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:53.281962  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:24:53.285300  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.205636  7103 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:53.205636  7031 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:53.253901  7195 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 12:24:53.295370  7190 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:53.252162  7076 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:53.273846  7188 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4.
I20260709 12:24:53.280239  7023 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260709 12:24:53.250133  7106 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.258901  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:53.305411  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:24:53.303197  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 172972us
I20260709 12:24:53.258142  7160 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:53.280664  7078 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:53.315011  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:53.317788  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.280964  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:53.254078  7197 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:53.302337  7190 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.303809  7177 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:53.291225  7075 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:53.326031  5628 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:53.307680  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:53.319634  5627 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:24:53.371225  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49966.9us
I20260709 12:24:53.386562  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.363982  6274 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:53.319895  7017 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:24:53.370196  7166 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:24:53.395527  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.395859  5703 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:53.319247  6271 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
W20260709 12:24:53.405635  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.401168  6278 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:53.371031  7177 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.411031  7041 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:53.410904  7017 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:53.411980  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.330161  7197 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:24:53.409430  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 127545us
I20260709 12:24:53.366992  7190 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:53.392846  7124 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:53.415719  7106 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:53.418756  5707 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:24:53.402729  5552 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:24:53.410540  7131 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:24:53.366852  7188 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 12:24:53.423362  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.421872  6273 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:53.429966  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:53.389676  7075 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.436216  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:53.439422  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.442096  6275 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:53.426424  7135 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.425239  7019 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:53.427094  7197 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:24:53.441048  7202 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.183s	user 0.002s	sys 0.000s
I20260709 12:24:53.421594  7177 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.472390  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.466284  7017 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:53.443208  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:53.473649  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:53.474624  5706 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
W20260709 12:24:53.467975  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 342514us
I20260709 12:24:53.437426  7111 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:53.448947  7041 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.477725  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.490430  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:53.494177  7174 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:24:53.494707  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 213344us
I20260709 12:24:53.498106  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.514535  5602 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.515781  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.516299  7078 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:24:53.516788  7150 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.516336  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.518558  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.519464  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.518141  7150 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.523809  5675 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.473462  7135 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.425369  7160 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.526508  7127 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:24:53.486181  7188 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.522648  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.523203  7093 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:53.482613  7111 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:53.525547  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.513036  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:53.528924  7127 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:53.518141  7205 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.546769  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:53.553378  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.476105  7131 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260709 12:24:53.560187  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.482565  7023 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20260709 12:24:53.563457  7177 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:53.524514  7078 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:24:53.563609  7037 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.812s	user 0.009s	sys 0.002s
I20260709 12:24:53.569973  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.525115  7214 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:53.577054  7037 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.579531  7037 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:53.579733  5635 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:53.580648  5557 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:53.545689  7093 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:24:53.583606  7213 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 12:24:53.583820  7093 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:53.584602  7167 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:53.584698  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:53.585222  7167 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:53.585568  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:53.581185  5635 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:53.586221  5555 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:53.552951  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:53.552663  7207 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:53.593425  5558 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:24:53.594309  5635 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:35614
I20260709 12:24:53.596268  5635 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:53.596748  5635 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:53.572966  7115 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:24:53.566840  7124 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.574671  7076 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:53.573288  7131 consensus_queue.cc:1243] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:24:53.603139  7199 raft_consensus.cc:2764] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260709 12:24:53.552100  7188 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:53.589501  7037 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:53.589788  7019 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.590193  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:53.617683  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:53.555914  7133 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:53.612671  7163 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.530s	user 0.011s	sys 0.000s
I20260709 12:24:53.620615  7037 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.642987  7196 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:53.643654  7196 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:53.625880  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:53.637686  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:24:53.623199  7132 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.788s	user 0.008s	sys 0.005s
I20260709 12:24:53.644991  5630 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:53.639575  5556 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:53.649000  7037 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:53.650504  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:53.651829  5458 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35600 (request call id 273) took 1989 ms. Trace:
I20260709 12:24:53.647536  7075 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:53.608788  7041 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 12:24:53.656275  5636 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:53.654855  7192 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.660076  5457 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 272) took 1828 ms. Trace:
I20260709 12:24:53.661505  5456 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 2, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 2 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:53.627233  7115 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.659062  7017 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:53.665742  5630 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
W20260709 12:24:53.660055  7069 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.579s	user 0.008s	sys 0.002s
I20260709 12:24:53.667003  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.656370  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 184) took 1138 ms. Trace:
I20260709 12:24:53.669781  7186 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:53.683830  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:53.690075  7186 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:53.691058  7186 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.691458  7053 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:53.672082  7075 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 12:24:53.672647  5619 tablet_service.cc:1558] Processing DeleteTablet for tablet ccec0cc7269743d6ad581695ea601d1f with delete_type TABLET_DATA_TOMBSTONED (TS 4a1c5e6dd8e249bf82d38f90fa37b43d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35612
I20260709 12:24:53.682578  5557 rpcz_store.cc:276] 0709 12:24:52.518244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.518375 (+   131us) service_pool.cc:224] Handling call
0709 12:24:52.518694 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:53.581025 (+1062331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.656030 (+ 75005us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540044dc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:53.656257 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2441724160}
I20260709 12:24:53.686542  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:24:53.670668  7169 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.639s	user 0.009s	sys 0.000s
I20260709 12:24:53.668926  7124 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.712055  7169 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.712211  7124 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.712970  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:53.713809  5553 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:24:53.714108  5708 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
W20260709 12:24:53.714352  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.692502  7186 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:53.649545  7093 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:53.723069  7169 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:53.664376  7076 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260709 12:24:53.674773  5457 rpcz_store.cc:276] 0709 12:24:51.832008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.832160 (+   152us) service_pool.cc:224] Handling call
0709 12:24:52.045347 (+213187us) spinlock_profiling.cc:243] Waited 42.4 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5dc090 00007f4ffa67af3a 00007f5000430e9c 00007f500052f04e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc 00007f4ffe9ef392 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:53.659995 (+1614648us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:52.853612 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:52.854114 (+   502us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:52.854159 (+    45us) write_op.cc:435] Acquired schema lock
0709 12:24:52.854184 (+    25us) tablet.cc:662] Decoding operations
0709 12:24:52.854596 (+   412us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:52.854645 (+    49us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:52.854680 (+    35us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:24:52.854978 (+   298us) tablet.cc:701] Row locks acquired
0709 12:24:52.854998 (+    20us) write_op.cc:260] PREPARE: finished
0709 12:24:52.855137 (+   139us) write_op.cc:270] Start()
0709 12:24:52.855247 (+   110us) write_op.cc:276] Timestamp: P: 1783599892855094 usec, L: 0
0709 12:24:52.855266 (+    19us) op_driver.cc:348] REPLICATION: starting
0709 12:24:52.855862 (+   596us) log.cc:844] Serialized 906 byte log entry
0709 12:24:53.170598 (+314736us) op_driver.cc:464] REPLICATION: finished
0709 12:24:53.287811 (+117213us) write_op.cc:301] APPLY: starting
0709 12:24:53.287888 (+    77us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:53.288003 (+   115us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:53.288066 (+    63us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:53.288271 (+   205us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:53.288445 (+   174us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 12:24:53.288474 (+    29us) write_op.cc:312] APPLY: finished
0709 12:24:53.288672 (+   198us) log.cc:844] Serialized 29 byte log entry
0709 12:24:53.288796 (+   124us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:53.288852 (+    56us) write_op.cc:454] Released schema lock
0709 12:24:53.288876 (+    24us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":672462,"reported_tablets":2,"spinlock_wait_cycles":97626880,"child_traces":[["op",{"apply.queue_time_us":115038,"apply.run_cpu_time_us":1279,"apply.run_wall_time_us":1280,"mutex_wait_us":4150,"num_ops":2,"prepare.queue_time_us":2329,"prepare.run_cpu_time_us":2486,"prepare.run_wall_time_us":2487,"replication_time_us":289770,"thread_start_us":137981,"threads_started":1}]]}
I20260709 12:24:53.711004  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.692651  7017 consensus_queue.cc:1243] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:24:53.717252  5629 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
W20260709 12:24:53.708127  7118 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.584s	user 0.005s	sys 0.006s
I20260709 12:24:53.725409  5628 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
W20260709 12:24:53.709734  7164 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.694s	user 0.007s	sys 0.006s
I20260709 12:24:53.608428  7208 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.117s
I20260709 12:24:53.728552  7106 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 12:24:53.739614  7209 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:24:53.740324  7206 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
W20260709 12:24:53.741006  7170 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.504s	user 0.010s	sys 0.000s
I20260709 12:24:53.620003  7161 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:53.705701  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:53.734258  5635 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.752903  5630 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4. Ops: []
I20260709 12:24:53.754276  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:53.674333  7133 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:24:53.733881  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 66984.4us
I20260709 12:24:53.747702  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:53.749245  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:53.647079  7127 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:53.727969  7078 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:24:53.747394  7089 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.683s	user 0.004s	sys 0.008s
W20260709 12:24:53.796798  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52014us
W20260709 12:24:53.800696  5710 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.052s	sys 0.088s
W20260709 12:24:53.802963  5710 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.052s	sys 0.088s
I20260709 12:24:53.803509  5635 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:24:53.806365  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 285759us
I20260709 12:24:53.807500  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.728411  7075 consensus_queue.cc:1243] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:24:53.755920  7106 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:53.809777  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.809922  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.652254  5458 rpcz_store.cc:276] 0709 12:24:51.662205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.667777 (+  5572us) service_pool.cc:224] Handling call
0709 12:24:53.651771 (+1983994us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:52.748645 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:52.749206 (+   561us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:52.749244 (+    38us) write_op.cc:435] Acquired schema lock
0709 12:24:52.749263 (+    19us) tablet.cc:662] Decoding operations
0709 12:24:52.749700 (+   437us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:52.749785 (+    85us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:52.749813 (+    28us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:24:52.750153 (+   340us) tablet.cc:701] Row locks acquired
0709 12:24:52.750172 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:24:52.750310 (+   138us) write_op.cc:270] Start()
0709 12:24:52.750429 (+   119us) write_op.cc:276] Timestamp: P: 1783599892750272 usec, L: 0
0709 12:24:52.750450 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 12:24:52.795737 (+ 45287us) log.cc:844] Serialized 907 byte log entry
0709 12:24:53.062235 (+266498us) op_driver.cc:464] REPLICATION: finished
0709 12:24:53.190985 (+128750us) write_op.cc:301] APPLY: starting
0709 12:24:53.194078 (+  3093us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:53.226730 (+ 32652us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:53.258466 (+ 31736us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:53.286318 (+ 27852us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:53.286480 (+   162us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 12:24:53.286524 (+    44us) write_op.cc:312] APPLY: finished
0709 12:24:53.286763 (+   239us) log.cc:844] Serialized 29 byte log entry
0709 12:24:53.286934 (+   171us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:53.287017 (+    83us) write_op.cc:454] Released schema lock
0709 12:24:53.287056 (+    39us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:53.287404 (+   348us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:53.287430 (+    26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":180761,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":114440,"apply.run_cpu_time_us":2475,"apply.run_wall_time_us":98588,"num_ops":2,"prepare.queue_time_us":354671,"prepare.run_cpu_time_us":3415,"prepare.run_wall_time_us":56552,"replication_time_us":311639,"thread_start_us":214595,"threads_started":2}]]}
I20260709 12:24:53.769018  7093 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:53.770390  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:53.813027  5708 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:53.777438  7188 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:53.778919  7158 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:53.785547  7124 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:53.757460  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1299) took 1090 ms. Trace:
I20260709 12:24:53.794672  7203 raft_consensus.cc:993] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
W20260709 12:24:53.795723  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.049s	sys 0.071s
W20260709 12:24:53.835321  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.049s	sys 0.071s
I20260709 12:24:53.811345  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:53.812914  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:53.813206  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:53.665555  7132 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:53.815454  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.828774  5558 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:53.745730  7118 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:53.811662  7147 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.781s	user 0.011s	sys 0.001s
I20260709 12:24:53.741408  7170 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.746711  7164 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.834064  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.646703  7163 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:53.802610  7169 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:53.840790  5636 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:24:53.841370  7132 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:53.841595  7161 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:53.831300  5629 rpcz_store.cc:276] 0709 12:24:52.666653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.771135 (+104482us) service_pool.cc:224] Handling call
0709 12:24:53.757276 (+986141us) spinlock_profiling.cc:243] Waited 877 ms on lock 0x7b540032bed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:53.757397 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2016617344}
I20260709 12:24:53.843565  7215 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:24:53.801092  7196 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:53.847995  5445 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 2acd6f5df0204b2295547e021cd8cb9f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4a1c5e6dd8e249bf82d38f90fa37b43d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:53.848531  7212 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:53.848824  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:53.813850  7205 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.850244  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:53.850608  5554 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:24:53.813287  7106 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:53.854436  5708 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
W20260709 12:24:53.855868  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44571us
I20260709 12:24:53.863487  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.864662  5631 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:24:53.864708  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 318030us
I20260709 12:24:53.876080  5561 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:53.876446  5524 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.876101  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.877955  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:24:53.839300  7175 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.661s	user 0.011s	sys 0.000s
I20260709 12:24:53.832581  7075 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.870118  5555 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:53.859169  7023 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:53.871447  5635 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:53.858934  7076 raft_consensus.cc:1081] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:24:53.849941  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:53.865483  7147 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.882280  5524 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.882483  7037 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:53.880478  5707 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:24:53.883811  7103 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.885474  7203 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:53.882477  5560 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:53.886809  7106 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:53.859064  7188 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 12:24:53.879614  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:53.892536  7132 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.889986  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:24:53.945292  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55362.6us
I20260709 12:24:53.945750  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:53.945952  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:53.889956  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:24:53.902187  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:53.947054  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.806988  7089 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.918144  6275 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:53.916535  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:53.829084  7111 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.859932  7176 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 12:24:53.931493  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:53.939894  5635 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:53.888682  7212 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.856513  7208 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:24:53.942866  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63363.8us
I20260709 12:24:53.892189  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:53.938846  5629 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:53.914156  5630 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:24:53.870786  7190 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:53.949548  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.971161  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:53.955065  7019 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260709 12:24:53.973757  7053 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:53.975574  6277 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: [1.4-1.4]
W20260709 12:24:53.978761  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:53.979319  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.889302  7175 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:52.885207  7178 consensus_queue.cc:1175] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f613ba929b1046d3b5aed909414a95bf" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:53.987978  7190 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:53.975507  7191 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:53.988232  7075 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:53.988770  7191 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.989202  7191 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:53.988895  5709 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:53.959512  7147 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:24:53.993021  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.961591  7014 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:53.956265  7215 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.981442  7135 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:54.007947  7131 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:53.967653  7212 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:54.007089  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1290) took 1415 ms. Trace:
I20260709 12:24:54.020331  5559 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:54.018337  5631 rpcz_store.cc:276] 0709 12:24:52.591478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.592666 (+  1188us) service_pool.cc:224] Handling call
0709 12:24:52.635448 (+ 42782us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:53.875908 (+1240460us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.006824 (+130916us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400438854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:54.006998 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2810340736}
W20260709 12:24:53.996426  7174 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.502s	user 0.010s	sys 0.000s
I20260709 12:24:54.033790  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1359) took 1021 ms. Trace:
I20260709 12:24:54.040669  5560 rpcz_store.cc:276] 0709 12:24:53.012431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.012616 (+   185us) service_pool.cc:224] Handling call
0709 12:24:53.013030 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:53.939257 (+926227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.033524 (+ 94267us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b540044f054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:54.033728 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1999126272}
I20260709 12:24:54.034471  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 428) took 1072 ms. Trace:
I20260709 12:24:53.990653  7160 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:53.990793  7132 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:24:54.039475  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 149616us
I20260709 12:24:53.970558  7017 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:54.033895  7131 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:54.042161  6277 rpcz_store.cc:276] 0709 12:24:52.962121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.962391 (+   270us) service_pool.cc:224] Handling call
0709 12:24:52.963039 (+   648us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:53.975958 (+1012919us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.034131 (+ 58173us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400773154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:54.034400 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2327729920}
I20260709 12:24:54.013870  7014 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:54.042548  7174 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:54.033332  7147 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:54.061504  7163 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:54.061543  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:54.079113  7209 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.990337  7195 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260709 12:24:54.041458  5706 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:54.080513  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:54.081283  7209 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:54.081552  7069 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:54.081800  7041 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:54.082202  7069 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:54.082613  7041 consensus_queue.cc:1243] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:24:54.033411  5556 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:24:54.067512  7217 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:53.992604  7114 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 12:24:54.017871  7169 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:54.041234  7211 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260709 12:24:54.067739  7131 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:54.055718  7106 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:54.074148  7014 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:54.059619  5630 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:24:54.074994  5561 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:54.082490  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:54.068055  7220 tablet_replica.cc:333] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:24:54.073489  7158 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:54.100682  5675 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:54.141443  7209 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:54.145120  7223 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:24:54.146363  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.160238  7198 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 12:24:54.143817  7164 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:54.168720  5708 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:54.077840  7135 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:54.140518  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:54.177856  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:54.179471  5675 leader_election.cc:400] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4.
I20260709 12:24:54.100625  7118 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:54.070883  7017 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.138721  7131 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:54.164114  5633 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:54.172463  7089 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:54.173216  7111 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:54.155543  7167 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.559s	user 0.011s	sys 0.000s
W20260709 12:24:54.175688  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.180994  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:24:54.181897  7210 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.208s	user 0.001s	sys 0.000s
I20260709 12:24:54.181823  7205 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:54.223559  7169 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.198913  7175 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:54.222158  7014 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.231168  7014 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:54.222699  7019 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:54.222934  7017 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:54.232686  5628 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:24:54.236760  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1362) took 1058 ms. Trace:
I20260709 12:24:54.237242  5559 rpcz_store.cc:276] 0709 12:24:53.178258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.178544 (+   286us) service_pool.cc:224] Handling call
0709 12:24:53.178920 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.020909 (+841989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.232327 (+211418us) spinlock_profiling.cc:243] Waited 841 ms on lock 0x7b540023f354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:54.236684 (+  4357us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1934496128}
I20260709 12:24:54.227120  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:54.185298  7190 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:54.222162  5675 leader_election.cc:400] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4.
I20260709 12:24:54.190778  7223 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:54.235816  7208 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:54.213876  7212 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.262686  5635 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:35630
I20260709 12:24:54.272162  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1461) took 1416 ms. Trace:
I20260709 12:24:54.273326  5635 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:54.273770  5635 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:54.235852  6275 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:24:54.191639  7135 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:54.177377  7147 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:54.262270  5628 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:54.280803  5634 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 12:24:54.281852  5634 pending_rounds.cc:85] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 2
I20260709 12:24:54.282162  5634 pending_rounds.cc:107] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 12:24:54.282470  5634 raft_consensus.cc:2889] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:54.285115  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:54.285471  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:54.291658  5709 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 2: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:24:54.291687  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 434638us
I20260709 12:24:54.303891  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:54.304790  5559 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:24:54.225962  7167 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.273548  5710 rpcz_store.cc:276] 0709 12:24:52.835545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.856866 (+ 21321us) service_pool.cc:224] Handling call
0709 12:24:52.880216 (+ 23350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:52.888153 (+  7937us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.888184 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:52.895629 (+  7445us) log.cc:844] Serialized 26 byte log entry
0709 12:24:53.179558 (+283929us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x7b18000bb3c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:24:53.803405 (+623847us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:24:53.803466 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.803561 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:54.251864 (+448303us) raft_consensus.cc:1685] finished
0709 12:24:54.251885 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.252058 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20794,"raft.queue_time_us":8818,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":12979,"spinlock_wait_cycles":2539392,"thread_start_us":623624,"threads_started":1,"wal-append.queue_time_us":986012}
I20260709 12:24:54.254223  7208 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:54.308321  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:54.276530  7170 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:54.249540  7014 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260709 12:24:54.286389  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:54.313035  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:54.320966  6277 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:54.308990  6278 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:54.247999  7111 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 12:24:54.252322  7205 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260709 12:24:54.322889  5705 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:54.285953  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:54.325150  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.325513  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:54.334615  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:54.327405  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:54.334205  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:54.400583  6279 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:54.399909  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:24:54.352339  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.395511  5706 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 2: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:54.325994  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:54.373148  5635 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.411988  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:54.410648  7013 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.414902  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:54.362502  5559 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:24:54.400760  6280 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:54.329524  7023 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.093s
I20260709 12:24:54.415858  5710 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:24:54.375092  5456 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.027s	sys 0.031s
I20260709 12:24:54.414772  7013 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:54.415295  6272 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:54.394848  7053 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.415616  7160 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:54.334249  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:54.323625  7115 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.090s
W20260709 12:24:54.431763  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.323807  7203 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20260709 12:24:54.433207  5635 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:54.417546  5628 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 2: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:24:54.348964  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 335779us
I20260709 12:24:54.435599  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:24:54.435091  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:24:54.417199  5456 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.027s	sys 0.034s
I20260709 12:24:54.337442  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:54.436906  5635 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:24:54.439574  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 422474us
I20260709 12:24:54.442256  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:54.442682  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:54.443704  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:24:54.450853  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 123549us
W20260709 12:24:54.451884  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 127376us
I20260709 12:24:54.413597  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:54.412340  7212 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:54.414855  7205 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260709 12:24:54.437117  7132 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:54.493913  5707 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.546s	user 0.008s	sys 0.002s
I20260709 12:24:54.519984  5707 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
W20260709 12:24:54.463220  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.434038  5633 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:54.437842  7208 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:54.420619  7053 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.336768  7019 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.150s
I20260709 12:24:54.505699  7215 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.523293  7170 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.519778  7132 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:54.416661  7223 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.529343  7135 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:54.529681  6275 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:54.517055  7177 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:54.512899  7167 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:54.522477  7076 raft_consensus.cc:2764] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:24:54.533018  7243 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:54.533113  7209 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:24:54.533308  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:54.513895  7178 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:24:54.534452  7013 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:54.507691  7191 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.585482  7178 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:54.586572  7053 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:54.547410  5555 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
W20260709 12:24:54.549966  7197 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.074s	user 0.006s	sys 0.008s
W20260709 12:24:54.553220  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.591203  6276 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:54.507962  5635 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:54.491022  7023 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:54.569046  7196 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:54.568557  7243 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:54.570479  7132 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:54.550788  7177 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.571228  5701 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:24:54.571762  7244 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:24:54.572235  5629 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:24:54.564831  7215 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:54.592010  5552 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:54.592955  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:54.573446  7209 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:24:54.605443  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.551227  7170 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.587023  7191 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:54.590831  7197 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.586383  7178 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.593983  7192 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260709 12:24:54.617681  7243 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:54.622602  7243 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:24:54.623695  7133 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.829s	user 0.009s	sys 0.002s
I20260709 12:24:54.632225  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:54.632781  7103 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:54.635602  7103 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:54.634955  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:54.644109  7103 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:54.649262  7240 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260709 12:24:54.653888  7177 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:54.665709  7197 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:54.667001  7197 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:54.667479  7197 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:54.647181  7209 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:54.639441  7069 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:54.671432  7069 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:54.678140  7215 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:54.689724  7170 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:54.696184  7197 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.634582  7244 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:54.710985  7244 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:54.712381  7244 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.714263  7244 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:54.732319  7023 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:24:54.786302  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.634999  5633 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:24:54.793290  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:54.795594  5635 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:54.799127  5636 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:24:54.825558  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.830884  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:54.833042  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1320) took 1881 ms. Trace:
I20260709 12:24:54.888525  5634 rpcz_store.cc:276] 0709 12:24:52.951645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.019378 (+ 67733us) service_pool.cc:224] Handling call
0709 12:24:53.019704 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.282884 (+1263180us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.832312 (+549428us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b540009e2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:54.832796 (+   484us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2899431296}
I20260709 12:24:54.861805  5627 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:24:54.869431  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.872015  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:54.872674  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:24:54.873239  5445 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 2acd6f5df0204b2295547e021cd8cb9f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4a1c5e6dd8e249bf82d38f90fa37b43d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:24:54.877008  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.879267  5557 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:54.896996  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:54.898213  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:54.896936  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:54.899156  5701 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:54.899609  5701 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:54.900629  5704 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:54.901185  5701 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:54.901585  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:54.902297  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:54.902717  5552 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:54.904768  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:54.906121  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:54.906359  5552 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:54.909121  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:54.910259  5552 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 2: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:24:54.910707  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.915606  5557 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:54.919693  5632 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:24:54.920799  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:54.924574  5632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1626) took 2172 ms (client timeout 2167 ms). Trace:
I20260709 12:24:54.925938  5633 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:24:54.929065  5632 rpcz_store.cc:269] 0709 12:24:52.752092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.890383 (+138291us) service_pool.cc:224] Handling call
0709 12:24:54.924327 (+2033944us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b5400439c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:54.924471 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3856235264}
I20260709 12:24:54.933400  5558 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:54.937167  5558 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 4. Ops: []
I20260709 12:24:54.975970  7069 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:24:55.035748  5558 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:55.036500  5635 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
W20260709 12:24:55.051038  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.977085  7069 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:55.067867  6273 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
W20260709 12:24:55.078799  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.042s	sys 0.150s
W20260709 12:24:55.079384  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.042s	sys 0.150s
W20260709 12:24:55.080545  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.090917  7075 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.092415  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:55.090233  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:55.095104  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.026s	sys 0.046s
W20260709 12:24:55.095309  5676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.049s	sys 0.053s
W20260709 12:24:55.097901  5676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.049s	sys 0.053s
W20260709 12:24:55.098438  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 643495us
I20260709 12:24:55.098639  7191 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.099552  7191 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.102495  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:55.103638  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:55.104908  7041 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.105125  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:55.105826  7041 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:55.106367  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:55.108497  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:55.110198  5555 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 3: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:24:55.111035  7114 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:55.111594  7114 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:55.103881  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:55.097280  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.026s	sys 0.046s
I20260709 12:24:55.103693  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:55.120254  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:55.100185  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:55.120276  5559 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 2: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:24:55.112691  5557 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:55.128268  7041 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.129382  7257 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:55.130174  7257 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:55.130873  7189 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:55.133255  6235 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.021s	sys 0.018s
I20260709 12:24:55.133592  5631 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
W20260709 12:24:55.134158  5673 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.000s	sys 0.026s
W20260709 12:24:55.134559  5673 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.000s	sys 0.026s
I20260709 12:24:55.136714  7195 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:55.137188  7195 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:55.138271  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:55.133689  6235 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.021s	sys 0.019s
W20260709 12:24:55.146337  6234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.010s	sys 0.030s
I20260709 12:24:55.138603  7195 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:55.146962  6234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.011s	sys 0.031s
W20260709 12:24:55.147732  7205 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.002s	sys 0.000s
W20260709 12:24:55.148414  7205 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.003s	sys 0.000s
I20260709 12:24:55.148754  7191 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.147660  7195 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:55.142292  5553 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
W20260709 12:24:55.149641  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 660512us
W20260709 12:24:55.169509  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.131536  7257 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.179879  7229 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.158859  7075 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.180967  7229 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.182016  7191 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:55.175825  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:55.131807  7189 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:55.187305  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.189816  7191 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.190598  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:55.168478  5561 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "addbcf2919194d2e866b5e8826422851"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:47734
I20260709 12:24:55.192582  7216 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:24:55.192766  5561 raft_consensus.cc:606] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:55.193063  7216 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:55.193178  5561 raft_consensus.cc:614] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:55.192107  7196 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.189869  7089 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.219972  7114 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.221597  7177 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.222337  7177 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.224500  7196 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.226861  7196 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:24:55.234057  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.234586  7132 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.235244  7132 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.235700  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:55.236955  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:55.238759  7192 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:55.239434  7192 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.239900  7192 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:55.240108  7132 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:55.240509  7244 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:55.244396  7220 raft_consensus.cc:2243] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:24:55.244453  7133 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.246182  7170 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.246798  7133 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.247044  7170 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.248273  5631 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:55.205224  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:55.240935  7250 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:24:55.241310  7203 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.010s	sys 0.017s
W20260709 12:24:55.249512  7203 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.010s	sys 0.018s
I20260709 12:24:55.249670  7170 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.249984  7133 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:55.250972  7250 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:55.252280  7250 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.253118  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:55.254405  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:55.255307  6276 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:24:55.256390  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 66036.4us
I20260709 12:24:55.256667  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:55.257833  6279 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:55.259320  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:55.260144  6273 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:24:55.260938  7244 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:55.261756  7220 raft_consensus.cc:2272] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:24:55.203239  7041 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:55.265105  7041 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:24:55.288923  5560 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
W20260709 12:24:55.295379  5526 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.913s	user 0.014s	sys 0.044s
W20260709 12:24:55.296005  5526 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.996s	user 0.014s	sys 0.045s
W20260709 12:24:55.296510  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 997305us
W20260709 12:24:55.297041  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:55.295460  7223 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.012s	sys 0.027s
W20260709 12:24:55.297904  7223 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.013s	sys 0.027s
I20260709 12:24:55.302722  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:55.273562  7041 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.310985  7118 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:55.305999  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:24:55.311280  7124 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.039s	sys 0.054s
I20260709 12:24:55.311666  7223 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.312630  7196 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:24:55.314692  7244 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:55.314903  7197 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:55.314982  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:55.332824  5709 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:24:55.315891  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:55.335330  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:24:55.336670  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 285953us
I20260709 12:24:55.268085  7220 ts_tablet_manager.cc:1916] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:24:55.348742  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:55.268536  7089 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:55.350821  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:55.341563  7124 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.041s	sys 0.054s
W20260709 12:24:55.368649  7147 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.010s	sys 0.027s
W20260709 12:24:55.369287  7147 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.822s	user 0.010s	sys 0.028s
I20260709 12:24:55.331179  7177 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.368834  7244 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.350211  7196 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:24:55.349108  7189 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:55.284581  7069 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:55.389071  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 176249us
I20260709 12:24:55.374291  7041 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:55.381202  7196 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:24:55.402136  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.385141  7075 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:55.385224  7192 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.409933  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:55.420430  5634 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:24:55.419901  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:55.422820  7178 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:55.423482  7178 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.425779  7178 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:55.426424  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:55.427773  7208 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:55.428397  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:55.429291  7170 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:55.429775  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:55.432011  7177 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:55.429922  7170 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:55.432824  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:55.433686  7196 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:55.435094  7196 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:24:55.434911  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:55.436060  5706 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:55.438459  7089 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:55.438845  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.387959  7223 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:55.439283  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.414289  7191 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.417689  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:24:55.464924  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.418648  7229 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:55.461994  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:55.477133  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:55.479660  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:55.450345  7189 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:55.486819  6279 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:55.485009  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:55.418247  7041 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.452122  7118 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:55.502399  5636 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:55.467537  7075 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:55.503896  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.498003  6276 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:24:55.411347  7216 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.491047  7229 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:55.506203  7089 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.500240  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:55.517601  7132 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:55.517863  7089 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.514740  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:24:55.496785  7147 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:55.508715  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:55.518487  7132 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.525393  7265 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:55.525542  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:55.526067  7075 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:55.529110  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.534937  7075 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.518946  5601 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:55.545954  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:24:55.546202  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:55.547066  7211 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:55.547501  7211 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.547834  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:55.535214  7177 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:24:55.548147  7211 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:55.548521  7177 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.548923  7177 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:55.551656  7147 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:55.517705  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:24:55.528496  7189 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.537498  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:55.537894  7132 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:55.539904  7089 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.520046  5633 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:24:55.547478  7209 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:55.535568  7216 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:55.536072  7266 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.560774  7192 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:55.552728  7267 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:55.528275  7041 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:55.561982  5707 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:55.514264  7229 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:55.523066  6273 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:55.566831  5703 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 2: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:55.566749  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:55.567932  7019 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:55.567912  7272 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:24:55.568476  7014 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:55.558171  7223 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:55.558972  7265 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.621456  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:55.621978  7266 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:55.623080  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:55.582463  7189 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:55.618189  7014 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:55.624101  7176 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:24:55.625036  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58508.2us
I20260709 12:24:55.625597  7014 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:55.617268  7132 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.627103  7014 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:55.626823  7132 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:55.628463  7131 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:55.628793  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:55.629312  7132 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.629737  6275 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:55.617483  7019 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:55.629787  7131 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:55.630910  7019 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:55.631994  7019 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:55.633138  7103 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:55.633255  7271 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:55.633947  7271 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:55.583204  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:55.638166  5708 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:24:55.640138  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:55.641273  5633 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:55.583819  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:55.644980  5631 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:55.645107  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:24:55.646031  7189 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:55.646428  7189 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:24:55.646462  7242 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.604s	user 0.003s	sys 0.000s
I20260709 12:24:55.646876  7189 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:55.585041  7196 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:55.647723  7196 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:55.624114  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.588900  7147 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.562000  7211 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:55.589350  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 112366us
W20260709 12:24:55.592459  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:55.594635  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 118015us
I20260709 12:24:55.608883  5558 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:55.678442  7211 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.678944  7244 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:55.680006  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:55.611068  5704 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:55.681005  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:55.685891  7197 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:55.678503  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:24:55.690923  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.611591  7075 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:55.694161  7147 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:24:55.613852  5627 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:55.694792  7147 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.695217  7147 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:55.696661  7266 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.566401  7118 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:55.585290  7177 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.651154  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:24:55.654794  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.657200  5710 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:55.618201  7229 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:24:55.703163  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51309.9us
I20260709 12:24:55.623473  7265 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:55.619029  7208 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:55.659282  7196 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:55.669677  7271 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.649217  7272 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:55.673054  7165 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:24:55.703316  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53390us
I20260709 12:24:55.714192  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:55.681028  7191 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:24:55.700076  7266 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:55.720721  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:55.721495  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:55.724766  5601 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:24:55.725385  5704 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:55.731732  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:24:55.733091  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:55.733825  5630 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:55.735059  7177 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:55.734876  7266 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.735750  7243 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260709 12:24:55.736161  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:55.737007  7212 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:55.737236  7266 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:55.737479  7212 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.737996  7212 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:55.705091  7267 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:55.740684  7267 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:55.740746  7069 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:24:55.741256  7267 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:55.741299  7069 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.741910  7069 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:55.742405  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:55.744673  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 372512us
W20260709 12:24:55.745047  7261 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.449s	user 0.003s	sys 0.000s
I20260709 12:24:55.746531  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:55.746902  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:55.747696  5709 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:24:55.748682  7240 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:24:55.751731  6275 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:24:55.754547  7069 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.756579  7147 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.746937  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 437) took 1436 ms. Trace:
I20260709 12:24:55.758447  5701 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:55.757447  6274 rpcz_store.cc:276] 0709 12:24:54.310311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.321030 (+ 10719us) service_pool.cc:224] Handling call
0709 12:24:54.509108 (+188078us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:54.539612 (+ 30504us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.541348 (+  1736us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:54.950604 (+409256us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b2c00040818. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:24:54.952012 (+  1408us) log.cc:844] Serialized 540 byte log entry
0709 12:24:55.219285 (+267273us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:24:55.280316 (+ 61031us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.325420 (+ 45104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.744069 (+418649us) spinlock_profiling.cc:243] Waited 5.99 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:24:55.746529 (+  2460us) raft_consensus.cc:1685] finished
0709 12:24:55.746557 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.746729 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22325248,"thread_start_us":248691,"threads_started":1,"wal-append.queue_time_us":343273}
I20260709 12:24:55.726054  7165 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:55.760988  7165 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.762295  7165 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:24:55.741186  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:55.767320  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.770094  7069 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:55.768566  7147 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:55.772527  5558 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:24:55.772419  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:55.774559  5558 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:24:55.775497  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:55.776618  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:55.779021  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:55.779966  6274 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260709 12:24:55.780542  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:55.780984  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.789757  5560 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:55.791714  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:55.792537  5701 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:55.771502  7267 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.796375  7265 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:55.796607  7119 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:55.796866  7265 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.797637  7265 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:55.797610  7119 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.794533  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:55.799438  5560 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:55.800923  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:55.801721  5552 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:55.804868  5553 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:55.808761  7265 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.801676  7119 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.813529  5555 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:55.813643  7267 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:55.815694  7266 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:55.816430  7266 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:55.818264  7271 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:55.818910  7266 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.819121  7271 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:55.822081  7271 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:55.822610  7265 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:55.816084  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:55.725903  7191 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:55.832677  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 101019us
I20260709 12:24:55.834110  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:55.838483  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:24:55.842451  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.839375  5632 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:24:55.846053  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:55.846973  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:55.848572  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:55.848733  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:55.849283  5705 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:55.850304  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:55.851461  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:55.840049  7191 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:55.852695  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:55.853569  5633 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:55.856871  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:55.857532  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:55.858626  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:55.859046  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:55.859294  5635 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:55.859571  7271 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:24:55.861166  7271 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:55.862057  7178 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:55.862200  7272 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.862946  7178 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:24:55.864579  7199 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:24:55.857797  5636 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:24:55.856716  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 120636us
I20260709 12:24:55.866106  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:55.865296  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:55.866822  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:55.866918  6274 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:55.866350  7271 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:55.866182  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:55.868142  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:55.866132  7199 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:55.869958  7220 ts_tablet_manager.cc:1929] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:24:55.870085  7211 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:55.871868  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:55.867136  5702 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:55.732627  7192 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.156s
W20260709 12:24:55.974347  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.967401  5706 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:55.983283  6279 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:24:55.983768  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.874912  5635 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:55.874544  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:55.829533  5552 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:24:55.986398  5705 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:24:55.987457  5630 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:24:55.991146  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:55.991952  5706 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:55.940927  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:55.994002  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:55.994570  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:55.993243  5555 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:55.927155  5560 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:55.995193  5554 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:24:56.004318  5703 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:24:55.967724  6274 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4. Ops: []
I20260709 12:24:56.006690  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:56.007604  5703 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:24:56.009667  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 143617us
I20260709 12:24:56.012161  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:55.836791  7069 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:55.983106  7119 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.971637  7019 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:55.966145  7124 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:55.961714  5556 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:24:55.973172  7244 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:55.919656  7178 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:55.968777  7073 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:56.014575  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:56.022330  5706 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:24:55.990819  7075 raft_consensus.cc:2764] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:24:56.024755  5561 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:55.970855  7223 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.026873  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:56.004944  7220 log.cc:1199] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/wals/ccec0cc7269743d6ad581695ea601d1f
I20260709 12:24:56.003925  7211 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:56.036824  7274 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:56.002710  7041 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:24:56.014737  7132 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.036633  7069 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:55.872464  7205 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:55.887619  7147 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:56.036608  5559 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:24:55.993903  7191 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.041955  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:56.042346  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:56.044459  5445 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 2acd6f5df0204b2295547e021cd8cb9f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4a1c5e6dd8e249bf82d38f90fa37b43d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:55.919739  7208 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.009452  7189 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.027980  7133 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:56.033924  7103 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:56.039723  7266 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:56.045370  6275 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:56.042423  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.032379  7118 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:56.028254  7119 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.060782  7135 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:56.077193  5557 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:56.077518  7135 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.033671  7019 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.064793  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.104557  5555 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:56.039712  7223 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.122032  6273 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:56.096629  7133 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.122362  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.059146  7274 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.104794  7147 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:56.125461  6279 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:24:56.086194  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:56.068755  7170 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:24:56.144162  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 58012us
W20260709 12:24:56.128679  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77210.5us
I20260709 12:24:56.129904  7118 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:56.130877  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 105247us
I20260709 12:24:56.130988  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:56.131817  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.132000  7220 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35612 (request call id 77) took 2463 ms. Trace:
I20260709 12:24:56.096493  7278 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.046387  7272 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.107015  5552 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:56.071187  7216 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:56.134167  7229 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.135919  7191 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.047010  7244 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.127537  7266 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:56.127305  7053 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.883s	user 0.011s	sys 0.032s
I20260709 12:24:56.143113  5706 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:56.048763  7271 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.043402  7073 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.096475  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:56.053598  7178 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.071105  7131 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.150650  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:56.158298  5703 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:24:56.159480  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:56.061363  7132 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.064952  7124 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.068928  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:56.125116  7103 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.178732  5560 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:56.180405  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.068423  7205 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.154875  7196 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:56.190596  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:56.188128  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:56.279882  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:56.280640  5706 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:56.206691  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.207244  7119 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:56.168605  7272 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.207530  7073 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:56.169463  7014 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:24:56.219944  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.215055  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:56.225440  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.277837  6277 raft_consensus.cc:2955] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:56.284763  5560 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
W20260709 12:24:56.280336  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 89868.3us
W20260709 12:24:56.216106  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62328.7us
I20260709 12:24:56.221441  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:56.216498  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.199141  5710 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:24:56.213330  6279 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:24:56.279824  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 121397us
I20260709 12:24:56.285498  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:24:56.303656  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:56.290414  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.105546  7199 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:56.182773  7053 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.075s	user 0.012s	sys 0.032s
I20260709 12:24:56.165355  7243 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.115s
I20260709 12:24:56.149992  7220 rpcz_store.cc:276] 0709 12:24:53.668291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.672260 (+  3969us) service_pool.cc:224] Handling call
0709 12:24:54.008853 (+336593us) ts_tablet_manager.cc:1151] Deleting tablet ccec0cc7269743d6ad581695ea601d1f
0709 12:24:54.008943 (+    90us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:24:55.682016 (+1673073us) tablet_metadata.cc:665] Metadata flushed
0709 12:24:55.869792 (+187776us) tablet_metadata.cc:665] Metadata flushed
0709 12:24:56.131894 (+262102us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1143007,"tablet-delete.queue_time_us":251792,"thread_start_us":216199,"threads_started":1}
I20260709 12:24:56.211014  7178 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.218391  7132 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:56.320475  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.282930  6275 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:24:56.221276  7274 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:56.216962  6272 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:56.332769  5558 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260709 12:24:56.170746  7216 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.212990  7131 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.220140  7019 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:56.215435  7041 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:24:56.337715  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.222059  7069 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:56.218879  7103 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:56.221599  7135 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.337203  5705 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:56.338948  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:56.363813  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:24:56.365132  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:56.318039  7257 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.003s	sys 0.013s
I20260709 12:24:56.220958  7189 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:56.220140  7266 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.340480  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:56.375209  5704 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:56.212678  7229 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.283784  7177 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:56.208295  7208 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:56.218459  7124 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:56.344844  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:56.345292  5706 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:24:56.332641  7133 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.181645  7147 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.219023  7281 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.124s
I20260709 12:24:56.382817  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:56.383337  7281 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:24:56.383805  7281 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:56.348531  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:56.350330  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:56.384877  6278 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:24:56.351424  7276 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:56.385002  7281 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.385527  7066 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260709 12:24:56.386303  7281 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:56.355293  7223 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:56.389168  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:56.389895  7073 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:56.390288  7073 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:56.390836  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.391495  5560 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:56.391587  7114 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:56.218359  7250 raft_consensus.cc:2955] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:56.393590  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:56.356711  7244 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:56.343216  7230 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:56.358083  7118 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.394848  7230 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:56.391286  7131 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.395820  7230 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.396574  7177 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:56.397543  7177 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.398311  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:56.339140  7196 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.356729  5445 catalog_manager.cc:5028] TS 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): tablet ccec0cc7269743d6ad581695ea601d1f (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:24:56.399986  7189 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:24:56.400996  7196 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:56.322719  7285 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:24:56.212783  7278 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.402845  7285 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:56.403122  5555 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:56.403306  7066 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:56.367940  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:56.404211  7066 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.404078  7285 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.406088  7278 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:56.406490  7066 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:56.368150  7272 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:56.373502  7212 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.635s	user 0.010s	sys 0.002s
I20260709 12:24:56.407994  7212 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:56.374584  7257 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.004s	sys 0.014s
I20260709 12:24:56.391263  7073 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.179567  7211 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.408255  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:56.394227  5559 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:56.394870  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:56.373116  7205 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:24:56.400194  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 249581us
I20260709 12:24:56.384054  7147 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.410218  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:56.377326  5526 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.746s (SENT) [suppressed 33 similar messages]
I20260709 12:24:56.419237  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.415895  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:56.421923  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:56.422511  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:56.424419  5706 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
W20260709 12:24:56.425062  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 76625.5us
I20260709 12:24:56.435206  7165 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:56.435669  7165 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.435993  7165 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:56.439307  7073 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:56.443987  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:56.440900  6274 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:56.425446  7131 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:24:56.428371  7089 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:56.448431  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.450235  7131 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.453243  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.454555  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:56.455864  6277 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:24:56.477304  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:56.478086  5630 raft_consensus.cc:2368] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4.
I20260709 12:24:56.478621  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:56.468879  7285 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:56.479592  5709 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:56.479473  7285 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.480062  7041 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:56.481670  7285 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.485570  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:56.486649  7285 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:56.487360  7285 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:56.465986  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:56.490888  5559 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:24:56.490839  6276 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:56.504602  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:56.455186  7272 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:56.462028  7212 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.512100  7285 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.462560  7131 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:24:56.506995  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:56.528542  5705 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:56.528053  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:56.452533  7089 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:56.458105  7216 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:56.508424  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:56.535454  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:56.512871  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:56.514421  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:56.517976  5631 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:24:56.536979  5635 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:56.518075  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.520444  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:56.501425  7041 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:56.530335  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 153018us
I20260709 12:24:56.539983  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:56.533289  7248 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.527962  7075 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:56.542999  7053 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.543627  7075 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.543990  7053 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.544417  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:56.544859  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52546.6us
I20260709 12:24:56.545068  5631 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
W20260709 12:24:56.534451  7277 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.789s	user 0.002s	sys 0.002s
I20260709 12:24:56.518657  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:24:56.525960  7192 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260709 12:24:56.546533  7147 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:24:56.546945  7147 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.547297  7147 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:24:56.547516  7041 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.534909  5634 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:56.529621  5554 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:56.539937  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:56.546070  7075 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:56.540728  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.552109  5555 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 2: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:24:56.543110  7230 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:24:56.537339  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:56.543145  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:56.531555  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:56.553659  5705 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:56.535022  7089 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:56.556589  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:56.560511  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:56.564770  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.564641  6236 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:56.543058  7248 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.552910  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:56.556344  7075 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.566622  7114 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:24:56.574766  5634 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:24:56.550197  7133 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:56.555667  5633 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:24:56.574018  7267 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:56.566705  7230 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:56.566599  5635 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:24:56.583103  7041 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.563964  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:56.550443  7216 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:56.590772  5456 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 172) took 4598 ms. Trace:
I20260709 12:24:56.606616  5599 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:56.602674  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:56.621888  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.546113  7053 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.629812  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:56.630193  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:56.627316  7103 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.631135  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:56.631297  7103 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.631839  6276 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:56.632084  6273 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:56.632526  7205 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:24:56.598816  7257 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:24:56.634011  7205 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:24:56.591857  7192 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.634363  7205 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:56.630252  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:56.566097  7189 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:24:56.641258  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:56.603598  7132 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:24:56.647193  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 95360.5us
I20260709 12:24:56.598963  7267 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:56.601431  7248 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:56.604221  5458 catalog_manager.cc:5697] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 2 to 3, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 3 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:56.652078  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.634676  7192 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.653503  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.626911  7271 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:24:56.646210  7257 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260709 12:24:56.667097  7103 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:56.666752  5458 catalog_manager.cc:5697] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 3 to 4, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 4 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:56.667856  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:56.668316  7132 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.668746  7132 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:56.668653  5458 catalog_manager.cc:5697] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 1 to 2, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 2 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:56.670351  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:56.671023  6272 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:56.670704  5458 catalog_manager.cc:5697] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 1 to 2, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 2 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:56.600407  7230 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.661630  5632 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:24:56.661425  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.665405  7205 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.647374  7244 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.673182  7272 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:56.635435  7069 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:24:56.681404  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 116838us
I20260709 12:24:56.683143  5554 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:24:56.655227  5634 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:24:56.668795  7243 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:56.597756  7133 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.686319  5555 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:56.597347  7075 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:56.691604  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54541.4us
I20260709 12:24:56.711560  5676 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:56.713932  7147 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.715719  7147 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:56.664631  7267 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:56.700251  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:56.716831  7189 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:56.717322  7189 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:24:56.684949  5630 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:56.677510  7165 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.718331  7189 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.704394  5704 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:56.717006  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:56.622678  5456 rpcz_store.cc:276] 0709 12:24:51.992227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.992405 (+   178us) service_pool.cc:224] Handling call
0709 12:24:56.590325 (+4597920us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:54.484151 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:54.646888 (+162737us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:54.648000 (+  1112us) write_op.cc:435] Acquired schema lock
0709 12:24:54.648930 (+   930us) tablet.cc:662] Decoding operations
0709 12:24:54.704240 (+ 55310us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:54.708014 (+  3774us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:54.708058 (+    44us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:24:54.708343 (+   285us) tablet.cc:701] Row locks acquired
0709 12:24:54.708387 (+    44us) write_op.cc:260] PREPARE: finished
0709 12:24:54.708586 (+   199us) write_op.cc:270] Start()
0709 12:24:54.708732 (+   146us) write_op.cc:276] Timestamp: P: 1783599894708528 usec, L: 0
0709 12:24:54.708759 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 12:24:55.093785 (+385026us) log.cc:844] Serialized 527 byte log entry
0709 12:24:55.311070 (+217285us) spinlock_profiling.cc:243] Waited 6.37 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ff75680ac 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b
0709 12:24:55.537297 (+226227us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa66de28 00007f4ffe0a8581 00007f4ffe0a7d58 00007f4ffe0ac9f1 00007f4ffe0ac9a9 00007f4ffe0ac939 00007f4ffe0ac901 00007f4ffe0abb2d 0000000000388634 0000000000388479
0709 12:24:56.038946 (+501649us) op_driver.cc:464] REPLICATION: finished
0709 12:24:56.409314 (+370368us) spinlock_profiling.cc:243] Waited 28.8 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa66f32a 00007f4ffa5f4f14 00007f4ffa5f4ec3 00007f4ffa5f4e38 00007f4ffa5f4df3 00007f4ffa5f40dc 00007f4ffa61de16 00007f4ffa605cbb 00007f4ffa5f039d 00007f4ffa5f3d75
0709 12:24:56.464605 (+ 55291us) write_op.cc:301] APPLY: starting
0709 12:24:56.464821 (+   216us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:56.465016 (+   195us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:56.465064 (+    48us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:56.465481 (+   417us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:56.465771 (+   290us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:24:56.465872 (+   101us) write_op.cc:312] APPLY: finished
0709 12:24:56.466197 (+   325us) log.cc:844] Serialized 23 byte log entry
0709 12:24:56.467529 (+  1332us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:56.467654 (+   125us) write_op.cc:454] Released schema lock
0709 12:24:56.467722 (+    68us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:56.468816 (+  1094us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:56.468844 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":351421,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":398420,"apply.run_cpu_time_us":2582,"apply.run_wall_time_us":16557,"num_ops":1,"prepare.queue_time_us":739615,"prepare.run_cpu_time_us":48660,"prepare.run_wall_time_us":1103342,"raft.queue_time_us":319462,"raft.run_cpu_time_us":39022,"raft.run_wall_time_us":449014,"replication_time_us":1294579,"spinlock_wait_cycles":690503680,"thread_start_us":1545607,"threads_started":4,"wal-append.queue_time_us":436277}]]}
I20260709 12:24:56.723694  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:56.727921  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:56.733393  5555 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:56.736078  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:56.704289  7069 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:56.762671  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.731181  7073 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:56.769248  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:56.733191  6272 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:24:56.732962  7205 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:56.702872  7196 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260709 12:24:56.709314  7271 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260709 12:24:56.775355  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:56.767859  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:56.759860  7069 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:56.771414  7248 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:56.726991  7103 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 12:24:56.781924  7229 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:24:56.811748  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.782023  7075 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:56.766427  7089 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.766539  6275 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:24:56.768190  7243 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:56.819084  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:56.784307  7196 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:56.766461  7257 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:56.767608  7041 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:56.723237  7244 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.835543  7089 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.836377  7244 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:56.798237  7248 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.838743  7248 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:56.786537  6271 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:24:56.773429  7073 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.840482  7073 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:56.841746  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:56.845254  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.846740  7212 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.847333  7212 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:56.833999  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1346) took 1732 ms. Trace:
I20260709 12:24:56.848800  5556 rpcz_store.cc:276] 0709 12:24:55.101136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.110745 (+  9609us) service_pool.cc:224] Handling call
0709 12:24:56.833671 (+1722926us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b54003a89d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:56.833841 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1046739712}
I20260709 12:24:56.815987  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:56.852972  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:56.853617  5634 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:56.854626  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:56.823144  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:56.814960  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:24:56.829995  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 204953us
I20260709 12:24:56.771073  7131 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.833913  7133 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.851482  6272 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:24:56.875648  6277 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: []
I20260709 12:24:56.867437  5561 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:56.813264  7165 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:56.869962  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:56.901376  7287 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:56.902217  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:56.902336  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:56.902863  7287 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:56.903353  5634 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:56.904502  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:56.904104  7287 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:56.905261  6280 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
W20260709 12:24:56.907917  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 108025us
I20260709 12:24:56.910856  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.822872  7196 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:56.912536  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:56.937230  5630 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:24:56.937527  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:56.938191  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:24:56.953900  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 134878us
I20260709 12:24:56.954857  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:56.958117  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:56.964460  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:56.959177  5634 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:56.869200  7041 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:56.969262  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:56.877763  7257 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:56.973696  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:56.973923  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:56.974586  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:56.899187  7019 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:57.161290  7257 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.161882  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:57.161989  7019 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:57.162541  5634 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:24:57.162885  7019 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:57.163822  7287 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:56.972652  5554 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:24:56.933610  7135 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:56.991792  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:57.033179  7192 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:57.169574  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.170250  5554 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:24:56.921499  7133 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:56.962090  7303 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:57.044785  5526 leader_election.cc:400] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4.
I20260709 12:24:56.949012  7178 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:56.971585  7276 raft_consensus.cc:2764] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260709 12:24:57.162578  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 206231us
I20260709 12:24:57.165578  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:57.163880  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:57.076900  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 138821us
I20260709 12:24:57.174196  7196 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.167784  7287 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:57.175655  7320 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:57.165524  7308 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.334s	user 0.003s	sys 0.000s
I20260709 12:24:57.182509  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:56.997641  7243 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:57.183799  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 246473us
I20260709 12:24:57.153339  7311 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:57.189782  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:57.191062  7178 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.210996  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:57.214074  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 221654us
I20260709 12:24:57.214706  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:57.215070  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:57.215345  5709 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
W20260709 12:24:57.215590  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50076.2us
I20260709 12:24:57.216192  5675 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:24:57.218206  5706 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:24:57.209113  7170 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:24:57.223124  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:57.223342  7199 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:24:57.223907  5703 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:24:57.224429  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:57.224658  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:57.225077  7244 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:57.225535  7244 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:57.225775  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:57.226329  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:57.225972  7244 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:57.226428  5705 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:57.232367  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:57.187752  7133 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:57.222381  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:57.233707  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:57.234988  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:57.170280  7267 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.189685  7303 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:57.225392  5708 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:57.230772  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:57.236788  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:57.230770  7209 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:24:57.239289  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.212937  7311 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:57.217831  7124 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:57.236949  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:57.227327  7196 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:57.223132  7170 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:24:57.242380  7314 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:24:57.236801  7298 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:24:57.236521  7133 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:57.238427  7192 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:57.238731  7199 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:24:57.298544  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 118813us
I20260709 12:24:57.243024  7114 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:57.242870  7103 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:57.249706  7278 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:57.221055  7320 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.246809  7303 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:57.249698  7170 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:57.252871  7131 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.251581  7311 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:57.301366  7248 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.224685  7041 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.295506  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:57.320393  7304 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.293s	user 0.002s	sys 0.000s
I20260709 12:24:57.303073  7192 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:57.324097  7132 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.655s	user 0.016s	sys 0.005s
I20260709 12:24:57.310827  7199 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:57.320966  7298 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:24:57.369521  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 123680us
I20260709 12:24:57.376477  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:57.377636  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:57.374332  7192 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:24:57.372014  7312 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.192s	user 0.000s	sys 0.002s
I20260709 12:24:57.363745  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:57.362825  7229 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:57.376600  7199 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:24:57.392596  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 155713us
I20260709 12:24:57.331449  7287 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.352175  7248 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.379438  7320 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:57.389461  7278 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:24:57.391029  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 216) took 1044 ms. Trace:
I20260709 12:24:57.388424  7325 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.377417  7132 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.394506  7321 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:57.424086  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:57.388286  7131 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260709 12:24:57.428529  7329 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:57.410883  7278 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:57.410034  6272 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:57.415684  5552 rpcz_store.cc:276] 0709 12:24:56.347072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.352090 (+  5018us) service_pool.cc:224] Handling call
0709 12:24:56.352396 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.408160 (+ 55764us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.408197 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:56.408255 (+    58us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:56.408291 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.380006 (+971715us) spinlock_profiling.cc:243] Waited 55.5 ms on lock 0x7b54000ebad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:57.381109 (+  1103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.389106 (+  7997us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":127707520}
W20260709 12:24:57.452409  7073 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.611s	user 0.013s	sys 0.000s
I20260709 12:24:57.452965  7073 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.455123  7212 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:57.455899  7073 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:57.456806  5710 raft_consensus.cc:2368] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3.
I20260709 12:24:57.424897  7325 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.462956  7178 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:57.463321  7325 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.463528  7178 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:57.432397  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 225) took 1279 ms. Trace:
I20260709 12:24:57.437287  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:57.463866  7325 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:57.440418  7278 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:24:57.447886  5561 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
W20260709 12:24:57.439246  7310 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
W20260709 12:24:57.459985  7301 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.786s	user 0.007s	sys 0.008s
I20260709 12:24:57.464459  5707 rpcz_store.cc:276] 0709 12:24:56.145948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.405845 (+259897us) service_pool.cc:224] Handling call
0709 12:24:57.375183 (+969338us) spinlock_profiling.cc:243] Waited 7.14 ms on lock 0x7b5800053198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:24:57.385943 (+ 10760us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.395381 (+  9438us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.400928 (+  5547us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:57.404609 (+  3681us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:57.404772 (+   163us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.412034 (+  7262us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.424835 (+ 12801us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16414464}
W20260709 12:24:57.477182  7306 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:24:57.426469  7321 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:57.490305  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1355) took 1417 ms. Trace:
I20260709 12:24:57.510449  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:24:57.507910  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 128565us
I20260709 12:24:57.509729  6278 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:24:57.464015  7178 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.484735  7325 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:57.433775  7329 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:24:57.508549  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 282272us
I20260709 12:24:57.515733  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:57.521528  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:24:57.528609  7069 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.715s	user 0.010s	sys 0.007s
I20260709 12:24:57.529115  7069 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.530721  7170 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.532047  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:57.532295  5633 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:24:57.532761  5631 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:24:57.514863  7321 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:57.514379  7135 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:57.522648  7325 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:57.533903  7271 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:24:57.549043  7271 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:57.532526  7333 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.549482  7271 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:57.549751  7333 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.551913  7333 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.514245  5558 rpcz_store.cc:276] 0709 12:24:56.049872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.346912 (+297040us) service_pool.cc:224] Handling call
0709 12:24:56.840684 (+493772us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.841223 (+   539us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.841268 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:56.841337 (+    69us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:56.841389 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.841476 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.466814 (+625338us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b54003a89d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:57.467049 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5379,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"spinlock_wait_cycles":1098962176}
I20260709 12:24:57.523748  7329 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:57.507808  7298 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.530807  7133 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.547809  7209 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:57.548434  7337 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:24:57.548617  5628 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:57.552883  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:57.526819  7272 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:24:57.557376  7192 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.557732  7267 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.559243  5631 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:57.542685  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:57.568159  7244 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.577026  5702 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 2: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:57.571650  7337 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:24:57.578513  7244 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:57.566375  7333 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 12:24:57.579113  7337 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.580885  5627 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:24:57.584368  7205 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.803s	user 0.012s	sys 0.002s
I20260709 12:24:57.574139  7272 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:24:57.585197  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 357363us
I20260709 12:24:57.586007  7272 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.590199  7192 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:57.588456  7133 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:57.583964  7199 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.613623  7335 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.613945  7199 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.557809  7019 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.614198  7335 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.595639  5556 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:57.596904  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 453) took 1191 ms. Trace:
I20260709 12:24:57.548031  7303 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.604704  7248 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.616573  7133 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:57.618038  7019 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:57.624573  7192 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:57.625280  7192 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.625738  7335 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:57.620858  7271 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.599771  7311 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.628212  7281 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.621649  6275 rpcz_store.cc:276] 0709 12:24:56.405553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.454578 (+ 49025us) service_pool.cc:224] Handling call
0709 12:24:56.454928 (+   350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:56.830516 (+375588us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.596498 (+765982us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b54002eed54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:57.596812 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":611120384}
I20260709 12:24:57.629263  7339 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.630110  7192 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:57.633275  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:57.640170  7328 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.641114  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:57.643744  7281 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.644554  7287 raft_consensus.cc:2764] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:24:57.645136  5553 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:24:57.620858  7321 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.644042  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.645080  7339 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.661499  7189 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:24:57.664628  5552 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:24:57.666136  7271 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.654943  7216 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:57.671499  7216 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.672112  7303 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.672329  7321 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.631460  7133 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.680133  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.680698  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.681380  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.681460  5554 raft_consensus.cc:1813] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Handling vote request from an unknown peer 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:57.683683  5554 raft_consensus.cc:2410] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:24:57.682473  5559 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:24:57.686582  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.687187  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.655072  7328 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.699960  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.700919  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.765921  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.769333  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.770565  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.771790  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.773257  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:57.719187  7069 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:57.775068  7216 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.775466  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 227) took 1151 ms. Trace:
I20260709 12:24:57.705300  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.704918  7118 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:57.713215  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1337) took 1093 ms. Trace:
I20260709 12:24:57.795151  5633 rpcz_store.cc:276] 0709 12:24:56.619546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.620322 (+   776us) service_pool.cc:224] Handling call
0709 12:24:57.712994 (+1092672us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b54001cedd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:57.713148 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2072578816}
I20260709 12:24:57.778584  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.717744  7229 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:57.775898  5701 rpcz_store.cc:276] 0709 12:24:56.624442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.637882 (+ 13440us) service_pool.cc:224] Handling call
0709 12:24:57.774925 (+1137043us) spinlock_profiling.cc:243] Waited 834 ms on lock 0x7b5400452ed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:57.775369 (+   444us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1917226368}
I20260709 12:24:57.707280  7339 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.710521  7325 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.785467  7132 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.785674  7133 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:57.753324  7328 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:57.786638  5552 raft_consensus.cc:2376] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2. Re-sending same reply.
I20260709 12:24:57.807446  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.807528  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.715490  7281 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.714654  7278 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:57.787855  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1345) took 1079 ms. Trace:
I20260709 12:24:57.810369  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:57.810950  5561 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:24:57.811285  7278 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:57.808419  7216 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.812409  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.812561  7216 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:57.813532  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.814350  5552 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 3: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:24:57.789583  5556 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:24:57.790042  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:57.815363  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.810034  5630 rpcz_store.cc:276] 0709 12:24:56.708997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.718228 (+  9231us) service_pool.cc:224] Handling call
0709 12:24:56.718602 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:56.937972 (+219370us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.787574 (+849602us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b54000c1054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:57.787788 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":501860096}
I20260709 12:24:57.820320  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:57.820647  7297 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.154s	user 0.000s	sys 0.022s
I20260709 12:24:57.821477  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:57.822186  5630 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:24:57.822957  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:57.817346  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:57.823707  5630 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:24:57.823871  6275 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:57.815900  7250 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.825590  7250 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.827543  7250 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:57.828045  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:57.828727  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:57.816817  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 475) took 1085 ms. Trace:
I20260709 12:24:57.790746  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:57.716086  7311 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.838901  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:57.839715  5707 raft_consensus.cc:2368] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3.
I20260709 12:24:57.829716  6277 rpcz_store.cc:276] 0709 12:24:56.731025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.731166 (+   141us) service_pool.cc:224] Handling call
0709 12:24:56.875183 (+144017us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.876209 (+  1026us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.816582 (+940373us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54004c7a50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:57.816749 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":330284800}
W20260709 12:24:57.816344  6271 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 459) took 2565 ms (client timeout 1746 ms). Trace:
I20260709 12:24:57.840476  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:57.841223  5701 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:57.839057  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:57.842813  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1533) took 1648 ms. Trace:
I20260709 12:24:57.843220  5704 rpcz_store.cc:276] 0709 12:24:56.194454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.384101 (+189647us) service_pool.cc:224] Handling call
0709 12:24:57.842429 (+1458328us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b5400067054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:57.842754 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":645276800}
I20260709 12:24:57.818903  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:57.832693  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:57.815313  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:57.848558  5552 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260709 12:24:57.848948  5552 pending_rounds.cc:85] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 3
I20260709 12:24:57.849184  5552 pending_rounds.cc:107] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 12:24:57.849611  5552 raft_consensus.cc:2889] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:57.794709  7118 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:57.798945  5557 raft_consensus.cc:2410] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:24:57.820706  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.799156  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.803262  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:57.801832  7229 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:57.806093  7325 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:57.817677  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:57.793906  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:57.812100  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:57.832154  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:57.835511  5561 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:24:57.829140  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1672) took 1054 ms. Trace:
I20260709 12:24:57.838438  7189 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:57.799399  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:57.842046  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:57.851675  7118 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:57.708993  7329 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:58.115796  5602 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.626s (SENT)
I20260709 12:24:58.243198  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:24:58.143548  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 340331us
I20260709 12:24:57.860576  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:58.243664  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:58.328033  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:58.333659  5703 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:24:58.334342  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:58.161803  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:24:58.329691  5631 rpcz_store.cc:276] 0709 12:24:56.774317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.541420 (+767103us) service_pool.cc:224] Handling call
0709 12:24:57.545903 (+  4483us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.567727 (+ 21824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.828947 (+261220us) spinlock_profiling.cc:243] Waited 8.23 ms on lock 0x7b54000c1f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:57.829085 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18920832}
I20260709 12:24:58.338976  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:58.034869  7328 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.341766  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:58.341967  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:58.350667  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:58.343086  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:58.032115  7133 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.902751  7147 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.116122  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:58.330078  5707 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:58.129799  7353 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:58.393016  5553 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:24:57.937049  7276 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.910908  7278 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:24:58.405237  7192 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.422639  7192 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:58.425369  7192 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:58.427480  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:58.428332  5558 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:58.428742  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:58.410768  7147 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:58.433027  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1356) took 1034 ms. Trace:
I20260709 12:24:58.433436  5632 rpcz_store.cc:276] 0709 12:24:57.398523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.846024 (+447501us) service_pool.cc:224] Handling call
0709 12:24:57.846331 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.846881 (+   550us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.846912 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.847330 (+   418us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.142803 (+295473us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:24:58.142846 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.142920 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.432809 (+289889us) raft_consensus.cc:1685] finished
0709 12:24:58.432829 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.432965 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":35336,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":8596,"thread_start_us":87768,"threads_started":1,"wal-append.queue_time_us":543956}
I20260709 12:24:58.434608  7147 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:24:58.435065  7305 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.268s	user 0.009s	sys 0.017s
I20260709 12:24:58.436038  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 231) took 1500 ms. Trace:
I20260709 12:24:58.031857  7019 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.957196  5629 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:58.222566  7216 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:58.395397  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 279656us
W20260709 12:24:58.403556  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 60944us
I20260709 12:24:58.404379  5554 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:58.257525  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:24:57.840853  6271 rpcz_store.cc:269] 0709 12:24:55.251293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.252672 (+  1379us) service_pool.cc:224] Handling call
0709 12:24:57.816096 (+2563424us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54005f5d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:57.816231 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2919341440}
I20260709 12:24:58.349565  5701 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:58.341315  5633 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:24:58.410117  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 476) took 1628 ms. Trace:
I20260709 12:24:58.413525  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1352) took 1440 ms. Trace:
I20260709 12:24:58.414152  5635 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:58.418665  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:58.420341  7276 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:58.408330  7118 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:58.437193  5555 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:24:57.964531  7311 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:58.440218  5628 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:24:58.437739  5710 rpcz_store.cc:276] 0709 12:24:56.935754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.935888 (+   134us) service_pool.cc:224] Handling call
0709 12:24:58.435890 (+1500002us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b54001a8e54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:58.435988 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":664929024}
I20260709 12:24:58.406545  7351 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 12:24:58.456943  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1410) took 1544 ms. Trace:
I20260709 12:24:58.446417  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:58.442375  7019 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:58.457453  5555 rpcz_store.cc:276] 0709 12:24:56.912122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.912277 (+   155us) service_pool.cc:224] Handling call
0709 12:24:58.436786 (+1524509us) spinlock_profiling.cc:243] Waited 8.82 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:24:58.436967 (+   181us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.456559 (+ 19592us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.456870 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20279552}
W20260709 12:24:58.459576  5602 leader_election.cc:336] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.676s (SENT)
I20260709 12:24:58.466137  5636 raft_consensus.cc:2368] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3.
I20260709 12:24:58.466166  5602 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:58.391666  7328 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:58.475744  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 245) took 1529 ms. Trace:
I20260709 12:24:58.476274  5636 rpcz_store.cc:276] 0709 12:24:56.946862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.245400 (+298538us) service_pool.cc:224] Handling call
0709 12:24:58.446203 (+1200803us) spinlock_profiling.cc:243] Waited 1.65 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:24:58.475682 (+ 29479us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3784448}
I20260709 12:24:58.408221  7133 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:58.482676  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:58.464941  7014 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.462623  5552 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:58.462253  7351 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:24:58.457787  7250 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:58.497558  7354 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:24:58.506112  5636 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:58.441593  7353 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:58.488494  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:58.484319  7196 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:24:58.523110  5630 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.674s	user 0.013s	sys 0.053s
W20260709 12:24:58.501190  5602 raft_consensus.cc:2686] T ccec0cc7269743d6ad581695ea601d1f P 4a1c5e6dd8e249bf82d38f90fa37b43d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:24:58.450475  5627 rpcz_store.cc:276] 0709 12:24:56.973634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.231911 (+258277us) service_pool.cc:224] Handling call
0709 12:24:58.413299 (+1181388us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b540009e7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:58.413460 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":494072064}
I20260709 12:24:58.498136  7351 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:58.473302  7311 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:58.654663  5630 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.013s	sys 0.053s
I20260709 12:24:58.655364  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:58.675591  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:58.726008  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:58.727192  5628 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:24:58.730583  5555 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:58.732245  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:58.498255  7014 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:58.685575  7343 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.690035  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:58.753338  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:24:58.754101  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 294579us
I20260709 12:24:58.754217  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:58.760437  5636 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:24:58.509601  7250 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:58.753319  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:58.756326  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:58.764050  5555 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:58.766968  7351 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:58.768286  7351 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:24:58.448702  6273 rpcz_store.cc:276] 0709 12:24:56.781791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.782273 (+   482us) service_pool.cc:224] Handling call
0709 12:24:56.782732 (+   459us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.842437 (+ 59705us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.842491 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:56.842560 (+    69us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:24:56.842615 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.409861 (+1567246us) spinlock_profiling.cc:243] Waited 59.4 ms on lock 0x7b5400534dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:58.409907 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.410052 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":136606080}
I20260709 12:24:58.793727  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:24:58.793180  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 508994us
I20260709 12:24:58.798573  5709 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
W20260709 12:24:58.803745  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1516) took 1589 ms (client timeout 1947 ms). Trace:
W20260709 12:24:58.807237  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.816772  5636 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:24:58.817749  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:58.818048  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 249) took 1027 ms. Trace:
W20260709 12:24:58.804196  5702 rpcz_store.cc:269] 0709 12:24:57.214315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.220309 (+  5994us) service_pool.cc:224] Handling call
0709 12:24:58.798180 (+1577871us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b5400323a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:58.803686 (+  5506us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":757726464}
I20260709 12:24:58.818681  5636 rpcz_store.cc:276] 0709 12:24:57.790480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.766259 (+975779us) service_pool.cc:224] Handling call
0709 12:24:58.816328 (+ 50069us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.817636 (+  1308us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.817958 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:58.808609  5556 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260709 12:24:58.825084  5554 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:58.764870  5633 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:24:58.778867  6275 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.940s	user 0.011s	sys 0.000s
I20260709 12:24:58.838933  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:58.821353  7325 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.826082  5634 raft_consensus.cc:2368] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4.
I20260709 12:24:58.806962  7248 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.792928  7216 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.765372  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:58.762159  7196 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:58.835736  7358 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.429s	user 0.000s	sys 0.001s
I20260709 12:24:58.762275  7343 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:58.821563  5552 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:24:58.765864  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:24:58.845403  5560 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.022s	sys 0.057s
I20260709 12:24:58.821926  7329 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:58.838698  7276 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:24:58.855765  5560 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.023s	sys 0.057s
I20260709 12:24:58.860772  5634 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:58.901561  5634 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
W20260709 12:24:58.903337  6277 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.009s	sys 0.020s
W20260709 12:24:58.904013  6277 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.009s	sys 0.021s
I20260709 12:24:58.857066  7019 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:58.844389  6275 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:24:58.923178  5628 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: []
I20260709 12:24:58.846786  7325 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:58.858183  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1389) took 1003 ms. Trace:
I20260709 12:24:58.892768  7328 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:58.937520  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:24:58.938340  7355 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 12:24:58.938524  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 363) took 1081 ms. Trace:
I20260709 12:24:58.931852  7147 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.934304  7133 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:58.851094  7248 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:58.944276  5555 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:24:58.853250  7216 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:58.934828  7357 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:24:58.950881  7353 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:58.952622  7328 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:58.951996  7216 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:58.953969  7178 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:58.931457  7329 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 12:24:58.954730  7196 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:24:58.956274  5445 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 2acd6f5df0204b2295547e021cd8cb9f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4a1c5e6dd8e249bf82d38f90fa37b43d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:58.963090  7248 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:58.963865  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:58.932087  5552 rpcz_store.cc:276] 0709 12:24:57.854944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.960423 (+105479us) service_pool.cc:224] Handling call
0709 12:24:58.856767 (+896344us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:58.968492  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:58.968734  7311 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:58.970927  7343 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:58.972589  7363 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.973258  7363 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:58.975837  7363 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:58.980885  7250 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:58.933485  7019 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.981897  7019 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.033987  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:59.034857  5558 raft_consensus.cc:1813] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Handling vote request from an unknown peer 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:58.939966  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 458) took 1119 ms. Trace:
I20260709 12:24:58.939160  6272 rpcz_store.cc:276] 0709 12:24:57.857454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.882388 (+ 24934us) service_pool.cc:224] Handling call
0709 12:24:58.937351 (+1054963us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:24:58.938442 (+  1091us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":53428992}
I20260709 12:24:59.041291  5558 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 3 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20260709 12:24:59.087720  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 283441us
I20260709 12:24:59.029803  7019 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:24:59.113095  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 303121us
W20260709 12:24:59.114424  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 278806us
I20260709 12:24:59.115259  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:59.115417  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:58.986043  7147 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:59.126045  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 252) took 1334 ms. Trace:
I20260709 12:24:59.042272  6275 rpcz_store.cc:276] 0709 12:24:57.816008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.817090 (+  1082us) service_pool.cc:224] Handling call
0709 12:24:58.932538 (+1115448us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:59.137673  5526 leader_election.cc:400] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4.
I20260709 12:24:59.138167  5526 leader_election.cc:403] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:24:59.146885  5702 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:59.128753  5636 rpcz_store.cc:276] 0709 12:24:57.791245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.849886 (+1058641us) service_pool.cc:224] Handling call
0709 12:24:58.851981 (+  2095us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:59.029403 (+177422us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:59.029448 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:59.035687 (+  6239us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:24:59.035743 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.125620 (+ 89877us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:59.125695 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.125941 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":851,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":12325,"spinlock_wait_cycles":347616000}
I20260709 12:24:59.148281  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 247) took 1358 ms. Trace:
I20260709 12:24:59.147302  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:24:59.148676  5632 rpcz_store.cc:276] 0709 12:24:57.789877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.436286 (+646409us) service_pool.cc:224] Handling call
0709 12:24:58.437728 (+  1442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.754865 (+317137us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.754903 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:58.754963 (+    60us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:58.755004 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.148082 (+393078us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b540018b854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:59.148122 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.148232 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":728794368}
I20260709 12:24:59.150420  5628 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:24:59.152104  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.152995  5703 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:24:59.131937  7223 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:59.156101  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:59.149204  5701 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 12:24:59.141215  7276 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:59.205302  5632 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:59.139134  7351 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:24:59.166502  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:59.178735  5636 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:59.118573  7353 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260709 12:24:59.206673  5706 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:59.207371  5636 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:59.144620  7363 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:59.141187  7250 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:59.208307  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:59.213106  5703 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:59.211387  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.215229  5708 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:59.204658  5701 pending_rounds.cc:85] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 5
I20260709 12:24:59.217557  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.219053  5704 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:24:59.228274  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 241) took 2284 ms. Trace:
W20260709 12:24:59.229635  5553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1360) took 1550 ms (client timeout 1898 ms). Trace:
I20260709 12:24:59.230067  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1367) took 1548 ms. Trace:
W20260709 12:24:59.229079  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 469266us
I20260709 12:24:59.163996  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 366) took 1275 ms. Trace:
I20260709 12:24:58.239843  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:59.209631  7351 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:59.229990  5553 rpcz_store.cc:269] 0709 12:24:57.678927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.679160 (+   233us) service_pool.cc:224] Handling call
0709 12:24:59.229468 (+1550308us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b54000ec4d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:59.229578 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1589057280}
W20260709 12:24:59.257397  5602 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.790s (SENT)
I20260709 12:24:59.210958  7364 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.210095  7287 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:59.210268  7325 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:59.303143  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:59.228848  5635 rpcz_store.cc:276] 0709 12:24:56.944060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.574469 (+630409us) service_pool.cc:224] Handling call
0709 12:24:58.413966 (+839497us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.448363 (+ 34397us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.228030 (+779667us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b540009e7d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:59.228195 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1833260928}
I20260709 12:24:59.218499  5701 pending_rounds.cc:107] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:24:59.304798  5704 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:24:59.312718  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 269471us
I20260709 12:24:59.313227  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:24:59.317571  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.284322  5706 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:24:59.320456  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:59.314287  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 199110us
I20260709 12:24:59.327921  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:24:59.310734  7229 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.373s	user 0.009s	sys 0.006s
I20260709 12:24:59.264549  7351 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:24:59.331214  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.324657  7287 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.334909  7287 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:59.230445  5560 rpcz_store.cc:276] 0709 12:24:57.682288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.821150 (+138862us) service_pool.cc:224] Handling call
0709 12:24:57.821443 (+   293us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.821800 (+   357us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.821827 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.822291 (+   464us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.258195 (+435904us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:24:58.892790 (+634595us) log.cc:844] Serialized 14 byte log entry
0709 12:24:59.229757 (+336967us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.229834 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.229876 (+    42us) raft_consensus.cc:1685] finished
0709 12:24:59.229888 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.230013 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":678787,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":70024,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":217,"thread_start_us":1303198,"threads_started":3,"wal-append.queue_time_us":900370}
I20260709 12:24:59.205035  7223 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:59.345264  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:59.346980  7223 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:59.347797  7223 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:59.246454  6274 rpcz_store.cc:276] 0709 12:24:57.888928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.921801 (+ 32873us) service_pool.cc:224] Handling call
0709 12:24:59.109062 (+1187261us) spinlock_profiling.cc:243] Waited 66.9 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:24:59.114041 (+  4979us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:59.127935 (+ 13894us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:59.134230 (+  6295us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:59.137601 (+  3371us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:59.142619 (+  5018us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.147418 (+  4799us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.157922 (+ 10504us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":153916800}
I20260709 12:24:59.217535  7250 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:24:59.336898  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 79483.6us
I20260709 12:24:59.337805  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:59.327730  7329 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:59.316962  5701 raft_consensus.cc:2889] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:59.373176  5602 leader_election.cc:336] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.880s (SENT)
I20260709 12:24:59.314451  7368 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:59.341219  7362 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260709 12:24:59.344549  7019 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:24:59.354244  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 207002us
I20260709 12:24:59.354673  5708 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:59.355628  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:59.336282  5703 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:24:59.247257  7178 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:24:59.368206  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 258154us
I20260709 12:24:59.369454  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1355) took 1971 ms. Trace:
W20260709 12:24:59.340708  7366 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:24:59.376302  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1548) took 1531 ms. Trace:
I20260709 12:24:59.378403  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:59.379745  5632 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:24:59.352421  5554 raft_consensus.cc:2955] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:24:59.353534  7311 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:59.389797  6236 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:59.313730  7281 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260709 12:24:59.395236  7386 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:59.402611  7386 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.403136  7147 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:59.403895  7383 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:24:59.404510  7380 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:24:59.405057  7380 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:59.405489  7380 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:24:59.406603  7250 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:59.404995  7386 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:59.409166  7387 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:24:59.386952  5636 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:24:59.351898  7196 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:59.412714  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:59.412985  7196 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:59.414062  7361 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:24:59.414757  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 81994.3us
I20260709 12:24:59.414633  7361 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:59.415009  7196 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:59.226364  7363 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.314054  7364 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:59.393659  5560 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
W20260709 12:24:59.375923  7118 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.910s	user 0.007s	sys 0.006s
I20260709 12:24:59.393428  5709 rpcz_store.cc:276] 0709 12:24:57.833059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.866255 (+ 33196us) service_pool.cc:224] Handling call
0709 12:24:58.798330 (+932075us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.799041 (+   711us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.363840 (+564799us) spinlock_profiling.cc:243] Waited 932 ms on lock 0x7b5400323a50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:59.364039 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2142635264}
I20260709 12:24:59.367378  7373 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.401069  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.368027  7192 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.404574  7383 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:59.413226  7377 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:24:59.393569  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:59.389525  5630 rpcz_store.cc:276] 0709 12:24:57.397951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.844390 (+446439us) service_pool.cc:224] Handling call
0709 12:24:57.844743 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.845414 (+   671us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.845452 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.848586 (+  3134us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.721866 (+873280us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:24:59.247982 (+526116us) log.cc:844] Serialized 14 byte log entry
0709 12:24:59.369137 (+121155us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.369218 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.369258 (+    40us) raft_consensus.cc:1685] finished
0709 12:24:59.369270 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.369401 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4110,"prepare.queue_time_us":628358,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":71,"raft.queue_time_us":62254,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":2320,"thread_start_us":1483660,"threads_started":3,"wal-append.queue_time_us":977580}
W20260709 12:24:59.419725  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46627.2us
I20260709 12:24:59.354347  7229 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:59.378693  7223 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.595757  5703 raft_consensus.cc:1813] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Handling vote request from an unknown peer 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:24:59.548447  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.591456  5458 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 270) took 5577 ms. Trace:
I20260709 12:24:59.483407  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1398) took 1162 ms. Trace:
I20260709 12:24:59.598948  7328 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.486537  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:59.489645  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:59.456836  5704 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:59.505211  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:24:59.548995  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 131917us
I20260709 12:24:59.499416  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:59.596748  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:59.603165  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 103830us
I20260709 12:24:59.391592  7368 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:59.427369  5633 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.589s	user 0.006s	sys 0.001s
I20260709 12:24:59.603261  6236 leader_election.cc:400] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3.
I20260709 12:24:59.615252  6236 leader_election.cc:403] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:24:59.467878  7343 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:24:59.494452  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 362) took 1637 ms. Trace:
I20260709 12:24:59.521912  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:59.501526  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:59.605409  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:59.605975  5703 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 3 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20260709 12:24:59.627950  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 108043us
I20260709 12:24:59.401654  7325 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:24:59.627667  6277 rpcz_store.cc:276] 0709 12:24:57.857142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.857268 (+   126us) service_pool.cc:224] Handling call
0709 12:24:58.161919 (+304651us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.162536 (+   617us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.162576 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.257004 (+ 94428us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.920549 (+663545us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:58.920618 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.920716 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.494259 (+573543us) raft_consensus.cc:1685] finished
0709 12:24:59.494277 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.494398 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":129,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":37317,"spinlock_wait_cycles":23424,"thread_start_us":515005,"threads_started":1,"wal-append.queue_time_us":796305}
I20260709 12:24:59.597805  5559 rpcz_store.cc:276] 0709 12:24:58.320928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.368993 (+ 48065us) service_pool.cc:224] Handling call
0709 12:24:58.393744 (+ 24751us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.393999 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.394025 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.405179 (+ 11154us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.836472 (+431293us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:59.082241 (+245769us) log.cc:844] Serialized 14 byte log entry
0709 12:24:59.082592 (+   351us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.082687 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.483183 (+400496us) raft_consensus.cc:1685] finished
0709 12:24:59.483202 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.483345 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":101130,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":24,"raft.queue_time_us":28169,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":4986,"thread_start_us":650323,"threads_started":2,"wal-append.queue_time_us":639715}
W20260709 12:24:59.403895  7367 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.247s	user 0.000s	sys 0.000s
I20260709 12:24:59.601648  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:59.634747  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:24:59.503144  7371 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:24:59.388814  7329 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.646775  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.647243  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1394) took 1406 ms. Trace:
I20260709 12:24:59.620587  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:59.648568  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.647646  5557 rpcz_store.cc:276] 0709 12:24:58.241135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.365169 (+124034us) service_pool.cc:224] Handling call
0709 12:24:58.393827 (+ 28658us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.394004 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.394029 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.406536 (+ 12507us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.905478 (+498942us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:24:58.905546 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.905626 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.647024 (+741398us) raft_consensus.cc:1685] finished
0709 12:24:59.647047 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.647184 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":113536,"thread_start_us":456003,"threads_started":1,"wal-append.queue_time_us":679050}
I20260709 12:24:59.603842  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:59.580848  7386 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.650177  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1676) took 2124 ms. Trace:
I20260709 12:24:59.650266  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.648015  7329 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:59.491638  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:59.651481  5627 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:59.650686  5631 rpcz_store.cc:276] 0709 12:24:57.526092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.352136 (+826044us) service_pool.cc:224] Handling call
0709 12:24:58.417507 (+ 65371us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.417911 (+   404us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.417945 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.520901 (+102956us) log.cc:844] Serialized 26 byte log entry
0709 12:24:59.097027 (+576126us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:59.097079 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.097173 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.649994 (+552821us) raft_consensus.cc:1685] finished
0709 12:24:59.650014 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.650125 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":22857,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":11909,"thread_start_us":317861,"threads_started":1,"wal-append.queue_time_us":553044}
I20260709 12:24:59.627372  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:59.650202  5709 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:24:59.653180  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.654202  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.650457  7386 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.653205  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:59.655455  5702 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:59.613888  7133 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:59.656309  7386 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:59.656736  7133 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:59.656960  7386 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:24:59.657670  7133 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:59.495683  7285 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:59.658653  7285 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:24:59.425107  7361 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:59.425891  7281 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:59.660722  7281 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:24:59.631424  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:59.605444  5708 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
W20260709 12:24:59.521118  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 197814us
I20260709 12:24:59.641286  7382 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 12:24:59.662295  7281 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:59.601161  7192 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.663442  7281 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:59.428426  7311 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.645126  7257 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:59.609187  7328 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.496454  7363 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:59.649235  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.499723  7041 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:24:59.644282  7189 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 12:24:59.395380  7019 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.651914  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.482811  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.666934  5709 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:59.668089  6234 leader_election.cc:400] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3.
W20260709 12:24:59.757468  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 89580.6us
I20260709 12:24:59.597460  5458 rpcz_store.cc:276] 0709 12:24:54.014297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.014476 (+   179us) service_pool.cc:224] Handling call
0709 12:24:59.591383 (+5576907us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:57.224871 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:57.225518 (+   647us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:57.225577 (+    59us) write_op.cc:435] Acquired schema lock
0709 12:24:57.225594 (+    17us) tablet.cc:662] Decoding operations
0709 12:24:57.364379 (+138785us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:57.382839 (+ 18460us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:57.385994 (+  3155us) tablet.cc:685] Acquiring locks for 4 operations
0709 12:24:57.429291 (+ 43297us) tablet.cc:701] Row locks acquired
0709 12:24:57.429342 (+    51us) write_op.cc:260] PREPARE: finished
0709 12:24:57.429507 (+   165us) write_op.cc:270] Start()
0709 12:24:57.429648 (+   141us) write_op.cc:276] Timestamp: P: 1783599897429446 usec, L: 0
0709 12:24:57.429669 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 12:24:57.430353 (+   684us) log.cc:844] Serialized 1666 byte log entry
0709 12:24:57.559842 (+129489us) op_driver.cc:464] REPLICATION: finished
0709 12:24:57.677449 (+117607us) write_op.cc:301] APPLY: starting
0709 12:24:57.677670 (+   221us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:57.677950 (+   280us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:57.683257 (+  5307us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:57.711329 (+ 28072us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:57.711530 (+   201us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0709 12:24:57.711580 (+    50us) write_op.cc:312] APPLY: finished
0709 12:24:57.711835 (+   255us) log.cc:844] Serialized 41 byte log entry
0709 12:24:57.712021 (+   186us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:57.712130 (+   109us) write_op.cc:454] Released schema lock
0709 12:24:57.712204 (+    74us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:57.712701 (+   497us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:57.712735 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":9638,"reported_tablets":7,"spinlock_wait_cycles":11520,"child_traces":[["op",{"apply.queue_time_us":109706,"apply.run_cpu_time_us":3987,"apply.run_wall_time_us":35254,"num_ops":4,"prepare.queue_time_us":377681,"prepare.run_cpu_time_us":5361,"prepare.run_wall_time_us":205795,"replication_time_us":123503,"thread_start_us":119710,"threads_started":2}]]}
I20260709 12:24:59.672633  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.673808  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.693842  7393 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:59.487134  7250 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.144s
I20260709 12:24:59.693213  5702 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:59.400167  7178 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.769104  7229 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:59.769500  7178 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:24:59.697513  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 93274.4us
I20260709 12:24:59.770349  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:59.771420  7019 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:24:59.699541  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 94195.2us
I20260709 12:24:59.774406  7311 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:59.774616  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:59.776000  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.598651  7383 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.776633  5702 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:59.777523  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.777956  7014 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:59.778087  5702 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:59.778330  7383 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:24:59.779282  5702 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:59.780622  5635 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 1.
I20260709 12:24:59.781261  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:24:59.782354  7276 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.572s	user 0.010s	sys 0.000s
I20260709 12:24:59.782269  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:24:59.782771  7276 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4.
I20260709 12:24:59.783186  7328 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:59.624675  5633 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:24:59.673956  7257 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:59.745663  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 112976us
I20260709 12:24:59.796775  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.746299  5704 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:24:59.699967  7382 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.600654  7377 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:59.633929  7389 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3.
I20260709 12:24:59.759670  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:59.805065  6277 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:24:59.761565  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:59.763134  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:24:59.600328  7373 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.806054  5560 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:24:59.765290  5701 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:59.808924  5701 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:24:59.773109  7368 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:59.784116  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.785467  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:59.771421  5599 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:24:59.810490  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:59.782931  5631 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:59.793115  5702 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:24:59.793877  5558 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:59.794523  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:24:59.799615  5704 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:24:59.776517  7394 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:24:59.812037  5702 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:59.792397  7285 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:24:59.814327  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:59.815246  5555 raft_consensus.cc:2410] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:24:59.796460  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1680) took 1338 ms. Trace:
I20260709 12:24:59.813066  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.796522  7257 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:59.810014  5708 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:24:59.821662  5708 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:24:59.814520  5704 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:59.810701  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:24:59.815953  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:24:59.817600  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:59.819912  7398 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:59.820021  5703 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:24:59.755435  7041 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.805619  7192 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:24:59.822077  5708 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:59.802294  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1341) took 3122 ms. Trace:
W20260709 12:24:59.858639  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.896660  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:59.893033  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:59.889881  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:59.928550  6276 raft_consensus.cc:2410] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260709 12:24:59.925741  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 109846us
W20260709 12:24:59.897930  7351 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.530s	user 0.009s	sys 0.008s
I20260709 12:24:59.944545  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:24:59.936447  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.937114  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:24:59.944614  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:59.947614  5557 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
W20260709 12:24:59.936182  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 125402us
I20260709 12:24:59.883921  5701 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:24:59.860562  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:59.894850  7368 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.940237  5708 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.951151  6236 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:24:59.988096  5708 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:59.804636  7170 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:24:59.951695  5555 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:24:59.939450  7375 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:24:59.911044  7398 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:24:59.995091  7398 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:24:59.995847  7392 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 12:24:59.995702  7014 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:24:59.996805  7014 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:24:59.970875  7368 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:59.997175  7014 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:24:59.997263  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1423) took 1263 ms. Trace:
I20260709 12:24:59.998111  7014 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:24:59.997665  5554 rpcz_store.cc:276] 0709 12:24:58.733730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.831141 (+ 97411us) service_pool.cc:224] Handling call
0709 12:24:59.351929 (+520788us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:59.352223 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:24:59.409423 (+ 57200us) log.cc:844] Serialized 14 byte log entry
0709 12:24:59.478358 (+ 68935us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:59.478431 (+    73us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:24:59.478490 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.478594 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.997087 (+518493us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b54001af750. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:24:59.997208 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":96,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":111,"spinlock_wait_cycles":1174907776,"thread_start_us":68477,"threads_started":1,"wal-append.queue_time_us":199407}
I20260709 12:24:59.999699  7014 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:59.999837  7386 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:24:59.861502  6274 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:24:59.887852  7223 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:00.004854  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:59.813800  7178 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:24:59.952230  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.991787  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:59.811518  7382 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:24:59.993045  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 43794.1us
I20260709 12:24:59.821151  7285 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:24:59.956633  7383 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:24:59.995026  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:24:59.860982  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:00.001609  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:24:59.891870  7133 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:00.001263  5709 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "458143f1c77141febe7d9725625617c2"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:24:59.976267  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.991704  7375 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:59.816735  5633 rpcz_store.cc:276] 0709 12:24:58.458064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.486026 (+ 27962us) service_pool.cc:224] Handling call
0709 12:24:59.796396 (+1310370us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:00.000742  7364 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:00.006626  7170 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:00.042626  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:00.051371  6276 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
W20260709 12:25:00.009316  7380 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.603s	user 0.007s	sys 0.004s
I20260709 12:25:00.043815  5704 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:00.044463  5709 raft_consensus.cc:606] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:24:59.954192  7192 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260709 12:25:00.045614  6279 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:00.046207  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:00.048237  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:00.004722  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:24:59.950913  7351 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:24:59.989480  6275 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:00.050496  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:24:59.951575  7276 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.066s
I20260709 12:25:00.052636  5709 raft_consensus.cc:614] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:00.012826  6271 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4. Ops: []
I20260709 12:24:59.911275  5629 rpcz_store.cc:276] 0709 12:24:56.675496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.675637 (+   141us) service_pool.cc:224] Handling call
0709 12:24:56.675966 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.442395 (+1766429us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.755346 (+1312951us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b5400134f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:24:59.795100 (+ 39754us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2945821056}
I20260709 12:25:00.002137  7384 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.066s
I20260709 12:25:00.061331  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:00.059227  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:00.069278  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:00.069463  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 117175us
I20260709 12:25:00.071293  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:00.047031  7368 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:00.044909  7375 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:00.079048  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:00.069020  5701 raft_consensus.cc:2368] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3.
I20260709 12:25:00.049044  7329 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260709 12:25:00.091796  5561 raft_consensus.cc:2368] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5.
I20260709 12:25:00.062042  7311 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:00.047132  7383 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:00.053722  5627 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:25:00.057334  7363 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
W20260709 12:25:00.096510  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50388.9us
W20260709 12:25:00.100287  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.105535  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:00.049335  7229 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260709 12:25:00.106657  5704 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:25:00.104594  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 153521us
I20260709 12:25:00.098141  5709 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:00.075443  7364 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:00.055703  7118 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 12:25:00.109525  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:00.056274  7285 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260709 12:25:00.053157  7019 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 12:25:00.080894  7377 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:00.055961  7380 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:00.106000  5707 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:00.095014  7328 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:25:00.213707  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 104269us
I20260709 12:25:00.204818  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:00.729385  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 524664us
I20260709 12:25:00.218550  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:00.240736  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:00.375160  7416 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:25:00.486891  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1703) took 1070 ms. Trace:
W20260709 12:25:00.492820  5628 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.004s	sys 0.004s
W20260709 12:25:00.495728  5508 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 350439us
W20260709 12:25:00.207059  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 156679us
W20260709 12:25:00.381142  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:00.300885  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 231680us
I20260709 12:25:00.391958  5703 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:25:00.745498  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:00.745779  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:00.746824  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:00.747139  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:00.747435  5557 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:00.747745  5555 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:25:00.747869  5558 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:25:00.748275  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:00.748528  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:00.201886  5704 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:00.751601  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1569) took 1141 ms. Trace:
I20260709 12:25:00.365648  5708 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260709 12:25:00.753787  5708 pending_rounds.cc:85] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 4
I20260709 12:25:00.754127  5708 pending_rounds.cc:107] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260709 12:25:00.754438  5708 raft_consensus.cc:2889] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:00.755307  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 245) took 1281 ms. Trace:
I20260709 12:25:00.755689  5708 rpcz_store.cc:276] 0709 12:24:59.474490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.821287 (+346797us) service_pool.cc:224] Handling call
0709 12:24:59.821547 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.754974 (+933427us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.755235 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:00.301626  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 180871us
I20260709 12:25:00.215152  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:00.251472  6271 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:25:00.214174  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:00.789659  6274 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:00.204027  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:00.279186  6279 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:25:00.793318  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 578226us
I20260709 12:25:00.374574  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:00.795014  6272 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:00.450644  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:00.830101  6273 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:00.830914  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 481) took 1027 ms. Trace:
I20260709 12:25:00.557709  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:00.833555  5552 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 3 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:25:00.835243  7429 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:00.835829  5552 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:00.259495  7041 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:00.246089  7133 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:00.388029  7178 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:00.124104  7189 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.871423  7189 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:00.289745  7223 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:00.881642  7223 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:00.883086  7223 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:00.886026  7435 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:00.886469  7435 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:00.120590  7250 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:00.887391  7435 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:00.374022  7257 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.548s	user 0.010s	sys 0.001s
I20260709 12:25:00.890100  7435 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:00.890209  7257 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:00.555676  7276 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.899820  7276 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:00.295516  7311 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:00.901989  7311 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:00.903999  7276 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:00.208103  7329 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260709 12:25:00.282491  7351 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:00.490494  7361 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:00.913223  7361 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:00.914307  7361 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:00.915333  7361 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:00.915704  7311 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:00.293152  7368 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:00.550187  7373 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:00.390307  7375 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:00.306820  7380 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:00.276371  7382 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:00.466881  7383 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:00.444424  7384 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:00.925233  7384 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:00.927433  7384 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:00.367959  7386 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:00.224758  7394 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:25:00.382774  7393 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.588s	user 0.009s	sys 0.002s
I20260709 12:25:00.226203  7396 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:00.208403  7397 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:00.936195  7394 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:00.940126  7397 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:00.940968  7394 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:00.941485  7397 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:00.302722  7398 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:00.112969  7402 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:00.556471  7414 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.947921  7414 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:00.950337  7414 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:00.584718  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:00.594000  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.780s	user 0.006s	sys 0.016s
I20260709 12:25:00.951093  5553 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:25:00.951275  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.140s	user 0.007s	sys 0.016s
I20260709 12:25:00.595443  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1697) took 1275 ms. Trace:
W20260709 12:25:00.951761  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.951938  5635 rpcz_store.cc:276] 0709 12:24:59.319537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.319739 (+   202us) service_pool.cc:224] Handling call
0709 12:25:00.595248 (+1275509us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b540018d154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:00.595377 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":954240896}
I20260709 12:25:00.681954  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:00.691790  7389 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:00.957271  7389 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:00.958544  7389 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:00.959688  7389 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:25:00.696023  5561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1403) took 2233 ms (client timeout 2148 ms). Trace:
I20260709 12:25:00.717613  7409 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:00.212855  5631 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:00.960702  7409 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:00.961035  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1707) took 1179 ms. Trace:
I20260709 12:25:00.961440  5631 rpcz_store.cc:276] 0709 12:24:59.781805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.781997 (+   192us) service_pool.cc:224] Handling call
0709 12:25:00.960964 (+1178967us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:00.727941  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.729473  7229 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:25:00.962591  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.737274  7325 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:25:00.738570  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 520075us
I20260709 12:25:00.963792  5705 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90c64867bae6493c846d941815aa1074"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:43276
W20260709 12:25:00.960528  5561 rpcz_store.cc:269] 0709 12:24:58.463055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.463225 (+   170us) service_pool.cc:224] Handling call
0709 12:24:59.489496 (+1026271us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:00.695759 (+1206263us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x7b54002283d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:00.695922 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1133112448}
I20260709 12:25:00.964572  7257 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:00.736758  5630 rpcz_store.cc:276] 0709 12:24:59.416801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.636165 (+219364us) service_pool.cc:224] Handling call
0709 12:24:59.636584 (+   419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:59.694782 (+ 58198us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:59.694809 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:59.745507 (+ 50698us) log.cc:844] Serialized 26 byte log entry
0709 12:25:00.056982 (+311475us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:00.057083 (+   101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.057196 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:00.486591 (+429395us) raft_consensus.cc:1685] finished
0709 12:25:00.486607 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:00.486812 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3877,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":120,"thread_start_us":252285,"threads_started":1,"wal-append.queue_time_us":349184}
I20260709 12:25:00.749073  5555 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:25:00.746829  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:00.291575  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:00.752017  5704 rpcz_store.cc:276] 0709 12:24:59.609914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.145283 (+535369us) service_pool.cc:224] Handling call
0709 12:25:00.201644 (+ 56361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.751293 (+549649us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.751535 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:00.759783  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:00.763223  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:00.977818  5557 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
W20260709 12:25:00.978214  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 215083us
W20260709 12:25:00.979058  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 129319us
I20260709 12:25:00.978060  7257 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:00.777902  7410 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:00.778330  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:00.794160  7170 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:00.795814  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:00.981134  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:00.982378  7412 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:00.984165  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:00.800189  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:00.800915  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:00.985360  5634 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:00.984925  7410 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:00.796552  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:00.987290  7410 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:00.823207  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 92811.2us
I20260709 12:25:00.825697  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:00.988852  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 163236us
I20260709 12:25:00.989321  5675 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:00.989657  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:00.831384  6273 rpcz_store.cc:276] 0709 12:24:59.803852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.805495 (+  1643us) service_pool.cc:224] Handling call
0709 12:25:00.450427 (+644932us) spinlock_profiling.cc:243] Waited 4.89 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:00.830814 (+380387us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11254656}
I20260709 12:25:00.844835  5456 catalog_manager.cc:5697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 2 to 3, term changed from 1 to 2, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to <none>, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:00.866369  5560 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:25:00.874511  7429 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:25:01.003038  7257 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:01.003762  7395 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:01.004155  7395 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:01.004566  7395 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:00.874416  7189 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:01.005645  7250 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:25:01.007129  7398 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:01.007330  7250 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:00.892246  5702 raft_consensus.cc:2368] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3.
W20260709 12:25:00.239694  7281 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.576s	user 0.007s	sys 0.005s
I20260709 12:25:00.217741  7328 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:01.046550  7328 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:00.239904  7364 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:01.057938  7364 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:01.058880  7364 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.059791  7364 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:01.062459  7395 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:00.918603  7368 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:00.891836  7223 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:00.924248  7375 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:00.929032  7311 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:00.943518  7397 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:01.066452  7368 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:00.951792  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 1.14129e+06us
I20260709 12:25:01.067458  7368 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:01.067991  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:01.069113  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:01.069934  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:01.070947  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:00.962425  7229 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:01.072536  7398 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:25:01.073195  7398 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:00.131696  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:00.939258  7396 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:01.077818  7396 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.079816  7395 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:01.080529  7178 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:01.080940  7178 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:01.081293  7178 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:00.967137  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:01.085057  7133 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:25:01.085286  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 118209us
W20260709 12:25:00.737468  5628 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.004s	sys 0.004s
I20260709 12:25:00.967459  7409 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:01.087720  7133 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:01.088552  7409 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:01.088804  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 255) took 1422 ms. Trace:
I20260709 12:25:01.089161  5628 rpcz_store.cc:276] 0709 12:24:59.666238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.694719 (+ 28481us) service_pool.cc:224] Handling call
0709 12:24:59.695386 (+   667us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:59.798489 (+103103us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:59.798516 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:59.799293 (+   777us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.088375 (+1289082us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:01.088422 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.088507 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:01.088558 (+    51us) raft_consensus.cc:1685] finished
0709 12:25:01.088573 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:01.088731 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3121,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":71,"thread_start_us":1329267,"threads_started":2,"wal-append.queue_time_us":554299}
I20260709 12:25:01.091168  7229 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:01.096799  7402 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:01.098670  7402 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:01.099251  7402 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:25:01.076035  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 944386us
I20260709 12:25:00.975040  5553 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:00.938186  7393 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:00.977337  5706 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:01.087801  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:00.984431  7170 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:01.108165  7311 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:01.108551  7170 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:01.109386  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:00.806842  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:01.111737  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 304985us
I20260709 12:25:01.113063  7389 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:01.113417  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:01.114776  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.996711  7410 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:01.115379  7438 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:01.000044  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:01.116323  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 116347us
I20260709 12:25:01.117235  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:01.119329  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:00.995349  5524 leader_election.cc:336] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.508s (SENT)
I20260709 12:25:01.001034  5708 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:01.120265  7361 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:01.001868  7379 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:00.997952  5456 catalog_manager.cc:5697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 2 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:01.121193  7379 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:01.121218  7361 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:01.121894  7379 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:01.122277  5456 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 2, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 2 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:01.123211  7170 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.009609  5675 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:01.001672  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 144800us
I20260709 12:25:01.013002  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.013963  5631 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
W20260709 12:25:01.015779  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1573) took 1203 ms (client timeout 1508 ms). Trace:
I20260709 12:25:01.016389  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.012125  7412 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:01.114765  6237 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:00.988091  5632 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:25:01.121407  7329 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:01.122689  7361 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:01.010571  5634 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:01.010864  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:01.124924  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 115408us
I20260709 12:25:01.125212  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:01.126011  5630 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:25:01.126329  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:01.129522  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 118721us
I20260709 12:25:01.136641  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:01.131362  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:01.125348  5706 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
W20260709 12:25:01.126411  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54514.5us
I20260709 12:25:01.131219  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:01.126909  5702 rpcz_store.cc:269] 0709 12:24:59.812745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.812835 (+    90us) service_pool.cc:224] Handling call
0709 12:25:01.015584 (+1202749us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400309654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:01.015723 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2471930112}
I20260709 12:25:01.127300  7412 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:01.127746  7386 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:01.127456  5633 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 1.
I20260709 12:25:01.124315  5456 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 3, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 3 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:01.146495  5456 catalog_manager.cc:5697] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 2, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 2 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:01.152792  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:01.154596  5706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1533) took 1796 ms (client timeout 1598 ms). Trace:
W20260709 12:25:01.155257  5706 rpcz_store.cc:269] 0709 12:24:59.342459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.291165 (+948706us) service_pool.cc:224] Handling call
0709 12:25:01.139299 (+848134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:01.148851  5456 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 3 to 4, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 4 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:01.160787  5602 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:01.161044  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46378.7us
I20260709 12:25:01.161720  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:01.158097  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:25:01.165055  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 169736us
W20260709 12:25:01.167025  5524 leader_election.cc:336] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.528s (SENT)
I20260709 12:25:01.167449  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:01.168655  5524 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.534s (SENT)
W20260709 12:25:01.169453  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:01.169864  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.160064  5456 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 2 to 4, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 4 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:01.128700  7170 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:01.183496  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:01.186064  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 56011.4us
I20260709 12:25:01.235306  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:01.193207  5445 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 2acd6f5df0204b2295547e021cd8cb9f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4a1c5e6dd8e249bf82d38f90fa37b43d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:01.237689  5553 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:25:01.238188  5445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45050.1us
W20260709 12:25:01.235844  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49564.9us
I20260709 12:25:01.188928  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:01.127671  7351 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:01.153913  7311 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:01.113471  7389 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.243361  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.267498  5708 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:25:01.267908  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 107192us
I20260709 12:25:01.245483  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:01.144608  7412 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:01.250674  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 79919.9us
I20260709 12:25:01.160673  7223 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:01.235558  7375 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:01.193353  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:01.242673  7118 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:01.253996  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.254127  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:01.255242  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 70528.5us
I20260709 12:25:01.290146  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:01.127969  7329 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:01.127673  7410 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:25:01.257997  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:01.258479  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 100448us
W20260709 12:25:01.258656  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 71908.7us
I20260709 12:25:01.157027  7393 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:01.266171  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:01.270314  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:01.273977  7384 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:01.186735  7364 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:01.189622  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.157135  7386 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:01.279467  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.279155  7223 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.192847  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:01.194216  7435 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:01.128504  7427 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.126s
I20260709 12:25:01.248762  7383 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:01.282933  5557 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:01.258373  7380 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 12:25:01.172609  5456 catalog_manager.cc:5697] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 3, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 3 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:01.286551  5553 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:01.255366  7424 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:25:01.295615  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.296099  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.298092  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:01.160921  7394 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.116s
I20260709 12:25:01.304952  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.274746  7389 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:01.306129  5561 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:01.271843  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.307255  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:01.375367  5702 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:01.273702  7411 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:01.286095  7445 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 12:25:01.186879  7281 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:01.282729  7375 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.310326  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:01.316989  7383 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:01.317909  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47661.2us
I20260709 12:25:01.288089  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:01.292032  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:01.348091  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:01.279863  7429 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
W20260709 12:25:01.349380  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51408.7us
I20260709 12:25:01.350770  5708 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 2: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:01.391256  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:01.301963  7189 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.317935  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:01.307508  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 156977us
I20260709 12:25:01.376901  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.309916  7311 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:01.377003  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:01.349886  5456 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:01.378931  5709 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:25:01.300487  7351 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260709 12:25:01.318761  7223 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:25:01.384228  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 74400.6us
I20260709 12:25:01.241511  7430 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260709 12:25:01.316973  7014 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:01.320967  7384 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:01.390978  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:01.324640  7328 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:01.394508  6278 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:01.389549  7383 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:01.393404  5702 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:01.381705  7426 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:01.396957  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.446161  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:01.315480  7422 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20260709 12:25:01.396804  5456 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 3, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 3 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:01.387459  7375 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:25:01.379041  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 75579.7us
I20260709 12:25:01.378546  7364 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:01.392108  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.480837  5708 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:25:01.404453  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:01.388110  7373 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:01.439849  5561 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
W20260709 12:25:01.437528  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44111.7us
I20260709 12:25:01.482373  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:25:01.483098  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 78749.4us
I20260709 12:25:01.438475  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:01.441404  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48391.2us
I20260709 12:25:01.485116  5602 leader_election.cc:400] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5.
I20260709 12:25:01.397513  7410 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:01.443821  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:01.488611  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.400919  7438 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:01.387169  7424 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 12:25:01.492941  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:01.452762  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 51653.1us
I20260709 12:25:01.448422  5709 raft_consensus.cc:2368] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4.
I20260709 12:25:01.399610  7386 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:01.495693  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.496456  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:01.473663  7448 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:25:01.501415  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.512152  5706 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:25:01.516649  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.518386  5706 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:25:01.518567  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.495787  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.502064  5456 catalog_manager.cc:5697] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:01.509610  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:01.514755  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:01.520814  5630 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:01.520849  6278 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:01.519831  5702 raft_consensus.cc:2410] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:01.526276  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.501072  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:01.474895  7229 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.532644  7041 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:01.533375  7041 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:01.534425  7041 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:01.508913  7455 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:01.536815  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.536940  7455 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.537833  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:01.538556  7380 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:01.539710  7455 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:01.540323  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:01.474151  5552 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:01.542064  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.552747  5635 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:01.397832  7189 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.493261  7364 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:01.446636  7325 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:01.446943  7426 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.549669  5630 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:01.557874  5631 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:25:01.384119  7411 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.489212  7170 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:01.544835  5599 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:01.436700  7014 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.496239  7438 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:25:01.562839  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 285061us
I20260709 12:25:01.571256  7351 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.501269  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:01.402809  7329 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:01.499475  7409 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.644098  5703 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:01.597266  5635 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
W20260709 12:25:01.583472  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.448549  7433 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.152s
W20260709 12:25:01.603153  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 59992us
I20260709 12:25:01.653338  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:01.550453  7281 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:01.657900  7281 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:01.658890  7281 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.475523  7178 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.660981  7411 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:01.661844  7411 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:01.662168  7178 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:01.540521  7229 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:01.644070  7424 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260709 12:25:01.663187  5632 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 2: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:01.660182  7409 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.652161  7455 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 12:25:01.664649  7455 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:01.664989  7455 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:01.665014  7409 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:01.665335  7402 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.566s	user 0.010s	sys 0.003s
I20260709 12:25:01.665740  6276 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:25:01.666343  7281 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:01.636940  7351 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:01.670516  7351 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:01.644541  7452 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:01.672318  6280 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 2. Current term is 3. Ops: []
W20260709 12:25:01.605027  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 48184us
I20260709 12:25:01.673447  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:01.597210  5630 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:01.556463  7368 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:01.609400  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.678709  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.679828  5635 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:01.680837  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:01.682049  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:01.543092  7451 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.126s
I20260709 12:25:01.615300  7328 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:01.616339  7470 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:01.571480  7250 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:25:01.566224  5657 debug-util.cc:398] Leaking SignalData structure 0x7b08001643a0 after lost signal to thread 4359
W20260709 12:25:01.689239  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.628968  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:01.694097  5553 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:25:01.629480  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:01.598691  7426 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:01.501072  7410 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.635432  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:01.704659  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 69288.6us
I20260709 12:25:01.705826  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:01.633905  7438 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:01.684686  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:01.686343  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.630259  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:01.688651  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:01.599380  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.674255  5629 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:01.659009  7014 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:01.632655  7393 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260709 12:25:01.711002  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:01.712442  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.712287  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.706727  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:01.724685  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 179965us
I20260709 12:25:01.726245  6276 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:01.685969  7424 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:01.718235  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:01.719901  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 66626.7us
I20260709 12:25:01.688489  7257 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:25:01.711651  7397 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.645s	user 0.001s	sys 0.010s
I20260709 12:25:01.703325  7396 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:01.604957  7384 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.778854  6278 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:01.732340  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.734330  7325 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:01.735270  7118 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:01.665820  7455 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:01.736635  5627 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:01.737665  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:25:01.740630  5657 debug-util.cc:398] Leaking SignalData structure 0x7b08001e5860 after lost signal to thread 5493
I20260709 12:25:01.631335  7458 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.117s
I20260709 12:25:01.726418  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:25:01.814419  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 88078.2us
I20260709 12:25:01.815088  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:01.816507  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:25:01.768728  7465 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:25:01.665740  7402 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.692551  7470 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:01.776690  7468 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:25:01.715726  7328 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:01.778343  5705 raft_consensus.cc:606] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:01.688427  7368 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:01.723956  7409 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:01.746266  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.810359  5560 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:01.809811  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:01.812163  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 74569.5us
I20260709 12:25:01.742619  7257 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:01.748200  7463 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:01.819970  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:01.821638  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.814522  7398 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.823253  7461 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:01.869273  7461 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:01.869378  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:01.870093  6276 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:01.742738  7472 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:01.826735  7438 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.829470  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:01.817523  7118 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:01.797325  7325 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:01.838296  5705 raft_consensus.cc:614] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:01.847236  7410 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:01.848665  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:01.885386  5555 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:01.882982  7474 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:01.886409  7474 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:01.886873  7474 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:01.833915  7329 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.039s
I20260709 12:25:01.813158  7412 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:01.829138  7133 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:01.893018  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1438) took 1125 ms. Trace:
I20260709 12:25:01.893460  5552 rpcz_store.cc:276] 0709 12:25:00.767660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.842277 (+ 74617us) service_pool.cc:224] Handling call
0709 12:25:00.842636 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:01.555992 (+713356us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.892730 (+336738us) spinlock_profiling.cc:243] Waited 631 ms on lock 0x7b54000eb5d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:01.892950 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1451792768}
I20260709 12:25:01.861102  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:01.863299  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:01.863826  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:01.868235  7398 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:01.870958  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:01.876768  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:01.838076  7328 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:01.814323  7375 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:01.881198  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51844.9us
I20260709 12:25:01.810169  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.799098  7396 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:01.903164  5524 leader_election.cc:400] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3.
I20260709 12:25:01.884785  7364 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:01.870235  7461 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:01.864996  7257 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:01.887517  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:01.883358  7118 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:01.853243  7351 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.039s
I20260709 12:25:01.901867  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:01.947467  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45684.6us
I20260709 12:25:01.910218  7373 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:01.904385  7398 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:01.904731  7457 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:01.905442  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:01.908391  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:01.889565  7395 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:01.884529  5657 debug-util.cc:398] Leaking SignalData structure 0x7b08001af0a0 after lost signal to thread 5585
I20260709 12:25:01.839318  7223 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.899858  6276 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:01.913054  5599 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:01.880940  7463 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:01.928309  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:01.858745  7435 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.972970  5634 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:01.973110  7435 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.973347  7463 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.937044  7325 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:01.977408  7478 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:01.977964  7478 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:01.978348  7478 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:01.936273  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:01.981194  7470 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:01.981720  7395 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:01.982185  7424 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:01.983223  7375 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:01.983563  7368 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:01.983489  5560 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:01.970850  7351 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.947853  7118 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:01.888334  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.892239  7133 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:01.933077  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:01.889874  7458 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.943013  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:01.943848  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56439.9us
I20260709 12:25:01.962102  7364 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:25:01.985913  7379 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.863s	user 0.010s	sys 0.000s
I20260709 12:25:01.921231  7472 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:25:01.946246  7482 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:25:01.903275  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.948876  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:01.926070  7328 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:01.925850  7430 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:25:01.903573  7422 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:01.968896  5704 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:01.969214  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.928668  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:02.003170  5634 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:02.014155  7435 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:02.006529  5702 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:01.984292  7395 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:01.944262  5708 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:02.015532  6276 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
W20260709 12:25:01.984828  7383 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.506s	user 0.008s	sys 0.001s
W20260709 12:25:01.985682  5657 debug-util.cc:398] Leaking SignalData structure 0x7b080014d960 after lost signal to thread 5660
I20260709 12:25:02.027681  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:01.988523  7454 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:02.026508  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:01.987426  7223 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:02.030697  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 81892.7us
I20260709 12:25:01.989148  7402 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:02.007663  7398 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:02.011842  7133 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:02.036073  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:02.036600  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:02.065184  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:02.041329  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:02.004878  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 91898us
I20260709 12:25:02.025626  7375 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260709 12:25:02.066121  5634 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 2: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:02.066594  7375 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:02.066664  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:02.067026  7375 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:02.067518  7452 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:01.992103  7257 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.067919  7375 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.069183  7375 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:02.069952  7257 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:02.020856  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:02.018472  7458 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.071929  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:43276 (request call id 8) took 1108 ms. Trace:
I20260709 12:25:02.047703  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:02.072310  5705 rpcz_store.cc:276] 0709 12:25:00.963388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.963644 (+   256us) service_pool.cc:224] Handling call
0709 12:25:02.071723 (+1108079us) spinlock_profiling.cc:243] Waited 814 ms on lock 0x7b540026eb54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa66c853 00007f4fff63a60a 00007f4ff8bc07d4 00007f4ff8bc0762 00007f4ff8bc0691 00007f4ff8bc060c 00007f4ff8bbf8c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:02.071851 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1871615360}
I20260709 12:25:02.070003  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:02.073573  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:02.074195  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:02.048074  7438 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:02.049885  7472 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:02.050655  5657 debug-util.cc:398] Leaking SignalData structure 0x7b0800346dc0 after lost signal to thread 5734
I20260709 12:25:02.029699  7325 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:02.053931  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:02.075981  5657 debug-util.cc:398] Leaking SignalData structure 0x7b0800347840 after lost signal to thread 6304
W20260709 12:25:02.076295  5657 debug-util.cc:398] Leaking SignalData structure 0x7b08002ff140 after lost signal to thread 6305
I20260709 12:25:02.076249  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:02.077584  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:02.054502  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:02.078261  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:02.078598  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:02.079133  5710 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:02.033834  7276 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:02.072320  7458 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:02.074227  6271 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:25:02.082571  7276 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:02.083168  7276 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:02.037123  7470 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:01.988071  7351 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.063392  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:02.068010  7452 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:02.094309  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:02.056599  7383 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.099458  5710 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:02.101075  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:02.102262  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:02.103932  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:02.062856  7389 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.681s	user 0.011s	sys 0.000s
I20260709 12:25:02.107381  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:02.043212  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:02.064199  7394 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.127099  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:02.118446  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:02.027846  7379 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.134208  5635 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:02.127815  5705 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:02.128099  5631 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:02.129014  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:02.134030  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:02.094023  7398 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:02.101660  7397 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:02.135984  7394 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.109311  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:02.137841  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:02.142324  5710 raft_consensus.cc:2410] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:25:02.119714  7402 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:02.114024  7461 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:02.144830  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:02.139571  7398 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:02.145566  7383 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:02.146682  7379 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:02.147342  5627 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:02.038749  7396 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.148420  6274 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:02.148469  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:02.156939  5553 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:02.122545  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:02.150959  5635 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:02.057096  7478 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.161859  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:02.127655  7389 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.167116  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:02.170037  5599 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:02.151618  7351 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:02.171669  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:02.178715  7469 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:02.167309  7041 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:02.172338  7311 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:25:02.180560  7453 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:02.185212  6277 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:02.149191  7426 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.181051  7189 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:02.192446  7469 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:02.168859  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:02.203012  5705 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:02.209841  7469 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.181294  7461 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:02.213340  5553 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:02.168768  7452 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.231185  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:02.198191  7328 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.228914  7379 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.253113  5557 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:02.255635  7329 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:02.264457  7478 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:02.266273  7389 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:02.218359  7474 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.269836  7474 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:02.272504  7379 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.275525  5632 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:02.223340  7118 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.275672  7379 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:02.288758  7394 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:02.290493  7328 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:02.285595  6275 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:02.283203  7426 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:02.286643  7133 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.295397  7133 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:02.297039  6279 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:02.297966  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 466) took 1022 ms. Trace:
I20260709 12:25:02.298334  6279 rpcz_store.cc:276] 0709 12:25:01.275910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.278282 (+  2372us) service_pool.cc:224] Handling call
0709 12:25:02.296611 (+1018329us) spinlock_profiling.cc:243] Waited 3.35 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:25:02.296831 (+   220us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.297522 (+   691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.297894 (+   372us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7700864}
I20260709 12:25:02.261716  7393 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:02.303347  7393 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:02.304054  7276 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.304312  7393 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.256237  7329 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:02.301782  6237 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:02.310114  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:02.342540  5561 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:02.344837  7453 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:02.344748  7479 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:02.348557  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:02.345662  7479 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:02.351581  7475 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:02.326529  7426 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:02.366616  5561 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:02.381227  7276 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:02.382372  7118 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:02.339581  7398 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.384598  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:02.386054  7398 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:02.354071  7223 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:02.387701  5560 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:02.352600  7479 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.394212  7479 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:02.353528  7475 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:02.398684  7475 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.400727  7223 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:02.376471  7329 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.432809  7223 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.422477  7329 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:02.452894  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:02.455960  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:02.458406  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:02.459514  5559 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:02.463235  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:02.470852  5701 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:02.473865  5560 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:02.511745  5559 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:02.514272  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1580) took 1260 ms. Trace:
I20260709 12:25:02.514848  5704 rpcz_store.cc:276] 0709 12:25:01.253790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.253912 (+   122us) service_pool.cc:224] Handling call
0709 12:25:02.514052 (+1260140us) spinlock_profiling.cc:243] Waited 594 ms on lock 0x7b540026eb54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:02.514190 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1366877440}
I20260709 12:25:02.519347  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:02.520080  5704 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:02.526504  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:02.527359  7389 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.529477  7389 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.529937  5704 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:02.531567  5599 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:02.533166  5559 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:25:02.537719  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 255) took 1135 ms. Trace:
I20260709 12:25:02.538254  5702 rpcz_store.cc:276] 0709 12:25:01.402815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.525880 (+123065us) service_pool.cc:224] Handling call
0709 12:25:02.537476 (+1011596us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x7b5400067054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:02.537644 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1070975360}
I20260709 12:25:02.539148  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:02.539728  5702 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:02.542766  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:02.542680  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:02.540294  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:02.544559  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:02.545042  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:02.546980  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:02.547752  5559 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:02.547690  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:02.545783  5702 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:02.548767  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:02.549299  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.533253  7469 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:02.553725  5635 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:02.556136  5631 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:02.559816  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:02.564150  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:02.553812  5629 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:02.561689  7469 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:02.568651  5559 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:02.560509  7396 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:02.557997  7189 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260709 12:25:02.568033  7380 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:25:02.575204  5702 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:25:02.569273  7429 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:02.559173  7479 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 12:25:02.569152  7426 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260709 12:25:02.578281  5702 pending_rounds.cc:85] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 2
I20260709 12:25:02.581118  5702 pending_rounds.cc:107] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:25:02.581525  5702 raft_consensus.cc:2889] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:02.566738  7435 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260709 12:25:02.572005  7434 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:25:02.586244  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:02.575279  7396 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:02.587572  5702 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:02.590037  7248 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:02.590436  7248 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:02.591235  7248 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.592165  7248 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:02.580982  7429 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:02.585983  7389 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:02.600661  7477 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.749s	user 0.015s	sys 0.011s
I20260709 12:25:02.598824  5707 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:02.607013  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 263) took 1054 ms. Trace:
I20260709 12:25:02.588215  7452 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 12:25:02.607430  5709 rpcz_store.cc:276] 0709 12:25:01.552456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.626187 (+ 73731us) service_pool.cc:224] Handling call
0709 12:25:01.626465 (+   278us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.626883 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:01.626911 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:01.632772 (+  5861us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.838463 (+205691us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:01.882934 (+ 44471us) log.cc:844] Serialized 14 byte log entry
0709 12:25:01.888540 (+  5606us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.888627 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.600193 (+711566us) spinlock_profiling.cc:243] Waited 4.33 ms on lock 0x7b240025b878. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff740b9c4 00007f4ff7409acb 00007f4ff75c4dc9 00007f4ff75c4a55 00007f4ff75c409d 00007f4ffa1f610e 00007f4ffa20d88c 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221
0709 12:25:02.606752 (+  6559us) raft_consensus.cc:1685] finished
0709 12:25:02.606837 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.606958 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5286,"prepare.queue_time_us":91405,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":11772,"raft.queue_time_us":48036,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":38243,"spinlock_wait_cycles":9957120,"thread_start_us":160172,"threads_started":2,"wal-append.queue_time_us":168021}
I20260709 12:25:02.616046  5709 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:25:02.618690  7508 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
W20260709 12:25:02.619993  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.623243  7451 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:02.625654  7429 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.628865  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:02.602828  7469 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.630734  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:02.631646  6272 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:02.638202  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:02.640211  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:02.639251  6273 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:25:02.641237  7453 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:02.642226  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:02.645157  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:02.611479  7393 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 12:25:02.647387  7133 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:02.652215  7461 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:02.651196  7394 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:02.652689  7461 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:02.653609  7461 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.648187  7434 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260709 12:25:02.654630  7461 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:02.655128  7434 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:02.655575  7434 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:02.646332  7451 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:25:02.657995  5701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1582) took 1403 ms (client timeout 1735 ms). Trace:
W20260709 12:25:02.658504  5701 rpcz_store.cc:269] 0709 12:25:01.254451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.257864 (+  3413us) service_pool.cc:224] Handling call
0709 12:25:02.657793 (+1399929us) spinlock_profiling.cc:243] Waited 880 ms on lock 0x7b540006efd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:02.657931 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2022982912}
I20260709 12:25:02.608994  7396 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.665047  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:02.665189  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:02.665783  5629 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:02.678177  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1583) took 1420 ms. Trace:
I20260709 12:25:02.678735  5703 rpcz_store.cc:276] 0709 12:25:01.257480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.257743 (+   263us) service_pool.cc:224] Handling call
0709 12:25:02.677904 (+1420161us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b5400197dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:02.678092 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":796751104}
I20260709 12:25:02.685356  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:02.689630  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 475) took 1284 ms. Trace:
I20260709 12:25:02.690196  6275 rpcz_store.cc:276] 0709 12:25:01.404864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.446128 (+ 41264us) service_pool.cc:224] Handling call
0709 12:25:02.689405 (+1243277us) spinlock_profiling.cc:243] Waited 335 ms on lock 0x7b5400773154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:02.689553 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":770238208}
I20260709 12:25:02.656498  7434 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.701562  7434 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:02.703850  7411 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.704823  7411 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.657138  7223 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.706789  7223 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.707329  7433 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:02.707374  7411 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:02.707839  7433 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:02.708739  7433 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.709942  7433 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:25:02.714704  5675 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.554s (SENT) [suppressed 39 similar messages]
I20260709 12:25:02.656713  7414 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:25:02.723289  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.657330  7451 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.732710  5702 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:02.733448  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:02.736902  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:02.738503  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:02.742949  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:02.744469  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:02.734462  7429 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.747460  5634 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:02.732981  7133 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.735011  7451 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:02.748170  5634 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:02.728299  7414 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:02.749930  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:02.756774  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:02.762723  5628 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:02.758224  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:02.761641  5634 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.757519  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:02.758757  7429 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.769730  7429 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:02.770108  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:02.775097  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 470) took 1456 ms. Trace:
I20260709 12:25:02.775458  6277 rpcz_store.cc:276] 0709 12:25:01.318913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.319155 (+   242us) service_pool.cc:224] Handling call
0709 12:25:01.908232 (+589077us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:02.774937 (+866705us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b5400771d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:02.775050 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":512258688}
I20260709 12:25:02.760886  7133 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.786898  5635 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:02.791867  7459 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:02.790287  5627 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 2: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:02.788170  7453 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260709 12:25:02.801426  5629 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:02.820180  7454 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:02.820910  7454 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:02.832305  7133 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:02.838086  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:02.793824  7223 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:02.794914  7459 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:02.844950  7459 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.847914  7459 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:02.856745  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:02.857872  6277 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:25:02.859238  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 930581us
I20260709 12:25:02.860390  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 258) took 1355 ms. Trace:
I20260709 12:25:02.860487  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:02.861968  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:02.861042  5706 rpcz_store.cc:276] 0709 12:25:01.504832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.541464 (+ 36632us) service_pool.cc:224] Handling call
0709 12:25:02.692622 (+1151158us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:02.693013 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.693042 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:02.693555 (+   513us) log.cc:844] Serialized 26 byte log entry
0709 12:25:02.760449 (+ 66894us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:02.778051 (+ 17602us) log.cc:844] Serialized 14 byte log entry
0709 12:25:02.778363 (+   312us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.778464 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.795181 (+ 16717us) raft_consensus.cc:1685] finished
0709 12:25:02.795216 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.860123 (+ 64907us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400197dd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:02.860289 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1400,"prepare.queue_time_us":15778,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":2157,"raft.queue_time_us":665,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":4503,"spinlock_wait_cycles":2646555520,"thread_start_us":65666,"threads_started":1,"wal-append.queue_time_us":73328}
I20260709 12:25:02.869874  6277 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:02.872586  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:02.874509  7457 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:25:02.875156  7518 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:25:02.875346  7457 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:25:02.880203  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 859400us
W20260709 12:25:02.881431  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.881786  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:02.898332  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:02.903559  6280 raft_consensus.cc:2368] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3.
I20260709 12:25:02.890970  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:02.904670  6277 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
W20260709 12:25:02.885094  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.915823  5709 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:02.918715  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 255) took 1001 ms. Trace:
I20260709 12:25:02.919281  5554 rpcz_store.cc:276] 0709 12:25:01.916882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.917435 (+   553us) service_pool.cc:224] Handling call
0709 12:25:01.917849 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:01.918101 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:25:01.941816 (+ 23715us) log.cc:844] Serialized 14 byte log entry
0709 12:25:02.295974 (+354158us) spinlock_profiling.cc:243] Waited 6.26 ms on lock 0x7b18002dc9a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:25:02.918279 (+622305us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.918358 (+    79us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:02.918416 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.918501 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.918644 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":455012,"prepare.queue_time_us":23526,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"spinlock_wait_cycles":14393600,"thread_start_us":166962,"threads_started":1,"wal-append.queue_time_us":519646}
I20260709 12:25:02.924490  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:02.884871  7383 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.929771  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:02.897928  7170 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:02.900046  5633 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:02.902160  7422 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:02.936343  7170 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:02.939236  5707 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:02.939723  7170 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.932391  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:02.941262  7469 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:02.937762  7411 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:02.941315  7474 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:02.942608  7411 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:02.908721  7423 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:02.943364  7423 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.885151  7455 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:02.943744  7423 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:02.943591  7411 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.936837  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1375) took 1232 ms. Trace:
I20260709 12:25:02.910274  7463 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 12:25:02.944881  5633 rpcz_store.cc:276] 0709 12:25:01.703976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.899368 (+1195392us) service_pool.cc:224] Handling call
0709 12:25:02.899791 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.936571 (+ 36780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.936776 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:02.945822  7463 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:02.946218  7463 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:02.947175  7463 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.898051  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:02.942893  7470 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:02.948531  7383 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.949386  7470 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:02.950408  7470 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.951160  7383 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:02.909564  7451 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:02.955648  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:02.956230  5704 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:02.960479  7451 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:02.961784  5627 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:02.877216  7457 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:02.937144  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:02.963925  5704 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:02.900238  7281 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:02.961953  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:02.955955  5709 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:25:02.926373  7519 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:25:02.977316  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 262630us
I20260709 12:25:02.977968  7451 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:25:03.082412  5628 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:25:02.978799  7463 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:03.084200  7412 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.084946  7412 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:02.983992  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 467) took 1703 ms. Trace:
I20260709 12:25:03.088236  7412 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:03.088578  6274 rpcz_store.cc:276] 0709 12:25:01.281159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.283548 (+  2389us) service_pool.cc:224] Handling call
0709 12:25:01.355335 (+ 71787us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.161604 (+806269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.983697 (+822093us) spinlock_profiling.cc:243] Waited 777 ms on lock 0x7b54005eef54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:02.983920 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1788049664}
I20260709 12:25:02.984560  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:02.984771  5708 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1591) took 1685 ms (client timeout 2037 ms). Trace:
I20260709 12:25:02.985549  5705 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:25:02.986310  7414 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:02.986912  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.987468  7368 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.090396  6280 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:03.090835  7368 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:03.092015  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1546) took 1020 ms. Trace:
I20260709 12:25:02.990389  7455 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:03.092438  5705 rpcz_store.cc:276] 0709 12:25:02.071845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.233393 (+161548us) service_pool.cc:224] Handling call
0709 12:25:02.985168 (+751775us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:03.090460 (+105292us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.091826 (+  1366us) spinlock_profiling.cc:243] Waited 751 ms on lock 0x7b54000e0bd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:03.091959 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1727881088}
I20260709 12:25:02.990204  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:03.093067  7368 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:02.988022  7229 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.994352  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:03.003155  5629 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:02.996502  7223 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.012609  5710 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:03.012962  7474 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:03.014056  7170 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:03.014408  5635 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:03.020074  5456 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 173) took 3248 ms. Trace:
W20260709 12:25:03.112668  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 250781us
I20260709 12:25:03.113559  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:03.037305  7410 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.090274  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:03.093734  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 103649us
I20260709 12:25:03.123224  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:02.993674  5634 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:25:03.090279  5708 rpcz_store.cc:269] 0709 12:25:01.299626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.536463 (+236837us) service_pool.cc:224] Handling call
0709 12:25:02.984565 (+1448102us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b54000e0bd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:02.984713 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":934873344}
W20260709 12:25:03.095660  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.097815  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.127271  5553 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:03.099198  5555 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:03.127960  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:03.107347  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.129091  5559 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:03.132629  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:25:03.133208  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 138974us
I20260709 12:25:03.109265  5558 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:03.114872  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:03.121749  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:03.134802  5560 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:03.135975  5559 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:03.101136  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.136684  5561 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:03.145252  5560 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:03.146107  5554 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:03.149067  5458 catalog_manager.cc:5697] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 1 to 2, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 2 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:03.150337  5458 catalog_manager.cc:5697] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 1 to 2, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 2 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:03.151242  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:03.155397  5634 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:03.093915  7368 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:03.157733  5458 catalog_manager.cc:5697] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 1 to 2, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 2 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:03.098037  7398 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.165041  5628 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:03.165923  5635 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 12:25:03.162492  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.167099  5524 leader_election.cc:400] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4.
I20260709 12:25:03.162900  5526 leader_election.cc:400] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3.
I20260709 12:25:03.167810  5526 leader_election.cc:403] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:25:03.168051  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:03.172803  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:03.133198  7414 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:03.055477  7454 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:03.139963  7424 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:03.132459  7455 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:03.181916  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.189647  5559 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:03.191545  5561 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:03.112515  5456 rpcz_store.cc:276] 0709 12:24:59.771829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.772033 (+   204us) service_pool.cc:224] Handling call
0709 12:25:03.019967 (+3247934us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:01.822020 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:01.880190 (+ 58170us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:01.883464 (+  3274us) write_op.cc:435] Acquired schema lock
0709 12:25:01.887682 (+  4218us) tablet.cc:662] Decoding operations
0709 12:25:01.988851 (+101169us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:01.988930 (+    79us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:01.988956 (+    26us) tablet.cc:685] Acquiring locks for 11 operations
0709 12:25:01.989788 (+   832us) tablet.cc:701] Row locks acquired
0709 12:25:01.989806 (+    18us) write_op.cc:260] PREPARE: finished
0709 12:25:01.989948 (+   142us) write_op.cc:270] Start()
0709 12:25:01.990067 (+   119us) write_op.cc:276] Timestamp: P: 1783599901989907 usec, L: 0
0709 12:25:01.990096 (+    29us) op_driver.cc:348] REPLICATION: starting
0709 12:25:01.990852 (+   756us) log.cc:844] Serialized 4290 byte log entry
0709 12:25:02.572691 (+581839us) op_driver.cc:464] REPLICATION: finished
0709 12:25:02.614212 (+ 41521us) write_op.cc:301] APPLY: starting
0709 12:25:02.614295 (+    83us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:02.614509 (+   214us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:02.619235 (+  4726us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:02.622487 (+  3252us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:02.623758 (+  1271us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0709 12:25:02.623821 (+    63us) write_op.cc:312] APPLY: finished
0709 12:25:02.624289 (+   468us) log.cc:844] Serialized 83 byte log entry
0709 12:25:02.624528 (+   239us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:02.624717 (+   189us) write_op.cc:454] Released schema lock
0709 12:25:02.624787 (+    70us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:02.625200 (+   413us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:02.625226 (+    26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":369,"reported_tablets":14,"child_traces":[["op",{"apply.queue_time_us":40164,"apply.run_cpu_time_us":4211,"apply.run_wall_time_us":10960,"num_ops":11,"prepare.queue_time_us":248895,"prepare.run_cpu_time_us":31303,"prepare.run_wall_time_us":305054,"raft.queue_time_us":329561,"raft.run_cpu_time_us":23690,"raft.run_wall_time_us":40873,"replication_time_us":580647,"thread_start_us":677065,"threads_started":4,"wal-append.queue_time_us":226644}]]}
I20260709 12:25:03.189481  5458 catalog_manager.cc:5697] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 3 to 4, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 4 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:03.159152  7475 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:03.154400  7379 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:03.187338  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:03.120991  7229 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:03.153539  7389 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:03.185604  5634 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:03.154635  7422 raft_consensus.cc:2764] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:25:03.197556  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.201061  7379 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:03.137585  7423 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:03.202725  7368 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:03.206796  7383 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:03.138500  7461 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20260709 12:25:03.133811  7474 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:03.178581  7479 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260709 12:25:03.208823  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.188256  7434 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:03.120934  7223 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:03.210088  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 92928.2us
I20260709 12:25:03.196099  5458 catalog_manager.cc:5697] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:25:03.211714  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 43745.5us
I20260709 12:25:03.244896  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:03.244544  5458 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 2 to 3, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 3 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:03.245445  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.213243  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:03.246109  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:03.139714  7410 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:03.216900  7379 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:03.199047  7453 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.197469  7351 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3.
I20260709 12:25:03.229357  5554 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:25:03.207150  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:03.204262  7133 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:25:03.266439  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.266772  7133 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:03.204028  7433 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:03.268872  7433 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:03.203794  7452 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.269313  7133 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:03.269639  7452 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:03.194072  7531 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:03.236595  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:03.240453  5561 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
W20260709 12:25:03.243302  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 71348.7us
I20260709 12:25:03.247797  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:03.178643  7398 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:03.200673  5552 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:03.180028  7411 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 12:25:03.220255  7361 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:03.251888  5458 catalog_manager.cc:5697] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 2 to 1, leader changed from <none> to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 1 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:03.209441  7118 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.038s
I20260709 12:25:03.261749  7461 consensus_queue.cc:1243] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:25:03.264247  7223 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:03.266000  7410 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:03.226140  7475 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260709 12:25:03.243172  7328 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.218822  7281 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:25:03.275135  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:03.308846  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:03.283310  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.310343  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:03.306674  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:03.274334  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:03.313891  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:03.290024  6280 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:03.317865  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:03.323480  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:03.327346  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:03.252262  7409 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:03.311864  5634 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:03.278966  7118 consensus_queue.cc:1243] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:25:03.254875  7384 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:03.278017  7411 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:03.278657  7423 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:03.335238  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:03.348467  5560 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:25:03.350909  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:03.527030  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 191883us
I20260709 12:25:03.529332  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:03.529894  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:03.530045  7410 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:03.530652  5705 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:03.532704  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:03.533330  5704 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:25:03.534454  5707 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:03.536684  5555 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:03.530084  5710 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:03.535867  7410 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:25:03.537416  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.538369  5553 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:03.538648  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.272331  7452 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:03.269315  7433 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.539999  7433 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:03.345871  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:03.542201  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 196442us
W20260709 12:25:03.508483  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 262432us
I20260709 12:25:03.279105  7424 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:03.276013  7398 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:03.541599  7276 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:03.298621  7454 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:03.272657  7531 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:03.280032  7474 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:03.345961  7530 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:25:03.333137  7281 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:03.280665  7328 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:03.280143  7453 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:03.354391  7384 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:03.361802  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:03.363929  7411 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:25:03.481451  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 234450us
I20260709 12:25:03.345775  7412 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:03.324530  7470 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:03.325668  7351 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:03.302803  7368 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:03.510577  5602 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.821s (SENT)
I20260709 12:25:03.594355  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:03.510993  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 274537us
I20260709 12:25:03.333318  7422 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:03.598351  5709 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:03.351449  7457 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:03.599756  7457 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:03.277410  7479 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:03.600509  7479 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:03.601490  7479 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:03.333554  7414 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:03.514155  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.600229  7457 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:03.603391  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:03.513710  5559 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9ff38a877a89473392a3e27657516383"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:47734
I20260709 12:25:03.605186  5559 raft_consensus.cc:606] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:03.605526  5559 raft_consensus.cc:614] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:03.348977  7396 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:03.607479  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:03.607791  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:03.567548  7276 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:25:03.610522  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.550140  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:03.611024  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.611550  5555 raft_consensus.cc:2368] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7.
I20260709 12:25:03.551278  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.543429  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:03.300681  7229 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:03.561381  7223 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.565697  7475 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.622888  7475 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:03.512007  7409 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:03.624459  5628 raft_consensus.cc:1217] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Deduplicated request from leader. Original: 1.1->[2.2-2.2]   Dedup: 2.2->[]
W20260709 12:25:03.624817  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 81470.6us
I20260709 12:25:03.625008  5628 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:25:03.569289  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.588307  7384 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:03.639739  7459 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:03.640668  7543 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:03.642025  7409 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:03.576138  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.570287  7423 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:03.608532  5554 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:25:03.646696  5524 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.037s (SENT)
I20260709 12:25:03.580521  7531 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:25:03.615694  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 105146us
W20260709 12:25:03.648903  5602 leader_election.cc:336] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.898s (SENT)
W20260709 12:25:03.650326  5602 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.993s (SENT)
I20260709 12:25:03.650839  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:03.616488  5558 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:03.619230  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:03.621374  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.622179  7248 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:03.625576  5561 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:03.653807  5555 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:03.621676  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:03.630141  7223 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:03.570109  7542 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260709 12:25:03.636530  7229 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.2, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:03.637045  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:03.624918  7454 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:03.638777  5627 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:03.617988  7414 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:03.588121  7411 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:03.544874  5634 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:03.647193  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:03.657006  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:25:03.559340  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:03.682862  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 123654us
I20260709 12:25:03.659583  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:03.685274  5555 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:03.640169  7459 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:03.652202  5602 leader_election.cc:336] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.014s (SENT)
W20260709 12:25:03.638626  7170 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.505s	user 0.008s	sys 0.004s
I20260709 12:25:03.638624  7453 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:03.646572  7423 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.685623  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:03.708681  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:03.665704  7248 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:03.656260  7523 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:03.690539  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:03.690541  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.662164  7461 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:03.662892  7474 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:25:03.693213  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 163755us
I20260709 12:25:03.692174  5709 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:03.647876  7379 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:03.687845  5559 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:03.721694  5555 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:03.680862  7229 consensus_queue.cc:1243] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:25:03.694526  7459 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:03.694020  7455 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:03.727751  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:03.713119  7423 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:03.693830  7133 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:03.732671  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:03.662317  7457 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:03.767491  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:03.790256  5710 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:03.791657  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:03.693298  7414 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:03.806345  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.728983  7523 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:03.808382  5553 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:03.729225  7461 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:03.812233  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.820300  5561 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:03.820657  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.821679  7223 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:03.821614  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:03.822221  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.742563  7133 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.736745  7475 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:03.829083  5557 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:03.738068  7540 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.734933  7474 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:03.854463  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:03.827440  7475 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.832538  5561 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:03.818367  7386 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:03.852274  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.855016  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.880124  5557 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:03.825161  7133 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:03.857138  5558 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:03.823534  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.907824  5553 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:03.737841  7544 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:03.814468  7549 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:03.790449  7396 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:03.909637  5553 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:25:03.932195  6280 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:03.935913  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:03.937033  6277 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:03.937713  7562 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:03.938230  7562 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:03.943183  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.944314  5559 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
W20260709 12:25:03.947098  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 155564us
I20260709 12:25:03.948187  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:03.906152  5561 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:03.876959  7223 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:25:03.876542  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:03.979401  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.981470  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:03.878851  7328 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:03.878683  7386 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:03.878204  7451 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:03.986248  5559 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:03.988131  7170 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.989085  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:03.989959  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:03.993018  7248 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:03.994513  7434 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.994522  7396 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:03.983346  7394 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 12:25:03.809523  7276 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:04.006567  5555 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:04.045271  5555 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:25:04.046746  5555 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:04.067533  5555 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:03.934626  7549 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.828449  7523 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.010942  7229 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.004s
W20260709 12:25:04.011780  7529 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:25:03.883661  7475 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:04.026669  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 342384us
W20260709 12:25:04.027873  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.088343  5561 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 12:25:04.103929  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.040421  5559 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:25:04.107197  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:03.931432  7540 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:04.095103  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:04.117947  6277 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:04.014564  7384 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.042564  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:04.113619  5558 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:04.119776  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:04.075435  7394 consensus_queue.cc:1243] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:25:04.080523  7248 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.021471  7451 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:04.115937  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:04.114607  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:04.115319  6279 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:25:04.125280  5705 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:25:04.125600  5558 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:25:04.120792  5709 raft_consensus.cc:2410] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:25:04.122678  5559 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:04.121693  5706 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:25:04.128680  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:04.129261  5561 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:25:04.092855  7470 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260709 12:25:04.131014  6272 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:25:04.131928  7223 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:04.132541  7223 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.132658  7248 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.133862  7414 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:04.134593  7223 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:04.135017  7461 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:04.124455  7542 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260709 12:25:04.136427  7379 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:04.136569  7461 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:25:04.138271  7461 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:04.139245  7394 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:04.140269  7394 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:04.134593  7248 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:04.141109  5554 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:04.140856  7394 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.141959  7475 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:04.038600  7434 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.143225  7394 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:04.097455  7386 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:04.144454  7434 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:04.144711  7386 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:04.145696  6273 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:04.147421  7544 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:04.148123  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:04.148725  5707 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:25:04.150187  5710 raft_consensus.cc:1217] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[2.2-2.2]   Dedup: 2.2->[]
I20260709 12:25:04.150723  7523 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:04.102193  7433 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.562s	user 0.007s	sys 0.004s
I20260709 12:25:04.158474  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:04.150725  5710 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:25:04.157668  6276 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:04.082975  7549 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:04.127065  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:04.166795  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:04.021806  7170 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.159406  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:04.169495  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:04.165513  7429 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.170621  7429 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.171002  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:04.165023  7462 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.037580  7474 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.172961  7429 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:04.174317  7540 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:04.174401  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:04.120712  7552 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20260709 12:25:04.176185  7540 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:04.176576  7540 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:04.176944  7540 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:04.011313  7562 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.178905  7562 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:04.179433  7411 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:04.082681  7328 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20260709 12:25:04.165333  7434 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260709 12:25:04.181492  7434 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:04.181840  7434 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:04.181974  7455 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:04.093843  7422 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
I20260709 12:25:04.182895  7328 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:04.102378  7559 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:04.132119  7384 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:04.168084  5705 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:04.256464  7396 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:25:04.160552  5706 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:04.257510  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:04.167829  5555 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:04.261132  6272 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:25:04.262420  7552 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:04.262959  7552 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.186339  5704 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:04.172900  7462 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.237462  7429 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:04.254017  7547 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.243s	user 0.003s	sys 0.000s
W20260709 12:25:04.292127  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.293906  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:04.311046  5703 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:04.311177  5561 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
W20260709 12:25:04.332682  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 81336.7us
I20260709 12:25:04.334184  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:04.237170  7328 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:04.247779  7544 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:04.182968  7434 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.306125  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:04.379809  7546 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:04.183985  7433 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.247514  7459 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:04.388396  5708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.020s	sys 0.021s
W20260709 12:25:04.393466  5708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.021s	sys 0.021s
I20260709 12:25:04.352401  7414 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260709 12:25:04.404117  6280 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:25:04.406297  6275 raft_consensus.cc:2368] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5.
I20260709 12:25:04.405978  7133 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:04.299578  7475 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.095s
I20260709 12:25:04.319053  7429 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:04.399073  7409 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.024s	sys 0.023s
W20260709 12:25:04.398699  7393 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.033s	sys 0.017s
I20260709 12:25:04.375108  7544 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:04.444083  7368 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.006s	sys 0.013s
W20260709 12:25:04.434477  7409 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.024s	sys 0.023s
W20260709 12:25:04.444375  7393 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.034s	sys 0.017s
W20260709 12:25:04.452786  5557 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.008s	sys 0.011s
W20260709 12:25:04.466761  5557 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.009s	sys 0.011s
I20260709 12:25:04.432649  7564 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:04.457543  7368 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.006s	sys 0.013s
I20260709 12:25:04.465785  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:04.476538  7133 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:04.464538  7569 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:04.477555  7451 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:04.477833  7569 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.478848  7462 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:04.459996  7552 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:04.481463  5676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.008s	sys 0.014s
W20260709 12:25:04.481791  5676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.008s	sys 0.014s
W20260709 12:25:04.481556  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 439075us
I20260709 12:25:04.483201  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:04.484627  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.489333  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:04.464915  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1740) took 1154 ms. Trace:
I20260709 12:25:04.502803  5708 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:04.500121  7411 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:04.506062  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:04.550330  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:04.551044  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45071.4us
I20260709 12:25:04.551759  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:04.479965  7170 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:04.586169  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.572629  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:04.465175  7540 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.476613  7389 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:04.479404  7549 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:04.639986  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 67482.9us
W20260709 12:25:04.646809  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 312708us
I20260709 12:25:04.669322  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:04.669515  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:04.671654  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:04.661206  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 66127.7us
W20260709 12:25:04.661726  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.004s	sys 0.072s
W20260709 12:25:04.672616  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.004s	sys 0.073s
W20260709 12:25:04.672890  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 546422us
W20260709 12:25:04.661990  5600 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.044s	sys 0.033s
W20260709 12:25:04.673585  5600 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.044s	sys 0.033s
I20260709 12:25:04.675370  5635 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:25:04.709089  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:04.710136  5635 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:25:04.570732  7451 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:04.707062  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:04.744401  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:04.747880  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:04.751025  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.745812  5524 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:04.763908  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 94685.6us
I20260709 12:25:04.595252  7532 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:25:04.766220  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 93493.7us
W20260709 12:25:04.766855  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 93434.9us
I20260709 12:25:04.773422  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:04.596926  7565 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:04.775650  5709 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:04.642706  7389 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:04.660729  7368 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:04.706128  7559 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.660717  7398 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:04.705945  7409 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:04.738036  7453 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:04.737758  7414 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:04.737298  7459 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:04.740411  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:04.763329  7546 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:04.790812  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:04.839813  7577 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:04.840812  7276 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:04.841307  5601 leader_election.cc:400] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5.
I20260709 12:25:04.500926  5629 rpcz_store.cc:276] 0709 12:25:03.309126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.320762 (+ 11636us) service_pool.cc:224] Handling call
0709 12:25:03.415452 (+ 94690us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:03.522628 (+107176us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.522675 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:03.523356 (+   681us) log.cc:844] Serialized 52 byte log entry
0709 12:25:03.732885 (+209529us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:03.750269 (+ 17384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.791757 (+ 41488us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.253439 (+461682us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:04.447066 (+193627us) raft_consensus.cc:1685] finished
0709 12:25:04.447095 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.462007 (+ 14912us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3249,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":126,"spinlock_wait_cycles":5610368,"thread_start_us":186125,"threads_started":1,"wal-append.queue_time_us":354865}
I20260709 12:25:04.831816  7575 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:04.844599  7575 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:04.845031  7575 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:04.832542  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:04.846171  5633 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
W20260709 12:25:04.849217  7572 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.369s	user 0.001s	sys 0.000s
I20260709 12:25:04.777141  7474 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:04.850080  7474 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:04.850476  7474 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:25:04.850682  7582 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.298s	user 0.000s	sys 0.001s
I20260709 12:25:04.801573  7532 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.853125  7532 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.854212  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1419) took 1304 ms. Trace:
I20260709 12:25:04.854562  5557 rpcz_store.cc:276] 0709 12:25:03.549552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.881042 (+331490us) service_pool.cc:224] Handling call
0709 12:25:03.881478 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:03.881886 (+   408us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.881926 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:03.882526 (+   600us) log.cc:844] Serialized 26 byte log entry
0709 12:25:04.468608 (+586082us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:04.468649 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.468728 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.853994 (+385266us) raft_consensus.cc:1685] finished
0709 12:25:04.854013 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.854156 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8630,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":14410,"thread_start_us":585668,"threads_started":1,"wal-append.queue_time_us":969667}
I20260709 12:25:04.793375  7433 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:04.741643  7560 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
I20260709 12:25:04.794723  7569 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:04.743269  7451 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:25:04.831262  7570 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
I20260709 12:25:04.774163  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:04.869788  5554 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.003s	sys 0.014s
I20260709 12:25:04.813172  7453 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:25:04.881575  5554 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.003s	sys 0.014s
I20260709 12:25:04.871030  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:04.810292  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44195.6us
I20260709 12:25:04.481328  7564 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.857717  7532 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:04.857956  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:04.834985  7409 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:04.805558  7368 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.866112  7422 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:04.865478  7469 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:04.807590  7565 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:04.898048  6274 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:04.877662  7451 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:04.947631  7414 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:04.948896  7549 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:04.950296  7562 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.771s	user 0.009s	sys 0.003s
I20260709 12:25:04.920552  7469 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:04.952001  7469 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.810509  7398 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:04.924300  7565 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:04.957268  7398 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:04.959941  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:04.961716  7564 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:04.965543  7386 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.820s	user 0.010s	sys 0.000s
I20260709 12:25:04.804857  7559 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:04.968001  7559 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:04.921819  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1424) took 1355 ms. Trace:
I20260709 12:25:04.877983  7429 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:04.925367  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:04.894668  7544 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:04.886370  7453 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:04.935990  7452 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:04.937484  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:04.938483  7368 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:04.942232  7531 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.278s	user 0.011s	sys 0.002s
I20260709 12:25:04.942121  7540 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:04.942893  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:04.983153  5703 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:25:04.943078  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:04.983680  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:04.943876  7574 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.181s	user 0.000s	sys 0.000s
I20260709 12:25:04.944685  7412 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:25:04.945469  7568 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:25:04.924081  7389 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:04.975452  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 50210.3us
I20260709 12:25:04.974407  5553 rpcz_store.cc:276] 0709 12:25:03.566554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.950125 (+383571us) service_pool.cc:224] Handling call
0709 12:25:03.950550 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.950820 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:25:04.440535 (+489715us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.921204 (+480669us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:04.921324 (+   120us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:04.921446 (+   122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.921553 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.921743 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":498425,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":80443,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":110,"thread_start_us":918791,"threads_started":3,"wal-append.queue_time_us":498961}
I20260709 12:25:04.935268  7560 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:25:04.984081  5704 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:04.993162  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:04.986951  7389 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:04.985780  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:04.982890  7423 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.192s	user 0.012s	sys 0.001s
I20260709 12:25:04.979707  7559 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:05.003993  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:05.005134  5704 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:25:04.989790  7549 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:25:05.023926  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.965598  5707 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:04.996187  7562 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:25:05.031839  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46984.1us
W20260709 12:25:05.032143  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 286413us
I20260709 12:25:04.989993  7470 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260709 12:25:05.032907  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:04.987534  7531 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.002987  7532 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260709 12:25:05.044164  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.004997  7423 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.031595  7562 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:05.047835  5553 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:05.056571  7562 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:05.056747  7452 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:05.057323  7452 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:05.057807  7452 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:05.035601  7579 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260709 12:25:05.034161  7393 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260709 12:25:05.061260  7429 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.061812  7429 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.065006  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.065497  7429 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:05.043310  7422 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:05.066375  7422 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.067466  7470 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:05.039710  7409 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.069229  7409 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.070313  7544 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:05.070809  7544 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:05.071178  7409 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:05.051751  5555 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:05.043663  7412 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:05.035583  7368 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.072013  7412 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.072713  7368 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.026621  7451 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.074970  7368 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:05.075788  7451 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:05.045060  7590 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260709 12:25:05.048425  7475 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:05.076479  5635 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:25:05.032212  7386 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 12:25:05.074105  7412 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:05.050259  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.035663  7549 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:05.037842  7276 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260709 12:25:05.068271  7470 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.076949  7475 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:05.053081  7474 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.098301  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:05.098912  6277 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:05.078171  5557 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 3: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:05.066629  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:05.051195  7531 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:05.083374  5554 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:05.089807  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:05.111941  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:05.091984  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.113627  5553 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:05.093204  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:05.114890  5559 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:25:05.095422  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.100632  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:05.116937  6278 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 4: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:05.138983  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:05.140195  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:05.147001  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:05.148729  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 115919us
I20260709 12:25:05.098345  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.149874  5561 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:05.111585  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:05.151120  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:05.151305  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:05.144948  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 151899us
I20260709 12:25:05.151820  5553 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:05.106076  7475 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:05.154115  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.111586  7423 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:05.155018  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.105901  7412 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.071762  7544 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:05.126634  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:05.164955  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 365508us
I20260709 12:25:05.174064  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:05.166846  5559 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:05.153940  6277 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:05.149820  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.155098  7329 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.097044  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:05.155303  7422 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:05.184114  7329 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.188457  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:05.171506  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:05.193495  5636 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:25:05.223667  5636 pending_rounds.cc:85] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 1
I20260709 12:25:05.223750  6275 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:05.224040  5636 pending_rounds.cc:107] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:25:05.224311  5636 raft_consensus.cc:2889] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:05.224972  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:05.225497  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:05.225741  6275 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 2: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:05.226647  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:05.228024  6275 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:05.176908  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1609) took 1017 ms. Trace:
I20260709 12:25:05.179566  7588 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.178471  7451 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.199674  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:05.180667  7386 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:05.201542  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:05.202680  5704 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:25:05.194396  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:05.177505  7540 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260709 12:25:05.249270  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:05.172307  7412 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.192955  7579 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.195057  7474 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:05.206975  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:05.260768  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:05.261641  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:05.262549  5706 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:05.262781  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:05.203109  7549 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.264532  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:05.265448  5633 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:05.266640  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:05.214135  7558 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:05.267417  7558 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:05.268445  7558 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.269997  7558 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:05.217895  7575 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.219678  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:05.271505  6271 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:05.273006  7575 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:05.274150  7579 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.275807  5631 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:05.255970  7386 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:05.277359  7386 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.277794  7568 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:05.278339  7568 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.278924  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:05.279729  5631 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:05.247673  7588 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.281294  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:05.282071  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1618) took 1021 ms. Trace:
I20260709 12:25:05.282380  7588 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:05.282570  5705 rpcz_store.cc:276] 0709 12:25:04.261078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.267138 (+  6060us) service_pool.cc:224] Handling call
0709 12:25:05.281050 (+1013912us) spinlock_profiling.cc:243] Waited 1.09 ms on lock 0x7b5800053198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:25:05.281291 (+   241us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:05.281645 (+   354us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:05.281684 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:05.281737 (+    53us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:05.281787 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.281864 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:05.282003 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2506496}
I20260709 12:25:05.283452  5706 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:05.283993  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1617) took 1023 ms. Trace:
I20260709 12:25:05.284898  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:05.222831  7565 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.285889  5710 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:25:05.286515  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1616) took 1127 ms. Trace:
I20260709 12:25:05.287202  7565 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:05.236594  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.219237  7470 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:05.288910  7470 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:05.289280  7470 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:05.289628  7470 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:05.289839  6271 raft_consensus.cc:2955] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:05.212385  7410 raft_consensus.cc:2955] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:05.228607  7562 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.293969  7562 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:05.256665  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 489) took 1147 ms. Trace:
I20260709 12:25:05.294595  6276 rpcz_store.cc:276] 0709 12:25:04.109314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.110936 (+  1622us) service_pool.cc:224] Handling call
0709 12:25:05.256481 (+1145545us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x7b54005910d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:05.256602 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":74927360}
I20260709 12:25:05.240166  7423 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:05.295802  7423 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:05.182186  7559 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:05.296747  7423 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.251060  7451 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.297024  7559 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.298375  6272 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:05.299495  7451 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:05.296748  5630 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:05.301028  7579 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:05.284600  7412 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:05.301589  7329 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:05.280593  7568 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:05.262945  7434 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:05.284658  5706 rpcz_store.cc:276] 0709 12:25:04.260791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.266767 (+  5976us) service_pool.cc:224] Handling call
0709 12:25:05.261331 (+994564us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b5800053198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:05.283936 (+ 22605us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3016576}
I20260709 12:25:05.326167  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:05.326467  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51141.8us
I20260709 12:25:05.327216  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:05.327517  5706 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:05.299602  7452 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.300482  7429 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.241576  7397 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:05.300720  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:05.286885  5710 rpcz_store.cc:276] 0709 12:25:04.159446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.166806 (+  7360us) service_pool.cc:224] Handling call
0709 12:25:05.284714 (+1117908us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b5800053198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:05.286446 (+  1732us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37373824}
I20260709 12:25:05.221454  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:05.258801  7328 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.309192  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:05.326725  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:05.334779  6276 raft_consensus.cc:2368] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5.
I20260709 12:25:05.331169  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:05.335170  5705 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:05.336292  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:05.237421  5701 rpcz_store.cc:276] 0709 12:25:04.156087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.163060 (+  6973us) service_pool.cc:224] Handling call
0709 12:25:05.066448 (+903388us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b5800053198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:05.173621 (+107173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27036928}
I20260709 12:25:05.330231  7412 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:05.349169  7412 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.349560  7412 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:05.352952  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:05.348567  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:05.357266  5707 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
W20260709 12:25:05.361877  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 211206us
I20260709 12:25:05.364925  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:05.341861  7433 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:05.342563  7397 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:05.358669  7328 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:05.334748  7373 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
W20260709 12:25:05.392894  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61825.8us
I20260709 12:25:05.331051  7568 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260709 12:25:05.414727  6277 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:05.383908  7433 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.339754  7429 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.343626  7453 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.394449  7462 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:05.374990  7470 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.440999  7462 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.430984  7433 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:05.556397  7329 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:05.557662  7329 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.328720  5627 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:05.559341  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:05.561430  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:05.562282  7470 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:05.563155  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:05.562788  7397 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.267441  5633 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 1.
I20260709 12:25:05.562148  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:05.565778  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:05.566090  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:05.234550  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 137603us
I20260709 12:25:05.566999  5631 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:05.567764  5705 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
W20260709 12:25:05.501932  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.508617  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 130132us
I20260709 12:25:05.570384  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.571344  7453 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:05.571833  5552 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:05.419803  7562 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260709 12:25:05.584664  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:05.585911  5559 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:05.447175  7462 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:05.542361  7583 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:05.591207  5560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1411) took 2077 ms (client timeout 1821 ms). Trace:
W20260709 12:25:05.593703  7617 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:25:05.547885  7610 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:05.595065  7610 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.596642  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 281) took 1444 ms. Trace:
I20260709 12:25:05.597164  5555 rpcz_store.cc:276] 0709 12:25:04.151633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.260269 (+108636us) service_pool.cc:224] Handling call
0709 12:25:04.270550 (+ 10281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:05.071841 (+801291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.595991 (+524150us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b5400082854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:05.596380 (+   389us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1796059392}
I20260709 12:25:05.599205  7409 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:05.601447  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.602257  7610 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:05.548987  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 251410us
I20260709 12:25:05.605846  7429 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:05.607004  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.550364  5556 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.563s	user 0.004s	sys 0.009s
I20260709 12:25:05.608012  5556 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:05.609012  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:05.609510  5556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1433) took 1976 ms (client timeout 1508 ms). Trace:
I20260709 12:25:05.610100  5628 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:25:05.609989  5556 rpcz_store.cc:269] 0709 12:25:03.632782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.636610 (+  3828us) service_pool.cc:224] Handling call
0709 12:25:05.609259 (+1972649us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54001be754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:05.609428 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3006013696}
I20260709 12:25:05.542346  7414 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:05.611738  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 485) took 1516 ms. Trace:
I20260709 12:25:05.612314  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:05.614132  7393 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:05.614535  7393 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:05.414048  7170 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:05.617401  7409 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.619774  7457 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:05.414064  7475 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:05.500381  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.573383  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:05.549129  7451 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:05.582475  7582 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:25:05.590754  7583 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.618274  7170 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.573050  7549 raft_consensus.cc:2764] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:25:05.581874  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.613380  5558 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:05.612155  6280 rpcz_store.cc:276] 0709 12:25:04.094721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.095547 (+   826us) service_pool.cc:224] Handling call
0709 12:25:04.095894 (+   347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:04.404732 (+308838us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.611128 (+1206396us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54002a8354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:05.611514 (+   386us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":708062080}
W20260709 12:25:05.591832  5560 rpcz_store.cc:269] 0709 12:25:03.514014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.538399 (+ 24385us) service_pool.cc:224] Handling call
0709 12:25:05.590582 (+2052183us) spinlock_profiling.cc:243] Waited 924 ms on lock 0x7b5400026cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:05.590935 (+   353us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2124759808}
I20260709 12:25:05.550717  7559 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
I20260709 12:25:05.637362  7451 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.638531  7475 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:05.639698  5708 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:05.631830  7170 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:05.611740  7414 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.640118  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:05.643268  7414 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:05.644771  5553 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:05.644886  5561 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:05.645807  7229 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:05.646721  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:05.647570  7361 raft_consensus.cc:2764] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:25:05.649248  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:05.646168  7558 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:05.640146  7451 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:05.655397  7451 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.655300  7558 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.656396  7451 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.658422  7558 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:05.662176  7583 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:05.663479  7559 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.664515  7559 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.671175  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:05.673537  5709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1614) took 1514 ms (client timeout 1952 ms). Trace:
W20260709 12:25:05.674124  5709 rpcz_store.cc:269] 0709 12:25:04.158692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.159032 (+   340us) service_pool.cc:224] Handling call
0709 12:25:05.673040 (+1514008us) spinlock_profiling.cc:243] Waited 589 ms on lock 0x7b54003e6f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:05.673335 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1355056896}
I20260709 12:25:05.675421  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.676020  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:05.676313  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.677237  5554 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:05.675469  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:05.678318  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:05.676118  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:05.679210  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:05.679905  5706 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:05.646853  7229 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.681875  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:05.682670  7229 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:05.682716  5703 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:05.683593  7479 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:05.684607  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.684859  7611 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:05.685343  7611 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.685889  7611 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:05.679158  5554 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:05.680069  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.632025  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:05.615993  7393 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.686236  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.701769  7583 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:05.680051  7469 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:05.714385  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:05.653491  6273 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:05.651746  7329 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:05.673557  5552 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:05.732544  5560 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:25:05.734159  5559 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:25:05.745743  7559 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:05.752465  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 317675us
I20260709 12:25:05.755391  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.756494  5559 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:05.755777  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:05.755722  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:05.757961  5560 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:25:05.732229  5629 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:05.742280  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:05.772452  5561 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:25:05.730340  7409 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:05.760332  5627 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:05.770650  5601 leader_election.cc:400] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5.
I20260709 12:25:05.780501  5631 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:05.783989  7410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:38456 (request call id 78) took 13331 ms. Trace:
I20260709 12:25:05.783221  5601 leader_election.cc:403] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 12:25:05.787546  7608 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:05.788041  7608 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.788462  7608 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:05.796928  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:05.797842  5634 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:05.784613  7410 rpcz_store.cc:276] 0709 12:24:52.400096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.400263 (+   167us) service_pool.cc:224] Handling call
0709 12:24:52.578200 (+177937us) log.cc:844] Serialized 480 byte log entry
0709 12:24:52.813914 (+235714us) spinlock_profiling.cc:243] Waited 2.37 ms on lock 0x7b540034e2a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5eb142 00007f4ffa5cbeaa 00007f4ffa5d2d53 00007f4ffa5d2cd9 00007f4ffa5d2c69 00007f4ffa5d2c31 00007f4ffa5d1dfd 0000000000388634 0000000000388479 00007f4ff7585475
0709 12:25:05.721492 (+12907578us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24817,"raft.queue_time_us":21473,"raft.run_cpu_time_us":3109,"raft.run_wall_time_us":65353,"spinlock_wait_cycles":5450112,"thread_start_us":184474,"threads_started":1,"wal-append.queue_time_us":230390,"wal-append.run_cpu_time_us":1635,"wal-append.run_wall_time_us":1014049}
I20260709 12:25:05.756155  7610 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:05.784052  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:05.784065  5556 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 4: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:05.838881  7559 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:05.841989  5706 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
W20260709 12:25:05.836899  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 143500us
I20260709 12:25:05.853000  5557 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 12:25:05.855620  7619 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:25:05.857684  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:05.858822  5633 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:05.860632  5709 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
W20260709 12:25:05.861196  7621 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
I20260709 12:25:05.843431  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:05.862537  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 60706.5us
I20260709 12:25:05.864073  7409 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:05.869037  5553 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:05.869884  7409 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.875646  7409 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:05.876212  7393 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:25:05.857746  7618 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:25:05.886214  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:05.836581  7583 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:05.908704  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 154596us
W20260709 12:25:05.908989  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 63242.5us
W20260709 12:25:05.908752  5445 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 2acd6f5df0204b2295547e021cd8cb9f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:25:05.932535  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:05.933352  7412 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.583s	user 0.007s	sys 0.011s
I20260709 12:25:05.933913  7412 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:05.948827  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 189091us
I20260709 12:25:05.950518  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:05.952770  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:05.953689  5628 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:05.956615  7329 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.956739  7433 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.957512  7329 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:05.959507  7433 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:05.961329  7328 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:05.898422  7170 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.960186  7329 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:05.981140  5554 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:05.962029  7328 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.990626  5553 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:25:05.990247  7462 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.991319  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:05.992596  7595 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:05.992859  7462 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:05.993830  7462 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:05.994349  7595 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:05.995863  7595 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:06.001492  7578 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:06.001703  7592 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:06.002108  7578 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:06.002916  7569 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.976042  7229 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.996738  7595 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:06.003472  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:06.012430  7612 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:06.013656  7612 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:06.014043  7612 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:06.015305  7612 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:05.994287  7462 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:06.020750  5443 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet f0f7b2d2a7e34d629491c96dbfbc7fdc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:25:06.021108  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.017483  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1441) took 2382 ms. Trace:
I20260709 12:25:05.963327  7451 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:06.018790  7549 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:06.029428  7549 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.031800  7549 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:06.032580  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.033516  5628 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
W20260709 12:25:06.035795  7620 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
I20260709 12:25:06.023368  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:06.036607  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:06.037269  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:06.036710  7583 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:06.038635  7412 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:06.035208  7595 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:06.039760  7595 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:06.024529  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:06.040697  7434 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.710s	user 0.014s	sys 0.005s
I20260709 12:25:06.040812  5560 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:06.041728  7434 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.042392  7434 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.004056  7229 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:06.044785  7592 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:06.045429  7592 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:06.046463  7592 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:06.047008  7229 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:06.022832  5558 rpcz_store.cc:276] 0709 12:25:03.634759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.126539 (+491780us) service_pool.cc:224] Handling call
0709 12:25:05.612690 (+1486151us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:05.629153 (+ 16463us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.017126 (+387973us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54001be750. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:06.017397 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3416648448}
I20260709 12:25:06.049479  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:06.003520  7578 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:06.053107  5675 leader_election.cc:336] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.554s (SENT)
I20260709 12:25:06.053695  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:06.049263  7608 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.058538  7608 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:06.056284  7578 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.063419  7170 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:06.064798  7469 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:06.065418  7469 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.065807  7469 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.066937  7328 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:06.020982  7462 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:05.945732  6274 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
W20260709 12:25:06.110234  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.095829  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:05.987712  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:06.094897  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:06.112640  5560 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:06.100728  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.097424  7469 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.103731  5557 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:06.128567  5635 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:06.128733  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:06.125639  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:06.148761  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:06.151678  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:06.148838  6280 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:25:06.103816  7607 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:06.160231  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:06.161499  5553 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:06.040944  7544 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:06.155196  7549 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:06.156083  7612 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.181859  5554 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:06.182461  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:06.183822  5560 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.118006  7409 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:06.189960  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 406053us
I20260709 12:25:06.155244  7607 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:06.106892  7386 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.070s
I20260709 12:25:06.003681  7569 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.201109  5553 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:25:06.168324  7595 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:06.175554  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:06.213294  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 118486us
I20260709 12:25:06.213737  5552 raft_consensus.cc:2368] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5.
I20260709 12:25:06.213546  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:06.206962  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1610) took 2050 ms. Trace:
I20260709 12:25:06.179831  7229 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:06.215178  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 162108us
I20260709 12:25:06.215494  7229 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:06.217463  7386 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:06.217898  7229 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:06.218086  7386 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:06.219197  7386 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:06.210649  7595 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.220757  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:06.222066  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.223670  5629 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:06.224839  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:06.225447  5632 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.226509  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.228690  7607 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.230043  7607 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.230369  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:06.230979  5706 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:06.233140  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:06.207762  7397 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:06.238662  7434 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:06.240114  7469 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:06.240525  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:06.240924  7588 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:06.241396  6274 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:06.241176  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:06.198558  7559 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5.
I20260709 12:25:06.242231  5704 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:06.242298  6279 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:06.194434  7451 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:06.220865  7595 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.236649  7634 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:06.233248  7549 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.216001  5557 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 4: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:06.215324  5636 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:06.208532  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:06.214736  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:06.214430  5630 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:25:06.216107  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.255563  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:06.256246  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.230053  5709 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:06.246112  5635 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:06.247150  7409 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:06.182054  7611 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.259681  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:06.260056  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:06.264242  5706 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
W20260709 12:25:06.267336  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:06.266453  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 155564us
I20260709 12:25:06.270881  5705 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 2.
I20260709 12:25:06.248770  7397 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:06.269412  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:06.258792  7386 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:06.273815  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:06.214789  5702 rpcz_store.cc:276] 0709 12:25:04.156458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.156599 (+   141us) service_pool.cc:224] Handling call
0709 12:25:04.157027 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:04.164904 (+  7877us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:04.164937 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:04.169194 (+  4257us) log.cc:844] Serialized 26 byte log entry
0709 12:25:05.643703 (+1474509us) spinlock_profiling.cc:243] Waited 5.37 ms on lock 0x7b1800169a80. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:25:05.902703 (+259000us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:06.114919 (+212216us) log.cc:844] Serialized 14 byte log entry
0709 12:25:06.193960 (+ 79041us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.200607 (+  6647us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:06.200671 (+    64us) raft_consensus.cc:1685] finished
0709 12:25:06.206594 (+  5923us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.206758 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100717,"prepare.queue_time_us":135182,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":87198,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":6432,"spinlock_wait_cycles":12345216,"thread_start_us":118363,"threads_started":1,"wal-append.queue_time_us":1847237}
I20260709 12:25:06.260104  7451 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:06.277336  7457 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:06.301218  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:06.289680  5703 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
W20260709 12:25:06.298079  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 83436.9us
I20260709 12:25:06.311800  7614 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:06.312793  7614 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:06.315316  7614 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:06.315721  7600 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.320148  7540 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260709 12:25:06.301517  7397 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.323792  7397 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:06.328504  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:06.329814  7549 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:06.330016  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 51651.8us
I20260709 12:25:06.330610  7549 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:06.331055  7549 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:06.331498  7549 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.311950  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:06.284487  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:06.332805  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:06.366153  5702 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.366564  5634 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:06.337757  5560 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:06.371081  5560 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:06.367503  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:06.316423  7600 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:06.377311  7451 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:06.377509  7639 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.379464  7611 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:06.368577  7469 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:25:06.383960  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:06.369423  7632 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:25:06.347442  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 336287us
W20260709 12:25:06.369599  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 62286.9us
I20260709 12:25:06.387346  5557 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:25:06.345598  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 76341.4us
W20260709 12:25:06.390151  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 57794.8us
W20260709 12:25:06.392174  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.392820  5632 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:06.275785  7634 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.406194  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:06.414011  5632 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:06.423009  5630 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:06.294124  7462 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.431684  5557 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:06.388944  7397 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260709 12:25:06.434645  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:06.402264  5552 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:06.440035  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.440424  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.448315  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.449740  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.450423  5632 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:25:06.451610  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.452518  5632 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:06.455762  5630 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:06.422791  7632 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:06.403430  7578 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260709 12:25:06.462991  6277 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:06.469969  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:25:06.463514  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 130084us
I20260709 12:25:06.479223  7457 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.479866  7457 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:06.433528  7540 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.042s
I20260709 12:25:06.484941  7592 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:25:06.484992  7457 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:06.487576  7600 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:06.403235  7479 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20260709 12:25:06.488518  7643 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:06.417027  7639 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:06.491838  7639 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:06.492581  7578 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:06.502007  7578 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:06.503151  7578 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:06.499128  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:06.505653  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:06.506923  7588 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:06.507068  7457 raft_consensus.cc:2764] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:25:06.507468  7588 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:06.508492  7588 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:06.513973  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 78194.4us
I20260709 12:25:06.516748  5635 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
W20260709 12:25:06.516973  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 198713us
W20260709 12:25:06.519970  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 84999.2us
I20260709 12:25:06.521425  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:06.521752  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.522689  5631 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:06.526801  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.527654  5632 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:06.466495  7611 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:06.510365  7578 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:06.544675  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.434651  7614 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.097s
I20260709 12:25:06.548026  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:06.548786  6273 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:06.549099  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.550305  5701 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:06.552778  7569 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:06.553031  6280 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:06.531767  7639 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.515846  7549 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.652307  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:06.652566  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:06.655870  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 87291.2us
I20260709 12:25:06.670219  6280 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:06.571406  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.670373  5630 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:06.666684  5702 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
W20260709 12:25:06.671059  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:06.671552  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:06.655391  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 140975us
I20260709 12:25:06.653363  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:06.601543  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.541589  7611 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.676381  7611 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:06.572947  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:06.687845  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:06.689316  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:06.689669  7654 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:06.689045  7639 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.691031  5630 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:06.692269  7639 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:06.689227  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:06.693392  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:06.692225  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:06.694447  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 142075us
I20260709 12:25:06.637964  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:06.706399  5559 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.010s	sys 0.037s
W20260709 12:25:06.706789  5559 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.011s	sys 0.038s
W20260709 12:25:06.709841  5636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1406) took 1503 ms (client timeout 1762 ms). Trace:
I20260709 12:25:06.693430  6276 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:06.702456  7654 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:06.702819  7452 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:06.655514  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:06.705739  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:06.715586  5636 rpcz_store.cc:269] 0709 12:25:05.207782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.225284 (+ 17502us) service_pool.cc:224] Handling call
0709 12:25:06.709437 (+1484153us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b5400134f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:06.709690 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1910576512}
I20260709 12:25:06.729177  5632 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:06.724906  5630 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:06.730111  7655 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:06.733263  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:06.740006  7655 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:06.693415  7634 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:06.723256  7361 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:06.740990  7452 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:06.740839  7361 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:06.741607  7452 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:06.741124  7655 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:06.737619  7629 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.509s	user 0.000s	sys 0.002s
I20260709 12:25:06.752866  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:06.729554  7611 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:06.742071  7452 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.760026  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:06.572139  7632 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.751174  7462 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:06.766849  7373 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:06.767575  7634 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:06.767433  7373 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.767997  7634 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.768478  7634 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.769681  7373 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:06.765511  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50898.5us
I20260709 12:25:06.774742  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:06.760974  7611 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.806221  7578 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:06.806741  7578 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:06.807099  7578 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:06.802824  7611 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.824712  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:06.826687  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:06.811306  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:06.809965  5627 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:06.834944  7373 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:06.828696  7229 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:06.838202  7636 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.465s	user 0.001s	sys 0.001s
I20260709 12:25:06.842053  7577 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:06.842314  7361 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:06.842427  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:06.843422  7373 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:06.843559  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:06.843853  7549 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:06.829362  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:06.845178  7595 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.581s	user 0.009s	sys 0.002s
I20260709 12:25:06.845762  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:06.845664  7595 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.848585  7595 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:06.849908  7434 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:06.852681  7643 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:06.853493  7643 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:06.856305  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 421739us
I20260709 12:25:06.809950  6236 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:06.852137  6279 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 502) took 1751 ms (client timeout 2232 ms). Trace:
I20260709 12:25:06.860082  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 304) took 1068 ms. Trace:
W20260709 12:25:06.858237  6279 rpcz_store.cc:269] 0709 12:25:05.100759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.146756 (+ 45997us) service_pool.cc:224] Handling call
0709 12:25:06.851886 (+1705130us) spinlock_profiling.cc:243] Waited 903 ms on lock 0x7b54002e3954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:06.852037 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2076969728}
W20260709 12:25:06.862562  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52713.9us
I20260709 12:25:06.860534  5634 rpcz_store.cc:276] 0709 12:25:05.791905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.821866 (+ 29961us) service_pool.cc:224] Handling call
0709 12:25:06.859913 (+1038047us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b5400134554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:06.860020 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":810994048}
I20260709 12:25:06.863550  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:06.856127  7643 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:06.876749  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:06.878033  5561 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.876852  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:06.876487  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:06.880311  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:06.878124  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:06.881986  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:06.882628  6276 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.882834  7610 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:06.883208  7610 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:06.883572  7610 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:06.884653  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:06.885272  5556 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:06.885707  7569 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.878836  5555 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:06.879544  6277 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:06.890143  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:06.891067  7608 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:06.891443  7614 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:06.892129  7614 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:06.892802  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:06.893509  5701 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:06.894605  7614 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:06.876910  7373 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:06.891508  7608 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:06.898904  7608 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:06.899080  7600 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:25:06.900964  7600 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:06.901655  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:06.901790  7600 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:06.902438  5705 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:06.902316  7600 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:06.903462  7600 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:06.898283  7373 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.904603  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:06.904831  5702 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:25:06.905342  5557 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.907218  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:06.907506  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:06.907982  6274 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:06.908613  7559 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:06.909322  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:06.909615  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:06.910332  6273 raft_consensus.cc:2368] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4.
I20260709 12:25:06.910735  7559 raft_consensus.cc:2764] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260709 12:25:06.910995  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 488960us
I20260709 12:25:06.911757  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:06.912819  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:06.913194  5704 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:06.913412  5554 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.914579  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:06.915150  5702 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:25:06.915444  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:06.916524  7451 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:06.916823  6279 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:06.916962  7451 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:06.917464  7451 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:06.909318  7654 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260709 12:25:06.904683  7373 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:06.920533  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:06.923022  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49640.7us
W20260709 12:25:06.926152  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50191.7us
I20260709 12:25:06.929625  5702 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:06.962428  6276 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:06.964107  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:06.965040  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 89494.3us
I20260709 12:25:06.942503  5559 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:06.938324  5557 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:06.871207  7583 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.885190  7592 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:06.878710  7329 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:06.876288  7540 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:25:06.960124  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 79894us
I20260709 12:25:06.962215  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:06.956530  7479 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:06.967133  5561 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:06.962215  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:25:06.968636  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56964.2us
I20260709 12:25:06.889900  5457 catalog_manager.cc:5697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:06.963666  7453 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:06.975446  5601 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:06.991680  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:06.994453  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:06.971014  7595 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:07.087616  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:07.090179  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:07.090938  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 111530us
I20260709 12:25:07.102233  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:06.996052  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:07.014750  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:07.101200  5702 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 6: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:07.093470  5554 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:06.974390  7361 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.967319  7612 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:07.109302  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 113323us
I20260709 12:25:06.995582  7328 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:25:07.109920  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 95252.4us
W20260709 12:25:07.118069  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.995009  7600 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:06.978626  7329 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:06.981498  7540 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:06.976917  7643 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.156616  7643 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.159756  7643 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:06.977003  7592 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:06.983210  7453 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.131182  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:07.142217  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:06.974038  7583 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.972662  7608 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:06.997182  7479 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:07.098927  7429 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:07.087060  7559 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:07.144109  7328 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.140899  7612 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:07.098423  7558 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:25:07.148636  7361 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.152460  7329 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:07.152931  7540 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:07.103374  7469 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:07.131796  7423 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:06.995272  7639 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.145s
I20260709 12:25:07.168047  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:07.174182  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.174887  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.177762  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:07.222959  5557 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:07.131579  7462 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:07.175937  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.225006  5706 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:25:07.188951  5710 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:07.225131  5705 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:25:07.175580  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.175938  5458 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 273) took 5087 ms. Trace:
I20260709 12:25:07.167824  5457 catalog_manager.cc:5697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 4 to 5, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 5 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:07.229403  5708 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:25:07.170588  7583 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:07.181938  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:07.234460  6276 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:07.160300  7229 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:07.166743  7592 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:07.175846  7453 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:07.168579  7389 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:07.173993  7632 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:07.169351  7577 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:07.192915  7423 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.180490  7384 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:07.173861  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.178452  7328 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:25:07.251065  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 273136us
I20260709 12:25:07.224704  7640 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:07.270980  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:07.178139  7429 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:07.170773  5556 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:07.172367  7608 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:07.184201  7361 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:07.189076  7549 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:07.272260  7632 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:07.273474  6279 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:07.181344  7612 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:07.274281  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:07.292234  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:07.275071  5558 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:07.180658  7569 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:07.192293  7540 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.173276  7479 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:07.303428  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 206887us
W20260709 12:25:07.287469  5675 connection.cc:256] RPC call timeout handler was delayed by -1.1256662870000582s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22000000000000003s for a response
W20260709 12:25:07.282617  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 288577us
I20260709 12:25:07.184886  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.197388  7329 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.242573  7462 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:07.270311  7423 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:07.288417  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:07.288754  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:07.289415  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:07.308741  5601 leader_election.cc:400] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4.
I20260709 12:25:07.317283  5601 leader_election.cc:403] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:25:07.317651  5634 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:07.287048  7577 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:07.291056  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:07.292104  7614 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:07.292640  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:07.179522  7558 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:07.284610  5457 catalog_manager.cc:5697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:07.292771  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:07.293331  7361 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 12:25:07.297363  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.298430  7632 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:07.286072  7549 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:07.299386  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.299960  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.326326  5705 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:07.300978  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:07.276757  5703 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:07.334259  5557 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:07.335919  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:07.303849  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 321006us
I20260709 12:25:07.304384  5630 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:07.337316  7434 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:07.250350  7578 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:07.338914  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47928.2us
I20260709 12:25:07.310081  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:07.311928  5707 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:07.314060  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.293522  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:07.315928  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:07.306107  7634 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.537s	user 0.011s	sys 0.000s
I20260709 12:25:07.338091  5457 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 2 to 3, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 3 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:07.163421  7451 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.326916  6278 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:07.321838  7558 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:07.231482  5458 rpcz_store.cc:276] 0709 12:25:02.088288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.089805 (+  1517us) service_pool.cc:224] Handling call
0709 12:25:02.635134 (+545329us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5dc090 00007f4ffa67af3a 00007f5000430e9c 00007f500052f04e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc 00007f4ffe9ef392 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:07.175677 (+4540543us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:03.527467 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:03.528085 (+   618us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:03.528126 (+    41us) write_op.cc:435] Acquired schema lock
0709 12:25:03.528145 (+    19us) tablet.cc:662] Decoding operations
0709 12:25:03.533245 (+  5100us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:03.533303 (+    58us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:03.533337 (+    34us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:25:03.536290 (+  2953us) tablet.cc:701] Row locks acquired
0709 12:25:03.536313 (+    23us) write_op.cc:260] PREPARE: finished
0709 12:25:03.536481 (+   168us) write_op.cc:270] Start()
0709 12:25:03.536598 (+   117us) write_op.cc:276] Timestamp: P: 1783599903536434 usec, L: 0
0709 12:25:03.536633 (+    35us) op_driver.cc:348] REPLICATION: starting
0709 12:25:03.540013 (+  3380us) log.cc:844] Serialized 2862 byte log entry
0709 12:25:03.818134 (+278121us) op_driver.cc:464] REPLICATION: finished
0709 12:25:04.037445 (+219311us) write_op.cc:301] APPLY: starting
0709 12:25:04.037592 (+   147us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:04.037794 (+   202us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:04.043352 (+  5558us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:04.154551 (+111199us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:04.169107 (+ 14556us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0709 12:25:04.169173 (+    66us) write_op.cc:312] APPLY: finished
0709 12:25:04.169458 (+   285us) log.cc:844] Serialized 59 byte log entry
0709 12:25:04.169676 (+   218us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:04.169800 (+   124us) write_op.cc:454] Released schema lock
0709 12:25:04.169847 (+    47us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:04.170241 (+   394us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:04.170268 (+    27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":350950,"reported_tablets":25,"spinlock_wait_cycles":63074560,"child_traces":[["op",{"apply.queue_time_us":219035,"apply.run_cpu_time_us":4180,"apply.run_wall_time_us":132753,"num_ops":7,"prepare.queue_time_us":237233,"prepare.run_cpu_time_us":4172,"prepare.run_wall_time_us":12875,"replication_time_us":281295,"thread_start_us":404527,"threads_started":2}]]}
I20260709 12:25:07.357852  5706 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:25:07.319305  7608 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.028s
I20260709 12:25:07.314325  7462 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:07.321979  7610 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:07.361769  7610 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:07.362610  7610 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:07.363013  7610 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:07.319128  7229 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260709 12:25:07.357147  7434 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:07.365114  7451 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:07.355650  5457 catalog_manager.cc:5697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 2, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:07.365633  7434 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:07.367120  5457 catalog_manager.cc:5697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 2 to 3, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 3 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:25:07.368428  7429 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:07.369709  7578 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:07.369136  5457 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 4 to 6, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 6 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:07.357416  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.355597  7634 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:07.367995  5705 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:25:07.373764  7634 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:07.374408  7583 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:25:07.359378  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 257263us
W20260709 12:25:07.378441  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:07.378911  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.328056  7393 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4.
I20260709 12:25:07.327426  7549 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:07.387435  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.388183  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:07.345981  7569 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:07.396945  5561 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:07.326957  7595 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:07.413024  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:07.415738  5557 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:07.424726  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:07.425925  5559 raft_consensus.cc:2410] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:25:07.395903  7389 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:25:07.435096  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55568.7us
I20260709 12:25:07.400017  7569 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:07.401043  7384 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:07.435829  7614 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.434290  6273 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 4: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:07.448093  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:07.448962  5559 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
W20260709 12:25:07.451640  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 143780us
W20260709 12:25:07.453737  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 112275us
I20260709 12:25:07.454386  5560 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:25:07.455075  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:07.442960  7569 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:07.467692  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 129792us
I20260709 12:25:07.439340  5554 raft_consensus.cc:2955] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:07.413007  7583 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.039s
I20260709 12:25:07.446442  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:07.456830  7451 raft_consensus.cc:2764] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260709 12:25:07.480993  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:07.442718  7614 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.481817  5559 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:07.484175  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:07.478504  7386 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 12:25:07.477916  7384 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:07.487000  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 109971us
I20260709 12:25:07.487790  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:07.489375  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:07.436937  7373 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:07.497210  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:07.505528  5705 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:25:07.508718  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:07.509742  6277 raft_consensus.cc:2410] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:25:07.512329  7614 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:07.428601  7610 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:07.517951  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:07.482340  7668 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:07.440917  7558 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:07.531519  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:07.535254  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47574.5us
I20260709 12:25:07.540117  5602 leader_election.cc:400] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5.
I20260709 12:25:07.522454  7389 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:07.540959  6277 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 12:25:07.535024  7665 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
W20260709 12:25:07.551617  7452 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.775s	user 0.005s	sys 0.006s
I20260709 12:25:07.552175  7452 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.554821  7452 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:07.534909  7677 consensus_queue.cc:1175] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" responder_term: 5 status { last_received { term: 4 index: 6 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:07.515955  7595 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:07.521530  7668 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:07.481662  7453 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:07.520702  7329 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:07.524140  7672 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:07.458222  7577 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.568555  6272 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:25:07.536725  7611 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.716s	user 0.008s	sys 0.003s
I20260709 12:25:07.573836  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:07.539413  7610 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:07.589676  5636 raft_consensus.cc:2410] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:25:07.610400  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.607969  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:07.871263  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:25:07.883692  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 307047us
I20260709 12:25:07.882097  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:07.884981  5631 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:07.611619  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:07.791062  7655 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:07.886600  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:07.886727  7655 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:07.744411  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 214514us
I20260709 12:25:07.610571  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.708173  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:07.891302  5702 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:07.883133  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.611119  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.574640  7672 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.611940  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:07.895431  5706 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:07.898187  5708 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:07.935397  5635 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:07.814145  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:07.829341  7583 consensus_queue.cc:1243] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:25:07.868968  7684 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:07.864068  7588 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.857533  7361 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.675527  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:07.709101  7479 raft_consensus.cc:2764] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260709 12:25:07.830893  7683 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.942454  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.599472  7328 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.953238  5702 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:07.701399  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:07.923645  7423 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.599s	user 0.008s	sys 0.003s
W20260709 12:25:07.949007  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 55396us
I20260709 12:25:07.947440  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:07.922081  7595 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.955741  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:07.922380  7672 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:07.952834  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:07.941274  7600 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.608218  7611 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:07.814292  7569 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.938196  7457 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:07.960975  5703 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:07.962555  5559 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:07.963121  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:07.913751  7612 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 12:25:07.961220  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:07.947073  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:07.950904  5627 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:07.973599  5555 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:07.995347  5703 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:07.960784  5707 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:25:07.944478  7687 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:25:08.013662  7328 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:08.018530  7549 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.618s	user 0.009s	sys 0.001s
I20260709 12:25:08.019088  7549 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:08.021377  7689 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:25:08.021970  7569 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:08.022667  6271 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 4: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:25:07.974928  7701 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:25:08.027611  5702 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:07.957634  7684 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.975265  7373 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:07.958746  7361 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.961429  7588 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.965277  7423 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.966877  7595 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.970808  7600 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:07.963929  7683 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:08.030819  5552 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:07.993060  7457 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:08.038722  5556 raft_consensus.cc:2368] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4.
I20260709 12:25:08.072479  5553 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:25:07.972872  7685 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.084s
W20260709 12:25:08.067718  5601 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.731s (SENT) [suppressed 13 similar messages]
I20260709 12:25:08.044174  5560 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:08.067451  7453 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:08.042305  5701 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:08.142254  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:08.045038  7693 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.102s
W20260709 12:25:08.161538  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 482743us
I20260709 12:25:08.164007  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:08.189193  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:08.197551  6234 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:08.185601  7328 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.164918  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:08.198840  7614 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.007s	sys 0.010s
I20260709 12:25:08.205842  7373 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:08.207063  7549 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:08.279505  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 211782us
I20260709 12:25:08.195433  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:08.188761  7583 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:25:08.291139  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 321070us
I20260709 12:25:08.290050  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:08.287783  7685 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:08.251374  7655 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:08.291414  7614 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s	user 0.008s	sys 0.010s
I20260709 12:25:08.332885  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:08.375746  5702 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:25:08.333978  7699 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
W20260709 12:25:08.335813  7700 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.290s	user 0.000s	sys 0.001s
I20260709 12:25:08.336687  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:08.336851  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:08.297053  5627 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:08.294539  7328 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:08.328205  5556 raft_consensus.cc:2376] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3. Re-sending same reply.
I20260709 12:25:08.340327  7361 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:08.340799  7549 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:08.307024  7694 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
I20260709 12:25:08.373625  7457 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:08.348656  7712 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.395485  7549 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:08.396162  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:08.397039  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:08.397235  5557 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 4: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:08.287494  7632 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:08.398077  5629 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:08.398643  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:08.399221  5552 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 4: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:08.374796  7685 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:08.353374  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 429) took 1442 ms. Trace:
I20260709 12:25:08.356220  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:08.291963  7373 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:25:08.355175  7690 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260709 12:25:08.387938  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 185685us
I20260709 12:25:08.362386  7705 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.389103  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:08.388847  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:08.373600  7423 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:08.357148  7610 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.341748  7595 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:08.289436  7639 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.686331  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1635) took 1294 ms. Trace:
I20260709 12:25:08.748835  5701 rpcz_store.cc:276] 0709 12:25:07.392227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.396275 (+  4048us) service_pool.cc:224] Handling call
0709 12:25:07.431363 (+ 35088us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.157208 (+725845us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.685940 (+528732us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b5400323f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:08.686241 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1285331840}
I20260709 12:25:08.749572  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:08.750464  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:08.400625  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:08.676422  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:08.420297  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:08.578595  5702 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:08.647536  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1506) took 1249 ms. Trace:
I20260709 12:25:08.652899  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:08.460952  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:08.783530  5635 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:08.752071  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:08.482369  5558 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:08.718896  5708 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:08.653447  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:08.785182  5708 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:08.652922  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:25:08.787222  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 134430us
I20260709 12:25:08.784075  5706 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:25:08.438630  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 149455us
I20260709 12:25:08.751977  5709 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:08.713343  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:08.790835  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1642) took 1604 ms. Trace:
I20260709 12:25:08.791363  5561 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:08.791285  5709 rpcz_store.cc:276] 0709 12:25:07.186414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.324243 (+137829us) service_pool.cc:224] Handling call
0709 12:25:08.356060 (+1031817us) spinlock_profiling.cc:243] Waited 4.94 ms on lock 0x7b5800053198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:08.790777 (+434717us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11360000}
I20260709 12:25:08.631745  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:08.782981  5554 rpcz_store.cc:276] 0709 12:25:07.398011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.398482 (+   471us) service_pool.cc:224] Handling call
0709 12:25:07.423993 (+ 25511us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:07.424411 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:25:07.573962 (+149551us) log.cc:844] Serialized 14 byte log entry
0709 12:25:07.968899 (+394937us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:07.974340 (+  5441us) log.cc:844] Serialized 26 byte log entry
0709 12:25:07.974433 (+    93us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:08.306461 (+332028us) log.cc:844] Serialized 14 byte log entry
0709 12:25:08.365259 (+ 58798us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.365336 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:08.477858 (+112522us) raft_consensus.cc:1685] finished
0709 12:25:08.477888 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:08.647462 (+169574us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":418529,"prepare.queue_time_us":677985,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":1957,"raft.queue_time_us":42628,"raft.run_cpu_time_us":2298,"raft.run_wall_time_us":16188,"thread_start_us":559387,"threads_started":2,"wal-append.queue_time_us":390605}
W20260709 12:25:08.798439  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 309553us
W20260709 12:25:08.392549  7544 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.955s	user 0.014s	sys 0.014s
I20260709 12:25:08.792819  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:08.400174  7685 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:08.807459  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:08.808306  5632 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
W20260709 12:25:08.688375  7672 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.718s	user 0.012s	sys 0.000s
I20260709 12:25:08.785951  5708 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:08.829077  5628 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:08.830370  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1511) took 1385 ms. Trace:
W20260709 12:25:08.803139  7544 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.368s	user 0.014s	sys 0.037s
I20260709 12:25:08.799263  7429 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:08.393050  7577 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:08.388674  7713 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:08.460129  7361 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:08.814260  7709 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:08.856069  7361 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.856683  7361 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:08.856413  7577 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:08.856912  5630 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:08.814515  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:08.815161  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:08.801679  5701 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:08.594516  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:08.459632  5707 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:08.416618  7451 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.706928  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 513) took 1417 ms. Trace:
I20260709 12:25:08.820327  7677 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:08.824015  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:08.383877  7583 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:08.711393  7558 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:08.800587  7694 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:08.717252  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:08.802508  7610 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:08.752539  7373 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:08.396215  7712 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:08.859081  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 46155.3us
I20260709 12:25:08.861500  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:08.873907  5706 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:25:08.800730  7640 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.528s	user 0.002s	sys 0.016s
I20260709 12:25:08.646907  7588 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:08.681562  5629 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:25:08.662218  7684 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:08.827100  7384 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:08.603175  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:08.853958  5708 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:08.830762  5553 rpcz_store.cc:276] 0709 12:25:07.445545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.453574 (+  8029us) service_pool.cc:224] Handling call
0709 12:25:07.466987 (+ 13413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:08.088329 (+621342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.825072 (+736743us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b54000ea1d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:08.830307 (+  5235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1391634176}
I20260709 12:25:08.910087  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:08.837080  7549 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:08.910902  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:08.911075  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 109439us
I20260709 12:25:08.855396  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:08.503232  7600 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:08.875118  5599 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:08.833200  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:08.876884  5554 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:08.400512  6273 rpcz_store.cc:276] 0709 12:25:06.911473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.915170 (+  3697us) service_pool.cc:224] Handling call
0709 12:25:08.353044 (+1437874us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b54004fa6d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:08.353275 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1056358272}
I20260709 12:25:08.855530  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:08.860895  6274 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:08.611171  7683 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:08.913971  7720 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.913903  7384 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:08.914685  7720 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:08.915465  7583 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:08.820683  7666 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.501s	user 0.004s	sys 0.000s
I20260709 12:25:08.897639  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 299) took 1312 ms. Trace:
I20260709 12:25:08.916692  5707 rpcz_store.cc:276] 0709 12:25:07.579177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.588668 (+  9491us) service_pool.cc:224] Handling call
0709 12:25:08.891599 (+1302931us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:08.917842  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1632) took 1994 ms. Trace:
I20260709 12:25:08.918201  5704 rpcz_store.cc:276] 0709 12:25:06.923418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.923880 (+   462us) service_pool.cc:224] Handling call
0709 12:25:06.954269 (+ 30389us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:06.954725 (+   456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:06.954753 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:06.955310 (+   557us) log.cc:844] Serialized 52 byte log entry
0709 12:25:07.110095 (+154785us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:07.171070 (+ 60975us) log.cc:844] Serialized 14 byte log entry
0709 12:25:07.171318 (+   248us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.171388 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:08.820033 (+1648645us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:08.917551 (+ 97518us) raft_consensus.cc:1685] finished
0709 12:25:08.917656 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:08.917788 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":66334,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":6747,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":221,"spinlock_wait_cycles":6112512,"thread_start_us":154392,"threads_started":1,"wal-append.queue_time_us":318771}
I20260709 12:25:08.801192  7718 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.466977  7655 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.388s
I20260709 12:25:08.814270  7704 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:08.932331  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:08.803885  7434 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:08.791590  7639 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:25:08.894997  7640 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.608s	user 0.003s	sys 0.017s
W20260709 12:25:08.921720  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 46664.8us
I20260709 12:25:08.904244  5627 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:08.877729  7694 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.923908  5602 leader_election.cc:400] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4.
I20260709 12:25:08.924913  5630 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:08.925143  6280 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:25:08.927660  5553 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:08.928102  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:08.948716  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:08.861377  7723 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20260709 12:25:08.853333  7672 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:08.861272  7393 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:08.920881  7423 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.802732  7705 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:08.821020  7680 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.113s
I20260709 12:25:08.856056  7713 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:08.891598  7451 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:08.951550  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:08.935901  7704 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:08.979974  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:08.945497  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 85482.8us
I20260709 12:25:08.985976  5554 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
W20260709 12:25:08.988896  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 60918.5us
I20260709 12:25:08.989532  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:08.873656  6272 rpcz_store.cc:276] 0709 12:25:07.289951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.293042 (+  3091us) service_pool.cc:224] Handling call
0709 12:25:08.706703 (+1413661us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b5400711954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:08.706855 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":405653760}
I20260709 12:25:08.863739  7709 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:08.941231  7640 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:08.944706  7683 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.001850  5636 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:25:08.944561  7583 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:08.952142  7694 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:08.936566  7595 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20260709 12:25:09.027086  5631 raft_consensus.cc:2376] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3. Re-sending same reply.
I20260709 12:25:09.025854  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:09.028481  5553 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:08.941534  7434 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:08.996177  7393 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.035440  7683 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:09.025820  7329 raft_consensus.cc:2764] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:25:08.988734  7588 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:08.997494  7704 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:09.049790  5553 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:25:09.025317  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:09.064837  5554 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:09.052910  7683 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:09.054227  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.031890  7429 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:08.990170  7423 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:08.951259  7727 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.139s
I20260709 12:25:08.935645  7718 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.055135  7577 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:09.060002  7384 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:09.090713  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 142073us
I20260709 12:25:09.087285  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:09.092082  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:09.042739  7583 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:09.079813  5556 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:09.064039  7588 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.086997  5633 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:09.141654  7583 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:09.129343  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 149417us
I20260709 12:25:09.143054  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:09.110143  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:09.148185  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 206010us
I20260709 12:25:09.107654  7329 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:09.116253  7384 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.149183  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:09.090561  7717 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:09.107707  7731 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.131750  7610 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:09.138242  7655 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.149820  7684 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:09.173673  5708 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:09.174681  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:09.179438  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:09.152660  7329 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:09.183317  7329 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:09.186534  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1641) took 1695 ms. Trace:
I20260709 12:25:09.158335  7462 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.188781  7730 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:09.189345  7462 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.159842  7595 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.158883  7736 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.160735  7712 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:09.161768  7673 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.187021  5710 rpcz_store.cc:276] 0709 12:25:07.490841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.491051 (+   210us) service_pool.cc:224] Handling call
0709 12:25:07.517976 (+ 26925us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:08.822884 (+1304908us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b180040c0c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffa66a060 00007f4ffa66abfb 00007f4ffa67564b
0709 12:25:08.823067 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:08.823096 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:08.823650 (+   554us) log.cc:844] Serialized 26 byte log entry
0709 12:25:08.920760 (+ 97110us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:25:08.920800 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.920868 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:09.186136 (+265268us) raft_consensus.cc:1685] finished
0709 12:25:09.186165 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.186453 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":122457,"raft.queue_time_us":210721,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":35236,"spinlock_wait_cycles":23340928,"thread_start_us":302697,"threads_started":2,"wal-append.queue_time_us":259230}
I20260709 12:25:09.209829  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:09.210273  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 268172us
W20260709 12:25:09.191179  7733 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 12:25:09.197623  7479 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
W20260709 12:25:09.219950  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 127908us
I20260709 12:25:09.176951  7639 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
W20260709 12:25:09.244526  7725 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.253s	user 0.000s	sys 0.001s
I20260709 12:25:09.244995  7462 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:09.246234  7741 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.246794  7741 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.233580  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1477) took 1012 ms. Trace:
I20260709 12:25:09.120134  7640 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:09.239902  7735 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:25:09.158109  7384 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.198148  7583 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:25:09.256491  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77092.1us
I20260709 12:25:09.257377  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:09.212136  5554 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:09.259047  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:09.260299  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:09.261278  7462 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:09.261685  7462 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.262042  7462 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:09.262256  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:09.264883  5554 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:09.266191  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:09.266300  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:09.266747  5554 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:09.267107  7680 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:09.267779  7680 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:09.267959  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:09.269090  7610 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:09.269646  7610 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.270678  7610 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:09.267961  5554 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:25:09.212119  7672 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:09.204246  7721 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.268474  7680 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:09.218057  7730 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:09.179504  7655 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:09.234822  7717 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:09.302647  7755 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:09.304783  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:09.212563  7595 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.309417  5635 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:25:09.315851  5675 connection.cc:256] RPC call timeout handler was delayed by -1.374986897999861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20110000000000003s for a response
I20260709 12:25:09.230412  7712 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20260709 12:25:09.249094  7741 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:09.163280  7731 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:09.335597  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 186481us
I20260709 12:25:09.250526  7361 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:09.337493  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 276077us
I20260709 12:25:09.346065  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:09.346014  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:09.318701  7393 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:09.329108  7713 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:09.327869  7583 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:09.234449  7673 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.369122  7434 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:09.369225  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 376508us
I20260709 12:25:09.369526  7749 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:25:09.254359  7736 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.354476  7747 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.372977  5627 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:09.373514  5628 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:09.335364  7748 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:09.347824  7672 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.315899  7721 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.399708  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:09.249095  5552 rpcz_store.cc:276] 0709 12:25:08.217100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.816936 (+599836us) service_pool.cc:224] Handling call
0709 12:25:08.817310 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:08.861995 (+ 44685us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:08.862035 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:08.862570 (+   535us) log.cc:844] Serialized 26 byte log entry
0709 12:25:08.913603 (+ 51033us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:09.152408 (+238805us) log.cc:844] Serialized 14 byte log entry
0709 12:25:09.172870 (+ 20462us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.191328 (+ 18458us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:09.197332 (+  6004us) raft_consensus.cc:1685] finished
0709 12:25:09.197361 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.229045 (+ 31684us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40525,"prepare.queue_time_us":450633,"raft.queue_time_us":48095,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":6127,"thread_start_us":203482,"threads_started":2,"wal-append.queue_time_us":275805}
W20260709 12:25:09.370059  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 89354.8us
I20260709 12:25:09.371970  5558 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:09.382279  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:09.383004  7744 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:09.383507  7745 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:09.383908  7739 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:09.415378  7739 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.415761  7739 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:09.391470  7748 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.418889  7748 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:09.419243  7687 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:09.353600  7705 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:09.385092  7685 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.532s	user 0.011s	sys 0.000s
I20260709 12:25:09.386224  7731 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:09.374706  7724 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
W20260709 12:25:09.387531  5524 leader_election.cc:336] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.666s (SENT)
I20260709 12:25:09.429741  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:09.386904  5628 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:09.377632  5675 connection.cc:256] RPC call timeout handler was delayed by -1.3393142939999052s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1583s for a response
I20260709 12:25:09.371074  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:09.393959  7747 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.379761  7752 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260709 12:25:09.396718  7736 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:09.400584  5635 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:09.373251  7713 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:09.383512  7722 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.248s	user 0.001s	sys 0.000s
I20260709 12:25:09.410878  7744 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:09.411562  7730 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.414170  7451 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:09.414939  7384 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:09.415706  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:09.402302  7672 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.381338  7709 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:09.430943  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:09.434630  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:09.409221  7588 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:09.419852  7687 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:09.392611  7434 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
W20260709 12:25:09.457953  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.421025  7718 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:09.461292  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:09.384286  7742 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.479745  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:09.464182  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:09.466585  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:09.461179  5628 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.468797  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:09.482439  6276 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
W20260709 12:25:09.482318  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 46625.9us
I20260709 12:25:09.474475  7760 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:09.474361  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:09.475967  7672 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:09.421727  7583 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 12:25:09.476601  7673 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:09.477252  5556 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:09.427237  7685 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:09.489096  5629 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:09.491854  7687 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.502840  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:09.457494  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:09.485101  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54439.5us
I20260709 12:25:09.404222  7668 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:09.463127  7713 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.473681  7712 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.109s
I20260709 12:25:09.477890  7742 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:09.780602  7595 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:09.787076  6274 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:09.522173  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 312) took 1230 ms. Trace:
W20260709 12:25:09.749271  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 361729us
I20260709 12:25:09.790042  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:09.789152  5636 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:09.790226  5708 rpcz_store.cc:276] 0709 12:25:08.291441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.291570 (+   129us) service_pool.cc:224] Handling call
0709 12:25:08.328719 (+ 37149us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.177696 (+848977us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.521905 (+344209us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b5400067054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:09.522108 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":896802688}
W20260709 12:25:09.798476  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 392455us
I20260709 12:25:09.474397  7451 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:09.799038  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:09.781153  7329 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.597s	user 0.006s	sys 0.005s
W20260709 12:25:09.520285  7558 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.615s	user 0.009s	sys 0.000s
I20260709 12:25:09.798578  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:09.799911  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:09.847525  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49058.8us
I20260709 12:25:09.837304  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:09.504081  7384 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:09.489182  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:09.848642  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:09.787331  7744 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.765076  7760 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:09.780758  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:09.787951  7655 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:09.575095  7373 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.678s	user 0.012s	sys 0.000s
I20260709 12:25:09.778714  7721 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:09.853509  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 304) took 1035 ms. Trace:
I20260709 12:25:09.886296  5552 rpcz_store.cc:276] 0709 12:25:08.818148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.431402 (+613254us) service_pool.cc:224] Handling call
0709 12:25:09.562882 (+131480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.766029 (+203147us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.768770 (+  2741us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:09.781881 (+ 13111us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:09.788092 (+  6211us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.797720 (+  9628us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.850080 (+ 52360us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:09.874465  5710 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:09.878193  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:09.756142  7763 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:09.882055  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:09.893963  5636 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:09.562722  7673 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:25:09.883056  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 83240.2us
I20260709 12:25:09.864174  5632 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 12:25:09.897593  5632 pending_rounds.cc:85] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 5
I20260709 12:25:09.884282  7762 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:09.884148  5627 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:09.797998  5701 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 1.
I20260709 12:25:09.517997  7429 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:09.575361  7753 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:25:09.517378  6271 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 425) took 2825 ms (client timeout 1731 ms). Trace:
W20260709 12:25:09.882277  5675 connection.cc:256] RPC call timeout handler was delayed by -1.207014294000146s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17300000000000001s for a response
I20260709 12:25:09.540947  7704 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:09.849606  7758 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:09.899469  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:09.900023  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:09.923032  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:09.923482  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:09.903569  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:09.898202  5632 pending_rounds.cc:107] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:25:09.925436  5632 raft_consensus.cc:2889] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:09.854387  7687 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:09.852819  7384 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.891577  7558 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:09.850435  7329 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:09.916831  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:09.918041  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1426) took 1317 ms. Trace:
I20260709 12:25:09.904414  7583 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:09.919281  7747 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:09.897985  7721 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 12:25:09.921383  7705 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.910923  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:09.902963  7361 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:09.933614  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.934445  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:09.948148  5705 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:09.892438  7373 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:09.932549  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:09.970237  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.029s	sys 0.066s
I20260709 12:25:09.970723  5636 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:25:09.970924  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.029s	sys 0.066s
I20260709 12:25:09.970927  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:09.850023  7451 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:09.970513  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:09.950686  7384 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:09.907842  7429 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:25:09.949489  7462 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.684s	user 0.009s	sys 0.004s
I20260709 12:25:09.944585  7672 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:09.913328  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:09.940716  7610 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.669s	user 0.012s	sys 0.000s
I20260709 12:25:09.927295  7749 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:09.941053  7595 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:09.932987  7727 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.935616  7673 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:09.909983  7766 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:09.974355  5705 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
W20260709 12:25:09.912925  6271 rpcz_store.cc:269] 0709 12:25:06.692221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.692927 (+   706us) service_pool.cc:224] Handling call
0709 12:25:09.517079 (+2824152us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54002909d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:09.517258 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2993348480}
I20260709 12:25:09.948715  5627 raft_consensus.cc:1217] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Deduplicated request from leader. Original: 1.1->[2.2-3.3]   Dedup: 2.2->[3.3-3.3]
I20260709 12:25:09.976544  5701 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:09.976861  7373 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:09.976992  7451 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:09.980592  7540 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:09.968118  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 306) took 1143 ms. Trace:
I20260709 12:25:10.038357  5627 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4. Ops: [2.2-3.3]
I20260709 12:25:10.040058  7429 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.041354  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:10.042104  7429 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:09.980777  7742 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:09.981824  7694 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.891s	user 0.004s	sys 0.007s
I20260709 12:25:09.950279  5633 rpcz_store.cc:276] 0709 12:25:08.600980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.602852 (+  1872us) service_pool.cc:224] Handling call
0709 12:25:09.917811 (+1314959us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54001ce8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:09.917967 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":394223488}
I20260709 12:25:10.044598  5629 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:09.984046  7713 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:09.984721  7569 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:25:09.984834  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50518us
I20260709 12:25:09.985386  7329 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:10.046406  7704 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:09.986737  7769 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:09.989262  5636 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:09.992877  5708 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
W20260709 12:25:09.971169  7748 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.552s	user 0.014s	sys 0.001s
I20260709 12:25:09.928089  7753 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.039781  5556 rpcz_store.cc:276] 0709 12:25:08.818691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.873537 (+1054846us) service_pool.cc:224] Handling call
0709 12:25:09.903272 (+ 29735us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.912533 (+  9261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.914319 (+  1786us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:09.928185 (+ 13866us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:09.932131 (+  3946us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.948927 (+ 16796us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.961967 (+ 13040us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:10.015509  6272 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:09.991003  7760 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:10.034626  7739 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.618s	user 0.011s	sys 0.000s
I20260709 12:25:10.016553  7558 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.004623  7655 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.016991  7685 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:09.967787  5707 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:09.993757  5552 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:10.020083  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:10.017819  7610 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:10.038319  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 104450us
I20260709 12:25:09.968266  7705 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.047065  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:10.047176  7769 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.028368  7595 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:10.000962  7462 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.043875  7694 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.061599  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:10.024498  7727 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.048136  7748 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.072010  7704 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:10.072139  7595 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.073866  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 307) took 1254 ms. Trace:
I20260709 12:25:10.074748  5552 rpcz_store.cc:276] 0709 12:25:08.818954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.888882 (+1069928us) service_pool.cc:224] Handling call
0709 12:25:09.947865 (+ 58983us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.062212 (+114347us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.073654 (+ 11442us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:10.072741  7704 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:10.074020  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1517) took 1876 ms. Trace:
I20260709 12:25:10.078156  5635 raft_consensus.cc:2368] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6.
I20260709 12:25:10.079007  5559 rpcz_store.cc:276] 0709 12:25:08.198075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.807509 (+609434us) service_pool.cc:224] Handling call
0709 12:25:08.829264 (+ 21755us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.912630 (+ 83366us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:25:08.932158 (+ 19528us) log.cc:844] Serialized 14 byte log entry
0709 12:25:09.353630 (+421472us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:09.353715 (+    85us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:09.353783 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.001229 (+647446us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b180035adc0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:25:10.073794 (+ 72565us) spinlock_profiling.cc:243] Waited 83.1 ms on lock 0x7b54000271d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:10.073838 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.073965 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":34384,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":45648,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":1923,"spinlock_wait_cycles":197810816,"thread_start_us":360543,"threads_started":2,"wal-append.queue_time_us":146553}
I20260709 12:25:10.084013  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:10.086630  7610 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:10.074173  7705 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:10.090883  7748 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:10.117616  7611 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:10.093338  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 310) took 1273 ms. Trace:
I20260709 12:25:10.077728  7753 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:10.080407  7680 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.728s	user 0.005s	sys 0.007s
I20260709 12:25:10.095870  5552 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:10.062687  7655 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.060985  7558 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.095965  6278 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:10.068363  7747 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:25:10.125155  5559 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:10.125588  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:10.126057  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 309) took 1306 ms. Trace:
I20260709 12:25:10.121052  5560 rpcz_store.cc:276] 0709 12:25:08.819696 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.092233 (+1272537us) service_pool.cc:224] Handling call
0709 12:25:10.092688 (+   455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.092888 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:10.092925 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:10.092982 (+    57us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:10.093031 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.093100 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.093239 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:10.059950  7739 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:25:10.127084  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 65598.4us
I20260709 12:25:10.128755  7739 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:10.129364  7739 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:10.129522  7753 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:10.129839  7739 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:10.130337  7753 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.130817  7739 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.134763  7384 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:10.135885  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51944.1us
I20260709 12:25:10.126538  5559 rpcz_store.cc:276] 0709 12:25:08.819458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.081674 (+1262216us) service_pool.cc:224] Handling call
0709 12:25:10.121966 (+ 40292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.125741 (+  3775us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.125985 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:10.138886  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 311) took 1319 ms. Trace:
I20260709 12:25:10.139861  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:10.031028  7766 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.142565  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:10.143817  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:10.138039  7705 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:10.132527  7747 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:10.145565  7726 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:10.151109  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:10.142638  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:10.155812  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:10.165781  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:10.156896  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:10.166846  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:10.167016  6279 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:10.154495  7612 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:10.168777  7769 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:10.168927  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:10.149225  7685 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:25:10.172160  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.173189  7329 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:10.173795  7768 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:10.164613  7540 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:10.185863  7766 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:10.186177  7694 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.186841  7328 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.154727  7569 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.124713  7680 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.179723  7329 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.183328  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:10.140937  5627 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:10.198614  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:10.199779  5632 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:10.200577  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:10.200816  7685 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:25:10.201462  5627 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:10.202173  7709 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:10.202742  7709 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:10.165071  7704 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.203722  7709 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.175951  7610 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:10.156658  7747 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:10.165148  5629 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:25:10.165740  7748 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.193351  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.224702  7748 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.225828  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:10.226641  5702 raft_consensus.cc:2376] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5. Re-sending same reply.
I20260709 12:25:10.227635  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:10.227903  5627 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:10.194618  7687 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.194526  7329 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:10.228684  7687 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.195632  7384 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:10.196194  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.194571  7569 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.241681  7655 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.219120  7328 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:10.243644  7569 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.196774  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.197737  7753 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.197749  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.244973  5706 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:10.245121  5709 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:25:10.246290  5706 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:25:10.246698  5709 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:10.247452  5706 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:10.198727  7423 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.206797  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:10.208989  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:10.249583  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:10.250607  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:10.251163  5632 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:10.152568  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 308) took 1321 ms. Trace:
I20260709 12:25:10.166314  7726 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:10.202752  7727 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.212275  7709 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:10.212884  7705 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:10.213984  7611 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:10.214620  7718 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:25:10.194856  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.217303  7680 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:10.219484  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:10.220922  7558 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.221158  7229 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:10.222252  7452 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:10.222340  7729 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:25:10.222791  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.229278  7748 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.182888  7768 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.230753  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.192016  7612 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:10.235292  7687 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.235275  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:10.237252  7747 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.240792  7329 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.244061  7328 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.252322  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.319808  5632 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
W20260709 12:25:10.572399  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 352958us
I20260709 12:25:10.572924  5702 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:25:10.572937  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:10.573385  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:10.576191  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:10.526690  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.571795  5708 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
W20260709 12:25:10.578337  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 371560us
I20260709 12:25:10.579265  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:10.580128  7643 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:10.587754  7643 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:10.575228  7328 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:10.582922  7229 consensus_queue.cc:1243] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:25:10.594255  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:10.575678  7729 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:10.326987  7738 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:10.289068  7569 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:10.254662  5552 rpcz_store.cc:276] 0709 12:25:08.819219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.078361 (+1259142us) service_pool.cc:224] Handling call
0709 12:25:10.095391 (+ 17030us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.124230 (+ 28839us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.140945 (+ 16715us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:10.575579  7768 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:10.580940  7423 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:10.632344  7423 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:10.633420  7423 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:10.633653  7328 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.634768  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1478) took 2309 ms. Trace:
I20260709 12:25:10.635147  5553 rpcz_store.cc:276] 0709 12:25:08.325943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.032036 (+706093us) service_pool.cc:224] Handling call
0709 12:25:09.032500 (+   464us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.059042 (+ 26542us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.634525 (+1575483us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b540044f554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:10.634707 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38712064}
I20260709 12:25:10.327374  7726 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.636845  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1486) took 1773 ms. Trace:
I20260709 12:25:10.636981  7328 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.589712  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1663) took 1067 ms. Trace:
I20260709 12:25:10.582585  5701 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:10.637295  5553 rpcz_store.cc:276] 0709 12:25:08.863440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.636003 (+1772563us) service_pool.cc:224] Handling call
0709 12:25:10.636332 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.636533 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:10.636563 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:10.636602 (+    39us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:10.636638 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.636692 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.636788 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:10.588918  7612 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:10.639161  7612 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.618667  7643 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.641362  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:10.412909  7384 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:10.580276  7452 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.642318  5636 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:10.601366  7558 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:10.644173  5552 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:10.604077  7758 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.643641  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:10.644922  7384 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.645021  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1489) took 1765 ms. Trace:
I20260709 12:25:10.613595  7680 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:10.646227  7680 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:10.647526  7384 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:10.607781  7694 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260709 12:25:10.601716  7704 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:25:10.649082  7704 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:10.648636  7758 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.649492  7704 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:10.650470  7704 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.630295  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 519) took 1698 ms. Trace:
I20260709 12:25:10.651818  6277 rpcz_store.cc:276] 0709 12:25:08.932156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.932778 (+   622us) service_pool.cc:224] Handling call
0709 12:25:08.947693 (+ 14915us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.072742 (+125049us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.072773 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:09.072814 (+    41us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:09.072849 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.630072 (+1557223us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54002ce7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:10.630112 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.630237 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":287082240}
I20260709 12:25:10.644573  7558 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.655887  7558 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:10.639323  7329 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.644248  7452 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.651572  7758 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.660871  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:10.661160  5553 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:10.662163  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1525) took 1763 ms. Trace:
I20260709 12:25:10.662482  7329 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:10.663302  7329 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.663916  7329 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:10.637743  5705 rpcz_store.cc:276] 0709 12:25:09.522109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.029291 (+507182us) service_pool.cc:224] Handling call
0709 12:25:10.029769 (+   478us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.048122 (+ 18353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:10.048154 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:10.048203 (+    49us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:10.048231 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.589508 (+541277us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b5400074c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:10.589553 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.589659 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41656576}
I20260709 12:25:10.662586  5553 rpcz_store.cc:276] 0709 12:25:08.898762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.660497 (+1761735us) service_pool.cc:224] Handling call
0709 12:25:10.660909 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.661830 (+   921us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.662076 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:10.668406  7729 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.669620  7729 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:10.669983  5553 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:25:10.671558  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1490) took 1754 ms. Trace:
I20260709 12:25:10.671947  5553 rpcz_store.cc:276] 0709 12:25:08.916629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.669160 (+1752531us) service_pool.cc:224] Handling call
0709 12:25:10.669655 (+   495us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.670376 (+   721us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.670733 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:10.676656  5553 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:10.619436  7738 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.682855  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1787) took 1257 ms. Trace:
I20260709 12:25:10.683382  5635 rpcz_store.cc:276] 0709 12:25:09.425506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.836922 (+411416us) service_pool.cc:224] Handling call
0709 12:25:10.682634 (+845712us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b54000c2954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:10.682781 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":523967744}
I20260709 12:25:10.684304  7738 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:10.689837  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:10.691012  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.677762  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 312) took 1558 ms. Trace:
I20260709 12:25:10.693758  5553 rpcz_store.cc:276] 0709 12:25:09.119225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.675488 (+1556263us) service_pool.cc:224] Handling call
0709 12:25:10.676226 (+   738us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.677435 (+  1209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.677696 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:10.696261  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:10.697440  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:10.698228  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:10.701020  5526 leader_election.cc:400] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6.
I20260709 12:25:10.701363  5526 leader_election.cc:403] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:10.703676  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:10.707837  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.699958  7328 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:10.703207  7229 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:10.702800  7762 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:10.702917  7739 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:25:10.715425  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:10.716008  6277 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:10.718850  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:10.719175  7780 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20260709 12:25:10.720449  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:10.721678  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:10.722776  7453 raft_consensus.cc:2764] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:25:10.723076  7558 raft_consensus.cc:2764] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:25:10.722774  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:10.731546  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:10.731956  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.732559  5702 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:10.647454  7680 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.677837  5552 rpcz_store.cc:276] 0709 12:25:08.879858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.643531 (+1763673us) service_pool.cc:224] Handling call
0709 12:25:10.643951 (+   420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.644744 (+   793us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.644961 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:10.734617  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.737445  5706 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:10.703867  7768 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.739176  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:10.735503  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 165058us
I20260709 12:25:10.740731  5702 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:10.741791  7558 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:10.715283  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:10.746346  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.750571  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:10.729492  7569 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:10.759886  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.760377  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:10.760838  5704 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:10.777776  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:10.778580  6276 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:10.746404  7558 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.781765  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.784636  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:10.782568  5706 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:10.787168  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:10.787763  5704 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:10.788125  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 198434us
I20260709 12:25:10.747776  7705 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:10.759313  7747 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:10.790218  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:10.768626  7328 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.797351  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.797991  5706 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:10.749389  7766 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:25:10.781006  7558 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:10.808043  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:10.809511  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:10.775880  7569 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:10.822031  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.804091  7612 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:10.830173  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:10.835558  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1484) took 1973 ms. Trace:
I20260709 12:25:10.799348  7705 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.836935  5555 rpcz_store.cc:276] 0709 12:25:08.862569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.628408 (+1765839us) service_pool.cc:224] Handling call
0709 12:25:10.628881 (+   473us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.629175 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:25:10.800889 (+171714us) log.cc:844] Serialized 14 byte log entry
0709 12:25:10.835222 (+ 34333us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:10.835275 (+    53us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:10.835321 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.835388 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.835500 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3831,"raft.queue_time_us":3955,"thread_start_us":200421,"threads_started":2}
I20260709 12:25:10.839805  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.839423  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:10.801494  7328 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.858239  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:10.861765  7781 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:25:10.862121  7462 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:10.862692  7328 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:10.864303  7779 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
I20260709 12:25:10.868014  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 520) took 1435 ms. Trace:
I20260709 12:25:10.868450  6278 rpcz_store.cc:276] 0709 12:25:09.432568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.432751 (+   183us) service_pool.cc:224] Handling call
0709 12:25:10.867814 (+1435063us) spinlock_profiling.cc:243] Waited 608 ms on lock 0x7b5400343ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:10.867955 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1397311232}
I20260709 12:25:10.869547  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:10.869622  5627 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:10.917992  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:10.928334  7764 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.453s	user 0.009s	sys 0.000s
I20260709 12:25:10.922843  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:10.931800  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1480) took 2131 ms. Trace:
I20260709 12:25:10.921967  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.933414  5706 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 6: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:10.933808  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:10.935267  5704 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:10.932317  5554 rpcz_store.cc:276] 0709 12:25:08.800225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.281751 (+481526us) service_pool.cc:224] Handling call
0709 12:25:09.319909 (+ 38158us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:09.328532 (+  8623us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.328596 (+    64us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:09.335320 (+  6724us) log.cc:844] Serialized 26 byte log entry
0709 12:25:09.431854 (+ 96534us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:09.470582 (+ 38728us) log.cc:844] Serialized 14 byte log entry
0709 12:25:09.471015 (+   433us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.471773 (+   758us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.927696 (+1455923us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:10.931406 (+  3710us) raft_consensus.cc:1685] finished
0709 12:25:10.931530 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.931720 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":35052,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":343730,"raft.run_cpu_time_us":421,"raft.run_wall_time_us":10324,"spinlock_wait_cycles":232097152,"thread_start_us":58851,"threads_started":1,"wal-append.queue_time_us":107309}
I20260709 12:25:10.936559  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.924289  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.828958  7747 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:10.838604  7612 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:10.957774  7612 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:10.959431  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.853529  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:10.960880  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 107442us
I20260709 12:25:10.934902  6277 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:10.936417  5636 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:10.941805  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:10.925349  7778 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:25:10.965811  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:10.967486  5675 connection.cc:256] RPC call timeout handler was delayed by -1.9898070929998539s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18220000000000003s for a response
I20260709 12:25:10.968714  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:10.957549  7389 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:10.960285  5706 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:25:10.955106  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.860255  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:10.969870  5675 connection.cc:256] RPC call timeout handler was delayed by -1.8629070929996487s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19630000000000003s for a response
I20260709 12:25:10.971424  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:10.971707  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:10.973189  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1437) took 2077 ms. Trace:
I20260709 12:25:10.973084  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.972340  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:10.975194  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:10.977766  5636 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:25:10.978353  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:10.932701  7675 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:10.977536  5630 rpcz_store.cc:276] 0709 12:25:08.887835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.888028 (+   193us) service_pool.cc:224] Handling call
0709 12:25:08.888372 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.948301 (+ 59929us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.965072 (+2016771us) spinlock_profiling.cc:243] Waited 32.8 ms on lock 0x7b540032d2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:10.965199 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75425536}
I20260709 12:25:10.940279  7429 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:10.981293  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:10.986459  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.974443  7610 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:10.969792  7453 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:10.988760  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:10.989620  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:10.956651  7747 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.979938  7328 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:25:10.983404  7462 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:10.980195  7675 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.977033  7766 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:10.996563  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:10.986624  7718 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:10.998369  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.986876  7683 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:11.011499  7683 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:11.011175  7718 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:11.012799  7683 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:11.014091  7718 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:10.976557  7384 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260709 12:25:11.003145  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:10.992136  7747 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:10.988868  7768 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:11.007136  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.015959  7718 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:11.000408  7462 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:10.988989  7786 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:11.008494  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.030071  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.030455  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.004312  6278 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
W20260709 12:25:11.019052  7792 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 12:25:11.018931  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:11.032398  5706 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 2.
I20260709 12:25:10.992589  7453 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:11.020505  7328 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.021049  7705 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:11.003950  7675 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:11.045017  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:11.045562  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:11.045145  7328 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:11.046748  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:11.047362  5706 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:25:11.033979  5636 raft_consensus.cc:2368] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5.
I20260709 12:25:11.049922  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.054049  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 330) took 1753 ms. Trace:
I20260709 12:25:11.030859  7462 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:11.055377  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:11.054687  5634 rpcz_store.cc:276] 0709 12:25:09.300899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.301203 (+   304us) service_pool.cc:224] Handling call
0709 12:25:09.920479 (+619276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.920688 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.920719 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:09.920757 (+    38us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:09.920786 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.920835 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.053746 (+1132911us) spinlock_profiling.cc:243] Waited 595 ms on lock 0x7b54001ce8d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:11.053963 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1367600512}
I20260709 12:25:11.033914  7453 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:11.062757  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:11.076445  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.078703  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.085033  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:11.085222  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 84464.6us
I20260709 12:25:11.087384  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:11.087716  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 98183us
I20260709 12:25:11.090811  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.094086  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.097698  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:11.072428  7739 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6.
W20260709 12:25:11.099291  5675 leader_election.cc:336] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.762s (SENT)
I20260709 12:25:11.099893  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:11.051719  7329 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:11.101118  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.103482  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:11.106554  7704 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:11.108641  7709 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:25:11.108898  7798 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:25:11.110309  7328 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:11.111640  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:11.112525  5708 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:11.113461  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:11.104151  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.099864  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.116410  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:11.125011  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:11.125144  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62529.2us
I20260709 12:25:11.127715  6278 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:11.129032  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:11.130204  5634 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 4: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:11.133852  5706 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:11.135188  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:11.136405  5706 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:11.137210  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:11.137897  5706 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:11.139127  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:11.139690  5706 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:11.140674  7766 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:11.140926  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1515) took 2975 ms. Trace:
I20260709 12:25:11.141197  7766 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:11.141932  7766 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:11.122282  7462 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:11.140669  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:11.114660  7803 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:25:11.141324  5561 rpcz_store.cc:276] 0709 12:25:08.165950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.792741 (+626791us) service_pool.cc:224] Handling call
0709 12:25:08.793076 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:08.806948 (+ 13872us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:08.806981 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:08.824766 (+ 17785us) log.cc:844] Serialized 26 byte log entry
0709 12:25:10.927319 (+2102553us) spinlock_profiling.cc:243] Waited 6.18 ms on lock 0x7b1800454960. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:25:10.991581 (+ 64262us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:10.991635 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.991887 (+   252us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:11.127016 (+135129us) raft_consensus.cc:1685] finished
0709 12:25:11.127045 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.140863 (+ 13818us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16902,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":10122,"spinlock_wait_cycles":14219008,"thread_start_us":64071,"threads_started":1,"wal-append.queue_time_us":2184979}
I20260709 12:25:11.147280  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:11.146322  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:11.147878  5708 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:11.148267  5706 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:11.110363  7655 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:11.150204  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:11.150723  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:11.151017  5706 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:25:11.151626  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 56638us
I20260709 12:25:11.149014  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:11.154227  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:11.161212  5675 leader_election.cc:400] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5.
I20260709 12:25:11.162114  5675 leader_election.cc:403] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:11.162803  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:11.164180  5706 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:25:11.164961  5706 pending_rounds.cc:85] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 1
I20260709 12:25:11.164919  5632 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:11.165349  5706 pending_rounds.cc:107] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:25:11.165604  5706 raft_consensus.cc:2889] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:11.160158  7655 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:11.172089  7768 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:11.173110  7462 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:11.174940  6280 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:11.177600  5704 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:11.145008  7558 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:11.202173  5708 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:11.202610  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:11.147971  7675 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.208904  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:11.181135  7229 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:11.199182  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:11.173974  7655 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:11.229674  7655 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:11.206249  7386 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:11.207361  6278 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:11.233474  7747 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:11.235775  7747 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:11.194280  7718 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5.
I20260709 12:25:11.236528  7668 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:11.237653  7229 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:11.212826  7809 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:25:11.215464  7329 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:11.212720  5635 raft_consensus.cc:2368] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7.
I20260709 12:25:11.219520  7786 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:11.170516  7807 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
W20260709 12:25:11.224925  5558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 305) took 2406 ms (client timeout 2259 ms). Trace:
I20260709 12:25:11.225541  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 327) took 1025 ms. Trace:
I20260709 12:25:11.227016  7453 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:10.139243  5560 rpcz_store.cc:276] 0709 12:25:08.819921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.126928 (+1307007us) service_pool.cc:224] Handling call
0709 12:25:10.138242 (+ 11314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.138539 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:10.138570 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:10.138612 (+    42us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:10.138674 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.138732 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.138836 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":94}
I20260709 12:25:11.228344  5703 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:11.239861  7229 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:11.248207  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49282.4us
W20260709 12:25:11.248620  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:11.251032  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51973.3us
I20260709 12:25:11.252938  5634 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:25:11.245957  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 436) took 1905 ms. Trace:
W20260709 12:25:11.257819  5558 rpcz_store.cc:269] 0709 12:25:08.818437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.944651 (+126214us) service_pool.cc:224] Handling call
0709 12:25:11.224711 (+2280060us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b54001af754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:11.224849 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":870816640}
I20260709 12:25:11.266870  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:11.270017  5703 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:11.258024  5709 rpcz_store.cc:276] 0709 12:25:10.200044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.247650 (+ 47606us) service_pool.cc:224] Handling call
0709 12:25:10.248007 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.224718 (+976711us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:11.224748 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:11.224788 (+    40us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:11.224818 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.225311 (+   493us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b5400278654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:11.225349 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.225478 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2245776768}
I20260709 12:25:11.267457  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:11.264762  7809 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:11.279184  5709 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 1. Current term is 2. Ops: []
I20260709 12:25:11.268800  6272 rpcz_store.cc:276] 0709 12:25:09.340285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.342402 (+  2117us) service_pool.cc:224] Handling call
0709 12:25:09.391540 (+ 49138us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.059324 (+667784us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.245490 (+1186166us) spinlock_profiling.cc:243] Waited 594 ms on lock 0x7b5400591ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:11.245814 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1366069120}
I20260709 12:25:11.244280  7612 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.278944  7807 consensus_queue.cc:1243] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:25:11.276104  7393 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260709 12:25:11.282883  7540 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:11.298403  5705 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 4: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:11.291374  7583 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.279978  7796 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
W20260709 12:25:11.301213  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.296727  7809 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.299356  7675 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:11.300033  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:11.306716  7755 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:11.309600  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:11.313133  7229 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:11.313619  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 340) took 1187 ms. Trace:
I20260709 12:25:11.303539  7583 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.324077  5557 raft_consensus.cc:2410] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:11.332580  7684 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.333089  7684 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:11.334333  7734 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 2.153s	user 0.011s	sys 0.015s
W20260709 12:25:11.337205  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.338912  5632 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:11.315647  7755 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:11.340667  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.340500  7755 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.338970  7583 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:11.341836  7787 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.478s	user 0.003s	sys 0.000s
I20260709 12:25:11.319731  7328 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:25:11.319954  7808 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20260709 12:25:11.350360  5703 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:11.317879  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:11.349454  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.351775  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:11.342847  7453 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:11.319557  5629 rpcz_store.cc:276] 0709 12:25:10.126078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.128608 (+  2530us) service_pool.cc:224] Handling call
0709 12:25:10.128993 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.224099 (+ 95106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.313363 (+1089264us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b5400241154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:11.313556 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54006144}
I20260709 12:25:11.354331  7755 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:11.363065  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:11.365749  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:11.374804  5631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1435) took 2556 ms (client timeout 1762 ms). Trace:
I20260709 12:25:11.375146  6280 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:11.326761  7558 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:11.375458  5631 rpcz_store.cc:269] 0709 12:25:08.809164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.809272 (+   108us) service_pool.cc:224] Handling call
0709 12:25:11.365757 (+2556485us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b5400134f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:11.365867 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":388454144}
I20260709 12:25:11.377302  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:11.380401  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:11.381911  5703 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:11.383996  6278 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:11.326321  7766 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.325595  7655 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:11.387785  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 118682us
W20260709 12:25:11.389186  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 71450.9us
I20260709 12:25:11.391953  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:11.393151  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:11.380228  7393 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:11.394821  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:11.394865  7809 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:25:11.401571  7684 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:11.401760  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1652) took 2013 ms. Trace:
I20260709 12:25:11.365440  7675 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 12:25:11.396636  7668 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:11.400743  7683 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:11.401154  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.412782  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:11.402693  5634 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:11.383235  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:11.366217  7577 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 12:25:11.402315  7684 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:11.404886  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:11.402354  7747 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:11.421517  5558 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:11.406899  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:11.407315  7815 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:25:11.408334  5632 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:11.387423  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:11.392061  7329 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:11.421805  7684 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:11.427361  7684 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:11.427362  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:11.429143  5555 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:11.413369  5703 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:11.431722  5557 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1520) took 3065 ms (client timeout 1823 ms). Trace:
W20260709 12:25:11.432142  5557 rpcz_store.cc:269] 0709 12:25:08.366104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.398287 (+ 32183us) service_pool.cc:224] Handling call
0709 12:25:11.431511 (+3033224us) spinlock_profiling.cc:243] Waited 2.92 s on lock 0x7b540044eb54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:11.431649 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6725222144}
I20260709 12:25:11.433619  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 437) took 2092 ms. Trace:
I20260709 12:25:11.434032  6274 rpcz_store.cc:276] 0709 12:25:09.340751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.342471 (+  1720us) service_pool.cc:224] Handling call
0709 12:25:09.561999 (+219528us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.787576 (+225577us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.433414 (+1645838us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54002909d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:11.433555 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":348636928}
I20260709 12:25:11.398815  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.416754  5560 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:25:11.417825  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1651) took 2029 ms. Trace:
I20260709 12:25:11.441751  5710 rpcz_store.cc:276] 0709 12:25:09.388489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.390722 (+  2233us) service_pool.cc:224] Handling call
0709 12:25:09.393957 (+  3235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.889586 (+495629us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.417604 (+1528018us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b540026fa54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:11.417772 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1104407808}
I20260709 12:25:11.412130  7393 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:11.401633  7766 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:11.379612  7818 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:25:11.419337  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:11.423744  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.424831  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:11.422139  7747 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:11.435866  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:11.436170  5559 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 12:25:11.498237  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:11.498777  5559 pending_rounds.cc:85] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 2
I20260709 12:25:11.499074  5559 pending_rounds.cc:107] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 12:25:11.499353  5559 raft_consensus.cc:2889] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:11.499043  5558 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:11.437203  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:11.440131  5703 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:11.409533  7612 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:11.414652  7540 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.495728  5555 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:11.496554  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:11.501778  5561 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:25:11.501643  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 76898.4us
I20260709 12:25:11.405144  7739 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 12:25:11.419687  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:11.506323  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:11.424512  7229 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:11.513280  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:11.514180  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:11.515522  5710 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:11.518282  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:11.518873  7747 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.518998  7393 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:11.519804  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:11.520210  7747 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:11.521879  5710 raft_consensus.cc:2410] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:25:11.523244  7655 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:11.523787  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:11.525223  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:11.525897  5710 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:11.496335  7817 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:11.496629  6272 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:11.402272  5707 rpcz_store.cc:276] 0709 12:25:09.388836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.391298 (+  2462us) service_pool.cc:224] Handling call
0709 12:25:09.396309 (+  5011us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.063055 (+666746us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.401481 (+1338426us) spinlock_profiling.cc:243] Waited 555 ms on lock 0x7b54001991d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:11.401698 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1276185984}
I20260709 12:25:11.518428  7809 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:11.529286  6274 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46486
I20260709 12:25:11.530174  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:11.530882  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:11.531574  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:11.561470  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:11.562005  5561 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:11.562928  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:11.563442  5561 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:11.532748  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:11.523280  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:11.516520  5632 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 12:25:11.532776  5708 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:11.537926  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:11.571282  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:11.571892  5703 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:11.573395  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:11.574182  5702 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:11.539259  5631 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:11.541067  5555 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:11.541433  7229 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:11.577225  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:11.578334  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:11.578874  7229 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:11.550132  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:11.579675  5560 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:11.580011  5707 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:11.579660  7229 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:11.541234  7588 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:11.584437  7453 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.584995  7453 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.584718  7588 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.587450  7453 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:11.587708  7588 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:11.544634  7773 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:11.578893  5555 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:11.589722  5703 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:11.590852  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:11.591164  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:11.591584  5703 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:11.592459  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:11.598477  5561 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:11.599782  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:11.509850  7328 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:11.569175  5554 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:11.602953  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:11.606868  5555 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:11.607605  5705 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:25:11.615198  5702 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:11.615739  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:11.541348  7739 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:11.497835  7578 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:25:11.541764  7817 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:11.541857  5701 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:11.508580  7540 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.650096  5555 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:11.630649  5702 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:25:11.550988  7824 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:11.601665  7818 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.552393  7329 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:25:11.654075  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47938.4us
I20260709 12:25:11.609120  7453 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:11.656299  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.551739  7809 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:11.658078  5557 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:11.615295  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:11.615372  6274 raft_consensus.cc:606] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:11.649454  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:11.665110  5558 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:11.667071  7540 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:11.649858  7773 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:11.663057  7824 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.669795  7684 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.655965  5702 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:25:11.672571  5710 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:11.673561  5552 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 3: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:11.650148  5701 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:25:11.680555  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 140077us
I20260709 12:25:11.633126  7680 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260709 12:25:11.677510  6274 raft_consensus.cc:614] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:11.729949  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.738965  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:11.680255  5707 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
W20260709 12:25:11.732040  7797 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:25:11.743466  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:11.744621  7739 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:11.745157  7739 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.731595  7540 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:11.745565  7739 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:11.736239  7720 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:11.741647  5633 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:11.736861  7328 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260709 12:25:11.745970  7540 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:11.725237  7429 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
I20260709 12:25:11.749087  7540 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:11.752241  5553 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:25:11.678455  7453 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.685173  7773 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:11.758344  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.762621  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:11.764400  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:11.765115  7328 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:25:11.765514  7328 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:11.762389  7747 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.766284  7328 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.766799  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:11.766767  7747 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:11.767509  7328 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:25:11.768502  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1481) took 2942 ms. Trace:
I20260709 12:25:11.768858  5559 rpcz_store.cc:276] 0709 12:25:08.829015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.139119 (+1310104us) service_pool.cc:224] Handling call
0709 12:25:11.435717 (+1296598us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.507857 (+ 72140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.768289 (+260432us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b540044eb50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:11.768446 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2958304640}
I20260709 12:25:11.676293  7818 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.770182  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:11.772410  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:11.779271  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:11.780107  5675 leader_election.cc:400] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7.
I20260709 12:25:11.780441  5675 leader_election.cc:403] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 12:25:11.780531  6279 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:11.739880  7817 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:11.788480  7807 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260709 12:25:11.790823  7817 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:11.791519  7818 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:11.791622  6272 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:11.769524  7809 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:11.777886  7583 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:11.777881  7668 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:11.792914  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:11.777740  7612 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:11.794035  7612 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:11.794461  7612 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:11.794384  7739 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.795634  7830 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:11.796530  7818 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:11.797057  7818 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:11.797897  7818 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:11.806782  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.790022  5524 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:11.777320  7786 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:11.810432  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:11.810920  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:11.817087  7747 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:11.777419  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:11.787684  7329 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:11.821357  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:11.821585  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:11.833472  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:11.810721  7718 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7.
I20260709 12:25:11.822578  5552 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 4: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:11.784595  7540 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20260709 12:25:11.822718  7755 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:11.813787  7453 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:11.822728  7583 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:11.826072  7588 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 12:25:11.853242  7684 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:11.825829  7668 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260709 12:25:11.856487  7786 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:11.857717  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:11.845392  7739 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:11.860046  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.823108  7773 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:11.809202  5701 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:11.891007  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:11.884617  7833 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:25:11.896201  5559 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
W20260709 12:25:11.906896  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 97897.9us
W20260709 12:25:11.914156  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 251311us
I20260709 12:25:11.887622  7824 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:11.842267  7807 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260709 12:25:11.880604  7588 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:11.943367  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:11.894448  7830 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:11.966383  7389 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:25:11.957262  7836 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:11.897658  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:11.979081  7836 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:11.959244  7588 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:12.020560  5552 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:25:12.020917  7588 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:12.022678  7668 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:12.022336  7836 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:12.021270  7694 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:11.939707  7540 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.023094  7668 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:12.023873  7540 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:12.025035  7668 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:12.026081  7668 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:12.028649  7612 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:12.030982  7612 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:11.893716  7786 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:12.113613  7569 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:12.114291  7569 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:12.105430  7687 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:12.099263  5556 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:12.149178  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:12.203676  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 260382us
W20260709 12:25:12.331480  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:12.332213  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:12.336458  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:12.338447  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:12.352162  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.375463  7462 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.378208  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:25:12.195889  7834 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.238s	user 0.001s	sys 0.000s
I20260709 12:25:12.329142  7726 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:12.196672  7786 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:12.114636  7569 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.376133  7462 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:12.380115  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:12.427102  7462 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:12.427800  7462 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:12.119429  7687 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:25:12.396441  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.327142  7588 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:12.166632  7773 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:12.423684  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:12.442525  5557 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:12.347330  7684 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:12.328171  5701 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:12.405586  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:12.446592  5558 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:12.447764  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:12.431803  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.432860  5628 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:12.440775  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 101605us
I20260709 12:25:12.450671  7683 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:12.451900  7540 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:12.454176  7540 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:12.454679  7540 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:12.455555  7540 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:12.457702  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:12.459755  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:12.460718  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:12.461470  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:12.461645  5560 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:12.462423  7824 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:12.434808  7818 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.476569  5628 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 12:25:12.477161  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:12.478180  5557 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:12.430498  7569 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:12.428666  7462 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:12.483433  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.492784  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:12.493937  5676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.016s	sys 0.020s
I20260709 12:25:12.431144  7786 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.485596  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 334) took 3133 ms. Trace:
W20260709 12:25:12.494297  5676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.016s	sys 0.020s
W20260709 12:25:12.495154  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 557327us
I20260709 12:25:12.495639  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:12.496495  5560 raft_consensus.cc:2376] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6. Re-sending same reply.
I20260709 12:25:12.494146  6277 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:12.498358  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:12.494875  5628 rpcz_store.cc:276] 0709 12:25:09.344557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.344691 (+   134us) service_pool.cc:224] Handling call
0709 12:25:09.367678 (+ 22987us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.327670 (+2959992us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b58007001d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5ce0aa 00007f4ffa650447 00007f4ffa668fe7 00007f4ffa66a7de 00007f4ffa674b49 00007f4ffa675416 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4
0709 12:25:12.477230 (+149560us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.477677 (+   447us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3967792896}
I20260709 12:25:12.501356  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1812) took 1201 ms. Trace:
I20260709 12:25:12.512941  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:12.512799  5634 rpcz_store.cc:276] 0709 12:25:11.299502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.299777 (+   275us) service_pool.cc:224] Handling call
0709 12:25:11.300090 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.413382 (+113292us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.501039 (+1087657us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5400438854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:12.501276 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":235411840}
I20260709 12:25:12.477377  7452 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.521651  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:12.500591  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:12.535197  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:12.464596  7773 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:12.535429  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:12.545361  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:12.472440  7684 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:12.562506  7684 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:12.476629  7755 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260709 12:25:12.565841  5558 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
W20260709 12:25:12.537534  7668 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.511s	user 0.010s	sys 0.001s
I20260709 12:25:12.534144  7452 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:12.567720  7740 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260709 12:25:12.565932  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:12.569442  5560 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:12.570470  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:12.486367  7818 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:12.559089  7462 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:12.523950  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:12.524883  7687 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:12.510452  7786 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:12.586750  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:12.587423  5560 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:12.587581  5636 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:25:12.580834  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 200798us
I20260709 12:25:12.565932  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:12.594094  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:12.603209  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:12.604216  5560 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:12.605386  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:12.593932  7687 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:12.610699  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:12.614313  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:12.607190  7729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.015s	sys 0.056s
I20260709 12:25:12.595562  7847 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:12.618111  5552 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 4: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:12.642163  7709 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:12.661547  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:12.666549  5560 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 7
W20260709 12:25:12.662196  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 227936us
I20260709 12:25:12.663195  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1661) took 1851 ms. Trace:
W20260709 12:25:12.660507  7729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.015s	sys 0.057s
W20260709 12:25:12.661806  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.671487  5709 rpcz_store.cc:276] 0709 12:25:10.811912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.282591 (+470679us) service_pool.cc:224] Handling call
0709 12:25:11.289871 (+  7280us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:11.314813 (+ 24942us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:11.314844 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:11.317162 (+  2318us) log.cc:844] Serialized 26 byte log entry
0709 12:25:11.387836 (+ 70674us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:25:11.387873 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.662599 (+1274726us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b54002f4c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:12.662657 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:12.662717 (+    60us) raft_consensus.cc:1685] finished
0709 12:25:12.662732 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.663093 (+   361us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1898,"raft.queue_time_us":3075,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":7167,"spinlock_wait_cycles":28969216,"thread_start_us":61272,"threads_started":1,"wal-append.queue_time_us":69099,"wal-append.run_cpu_time_us":1582,"wal-append.run_wall_time_us":1002447}
I20260709 12:25:12.694337  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:12.675042  7683 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.778328  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:12.779160  5703 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:12.797031  5560 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:12.802269  7429 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.803011  7429 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:12.807395  7429 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:12.809134  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:12.809916  5709 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:12.772825  7452 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:12.810842  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:12.830487  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 295163us
I20260709 12:25:12.810582  7668 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.777657  7683 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:12.793815  7852 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:12.834321  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:12.845072  5630 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:12.846256  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:12.847291  5702 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:12.835266  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:12.814409  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:12.814947  7773 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:12.817783  7818 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:12.819406  7755 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.844223  5703 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:25:12.851600  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.832875  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:12.855530  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 262144us
I20260709 12:25:12.849979  7796 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.840236  7786 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:12.821885  7452 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.032s
W20260709 12:25:12.866936  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.823004  7569 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:12.852092  5701 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:12.872234  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:12.850667  7852 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:12.877979  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1680) took 2083 ms. Trace:
W20260709 12:25:12.870090  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.851959  7855 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:12.861421  7726 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.857193  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:12.879887  5706 rpcz_store.cc:276] 0709 12:25:10.794414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.205579 (+411165us) service_pool.cc:224] Handling call
0709 12:25:11.249219 (+ 43640us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.249382 (+   163us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:25:11.258618 (+  9236us) log.cc:844] Serialized 14 byte log entry
0709 12:25:12.346406 (+1087788us) spinlock_profiling.cc:243] Waited 2.97 ms on lock 0x7b1800444400. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:25:12.877503 (+531097us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:12.877584 (+    81us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:12.877648 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.877739 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.877901 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2216,"prepare.queue_time_us":9105,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":1046,"raft.queue_time_us":83711,"raft.run_wall_time_us":49818,"spinlock_wait_cycles":10032256,"thread_start_us":528291,"threads_started":2,"wal-append.queue_time_us":1470736}
I20260709 12:25:12.973546  6275 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:12.968802  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:12.977504  7859 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:25:12.978881  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 360) took 1015 ms. Trace:
I20260709 12:25:12.979229  5631 rpcz_store.cc:276] 0709 12:25:11.963759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.964166 (+   407us) service_pool.cc:224] Handling call
0709 12:25:11.968799 (+  4633us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:11.981223 (+ 12424us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:11.981270 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:12.444383 (+463113us) log.cc:844] Serialized 26 byte log entry
0709 12:25:12.731854 (+287471us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:12.731901 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.731988 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:12.978673 (+246685us) raft_consensus.cc:1685] finished
0709 12:25:12.978701 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.978826 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11938,"raft.queue_time_us":19858,"spinlock_wait_cycles":7278464,"thread_start_us":220775,"threads_started":1,"wal-append.queue_time_us":367941}
I20260709 12:25:12.980098  7569 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:12.981940  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1678) took 2195 ms. Trace:
W20260709 12:25:12.983448  5704 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1650) took 2185 ms (client timeout 1877 ms). Trace:
W20260709 12:25:13.005343  5704 rpcz_store.cc:269] 0709 12:25:10.798195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.938650 (+140455us) service_pool.cc:224] Handling call
0709 12:25:12.983281 (+2044631us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54000e0bd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:12.983382 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":383129984}
I20260709 12:25:12.984153  5707 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:12.984262  7861 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:12.844238  7668 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:12.985275  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 113174us
I20260709 12:25:13.006651  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.006932  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.006992  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:12.972213  7704 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260709 12:25:13.007688  5703 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:13.008514  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.009174  7683 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:13.009302  7861 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:13.009496  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:13.009786  7668 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:12.860466  7755 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.010571  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.010345  7861 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.011293  5703 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 6: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:13.011957  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:13.012033  7755 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:12.988965  7850 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 12:25:12.990518  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:12.864988  7858 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:12.996227  7773 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:12.997885  5552 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:25:12.999004  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.000782  5706 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:25:13.001430  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:12.965641  7852 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:13.020394  5635 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:13.001832  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:13.003415  5710 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:12.966495  7786 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:25:13.004148  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.025590  5702 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:13.027426  5702 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:12.982496  5701 rpcz_store.cc:276] 0709 12:25:10.786312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.172905 (+386593us) service_pool.cc:224] Handling call
0709 12:25:11.173220 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.533624 (+360404us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54000404a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa6741c0 00007f4ffa675231 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c
0709 12:25:12.964233 (+1430609us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.981867 (+ 17634us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":275582208}
W20260709 12:25:13.006225  6237 connection.cc:256] RPC call timeout handler was delayed by -1.3029757529998278s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1683s for a response
W20260709 12:25:13.032018  6237 connection.cc:256] RPC call timeout handler was delayed by -1.2089893639995353s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2646s for a response
W20260709 12:25:13.032610  6237 connection.cc:256] RPC call timeout handler was delayed by -1.2027970549997917s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22480000000000003s for a response
W20260709 12:25:13.033012  6237 connection.cc:256] RPC call timeout handler was delayed by -1.0358263519997308s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1946s for a response
W20260709 12:25:13.033399  6237 connection.cc:256] RPC call timeout handler was delayed by -1.0224091069999304s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21070000000000003s for a response
I20260709 12:25:13.033845  5702 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:25:12.987118  7726 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.005676  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 359) took 1043 ms. Trace:
I20260709 12:25:12.865435  7796 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.039373  5632 rpcz_store.cc:276] 0709 12:25:11.962854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.963191 (+   337us) service_pool.cc:224] Handling call
0709 12:25:11.969420 (+  6229us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.969667 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:25:12.677579 (+707912us) log.cc:844] Serialized 14 byte log entry
0709 12:25:13.005395 (+327816us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:13.005440 (+    45us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:13.005478 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.005534 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:13.005628 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":324770,"prepare.queue_time_us":406366,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":8236,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":2888,"thread_start_us":629018,"threads_started":2}
I20260709 12:25:13.042843  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.043457  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.043731  5703 raft_consensus.cc:2410] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:13.016916  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:13.017045  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.044564  5560 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:13.021870  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:13.023213  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 184230us
I20260709 12:25:13.023850  5709 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:13.045686  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:12.872601  7854 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.124s
I20260709 12:25:13.048084  5710 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 12:25:13.048506  5710 pending_rounds.cc:85] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 5
I20260709 12:25:13.018893  7847 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:25:13.048758  5710 pending_rounds.cc:107] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:25:13.046620  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:13.049446  5710 raft_consensus.cc:2889] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:13.012553  7853 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.048679  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.016388  7855 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:13.050336  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:13.053702  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 81736us
I20260709 12:25:13.020875  7569 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.053889  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:13.055426  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:13.057353  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.058212  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:13.063123  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:13.065201  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.065821  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:13.067719  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:13.069097  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:13.075321  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:13.097497  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:13.025710  7773 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.051196  7865 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:13.061725  7862 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:13.101545  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:13.063386  7752 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:13.096720  7540 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:13.107509  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1510) took 1118 ms. Trace:
I20260709 12:25:13.111826  5557 rpcz_store.cc:276] 0709 12:25:11.988749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.101287 (+1112538us) service_pool.cc:224] Handling call
0709 12:25:13.107452 (+  6165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:13.113372  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.069468  7854 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:13.062546  7588 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:13.070349  7643 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:13.068823  7786 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:13.062115  7569 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.064208  7729 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 12:25:13.058471  7832 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 12:25:13.117523  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:13.098254  7684 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:13.091804  7864 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:13.066747  7462 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 12:25:13.064040  7429 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 12:25:13.068504  7668 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260709 12:25:13.119560  7643 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.129312  5705 raft_consensus.cc:2376] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4. Re-sending same reply.
I20260709 12:25:13.069607  7755 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260709 12:25:13.088555  7229 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 12:25:13.101361  7870 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:25:13.134599  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 82870.8us
I20260709 12:25:13.097792  7683 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260709 12:25:13.135952  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.171280  5557 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:13.172537  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.091872  7845 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260709 12:25:13.111032  7540 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:13.150581  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 87557.5us
I20260709 12:25:13.187567  6237 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:13.116683  7854 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:13.129849  7384 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:13.130632  7862 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:13.177450  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.157675  7643 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:13.119274  7871 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 12:25:13.158226  7877 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:13.224444  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1531) took 1681 ms. Trace:
W20260709 12:25:13.171507  7868 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:25:13.233958  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.243134  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.211342  7540 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:13.238066  5555 rpcz_store.cc:276] 0709 12:25:11.542871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.616373 (+ 73502us) service_pool.cc:224] Handling call
0709 12:25:11.616723 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.651324 (+ 34601us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.224082 (+1572758us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b540044e654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:13.224342 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":76279552}
I20260709 12:25:13.245230  5557 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:13.256336  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.257257  5555 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:13.258435  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:13.261716  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.263145  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1542) took 1399 ms. Trace:
I20260709 12:25:13.263556  5555 rpcz_store.cc:276] 0709 12:25:11.864152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.258123 (+1393971us) service_pool.cc:224] Handling call
0709 12:25:13.263088 (+  4965us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:13.261716  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 164300us
I20260709 12:25:13.264484  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.266937  5555 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 6: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:13.267540  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1498) took 1440 ms. Trace:
I20260709 12:25:13.267905  5555 rpcz_store.cc:276] 0709 12:25:11.827316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.264208 (+1436892us) service_pool.cc:224] Handling call
0709 12:25:13.267473 (+  3265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:13.269001  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.269891  5555 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:13.269783  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:13.275035  5709 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:13.271781  7872 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:25:13.272269  7773 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:13.272905  7726 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:13.228694  7862 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:13.311537  5710 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:13.118551  7752 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.312717  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:13.231773  7865 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.263348  5557 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:13.244419  7384 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:13.020687  7858 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:13.305064  7874 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:25:13.306129  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.308977  5709 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:13.317056  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.317653  5709 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:13.322178  7858 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:13.274963  7878 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:13.063179  7853 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.314262  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:13.328694  5557 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:13.329505  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.308668  7882 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:25:13.309497  5676 connection.cc:256] RPC call timeout handler was delayed by -1.3115737389998685s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20299999999999999s for a response
I20260709 12:25:13.061046  7855 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.306928  7729 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:13.330397  5707 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:25:13.330466  6273 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:13.322211  7643 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:13.303228  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 132879us
I20260709 12:25:13.332047  7752 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:13.362073  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:13.363051  7858 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:13.363592  7858 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:13.363454  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:13.312642  7862 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:13.314544  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.339871  5555 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:13.364871  7858 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.367126  5561 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:13.347735  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:13.380777  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1530) took 1838 ms. Trace:
I20260709 12:25:13.374178  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:13.369832  5703 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:13.376696  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:13.378316  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1496) took 1554 ms. Trace:
I20260709 12:25:13.350361  7773 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.381311  5561 rpcz_store.cc:276] 0709 12:25:11.542659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.607825 (+ 65166us) service_pool.cc:224] Handling call
0709 12:25:13.366627 (+1758802us) spinlock_profiling.cc:243] Waited 7.35 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:25:13.366890 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.380405 (+ 13515us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.380695 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16909568}
I20260709 12:25:13.383980  5555 rpcz_store.cc:276] 0709 12:25:11.823705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.328780 (+1505075us) service_pool.cc:224] Handling call
0709 12:25:13.378251 (+ 49471us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:13.385430  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.385778  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.386232  5555 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:13.387598  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.388289  5555 raft_consensus.cc:2376] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7. Re-sending same reply.
I20260709 12:25:13.382437  6280 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:13.390566  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.392155  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.393697  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.394441  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.394886  5561 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:13.396780  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1652) took 2587 ms. Trace:
I20260709 12:25:13.397300  5708 rpcz_store.cc:276] 0709 12:25:10.802752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.214174 (+411422us) service_pool.cc:224] Handling call
0709 12:25:11.532597 (+318423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.570089 (+ 37492us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.389861 (+1819772us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54002f4250. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:13.390040 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":623362432}
I20260709 12:25:13.397506  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:13.400733  5557 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:13.395673  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.402760  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.402570  5561 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:13.403239  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:13.404490  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:13.303658  7854 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.405876  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.406410  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:13.406754  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.406646  5561 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:13.405067  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:13.378043  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:13.380059  7869 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:13.323089  7865 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.407905  7687 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 12:25:13.397429  7879 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:13.410279  5676 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:13.398723  7745 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:13.360544  7729 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.337446  7878 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.434551  7588 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:13.435928  7878 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:13.413978  5599 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:13.409965  7655 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:13.356465  7882 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.415244  5704 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:13.415258  7687 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:13.443490  7694 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:13.444911  7687 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:13.445196  7694 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:13.445745  7694 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:13.446215  7687 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.447883  7687 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:13.420625  7577 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:25:13.422873  5555 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:13.405920  7786 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:13.426864  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:13.428119  7745 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.452690  7384 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:13.453100  7384 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.453439  7384 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:13.453943  7864 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:13.436923  7869 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:13.455711  7869 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.457104  7845 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:25:13.432252  7718 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:13.458293  7845 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:13.458586  7718 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:13.458789  7845 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:13.460783  7852 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.463711  7852 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:13.465754  7882 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:13.466553  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.467321  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:13.469305  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:13.452169  7745 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:13.397861  7773 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.474208  7773 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:13.458303  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.433840  7643 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:13.412470  5701 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:13.412271  7861 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:13.437129  5557 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:13.443056  7655 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.416262  7865 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:13.459679  7845 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.477762  7729 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:13.536927  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:13.486255  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.628738  5560 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 4: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:25:13.511379  5676 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.030s (SENT) [suppressed 38 similar messages]
W20260709 12:25:13.535845  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 121930us
I20260709 12:25:13.630373  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:25:13.630575  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 443095us
I20260709 12:25:13.631397  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:13.632406  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:13.630375  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:13.641772  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:13.573599  7883 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:13.645526  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.646382  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:13.440115  7884 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.215s
I20260709 12:25:13.647517  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1541) took 1784 ms. Trace:
I20260709 12:25:13.648561  7462 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:13.647846  5709 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:13.647866  5560 rpcz_store.cc:276] 0709 12:25:11.863595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.646752 (+1783157us) service_pool.cc:224] Handling call
0709 12:25:13.647009 (+   257us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.647208 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:13.647242 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:13.647287 (+    45us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:13.647320 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.647373 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:13.647468 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:13.652875  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:13.653899  5560 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:13.654735  5709 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:13.600603  7773 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:13.642762  7807 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.603s	user 0.010s	sys 0.020s
I20260709 12:25:13.654660  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:13.632697  7677 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:25:13.639938  7655 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:13.649590  7861 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:13.657852  6237 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:13.658073  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.648800  6274 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:13.632782  7756 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 12:25:13.649243  7748 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:13.661068  7832 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:25:13.664505  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 153214us
I20260709 12:25:13.658928  7683 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:13.710692  7745 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.712021  7683 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.713728  7745 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:13.714083  7683 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:13.663321  7773 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.708734  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.716599  7773 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:13.717298  7832 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:25:13.717846  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.718197  7694 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.718283  7832 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.722203  5708 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:13.723227  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.723812  5708 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:13.711198  7871 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 12:25:13.709239  7677 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:13.701790  7882 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:13.649631  7883 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.653398  7462 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:13.733794  7883 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:13.733937  7462 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:13.662953  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:13.669057  7384 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.661429  7643 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.459679  7718 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.645117  7729 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:13.584309  7862 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.697496  6274 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:13.649650  7876 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 12:25:13.703260  7540 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.704479  7861 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.706007  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 544) took 1215 ms. Trace:
W20260709 12:25:13.707129  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49331.2us
I20260709 12:25:13.655113  7588 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:13.661597  7807 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.663s	user 0.011s	sys 0.020s
I20260709 12:25:13.659050  7830 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.645519  7845 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:13.732663  7752 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.655820  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1511) took 1545 ms. Trace:
I20260709 12:25:13.740694  5707 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:13.765102  5560 rpcz_store.cc:276] 0709 12:25:12.110134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.653490 (+1543356us) service_pool.cc:224] Handling call
0709 12:25:13.653733 (+   243us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.655599 (+  1866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.655764 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:13.736526  7855 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260709 12:25:13.702288  7865 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.745155  7748 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:13.749862  7694 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:13.741679  7643 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.754365  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:13.749718  7729 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:13.767488  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:13.818468  5702 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:25:13.760154  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1529) took 2217 ms. Trace:
I20260709 12:25:13.824162  5554 rpcz_store.cc:276] 0709 12:25:11.542424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.605596 (+ 63172us) service_pool.cc:224] Handling call
0709 12:25:13.759233 (+2153637us) spinlock_profiling.cc:243] Waited 9.45 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:25:13.759516 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.759717 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:13.759756 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:13.759802 (+    46us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:13.759835 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.759912 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:13.760068 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21731712}
I20260709 12:25:13.825860  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:13.756745  7876 consensus_queue.cc:1243] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:25:13.755137  7718 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:25:13.813820  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.834363  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:13.835952  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:13.835031  5554 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:13.826803  5560 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:13.768360  7865 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.819726  7756 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:13.821599  7773 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:25:13.860320  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.820097  7862 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:13.863420  5560 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:13.864821  5560 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:13.864907  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:13.865568  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1512) took 1669 ms. Trace:
I20260709 12:25:13.865931  5560 rpcz_store.cc:276] 0709 12:25:12.196419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.864380 (+1667961us) service_pool.cc:224] Handling call
0709 12:25:13.864659 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.865335 (+   676us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.865517 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:13.839787  7694 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:13.816201  7752 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.766081  7830 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.765434  7588 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.878476  5554 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:13.880649  5676 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:13.881403  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54929.3us
I20260709 12:25:13.882030  7862 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:13.882731  7862 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:13.883885  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1494) took 2060 ms. Trace:
I20260709 12:25:13.883975  7862 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.848187  7756 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:13.888597  7729 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.758100  6273 rpcz_store.cc:276] 0709 12:25:12.491008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.498924 (+  7916us) service_pool.cc:224] Handling call
0709 12:25:13.321378 (+822454us) spinlock_profiling.cc:243] Waited 3.59 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:25:13.321665 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.369578 (+ 47913us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.705690 (+336112us) spinlock_profiling.cc:243] Waited 4.95 ms on lock 0x7b54002e2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:13.705927 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19640960}
I20260709 12:25:13.890715  7886 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:25:13.893926  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.895077  7748 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.897882  7752 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:13.899410  7830 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:13.867174  7884 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:13.870500  7852 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:25:13.919051  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 164792us
I20260709 12:25:13.841704  6274 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:13.928144  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45619.1us
I20260709 12:25:13.919870  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:13.863441  7773 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:13.905496  7884 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:13.913600  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:13.884500  5561 rpcz_store.cc:276] 0709 12:25:11.823132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.407640 (+1584508us) service_pool.cc:224] Handling call
0709 12:25:13.408143 (+   503us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:13.413443 (+  5300us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:13.413470 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:13.416157 (+  2687us) log.cc:844] Serialized 26 byte log entry
0709 12:25:13.601308 (+185151us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:13.841698 (+240390us) log.cc:844] Serialized 14 byte log entry
0709 12:25:13.883457 (+ 41759us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.883572 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:13.883634 (+    62us) raft_consensus.cc:1685] finished
0709 12:25:13.883653 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:13.883813 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":288216,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":58114,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":1883,"thread_start_us":452273,"threads_started":3,"wal-append.queue_time_us":308852}
I20260709 12:25:13.916155  7384 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:13.932544  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:13.933646  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:13.934581  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:13.935112  5561 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:13.925328  7852 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:25:13.938301  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.939093  7876 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:13.944651  5561 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:13.952718  5561 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:25:13.931164  7683 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.953362  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1519) took 1754 ms. Trace:
I20260709 12:25:13.934092  7884 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:13.926630  7462 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:13.924553  7756 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:13.954196  5559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1491) took 2134 ms (client timeout 1737 ms). Trace:
W20260709 12:25:13.954681  5553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1533) took 2411 ms (client timeout 2030 ms). Trace:
I20260709 12:25:14.074517  5561 rpcz_store.cc:276] 0709 12:25:12.198821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.952198 (+1753377us) service_pool.cc:224] Handling call
0709 12:25:13.952565 (+   367us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:13.953136 (+   571us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.953313 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:13.967991  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.069814  7540 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:14.059155  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:14.074741  5559 rpcz_store.cc:269] 0709 12:25:11.820310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.820531 (+   221us) service_pool.cc:224] Handling call
0709 12:25:13.953960 (+2133429us) spinlock_profiling.cc:243] Waited 61.7 ms on lock 0x7b5400082354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:13.954106 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":141921664}
I20260709 12:25:14.079236  5559 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
W20260709 12:25:14.075271  5553 rpcz_store.cc:269] 0709 12:25:11.543279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.561155 (+ 17876us) service_pool.cc:224] Handling call
0709 12:25:13.954493 (+2393338us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54000eb5d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:13.954617 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":436593920}
I20260709 12:25:14.083084  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1522) took 1883 ms. Trace:
I20260709 12:25:14.084237  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1521) took 1882 ms. Trace:
I20260709 12:25:14.084509  5553 rpcz_store.cc:276] 0709 12:25:12.199703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.082368 (+1882665us) service_pool.cc:224] Handling call
0709 12:25:14.082639 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.082805 (+   166us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.082828 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:14.082868 (+    40us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:14.082901 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.082948 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.083036 (+    88us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:14.086786  5559 rpcz_store.cc:276] 0709 12:25:12.199408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.078715 (+1879307us) service_pool.cc:224] Handling call
0709 12:25:14.079041 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.081588 (+  2547us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.081767 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:14.088063  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:14.088630  5553 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:14.091375  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1535) took 1680 ms. Trace:
I20260709 12:25:13.938288  7729 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:13.948505  7830 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:13.949792  6273 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46486
I20260709 12:25:14.091805  5553 rpcz_store.cc:276] 0709 12:25:12.411268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.090586 (+1679318us) service_pool.cc:224] Handling call
0709 12:25:14.090877 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.091051 (+   174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.091083 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:14.091122 (+    39us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:14.091158 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.091215 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.091323 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:14.091624  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:14.094992  5553 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:25:14.096271  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1536) took 1684 ms. Trace:
I20260709 12:25:14.096642  5553 rpcz_store.cc:276] 0709 12:25:12.411558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.094516 (+1682958us) service_pool.cc:224] Handling call
0709 12:25:14.094778 (+   262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.095998 (+  1220us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.096213 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:14.076818  7643 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:14.074327  7876 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:14.099319  5553 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260709 12:25:14.098976  5559 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:13.940593  7655 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:14.095338  6273 raft_consensus.cc:606] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:14.099781  5553 pending_rounds.cc:85] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 1
I20260709 12:25:14.091107  7540 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:14.102133  5553 pending_rounds.cc:107] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:25:14.102977  5553 raft_consensus.cc:2889] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:14.093696  7729 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:14.097939  7756 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:14.100704  7876 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:14.110642  5631 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:25:14.101905  6273 raft_consensus.cc:614] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:14.113933  7773 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:14.098434  7848 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:13.953540  7897 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.155s
I20260709 12:25:14.106582  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1545) took 1695 ms. Trace:
I20260709 12:25:14.049350  7904 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260709 12:25:14.117341  6273 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e20e319f46504838a20c558a66692da6"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:46486
I20260709 12:25:14.118177  6273 raft_consensus.cc:606] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:14.119369  7588 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:14.120265  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:25:14.120918  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1697) took 1072 ms. Trace:
I20260709 12:25:14.121487  5703 rpcz_store.cc:276] 0709 12:25:13.048248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.048407 (+   159us) service_pool.cc:224] Handling call
0709 12:25:14.120722 (+1072315us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b54002f5b54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:14.120850 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":731170048}
I20260709 12:25:14.120260  7718 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:14.123375  7718 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:14.128238  7718 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:14.128842  7903 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:25:14.115576  5553 rpcz_store.cc:276] 0709 12:25:12.411624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.097245 (+1685621us) service_pool.cc:224] Handling call
0709 12:25:14.097458 (+   213us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.106225 (+  8767us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.106507 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:14.133097  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:14.131711  7876 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:14.140712  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:14.142180  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:14.146387  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:14.099833  7852 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:14.151772  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:14.151846  6279 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:14.153139  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:14.153752  5701 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:25:14.155443  5553 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:14.156632  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1546) took 1744 ms. Trace:
I20260709 12:25:14.157158  5553 rpcz_store.cc:276] 0709 12:25:12.411893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.134752 (+1722859us) service_pool.cc:224] Handling call
0709 12:25:14.135031 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.156241 (+ 21210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.156552 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:14.086565  7683 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:14.114711  7540 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:14.154275  7883 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:14.135221  7848 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.159324  7882 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:14.159183  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:14.106493  7865 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:14.159873  7885 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:14.200398  7885 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.200994  7885 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:14.202661  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:14.203527  7738 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:14.203918  7738 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:14.204281  7738 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:14.204034  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:14.160804  6271 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:14.161499  7870 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:14.162864  5707 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:25:14.163226  7462 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:14.163569  7881 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.927s	user 0.006s	sys 0.000s
I20260709 12:25:14.163416  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:14.220142  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:14.164893  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:14.165076  6280 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:14.221043  5707 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:14.221259  5629 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:14.165506  7694 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:25:14.222071  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1816) took 2098 ms (client timeout 1865 ms). Trace:
I20260709 12:25:14.165720  5635 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:14.222862  5629 rpcz_store.cc:269] 0709 12:25:12.123540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.149997 (+ 26457us) service_pool.cc:224] Handling call
0709 12:25:14.164706 (+2014709us) spinlock_profiling.cc:243] Waited 3.76 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:14.221978 (+ 57272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8657152}
I20260709 12:25:14.105796  7729 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:14.226181  5561 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:14.220788  6278 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:14.220744  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:14.169466  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:14.171041  7897 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.190497  7540 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:14.232093  7897 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:14.234725  7897 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:14.192025  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 1.14776e+06us
I20260709 12:25:14.237365  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:14.239570  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:14.216343  6276 raft_consensus.cc:2368] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6.
I20260709 12:25:14.220201  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:14.159773  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:14.189527  7848 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:14.234624  7883 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:14.251392  5560 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:14.246352  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 533) took 1381 ms. Trace:
I20260709 12:25:14.189195  7773 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.262828  7655 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:14.250718  7869 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:14.242084  7462 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 12:25:14.261618  7639 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:14.253782  6276 rpcz_store.cc:276] 0709 12:25:12.864782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.865187 (+   405us) service_pool.cc:224] Handling call
0709 12:25:14.203812 (+1338625us) spinlock_profiling.cc:243] Waited 6.64 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:14.246281 (+ 42469us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15275904}
I20260709 12:25:14.263008  7773 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:14.258328  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:14.274068  6279 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:25:14.265492  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.213251  7588 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.277113  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:14.265998  7882 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:14.281247  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:14.281360  7884 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:14.283114  5553 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:25:14.287798  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:14.253157  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:14.290503  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:14.291193  5703 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:14.295651  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 340) took 1021 ms. Trace:
I20260709 12:25:14.296173  5704 rpcz_store.cc:276] 0709 12:25:13.274411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.274749 (+   338us) service_pool.cc:224] Handling call
0709 12:25:14.295465 (+1020716us) spinlock_profiling.cc:243] Waited 99.3 ms on lock 0x7b54002e81d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:14.295579 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":228355456}
I20260709 12:25:14.297022  7899 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.297876  5704 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:14.297655  7899 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:14.301239  7684 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:14.301695  7853 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:14.303308  5558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1501) took 2475 ms (client timeout 1882 ms). Trace:
I20260709 12:25:14.303810  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:14.301714  7684 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:14.296065  7832 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:14.300251  7899 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:14.306282  7845 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:14.308674  5706 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 6: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:14.286727  7588 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:14.332422  5628 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:14.292239  5552 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260709 12:25:14.304029  5558 rpcz_store.cc:269] 0709 12:25:11.828220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.447436 (+619216us) service_pool.cc:224] Handling call
0709 12:25:14.303115 (+1855679us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54000eb0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:14.303227 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":269822720}
W20260709 12:25:14.312049  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.313418  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:14.312701  5632 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:14.328073  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1545) took 1329 ms. Trace:
I20260709 12:25:14.318300  7683 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:14.329016  5636 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:14.341514  6279 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:14.309792  7869 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:14.354598  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:14.344930  5553 rpcz_store.cc:276] 0709 12:25:12.998270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.274277 (+1276007us) service_pool.cc:224] Handling call
0709 12:25:14.282744 (+  8467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:14.327115 (+ 44371us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.327996 (+   881us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:14.328420  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.304100  5703 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:14.357164  5558 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:14.357940  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1553) took 1048 ms. Trace:
I20260709 12:25:14.358352  5558 rpcz_store.cc:276] 0709 12:25:13.309225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.343534 (+1034309us) service_pool.cc:224] Handling call
0709 12:25:14.357002 (+ 13468us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.357683 (+   681us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.357868 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:14.331990  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:14.367820  5599 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:14.330859  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:14.353705  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:14.374184  5628 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:14.345109  5704 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:14.376787  7773 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:14.377791  7684 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:14.378770  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1520) took 2179 ms. Trace:
I20260709 12:25:14.379150  5561 rpcz_store.cc:276] 0709 12:25:12.199128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.076139 (+1877011us) service_pool.cc:224] Handling call
0709 12:25:14.076473 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.226698 (+150225us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.378539 (+151841us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b540023f854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:14.378713 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":343733632}
I20260709 12:25:14.379993  7796 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:14.381073  5629 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:14.381459  7796 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:14.382071  7796 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:14.383798  7726 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:14.379145  7684 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:14.385530  5708 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:14.331045  7845 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.370976  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1516) took 2173 ms. Trace:
W20260709 12:25:14.348560  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 396939us
I20260709 12:25:14.391947  5558 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:14.389626  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:14.394841  5704 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:14.396188  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:14.391394  5554 rpcz_store.cc:276] 0709 12:25:12.197889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.887540 (+1689651us) service_pool.cc:224] Handling call
0709 12:25:13.887958 (+   418us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:13.902691 (+ 14733us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:13.902735 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:13.904071 (+  1336us) log.cc:844] Serialized 26 byte log entry
0709 12:25:14.265161 (+361090us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b180022ffc0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:25:14.332570 (+ 67409us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:14.352230 (+ 19660us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.352469 (+   239us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.352580 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:14.352634 (+    54us) raft_consensus.cc:1685] finished
0709 12:25:14.370725 (+ 18091us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.370900 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4251,"prepare.queue_time_us":19507,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":5038,"raft.queue_time_us":19195,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":4458,"spinlock_wait_cycles":8724224,"thread_start_us":67193,"threads_started":1,"wal-append.queue_time_us":422197}
I20260709 12:25:14.398339  5554 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:25:14.399334  5554 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: [3.5-4.6]
W20260709 12:25:14.400524  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.392661  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:14.329274  7748 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 12:25:14.404116  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:14.353675  7900 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:14.400205  5554 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:25:14.410874  5628 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:14.354307  7869 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:14.331403  7832 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:14.416447  6279 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:14.384222  7726 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:14.390686  7845 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:14.421049  7643 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:14.374641  7861 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.390856  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1549) took 1349 ms. Trace:
W20260709 12:25:14.492654  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:14.506575  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:14.425743  7916 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:25:14.411966  7773 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:14.423300  7869 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:14.422363  7748 consensus_queue.cc:1243] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:25:14.418165  7904 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20260709 12:25:14.382958  7796 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:14.505779  7852 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.493335  7882 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 12:25:14.521513  7773 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:14.422510  7900 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:14.509294  7861 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:14.536818  7639 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:14.569240  7639 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.569610  7639 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:14.577116  7913 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:25:14.551313  7876 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:14.582662  7885 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:14.582978  7540 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.582890  7639 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:14.584824  7540 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:14.586107  7861 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:14.586869  7540 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:14.507665  5552 rpcz_store.cc:276] 0709 12:25:13.041975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.291506 (+1249531us) service_pool.cc:224] Handling call
0709 12:25:14.291950 (+   444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:14.345490 (+ 53540us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.389066 (+ 43576us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:14.606787  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 221820us
I20260709 12:25:14.562124  7773 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:14.551352  7853 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:14.612939  7869 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:14.613165  7853 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:14.562887  7683 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:25:14.618283  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:14.620525  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.617448  5601 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:14.583644  7884 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:14.549718  7852 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:14.534933  7726 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:14.652522  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:14.651481  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:14.756910  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 333082us
W20260709 12:25:14.675340  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44703.9us
I20260709 12:25:15.035830  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 335) took 1769 ms. Trace:
I20260709 12:25:15.036556  5702 rpcz_store.cc:276] 0709 12:25:13.265984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.266140 (+   156us) service_pool.cc:224] Handling call
0709 12:25:13.266563 (+   423us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:13.823850 (+557287us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.862298 (+1038448us) spinlock_profiling.cc:243] Waited 552 ms on lock 0x7b5400067054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:15.035745 (+173447us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1268627712}
I20260709 12:25:15.038619  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:15.033273  6280 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:25:14.882496  7912 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.377s	user 0.001s	sys 0.000s
I20260709 12:25:15.031371  6276 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:15.059799  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:14.663108  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.060616  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1525) took 2828 ms. Trace:
W20260709 12:25:14.893702  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.060627  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:15.062016  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.063517  5702 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:14.853657  5552 pending_rounds.cc:85] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 6
I20260709 12:25:15.068955  5552 pending_rounds.cc:107] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:25:15.069003  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:14.889498  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.029662  7925 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.302s	user 0.000s	sys 0.000s
W20260709 12:25:15.013855  7684 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.629s	user 0.012s	sys 0.000s
W20260709 12:25:14.930367  7915 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.315s	user 0.001s	sys 0.000s
W20260709 12:25:15.030475  7738 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.826s	user 0.012s	sys 0.000s
W20260709 12:25:14.873224  5707 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.651s	user 0.008s	sys 0.007s
I20260709 12:25:14.857980  7861 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:15.075433  7726 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:15.032182  7809 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:15.069844  7904 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.076681  7908 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:15.076829  7540 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:15.083204  7729 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.830s	user 0.006s	sys 0.004s
I20260709 12:25:15.056867  7683 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.732028  7569 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:14.640842  7847 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:15.063565  7911 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.530s	user 0.002s	sys 0.000s
I20260709 12:25:15.085665  7870 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:14.692625  7921 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:25:15.065224  7869 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:14.632223  5701 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
W20260709 12:25:14.731213  7918 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.163s	user 0.000s	sys 0.000s
W20260709 12:25:15.138820  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.139529  7773 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.513s	user 0.009s	sys 0.000s
I20260709 12:25:15.114071  7540 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:14.692957  7883 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.398s
I20260709 12:25:15.092027  7853 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:14.643215  7588 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:15.069360  5552 raft_consensus.cc:2889] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:14.767041  5630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1442) took 3758 ms (client timeout 1601 ms). Trace:
W20260709 12:25:15.153277  5630 rpcz_store.cc:269] 0709 12:25:11.008404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.029818 (+ 21414us) service_pool.cc:224] Handling call
0709 12:25:14.766781 (+3736963us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b540032d7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:14.766940 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4172428544}
I20260709 12:25:14.778949  5704 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:15.095160  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:15.155972  5635 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
W20260709 12:25:15.157495  5602 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.008s	sys 0.035s
W20260709 12:25:15.158154  5602 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.008s	sys 0.035s
W20260709 12:25:14.764876  5706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1694) took 1748 ms (client timeout 1720 ms). Trace:
W20260709 12:25:15.158433  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 530726us
W20260709 12:25:15.158807  5706 rpcz_store.cc:269] 0709 12:25:13.015970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.020055 (+  4085us) service_pool.cc:224] Handling call
0709 12:25:14.764507 (+1744452us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54001991d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:14.764762 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2907974400}
I20260709 12:25:15.061594  5559 rpcz_store.cc:276] 0709 12:25:12.232752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.087913 (+1855161us) service_pool.cc:224] Handling call
0709 12:25:14.092689 (+  4776us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.101505 (+  8816us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.060125 (+958620us) spinlock_profiling.cc:243] Waited 6.1 ms on lock 0x7b54003c46d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:15.060292 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14030720}
W20260709 12:25:15.097429  5675 leader_election.cc:336] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.601s (SENT)
I20260709 12:25:15.101486  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:15.157409  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:14.763093  7900 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:14.710323  7639 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:15.116822  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52687.3us
I20260709 12:25:15.118140  7429 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:14.730144  7832 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:15.097679  7861 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:15.106490  7904 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:15.121935  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 363) took 1284 ms. Trace:
W20260709 12:25:15.125233  7848 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.856s	user 0.012s	sys 0.000s
I20260709 12:25:15.128863  7569 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.129293  7847 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:15.096432  5707 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:15.106632  7908 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.119565  7683 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:15.093503  7738 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:15.140105  7773 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:15.147760  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:14.670632  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:14.766295  7612 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:15.163717  6276 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:15.130690  7870 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:15.116894  7729 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:15.370438  5553 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.837s	user 0.000s	sys 0.018s
W20260709 12:25:15.546424  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 64637.4us
W20260709 12:25:15.540534  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.561340  5553 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.028s	user 0.004s	sys 0.021s
W20260709 12:25:15.561549  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 464141us
I20260709 12:25:15.569253  5675 leader_election.cc:400] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6.
I20260709 12:25:15.569761  5675 leader_election.cc:403] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:25:15.169087  7925 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:25:15.577159  5703 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
W20260709 12:25:15.577183  5602 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.748s (SENT)
I20260709 12:25:15.609277  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:15.419860  5706 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:25:15.612982  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.544415  5705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1680) took 2418 ms (client timeout 1877 ms). Trace:
W20260709 12:25:15.614518  5705 rpcz_store.cc:269] 0709 12:25:13.001329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.001486 (+   157us) service_pool.cc:224] Handling call
0709 12:25:15.419367 (+2417881us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54000e0bd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:15.419530 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":246452608}
W20260709 12:25:15.615412  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.578981  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:15.573980  5631 rpcz_store.cc:276] 0709 12:25:13.837201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.842815 (+  5614us) service_pool.cc:224] Handling call
0709 12:25:13.864470 (+ 21655us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:14.111111 (+246641us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.121582 (+1010471us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b540009e7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:15.121853 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":565527936}
W20260709 12:25:15.544281  7384 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.017s	sys 0.002s
I20260709 12:25:15.622236  5558 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20260709 12:25:15.616833  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:15.303324  7900 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.422637  7899 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.318655  7639 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:15.641090  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.654691  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 337) took 1662 ms. Trace:
I20260709 12:25:15.656000  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:15.658476  5674 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.006s	sys 0.030s
I20260709 12:25:15.179224  7883 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:15.560796  7612 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:15.560204  7540 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:15.224655  7462 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:15.648464  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:15.560519  7852 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:15.642243  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 65055.1us
W20260709 12:25:15.684978  5602 leader_election.cc:336] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.790s (SENT)
W20260709 12:25:15.685249  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.685506  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:15.565933  7848 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:15.658890  5674 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.006s	sys 0.030s
I20260709 12:25:15.647778  7588 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:15.592334  7908 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:15.655279  5554 rpcz_store.cc:276] 0709 12:25:13.992641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.414547 (+421906us) service_pool.cc:224] Handling call
0709 12:25:14.414896 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:14.419628 (+  4732us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.419660 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:14.420166 (+   506us) log.cc:844] Serialized 26 byte log entry
0709 12:25:14.526737 (+106571us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:14.553702 (+ 26965us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.554775 (+  1073us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.554879 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:15.079289 (+524410us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b1800375760. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:25:15.646552 (+567263us) raft_consensus.cc:1685] finished
0709 12:25:15.654437 (+  7885us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.654614 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3831,"prepare.queue_time_us":34042,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":81,"raft.queue_time_us":4384,"raft.run_cpu_time_us":447,"raft.run_wall_time_us":38343,"spinlock_wait_cycles":6659840,"thread_start_us":106069,"threads_started":1,"wal-append.queue_time_us":191303}
W20260709 12:25:15.614112  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 350) took 1396 ms (client timeout 1561 ms). Trace:
I20260709 12:25:15.639132  7884 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:15.689920  5706 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:25:15.658759  5631 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:15.648168  7921 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:15.613953  7890 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6.
W20260709 12:25:15.700969  6280 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.004s	sys 0.008s
W20260709 12:25:15.743597  6280 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.004s	sys 0.008s
W20260709 12:25:15.356699  5710 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.014s	sys 0.010s
W20260709 12:25:15.745836  5710 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.052s	user 0.014s	sys 0.013s
I20260709 12:25:15.229473  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 361) took 1393 ms. Trace:
I20260709 12:25:15.172276  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:15.703876  5561 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.005s	sys 0.007s
W20260709 12:25:15.977439  5561 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.832s	user 0.005s	sys 0.007s
W20260709 12:25:15.705394  5602 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.882s (SENT)
I20260709 12:25:15.977927  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:25:15.979640  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 274345us
W20260709 12:25:15.980564  5602 leader_election.cc:336] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.905s (SENT)
I20260709 12:25:15.981241  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:15.179207  7927 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.484s
I20260709 12:25:15.983880  7927 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:15.984865  7927 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:15.680719  6272 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:15.728502  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:15.661439  7883 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:15.987560  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:15.987236  7883 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:15.989020  7883 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:15.989310  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:15.990469  6276 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:15.226765  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:15.792996  7884 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:25:15.993495  6277 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 534) took 3125 ms (client timeout 1968 ms). Trace:
W20260709 12:25:15.994117  6277 rpcz_store.cc:269] 0709 12:25:12.868307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.868653 (+   346us) service_pool.cc:224] Handling call
0709 12:25:15.226459 (+2357806us) spinlock_profiling.cc:243] Waited 3.3 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:15.993390 (+766931us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7589120}
W20260709 12:25:15.788846  7845 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.281s	user 0.011s	sys 0.003s
I20260709 12:25:15.996903  7845 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:15.717000  5707 rpcz_store.cc:269] 0709 12:25:14.212237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.219767 (+  7530us) service_pool.cc:224] Handling call
0709 12:25:15.609023 (+1389256us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:15.856927  7683 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:15.720484  7393 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:15.791657  7738 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:15.399513  7577 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:15.712827  7852 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:15.696612  7540 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:15.727699  7853 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.774098  7643 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.661618  7639 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:15.781539  7847 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:15.713054  7588 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:15.866768  7848 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:15.696157  7462 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:15.862334  7612 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:15.649282  7900 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:15.861049  7938 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.438s	user 0.000s	sys 0.001s
I20260709 12:25:15.953949  7869 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:15.954979  7773 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:15.957413  7729 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:15.867556  7890 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:15.959493  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.803416  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 473) took 1127 ms. Trace:
I20260709 12:25:15.655347  7899 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:15.780226  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1708) took 2135 ms. Trace:
I20260709 12:25:16.094017  5709 rpcz_store.cc:276] 0709 12:25:13.644803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.662360 (+ 17557us) service_pool.cc:224] Handling call
0709 12:25:15.098449 (+1436089us) spinlock_profiling.cc:243] Waited 962 us on lock 0x7b5800053198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:25:15.098680 (+   231us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.099326 (+   646us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.099370 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.111782 (+ 12412us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.621202 (+509420us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:25:15.621256 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.621345 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:15.779880 (+158535us) raft_consensus.cc:1685] finished
0709 12:25:15.779903 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.780155 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":29387,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":117,"spinlock_wait_cycles":2212224,"thread_start_us":446463,"threads_started":1,"wal-append.queue_time_us":629040}
I20260709 12:25:15.770529  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:15.972553  6237 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.069s	sys 0.112s
W20260709 12:25:16.096693  6237 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.793s	user 0.069s	sys 0.112s
I20260709 12:25:16.096751  5631 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
W20260709 12:25:16.097203  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 804427us
I20260709 12:25:15.976743  5632 rpcz_store.cc:276] 0709 12:25:13.835082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.835244 (+   162us) service_pool.cc:224] Handling call
0709 12:25:13.840102 (+  4858us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.344243 (+504141us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.228231 (+883988us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x7b54001cf2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:15.228580 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1086169856}
I20260709 12:25:15.993960  7884 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:15.767920  7855 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:16.003297  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:15.701798  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:16.101125  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:16.101752  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1828) took 1170 ms. Trace:
I20260709 12:25:16.102108  5630 rpcz_store.cc:276] 0709 12:25:14.931640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.154899 (+223259us) service_pool.cc:224] Handling call
0709 12:25:16.101693 (+946794us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:16.102700  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:16.103080  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:16.103605  5632 raft_consensus.cc:2410] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:16.103752  5630 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
W20260709 12:25:16.104287  7941 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:25:16.040126  7588 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:16.104717  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:16.105331  5630 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:16.105582  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1824) took 1334 ms. Trace:
I20260709 12:25:16.105921  5628 rpcz_store.cc:276] 0709 12:25:14.771185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.776832 (+  5647us) service_pool.cc:224] Handling call
0709 12:25:14.777371 (+   539us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:15.067143 (+289772us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.067188 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:15.075719 (+  8531us) log.cc:844] Serialized 52 byte log entry
0709 12:25:15.350315 (+274596us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:25:15.350373 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.350472 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:16.105381 (+754909us) raft_consensus.cc:1685] finished
0709 12:25:16.105400 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.105528 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75027,"raft.queue_time_us":279837,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":25925,"thread_start_us":274155,"threads_started":1,"wal-append.queue_time_us":932291}
I20260709 12:25:16.107649  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:16.108378  5628 raft_consensus.cc:2410] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:25:16.101137  7855 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:16.005848  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:16.109150  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1823) took 1338 ms. Trace:
I20260709 12:25:16.106105  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:16.109987  7612 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:16.110859  7639 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:16.111449  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:16.112697  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:15.957425  7809 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.854s	user 0.007s	sys 0.004s
I20260709 12:25:16.013367  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:16.113448  7818 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.113869  6276 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:16.033604  7853 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.113967  7818 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.114795  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:16.026978  7852 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.115748  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:15.954912  7939 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:25:16.033880  7540 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.116428  5632 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:15.881524  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 554) took 1193 ms. Trace:
I20260709 12:25:16.117918  6275 rpcz_store.cc:276] 0709 12:25:14.669688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.669953 (+   265us) service_pool.cc:224] Handling call
0709 12:25:15.854160 (+1184207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:16.116371  5630 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:16.119197  7899 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:16.119748  5630 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:16.120469  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1826) took 1348 ms. Trace:
I20260709 12:25:15.722050  7569 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:16.025374  7726 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.928s	user 0.009s	sys 0.002s
I20260709 12:25:16.115623  6276 raft_consensus.cc:2410] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:25:16.118026  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:15.972776  7870 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.121469  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 353) took 1513 ms. Trace:
I20260709 12:25:16.109601  6277 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 4: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:16.047807  7946 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.133805  5710 rpcz_store.cc:276] 0709 12:25:14.608434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.608605 (+   171us) service_pool.cc:224] Handling call
0709 12:25:14.608947 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:14.615132 (+  6185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.615178 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:14.626869 (+ 11691us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.948751 (+1321882us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:15.948806 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.948904 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:16.121175 (+172271us) raft_consensus.cc:1685] finished
0709 12:25:16.121206 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.121391 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1297,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":126,"thread_start_us":1052161,"threads_started":1,"wal-append.queue_time_us":1096219}
I20260709 12:25:16.109612  7855 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.120815  5630 rpcz_store.cc:276] 0709 12:25:14.771890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.119247 (+1347357us) service_pool.cc:224] Handling call
0709 12:25:16.119549 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.120187 (+   638us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.120408 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:16.319236  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.321282  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1834) took 1046 ms. Trace:
W20260709 12:25:16.309680  5552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.011s	sys 0.013s
W20260709 12:25:16.310464  5553 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.007s	sys 0.018s
W20260709 12:25:16.322626  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 341905us
W20260709 12:25:16.323108  5552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.011s	sys 0.014s
W20260709 12:25:16.323227  5553 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.008s	sys 0.019s
I20260709 12:25:16.132324  5627 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:16.325440  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1554) took 3015 ms. Trace:
W20260709 12:25:16.313220  5527 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.008s	sys 0.013s
W20260709 12:25:16.071048  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.109108  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:16.326227  5527 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.009s	sys 0.013s
W20260709 12:25:16.327936  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.328866  5628 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:25:16.325961  5553 rpcz_store.cc:276] 0709 12:25:13.309524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.355687 (+1046163us) service_pool.cc:224] Handling call
0709 12:25:14.386869 (+ 31182us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.387127 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:25:14.532764 (+145637us) log.cc:844] Serialized 14 byte log entry
0709 12:25:16.324169 (+1791405us) log.cc:844] Serialized 14 byte log entry
0709 12:25:16.324646 (+   477us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:16.324763 (+   117us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:16.324899 (+   136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.325034 (+   135us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.325301 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26951,"prepare.queue_time_us":830546,"prepare.run_cpu_time_us":306,"prepare.run_wall_time_us":3876,"raft.queue_time_us":15064,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":21799,"thread_start_us":1850068,"threads_started":3,"wal-append.queue_time_us":1163564}
I20260709 12:25:16.132750  5629 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:16.322082  5630 rpcz_store.cc:276] 0709 12:25:15.274562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.318674 (+1044112us) service_pool.cc:224] Handling call
0709 12:25:16.319715 (+  1041us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.320301 (+   586us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.320387 (+    86us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:16.320491 (+   104us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:16.320588 (+    97us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.320724 (+   136us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.320973 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:16.324020  5602 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.931s (SENT)
I20260709 12:25:16.334249  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:16.325968  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 339) took 2267 ms. Trace:
I20260709 12:25:16.335373  5552 rpcz_store.cc:276] 0709 12:25:14.058416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.610718 (+552302us) service_pool.cc:224] Handling call
0709 12:25:14.728420 (+117702us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:15.338269 (+609849us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.361352 (+ 23083us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:15.618094 (+256742us) log.cc:844] Serialized 52 byte log entry
0709 12:25:16.324197 (+706103us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:16.325004 (+   807us) log.cc:844] Serialized 14 byte log entry
0709 12:25:16.325409 (+   405us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.325555 (+   146us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:16.325633 (+    78us) raft_consensus.cc:1685] finished
0709 12:25:16.325653 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.325878 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6494,"raft.queue_time_us":26106,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":35314,"thread_start_us":600807,"threads_started":1,"wal-append.queue_time_us":599023}
I20260709 12:25:16.131050  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:16.130822  5632 raft_consensus.cc:2410] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:25:16.342114  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1829) took 1275 ms. Trace:
I20260709 12:25:16.109573  5633 rpcz_store.cc:276] 0709 12:25:14.770825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.771070 (+   245us) service_pool.cc:224] Handling call
0709 12:25:14.771660 (+   590us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:14.772314 (+   654us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.772388 (+    74us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:14.773135 (+   747us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.117835 (+344700us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:15.117890 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.117982 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:16.108916 (+990934us) raft_consensus.cc:1685] finished
0709 12:25:16.108942 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.109090 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":208261,"raft.queue_time_us":77266,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":3675,"thread_start_us":344167,"threads_started":1,"wal-append.queue_time_us":467646}
I20260709 12:25:16.342625  5632 rpcz_store.cc:276] 0709 12:25:15.066193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.117741 (+1051548us) service_pool.cc:224] Handling call
0709 12:25:16.341862 (+224121us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:16.343911  5631 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:25:16.344497  5631 pending_rounds.cc:85] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 6
I20260709 12:25:16.344825  5631 pending_rounds.cc:107] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:25:16.345177  5631 raft_consensus.cc:2889] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:16.345340  5602 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.966s (SENT)
W20260709 12:25:16.347596  5602 leader_election.cc:336] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.032s (SENT)
I20260709 12:25:16.349964  5635 raft_consensus.cc:2410] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20260709 12:25:16.138543  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:16.351985  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:16.353755  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:16.354410  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.356303  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:16.357323  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:16.357812  5701 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:16.359452  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:16.361234  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:16.362316  5706 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:16.146931  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:16.365540  5709 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:16.170708  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 131986us
I20260709 12:25:16.119640  6275 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:25:16.202953  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.141021  6276 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:16.240748  7462 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:16.135636  6277 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:16.378228  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:16.378599  6277 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:16.090243  6278 rpcz_store.cc:276] 0709 12:25:14.676077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.676229 (+   152us) service_pool.cc:224] Handling call
0709 12:25:15.034180 (+357951us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.098118 (+ 63938us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.098154 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.098602 (+   448us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.361762 (+263160us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:15.361861 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.361991 (+   130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:15.803069 (+441078us) raft_consensus.cc:1685] finished
0709 12:25:15.803112 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.803324 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":147800,"raft.queue_time_us":80880,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":73,"thread_start_us":242707,"threads_started":1,"wal-append.queue_time_us":677295}
I20260709 12:25:16.185016  7384 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20260709 12:25:16.253429  7569 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:16.025736  7577 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.266027  7588 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.033828  7643 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.348196  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:25:16.442250  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 94766.8us
I20260709 12:25:16.239531  7818 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:16.088102  7847 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:16.040529  7845 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:16.446887  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:16.089381  7848 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:16.219581  7853 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:16.255435  7854 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:16.449429  7848 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.239714  7861 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:16.099694  7869 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.446962  7847 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.455840  7847 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:16.219962  7852 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:16.120247  7890 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.459515  7890 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:16.256516  7898 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260709 12:25:16.177147  7870 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:16.460852  7898 consensus_queue.cc:1243] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:25:16.461730  7898 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:16.462107  7898 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:16.462821  7898 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:16.272601  7908 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.542s	user 0.011s	sys 0.000s
I20260709 12:25:16.463981  7908 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.267823  7900 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:16.221308  7948 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:25:16.263059  7957 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 12:25:16.455976  7869 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:16.480711  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 338) took 2444 ms. Trace:
I20260709 12:25:16.098737  7959 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:16.481261  5561 rpcz_store.cc:276] 0709 12:25:14.036619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.563124 (+526505us) service_pool.cc:224] Handling call
0709 12:25:14.580761 (+ 17637us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:14.727152 (+146391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.727180 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.099359 (+372179us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.772433 (+673074us) spinlock_profiling.cc:243] Waited 6.54 ms on lock 0x7b480027e958. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:25:16.004545 (+232112us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:16.004616 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.004726 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:16.479979 (+475253us) raft_consensus.cc:1685] finished
0709 12:25:16.480037 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.480478 (+   441us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":264,"raft.run_cpu_time_us":34819,"raft.run_wall_time_us":1191248,"spinlock_wait_cycles":15044864,"thread_start_us":995516,"threads_started":2,"wal-append.queue_time_us":906198}
I20260709 12:25:16.482560  7959 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.483741  7908 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:16.239972  7961 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.263315  7855 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:16.394230  7683 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.386685  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:16.490633  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1712) took 2347 ms. Trace:
I20260709 12:25:16.324064  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:16.404166  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0482729750001454s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1878s for a response
I20260709 12:25:16.407285  7612 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:16.410089  7956 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.413362  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:16.420799  7921 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.421710  7968 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.423580  7883 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:16.422413  5701 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:16.431465  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:16.431839  7954 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:16.433490  7577 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:16.491290  5708 rpcz_store.cc:276] 0709 12:25:14.143338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.143471 (+   133us) service_pool.cc:224] Handling call
0709 12:25:14.143800 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.386154 (+242354us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.489878 (+2103724us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b540006efd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:16.490364 (+   486us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":555324672}
W20260709 12:25:16.453456  5627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 366) took 2607 ms (client timeout 2710 ms). Trace:
I20260709 12:25:16.462591  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:16.177424  7946 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.251804  7904 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:16.448305  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54998.7us
I20260709 12:25:16.466671  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:16.474861  7848 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:16.483320  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:16.484925  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:16.558243  7643 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:16.558919  7796 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:16.462425  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:16.502883  7959 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:16.503505  7683 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.504629  7921 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.505733  7956 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.590818  7885 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:16.506893  6278 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
W20260709 12:25:16.507755  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:16.508143  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0211646230000042s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
I20260709 12:25:16.508808  7612 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.510125  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:16.513010  7968 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.607719  7963 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:25:16.608255  7945 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.652s	user 0.000s	sys 0.001s
W20260709 12:25:16.444818  5555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 331) took 3428 ms (client timeout 2166 ms). Trace:
I20260709 12:25:16.611002  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 555) took 1576 ms. Trace:
I20260709 12:25:16.611459  6279 rpcz_store.cc:276] 0709 12:25:15.035014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.036616 (+  1602us) service_pool.cc:224] Handling call
0709 12:25:15.037051 (+   435us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:15.097398 (+ 60347us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.097436 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:15.103291 (+  5855us) log.cc:844] Serialized 52 byte log entry
0709 12:25:15.576641 (+473350us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:15.576693 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.576762 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:16.607135 (+1030373us) spinlock_profiling.cc:243] Waited 4.65 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:16.610495 (+  3360us) raft_consensus.cc:1685] finished
0709 12:25:16.610519 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.610793 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":59810,"raft.queue_time_us":62443,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":8635,"spinlock_wait_cycles":10702080,"thread_start_us":457679,"threads_started":1,"wal-append.queue_time_us":839766}
W20260709 12:25:16.523192  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:16.616837  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.617290  7968 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:16.520754  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 197126us
W20260709 12:25:16.525146  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 80223.9us
I20260709 12:25:16.521055  7908 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:25:16.522713  5627 rpcz_store.cc:269] 0709 12:25:13.845743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.853265 (+  7522us) service_pool.cc:224] Handling call
0709 12:25:16.453015 (+2599750us) spinlock_profiling.cc:243] Waited 2.19 s on lock 0x7b540032c8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:16.453212 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5034234752}
W20260709 12:25:16.536610  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45282.8us
I20260709 12:25:16.545887  7588 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:16.547412  5706 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:16.535663  7961 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.523347  7869 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.543953  5707 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:16.551630  5709 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:16.572916  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:16.573714  5632 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:16.574141  5635 raft_consensus.cc:2410] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:25:16.521123  7954 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.579442  7683 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:16.580461  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.588454  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:16.589465  7957 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.599500  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:16.631381  7957 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.632388  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:16.633240  5627 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:16.601876  7612 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:16.602206  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:16.636312  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.636675  7954 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:16.638314  7957 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:16.639046  7761 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.603096  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:16.610674  5555 rpcz_store.cc:269] 0709 12:25:13.014023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.389052 (+375029us) service_pool.cc:224] Handling call
0709 12:25:16.442524 (+3053472us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b54000ebad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:16.442809 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":72301312}
I20260709 12:25:16.641268  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:16.584188  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:16.625490  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.623490  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:16.619316  7908 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.650225  7869 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.653306  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1513) took 4456 ms. Trace:
I20260709 12:25:16.654017  7869 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:16.656005  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:16.657598  7683 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:25:16.657671  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1825) took 1886 ms (client timeout 1720 ms). Trace:
I20260709 12:25:16.658336  7683 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.658617  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:16.658396  5629 rpcz_store.cc:269] 0709 12:25:14.771527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.850813 (+ 79286us) service_pool.cc:224] Handling call
0709 12:25:16.657307 (+1806494us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400241654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:16.657470 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2461138688}
I20260709 12:25:16.659174  7683 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:16.668489  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:16.685807  5706 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 6: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:16.661437  5634 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:16.669133  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:16.711000  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:16.666124  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:16.653823  5560 rpcz_store.cc:276] 0709 12:25:12.196881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.866940 (+1670059us) service_pool.cc:224] Handling call
0709 12:25:14.251091 (+384151us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.251981 (+   890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.652883 (+2400902us) spinlock_profiling.cc:243] Waited 384 ms on lock 0x7b54003a98d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:16.653093 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":882549504}
I20260709 12:25:16.668484  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:16.664407  5602 leader_election.cc:336] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.094s (SENT)
I20260709 12:25:16.660282  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:16.645951  7908 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:16.640028  7761 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.660576  7643 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:25:16.733104  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 64933.2us
I20260709 12:25:16.731259  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:16.660558  7853 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:16.733095  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:16.663911  6277 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:16.738550  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.670241  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 475) took 1993 ms. Trace:
W20260709 12:25:16.763955  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 103790us
I20260709 12:25:16.765712  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:16.766289  7900 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:16.766808  7900 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:16.767233  7900 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:16.767452  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:16.768060  7773 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:16.768908  7773 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:16.769292  7773 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.769662  7773 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:16.662184  7961 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:16.775444  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 56824.1us
I20260709 12:25:16.772920  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:16.789551  7752 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:16.791285  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:16.791914  6278 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:16.794159  7929 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:16.794409  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:16.794906  7929 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:16.796499  7946 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:16.796880  7901 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:16.797462  7901 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:25:16.797914  7977 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
I20260709 12:25:16.798816  5709 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:16.798532  7901 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.800221  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:16.801165  5709 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:16.738061  7963 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:25:16.808821  7973 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:25:16.733619  7847 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:16.815158  7847 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.779422  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:16.821270  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:16.823906  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45328.6us
I20260709 12:25:16.783630  7853 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:16.830744  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:16.834638  6277 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:16.809852  7963 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:16.838716  7963 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:16.839740  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:16.840493  7904 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:16.841087  7904 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.843272  7904 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:16.844503  7655 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.845319  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1718) took 2660 ms. Trace:
I20260709 12:25:16.845194  7655 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.846176  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 354) took 2236 ms. Trace:
I20260709 12:25:16.847208  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1723) took 2458 ms. Trace:
I20260709 12:25:16.847704  7655 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:16.848496  5631 raft_consensus.cc:2410] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:16.847577  5704 rpcz_store.cc:276] 0709 12:25:14.388269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.395559 (+  7290us) service_pool.cc:224] Handling call
0709 12:25:14.778660 (+383101us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:15.155276 (+376616us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.846847 (+1691571us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54001991d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:16.847056 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":880239872}
I20260709 12:25:16.850464  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:16.851982  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:16.853251  5704 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:16.771637  6272 rpcz_store.cc:276] 0709 12:25:14.676784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.677063 (+   279us) service_pool.cc:224] Handling call
0709 12:25:14.677669 (+   606us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:15.985868 (+1308199us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.669743 (+683875us) spinlock_profiling.cc:243] Waited 969 ms on lock 0x7b54002909d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:16.670035 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2229666176}
I20260709 12:25:16.854612  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:16.855785  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:16.796098  7929 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.858327  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:16.845742  5703 rpcz_store.cc:276] 0709 12:25:14.185249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.356850 (+171601us) service_pool.cc:224] Handling call
0709 12:25:15.576990 (+1220140us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:15.593059 (+ 16069us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.844925 (+1251866us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54001a8e50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:16.845143 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2766001024}
I20260709 12:25:16.809651  7956 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:16.846566  5702 rpcz_store.cc:276] 0709 12:25:14.610061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.064239 (+454178us) service_pool.cc:224] Handling call
0709 12:25:15.064608 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:15.948117 (+883509us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.948174 (+    57us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:15.948246 (+    72us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:15.948311 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.845762 (+897451us) spinlock_profiling.cc:243] Waited 672 ms on lock 0x7b540006f754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:16.845824 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.846032 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1545008512}
I20260709 12:25:16.866091  6274 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:16.868145  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 133020us
I20260709 12:25:16.866243  5707 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
W20260709 12:25:16.874483  5524 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.877s (SENT)
I20260709 12:25:16.874519  7462 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:16.876153  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 333) took 2938 ms. Trace:
I20260709 12:25:16.876554  5558 rpcz_store.cc:276] 0709 12:25:13.938100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.392996 (+454896us) service_pool.cc:224] Handling call
0709 12:25:14.417323 (+ 24327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.636849 (+1219526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.875733 (+1238884us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54003a89d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:16.875987 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2770299648}
I20260709 12:25:16.917620  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:16.922601  5524 connection.cc:256] RPC call timeout handler was delayed by -2.2996979609997652s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1646s for a response
I20260709 12:25:16.894048  5706 raft_consensus.cc:2376] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5. Re-sending same reply.
I20260709 12:25:16.925252  5709 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:16.893232  7904 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
W20260709 12:25:16.933177  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 59771us
W20260709 12:25:16.936118  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.935858  7853 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.940476  7577 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:25:16.942646  7898 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:16.945541  7761 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:16.946184  7761 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:25:16.923318  5524 connection.cc:256] RPC call timeout handler was delayed by -2.2627825949998623s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
W20260709 12:25:16.955829  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 126436us
W20260709 12:25:17.030592  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 97633.2us
I20260709 12:25:16.954679  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:17.032379  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:16.954499  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:17.033237  5704 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:17.043205  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:17.043859  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 159543us
I20260709 12:25:17.034384  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:16.946550  7761 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:17.062834  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:17.064465  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 400135us
I20260709 12:25:17.054867  7643 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:17.073199  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 118169us
I20260709 12:25:17.073593  7643 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.075706  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:17.090312  7739 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.091034  7739 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:16.917995  7922 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.340s
I20260709 12:25:17.093441  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:17.094931  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:17.101154  7982 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.281s	user 0.001s	sys 0.000s
I20260709 12:25:17.102119  7919 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:17.054590  5627 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:17.107916  7745 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:16.943827  7963 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.110512  7963 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:17.111212  6279 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:17.112713  7588 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:17.112970  7904 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.113224  7588 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:17.113360  7991 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.114459  7588 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.117939  7683 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.118953  7588 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:25:17.121224  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.096153  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:17.125489  5701 raft_consensus.cc:1813] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Handling vote request from an unknown peer 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:17.126078  5701 raft_consensus.cc:2368] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4.
W20260709 12:25:17.040915  7935 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.422s	user 0.006s	sys 0.012s
I20260709 12:25:17.115564  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.116061  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.116451  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.113960  7991 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.116851  7904 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.045368  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:17.131184  5555 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:17.124557  7959 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260709 12:25:17.156774  7683 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:17.156680  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.159588  7904 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:17.160934  5552 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:25:17.161785  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.162068  5701 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:17.163794  5702 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:17.058271  7957 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.108s
I20260709 12:25:17.168612  5561 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:17.158275  5557 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
W20260709 12:25:17.170575  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 92563.5us
W20260709 12:25:17.171556  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 95920.7us
W20260709 12:25:17.177063  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.191363  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.159699  7900 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.192188  7991 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:17.192714  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.193171  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:17.193483  7858 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:17.194437  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:17.231875  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46720.5us
I20260709 12:25:17.187119  5703 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:17.195566  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:17.199402  5559 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:17.178711  7954 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:17.201568  5602 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.142s (SENT)
I20260709 12:25:17.226089  6277 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
W20260709 12:25:17.230013  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.231247  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 58261.8us
I20260709 12:25:17.233734  5702 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:17.234706  5555 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:17.077071  7956 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.151s
I20260709 12:25:17.240628  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:17.198096  7847 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:17.237845  6277 pending_rounds.cc:85] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 5
I20260709 12:25:17.243898  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:17.259768  5561 raft_consensus.cc:2368] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6.
I20260709 12:25:17.260701  7870 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:17.260852  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.245728  5560 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:17.261852  7897 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:17.262220  7897 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:17.258857  6277 pending_rounds.cc:107] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:25:17.246922  7898 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:17.263058  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.247485  7761 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.249259  7643 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:17.230556  7922 consensus_queue.cc:1243] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:25:17.252475  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:17.253966  7883 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.739s	user 0.009s	sys 0.006s
I20260709 12:25:17.280036  7883 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:17.280395  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:17.282004  5560 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:17.256167  7773 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.257223  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:17.257810  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57197us
I20260709 12:25:17.257688  7884 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260709 12:25:17.261274  7870 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.263589  6277 raft_consensus.cc:2889] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:17.289122  7870 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:17.289724  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 544) took 1562 ms. Trace:
I20260709 12:25:17.263036  7897 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.272895  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:17.290218  6277 rpcz_store.cc:276] 0709 12:25:15.727401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.134973 (+407572us) service_pool.cc:224] Handling call
0709 12:25:16.135450 (+   477us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.663460 (+528010us) spinlock_profiling.cc:243] Waited 87.9 ms on lock 0x7b5400772020. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5f006f 00007f4ffa668f60 00007f4ffa66a7de 00007f4ffa674b49 00007f4ffa675416 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62
0709 12:25:17.289352 (+625892us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.289655 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":202068736}
I20260709 12:25:17.291515  7883 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:17.292043  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.292804  5559 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:17.293830  7773 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:17.294785  7933 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260709 12:25:17.298115  7898 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:25:17.301034  7883 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:17.302186  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:17.303165  7883 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:17.303956  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:17.244777  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.306130  5557 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:17.306062  5555 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:17.273814  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:17.226701  7921 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:17.276221  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:17.179678  7908 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.254688  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:17.283663  7739 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:17.289976  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 104598us
I20260709 12:25:17.292081  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.307405  5703 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
W20260709 12:25:17.274461  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 73432.1us
I20260709 12:25:17.293776  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.199038  7904 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.314885  5553 raft_consensus.cc:2368] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6.
W20260709 12:25:17.320026  5602 leader_election.cc:336] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.169s (SENT)
I20260709 12:25:17.323594  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.324680  5561 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:17.304749  7761 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:17.325225  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.286239  7847 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:17.325250  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:17.328615  7919 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:17.329175  7908 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:17.329224  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:17.329651  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 360) took 1749 ms. Trace:
I20260709 12:25:17.330374  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:17.332432  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.336649  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.337227  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.335124  5553 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:17.344588  7963 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.344815  7612 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:17.345238  7963 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.347667  7963 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:17.335969  5705 rpcz_store.cc:276] 0709 12:25:15.579923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.659845 (+ 79922us) service_pool.cc:224] Handling call
0709 12:25:15.714944 (+ 55099us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.120728 (+405784us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:25:16.423396 (+302668us) log.cc:844] Serialized 14 byte log entry
0709 12:25:16.560447 (+137051us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:16.583698 (+ 23251us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:16.584277 (+   579us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.329221 (+744944us) spinlock_profiling.cc:243] Waited 405 ms on lock 0x7b5400075654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:17.329272 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.329450 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27387,"prepare.queue_time_us":339635,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":100,"raft.queue_time_us":24960,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":142,"spinlock_wait_cycles":933212288,"thread_start_us":370593,"threads_started":2,"wal-append.queue_time_us":125971}
I20260709 12:25:17.348965  7540 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:17.281831  7954 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.348780  5557 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:17.350723  5702 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:17.354557  5701 raft_consensus.cc:2410] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260709 12:25:17.355132  5602 leader_election.cc:336] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.185s (SENT)
I20260709 12:25:17.356542  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:17.340662  7922 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:25:17.359294  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.352262  6276 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:17.359299  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.355486  5703 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:17.360592  5553 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:17.345266  7612 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:17.333984  7959 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.028s
I20260709 12:25:17.368955  7921 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:17.350694  7847 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.372315  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:17.372584  7540 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:25:17.373790  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:17.386453  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:17.396773  5703 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:25:17.374589  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:17.377262  6274 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 12:25:17.380880  5554 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:25:17.381383  7922 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:17.383316  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:17.386687  5556 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:17.343926  7904 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.382669  7612 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:17.400163  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:25:17.405030  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49931.1us
I20260709 12:25:17.391443  7900 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:17.402796  6274 pending_rounds.cc:85] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 5
I20260709 12:25:17.413897  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:17.415305  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 188542us
W20260709 12:25:17.419389  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 48972us
I20260709 12:25:17.420485  5559 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:17.408347  5707 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:17.429651  5524 connection.cc:256] RPC call timeout handler was delayed by -1.11685482699977s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
W20260709 12:25:17.430233  5524 connection.cc:256] RPC call timeout handler was delayed by -1.1060548269997525s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1534s for a response
I20260709 12:25:17.386687  7870 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.417311  6274 pending_rounds.cc:107] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:25:17.418767  7853 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:17.402843  7921 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:17.419519  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:17.442677  5602 leader_election.cc:336] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.241s (SENT)
I20260709 12:25:17.443941  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:17.445786  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:17.426944  7588 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:17.446658  5632 raft_consensus.cc:2376] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6. Re-sending same reply.
I20260709 12:25:17.433631  7900 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.434804  7540 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:25:17.455936  7876 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.013s	sys 0.013s
W20260709 12:25:17.456650  7950 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.246s	user 0.001s	sys 0.001s
W20260709 12:25:17.457803  7993 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:25:17.436291  6274 raft_consensus.cc:2889] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:17.466089  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 573982us
I20260709 12:25:17.461970  5552 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 3: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:25:17.477854  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.465705  7876 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.013s	sys 0.013s
I20260709 12:25:17.486713  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:17.490892  5631 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:17.466331  7540 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:25:17.480749  7999 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
W20260709 12:25:17.492092  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 60278.4us
I20260709 12:25:17.474004  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:17.483600  7959 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:17.468580  7900 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.503472  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:17.531109  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 158897us
I20260709 12:25:17.536341  5559 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:17.515853  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 361) took 1370 ms. Trace:
I20260709 12:25:17.536682  5557 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:17.516942  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:17.517206  5703 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:25:17.537969  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1730) took 1078 ms. Trace:
I20260709 12:25:17.538312  5703 rpcz_store.cc:276] 0709 12:25:16.459736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.436497 (+976761us) service_pool.cc:224] Handling call
0709 12:25:17.516774 (+ 80277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:17.537665 (+ 20891us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.537900 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:17.496236  7540 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:17.517271  5524 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:17.517853  7655 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:17.519096  7908 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:25:17.519506  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.552784  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.553654  5559 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:17.553251  5555 raft_consensus.cc:2376] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4. Re-sending same reply.
I20260709 12:25:17.511790  7994 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:25:17.525312  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 556) took 2468 ms. Trace:
I20260709 12:25:17.483012  7588 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:17.558351  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 120582us
I20260709 12:25:17.527166  7847 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:17.527830  5705 raft_consensus.cc:1217] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Deduplicated request from leader. Original: 2.2->[3.3-3.3]   Dedup: 3.3->[]
I20260709 12:25:17.560410  5705 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:25:17.508476  7729 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.536509  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:17.511386  7393 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:17.515761  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:17.541255  5633 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:17.567560  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1731) took 1094 ms. Trace:
I20260709 12:25:17.568037  5705 rpcz_store.cc:276] 0709 12:25:16.473125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.527216 (+1054091us) service_pool.cc:224] Handling call
0709 12:25:17.527624 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:17.567272 (+ 39648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.567501 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:17.537123  5710 rpcz_store.cc:276] 0709 12:25:16.145825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.146551 (+   726us) service_pool.cc:224] Handling call
0709 12:25:16.147098 (+   547us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.484332 (+337234us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.485626 (+  1294us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:16.491703 (+  6077us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:16.491781 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.515487 (+1023706us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b5400278654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:17.515529 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.515700 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":774465536}
I20260709 12:25:17.569054  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.546660  6271 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:17.549610  7921 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:25:17.526954  7954 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:17.528303  7904 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:17.552421  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 466) took 3908 ms. Trace:
I20260709 12:25:17.580992  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.556602  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:17.501345  7922 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.544416  7655 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:17.553615  7870 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:17.595355  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:17.596086  5558 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:17.579411  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:17.569706  5557 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:17.585523  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1733) took 1036 ms. Trace:
I20260709 12:25:17.604100  5704 rpcz_store.cc:276] 0709 12:25:16.550426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.544334 (+993908us) service_pool.cc:224] Handling call
0709 12:25:17.566597 (+ 22263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:17.581706 (+ 15109us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:17.581753 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:17.585197 (+  3444us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:17.585247 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.585308 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.585440 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:17.567384  7393 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.587518  7858 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.587930  7954 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:17.589660  7967 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.570515  7729 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.594051  7588 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:17.556023  6280 rpcz_store.cc:276] 0709 12:25:15.057011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.059663 (+  2652us) service_pool.cc:224] Handling call
0709 12:25:15.060090 (+   427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.161962 (+101872us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.162028 (+    66us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.162718 (+   690us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.949902 (+787184us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:15.949965 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.950063 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.071972 (+1121909us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b1800084e80. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:25:17.524640 (+452668us) raft_consensus.cc:1685] finished
0709 12:25:17.524668 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.525025 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":203484,"raft.queue_time_us":102568,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":11385,"spinlock_wait_cycles":58132352,"thread_start_us":566183,"threads_started":1,"wal-append.queue_time_us":614164}
I20260709 12:25:17.574208  7761 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.596544  7998 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:25:17.596964  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 79804us
I20260709 12:25:17.597481  7655 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.599076  5710 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:17.611752  5557 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:17.600410  7900 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:17.613775  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.582749  7959 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:17.600623  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:17.601203  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 85611.5us
W20260709 12:25:17.602152  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56844.4us
I20260709 12:25:17.606186  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:17.570600  7847 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 12:25:17.605844  7393 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:17.612335  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1736) took 1054 ms. Trace:
W20260709 12:25:17.615840  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 56683.1us
W20260709 12:25:17.617782  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.597506  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:17.614291  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 112126us
I20260709 12:25:17.580705  6274 rpcz_store.cc:276] 0709 12:25:13.629036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.629788 (+   752us) service_pool.cc:224] Handling call
0709 12:25:13.648654 (+ 18866us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.852211 (+3203557us) spinlock_profiling.cc:243] Waited 2.61 s on lock 0x7b58007322d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5ce0aa 00007f4ffa650447 00007f4ffa668fe7 00007f4ffa66a7de 00007f4ffa674b49 00007f4ffa675416 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4
0709 12:25:17.494136 (+641925us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.536963 (+ 42827us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6000034048}
I20260709 12:25:17.619866  7761 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.620252  5707 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:17.633507  7748 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260709 12:25:17.635299  5707 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:17.621270  7853 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:17.616914  7959 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.636571  7885 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.625984  7832 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:17.637207  7832 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:17.638335  5705 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:17.638057  7832 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.639061  7832 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:17.637207  7885 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.642290  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:17.642763  7885 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:17.643090  5709 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:17.636571  7959 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:17.619419  5710 rpcz_store.cc:276] 0709 12:25:16.557791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.569386 (+1011595us) service_pool.cc:224] Handling call
0709 12:25:17.578963 (+  9577us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:17.612098 (+ 33135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.612288 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:17.608464  7858 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.649035  8001 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:17.609189  7963 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:17.632306  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:17.635382  7761 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:17.611658  7967 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.636289  7853 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:17.609237  7954 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.609390  8003 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:17.628492  7655 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:17.673774  5558 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:17.660557  6237 leader_election.cc:400] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6.
I20260709 12:25:17.678875  6237 leader_election.cc:403] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:25:17.653270  5633 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:17.672967  7963 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:17.758863  7998 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:25:17.772644  7963 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:17.758019  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 139942us
I20260709 12:25:17.774235  7847 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:17.774652  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:17.775352  7847 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:17.775760  7847 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:17.776911  8006 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:17.776738  7847 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.777310  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:17.777688  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:17.676530  8008 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.144s
I20260709 12:25:17.757397  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:17.757936  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 128420us
W20260709 12:25:17.765086  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.795264  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.819968  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.820619  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.796705  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.822018  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.835541  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:17.769471  8007 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
W20260709 12:25:17.759099  8004 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.141s	user 0.000s	sys 0.000s
I20260709 12:25:17.677924  7954 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:17.853981  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:17.854640  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.640146  7612 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.674396  7853 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:25:17.869987  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 93679.8us
I20260709 12:25:17.773937  7963 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:17.656112  7858 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:17.651192  5552 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:17.762022  7967 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:17.678390  8003 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:17.898758  8003 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:17.898816  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1732) took 1369 ms. Trace:
I20260709 12:25:17.760291  7933 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:17.856266  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.852936  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 246809us
W20260709 12:25:17.876871  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.884186  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48716.3us
I20260709 12:25:17.885047  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.886065  7929 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:17.889150  7920 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.865664  7761 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:17.886776  7998 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:17.893079  7393 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.894578  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:17.870198  8001 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:17.894865  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.916441  5554 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:17.830024  7959 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.896662  7612 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:17.886392  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:17.897277  7961 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:17.897430  8010 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.874233  7853 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:17.899369  5703 rpcz_store.cc:276] 0709 12:25:16.529503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.541681 (+1012178us) service_pool.cc:224] Handling call
0709 12:25:17.542145 (+   464us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:17.542550 (+   405us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:17.542576 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:17.543124 (+   548us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.613897 (+ 70773us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:17.613922 (+    25us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.613964 (+    42us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.898482 (+284518us) raft_consensus.cc:1685] finished
0709 12:25:17.898502 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.898742 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2637,"raft.queue_time_us":17272,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":5286,"thread_start_us":70365,"threads_started":1,"wal-append.queue_time_us":72671}
I20260709 12:25:17.911449  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:17.887797  8008 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:17.882398  7847 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:17.914747  6278 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:17.869863  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:17.910332  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:17.913800  7920 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.926549  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.928128  5558 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 6: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:17.908972  7929 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:17.960564  7922 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:17.961196  7929 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.962334  7929 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:17.962405  7612 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.963001  7612 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.914224  7967 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.965072  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:17.872398  7540 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.965843  5552 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:17.967510  7540 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:17.923641  7961 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:17.939188  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1698) took 1086 ms. Trace:
I20260709 12:25:17.940533  6274 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:17.929394  8008 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:17.969821  7393 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:17.935685  5554 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 4: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:17.970184  8008 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.971293  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.971995  7920 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:17.924104  7761 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:17.972697  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:17.948599  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:17.976341  5703 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:17.948896  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:17.949606  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:17.950330  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:17.953533  7959 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:17.953992  8010 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:17.955610  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.955758  7729 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:17.964905  7967 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.990767  7967 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:17.991904  5631 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:25:17.969332  7961 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:17.993429  7961 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:17.944571  7900 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:17.948118  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:17.997817  5629 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:17.947270  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:17.998996  5633 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:17.968829  5707 rpcz_store.cc:276] 0709 12:25:16.853010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.644424 (+791414us) service_pool.cc:224] Handling call
0709 12:25:17.644779 (+   355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:17.645172 (+   393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:17.645203 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:17.645709 (+   506us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.850329 (+204620us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:17.850393 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.850498 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.938908 (+ 88410us) raft_consensus.cc:1685] finished
0709 12:25:17.938935 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.939099 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1125,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":115,"thread_start_us":204147,"threads_started":1,"wal-append.queue_time_us":283150}
I20260709 12:25:18.000597  5707 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 12:25:18.000939  5707 pending_rounds.cc:85] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 3
I20260709 12:25:18.001233  5707 pending_rounds.cc:107] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:25:18.001456  5707 raft_consensus.cc:2889] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:18.002293  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1745) took 1083 ms. Trace:
I20260709 12:25:18.002722  5707 rpcz_store.cc:276] 0709 12:25:16.922848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.999892 (+1077044us) service_pool.cc:224] Handling call
0709 12:25:18.000221 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.001880 (+  1659us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.002201 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:17.925124  8001 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:17.972160  8008 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:18.008010  7577 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:25:18.009214  5599 leader_election.cc:400] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4.
I20260709 12:25:18.009594  5599 leader_election.cc:403] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:25:18.005498  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.009554  7577 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:18.010511  7577 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:18.011334  7921 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:25:18.011497  7577 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:18.012629  7745 consensus_queue.cc:1175] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f613ba929b1046d3b5aed909414a95bf" responder_term: 2 status { last_received { term: 2 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 2 index: 2 } }
I20260709 12:25:18.013479  7745 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:18.013988  7577 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:18.005515  8001 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:18.004022  7612 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:18.019979  7921 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4.
I20260709 12:25:18.022006  5635 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:18.023885  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 369) took 1297 ms. Trace:
I20260709 12:25:18.024150  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.025007  5627 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:18.024318  5704 rpcz_store.cc:276] 0709 12:25:16.726290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.607457 (+881167us) service_pool.cc:224] Handling call
0709 12:25:17.613155 (+  5698us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:17.631363 (+ 18208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:17.631393 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:17.632147 (+   754us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.873283 (+241136us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:17.873327 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.873398 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:18.023665 (+150267us) raft_consensus.cc:1685] finished
0709 12:25:18.023685 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:18.023818 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13568,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":141,"thread_start_us":231897,"threads_started":1,"wal-append.queue_time_us":314707}
I20260709 12:25:18.026070  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.027746  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:18.028519  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.029153  5635 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:18.028630  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1705) took 1173 ms. Trace:
I20260709 12:25:18.030019  5706 rpcz_store.cc:276] 0709 12:25:16.855111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.027507 (+1172396us) service_pool.cc:224] Handling call
0709 12:25:18.028546 (+  1039us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:18.032023  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:18.032604  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:18.033195  5636 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:18.033293  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:18.034432  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:18.034940  5553 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:18.037760  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:17.926841  5701 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:18.043066  7761 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:18.044010  7885 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20260709 12:25:18.044374  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:17.925136  7858 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.121s
I20260709 12:25:17.455610  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:18.046433  7858 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:18.046857  7858 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:18.047564  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1734) took 1490 ms. Trace:
I20260709 12:25:18.048388  5705 rpcz_store.cc:276] 0709 12:25:16.557011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.568736 (+1011725us) service_pool.cc:224] Handling call
0709 12:25:17.638170 (+ 69434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:17.638706 (+   536us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.047219 (+408513us) spinlock_profiling.cc:243] Waited 69.1 ms on lock 0x7b54001987d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:18.047457 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":158929152}
I20260709 12:25:18.049759  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:18.050413  5556 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:18.027683  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.052578  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:18.053367  5557 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:18.056057  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.058772  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:18.057080  5634 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:18.059981  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:18.060849  5558 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:18.056667  5554 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
W20260709 12:25:18.063444  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.064157  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:18.064234  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:18.065263  5557 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 6: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:17.981389  7922 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:18.072247  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:18.069918  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:18.073614  5557 raft_consensus.cc:1813] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Handling vote request from an unknown peer 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:18.072469  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:18.068873  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:18.070096  5633 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:18.081058  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:18.082441  5557 raft_consensus.cc:2368] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4.
I20260709 12:25:18.075584  5634 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:18.058820  7729 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:18.068146  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.076982  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:18.082911  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.086692  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:18.074801  5558 raft_consensus.cc:2410] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:25:18.088835  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:18.089605  5704 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:18.093323  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.064805  7961 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:18.071897  7588 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:18.104130  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.107369  5636 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:18.100485  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:18.071966  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:18.114854  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 346925us
I20260709 12:25:18.122820  5554 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:18.126839  5557 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:18.079187  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:18.130396  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:18.130712  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:18.131493  6279 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:18.047817  7858 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:18.072136  7885 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260709 12:25:18.141283  7858 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:18.136090  7748 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:18.142082  7748 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.142841  7748 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:18.136286  7959 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.142091  5627 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:25:18.144718  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:18.144944  7959 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:18.147177  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.147837  7959 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:18.148521  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.149362  7612 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:25:18.150019  7897 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:18.135583  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:18.135653  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:18.141366  7885 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:18.114365  7961 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:18.173998  5557 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:18.151942  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:18.075147  7761 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.062s
I20260709 12:25:18.204300  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:18.206882  7959 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:25:18.208567  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 56822.9us
I20260709 12:25:18.225245  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:18.225981  7996 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:18.226446  7996 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:18.228242  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 343) took 1034 ms. Trace:
I20260709 12:25:18.094317  7577 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20260709 12:25:18.230165  7998 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:18.211270  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:18.212097  7922 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:18.213398  7957 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:18.245960  5702 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:18.217859  8010 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:18.220173  7761 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:18.220924  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:18.173774  7885 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:18.099972  7729 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:18.220932  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 342) took 1027 ms. Trace:
I20260709 12:25:18.174990  5554 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:25:18.310933  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 341) took 1118 ms. Trace:
W20260709 12:25:18.237627  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 102746us
I20260709 12:25:18.243671  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.228736  5561 rpcz_store.cc:276] 0709 12:25:17.193396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.260598 (+ 67202us) service_pool.cc:224] Handling call
0709 12:25:18.228022 (+967424us) spinlock_profiling.cc:243] Waited 63 ms on lock 0x7b54001be254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:18.228174 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":144848128}
I20260709 12:25:18.313251  5635 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:18.314421  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.245612  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:18.254446  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1744) took 1398 ms. Trace:
I20260709 12:25:18.195791  7900 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:18.213639  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:18.259869  5707 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
W20260709 12:25:18.266691  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.305043  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:18.152880  7588 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:18.310492  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:18.257187  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:18.243690  5556 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:18.315765  5703 rpcz_store.cc:276] 0709 12:25:16.860232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.926347 (+1066115us) service_pool.cc:224] Handling call
0709 12:25:17.942439 (+ 16092us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:17.983335 (+ 40896us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.254059 (+270724us) spinlock_profiling.cc:243] Waited 31.2 ms on lock 0x7b5400074c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:18.254352 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":71833472}
I20260709 12:25:18.246344  7957 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:18.269758  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.268239  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.311300  5554 rpcz_store.cc:276] 0709 12:25:17.192684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.124120 (+931436us) service_pool.cc:224] Handling call
0709 12:25:18.157337 (+ 33217us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.310673 (+153336us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.310875 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:18.798991  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1701) took 1945 ms. Trace:
W20260709 12:25:18.759119  5630 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.010s	sys 0.013s
W20260709 12:25:18.802225  5630 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.010s	sys 0.013s
I20260709 12:25:18.310286  5557 rpcz_store.cc:276] 0709 12:25:17.193061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.128274 (+935213us) service_pool.cc:224] Handling call
0709 12:25:18.173815 (+ 45541us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.220513 (+ 46698us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.220790 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:18.802515  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1699) took 1949 ms. Trace:
W20260709 12:25:18.304190  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 367612us
W20260709 12:25:18.423214  5674 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.022s	sys 0.024s
I20260709 12:25:18.436615  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.361169  6279 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:18.383948  5553 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
W20260709 12:25:18.622552  5634 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.523s	user 0.002s	sys 0.011s
I20260709 12:25:18.342375  5561 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:18.805920  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1560) took 1182 ms. Trace:
I20260709 12:25:18.372023  5635 raft_consensus.cc:2368] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4.
I20260709 12:25:18.342720  5559 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:18.806463  5553 rpcz_store.cc:276] 0709 12:25:17.623330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.034199 (+410869us) service_pool.cc:224] Handling call
0709 12:25:18.805863 (+771664us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:18.807344  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 344) took 1613 ms. Trace:
W20260709 12:25:18.804872  5674 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.889s	user 0.039s	sys 0.032s
W20260709 12:25:18.337631  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:18.481858  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:18.812958  5675 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.571s (SENT) [suppressed 44 similar messages]
W20260709 12:25:18.819168  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.803295  5633 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
W20260709 12:25:18.817322  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.807746  5559 rpcz_store.cc:276] 0709 12:25:17.193729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.307956 (+1114227us) service_pool.cc:224] Handling call
0709 12:25:18.342388 (+ 34432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.807111 (+464723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.807290 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:18.359498  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47271.2us
I20260709 12:25:18.802992  5709 rpcz_store.cc:276] 0709 12:25:16.853366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.643749 (+790383us) service_pool.cc:224] Handling call
0709 12:25:17.644093 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:17.814822 (+170729us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:17.814860 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:17.873116 (+ 58256us) log.cc:844] Serialized 26 byte log entry
0709 12:25:18.076939 (+203823us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:25:18.076978 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.077044 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:18.802262 (+725218us) raft_consensus.cc:1685] finished
0709 12:25:18.802285 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:18.802459 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":108606,"raft.run_cpu_time_us":68,"raft.run_wall_time_us":68,"thread_start_us":183804,"threads_started":1,"wal-append.queue_time_us":316812}
I20260709 12:25:18.343286  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:18.806345  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 347) took 1611 ms. Trace:
I20260709 12:25:18.268128  7612 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:18.822616  5556 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:18.839825  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:18.742939  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.005s	sys 0.017s
I20260709 12:25:18.839875  5706 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:18.840531  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:18.821138  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 558986us
I20260709 12:25:18.822050  5636 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:18.822345  5553 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260709 12:25:18.841807  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 496312us
I20260709 12:25:18.846561  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1593) took 1558 ms. Trace:
I20260709 12:25:18.842296  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:18.363229  6271 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:25:18.844128  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1592) took 1556 ms. Trace:
I20260709 12:25:18.839944  5561 rpcz_store.cc:276] 0709 12:25:17.194815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.341471 (+1146656us) service_pool.cc:224] Handling call
0709 12:25:18.341850 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.806115 (+464265us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.806288 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:18.805931  5634 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
W20260709 12:25:18.452629  7963 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.553s	user 0.011s	sys 0.000s
I20260709 12:25:18.846668  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:18.847126  5553 rpcz_store.cc:276] 0709 12:25:17.288615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.821781 (+1533166us) service_pool.cc:224] Handling call
0709 12:25:18.822126 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.846271 (+ 24145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.846480 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:18.842541  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:18.846722  6280 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.016s	sys 0.020s
W20260709 12:25:18.843918  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.007s	sys 0.018s
I20260709 12:25:18.801661  5710 rpcz_store.cc:276] 0709 12:25:16.853960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.648772 (+794812us) service_pool.cc:224] Handling call
0709 12:25:17.649149 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:17.649629 (+   480us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:17.649657 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:17.650142 (+   485us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.916009 (+265867us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:17.916054 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.916128 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:18.798750 (+882622us) raft_consensus.cc:1685] finished
0709 12:25:18.798772 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:18.798919 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1219,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":97583,"thread_start_us":265476,"threads_started":1,"wal-append.queue_time_us":465457}
W20260709 12:25:18.573674  8008 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.568s	user 0.005s	sys 0.007s
I20260709 12:25:18.268028  7761 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:18.767692  7393 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:18.869163  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 782542us
W20260709 12:25:18.374207  7655 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.596s	user 0.013s	sys 0.006s
I20260709 12:25:18.850943  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:18.579708  5552 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.613s	user 0.016s	sys 0.001s
I20260709 12:25:18.350142  7933 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6.
I20260709 12:25:18.434154  7900 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:18.571354  8003 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.672s	user 0.009s	sys 0.006s
I20260709 12:25:18.849061  6278 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:18.882190  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.882920  5631 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:18.341001  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:18.860642  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:18.885712  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:18.885947  5706 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:25:18.452944  7577 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:25:18.846024  7954 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.964s	user 0.016s	sys 0.006s
I20260709 12:25:18.351181  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:18.862252  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.893615  5636 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:18.854554  5558 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:25:18.855680  6280 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.016s	sys 0.020s
I20260709 12:25:18.855041  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.364028  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:18.894898  5635 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
W20260709 12:25:18.900938  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.867981  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:18.901680  5702 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:18.849880  5703 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:25:18.871443  5704 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.781s	user 0.008s	sys 0.003s
I20260709 12:25:18.850840  5556 rpcz_store.cc:276] 0709 12:25:17.288132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.822153 (+1534021us) service_pool.cc:224] Handling call
0709 12:25:18.822408 (+   255us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.843881 (+ 21473us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.844073 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:18.845506  7612 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:25:18.853485  7897 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.703s	user 0.013s	sys 0.001s
I20260709 12:25:18.909790  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.877175  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.879969  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:18.884120  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:18.887059  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.887694  5632 raft_consensus.cc:2410] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:18.888070  5710 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:18.888960  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:18.862112  7832 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:18.359148  7853 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:18.915697  5560 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:18.345798  8001 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.533s
I20260709 12:25:18.856555  7961 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:18.898846  6274 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:18.365300  7885 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:18.900295  5559 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:18.876169  7393 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:18.933200  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 360) took 1589 ms. Trace:
W20260709 12:25:18.903646  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.933559  5559 rpcz_store.cc:276] 0709 12:25:17.343723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.889113 (+1545390us) service_pool.cc:224] Handling call
0709 12:25:18.899992 (+ 10879us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.932911 (+ 32919us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.933141 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:18.934103  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.902499  5704 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:18.875975  5552 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:18.912710  7612 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:18.917562  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.918154  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:18.869817  8006 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 12:25:18.897253  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.950852  5631 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:18.926442  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:18.927341  5556 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
W20260709 12:25:18.928258  7858 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.786s	user 0.015s	sys 0.006s
I20260709 12:25:18.928179  5458 catalog_manager.cc:5697] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:18.952943  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1555) took 1331 ms. Trace:
I20260709 12:25:18.933981  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.936233  5557 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:18.889185  8003 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:18.938474  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:18.960456  5707 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:18.898833  7577 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:18.939306  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:18.949350  5708 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:18.915437  7729 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:18.889508  7900 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:18.952742  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:18.954097  5556 rpcz_store.cc:276] 0709 12:25:17.622012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.912656 (+1290644us) service_pool.cc:224] Handling call
0709 12:25:18.927190 (+ 14534us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.952712 (+ 25522us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.952893 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:18.933525  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 53630.8us
W20260709 12:25:18.959776  5628 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.747s	user 0.009s	sys 0.000s
I20260709 12:25:18.914613  5633 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:18.920233  8030 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:25:18.970863  5457 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35600 (request call id 274) took 15925 ms (client timeout 15000 ms). Trace:
I20260709 12:25:18.970973  8030 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:18.908617  7933 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 12:25:18.886148  7996 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:18.983413  5458 catalog_manager.cc:5697] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 2 to 3, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 3 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:18.984536  7729 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:18.984611  7963 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:18.985090  7729 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:25:18.985707  7748 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.842s	user 0.014s	sys 0.008s
I20260709 12:25:18.985317  5458 catalog_manager.cc:5697] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 4 to 5, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 5 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:25:18.985801  7963 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:18.986004  7729 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:18.986153  7748 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:18.987221  5627 raft_consensus.cc:1813] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Handling vote request from an unknown peer f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:18.987704  5627 raft_consensus.cc:2368] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6.
I20260709 12:25:18.988139  7729 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:18.924736  7832 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:18.896646  7954 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:18.959964  6272 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:18.967092  5559 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:25:18.970093  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:19.002020  5555 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:18.974155  5636 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:19.002449  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1556) took 1380 ms. Trace:
I20260709 12:25:18.975312  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:18.969782  5628 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:19.002978  5559 rpcz_store.cc:276] 0709 12:25:17.622264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.942621 (+1320357us) service_pool.cc:224] Handling call
0709 12:25:18.966639 (+ 24018us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.002143 (+ 35504us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.002385 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:18.971298  5457 rpcz_store.cc:269] 0709 12:25:02.955151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.955324 (+   173us) service_pool.cc:224] Handling call
0709 12:25:18.873184 (+15917860us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:07.568238 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:07.568970 (+   732us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:07.569021 (+    51us) write_op.cc:435] Acquired schema lock
0709 12:25:07.569041 (+    20us) tablet.cc:662] Decoding operations
0709 12:25:07.896295 (+327254us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:07.937388 (+ 41093us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:07.942927 (+  5539us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:25:08.157372 (+214445us) tablet.cc:701] Row locks acquired
0709 12:25:08.198706 (+ 41334us) write_op.cc:260] PREPARE: finished
0709 12:25:08.252090 (+ 53384us) write_op.cc:270] Start()
0709 12:25:08.372987 (+120897us) write_op.cc:276] Timestamp: P: 1783599908211921 usec, L: 0
0709 12:25:08.383658 (+ 10671us) op_driver.cc:348] REPLICATION: starting
0709 12:25:08.611441 (+227783us) log.cc:844] Serialized 2792 byte log entry
0709 12:25:08.865028 (+253587us) spinlock_profiling.cc:243] Waited 6.61 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ff75680ac 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b
0709 12:25:11.333121 (+2468093us) spinlock_profiling.cc:243] Waited 5.62 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:11.536988 (+203867us) op_driver.cc:464] REPLICATION: finished
0709 12:25:11.574723 (+ 37735us) write_op.cc:301] APPLY: starting
0709 12:25:11.574820 (+    97us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:11.575029 (+   209us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:11.575070 (+    41us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:11.576204 (+  1134us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:11.576551 (+   347us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0709 12:25:11.576593 (+    42us) write_op.cc:312] APPLY: finished
0709 12:25:11.576904 (+   311us) log.cc:844] Serialized 59 byte log entry
0709 12:25:11.577095 (+   191us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:11.577212 (+   117us) write_op.cc:454] Released schema lock
0709 12:25:11.577273 (+    61us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:11.620853 (+ 43580us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:11.630968 (+ 10115us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":34959,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":37454,"apply.run_cpu_time_us":2956,"apply.run_wall_time_us":6106,"num_ops":7,"prepare.queue_time_us":82521,"prepare.run_cpu_time_us":70346,"prepare.run_wall_time_us":1514342,"raft.queue_time_us":169327,"raft.run_cpu_time_us":12916,"raft.run_wall_time_us":144533,"replication_time_us":3150466,"spinlock_wait_cycles":28133504,"thread_start_us":611040,"threads_started":4,"wal-append.queue_time_us":483297,"wal-append.run_cpu_time_us":45077,"wal-append.run_wall_time_us":3460460}]]}
I20260709 12:25:18.981743  7393 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:18.989871  5602 connection.cc:256] RPC call timeout handler was delayed by -2.2818504319998283s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17010000000000003s for a response
I20260709 12:25:18.968400  5706 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:18.966080  5710 raft_consensus.cc:2376] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7. Re-sending same reply.
I20260709 12:25:18.993960  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:18.992548  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:19.059736  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:19.060503  5706 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:18.998387  7900 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:18.958938  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1561) took 1325 ms. Trace:
I20260709 12:25:18.987221  5458 catalog_manager.cc:5697] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 2 to 3, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 3 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:19.062175  6276 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:19.062605  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:19.063380  5458 catalog_manager.cc:5697] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 3 to 4, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 4 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:25:19.064193  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:19.064854  5635 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:19.065762  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:19.065398  5458 catalog_manager.cc:5697] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 1 to 3, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 3 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:19.066411  5635 raft_consensus.cc:2368] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4.
I20260709 12:25:19.067237  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:19.067792  5635 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:19.067312  5458 catalog_manager.cc:5697] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:25:18.897853  7957 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:19.075791  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:19.073184  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:19.078403  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:19.077762  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:19.078670  5631 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:19.084921  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:19.079181  5559 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:19.087519  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:19.088501  5710 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:19.089956  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:19.090833  5706 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:19.069064  7963 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:19.092839  7963 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:19.093842  7963 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:19.095459  7963 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:19.096694  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:19.097375  5629 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:19.098363  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:19.058894  7858 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:19.105798  7858 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:18.932947  7761 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:19.117242  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:19.125392  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:19.126221  5703 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:19.068929  7745 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:19.134815  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:19.138304  5707 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:19.139212  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1589) took 1947 ms. Trace:
I20260709 12:25:19.139679  5558 rpcz_store.cc:276] 0709 12:25:17.191724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.100936 (+909212us) service_pool.cc:224] Handling call
0709 12:25:18.132981 (+ 32045us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.894543 (+761562us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.138994 (+244451us) spinlock_profiling.cc:243] Waited 716 ms on lock 0x7b54000ea1d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:19.139147 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1646640384}
W20260709 12:25:19.140261  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.143640  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:19.146158  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 271937us
W20260709 12:25:19.222790  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 59038.7us
I20260709 12:25:19.159169  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1860) took 1135 ms. Trace:
I20260709 12:25:19.245769  5632 rpcz_store.cc:276] 0709 12:25:18.023722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.025841 (+  2119us) service_pool.cc:224] Handling call
0709 12:25:19.141472 (+1115631us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b5400438d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:19.159073 (+ 17601us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1979570048}
I20260709 12:25:19.246117  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:19.120872  8019 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:25:19.252745  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 169329us
I20260709 12:25:19.264598  5707 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:19.266613  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:19.181458  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:19.299628  5635 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:19.251746  8006 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:19.263445  7588 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:19.137024  7957 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:19.069342  5552 rpcz_store.cc:276] 0709 12:25:17.623608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.919368 (+295760us) service_pool.cc:224] Handling call
0709 12:25:18.948971 (+1029603us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:19.312139  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 227357us
I20260709 12:25:19.145008  7761 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:19.273144  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.274577  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:19.268252  5458 catalog_manager.cc:5697] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 2 to 3, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 3 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:25:19.267544  5558 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:19.121229  8008 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.143s
I20260709 12:25:19.282174  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1864) took 1201 ms. Trace:
I20260709 12:25:19.282574  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:19.274122  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:19.119621  7900 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:19.329849  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:19.273437  7959 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:19.120196  7858 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:19.159682  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:19.314996  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:19.337172  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63096.8us
W20260709 12:25:19.318572  5602 connection.cc:256] RPC call timeout handler was delayed by -2.1270504320000327s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18730000000000002s for a response
W20260709 12:25:19.331944  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49554.1us
W20260709 12:25:19.329030  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 338566us
I20260709 12:25:19.332702  5707 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:19.334367  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:19.342846  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1491) took 1193 ms. Trace:
I20260709 12:25:19.343312  5635 rpcz_store.cc:276] 0709 12:25:18.149036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.318977 (+1169941us) service_pool.cc:224] Handling call
0709 12:25:19.342763 (+ 23786us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:19.345965  5706 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:19.347800  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:19.348598  5558 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:19.349647  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:19.351435  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.357012  5559 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 4: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:19.248495  7745 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:19.327356  5630 rpcz_store.cc:276] 0709 12:25:18.065792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.065909 (+   117us) service_pool.cc:224] Handling call
0709 12:25:18.066129 (+   220us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.069633 (+  3504us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:25:18.802600 (+732967us) log.cc:844] Serialized 14 byte log entry
0709 12:25:19.266980 (+464380us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:19.267062 (+    82us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:19.267137 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.267310 (+   173us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.267454 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":733597,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":6128,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":8930,"thread_start_us":1099877,"threads_started":2}
I20260709 12:25:19.360745  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:19.372566  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.377379  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:19.382897  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:19.384225  5631 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:19.325340  8030 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:19.327584  7954 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:19.424890  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:19.272935  7832 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:19.431406  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:19.425823  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:19.432564  5559 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:19.336835  7959 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:19.425246  5458 catalog_manager.cc:5697] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 2 to 3, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 3 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:19.357959  7900 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:19.320073  8006 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:19.441455  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 64710.7us
W20260709 12:25:19.451728  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.457060  5558 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:19.465273  5630 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260709 12:25:19.486861  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.605525  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:19.605525  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:19.471932  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 565) took 1269 ms. Trace:
I20260709 12:25:19.646235  5707 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:19.467178  5632 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:25:19.641959  5602 connection.cc:256] RPC call timeout handler was delayed by -1.8840504320000946s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2143s for a response
I20260709 12:25:19.474443  5458 catalog_manager.cc:5697] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 1 to 2, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 2 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
W20260709 12:25:19.651129  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 516390us
I20260709 12:25:19.642578  5702 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
W20260709 12:25:19.650970  5602 connection.cc:256] RPC call timeout handler was delayed by -1.8619033390000368s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30970000000000003s for a response
W20260709 12:25:19.652706  5602 connection.cc:256] RPC call timeout handler was delayed by -1.8390504320000218s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21930000000000002s for a response
I20260709 12:25:19.651180  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1488) took 1503 ms. Trace:
I20260709 12:25:19.387450  8034 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:19.640875  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:19.670995  5631 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:19.659664  5632 rpcz_store.cc:276] 0709 12:25:18.148113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.432469 (+1284356us) service_pool.cc:224] Handling call
0709 12:25:19.466833 (+ 34364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.650816 (+183983us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.651099 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:19.543428  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 193862us
I20260709 12:25:19.481115  5703 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:25:19.646540  6280 rpcz_store.cc:276] 0709 12:25:18.202246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.228597 (+ 26351us) service_pool.cc:224] Handling call
0709 12:25:18.228968 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.229285 (+   317us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:25:18.894751 (+665466us) log.cc:844] Serialized 14 byte log entry
0709 12:25:19.464997 (+570246us) log.cc:844] Serialized 14 byte log entry
0709 12:25:19.471457 (+  6460us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:19.471524 (+    67us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:19.471603 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.471689 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.471852 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61103,"prepare.queue_time_us":683782,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":2270,"thread_start_us":1143062,"threads_started":3,"wal-append.queue_time_us":647789}
W20260709 12:25:19.665087  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 222547us
W20260709 12:25:19.682334  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 340638us
I20260709 12:25:19.677021  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1486) took 1529 ms. Trace:
I20260709 12:25:19.431423  8003 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:19.434774  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:19.686241  5558 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:25:19.687819  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 398) took 1415 ms. Trace:
I20260709 12:25:19.687870  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:19.703328  5707 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 2.
I20260709 12:25:19.703135  5632 rpcz_store.cc:276] 0709 12:25:18.272689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.685927 (+1413238us) service_pool.cc:224] Handling call
0709 12:25:19.686510 (+   583us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.687264 (+   754us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:19.687323 (+    59us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:19.687397 (+    74us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:19.687481 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.687605 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.687764 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:19.704514  5707 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:19.687409  5630 rpcz_store.cc:276] 0709 12:25:18.147601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.428753 (+1281152us) service_pool.cc:224] Handling call
0709 12:25:19.441520 (+ 12767us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.676730 (+235210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.676954 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:19.605401  8038 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 12:25:19.440763  7959 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:19.698927  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:19.766333  5661 heartbeater.cc:646] Failed to heartbeat to 127.4.65.62:38921 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.4.65.62:38921 timed out after 15.000s (SENT)
W20260709 12:25:19.766418  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 67562.6us
I20260709 12:25:19.767176  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:19.771680  8054 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.772339  8054 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:19.697429  8034 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:19.774266  8034 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:19.775135  8054 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:19.606204  7884 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.778419  7884 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:19.466670  8030 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.279s
I20260709 12:25:19.745349  8057 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:19.784691  8057 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:19.785073  8057 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:19.745126  8043 raft_consensus.cc:2764] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:25:19.786290  8043 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:25:19.681203  8008 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20260709 12:25:19.788237  7884 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:19.517181  7954 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:25:19.747690  8032 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.203s	user 0.000s	sys 0.000s
I20260709 12:25:19.748588  5458 catalog_manager.cc:5697] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 2 to 3, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 3 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:19.790364  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1730) took 1719 ms. Trace:
I20260709 12:25:19.790927  5458 catalog_manager.cc:5697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 2 to 3, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 3 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:25:19.790809  5701 rpcz_store.cc:276] 0709 12:25:18.073657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.474465 (+400808us) service_pool.cc:224] Handling call
0709 12:25:18.697035 (+222570us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.836090 (+139055us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.840394 (+  4304us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:18.908231 (+ 67837us) log.cc:844] Serialized 26 byte log entry
0709 12:25:19.312213 (+403982us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:19.314434 (+  2221us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.326938 (+ 12504us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.790074 (+463136us) raft_consensus.cc:1685] finished
0709 12:25:19.790110 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.790292 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19180,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":16097,"thread_start_us":300728,"threads_started":1,"wal-append.queue_time_us":450689}
I20260709 12:25:19.749917  5552 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:19.792320  5458 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 1 to 4, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 4 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:19.794032  7963 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.698s	user 0.012s	sys 0.001s
I20260709 12:25:19.794070  5458 catalog_manager.cc:5697] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 1 to 2, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 2 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:19.750288  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1558) took 2082 ms. Trace:
I20260709 12:25:19.796175  5458 catalog_manager.cc:5697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf reported cstate change: config changed from index 3 to 4, term changed from 1 to 3, VOTER 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) evicted. New cstate: current_term: 3 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:19.796599  5558 rpcz_store.cc:276] 0709 12:25:17.622799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.685386 (+2062587us) service_pool.cc:224] Handling call
0709 12:25:19.685807 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.704622 (+ 18815us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.704808 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:19.797919  5458 catalog_manager.cc:5697] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 2 to 3, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 3 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:25:19.798595  5558 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:19.735265  7745 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:19.753013  8037 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.240s	user 0.000s	sys 0.000s
I20260709 12:25:19.823624  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 358) took 2506 ms. Trace:
I20260709 12:25:19.757673  8006 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:19.758033  7729 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.767s	user 0.007s	sys 0.002s
W20260709 12:25:19.739634  6274 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.790s	user 0.007s	sys 0.000s
I20260709 12:25:19.688504  8036 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:19.685173  7393 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:25:19.759462  7929 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.797s	user 0.016s	sys 0.004s
I20260709 12:25:19.743070  7900 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:19.763733  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:19.765487  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 400) took 1472 ms. Trace:
W20260709 12:25:19.780781  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.859673  5630 rpcz_store.cc:276] 0709 12:25:18.292838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.705188 (+1412350us) service_pool.cc:224] Handling call
0709 12:25:19.764739 (+ 59551us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.765008 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:19.765049 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:19.765109 (+    60us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:19.765191 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.765266 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.765406 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:19.745563  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:19.861632  5559 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:19.488904  8041 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:19.681106  7922 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:25:19.737335  7612 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.765s	user 0.009s	sys 0.001s
I20260709 12:25:19.789155  7954 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:25:19.748729  8042 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
W20260709 12:25:19.799036  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 356854us
W20260709 12:25:19.804586  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 82821.8us
I20260709 12:25:19.805295  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1562) took 2175 ms. Trace:
I20260709 12:25:19.866854  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:19.867257  5558 rpcz_store.cc:276] 0709 12:25:17.623864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.798078 (+2174214us) service_pool.cc:224] Handling call
0709 12:25:19.798436 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.799039 (+   603us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.799286 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:19.867825  5630 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260709 12:25:19.807085  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.868786  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 401) took 1575 ms. Trace:
I20260709 12:25:19.606767  8047 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:19.824057  5554 rpcz_store.cc:276] 0709 12:25:17.317046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.841914 (+1524868us) service_pool.cc:224] Handling call
0709 12:25:18.857003 (+ 15089us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.857437 (+   434us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:25:19.123478 (+266041us) log.cc:844] Serialized 14 byte log entry
0709 12:25:19.823201 (+699723us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:19.823276 (+    75us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:19.823338 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.823403 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.823567 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40529,"prepare.queue_time_us":487149,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":3952,"raft.queue_time_us":308293,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":102,"thread_start_us":826209,"threads_started":3,"wal-append.queue_time_us":645197}
W20260709 12:25:19.718146  8025 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.002s	sys 0.011s
I20260709 12:25:19.862555  7745 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:19.869253  5630 rpcz_store.cc:276] 0709 12:25:18.293185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.867141 (+1573956us) service_pool.cc:224] Handling call
0709 12:25:19.867488 (+   347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:19.868427 (+   939us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.868702 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:19.870544  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 103459us
I20260709 12:25:19.879033  6279 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
W20260709 12:25:19.873591  8025 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.002s	sys 0.015s
W20260709 12:25:19.884979  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 179420us
I20260709 12:25:19.853101  6274 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:19.853036  8061 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:19.887419  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 402) took 1594 ms. Trace:
I20260709 12:25:19.865126  8041 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:25:19.887881  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:19.896080  6274 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 490) took 1800 ms (client timeout 1606 ms). Trace:
W20260709 12:25:19.897817  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.890420  5630 rpcz_store.cc:276] 0709 12:25:18.293469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.884262 (+1590793us) service_pool.cc:224] Handling call
0709 12:25:19.886182 (+  1920us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.886558 (+   376us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:19.887068 (+   510us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:19.887127 (+    59us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:19.887175 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.887229 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.887357 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:19.889063  5554 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20260709 12:25:19.888192  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:19.948982  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72555.6us
W20260709 12:25:19.885259  7957 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.563s	user 0.005s	sys 0.004s
I20260709 12:25:20.012926  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 362) took 2056 ms. Trace:
I20260709 12:25:20.013527  5554 rpcz_store.cc:276] 0709 12:25:17.956656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.874241 (+1917585us) service_pool.cc:224] Handling call
0709 12:25:19.888766 (+ 14525us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:20.012343 (+123577us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.012858 (+   515us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:20.018471  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:20.019899  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:19.871361  8047 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:19.873963  7612 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:19.954280  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 85021.7us
W20260709 12:25:19.960207  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:20.031493  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 164727us
I20260709 12:25:19.883116  8008 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:19.885738  7954 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:19.863538  7393 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:19.844199  8057 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:19.875633  8052 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.179s
I20260709 12:25:19.875488  7922 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:19.865396  8036 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:20.049978  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:20.328511  5676 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:25:20.307540  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 440510us
W20260709 12:25:20.330348  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:20.331621  5552 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.539s	user 0.006s	sys 0.000s
I20260709 12:25:20.332249  5552 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:20.333333  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1601) took 1365 ms. Trace:
I20260709 12:25:20.334234  5552 rpcz_store.cc:276] 0709 12:25:18.968186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.376037 (+407851us) service_pool.cc:224] Handling call
0709 12:25:20.333250 (+957213us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:20.331451  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 310127us
I20260709 12:25:20.183404  6280 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:20.336324  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:19.887070  8050 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:19.991789  8042 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
W20260709 12:25:20.337177  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 573514us
W20260709 12:25:20.308858  7761 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.985s	user 0.011s	sys 0.000s
I20260709 12:25:20.363299  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:20.367717  5554 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:25:19.989702  7963 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.388947  7963 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:20.389932  7393 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:20.393383  8071 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:25:20.395462  8060 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.382s	user 0.001s	sys 0.000s
W20260709 12:25:20.395660  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.226680  7959 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:20.398416  8025 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:20.336102  8047 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:20.260627  8065 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:20.400607  8065 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:20.401425  8065 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:20.402298  8052 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:20.262997  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 558) took 1461 ms. Trace:
I20260709 12:25:20.372539  5710 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:25:20.385063  5552 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:20.385512  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 364) took 2428 ms. Trace:
I20260709 12:25:20.403687  5554 rpcz_store.cc:276] 0709 12:25:17.957140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.366905 (+2409765us) service_pool.cc:224] Handling call
0709 12:25:20.367399 (+   494us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:20.385144 (+ 17745us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.385439 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:20.404584  5635 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:25:20.363782  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:20.349146  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:20.402462  8065 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:20.405560  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56540.9us
W20260709 12:25:20.407037  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.988942  8055 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:25:20.416496  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 85414.5us
I20260709 12:25:20.417375  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:20.050071  8003 raft_consensus.cc:2764] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:25:19.993093  8041 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:20.218734  8072 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:20.421208  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:20.423478  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 535381us
W20260709 12:25:20.445919  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 82183.6us
I20260709 12:25:20.448187  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:20.391978  7963 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:20.418107  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:20.445912  5552 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:25:20.184008  8030 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.234s
W20260709 12:25:20.454193  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 82696.6us
W20260709 12:25:19.992830  6274 rpcz_store.cc:269] 0709 12:25:18.095298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.100127 (+  4829us) service_pool.cc:224] Handling call
0709 12:25:18.351008 (+250881us) spinlock_profiling.cc:243] Waited 54.1 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754de9c 00007f4ff754da5b 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:19.895899 (+1544891us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":124413184}
I20260709 12:25:20.458047  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 365) took 2492 ms. Trace:
I20260709 12:25:20.332476  8058 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.149s
I20260709 12:25:20.362995  8008 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:20.415740  7959 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:20.462960  5552 rpcz_store.cc:276] 0709 12:25:17.957383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.445268 (+2487885us) service_pool.cc:224] Handling call
0709 12:25:20.445673 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:20.449761 (+  4088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.450107 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:20.403348  8052 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.054s
I20260709 12:25:20.419307  7957 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:20.469324  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:20.450862  7393 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:20.451017  7761 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:20.488642  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:20.467718  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:20.490343  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.371073  8050 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:20.464643  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.490660  8068 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:20.490538  8039 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 12:25:20.461988  8062 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:20.491659  8057 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:20.468658  7963 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:20.492690  8069 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:20.493119  7954 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:20.496495  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:20.497300  8053 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:25:20.502072  7612 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:20.457475  8072 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:20.446187  8066 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:25:20.505703  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 169474us
I20260709 12:25:20.523655  8074 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:20.526403  8030 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:20.520267  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 557) took 2280 ms. Trace:
I20260709 12:25:20.467933  8061 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:20.505926  7393 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:20.533937  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45416.2us
I20260709 12:25:20.553786  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:20.507720  8006 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:20.459409  8063 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
W20260709 12:25:20.538512  5675 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.629s (SENT)
W20260709 12:25:20.538686  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 71081.7us
I20260709 12:25:20.569895  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:20.539647  8049 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.532711  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:20.505579  8003 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 12:25:20.507921  8067 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:25:20.519906  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 68314.7us
I20260709 12:25:20.525458  8062 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:20.507504  7957 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:20.572398  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:20.542652  7761 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:20.528405  8068 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:20.497334  8052 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.042s
I20260709 12:25:20.567658  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:20.531993  8069 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.536029  8059 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:20.549321  6277 rpcz_store.cc:276] 0709 12:25:18.239640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.267168 (+ 27528us) service_pool.cc:224] Handling call
0709 12:25:18.329845 (+ 62677us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.462447 (+132602us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:25:19.059372 (+596925us) log.cc:844] Serialized 14 byte log entry
0709 12:25:19.393404 (+334032us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:19.393989 (+   585us) log.cc:844] Serialized 26 byte log entry
0709 12:25:19.394161 (+   172us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:19.394215 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.514052 (+1119837us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b5400771d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:20.514120 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:20.514187 (+    67us) raft_consensus.cc:1685] finished
0709 12:25:20.520056 (+  5869us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.520210 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18765,"prepare.queue_time_us":479690,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":304522,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":3786,"spinlock_wait_cycles":286056832,"thread_start_us":494645,"threads_started":3,"wal-append.queue_time_us":530346}
I20260709 12:25:20.560825  8055 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:20.601785  8055 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:20.603149  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 494) took 1458 ms. Trace:
I20260709 12:25:20.603672  8061 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:20.608984  6272 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:25:20.578526  8062 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:20.548996  8066 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.576476  8057 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:20.548858  8074 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.582049  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:20.562728  7393 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:20.579525  8049 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:20.625723  7745 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.747s	user 0.007s	sys 0.005s
I20260709 12:25:20.626207  7745 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:20.626904  7745 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.627372  7745 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:20.628093  8049 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:20.549144  8030 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:20.590798  8008 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:20.591346  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:20.563184  8006 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:20.631692  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 165317us
I20260709 12:25:20.586571  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:20.589643  8069 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:20.584648  8067 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:20.594400  8078 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.012s
I20260709 12:25:20.595042  8003 raft_consensus.cc:2764] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:25:20.595263  8076 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:20.595073  8077 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:20.612905  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:20.403059  6273 rpcz_store.cc:276] 0709 12:25:18.799540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.814385 (+ 14845us) service_pool.cc:224] Handling call
0709 12:25:18.886324 (+ 71939us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.903486 (+ 17162us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:25:19.291735 (+388249us) log.cc:844] Serialized 14 byte log entry
0709 12:25:19.949686 (+657951us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.019185 (+ 69499us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:20.054741 (+ 35556us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.221567 (+166826us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.261027 (+ 39460us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":146969,"prepare.queue_time_us":69665,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":229330,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":170880,"thread_start_us":696037,"threads_started":2,"wal-append.queue_time_us":555613}
I20260709 12:25:20.579962  8058 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:20.629873  8074 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:20.634035  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:20.592840  8056 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.611424  7922 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:20.634588  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1505) took 2166 ms. Trace:
W20260709 12:25:20.641408  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54925.6us
I20260709 12:25:20.630244  8066 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:20.633294  8006 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:20.668943  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 86991.5us
I20260709 12:25:20.661432  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:20.663543  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:20.640720  8052 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.663944  5630 rpcz_store.cc:276] 0709 12:25:18.467824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.993666 (+1525842us) service_pool.cc:224] Handling call
0709 12:25:20.327226 (+333560us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.369242 (+ 42016us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:25:20.514216 (+144974us) log.cc:844] Serialized 14 byte log entry
0709 12:25:20.634212 (+119996us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.634265 (+    53us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:20.634336 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.634411 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.634532 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":259789,"raft.queue_time_us":22375,"thread_start_us":196525,"threads_started":2}
I20260709 12:25:20.653290  8008 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:20.685360  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:20.656387  8057 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:20.658843  8059 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:20.665210  7588 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:20.694571  5634 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
W20260709 12:25:20.693140  5675 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.732s (SENT)
I20260709 12:25:20.672591  7922 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:20.699879  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1508) took 1959 ms. Trace:
I20260709 12:25:20.697139  8085 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:20.702234  8085 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:20.705535  8085 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:20.706204  8064 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.706735  8064 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:20.713059  5630 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:25:20.713862  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1507) took 2055 ms. Trace:
I20260709 12:25:20.714334  5630 rpcz_store.cc:276] 0709 12:25:18.658914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.700129 (+2041215us) service_pool.cc:224] Handling call
0709 12:25:20.704168 (+  4039us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.713550 (+  9382us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.713795 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:20.701520  5634 rpcz_store.cc:276] 0709 12:25:18.740947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.694002 (+1953055us) service_pool.cc:224] Handling call
0709 12:25:20.694384 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.699544 (+  5160us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.699804 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:20.687662  8003 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:20.698325  7588 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.690770  8008 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.649326  8072 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:20.708196  6274 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:20.730930  7745 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:20.698611  8059 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.732318  8059 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:20.700907  7922 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:20.733413  7922 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:20.733852  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:20.734651  5561 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:20.735769  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:20.737746  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:20.738641  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 498) took 1478 ms. Trace:
I20260709 12:25:20.745333  8067 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:20.745720  8059 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:20.748306  8059 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:20.748684  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 65207.6us
I20260709 12:25:20.749550  5602 leader_election.cc:400] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4.
I20260709 12:25:20.750444  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:20.751227  7745 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:20.663271  8062 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:20.760174  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:20.739099  6278 rpcz_store.cc:276] 0709 12:25:19.259786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.289973 (+ 30187us) service_pool.cc:224] Handling call
0709 12:25:20.737581 (+1447608us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:20.738576 (+   995us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25891072}
I20260709 12:25:20.736339  5561 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:20.610845  8068 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:20.793311  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.797303  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:20.798256  5561 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:20.798938  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1597) took 2865 ms. Trace:
I20260709 12:25:20.800038  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:25:20.785105  6276 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 567) took 2524 ms (client timeout 1983 ms). Trace:
W20260709 12:25:20.802330  6276 rpcz_store.cc:269] 0709 12:25:18.246962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.247149 (+   187us) service_pool.cc:224] Handling call
0709 12:25:20.770997 (+2523848us) spinlock_profiling.cc:243] Waited 89.1 ms on lock 0x7b54005eef54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:20.771137 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":204817664}
I20260709 12:25:20.807648  5634 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:20.808390  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1519) took 1837 ms. Trace:
I20260709 12:25:20.808750  5634 rpcz_store.cc:276] 0709 12:25:18.970708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.723610 (+1752902us) service_pool.cc:224] Handling call
0709 12:25:20.807380 (+ 83770us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.808123 (+   743us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.808309 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:20.793891  8053 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:20.810405  8053 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.810743  8053 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:20.811614  7967 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.667912  8056 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:20.759598  8008 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:20.813875  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1511) took 1941 ms. Trace:
I20260709 12:25:20.683162  8052 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:20.818198  5630 rpcz_store.cc:276] 0709 12:25:18.872778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.715152 (+1842374us) service_pool.cc:224] Handling call
0709 12:25:20.715523 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.813479 (+ 97956us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:20.813514 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.813564 (+    50us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:20.813616 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.813680 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.813815 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:20.727756  7588 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:20.801590  5558 rpcz_store.cc:276] 0709 12:25:17.933765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.873825 (+1940060us) service_pool.cc:224] Handling call
0709 12:25:19.884948 (+ 11123us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:20.307246 (+422298us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:25:20.366841 (+ 59595us) log.cc:844] Serialized 14 byte log entry
0709 12:25:20.641780 (+274939us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:20.657691 (+ 15911us) log.cc:844] Serialized 52 byte log entry
0709 12:25:20.657872 (+   181us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:20.658091 (+   219us) log.cc:844] Serialized 14 byte log entry
0709 12:25:20.797783 (+139692us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.798550 (+   767us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:20.798712 (+   162us) raft_consensus.cc:1685] finished
0709 12:25:20.798733 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.798883 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":97891,"prepare.queue_time_us":158244,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":1562,"raft.queue_time_us":563292,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":201,"spinlock_wait_cycles":8256768,"thread_start_us":700504,"threads_started":4,"wal-append.queue_time_us":188430}
I20260709 12:25:20.734654  7922 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:20.694185  8030 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:20.759452  8003 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:20.812165  7967 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:20.827090  5628 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:20.827235  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:20.839917  5558 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:20.833194  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:20.840826  5630 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:20.832548  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:20.804400  8076 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:25:20.867861  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 196791us
I20260709 12:25:20.869308  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:20.815207  8084 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.815614  8078 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.816581  8058 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:20.874864  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:20.878404  5628 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
W20260709 12:25:20.879413  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 208328us
I20260709 12:25:20.879648  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:20.864324  8050 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:20.940816  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 124948us
I20260709 12:25:20.835640  8088 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.910471  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:20.878096  8058 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.912297  5628 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:20.818003  8047 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:20.949590  8047 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:20.875912  7959 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
W20260709 12:25:20.919778  5636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1512) took 2027 ms (client timeout 1615 ms). Trace:
I20260709 12:25:20.921068  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 363) took 2964 ms. Trace:
W20260709 12:25:20.951261  5636 rpcz_store.cc:269] 0709 12:25:18.892666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.917280 (+ 24614us) service_pool.cc:224] Handling call
0709 12:25:20.919576 (+2002296us) spinlock_profiling.cc:243] Waited 20.5 ms on lock 0x7b54001ce8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:20.919691 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47175040}
I20260709 12:25:20.951347  5553 rpcz_store.cc:276] 0709 12:25:17.956902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.366393 (+2409491us) service_pool.cc:224] Handling call
0709 12:25:20.396312 (+ 29919us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.542510 (+146198us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:20.542541 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.542585 (+    44us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:20.542634 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.920759 (+378125us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b540023ffd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:20.920823 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.920988 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":335645440}
I20260709 12:25:20.953191  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:20.953850  5553 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:20.954857  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:20.955123  8022 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 2.211s	user 0.005s	sys 0.015s
W20260709 12:25:20.922495  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:20.926735  8090 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 12:25:20.908183  8092 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:20.957247  8092 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:20.958276  7921 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:25:20.879896  8078 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:20.960610  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 359) took 3617 ms. Trace:
I20260709 12:25:20.961200  5557 rpcz_store.cc:276] 0709 12:25:17.343395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.858988 (+1515593us) service_pool.cc:224] Handling call
0709 12:25:18.935970 (+ 76982us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.953894 (+ 17924us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.960224 (+2006330us) spinlock_profiling.cc:243] Waited 63.1 ms on lock 0x7b54000ebfd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:20.960508 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":145101056}
I20260709 12:25:20.932662  8003 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:20.610466  6275 rpcz_store.cc:276] 0709 12:25:19.147155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.159288 (+ 12133us) service_pool.cc:224] Handling call
0709 12:25:19.314225 (+154937us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:19.799077 (+484852us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:19.805800 (+  6723us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:19.885499 (+ 79699us) log.cc:844] Serialized 26 byte log entry
0709 12:25:20.487568 (+602069us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:20.492662 (+  5094us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.492739 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:20.602844 (+110105us) raft_consensus.cc:1685] finished
0709 12:25:20.602864 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.603077 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":185837,"raft.queue_time_us":448717,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":22383,"thread_start_us":531801,"threads_started":2,"wal-append.queue_time_us":675028}
I20260709 12:25:20.910259  7920 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.969283  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:20.969604  8093 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.945022  8063 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 12:25:20.970283  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 497) took 1721 ms. Trace:
I20260709 12:25:20.970124  8093 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:20.970686  6271 rpcz_store.cc:276] 0709 12:25:19.248604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.259470 (+ 10866us) service_pool.cc:224] Handling call
0709 12:25:20.969169 (+1709699us) spinlock_profiling.cc:243] Waited 2.06 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754deee 000000000036eb39 00007f4fff5c11e0 00007f4fff68a0c4 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:20.970214 (+  1045us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4739328}
I20260709 12:25:20.971732  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:20.973230  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 399) took 2700 ms. Trace:
I20260709 12:25:20.973603  5632 rpcz_store.cc:276] 0709 12:25:18.274473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.704203 (+1429730us) service_pool.cc:224] Handling call
0709 12:25:19.704442 (+   239us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.378943 (+674501us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:20.379035 (+    92us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.379111 (+    76us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:20.379183 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.973008 (+593825us) spinlock_profiling.cc:243] Waited 674 ms on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:20.973062 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.973174 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1550243200}
I20260709 12:25:20.963289  5553 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:20.964144  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:20.952298  5561 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:20.946552  8064 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:20.969202  7920 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:20.939270  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 277908us
I20260709 12:25:20.968659  8078 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:20.962575  8058 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:20.880015  8084 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:21.001601  8050 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
W20260709 12:25:21.004051  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 62392.2us
I20260709 12:25:21.005367  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:21.020659  5561 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:21.022648  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:25:21.006175  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 64065.3us
W20260709 12:25:21.011070  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.960144  8092 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:20.975418  8093 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:21.025666  5526 connection.cc:256] RPC call timeout handler was delayed by -1.5589853689998563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1673s for a response
I20260709 12:25:21.026027  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:21.028016  5526 connection.cc:256] RPC call timeout handler was delayed by -1.27998536899986s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19830000000000003s for a response
I20260709 12:25:20.991148  8067 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.028887  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:20.943636  8053 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:20.979082  7921 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260709 12:25:21.030985  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:21.031713  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:21.032841  8039 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:21.033466  7967 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:21.032043  8078 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:21.041123  8078 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:21.041182  7957 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:21.041649  8078 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:21.033803  8101 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:21.044693  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1742) took 4188 ms. Trace:
I20260709 12:25:21.045357  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:21.045725  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:21.046047  5705 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:21.046276  5709 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:21.047327  8072 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:21.048076  7853 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:21.048619  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:21.045171  5708 rpcz_store.cc:276] 0709 12:25:16.855899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.920956 (+1065057us) service_pool.cc:224] Handling call
0709 12:25:17.934051 (+ 13095us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.966641 (+1032590us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.044213 (+2077572us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400210f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:21.044610 (+   397us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2318069504}
I20260709 12:25:20.964498  8088 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:21.043771  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:21.050796  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 355) took 3799 ms. Trace:
W20260709 12:25:21.048098  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.052820  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:21.052748  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:21.038923  8008 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:21.053359  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.054052  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:21.037070  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:21.057057  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:21.051345  5555 rpcz_store.cc:276] 0709 12:25:17.251694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.600311 (+348617us) service_pool.cc:224] Handling call
0709 12:25:21.050600 (+3450289us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001af754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:21.050728 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3184665984}
I20260709 12:25:21.051980  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:21.014098  8047 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260709 12:25:21.070497  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:21.078351  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:21.015743  8064 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:25:21.065254  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:21.069523  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:21.086750  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:21.080189  5633 raft_consensus.cc:2376] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4. Re-sending same reply.
I20260709 12:25:21.096081  5555 raft_consensus.cc:2376] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4. Re-sending same reply.
I20260709 12:25:21.077306  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:21.036043  8093 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:21.100409  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:21.101598  6276 raft_consensus.cc:2410] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 12:25:21.073701  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:21.102402  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:21.102777  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:21.103319  5632 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:25:21.103577  5709 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:21.048667  8062 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:21.104246  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:21.067263  8055 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:21.106401  5701 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:21.021854  8069 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:21.107165  8099 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260709 12:25:21.107714  7920 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:21.165840  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:21.108177  8101 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:25:21.108425  8041 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.649s	user 0.001s	sys 0.010s
I20260709 12:25:21.084962  7921 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:21.109246  7922 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:21.109391  8088 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:21.109797  7745 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:21.110718  7967 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:21.112540  7853 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:21.113893  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:21.114750  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:21.115363  8006 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:21.115736  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:21.115388  8072 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:21.116236  8093 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.074252  8066 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:21.117620  8053 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:21.074347  8008 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:21.060959  7959 raft_consensus.cc:2764] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:25:21.154877  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:25:21.203282  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48523.4us
I20260709 12:25:21.203069  8093 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:21.155158  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:21.174978  7921 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:21.159226  5709 raft_consensus.cc:2410] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:21.184263  7745 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:21.204476  7921 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:21.204684  7745 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:21.205358  7967 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:21.144764  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:21.205897  7967 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:21.206373  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:21.206645  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:21.207095  5701 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:21.207284  5633 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:21.161907  5675 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:21.160774  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:21.207978  7853 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:25:21.162266  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 58138.3us
I20260709 12:25:21.209321  7853 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:21.212195  5628 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:21.162724  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:21.166400  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:21.212908  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.165380  8059 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:21.182827  8041 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:21.247747  8066 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:21.166400  5703 raft_consensus.cc:2376] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6. Re-sending same reply.
I20260709 12:25:21.080116  8085 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:21.076764  8056 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:21.076630  8036 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:21.119233  7957 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:21.174973  7761 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.591s	user 0.010s	sys 0.000s
I20260709 12:25:21.194600  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:21.181393  7922 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.168445  8101 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:21.188889  7920 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.014s
I20260709 12:25:21.201040  8006 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:21.206853  7967 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:21.175909  8052 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:21.213270  5702 raft_consensus.cc:2376] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6. Re-sending same reply.
I20260709 12:25:21.072682  8043 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:21.164492  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:21.168597  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:21.163769  8055 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:21.049351  8067 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:21.289547  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 85496.3us
W20260709 12:25:21.342308  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:21.289786  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 128105us
I20260709 12:25:21.371047  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 496) took 2220 ms. Trace:
W20260709 12:25:21.373970  5627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1484) took 3226 ms (client timeout 2071 ms). Trace:
W20260709 12:25:21.289672  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 134607us
W20260709 12:25:21.355698  6236 connection.cc:256] RPC call timeout handler was delayed by -1.9462188699999388s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20850000000000002s for a response
I20260709 12:25:21.350894  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:21.463238  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.360102  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 572) took 1657 ms. Trace:
W20260709 12:25:21.420852  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 366847us
I20260709 12:25:21.351881  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:21.461205  5627 rpcz_store.cc:269] 0709 12:25:18.147042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.148197 (+  1155us) service_pool.cc:224] Handling call
0709 12:25:21.373710 (+3225513us) spinlock_profiling.cc:243] Waited 838 ms on lock 0x7b54001cedd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:21.373886 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1927401728}
W20260709 12:25:21.288726  5560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1582) took 4124 ms (client timeout 2012 ms). Trace:
I20260709 12:25:21.250370  5552 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:21.373214  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:21.308804  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.311511  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:21.464476  5630 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.502s	user 0.011s	sys 0.000s
I20260709 12:25:21.490154  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:21.459278  5555 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:21.463943  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:21.538380  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:21.701678  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.702878  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:21.701355  6280 rpcz_store.cc:276] 0709 12:25:19.702808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.703058 (+   250us) service_pool.cc:224] Handling call
0709 12:25:19.822770 (+119712us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.339050 (+516280us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.359765 (+1020715us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b5400055854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:21.360022 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":522297216}
W20260709 12:25:21.489305  5710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1757) took 2550 ms (client timeout 2684 ms). Trace:
I20260709 12:25:21.699347  5630 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
W20260709 12:25:21.700837  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 239873us
W20260709 12:25:21.697427  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 487435us
I20260709 12:25:21.706036  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:21.704478  5710 rpcz_store.cc:269] 0709 12:25:18.938473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.120329 (+181856us) service_pool.cc:224] Handling call
0709 12:25:21.489078 (+2368749us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400075654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:21.489232 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2823351168}
I20260709 12:25:21.700958  5709 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
W20260709 12:25:21.699782  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.707926  5701 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
W20260709 12:25:21.377233  5635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1494) took 3227 ms (client timeout 2806 ms). Trace:
I20260709 12:25:21.460887  6279 rpcz_store.cc:276] 0709 12:25:19.150217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.243414 (+ 93197us) service_pool.cc:224] Handling call
0709 12:25:19.273284 (+ 29870us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.879471 (+606187us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.370521 (+1491050us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b54005910d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:21.370904 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1392652672}
W20260709 12:25:21.709842  5635 rpcz_store.cc:269] 0709 12:25:18.149889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.374783 (+1224894us) service_pool.cc:224] Handling call
0709 12:25:21.376916 (+2002133us) spinlock_profiling.cc:243] Waited 974 ms on lock 0x7b54001cf2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:21.377113 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2239705856}
W20260709 12:25:21.710227  5602 leader_election.cc:336] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.477s (SENT)
I20260709 12:25:21.709862  5708 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:21.710960  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:21.516144  5556 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:21.254896  5707 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:25:21.461222  5601 connection.cc:256] RPC call timeout handler was delayed by -1.9186537670002508s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16060000000000002s for a response
W20260709 12:25:21.712936  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:21.702215  5560 rpcz_store.cc:269] 0709 12:25:17.163921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.291664 (+127743us) service_pool.cc:224] Handling call
0709 12:25:21.288529 (+3996865us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b54001bec54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:21.288643 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3680821632}
I20260709 12:25:21.380038  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:21.309379  7393 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.687s	user 0.003s	sys 0.009s
I20260709 12:25:21.317214  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:21.372068  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:21.819597  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 58492.4us
I20260709 12:25:22.298434  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1525) took 1484 ms. Trace:
I20260709 12:25:21.358345  7996 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:21.864943  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:21.638989  8058 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.612s	user 0.010s	sys 0.002s
W20260709 12:25:22.297849  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 571112us
W20260709 12:25:22.299638  5556 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.588s	user 0.016s	sys 0.031s
I20260709 12:25:21.293857  8008 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:22.298882  5636 rpcz_store.cc:276] 0709 12:25:20.813754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.952583 (+138829us) service_pool.cc:224] Handling call
0709 12:25:20.952940 (+   357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.061731 (+108791us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.061758 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.062417 (+   659us) log.cc:844] Serialized 26 byte log entry
0709 12:25:21.592139 (+529722us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:21.592202 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.592315 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.289548 (+697233us) raft_consensus.cc:1685] finished
0709 12:25:22.289580 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.298358 (+  8778us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":61791,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":5143,"thread_start_us":581071,"threads_started":2,"wal-append.queue_time_us":1000856}
I20260709 12:25:21.761008  8056 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.820446  8068 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:21.487314  7748 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:22.301510  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:22.350764  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 49334.4us
I20260709 12:25:21.249810  8047 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.859973  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:21.465330  8101 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:21.860055  8079 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.858562  8003 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.485251  8052 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.920253  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:21.716338  8055 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:21.205813  8093 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:21.820772  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:21.832254  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:21.571076  7588 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.698s	user 0.012s	sys 0.000s
I20260709 12:25:21.820007  8066 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.467325  7957 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.373839  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:21.818783  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:22.382447  5707 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:22.383059  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1779) took 1208 ms. Trace:
I20260709 12:25:22.383481  5707 rpcz_store.cc:276] 0709 12:25:21.174548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.818443 (+643895us) service_pool.cc:224] Handling call
0709 12:25:22.382989 (+564546us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:21.368729  8084 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:21.637439  8074 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.946s	user 0.011s	sys 0.001s
I20260709 12:25:21.488300  7577 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:21.293812  8069 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:22.306653  8030 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:21.820150  8099 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:21.891185  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:22.389273  5627 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 2.
I20260709 12:25:21.820178  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:21.535252  8100 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.502s	user 0.000s	sys 0.004s
I20260709 12:25:21.374464  8034 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.309145  6273 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:21.462845  8085 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:21.918562  7612 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:21.461723  7922 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:21.466027  7761 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:21.692564  8053 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:21.248999  8041 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:22.057612  5599 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.500s (SENT)
W20260709 12:25:22.412237  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 354684us
W20260709 12:25:22.238353  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:22.140434  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:22.299511  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1531) took 1230 ms. Trace:
W20260709 12:25:22.333632  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 623385us
I20260709 12:25:22.343086  5556 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:21.876333  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:21.820487  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:22.282218  8113 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.558s	user 0.000s	sys 0.002s
I20260709 12:25:21.383380  8088 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:21.731817  8036 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:25:22.172187  8109 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.245s	user 0.001s	sys 0.000s
W20260709 12:25:22.156299  7921 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.934s	user 0.007s	sys 0.004s
I20260709 12:25:22.355453  6280 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:22.268221  8067 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:21.918977  8043 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.344566  7745 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.358425  8056 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.359334  7748 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:22.350536  7884 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.368911  5702 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:25:21.632143  8078 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.590s	user 0.000s	sys 0.009s
I20260709 12:25:21.911890  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:22.374716  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:22.381174  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1599) took 1239 ms. Trace:
I20260709 12:25:22.379367  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 563) took 1009 ms. Trace:
I20260709 12:25:22.391153  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:22.340180  7393 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:22.385874  8111 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.668s	user 0.004s	sys 0.002s
I20260709 12:25:22.373216  8068 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:22.352799  8058 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:22.400652  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:22.400988  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1584) took 1464 ms. Trace:
W20260709 12:25:22.400341  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.686s	user 0.005s	sys 0.013s
I20260709 12:25:22.403170  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 578) took 1302 ms. Trace:
I20260709 12:25:22.399259  8034 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.397753  7612 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:22.407172  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:22.408028  5561 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
W20260709 12:25:22.409718  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 57394.2us
I20260709 12:25:22.410916  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:22.416491  5555 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:22.414428  5628 rpcz_store.cc:276] 0709 12:25:21.069032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.069192 (+   160us) service_pool.cc:224] Handling call
0709 12:25:22.299058 (+1229866us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b540009ecd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:22.299441 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":262814592}
I20260709 12:25:22.463224  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1777) took 1289 ms. Trace:
W20260709 12:25:22.462558  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:22.514947  5525 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49030 dropped due to backpressure: service queue is full with 50 items
W20260709 12:25:22.542889  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:22.549227  5555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1598) took 1409 ms (client timeout 1755 ms). Trace:
W20260709 12:25:22.549635  5555 rpcz_store.cc:269] 0709 12:25:21.139841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.725416 (+585575us) service_pool.cc:224] Handling call
0709 12:25:22.549170 (+823754us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:22.550117  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.005s	sys 0.017s
W20260709 12:25:22.550706  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 844692us
I20260709 12:25:22.548811  5554 rpcz_store.cc:276] 0709 12:25:21.141438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.831879 (+690441us) service_pool.cc:224] Handling call
0709 12:25:22.381099 (+549220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.551458  6237 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:22.551676  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:22.551946  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:22.400144  8084 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:25:22.554463  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 73001.2us
I20260709 12:25:22.555336  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:22.555405  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1535) took 1468 ms. Trace:
I20260709 12:25:22.371316  8047 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:22.377724  8079 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.377547  8003 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.386340  7588 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:22.476673  7870 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 3.752s	user 0.049s	sys 0.053s
W20260709 12:25:22.478824  8112 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:25:22.479271  7921 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.477252  8036 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.592376  5705 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:25:22.386289  8066 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.388523  8074 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:22.464174  8088 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:22.548995  5560 rpcz_store.cc:276] 0709 12:25:20.936549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.731659 (+795110us) service_pool.cc:224] Handling call
0709 12:25:22.400910 (+669251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.597422  5558 raft_consensus.cc:2368] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4.
I20260709 12:25:22.555730  5627 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:22.557744  5707 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:22.557880  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1539) took 1087 ms. Trace:
I20260709 12:25:22.500747  8067 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:25:22.421150  7961 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 3.128s	user 0.017s	sys 0.055s
I20260709 12:25:22.552773  5710 rpcz_store.cc:276] 0709 12:25:21.174059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.819083 (+645024us) service_pool.cc:224] Handling call
0709 12:25:22.463118 (+644035us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.549021  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:22.568933  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:22.556427  5632 rpcz_store.cc:276] 0709 12:25:21.087282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.485468 (+398186us) service_pool.cc:224] Handling call
0709 12:25:21.490775 (+  5307us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:21.491288 (+   513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.491330 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:21.492173 (+   843us) log.cc:844] Serialized 52 byte log entry
0709 12:25:21.645894 (+153721us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:21.645953 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.646058 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.481448 (+835390us) raft_consensus.cc:1685] finished
0709 12:25:22.481479 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.555343 (+ 73864us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1722,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":123,"thread_start_us":153079,"threads_started":1,"wal-append.queue_time_us":230965}
W20260709 12:25:22.571327  5602 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.838s (SENT)
I20260709 12:25:22.572964  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1565) took 4922 ms. Trace:
I20260709 12:25:22.607016  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:22.499922  7957 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:22.590184  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:22.607121  5559 rpcz_store.cc:276] 0709 12:25:17.650106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.862640 (+2212534us) service_pool.cc:224] Handling call
0709 12:25:21.050793 (+1188153us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.051085 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.051113 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:21.051151 (+    38us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:21.051186 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.051238 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.572786 (+1521548us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54001af750. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:22.572901 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":668,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":23778,"spinlock_wait_cycles":2727224064}
W20260709 12:25:22.608536  5555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1589) took 1665 ms (client timeout 1966 ms). Trace:
I20260709 12:25:22.548532  6272 rpcz_store.cc:276] 0709 12:25:21.369898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.371636 (+  1738us) service_pool.cc:224] Handling call
0709 12:25:22.379281 (+1007645us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.553154  8092 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.529462  7748 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:22.554209  7996 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.595546  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:22.598201  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1768) took 1427 ms. Trace:
I20260709 12:25:22.598084  5636 rpcz_store.cc:276] 0709 12:25:21.470449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.354151 (+883702us) service_pool.cc:224] Handling call
0709 12:25:22.557822 (+203671us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.377173  8052 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:22.586839  7870 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.906s	user 0.050s	sys 0.053s
I20260709 12:25:22.633347  7745 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:22.634145  7612 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.635198  7612 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:22.636421  8092 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:22.637666  8052 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:22.571064  8051 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:22.644825  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49404.3us
I20260709 12:25:22.570912  7832 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:22.648234  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:22.648594  7832 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:22.649070  7832 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:22.649150  7870 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:22.649669  7870 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:22.649821  8079 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:22.650058  7870 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:22.650422  8079 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.650873  8079 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:22.592136  6274 raft_consensus.cc:2368] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5.
I20260709 12:25:22.590618  7967 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.660426  7967 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.611903  8067 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:22.571223  8121 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.662133  8121 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:22.662863  8067 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:22.663504  8067 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.578496  8104 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:22.664575  8104 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:22.664875  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1600) took 3697 ms. Trace:
I20260709 12:25:22.665062  8104 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:22.665292  5552 rpcz_store.cc:276] 0709 12:25:18.967900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.513467 (+1545567us) service_pool.cc:224] Handling call
0709 12:25:20.526219 (+ 12752us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.700325 (+1174106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.664658 (+964333us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b540016ba54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:22.664819 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1664847744}
I20260709 12:25:22.667114  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:22.667712  5552 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:22.669286  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 387) took 1618 ms. Trace:
I20260709 12:25:22.669767  5705 rpcz_store.cc:276] 0709 12:25:21.050986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.051435 (+   449us) service_pool.cc:224] Handling call
0709 12:25:21.051723 (+   288us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:22.594422 (+1542699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.668983 (+ 74561us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54002d0fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:22.669213 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3539222784}
I20260709 12:25:22.671341  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:22.608934  5555 rpcz_store.cc:269] 0709 12:25:20.943340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.564976 (+1621636us) service_pool.cc:224] Handling call
0709 12:25:22.608467 (+ 43491us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.591301  5634 raft_consensus.cc:2368] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5.
I20260709 12:25:22.659993  6277 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:25:22.674419  5634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1523) took 1861 ms (client timeout 1919 ms). Trace:
I20260709 12:25:22.672062  5705 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:22.675148  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1773) took 1502 ms. Trace:
W20260709 12:25:22.674782  5634 rpcz_store.cc:269] 0709 12:25:20.813197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.813374 (+   177us) service_pool.cc:224] Handling call
0709 12:25:22.674235 (+1860861us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b54003afa54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:22.674361 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3970344704}
I20260709 12:25:22.675388  5705 rpcz_store.cc:276] 0709 12:25:21.173132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.671023 (+1497891us) service_pool.cc:224] Handling call
0709 12:25:22.675108 (+  4085us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.588030  8093 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.613576  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:22.614513  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45651.3us
I20260709 12:25:22.591441  8084 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:22.527515  7884 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.550587  8078 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.500993  8043 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.547868  6273 rpcz_store.cc:276] 0709 12:25:21.100787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.160024 (+ 59237us) service_pool.cc:224] Handling call
0709 12:25:22.403093 (+1243069us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10496}
I20260709 12:25:22.619657  5557 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:22.621727  5707 rpcz_store.cc:276] 0709 12:25:21.171135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.390801 (+1219666us) service_pool.cc:224] Handling call
0709 12:25:22.598141 (+207340us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.625422  8066 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:22.705729  7922 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.707145  8078 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:22.624527  8034 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:22.605451  7961 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.323s	user 0.018s	sys 0.055s
I20260709 12:25:22.625384  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:22.710448  5710 raft_consensus.cc:2376] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4. Re-sending same reply.
W20260709 12:25:22.711001  5710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1770) took 1538 ms (client timeout 1935 ms). Trace:
W20260709 12:25:22.711376  5710 rpcz_store.cc:269] 0709 12:25:21.172317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.613130 (+1440813us) service_pool.cc:224] Handling call
0709 12:25:22.710924 (+ 97794us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.627360  8074 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:22.627744  6271 raft_consensus.cc:2410] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:25:22.629108  8056 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:22.629719  7393 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:22.494668  8085 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.688793  8093 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.708802  8084 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:22.717588  8085 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:22.718426  7884 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:22.719027  7748 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.550940  5525 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49030 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1590)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1573)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1587)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1585)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1574)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1580)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1593)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1591)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1600)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1601)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1569)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1592)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1603)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1605)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1606)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1616)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1617)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1607)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1622)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1579)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1614)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1619)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1618)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 372)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1608)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1610)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1611)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1613)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1581)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1582)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1583)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1586)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1597)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1623)
I20260709 12:25:22.630544  7588 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:22.569737  8068 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.726233  7761 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.726840  7761 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:22.727257  8068 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:22.728986  8079 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:22.729434  8047 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:22.630940  8110 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:25:22.631583  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:22.639294  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 82172us
I20260709 12:25:22.733991  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1760) took 3675 ms. Trace:
I20260709 12:25:22.734392  5706 rpcz_store.cc:276] 0709 12:25:19.059064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.347155 (+288091us) service_pool.cc:224] Handling call
0709 12:25:19.386858 (+ 39703us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:19.978467 (+591609us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:19.978520 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:19.981578 (+  3058us) log.cc:844] Serialized 26 byte log entry
0709 12:25:21.381408 (+1399830us) spinlock_profiling.cc:243] Waited 7.46 ms on lock 0x7b180029b540. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:25:21.716892 (+335484us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:21.716959 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.717075 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:22.733694 (+1016619us) raft_consensus.cc:1685] finished
0709 12:25:22.733718 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.733932 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":79473,"raft.queue_time_us":416917,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":11125,"spinlock_wait_cycles":17147648,"thread_start_us":798669,"threads_started":2,"wal-append.queue_time_us":2431047}
I20260709 12:25:22.735603  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:22.735761  8030 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.736305  8030 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:22.737252  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:22.643603  8036 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:22.643329  7996 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:22.738281  8030 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:22.707845  7922 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:22.735641  7832 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.616951  8124 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:22.739534  8051 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:22.659390  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:22.676076  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:22.740190  6280 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:22.740567  5705 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:22.675956  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:22.741160  5634 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:22.741189  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1751) took 1583 ms. Trace:
I20260709 12:25:22.741680  5705 rpcz_store.cc:276] 0709 12:25:21.158106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.675852 (+1517746us) service_pool.cc:224] Handling call
0709 12:25:22.741121 (+ 65269us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.742612  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:22.685423  5601 leader_election.cc:400] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5.
I20260709 12:25:22.736146  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:22.743515  5601 leader_election.cc:403] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:25:22.668161  5552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1587) took 1725 ms (client timeout 2036 ms). Trace:
W20260709 12:25:22.686081  5675 connection.cc:256] RPC call timeout handler was delayed by -1.2712436469996646s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19190000000000002s for a response
I20260709 12:25:22.648514  7885 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:22.687187  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:22.687407  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1753) took 1528 ms (client timeout 1812 ms). Trace:
I20260709 12:25:22.609678  8125 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:22.688159  5632 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:22.622854  7959 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:22.690810  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:22.690958  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 94391.6us
W20260709 12:25:22.749938  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 67809.9us
I20260709 12:25:22.663930  8067 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:22.692459  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:22.623270  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:22.751678  5707 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:22.751801  5627 raft_consensus.cc:2368] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6.
I20260709 12:25:22.752108  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1750) took 1594 ms. Trace:
I20260709 12:25:22.749596  8030 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:22.752533  5707 rpcz_store.cc:276] 0709 12:25:21.157878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.692108 (+1534230us) service_pool.cc:224] Handling call
0709 12:25:22.752045 (+ 59937us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.753494  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:22.754248  5707 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:25:22.754284  5634 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:22.754730  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1778) took 1580 ms. Trace:
I20260709 12:25:22.755034  5707 rpcz_store.cc:276] 0709 12:25:21.174312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.753256 (+1578944us) service_pool.cc:224] Handling call
0709 12:25:22.754680 (+  1424us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.755257  8079 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:22.693145  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:22.690510  7933 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260709 12:25:22.693823  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:22.694254  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:22.756745  7933 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:22.701330  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:22.756934  6279 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46486
I20260709 12:25:22.757615  6276 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:22.757716  6279 raft_consensus.cc:606] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:22.758123  6279 raft_consensus.cc:614] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:22.757014  6274 raft_consensus.cc:2410] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:25:22.748837  8066 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:22.702515  7967 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:22.759011  8067 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:22.759773  8079 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:25:22.760409  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:22.681098  7921 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:22.759099  8066 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:22.678784  8121 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:22.761175  8102 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:22.761340  8066 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:22.658021  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:22.659368  7957 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:22.740185  6272 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:22.739395  7996 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:22.750411  8125 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.676580  5554 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:22.764273  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:22.773010  5701 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:22.773366  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:22.773641  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1744) took 1617 ms. Trace:
I20260709 12:25:22.774088  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:22.774689  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 91616.7us
I20260709 12:25:22.765556  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:22.775504  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:22.777657  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:22.778477  7996 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:22.780148  7996 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:25:22.767537  5553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1578) took 1844 ms (client timeout 1505 ms). Trace:
W20260709 12:25:22.781656  5553 rpcz_store.cc:269] 0709 12:25:20.923350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.954590 (+ 31240us) service_pool.cc:224] Handling call
0709 12:25:22.767322 (+1812732us) spinlock_profiling.cc:243] Waited 7.82 ms on lock 0x7b54002279d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:22.767425 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17990016}
I20260709 12:25:22.713418  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:22.734090  5636 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
W20260709 12:25:22.773398  5630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1529) took 1958 ms (client timeout 2059 ms). Trace:
W20260709 12:25:22.783701  5630 rpcz_store.cc:269] 0709 12:25:20.814711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.869033 (+ 54322us) service_pool.cc:224] Handling call
0709 12:25:22.773252 (+1904219us) spinlock_profiling.cc:243] Waited 90.8 ms on lock 0x7b5400438d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:22.773345 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":208813824}
I20260709 12:25:22.774953  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1745) took 1618 ms. Trace:
I20260709 12:25:22.787570  5709 rpcz_store.cc:276] 0709 12:25:21.156502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.773853 (+1617351us) service_pool.cc:224] Handling call
0709 12:25:22.774900 (+  1047us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.744014  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1784) took 1044 ms. Trace:
W20260709 12:25:22.744131  5552 rpcz_store.cc:269] 0709 12:25:20.942789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.666833 (+1724044us) service_pool.cc:224] Handling call
0709 12:25:22.668093 (+  1260us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:22.746033  5702 rpcz_store.cc:269] 0709 12:25:21.158663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.458352 (+299689us) service_pool.cc:224] Handling call
0709 12:25:22.687222 (+1228870us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b54001a8e54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:22.687352 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1537034624}
I20260709 12:25:22.788712  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:22.789350  5706 rpcz_store.cc:276] 0709 12:25:21.699336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.735883 (+1036547us) service_pool.cc:224] Handling call
0709 12:25:22.743960 (+  8077us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.790087  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:22.751106  5599 leader_election.cc:336] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.931s (SENT)
I20260709 12:25:22.774261  5632 raft_consensus.cc:2410] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
W20260709 12:25:22.790200  5602 leader_election.cc:336] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 3.666s (SENT)
W20260709 12:25:22.790982  5599 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.935s (SENT)
W20260709 12:25:22.791563  5599 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.035s (SENT)
I20260709 12:25:22.757162  7933 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:22.751278  7959 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:22.699713  8101 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.743280  5705 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:22.805819  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:22.806384  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1746) took 1649 ms. Trace:
I20260709 12:25:22.756044  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:22.807119  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:22.806965  5705 rpcz_store.cc:276] 0709 12:25:21.156742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.742366 (+1585624us) service_pool.cc:224] Handling call
0709 12:25:22.806307 (+ 63941us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.807862  5555 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:22.807776  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:22.808409  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:22.808689  5707 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:22.809010  5630 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:22.730607  5525 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:58854 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1585)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1574)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1580)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1593)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1591)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1600)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1601)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1569)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1592)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1603)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1605)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1606)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1616)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1617)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1607)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1622)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1579)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1614)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1619)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1618)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 372)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1608)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1610)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1611)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1613)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1581)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1582)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1583)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1586)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1597)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1623)
I20260709 12:25:22.809553  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:22.808120  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:22.810209  5707 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:22.810339  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:22.810623  5705 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:22.793212  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:22.810979  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:22.812299  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:22.788161  5554 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:22.774163  5701 rpcz_store.cc:276] 0709 12:25:21.156224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.763755 (+1607531us) service_pool.cc:224] Handling call
0709 12:25:22.773558 (+  9803us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.769903  8078 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 12:25:22.816161  7933 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.817438  7933 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:22.798458  5710 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:22.803349  8039 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.792033  8124 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:25:22.739809  8104 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:22.805840  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:22.838532  5706 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:22.809805  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:22.795588  7922 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.842264  5704 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:22.794034  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:22.744585  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:22.814080  5555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1574) took 1892 ms (client timeout 2103 ms). Trace:
W20260709 12:25:22.843237  5555 rpcz_store.cc:269] 0709 12:25:20.922232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.783293 (+1861061us) service_pool.cc:224] Handling call
0709 12:25:22.814035 (+ 30742us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.815162  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:22.844267  5702 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:22.761711  8102 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:22.766508  8058 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:22.765311  7957 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:22.816295  5552 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:22.835954  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:22.860270  5552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1580) took 1924 ms (client timeout 2154 ms). Trace:
W20260709 12:25:22.860998  5552 rpcz_store.cc:269] 0709 12:25:20.935507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.788989 (+1853482us) service_pool.cc:224] Handling call
0709 12:25:22.860186 (+ 71197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.820091  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1782) took 1568 ms. Trace:
I20260709 12:25:22.862627  5707 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:22.844537  8050 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.811105  7959 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:22.810958  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.434340  5553 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
W20260709 12:25:23.434967  5553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1585) took 2498 ms (client timeout 2056 ms). Trace:
I20260709 12:25:22.840409  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:22.791992  5709 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:22.843613  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:22.845065  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:23.455660  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 390) took 1110 ms. Trace:
W20260709 12:25:22.852936  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 58967.5us
I20260709 12:25:23.456151  5709 rpcz_store.cc:276] 0709 12:25:22.344869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.789857 (+444988us) service_pool.cc:224] Handling call
0709 12:25:23.455583 (+665726us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.457417  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:22.854162  8102 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:22.835378  8036 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.841614  7922 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:22.839926  5525 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:58854 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1593)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1591)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1600)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1601)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1569)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1592)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1603)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1605)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1606)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1616)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1617)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1607)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1622)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1579)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1614)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1619)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1618)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 372)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1608)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1610)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1611)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1613)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1581)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1582)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1583)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1586)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1597)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1623)
I20260709 12:25:22.813817  8047 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:22.838754  8078 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:22.861935  5710 rpcz_store.cc:276] 0709 12:25:21.251597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.712104 (+1460507us) service_pool.cc:224] Handling call
0709 12:25:22.820010 (+107906us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:22.780084  5524 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49044 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1593)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1591)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1600)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1601)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1569)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1592)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1603)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1605)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1606)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1616)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1617)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1607)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1622)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1579)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1614)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 377)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1619)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1618)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 372)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1608)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1610)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1611)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1613)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1581)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1582)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1583)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1586)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1597)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1623)
I20260709 12:25:23.462842  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:22.822305  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:22.856102  8093 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:23.092999  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:22.864065  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 73927.4us
W20260709 12:25:23.381666  5560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1590) took 2435 ms (client timeout 1988 ms). Trace:
W20260709 12:25:23.485481  5560 rpcz_store.cc:269] 0709 12:25:20.946119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.037319 (+2091200us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:23.381599 (+344280us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:25:23.477305  5524 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:49044 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1600)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1601)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1569)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1592)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1603)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1605)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1606)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1616)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1617)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1607)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1622)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1579)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1614)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1619)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1611)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1618)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1610)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 372)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1608)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1610)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1611)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1613)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1581)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1582)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1583)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1586)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1597)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1623)
W20260709 12:25:23.037271  8139 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 12:25:22.851332  8068 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:22.875278  5706 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
W20260709 12:25:23.488837  5560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1600) took 1722 ms (client timeout 1500 ms). Trace:
I20260709 12:25:22.865337  8104 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:22.959189  8041 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.870493  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:22.875324  5707 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:23.332105  8101 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:23.383419  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:22.869505  5629 raft_consensus.cc:2410] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:25:23.086503  5705 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:23.489208  5524 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:49044 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1601)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1569)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1592)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1603)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1605)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1606)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1616)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1617)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1607)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1622)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1579)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1614)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1613)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1619)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1611)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1618)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1610)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 372)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1608)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1610)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1611)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1613)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1581)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1582)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1583)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1586)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1597)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1623)
I20260709 12:25:22.864167  7957 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:23.385748  7748 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.014s	sys 0.020s
W20260709 12:25:23.403049  5561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1594) took 2455 ms (client timeout 1829 ms). Trace:
W20260709 12:25:23.400565  6272 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.635s	user 0.012s	sys 0.000s
I20260709 12:25:23.392948  7832 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:23.424718  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:22.839864  8039 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:23.451500  8088 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:23.454227  5704 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1742) took 2348 ms (client timeout 2800 ms). Trace:
W20260709 12:25:23.435343  5553 rpcz_store.cc:269] 0709 12:25:20.936786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.782611 (+1845825us) service_pool.cc:224] Handling call
0709 12:25:23.434873 (+652262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.454071  5630 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:23.421345  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:22.852460  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:23.455957  5701 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:23.504235  5553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1601) took 1739 ms (client timeout 1505 ms). Trace:
I20260709 12:25:23.459307  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:23.466396  6277 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:23.504575  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:23.460340  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:23.460587  8036 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:23.462464  7922 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:23.466513  5525 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49030 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1600)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1601)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1569)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1592)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1603)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1605)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1606)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1616)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1617)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1607)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1622)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1579)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1614)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1619)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1611)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1618)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1610)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 372)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1608)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1610)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1611)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1613)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1581)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1582)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1583)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1586)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1597)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1623)
W20260709 12:25:23.470530  7933 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.653s	user 0.009s	sys 0.002s
W20260709 12:25:23.472007  7996 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.691s	user 0.006s	sys 0.006s
I20260709 12:25:23.458033  8050 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:23.472882  8066 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.711s	user 0.012s	sys 0.000s
I20260709 12:25:23.473264  8047 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:23.460294  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 397) took 5192 ms. Trace:
I20260709 12:25:23.474124  8089 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:23.194391  7897 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.501691  6272 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:23.532223  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:23.476737  8124 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.625s	user 0.009s	sys 0.004s
W20260709 12:25:23.480675  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 658390us
I20260709 12:25:23.435559  8125 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:23.536166  8003 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:23.536247  7897 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:23.537668  8036 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:23.039400  8134 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.179s	user 0.000s	sys 0.000s
I20260709 12:25:23.489315  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 397) took 1129 ms. Trace:
W20260709 12:25:23.489269  5560 rpcz_store.cc:269] 0709 12:25:21.764031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.486541 (+1722510us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:23.486662 (+   121us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:25:23.489950  5602 leader_election.cc:336] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.988s (SENT)
W20260709 12:25:23.381294  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 533567us
I20260709 12:25:23.540838  5706 rpcz_store.cc:276] 0709 12:25:22.360225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.790012 (+429787us) service_pool.cc:224] Handling call
0709 12:25:23.489251 (+699239us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.543087  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:23.495128  5552 raft_consensus.cc:2410] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:23.498260  5707 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:25:23.548118  5552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1591) took 2601 ms (client timeout 2204 ms). Trace:
I20260709 12:25:23.485033  8078 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.500277  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:23.487751  7961 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.498162  7858 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.497582  5524 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:49044 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1601)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1569)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1592)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1603)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1605)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1606)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1616)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1617)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1607)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1622)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1579)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1614)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1613)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1619)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1611)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1618)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1610)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 372)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1608)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1610)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1611)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1613)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1581)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1582)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1583)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1586)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1597)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1623)
I20260709 12:25:23.491082  8030 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:23.504113  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1747) took 2347 ms. Trace:
I20260709 12:25:23.551687  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:23.505062  5628 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:23.556594  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:23.556643  5706 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:25:23.557245  5628 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:23.496342  8056 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:23.513648  5630 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:23.516232  7959 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:23.514138  5704 rpcz_store.cc:269] 0709 12:25:21.105858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.105993 (+   135us) service_pool.cc:224] Handling call
0709 12:25:23.445425 (+2339432us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b5400309654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:23.454076 (+  8651us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3790426880}
W20260709 12:25:23.504666  5553 rpcz_store.cc:269] 0709 12:25:21.764400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.504055 (+1739655us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:23.504178 (+   123us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:25:23.498296  8041 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:23.558416  8030 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.571556  8030 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:23.572888  8041 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:23.515625  8088 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:23.532853  8089 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:23.575919  8053 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:25:23.518618  5525 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49030 dropped due to backpressure: service queue is full with 50 items [suppressed 7 similar messages]
I20260709 12:25:23.576452  8053 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:23.577260  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 90620.3us
I20260709 12:25:23.552341  8124 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5.
I20260709 12:25:23.551971  8078 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:23.576647  5525 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49030 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1592)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1603)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1605)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1606)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1616)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1617)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1607)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1622)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1579)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1614)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1613)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1617)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1619)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1611)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1626)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1618)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1610)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 372)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1608)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1610)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1611)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1613)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1581)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1582)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1583)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1586)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1597)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1623)
I20260709 12:25:23.527297  5631 rpcz_store.cc:276] 0709 12:25:18.264866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.690679 (+1425813us) service_pool.cc:224] Handling call
0709 12:25:20.919691 (+1229012us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.919894 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:20.919918 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.919961 (+    43us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:20.920000 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.920053 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.456750 (+2536697us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54001ce8d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:23.456872 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2823869824}
I20260709 12:25:23.559015  7961 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:23.581332  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:23.038201  7870 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:23.535184  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:23.495096  8068 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:23.496332  5705 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:23.583359  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1752) took 2425 ms. Trace:
W20260709 12:25:23.498651  5561 rpcz_store.cc:269] 0709 12:25:20.947335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.025748 (+ 78413us) service_pool.cc:224] Handling call
0709 12:25:23.402740 (+2376992us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b540023f354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:23.402928 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3173070208}
I20260709 12:25:23.583660  7870 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:23.584235  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.584730  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:23.585005  5561 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:25:23.585446  5561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1592) took 2638 ms (client timeout 2810 ms). Trace:
W20260709 12:25:23.585878  5558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1609) took 2892 ms (client timeout 2176 ms). Trace:
W20260709 12:25:23.585789  5561 rpcz_store.cc:269] 0709 12:25:20.946710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.583998 (+2637288us) service_pool.cc:224] Handling call
0709 12:25:23.585391 (+  1393us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.586828  5636 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:23.587062  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.550328  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1875) took 1187 ms. Trace:
I20260709 12:25:23.548889  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1755) took 2389 ms. Trace:
I20260709 12:25:23.590986  5635 rpcz_store.cc:276] 0709 12:25:22.363116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.494029 (+1130913us) service_pool.cc:224] Handling call
0709 12:25:23.550241 (+ 56212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.592336  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:23.590587  5558 rpcz_store.cc:269] 0709 12:25:20.693190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.883107 (+189917us) service_pool.cc:224] Handling call
0709 12:25:23.585688 (+2702581us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b54003c46d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:23.585807 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3801333504}
W20260709 12:25:23.548862  5552 rpcz_store.cc:269] 0709 12:25:20.946407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.861776 (+1915369us) service_pool.cc:224] Handling call
0709 12:25:23.548034 (+686258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.557608  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1772) took 2384 ms. Trace:
I20260709 12:25:23.558594  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1873) took 1196 ms. Trace:
I20260709 12:25:23.593963  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.594235  5628 rpcz_store.cc:276] 0709 12:25:22.362546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.556195 (+1193649us) service_pool.cc:224] Handling call
0709 12:25:23.558505 (+  2310us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.594841  5558 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
W20260709 12:25:23.595479  5558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1570) took 2677 ms (client timeout 2850 ms). Trace:
W20260709 12:25:23.498572  7748 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.014s	sys 0.020s
W20260709 12:25:23.561558  5560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1569) took 2637 ms (client timeout 2477 ms). Trace:
W20260709 12:25:23.596509  5560 rpcz_store.cc:269] 0709 12:25:20.917675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.554592 (+2636917us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:23.554748 (+   156us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:25:23.597711  5558 rpcz_store.cc:269] 0709 12:25:20.918111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.593660 (+2675549us) service_pool.cc:224] Handling call
0709 12:25:23.595406 (+  1746us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.555445  8003 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:23.570566  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:23.605047  5704 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:23.588490  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 414) took 1238 ms. Trace:
I20260709 12:25:23.606245  5636 rpcz_store.cc:276] 0709 12:25:22.349673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.783924 (+434251us) service_pool.cc:224] Handling call
0709 12:25:23.588261 (+804337us) spinlock_profiling.cc:243] Waited 775 ms on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:23.588415 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1782519040}
I20260709 12:25:23.583702  5705 rpcz_store.cc:276] 0709 12:25:21.158372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.495708 (+2337336us) service_pool.cc:224] Handling call
0709 12:25:23.496008 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.583120 (+ 87112us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.583306 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.608038  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:23.608803  5705 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:23.609844  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:23.589308  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54251.8us
I20260709 12:25:23.612053  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:23.593168  5635 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 6: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:23.554960  5708 rpcz_store.cc:276] 0709 12:25:21.156996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.406748 (+2249752us) service_pool.cc:224] Handling call
0709 12:25:23.504056 (+ 97308us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.593914  5706 rpcz_store.cc:276] 0709 12:25:21.172884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.555952 (+2383068us) service_pool.cc:224] Handling call
0709 12:25:23.556295 (+   343us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.557247 (+   952us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.557524 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.559794  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:23.591735  5707 rpcz_store.cc:276] 0709 12:25:21.159232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.497583 (+2338351us) service_pool.cc:224] Handling call
0709 12:25:23.497877 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.548196 (+ 50319us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.548788 (+   592us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.599318  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.524289  7996 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:23.563180  5524 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:49044 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1592)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1577)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1570)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1603)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1605)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1606)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1616)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1617)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 378)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1607)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1622)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 387)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 384)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1579)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1614)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 386)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 385)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 379)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 380)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1613)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 381)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1617)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1619)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1611)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1626)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1618)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 383)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1610)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1620)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 367)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 368)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 369)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 372)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 373)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1608)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1610)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 376)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1611)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1613)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1581)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1582)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1583)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1586)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1597)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1615)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1621)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1623)
W20260709 12:25:23.591089  5561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1577) took 2668 ms (client timeout 2838 ms). Trace:
I20260709 12:25:23.595176  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:23.621526  5561 rpcz_store.cc:269] 0709 12:25:20.923120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.586850 (+2663730us) service_pool.cc:224] Handling call
0709 12:25:23.591010 (+  4160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.622018  5628 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:25:23.622169  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 413) took 1276 ms. Trace:
I20260709 12:25:23.602084  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:23.602090  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:23.616235  5558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1605) took 1715 ms (client timeout 2035 ms). Trace:
I20260709 12:25:23.623072  5634 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:23.617784  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:23.622735  5635 rpcz_store.cc:276] 0709 12:25:22.346026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.592081 (+1246055us) service_pool.cc:224] Handling call
0709 12:25:23.622087 (+ 30006us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.622537  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.627043  5561 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:23.625799  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1870) took 1264 ms. Trace:
W20260709 12:25:23.623009  5558 rpcz_store.cc:269] 0709 12:25:21.900838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.598987 (+1698149us) service_pool.cc:224] Handling call
0709 12:25:23.616155 (+ 17168us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.628440  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:23.628650  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:23.629202  5558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1616) took 1271 ms (client timeout 1692 ms). Trace:
I20260709 12:25:23.629617  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:23.621517  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:23.629534  5558 rpcz_store.cc:269] 0709 12:25:22.358087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.628143 (+1270056us) service_pool.cc:224] Handling call
0709 12:25:23.629149 (+  1006us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.631461  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:23.631587  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.631618  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:23.632237  5558 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:23.632000  5628 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:23.632881  5630 rpcz_store.cc:276] 0709 12:25:22.361449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.617457 (+1256008us) service_pool.cc:224] Handling call
0709 12:25:23.625698 (+  8241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.627131  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:23.530014  8066 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:23.633097  5705 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 4: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:23.633998  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:23.635815  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.645725  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:23.646500  5706 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:23.634306  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:23.646742  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:23.633136  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1608) took 1594 ms. Trace:
I20260709 12:25:23.561022  8036 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:23.650437  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1749) took 2492 ms. Trace:
I20260709 12:25:23.650813  5704 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:25:23.641546  5561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1606) took 1701 ms (client timeout 2103 ms). Trace:
I20260709 12:25:23.651507  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1776) took 2477 ms. Trace:
I20260709 12:25:23.641976  5635 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:23.560556  7897 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:23.647679  5558 rpcz_store.cc:276] 0709 12:25:22.039194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.631381 (+1592187us) service_pool.cc:224] Handling call
0709 12:25:23.633082 (+  1701us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.654496  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:23.655308  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:23.656637  5634 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:23.657229  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1884) took 1007 ms. Trace:
I20260709 12:25:23.657532  5634 rpcz_store.cc:276] 0709 12:25:22.649573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.601775 (+952202us) service_pool.cc:224] Handling call
0709 12:25:23.657170 (+ 55395us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.658737  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:23.592559  8039 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:23.659835  5634 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:23.648034  5636 raft_consensus.cc:2410] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:25:23.632488  6274 raft_consensus.cc:2410] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260709 12:25:23.561259  6275 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 574) took 2461 ms (client timeout 2660 ms). Trace:
I20260709 12:25:23.584264  7870 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:23.562084  8125 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:23.650027  7748 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:25:23.651459  5561 rpcz_store.cc:269] 0709 12:25:21.939850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.622247 (+1682397us) service_pool.cc:224] Handling call
0709 12:25:23.641463 (+ 19216us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:23.672839  5557 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1575) took 2750 ms (client timeout 1744 ms). Trace:
I20260709 12:25:23.672935  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:23.673166  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:23.673581  5561 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:23.673295  5557 rpcz_store.cc:269] 0709 12:25:20.922615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.962346 (+ 39731us) service_pool.cc:224] Handling call
0709 12:25:23.672629 (+2710283us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b54003a8ed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:23.672770 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3787980032}
W20260709 12:25:23.667019  6275 rpcz_store.cc:269] 0709 12:25:21.099618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.118145 (+ 18527us) service_pool.cc:224] Handling call
0709 12:25:23.416804 (+2298659us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754deee 000000000036eb39 00007f4fff5c11e0 00007f4fff68a0c4 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:23.556503 (+139699us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48900480}
I20260709 12:25:23.650470  8036 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:23.649140  8142 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:23.652895  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1874) took 1290 ms. Trace:
I20260709 12:25:23.653625  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:23.680928  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 103067us
I20260709 12:25:23.655329  5558 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:23.662158  5705 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:25:23.682044  5635 rpcz_store.cc:276] 0709 12:25:22.362857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.626684 (+1263827us) service_pool.cc:224] Handling call
0709 12:25:23.652836 (+ 26152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.684180  5635 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:23.558257  7858 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:23.664240  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1871) took 1302 ms. Trace:
I20260709 12:25:23.689970  5634 rpcz_store.cc:276] 0709 12:25:22.361798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.658471 (+1296673us) service_pool.cc:224] Handling call
0709 12:25:23.664173 (+  5702us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.691828  5634 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:25:23.692809  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 406) took 1376 ms. Trace:
I20260709 12:25:23.667510  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:23.693305  5634 rpcz_store.cc:276] 0709 12:25:22.316610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.690998 (+1374388us) service_pool.cc:224] Handling call
0709 12:25:23.691520 (+   522us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.692424 (+   904us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.692739 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.651546  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:23.653512  7922 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:23.652189  5704 rpcz_store.cc:276] 0709 12:25:21.173829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.630456 (+2456627us) service_pool.cc:224] Handling call
0709 12:25:23.630697 (+   241us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.651259 (+ 20562us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.651453 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:23.682250  5602 leader_election.cc:336] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.204s (SENT)
I20260709 12:25:23.682569  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 392) took 1323 ms. Trace:
I20260709 12:25:23.645849  5552 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:23.696488  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:23.696794  5552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1603) took 1816 ms (client timeout 1931 ms). Trace:
I20260709 12:25:23.696478  5705 rpcz_store.cc:276] 0709 12:25:22.358826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.661437 (+1302611us) service_pool.cc:224] Handling call
0709 12:25:23.661861 (+   424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.682158 (+ 20297us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.682485 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:23.697296  5552 rpcz_store.cc:269] 0709 12:25:21.880892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.593766 (+1712874us) service_pool.cc:224] Handling call
0709 12:25:23.696726 (+102960us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.698417  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:23.650851  5710 rpcz_store.cc:276] 0709 12:25:21.157652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.465640 (+2307988us) service_pool.cc:224] Handling call
0709 12:25:23.493377 (+ 27737us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.493578 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:25:23.561637 (+ 68059us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.650098 (+ 88461us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.650159 (+    61us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:23.650211 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.650277 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.650387 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":93,"thread_start_us":144627,"threads_started":2}
I20260709 12:25:23.686646  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 405) took 1370 ms. Trace:
I20260709 12:25:23.668531  5628 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:23.692996  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1617) took 1320 ms. Trace:
I20260709 12:25:23.634891  8003 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:25:23.665629  5560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1604) took 1767 ms (client timeout 1935 ms). Trace:
I20260709 12:25:23.694276  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:23.701318  5560 rpcz_store.cc:269] 0709 12:25:21.898756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.597371 (+1698615us) service_pool.cc:224] Handling call
0709 12:25:23.665526 (+ 68155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.700165  5558 rpcz_store.cc:276] 0709 12:25:22.372840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.654231 (+1281391us) service_pool.cc:224] Handling call
0709 12:25:23.692949 (+ 38718us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.702339  5557 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:23.700055  5635 rpcz_store.cc:276] 0709 12:25:22.316253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.683565 (+1367312us) service_pool.cc:224] Handling call
0709 12:25:23.683938 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.686264 (+  2326us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.686565 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.703009  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.700102  5630 raft_consensus.cc:2410] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:25:23.703806  5558 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:23.703013  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1607) took 1728 ms. Trace:
I20260709 12:25:23.704670  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1622) took 1314 ms. Trace:
I20260709 12:25:23.705371  5557 rpcz_store.cc:276] 0709 12:25:21.974370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.674265 (+1699895us) service_pool.cc:224] Handling call
0709 12:25:23.702947 (+ 28682us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.705467  5558 rpcz_store.cc:276] 0709 12:25:22.391177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.702685 (+1311508us) service_pool.cc:224] Handling call
0709 12:25:23.704575 (+  1890us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.706969  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.708045  5558 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:23.708191  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 416) took 1347 ms. Trace:
I20260709 12:25:23.710016  5636 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:25:23.709282  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.708841  5628 rpcz_store.cc:276] 0709 12:25:22.360964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.707127 (+1346163us) service_pool.cc:224] Handling call
0709 12:25:23.707457 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.707764 (+   307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.707802 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.707860 (+    58us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:23.707932 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.708004 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.708126 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.711910  5634 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:25:23.712527  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 409) took 1395 ms. Trace:
I20260709 12:25:23.712631  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1877) took 1349 ms. Trace:
I20260709 12:25:23.712857  5634 rpcz_store.cc:276] 0709 12:25:22.317475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.703318 (+1385843us) service_pool.cc:224] Handling call
0709 12:25:23.703636 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.712251 (+  8615us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.712447 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.712984  5628 rpcz_store.cc:276] 0709 12:25:22.363651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.711852 (+1348201us) service_pool.cc:224] Handling call
0709 12:25:23.712123 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.712286 (+   163us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.712309 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.712343 (+    34us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:23.712409 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.712457 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.712584 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.671427  8125 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.714701  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:23.715340  5708 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:23.710918  5558 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:23.716632  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.717303  5558 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 4: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:23.717551  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1869) took 1356 ms. Trace:
W20260709 12:25:23.717721  5558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1579) took 2794 ms (client timeout 3666 ms). Trace:
I20260709 12:25:23.717913  5630 rpcz_store.cc:276] 0709 12:25:22.361228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.651293 (+1290065us) service_pool.cc:224] Handling call
0709 12:25:23.717494 (+ 66201us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:23.718082  5558 rpcz_store.cc:269] 0709 12:25:20.923593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.716411 (+2792818us) service_pool.cc:224] Handling call
0709 12:25:23.717670 (+  1259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.719272  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1882) took 1141 ms. Trace:
I20260709 12:25:23.719727  5628 rpcz_store.cc:276] 0709 12:25:22.578363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.713954 (+1135591us) service_pool.cc:224] Handling call
0709 12:25:23.718815 (+  4861us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.718998 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.719023 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.719059 (+    36us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:23.719107 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.719149 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.719223 (+    74us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.724702  5561 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:23.725975  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.726763  5561 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:23.727406  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1614) took 1371 ms. Trace:
I20260709 12:25:23.676050  8089 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:23.731483  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 407) took 1393 ms. Trace:
I20260709 12:25:23.741698  5636 rpcz_store.cc:276] 0709 12:25:22.316889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.693976 (+1377087us) service_pool.cc:224] Handling call
0709 12:25:23.709786 (+ 15810us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.710439 (+   653us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.710632 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.734043  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 395) took 1374 ms. Trace:
I20260709 12:25:23.743003  5710 rpcz_store.cc:276] 0709 12:25:22.359683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.699775 (+1340092us) service_pool.cc:224] Handling call
0709 12:25:23.733312 (+ 33537us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.733598 (+   286us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.733637 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.733697 (+    60us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:23.733752 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.733815 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.733944 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.727802  5561 rpcz_store.cc:276] 0709 12:25:22.356598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.725676 (+1369078us) service_pool.cc:224] Handling call
0709 12:25:23.727338 (+  1662us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.745457  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.691395  8036 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:23.739794  5632 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:23.795856  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.695132  7748 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:23.797704  5561 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:23.802675  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 379) took 1073 ms. Trace:
I20260709 12:25:23.796218  5708 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:23.803113  5561 rpcz_store.cc:276] 0709 12:25:22.729036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.795502 (+1066466us) service_pool.cc:224] Handling call
0709 12:25:23.802594 (+  7092us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.803829  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 396) took 1443 ms. Trace:
I20260709 12:25:23.804203  5708 rpcz_store.cc:276] 0709 12:25:22.359959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.715904 (+1355945us) service_pool.cc:224] Handling call
0709 12:25:23.796037 (+ 80133us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.803563 (+  7526us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.803769 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.804705  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:23.701375  7922 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.805683  5561 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:23.804214  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.806777  5552 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
W20260709 12:25:23.809562  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 127310us
I20260709 12:25:23.803270  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1771) took 2630 ms. Trace:
W20260709 12:25:23.818305  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.806s (SENT) [suppressed 79 similar messages]
I20260709 12:25:23.818981  5561 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:23.819796  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 380) took 1090 ms. Trace:
I20260709 12:25:23.820236  5561 rpcz_store.cc:276] 0709 12:25:22.729318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.804372 (+1075054us) service_pool.cc:224] Handling call
0709 12:25:23.819710 (+ 15338us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.825861  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1748) took 2668 ms. Trace:
I20260709 12:25:23.821902  5710 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 12:25:23.820858  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 218833us
I20260709 12:25:23.826409  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:23.828001  5561 raft_consensus.cc:2376] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6. Re-sending same reply.
W20260709 12:25:23.828768  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 207316us
I20260709 12:25:23.828989  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:23.829629  5561 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:23.830130  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:23.826438  5709 rpcz_store.cc:276] 0709 12:25:21.157407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.457051 (+2299644us) service_pool.cc:224] Handling call
0709 12:25:23.457536 (+   485us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.457782 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:25:23.597172 (+139390us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.825459 (+228287us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.825522 (+    63us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:23.825573 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.825645 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.825793 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10576,"prepare.queue_time_us":196191,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"thread_start_us":352440,"threads_started":2}
I20260709 12:25:23.832727  5709 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:23.833149  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:23.833676  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 400) took 1472 ms. Trace:
I20260709 12:25:23.834429  5709 rpcz_store.cc:276] 0709 12:25:22.361045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.832109 (+1471064us) service_pool.cc:224] Handling call
0709 12:25:23.832531 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.833327 (+   796us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.833600 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.834496  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:23.835796  5552 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:23.728842  8125 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:23.827346  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 399) took 1466 ms. Trace:
I20260709 12:25:23.838994  5710 rpcz_store.cc:276] 0709 12:25:22.360796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.744507 (+1383711us) service_pool.cc:224] Handling call
0709 12:25:23.821435 (+ 76928us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.827059 (+  5624us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.827273 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:23.813031  5602 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.500s (SENT)
I20260709 12:25:23.841300  5710 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:23.842726  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1785) took 1045 ms. Trace:
I20260709 12:25:23.843541  5710 rpcz_store.cc:276] 0709 12:25:22.797033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.840532 (+1043499us) service_pool.cc:224] Handling call
0709 12:25:23.840989 (+   457us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.842139 (+  1150us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.842607 (+   468us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.841243  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:23.847038  5602 leader_election.cc:336] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.505s (SENT)
I20260709 12:25:23.747747  7870 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:23.848119  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:23.839191  5709 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:23.849179  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 401) took 1487 ms. Trace:
I20260709 12:25:23.849550  5709 rpcz_store.cc:276] 0709 12:25:22.361294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.838455 (+1477161us) service_pool.cc:224] Handling call
0709 12:25:23.838925 (+   470us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.848839 (+  9914us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.849102 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:23.852035  5602 leader_election.cc:336] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.744s (SENT)
I20260709 12:25:23.852813  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:23.856036  5710 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:25:23.857549  5561 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:23.858120  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 381) took 1128 ms. Trace:
I20260709 12:25:23.858410  5561 rpcz_store.cc:276] 0709 12:25:22.729578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.828710 (+1099132us) service_pool.cc:224] Handling call
0709 12:25:23.858064 (+ 29354us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.796506  8101 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.859316  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.864588  5552 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:23.815869  7922 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:23.869619  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:23.872851  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:23.873602  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:23.874274  5552 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:23.874873  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:23.875973  7996 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:23.876534  8089 raft_consensus.cc:2764] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:25:23.878659  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:23.879757  8089 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:23.881496  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:23.881749  5552 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:23.884210  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.885435  5552 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 4: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:23.886466  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:23.887786  5552 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:23.892856  5701 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:23.899576  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1786) took 1102 ms. Trace:
I20260709 12:25:23.900173  5701 rpcz_store.cc:276] 0709 12:25:22.797259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.836162 (+ 38903us) service_pool.cc:224] Handling call
0709 12:25:23.899497 (+1063335us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.904690  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.905516  5561 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:23.906563  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:23.907071  5561 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:23.908571  7897 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:23.910166  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:23.910351  7897 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:23.911106  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:23.911809  8143 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:23.912215  8143 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.912631  8143 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:23.912824  8039 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:23.913286  8039 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.913674  8039 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:23.849596  8142 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.068s
I20260709 12:25:23.914721  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:23.915280  8142 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:23.915302  7996 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:23.915823  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:23.916771  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:23.916194  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:23.917670  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1774) took 2744 ms. Trace:
I20260709 12:25:23.915925  8142 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:23.912631  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:23.918033  5707 rpcz_store.cc:276] 0709 12:25:21.173367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.614532 (+2441165us) service_pool.cc:224] Handling call
0709 12:25:23.614957 (+   425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.615245 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.615287 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.646313 (+ 31026us) log.cc:844] Serialized 26 byte log entry
0709 12:25:23.731572 (+ 85259us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:23.731637 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.731755 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.917477 (+185722us) raft_consensus.cc:1685] finished
0709 12:25:23.917496 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.917617 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":50699,"threads_started":1,"wal-append.queue_time_us":180417}
I20260709 12:25:23.915465  6278 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:23.921351  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:23.898540  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0086478219996025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1875s for a response
W20260709 12:25:23.927946  5675 leader_election.cc:336] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.919s (SENT)
I20260709 12:25:23.919008  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:23.932473  5552 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:23.934228  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1619) took 1560 ms. Trace:
I20260709 12:25:23.934783  5552 rpcz_store.cc:276] 0709 12:25:22.374228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.886172 (+1511944us) service_pool.cc:224] Handling call
0709 12:25:23.934142 (+ 47970us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.936722  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.942679  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1783) took 2244 ms. Trace:
I20260709 12:25:23.916060  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:23.943240  5706 rpcz_store.cc:276] 0709 12:25:21.698575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.656702 (+1958127us) service_pool.cc:224] Handling call
0709 12:25:23.669871 (+ 13169us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.670260 (+   389us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.670287 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.681650 (+ 11363us) log.cc:844] Serialized 26 byte log entry
0709 12:25:23.931549 (+249899us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:23.931594 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.931663 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.942358 (+ 10695us) raft_consensus.cc:1685] finished
0709 12:25:23.942385 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.942592 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1423,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":5461,"thread_start_us":184468,"threads_started":1,"wal-append.queue_time_us":229596}
I20260709 12:25:23.928718  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:23.959213  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.960299  5552 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:23.944825  5631 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:23.964203  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.965922  5552 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:23.966813  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1618) took 1593 ms. Trace:
I20260709 12:25:23.967389  5552 rpcz_store.cc:276] 0709 12:25:22.373135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.963744 (+1590609us) service_pool.cc:224] Handling call
0709 12:25:23.966718 (+  2974us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.912751  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:23.969127  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 394) took 1609 ms. Trace:
I20260709 12:25:23.962777  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:23.973923  5561 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:23.975052  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.919926  7870 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:25:23.971107  5705 rpcz_store.cc:276] 0709 12:25:22.359395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.698264 (+1338869us) service_pool.cc:224] Handling call
0709 12:25:23.698572 (+   308us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.698893 (+   321us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.698919 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.699365 (+   446us) log.cc:844] Serialized 26 byte log entry
0709 12:25:23.944394 (+245029us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:23.944442 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.944522 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.968878 (+ 24356us) raft_consensus.cc:1685] finished
0709 12:25:23.968898 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.969058 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":800,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":15461,"thread_start_us":108389,"threads_started":1,"wal-append.queue_time_us":198583}
I20260709 12:25:23.927567  7996 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:25:23.980095  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 131989us
I20260709 12:25:23.976114  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:23.981631  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 383) took 1251 ms. Trace:
I20260709 12:25:23.982169  5552 rpcz_store.cc:276] 0709 12:25:22.730099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.975785 (+1245686us) service_pool.cc:224] Handling call
0709 12:25:23.981549 (+  5764us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:23.943078  8119 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:25:23.983111  5602 leader_election.cc:336] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.931s (SENT)
I20260709 12:25:23.961956  7933 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:23.883549  8101 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:23.984105  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:23.971755  7959 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:23.961728  7961 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:23.979666  8084 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:23.932550  8144 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:23.979401  7996 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.975844  8152 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:23.993549  5636 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260709 12:25:24.002732  5636 pending_rounds.cc:85] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 3
I20260709 12:25:24.003012  5636 pending_rounds.cc:107] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260709 12:25:24.003257  5636 raft_consensus.cc:2889] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:23.992321  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:23.996820  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1493) took 5842 ms (client timeout 2625 ms). Trace:
I20260709 12:25:23.983949  8119 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:23.998911  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 70994.6us
I20260709 12:25:23.985651  8066 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:23.984421  7870 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:23.997293  6272 raft_consensus.cc:2410] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260709 12:25:24.004299  5629 rpcz_store.cc:269] 0709 12:25:18.149614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.098087 (+948473us) service_pool.cc:224] Handling call
0709 12:25:23.989763 (+4891676us) spinlock_profiling.cc:243] Waited 3.77 s on lock 0x7b54000c1054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:23.991552 (+  1789us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8670273408}
I20260709 12:25:23.989449  8158 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:24.011157  8158 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:24.012125  5629 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:24.013077  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1549) took 1224 ms. Trace:
I20260709 12:25:23.997933  8144 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:24.013921  8144 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:24.001295  7996 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:23.974437  7748 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.021054  8161 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:24.010088  8119 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:24.024493  5629 rpcz_store.cc:276] 0709 12:25:22.788665 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.011152 (+1222487us) service_pool.cc:224] Handling call
0709 12:25:24.011638 (+   486us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.012696 (+  1058us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.013004 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:24.026746  5629 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:25:23.932850  8003 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:24.030107  5675 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.059s (SENT)
W20260709 12:25:24.030371  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.031424  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1545) took 1244 ms. Trace:
I20260709 12:25:24.032193  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1550) took 1243 ms. Trace:
I20260709 12:25:24.033042  5629 rpcz_store.cc:276] 0709 12:25:22.788962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.025913 (+1236951us) service_pool.cc:224] Handling call
0709 12:25:24.026257 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.031765 (+  5508us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.032105 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:24.032025  5630 rpcz_store.cc:276] 0709 12:25:22.787022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.722047 (+935025us) service_pool.cc:224] Handling call
0709 12:25:23.722247 (+   200us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.797974 (+ 75727us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.798007 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.819500 (+ 21493us) log.cc:844] Serialized 26 byte log entry
0709 12:25:23.973026 (+153526us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:23.973069 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.973138 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.031091 (+ 57953us) raft_consensus.cc:1685] finished
0709 12:25:24.031137 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.031357 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14172,"raft.queue_time_us":25037,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":18531,"thread_start_us":101195,"threads_started":1,"wal-append.queue_time_us":168313}
I20260709 12:25:24.037740  6280 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:24.005306  5561 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:24.041527  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:24.008821  7954 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.061949  5629 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
W20260709 12:25:24.065165  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 217122us
I20260709 12:25:24.065804  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1551) took 1276 ms. Trace:
I20260709 12:25:24.066242  5629 rpcz_store.cc:276] 0709 12:25:22.789544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.034172 (+1244628us) service_pool.cc:224] Handling call
0709 12:25:24.061701 (+ 27529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.065490 (+  3789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.065751 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:24.066370  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 56951.4us
I20260709 12:25:24.046294  8053 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.071074  8101 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:24.071734  8053 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:24.072737  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:24.073318  5561 raft_consensus.cc:2376] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5. Re-sending same reply.
I20260709 12:25:24.074189  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:24.068856  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:24.074743  5561 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:24.075600  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:24.075619  7959 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:24.076273  5561 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:24.078662  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:24.048040  7832 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:24.012740  8158 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.081863  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:24.046334  8152 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:25:24.082270  8030 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.510s	user 0.011s	sys 0.000s
I20260709 12:25:24.065706  8151 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:24.088251  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:24.102011  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:24.086988  7959 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:24.106894  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:24.107236  7959 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.107683  7959 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:24.108170  5555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1593) took 3161 ms (client timeout 2152 ms). Trace:
W20260709 12:25:24.108593  5555 rpcz_store.cc:269] 0709 12:25:20.946993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.107971 (+3160978us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:24.108107 (+   136us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:25:24.109504  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:24.110253  5555 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
W20260709 12:25:24.111558  8150 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 12:25:24.116118  7858 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:24.111083  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:24.101229  7870 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.101070  8030 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.125768  5635 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:24.003320  8142 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.115881  7996 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:24.121541  8104 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:24.131458  8158 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:24.134981  8164 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260709 12:25:24.135259  5632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 415) took 1780 ms (client timeout 2057 ms). Trace:
I20260709 12:25:24.078572  7954 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.096503  8152 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:24.130257  8153 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 12:25:24.133769  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:24.163326  5561 raft_consensus.cc:2376] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6. Re-sending same reply.
W20260709 12:25:24.155359  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 172262us
I20260709 12:25:24.106909  7897 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.033s
W20260709 12:25:24.160898  5632 rpcz_store.cc:269] 0709 12:25:22.354426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.843364 (+488938us) service_pool.cc:224] Handling call
0709 12:25:24.135078 (+1291714us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54001cf7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:24.135195 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":282985216}
W20260709 12:25:24.165024  5602 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.935s (SENT)
I20260709 12:25:24.166865  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.162566  8039 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.169512  8039 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:24.161111  8094 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.012s
I20260709 12:25:24.173269  8094 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:24.173063  8152 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.176422  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:24.176753  7957 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.673s	user 0.007s	sys 0.004s
I20260709 12:25:24.177175  6272 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:24.177619  7957 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.178642  6273 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:24.173614  8094 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:24.178937  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:24.128404  7922 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:24.183334  5602 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.035s (SENT)
I20260709 12:25:24.184080  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.181213  8101 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:24.192893  8101 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:24.193197  8078 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.613s	user 0.013s	sys 0.000s
I20260709 12:25:24.193288  8101 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:24.193850  8078 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:24.182406  8094 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.197646  8094 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:24.199561  8119 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.205706  7954 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:24.206718  8036 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.207461  7996 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:24.161196  8104 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:24.208118  6275 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:24.208841  8104 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:24.208984  8101 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.211722  8125 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.214459  8101 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:24.217458  5634 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:24.216557  8168 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:25:24.221503  5602 leader_election.cc:336] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.103s (SENT)
I20260709 12:25:24.226080  5561 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:25:24.228456  6280 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:24.233281  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1548) took 1445 ms. Trace:
I20260709 12:25:24.233884  5636 rpcz_store.cc:276] 0709 12:25:22.788296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.742567 (+954271us) service_pool.cc:224] Handling call
0709 12:25:23.989941 (+247374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.010478 (+ 20537us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.233006 (+222528us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b54000c1050. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:24.233201 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":567575936}
I20260709 12:25:24.229245  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:24.235836  6272 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:24.235827  5631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1886) took 1546 ms (client timeout 1957 ms). Trace:
W20260709 12:25:24.236833  5631 rpcz_store.cc:269] 0709 12:25:22.689716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.581061 (+891345us) service_pool.cc:224] Handling call
0709 12:25:24.235625 (+654564us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b540008c0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:24.235749 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":834512768}
I20260709 12:25:24.237586  7922 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:24.237951  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 884029us
I20260709 12:25:24.238236  7922 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:25:24.238722  5555 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:24.239655  8119 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:24.239832  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1620) took 1855 ms. Trace:
I20260709 12:25:24.240262  5555 rpcz_store.cc:276] 0709 12:25:22.384783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.110818 (+1726035us) service_pool.cc:224] Handling call
0709 12:25:24.239666 (+128848us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54000ebad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:24.239780 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":278479232}
I20260709 12:25:24.241385  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 410) took 1923 ms. Trace:
I20260709 12:25:24.241477  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:24.241811  5635 rpcz_store.cc:276] 0709 12:25:22.318174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.703463 (+1385289us) service_pool.cc:224] Handling call
0709 12:25:23.703846 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.126235 (+422389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.241149 (+114914us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b54001cf2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:24.241327 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":969836288}
I20260709 12:25:24.242702  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:24.243285  5555 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:24.230386  8162 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:24.246168  8089 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:24.233317  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 367) took 5244 ms. Trace:
I20260709 12:25:24.247054  5561 rpcz_store.cc:276] 0709 12:25:18.988936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.180875 (+5191939us) service_pool.cc:224] Handling call
0709 12:25:24.225739 (+ 44864us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.233002 (+  7263us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.233255 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:24.249001  5602 leader_election.cc:336] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.154s (SENT)
I20260709 12:25:24.249419  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:24.250927  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:24.230459  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1544) took 1443 ms. Trace:
I20260709 12:25:24.251996  5628 rpcz_store.cc:276] 0709 12:25:22.786721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.720581 (+933860us) service_pool.cc:224] Handling call
0709 12:25:23.846171 (+125590us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.920565 (+ 74394us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.920596 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.973934 (+ 53338us) log.cc:844] Serialized 26 byte log entry
0709 12:25:24.062063 (+ 88129us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:24.062100 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.062175 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.230157 (+167982us) raft_consensus.cc:1685] finished
0709 12:25:24.230189 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.230374 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12,"raft.queue_time_us":67459,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"thread_start_us":143893,"threads_started":2,"wal-append.queue_time_us":148420}
W20260709 12:25:24.258831  5602 leader_election.cc:336] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:49044 dropped due to backpressure: service queue is full with 50 items
I20260709 12:25:24.205746  7957 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:24.219468  8078 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:24.264267  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.207901  8143 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.261946  8051 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:24.282941  5561 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:24.283799  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 369) took 4379 ms. Trace:
I20260709 12:25:24.284368  5561 rpcz_store.cc:276] 0709 12:25:19.904913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.267349 (+4362436us) service_pool.cc:224] Handling call
0709 12:25:24.282482 (+ 15133us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.283486 (+  1004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.283729 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:24.264654  8039 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:24.276453  8036 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:23.814237  5702 rpcz_store.cc:276] 0709 12:25:21.172642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.523312 (+2350670us) service_pool.cc:224] Handling call
0709 12:25:23.533795 (+ 10483us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.534063 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:25:23.619832 (+ 85769us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.802713 (+182881us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.802807 (+    94us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:23.802902 (+    95us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.803009 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.803185 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":155497,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":2719,"thread_start_us":236404,"threads_started":2}
I20260709 12:25:24.290665  8078 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:24.290966  7959 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:24.293987  8043 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:24.267501  8053 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:25:24.305927  8039 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:24.308408  8039 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.312795  8162 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.313349  8162 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:24.314483  5553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1602) took 2494 ms (client timeout 1744 ms). Trace:
I20260709 12:25:24.315274  7959 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:24.315583  8068 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:24.315069  5553 rpcz_store.cc:269] 0709 12:25:21.826000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.314225 (+2488225us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:24.314393 (+   168us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:25:24.316833  5553 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:25:24.317435  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 373) took 3700 ms. Trace:
I20260709 12:25:24.317842  8050 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:24.321672  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:24.317727  5553 rpcz_store.cc:276] 0709 12:25:20.616763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.316252 (+3699489us) service_pool.cc:224] Handling call
0709 12:25:24.316613 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.317209 (+   596us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.317384 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:24.327224  6276 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:24.327083  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:24.327839  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 69051.5us
I20260709 12:25:24.292831  5555 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:24.332472  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:24.306610  8125 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:24.296649  8051 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:24.336688  5555 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: [3.5-5.6]
W20260709 12:25:24.337687  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.337798  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1610) took 3638 ms. Trace:
I20260709 12:25:24.337558  8051 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:24.338181  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.334492  5553 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:25:24.338236  5555 rpcz_store.cc:276] 0709 12:25:20.699079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.336047 (+3636968us) service_pool.cc:224] Handling call
0709 12:25:24.336432 (+   385us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:24.337421 (+   989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.337719 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:24.335206  8152 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:24.339298  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1608) took 3650 ms. Trace:
I20260709 12:25:24.339736  5553 rpcz_store.cc:276] 0709 12:25:20.689243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.329352 (+3640109us) service_pool.cc:224] Handling call
0709 12:25:24.334223 (+  4871us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.339022 (+  4799us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.339247 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:24.281921  8144 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.341413  5555 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:25:24.318842  8143 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:24.346652  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 372) took 3730 ms. Trace:
I20260709 12:25:24.347858  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 376) took 3637 ms. Trace:
I20260709 12:25:24.348570  5555 rpcz_store.cc:276] 0709 12:25:20.704291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.339184 (+3634893us) service_pool.cc:224] Handling call
0709 12:25:24.341016 (+  1832us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.341769 (+   753us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.341942 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:24.348215  5561 rpcz_store.cc:276] 0709 12:25:20.616491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.324052 (+3707561us) service_pool.cc:224] Handling call
0709 12:25:24.346182 (+ 22130us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.346352 (+   170us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:24.346376 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:24.346418 (+    42us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:24.346454 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.346506 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.346599 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:24.350329  5561 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:25:24.351226  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.352396  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.352675  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.353029  5631 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:24.353205  5635 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:24.355309  6277 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:24.282871  7933 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:24.366267  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.381878  6280 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:24.366688  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.356732  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1581) took 3421 ms. Trace:
I20260709 12:25:24.383494  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:24.383211  5561 rpcz_store.cc:276] 0709 12:25:20.935811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.349803 (+3413992us) service_pool.cc:224] Handling call
0709 12:25:24.350112 (+   309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.356491 (+  6379us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.356678 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:24.385020  6277 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:24.384791  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.385059  5561 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:24.383242  5633 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:24.386260  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.386520  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.387573  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1582) took 3451 ms. Trace:
W20260709 12:25:24.345373  8160 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.482s	user 0.000s	sys 0.001s
I20260709 12:25:24.367295  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.387943  5561 rpcz_store.cc:276] 0709 12:25:20.936046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.384658 (+3448612us) service_pool.cc:224] Handling call
0709 12:25:24.384905 (+   247us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.385886 (+   981us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.387515 (+  1629us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:24.384195  6279 raft_consensus.cc:2368] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5.
I20260709 12:25:24.386763  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.357457  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.389302  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.389367  5633 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:24.389547  5635 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:24.390504  5636 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:25:24.390597  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.315390  8162 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:24.392721  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:24.393123  5630 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:24.394050  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.395123  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.395561  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.395833  5627 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:24.395865  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.396926  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.347913  8036 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:24.402500  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:24.404896  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.405464  5636 raft_consensus.cc:2376] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4. Re-sending same reply.
I20260709 12:25:24.409996  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:24.306679  8078 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.414337  5633 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:24.403611  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.415175  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.416261  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:24.416515  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.416299  5630 raft_consensus.cc:2376] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5. Re-sending same reply.
I20260709 12:25:24.417779  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.417901  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.418872  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.419273  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.420748  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.421497  5627 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:24.386977  7900 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:24.423378  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:24.422096  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.416877  5636 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:24.424429  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:24.424778  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b21feec8630479b94ed9384256483fc"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:25:24.422456  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.425443  5630 raft_consensus.cc:606] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.426277  5630 raft_consensus.cc:614] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:24.427789  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.398653  8119 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:24.426023  5631 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:24.431619  6279 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 7: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:24.404335  8036 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:24.408257  8089 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:24.431921  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.433430  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:24.430609  8142 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:25:24.437371  8171 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:25:24.438891  7996 raft_consensus.cc:2764] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:25:24.439206  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:24.439725  8089 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:24.440318  7900 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:24.440783  7900 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.441286  7900 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:24.440842  8089 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:24.446233  5559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1573) took 3524 ms (client timeout 2035 ms). Trace:
W20260709 12:25:24.446936  5559 rpcz_store.cc:269] 0709 12:25:20.921850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.445932 (+3524082us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:24.446073 (+   141us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:25:24.448992  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.414569  8143 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.434146  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 398) took 2073 ms. Trace:
I20260709 12:25:24.387079  7957 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 12:25:24.423156  7897 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:24.455139  5704 rpcz_store.cc:276] 0709 12:25:22.360498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.720790 (+1360292us) service_pool.cc:224] Handling call
0709 12:25:23.721064 (+   274us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.721371 (+   307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.721397 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.721747 (+   350us) log.cc:844] Serialized 26 byte log entry
0709 12:25:23.859291 (+137544us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:23.859330 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.859398 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.335735 (+476337us) spinlock_profiling.cc:243] Waited 5.22 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:24.430633 (+ 94898us) raft_consensus.cc:1685] finished
0709 12:25:24.430656 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.434064 (+  3408us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":117,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":71879,"spinlock_wait_cycles":11997056,"thread_start_us":10773,"threads_started":1,"wal-append.queue_time_us":34318}
I20260709 12:25:24.450249  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.450695  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1881) took 2062 ms. Trace:
I20260709 12:25:24.457762  5634 rpcz_store.cc:276] 0709 12:25:22.388190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.713699 (+1325509us) service_pool.cc:224] Handling call
0709 12:25:23.713954 (+   255us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.224908 (+510954us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.450387 (+225479us) spinlock_profiling.cc:243] Waited 503 ms on lock 0x7b54001f2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:24.450621 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1157491456}
I20260709 12:25:24.451817  6277 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:24.453166  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.453871  5559 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:25:24.461681  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1586) took 3524 ms. Trace:
I20260709 12:25:24.462203  5559 rpcz_store.cc:276] 0709 12:25:20.937077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.447810 (+3510733us) service_pool.cc:224] Handling call
0709 12:25:24.448285 (+   475us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.459935 (+ 11650us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.461610 (+  1675us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:24.462891  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.476092  5559 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:25:24.449416  7900 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.478034  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:24.467465  8119 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:24.481381  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1597) took 3502 ms. Trace:
I20260709 12:25:24.665653  5559 rpcz_store.cc:276] 0709 12:25:20.978834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.463028 (+3484194us) service_pool.cc:224] Handling call
0709 12:25:24.475847 (+ 12819us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.481111 (+  5264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.481324 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:24.464303  8143 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.471434  8170 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:25:24.505760  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.498394  8036 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:24.676694  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 198759us
I20260709 12:25:24.677667  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:24.615978  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 260978us
I20260709 12:25:24.632092  8051 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:24.642127  8078 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:24.482051  8095 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:24.455183  7933 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:24.649313  8176 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.229s	user 0.009s	sys 0.000s
I20260709 12:25:24.665410  8119 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.652114  5632 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:24.658420  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:25:24.707234  5630 raft_consensus.cc:606] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.707580  5630 raft_consensus.cc:614] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:24.680817  7897 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.660434  7900 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:24.681396  5559 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260709 12:25:24.688179  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 238607us
I20260709 12:25:24.651765  8144 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:24.704623  8036 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:24.672755  6272 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
W20260709 12:25:24.709467  5676 connection.cc:256] RPC call timeout handler was delayed by -1.05502030200023s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1875s for a response
W20260709 12:25:24.713091  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.702523  8043 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:25:24.714821  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.715097  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1615) took 2357 ms. Trace:
I20260709 12:25:24.715608  5559 rpcz_store.cc:276] 0709 12:25:22.357837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.666672 (+2308835us) service_pool.cc:224] Handling call
0709 12:25:24.681064 (+ 14392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.714766 (+ 33702us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.715028 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:24.718019  8043 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.718580  8043 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:24.719266  7832 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.719862  7832 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:24.726065  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 368) took 5721 ms. Trace:
I20260709 12:25:24.727569  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:25:24.729157  8143 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:24.726629  5552 rpcz_store.cc:276] 0709 12:25:19.004861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.245661 (+5240800us) service_pool.cc:224] Handling call
0709 12:25:24.245951 (+   290us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.246292 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:24.246319 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:24.246765 (+   446us) log.cc:844] Serialized 26 byte log entry
0709 12:25:24.287109 (+ 40344us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:24.419518 (+132409us) log.cc:844] Serialized 14 byte log entry
0709 12:25:24.426515 (+  6997us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.426620 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.725621 (+299001us) raft_consensus.cc:1685] finished
0709 12:25:24.725644 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.725981 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3768,"prepare.queue_time_us":169754,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":1377,"raft.queue_time_us":4886,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":4222,"thread_start_us":136954,"threads_started":2,"wal-append.queue_time_us":102851}
I20260709 12:25:24.731627  8053 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:24.732474  8095 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:24.733508  8050 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.734004  8050 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.692229  7996 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:24.738075  7832 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.741389  7761 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:24.708474  8119 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:24.747498  7832 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:24.702132  8170 consensus_queue.cc:1175] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" responder_term: 7 status { last_received { term: 6 index: 7 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:24.711377  7897 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:24.752161  7897 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:24.757521  5554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1607) took 4068 ms (client timeout 2235 ms). Trace:
W20260709 12:25:24.758245  5554 rpcz_store.cc:269] 0709 12:25:20.689013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.551705 (+1862692us) service_pool.cc:224] Handling call
0709 12:25:24.757257 (+2205552us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b540044f554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:24.757422 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":284783872}
I20260709 12:25:24.760582  5559 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:24.761289  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1621) took 2376 ms. Trace:
I20260709 12:25:24.761596  5559 rpcz_store.cc:276] 0709 12:25:22.390058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.716435 (+2326377us) service_pool.cc:224] Handling call
0709 12:25:24.760252 (+ 43817us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.760944 (+   692us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.761229 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3968}
I20260709 12:25:24.764786  5561 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:24.766978  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1583) took 3830 ms. Trace:
I20260709 12:25:24.767452  5561 rpcz_store.cc:276] 0709 12:25:20.936279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.391215 (+3454936us) service_pool.cc:224] Handling call
0709 12:25:24.764492 (+373277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.765499 (+  1007us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.766668 (+  1169us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b540044f550. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:24.766885 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":857716480}
I20260709 12:25:24.732093  8053 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:24.736073  8050 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:24.775949  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:25:24.774736  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:24.787031  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.810177  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:24.768319  8119 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:24.816740  8192 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:25:24.820833  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 490032us
I20260709 12:25:24.821290  5559 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: []
W20260709 12:25:24.824668  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 357116us
I20260709 12:25:24.830096  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.830603  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.830799  5632 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:24.832373  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:24.837970  5630 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:24.842708  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.843907  5630 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260709 12:25:24.849301  5556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1572) took 3928 ms (client timeout 1935 ms). Trace:
I20260709 12:25:24.850215  5634 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:24.850667  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.769073  8053 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.851414  5632 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:24.863903  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.868398  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.868842  5634 raft_consensus.cc:2368] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7.
I20260709 12:25:24.866647  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:24.849897  5556 rpcz_store.cc:269] 0709 12:25:20.921416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.182533 (+261117us) service_pool.cc:224] Handling call
0709 12:25:24.849042 (+3666509us) spinlock_profiling.cc:243] Waited 57.5 ms on lock 0x7b5400026cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:24.849202 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":132140416}
I20260709 12:25:24.873327  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:24.819872  8192 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.012s
I20260709 12:25:24.782913  8092 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:24.878526  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.883687  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.820709  8050 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.880272  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.890057  5634 raft_consensus.cc:2376] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7. Re-sending same reply.
W20260709 12:25:24.881039  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 170065us
I20260709 12:25:24.882071  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.882308  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.883782  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.879482  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.893956  5630 raft_consensus.cc:2368] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6.
I20260709 12:25:24.894778  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.895303  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.896087  5627 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:24.773641  7996 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:24.888554  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.894369  5632 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:24.831485  7761 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.892665  5633 raft_consensus.cc:2376] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4. Re-sending same reply.
I20260709 12:25:24.861609  7959 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:24.898517  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.899407  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:24.909204  5632 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:25:24.900390  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 190471us
I20260709 12:25:24.911007  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:24.912825  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.912880  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:24.886624  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:24.909770  5634 raft_consensus.cc:2368] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5.
W20260709 12:25:24.921139  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.921432  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.915532  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.922286  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:24.922834  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:24.883188  5561 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:24.894332  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:24.924610  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.925321  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:24.926797  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:24.927598  5632 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:24.879627  8036 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260709 12:25:24.880156  7832 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:25:24.886415  8103 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 12:25:24.935458  5636 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:24.937119  5633 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:25:24.937953  6279 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:25:24.928221  8092 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:24.898146  8050 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.928771  7900 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:24.920197  7959 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:24.935262  8119 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:24.915102  8051 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 12:25:24.960563  8144 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:24.961611  8050 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:24.914772  7957 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 12:25:24.933086  7897 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:25:24.977038  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.921595  8089 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:24.940138  8142 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 12:25:24.915766  7761 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } attrs { replace: true } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.977272  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1628) took 1389 ms. Trace:
I20260709 12:25:24.950132  8127 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:24.992970  8144 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:24.989624  5554 rpcz_store.cc:276] 0709 12:25:23.587506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.759687 (+1172181us) service_pool.cc:224] Handling call
0709 12:25:24.760197 (+   510us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.792892 (+ 32695us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:25:24.831288 (+ 38396us) log.cc:844] Serialized 14 byte log entry
0709 12:25:24.976912 (+145624us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:24.976964 (+    52us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:24.977020 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.977085 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.977217 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2186,"prepare.queue_time_us":47297,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":2291,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":2433,"thread_start_us":143038,"threads_started":1}
I20260709 12:25:24.994122  8144 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:24.974639  8088 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260709 12:25:24.997463  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:25.007377  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:25.007783  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1623) took 2616 ms. Trace:
W20260709 12:25:25.009428  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 186873us
I20260709 12:25:24.974048  8092 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:25.018114  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:25.018790  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:25.006371  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:25.019814  5636 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:25.021111  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:25.021385  5561 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:25.021893  5634 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:25.021718  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:25.022329  5632 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:25.022568  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:25.023527  5561 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:24.979528  7832 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:25.008183  5552 rpcz_store.cc:276] 0709 12:25:22.395147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.730834 (+2335687us) service_pool.cc:224] Handling call
0709 12:25:24.731163 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.731369 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:25:24.731645 (+   276us) log.cc:844] Serialized 14 byte log entry
0709 12:25:25.007254 (+275609us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:25.007315 (+    61us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:25.007373 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.007560 (+   187us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.007716 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22640,"prepare.queue_time_us":183,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"spinlock_wait_cycles":34560,"thread_start_us":252511,"threads_started":1}
I20260709 12:25:25.093441  5526 leader_election.cc:400] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5.
I20260709 12:25:25.094118  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:25.094836  5552 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:25.094964  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:25.028855  7897 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 12:25:25.038388  8047 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:25.005724  6274 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:25.038810  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:25.039598  8092 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:24.974411  7858 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
I20260709 12:25:25.043154  7761 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:25.044432  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:25.051120  8195 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:25.059136  7959 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:25.060647  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.060474  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:25.063446  5628 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:25.025547  8093 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:25.094264  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:25.037221  5632 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:25.097026  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:25.096096  7832 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:25.109215  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 181997us
I20260709 12:25:25.114518  5631 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:25.108481  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 69767.6us
I20260709 12:25:25.117920  5561 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:25.118076  6236 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:25.118438  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:25.100812  7996 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:25.122583  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:25.126191  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:25.124066  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:25.119161  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:25.111258  8047 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.098134  8119 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:25:25.113543  8195 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:25.137081  5635 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:25:25.115689  8093 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.138495  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:25.119611  7832 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:25.096872  8194 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260709 12:25:25.140929  5628 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:25.133746  8199 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:25:25.160859  6234 leader_election.cc:336] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.626s (SENT)
I20260709 12:25:25.137725  7996 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:25.141774  8093 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:25.142202  8047 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:25.167991  8143 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:25.178115  8103 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:25:25.204339  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 87023.4us
I20260709 12:25:25.184026  7996 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:25.174588  8200 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:25.166918  8199 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:25:25.195606  8198 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
W20260709 12:25:25.193471  6271 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 573) took 4094 ms (client timeout 4046 ms). Trace:
I20260709 12:25:25.210875  8195 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:25.210779  7996 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:25.227629  8208 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.013s
W20260709 12:25:25.212915  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 86801.5us
I20260709 12:25:25.212013  8207 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 12:25:25.240047  8207 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:25.240669  8188 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 12:25:25.240900  8207 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:25.241990  8185 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 12:25:25.243131  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1613) took 4524 ms. Trace:
W20260709 12:25:25.243355  8196 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:25:25.211357  8103 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:25.244804  8088 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:25.244966  8103 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:25.245643  8207 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:25.245724  8088 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:25.237485  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.248517  8205 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:25.249366  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:25.249667  8205 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:25.250020  8205 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:25.250471  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:25.250835  8205 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:25.252089  8205 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:25.257114  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:25.257788  8212 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:25.258430  5633 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:25:25.258695  8212 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.259153  8212 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:25.259548  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:25.229601  8197 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:25:25.262565  8183 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:25.230525  8036 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 12:25:25.263180  8183 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:25.265801  8200 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:25.267899  7933 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:25.268440  7933 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:25.269829  8119 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:25.269433  7933 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:25.270407  8119 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:25.230608  8047 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:25.243489  5555 rpcz_store.cc:276] 0709 12:25:20.718745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.349655 (+3630910us) service_pool.cc:224] Handling call
0709 12:25:24.387624 (+ 37969us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:24.405072 (+ 17448us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:24.405102 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:24.423349 (+ 18247us) log.cc:844] Serialized 52 byte log entry
0709 12:25:24.949202 (+525853us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:24.949241 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.949311 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:25.242894 (+293583us) raft_consensus.cc:1685] finished
0709 12:25:25.242912 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.243075 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3817,"raft.queue_time_us":12615,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":11000,"thread_start_us":492196,"threads_started":1,"wal-append.queue_time_us":632799}
I20260709 12:25:25.271343  8119 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:25.272372  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:25.272984  5555 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:25.234534  8193 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
W20260709 12:25:25.227042  6271 rpcz_store.cc:269] 0709 12:25:21.099322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.155072 (+ 55750us) service_pool.cc:224] Handling call
0709 12:25:21.920059 (+764987us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754de9c 00007f4ff754da5b 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:25.189216 (+3269157us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54000b54d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:25.193376 (+  4160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":462950528}
I20260709 12:25:25.230618  8143 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:25.230796  8030 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:25.234381  8199 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:25.233546  7996 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:25.276621  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 519) took 1041 ms. Trace:
I20260709 12:25:25.277094  6280 rpcz_store.cc:276] 0709 12:25:24.235469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.245695 (+ 10226us) service_pool.cc:224] Handling call
0709 12:25:24.246001 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.382695 (+136694us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.276369 (+893674us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b5400534dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:25.276550 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":312008960}
I20260709 12:25:25.279296  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:25.280149  5552 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:25.227981  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 64132.5us
I20260709 12:25:25.287357  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:25.288771  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:25.275971  8143 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:25.286633  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:25.302012  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:25.297297  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:25.251959  8088 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:25.310817  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:25.346200  8089 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:25.346776  8219 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:25:25.353150  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54591.4us
I20260709 12:25:25.308815  8200 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260709 12:25:25.296458  8199 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:25.308960  8030 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 12:25:25.308835  7761 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 12:25:25.360703  8215 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:25.307843  7996 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:25.324018  5602 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.829s (SENT)
I20260709 12:25:25.307463  8047 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:25.312528  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.359694  5631 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:25.309274  8039 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:25.309051  8194 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:25:25.361522  7761 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:25:25.363729  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 235796us
W20260709 12:25:25.378993  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 68268.8us
W20260709 12:25:25.364392  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.369302  5634 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:25:25.296236  8036 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260709 12:25:25.331521  7858 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:25.360806  8030 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:25.315575  5555 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:25.326937  5552 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:25.377879  5554 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:25.380107  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:25.384178  5631 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:25:25.383998  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:25.386615  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:25.387693  8221 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:25:25.371884  8043 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.653s	user 0.007s	sys 0.014s
I20260709 12:25:25.393409  5705 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:25.309573  7832 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:25.392182  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:25.411382  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:25.411981  5708 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:25:25.412338  5708 pending_rounds.cc:85] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 6
I20260709 12:25:25.412644  5708 pending_rounds.cc:107] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:25:25.412475  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:25.413028  5708 raft_consensus.cc:2889] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:25.413390  5552 raft_consensus.cc:2368] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5.
I20260709 12:25:25.413331  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:25.414214  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:25.309212  8207 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20260709 12:25:25.392433  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:25.393193  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:25.418586  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:25.419047  5709 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:25.397871  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:25.419240  5702 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:25.419704  5701 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:25.405251  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:25.406183  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:25.422717  5710 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:25.406471  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:25.371337  8039 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.408576  5628 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:25:25.425598  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:25.411923  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:25.379091  7761 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:25.425515  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:25.415469  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:25.459638  5706 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:25.452783  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 60441.6us
I20260709 12:25:25.457180  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:25.463022  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:25.465021  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:25.466003  5555 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:25.468783  5705 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:25.396278  8219 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:25.331604  8183 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:25.393910  8030 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:25.487308  5702 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:25.421556  8221 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:25.502418  8219 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:25.468616  7957 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:25.503702  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:25.426148  8194 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.436834  8039 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:25.470296  8170 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:25.412555  8043 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:25.396440  7858 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:25.436160  8092 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
W20260709 12:25:25.516675  8222 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.129s	user 0.000s	sys 0.002s
I20260709 12:25:25.518765  7959 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:25.518837  8224 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:25.533670  5628 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:25:25.539899  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:25.577667  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 197655us
I20260709 12:25:25.721450  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:25.589938  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:25:25.603125  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 223675us
I20260709 12:25:25.615252  5706 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:25.668298  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:25.668522  5560 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:25.670014  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:25.725718  5627 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:25.585039  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 516) took 2004 ms. Trace:
I20260709 12:25:25.731180  5556 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:25.531322  8030 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:25.720904  5558 raft_consensus.cc:2376] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4. Re-sending same reply.
I20260709 12:25:25.752678  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:25.753466  5702 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:25.753698  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:25.754468  5558 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:25.761053  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:25.755441  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:25.793609  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:25.794847  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:25.795817  5558 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:25.797520  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:25.799536  5636 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:25.800516  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:25.801210  5555 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:25.782965  5705 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:25.806721  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:25.694554  8170 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:25.798233  5708 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:25.663937  7761 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:25.719830  8224 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:25.838792  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:25.848454  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:25.848722  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:25.853943  5708 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:25.854753  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:25.855036  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:25.855396  5704 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:25.807472  5558 raft_consensus.cc:2368] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6.
I20260709 12:25:25.853240  5703 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:25.857992  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1630) took 1030 ms. Trace:
I20260709 12:25:25.858333  5558 rpcz_store.cc:276] 0709 12:25:24.828058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.806436 (+978378us) service_pool.cc:224] Handling call
0709 12:25:25.857931 (+ 51495us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.853874  5709 raft_consensus.cc:2376] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6. Re-sending same reply.
I20260709 12:25:25.859721  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:25.861902  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:25.878289  5706 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:25.878535  5627 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:25.808807  8030 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:25.805552  8226 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:25.819387  7832 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:25.873133  5704 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:25.810995  8207 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260709 12:25:25.852335  8041 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:25.880035  5558 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:25.874742  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:25.874791  5705 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:25.892901  5703 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:25.728812  8194 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:25.841375  7959 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 12:25:25.873045  8151 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:25.897959  5709 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:25.883159  5702 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:25.870738  7761 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:25.730189  6273 rpcz_store.cc:276] 0709 12:25:23.601070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.622287 (+ 21217us) service_pool.cc:224] Handling call
0709 12:25:23.664024 (+ 41737us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.179010 (+514986us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.529144 (+1350134us) spinlock_profiling.cc:243] Waited 512 ms on lock 0x7b54006bdad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:25.565354 (+ 36210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1176612096}
I20260709 12:25:25.882484  8226 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:25.904234  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 314357us
I20260709 12:25:25.905211  5555 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:25.898819  8041 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:25.925355  8053 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:25.926366  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1611) took 5208 ms. Trace:
I20260709 12:25:25.926776  5553 rpcz_store.cc:276] 0709 12:25:20.718116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.340444 (+3622328us) service_pool.cc:224] Handling call
0709 12:25:24.340805 (+   361us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:24.342214 (+  1409us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:24.342241 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:24.353320 (+ 11079us) log.cc:844] Serialized 52 byte log entry
0709 12:25:24.388078 (+ 34758us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:24.388114 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.388175 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:25.792814 (+1404639us) spinlock_profiling.cc:243] Waited 2.05 ms on lock 0x7b1800375400. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:25:25.926002 (+133188us) raft_consensus.cc:1685] finished
0709 12:25:25.926023 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.926301 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":240,"raft.queue_time_us":322,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":36113,"spinlock_wait_cycles":4704640,"thread_start_us":34398,"threads_started":1,"wal-append.queue_time_us":33223}
I20260709 12:25:25.895756  5706 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
W20260709 12:25:25.929315  8231 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:25:25.931154  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:25.916141  5558 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:25.933250  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1627) took 1106 ms. Trace:
I20260709 12:25:25.933666  5558 rpcz_store.cc:276] 0709 12:25:24.827082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.859082 (+1032000us) service_pool.cc:224] Handling call
0709 12:25:25.933190 (+ 74108us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:25.906291  8220 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.370s	user 0.000s	sys 0.001s
W20260709 12:25:25.903537  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.901983  8151 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:25:25.930766  7996 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.562s	user 0.012s	sys 0.001s
I20260709 12:25:25.943985  7858 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:25.999567  5557 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:25.942093  8078 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 12:25:25.945814  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 391) took 2205 ms. Trace:
I20260709 12:25:26.006284  5555 rpcz_store.cc:276] 0709 12:25:23.740238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.932145 (+2191907us) service_pool.cc:224] Handling call
0709 12:25:25.945306 (+ 13161us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.945484 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:25.945511 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:25.945550 (+    39us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:25.945589 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.945639 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.945757 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:26.007993  5555 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:25:26.008432  5710 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.005s	sys 0.009s
I20260709 12:25:26.008615  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1633) took 1215 ms. Trace:
I20260709 12:25:26.009007  5555 rpcz_store.cc:276] 0709 12:25:24.793483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.007278 (+1213795us) service_pool.cc:224] Handling call
0709 12:25:26.007607 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.008377 (+   770us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.008565 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.931627  8226 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:26.003558  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.008767  5710 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.005s	sys 0.009s
I20260709 12:25:25.957629  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:26.000931  5560 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
W20260709 12:25:26.013324  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55847.1us
I20260709 12:25:26.013447  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1641) took 1064 ms. Trace:
I20260709 12:25:26.013818  5560 rpcz_store.cc:276] 0709 12:25:24.949310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.472492 (+523182us) service_pool.cc:224] Handling call
0709 12:25:26.013390 (+540898us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:26.013980  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:26.011013  5555 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:26.014925  5560 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:26.026432  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:26.029141  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1634) took 1200 ms. Trace:
I20260709 12:25:26.029678  5560 rpcz_store.cc:276] 0709 12:25:24.829121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.014507 (+1185386us) service_pool.cc:224] Handling call
0709 12:25:26.014783 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.028242 (+ 13459us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.029059 (+   817us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.947991  8151 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:26.054888  5560 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:25:26.058244  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1636) took 1123 ms. Trace:
I20260709 12:25:26.058653  5560 rpcz_store.cc:276] 0709 12:25:24.934786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.030712 (+1095926us) service_pool.cc:224] Handling call
0709 12:25:26.054405 (+ 23693us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:26.057949 (+  3544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.058183 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.867599  8219 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.033141  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:26.055995  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1631) took 1227 ms. Trace:
W20260709 12:25:26.008975  8158 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.009s	sys 0.003s
I20260709 12:25:26.077294  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1631) took 2433 ms. Trace:
I20260709 12:25:26.072369  5555 rpcz_store.cc:276] 0709 12:25:24.828373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.009825 (+1181452us) service_pool.cc:224] Handling call
0709 12:25:26.010697 (+   872us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.055483 (+ 44786us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.055925 (+   442us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:26.078235  5559 rpcz_store.cc:276] 0709 12:25:23.644121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.762397 (+1118276us) service_pool.cc:224] Handling call
0709 12:25:24.762738 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.821727 (+ 58989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.077029 (+1255302us) spinlock_profiling.cc:243] Waited 58.3 ms on lock 0x7b54000271d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:26.077219 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":134126976}
I20260709 12:25:26.031206  7858 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:25:26.083014  8232 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 12:25:26.084067  8124 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:25.901403  8224 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.180637  8224 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:25.877108  8221 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.123765  8194 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:26.124270  8205 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.869s	user 0.011s	sys 0.002s
I20260709 12:25:26.104082  5555 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:26.025985  8236 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.144s
I20260709 12:25:26.103591  5559 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:25:26.197422  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:26.150408  8234 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 12:25:25.907984  8170 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.025791  8233 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.171s
W20260709 12:25:26.073326  8158 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.025s	sys 0.005s
I20260709 12:25:26.056782  8078 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:26.102100  7858 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:26.158049  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:26.125712  8240 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:26.178788  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 145748us
I20260709 12:25:26.222757  5602 leader_election.cc:400] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5.
I20260709 12:25:26.223637  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:26.026194  8230 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.097s
W20260709 12:25:26.178001  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 381227us
I20260709 12:25:26.230779  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:26.184725  8124 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:26.231993  8124 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.116573  5560 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:26.235642  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 403) took 1177 ms. Trace:
I20260709 12:25:26.200083  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:26.205904  5555 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:25:26.237032  5559 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:26.237182  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1639) took 1001 ms. Trace:
W20260709 12:25:26.189821  8242 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:25:26.237813  8251 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:26.238003  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:26.215888  8078 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:26.239182  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:26.237545  5555 rpcz_store.cc:276] 0709 12:25:25.236303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.088630 (+852327us) service_pool.cc:224] Handling call
0709 12:25:26.103941 (+ 15311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.236920 (+132979us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.237129 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:26.210578  8103 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.962s	user 0.013s	sys 0.000s
I20260709 12:25:26.222245  7858 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:26.240692  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:26.250401  5555 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:25:26.190410  8212 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.931s	user 0.005s	sys 0.015s
I20260709 12:25:26.229101  8240 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.235989  5560 rpcz_store.cc:276] 0709 12:25:25.057758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.072368 (+1014610us) service_pool.cc:224] Handling call
0709 12:25:26.101328 (+ 28960us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.235364 (+134036us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.235581 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:26.240106  5559 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:26.245636  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:26.258836  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 436) took 1137 ms. Trace:
I20260709 12:25:26.259244  5636 rpcz_store.cc:276] 0709 12:25:25.121445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.125803 (+  4358us) service_pool.cc:224] Handling call
0709 12:25:25.126053 (+   250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.800041 (+673988us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.258595 (+458554us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b54001ce8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:26.258776 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1548314368}
I20260709 12:25:26.262389  5675 leader_election.cc:400] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7.
I20260709 12:25:26.263605  5675 leader_election.cc:403] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:26.253724  5555 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:26.264755  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 405) took 1007 ms. Trace:
I20260709 12:25:26.239205  8078 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:26.265133  5555 rpcz_store.cc:276] 0709 12:25:25.257679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.251863 (+994184us) service_pool.cc:224] Handling call
0709 12:25:26.252197 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.264399 (+ 12202us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.264684 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:26.245340  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 87378us
I20260709 12:25:26.244411  8205 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.249401  8051 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.254886  8227 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:26.275949  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:26.277264  8229 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 12:25:26.283730  8228 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7.
I20260709 12:25:26.291797  5707 raft_consensus.cc:1813] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Handling vote request from an unknown peer 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:26.285616  8195 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:26.269224  8237 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:26.263341  8208 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:26.250504  8245 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:26.253872  8212 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.268631  8194 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:26.304171  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 80572.2us
I20260709 12:25:26.306830  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:26.313405  8050 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:26.313948  8050 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:26.314852  8050 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:26.316864  8244 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260709 12:25:26.318779  8056 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:26.319542  8056 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:26.286276  8205 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.319850  8221 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:26.323709  8066 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:26.324141  8066 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:26.320906  8229 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:26.324844  8229 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:26.325433  8205 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:26.325290  8066 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.325822  8229 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.300518  8054 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:26.326783  8066 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:26.333124  7761 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:26.321082  8227 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:26.346061  8227 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.327124  8229 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:26.348641  5701 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:25:26.316188  8050 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:26.354646  8043 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:26.318516  8194 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:26.333118  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.319994  8249 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:25:26.327098  8054 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.293107  8051 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.250847  8103 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.337839  5602 leader_election.cc:400] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6.
I20260709 12:25:26.321770  8102 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:26.320665  8056 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.374464  8254 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.374709  8056 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:26.375124  8254 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.377743  8254 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:26.343894  8199 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.978s	user 0.010s	sys 0.004s
I20260709 12:25:26.305419  8208 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:26.368916  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 97996.9us
I20260709 12:25:26.375891  7858 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.398070  7858 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:26.398990  8212 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:26.380996  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:26.386161  8208 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:26.427392  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:26.370157  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 94323.2us
W20260709 12:25:26.432463  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.388854  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.393905  8103 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:26.394436  5705 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:26.370707  8245 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:26.388196  8228 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:25:26.434639  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.395462  8047 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.025s	user 0.008s	sys 0.007s
W20260709 12:25:26.435801  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.437961  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:26.386880  8199 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:26.447355  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:26.323148  8226 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.434123  7858 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:26.435627  8043 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.437522  8229 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:26.486011  5675 leader_election.cc:400] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6.
I20260709 12:25:26.487622  5675 leader_election.cc:403] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:26.445634  8047 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:26.457566  8142 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:26.439317  8124 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.046s
W20260709 12:25:26.488266  8039 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.790s	user 0.002s	sys 0.009s
I20260709 12:25:26.474622  8103 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:26.457366  8195 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:26.480692  8217 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260709 12:25:26.486057  8229 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:26.445168  8245 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:26.509182  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 56027.9us
I20260709 12:25:26.489939  7858 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:26.494798  8240 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:26.517616  8217 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:26.517987  8257 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:26.506775  8039 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.489812  8078 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.537489  8078 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:26.537575  8039 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:26.523232  7858 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:26.540102  8030 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.657s	user 0.011s	sys 0.000s
I20260709 12:25:26.487393  8043 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:26.505613  8247 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.288s	user 0.001s	sys 0.001s
W20260709 12:25:26.516268  8093 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.337s	user 0.027s	sys 0.019s
I20260709 12:25:26.557689  8043 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:26.558326  8093 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.509689  8227 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6.
I20260709 12:25:26.544584  8054 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:26.561272  8210 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.562028  8210 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.562889  8151 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.564296  8210 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:26.545596  8226 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:26.547549  8252 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.257s	user 0.000s	sys 0.001s
I20260709 12:25:26.547189  8078 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:26.566694  8078 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:26.567101  8078 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:26.549891  8041 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:26.551178  8245 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:26.527247  8235 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.393s	user 0.002s	sys 0.000s
I20260709 12:25:26.568018  8078 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.570318  8078 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:26.568418  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52212.7us
I20260709 12:25:26.571687  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 593) took 1283 ms. Trace:
I20260709 12:25:26.572299  6274 rpcz_store.cc:276] 0709 12:25:25.288305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.291638 (+  3333us) service_pool.cc:224] Handling call
0709 12:25:25.361990 (+ 70352us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:25.424614 (+ 62624us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:25.452518 (+ 27904us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:25.527859 (+ 75341us) log.cc:844] Serialized 26 byte log entry
0709 12:25:25.905955 (+378096us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:25.908777 (+  2822us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.923643 (+ 14866us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:26.526557 (+602914us) spinlock_profiling.cc:243] Waited 8.46 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:26.571289 (+ 44732us) raft_consensus.cc:1685] finished
0709 12:25:26.571341 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.571614 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20850,"raft.queue_time_us":47557,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"spinlock_wait_cycles":19465600,"thread_start_us":218201,"threads_started":1,"wal-append.queue_time_us":530000}
I20260709 12:25:26.564854  8093 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:26.531639  5708 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:26.553993  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 435) took 1433 ms. Trace:
I20260709 12:25:26.574981  5634 rpcz_store.cc:276] 0709 12:25:25.120851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.123184 (+  2333us) service_pool.cc:224] Handling call
0709 12:25:25.369078 (+245894us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:25.388845 (+ 19767us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.553787 (+1164942us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b5400439c50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:26.553933 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":564624640}
I20260709 12:25:26.573932  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:26.512595  8195 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.578339  8226 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:25:26.579602  8195 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:26.579879  8151 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:26.580000  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1804) took 1180 ms. Trace:
I20260709 12:25:26.583130  5707 rpcz_store.cc:276] 0709 12:25:25.399715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.425269 (+ 25554us) service_pool.cc:224] Handling call
0709 12:25:26.579797 (+1154528us) spinlock_profiling.cc:243] Waited 825 ms on lock 0x7b5400309654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:26.579944 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1896900992}
I20260709 12:25:26.581696  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:26.584846  7900 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:26.585172  8226 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:26.586071  8127 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:26.586462  8127 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:26.588748  5703 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:26.514684  8229 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:26.585243  7900 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:26.591589  5675 leader_election.cc:400] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5.
I20260709 12:25:26.592036  5675 leader_election.cc:403] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:26.591979  7900 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.593178  7900 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:26.593371  8224 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5.
I20260709 12:25:26.587489  8127 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.595335  8127 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:26.595450  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:26.598042  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.599884  8051 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:26.598452  8195 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.599407  8224 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:26.600772  8195 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:26.533519  8217 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:26.603001  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1802) took 1204 ms. Trace:
I20260709 12:25:26.499176  8142 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:26.608856  8253 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.305s	user 0.002s	sys 0.002s
I20260709 12:25:26.608582  5701 rpcz_store.cc:276] 0709 12:25:25.399104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.420276 (+ 21172us) service_pool.cc:224] Handling call
0709 12:25:25.458363 (+ 38087us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:26.349105 (+890742us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.602744 (+253639us) spinlock_profiling.cc:243] Waited 890 ms on lock 0x7b54001a9354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:26.602941 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2046808960}
I20260709 12:25:26.610450  8142 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:26.608299  8217 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.611898  8230 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:26.602187  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:26.612977  8230 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:26.613997  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:26.543396  8194 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:26.612144  8217 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:26.618765  6273 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:26.620611  7933 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:26.624339  8051 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:25:26.633287  6234 connection.cc:256] RPC call timeout handler was delayed by -1.1617981760000475s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2219s for a response
I20260709 12:25:26.625622  8051 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:26.636014  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:26.640129  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:26.641392  6280 raft_consensus.cc:2410] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:26.641145  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:26.658197  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:26.660758  6273 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:26.634442  8200 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:25:26.614544  8230 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:26.671720  8219 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.010s	sys 0.021s
I20260709 12:25:26.670274  8088 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.686033  8216 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:25:26.678208  8195 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:26.685915  8088 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.685479  8230 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:26.684234  8219 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.011s	sys 0.021s
I20260709 12:25:26.671362  8208 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:26.709823  8208 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:26.747607  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.749495  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.755064  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.755497  5558 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: []
W20260709 12:25:26.757078  5556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1625) took 1931 ms (client timeout 1541 ms). Trace:
I20260709 12:25:26.757707  5561 raft_consensus.cc:2368] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6.
I20260709 12:25:26.757635  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 406) took 1465 ms. Trace:
W20260709 12:25:26.757758  5556 rpcz_store.cc:269] 0709 12:25:24.825402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.873085 (+ 47683us) service_pool.cc:224] Handling call
0709 12:25:26.756335 (+1883250us) spinlock_profiling.cc:243] Waited 857 ms on lock 0x7b540044e154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:26.756746 (+   411us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1970635136}
I20260709 12:25:26.760483  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:26.759002  5560 rpcz_store.cc:276] 0709 12:25:25.291731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.273853 (+982122us) service_pool.cc:224] Handling call
0709 12:25:26.294090 (+ 20237us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:26.296678 (+  2588us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:26.296740 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:26.306222 (+  9482us) log.cc:844] Serialized 26 byte log entry
0709 12:25:26.494160 (+187938us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:26.497010 (+  2850us) log.cc:844] Serialized 14 byte log entry
0709 12:25:26.501001 (+  3991us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.501076 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:26.757031 (+255955us) raft_consensus.cc:1685] finished
0709 12:25:26.757101 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.757419 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1987,"prepare.queue_time_us":2751,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":6577,"raft.run_cpu_time_us":355,"raft.run_wall_time_us":21122,"thread_start_us":96547,"threads_started":1,"wal-append.queue_time_us":168147}
I20260709 12:25:26.761799  5556 raft_consensus.cc:2376] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7. Re-sending same reply.
I20260709 12:25:26.762941  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:26.763283  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:26.764307  5556 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:26.766240  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:26.767534  5556 raft_consensus.cc:2376] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5. Re-sending same reply.
I20260709 12:25:26.767722  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:26.787227  5552 raft_consensus.cc:1813] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Handling vote request from an unknown peer 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:26.769371  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:26.789716  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1634) took 1995 ms. Trace:
I20260709 12:25:26.790218  5553 rpcz_store.cc:276] 0709 12:25:24.793818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.928515 (+1134697us) service_pool.cc:224] Handling call
0709 12:25:26.769030 (+840515us) spinlock_profiling.cc:243] Waited 5.33 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:26.789610 (+ 20580us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12261248}
I20260709 12:25:26.791725  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:26.768769  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:26.770762  5554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1628) took 1943 ms (client timeout 1690 ms). Trace:
I20260709 12:25:26.792328  5632 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: []
W20260709 12:25:26.792677  5554 rpcz_store.cc:269] 0709 12:25:24.827478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.994766 (+167288us) service_pool.cc:224] Handling call
0709 12:25:26.770367 (+1775601us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b54003a89d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:26.770565 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":835654016}
W20260709 12:25:26.794427  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 355793us
I20260709 12:25:26.794900  5554 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:26.778086  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:26.781464  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.782873  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:26.783540  5557 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1642) took 1813 ms (client timeout 1858 ms). Trace:
I20260709 12:25:26.784307  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:26.785022  5628 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:25:26.785008  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:26.783612  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:26.786072  5636 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260709 12:25:26.788501  5561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1635) took 1994 ms (client timeout 2012 ms). Trace:
I20260709 12:25:26.774120  5629 raft_consensus.cc:2410] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:25:26.772765  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:26.798169  5555 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
W20260709 12:25:26.801642  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1898) took 2435 ms (client timeout 2865 ms). Trace:
I20260709 12:25:26.802793  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 394) took 2894 ms. Trace:
I20260709 12:25:26.807775  5630 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:25:26.693683  8183 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:26.695006  8224 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:26.691597  8229 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:26.802605  5629 rpcz_store.cc:269] 0709 12:25:24.366427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.366529 (+   102us) service_pool.cc:224] Handling call
0709 12:25:26.801358 (+2434829us) spinlock_profiling.cc:243] Waited 2.38 s on lock 0x7b5400242054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:26.801561 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5463130368}
I20260709 12:25:26.818065  5628 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:25:26.906772  5703 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:26.813207  5631 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:26.809549  5552 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:25:26.846107  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.915800  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.797446  5561 rpcz_store.cc:269] 0709 12:25:24.794236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.118900 (+324664us) service_pool.cc:224] Handling call
0709 12:25:26.788234 (+1669334us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b54001bec54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:26.788432 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3746974080}
I20260709 12:25:24.785887  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:26.810392  5627 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
W20260709 12:25:26.992834  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.995172  5599 connection.cc:256] RPC call timeout handler was delayed by -1.2714379389999522s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15410000000000001s for a response
I20260709 12:25:26.812474  5559 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:26.809852  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1571) took 1919 ms. Trace:
I20260709 12:25:27.015281  5556 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:26.803385  5558 rpcz_store.cc:276] 0709 12:25:23.908336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.934556 (+2026220us) service_pool.cc:224] Handling call
0709 12:25:25.934851 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.756471 (+821620us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.802185 (+ 45714us) spinlock_profiling.cc:243] Waited 820 ms on lock 0x7b54002283d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:26.802689 (+   504us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1885211520}
I20260709 12:25:27.018635  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:26.808410  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:26.796142  5557 rpcz_store.cc:269] 0709 12:25:24.970305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.256811 (+286506us) service_pool.cc:224] Handling call
0709 12:25:26.782944 (+1526133us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b54000ea6d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:26.783198 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1705687936}
I20260709 12:25:27.022383  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:27.015286  5630 rpcz_store.cc:276] 0709 12:25:24.889958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.899516 (+  9558us) service_pool.cc:224] Handling call
0709 12:25:26.806886 (+1907370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.808434 (+  1548us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.809505 (+  1071us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b5400242050. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:26.809749 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4385443584}
I20260709 12:25:26.980691  5456 catalog_manager.cc:5697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:27.028052  5561 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:27.014652  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1638) took 1971 ms. Trace:
I20260709 12:25:26.968869  8043 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:27.030774  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:26.916261  7996 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:27.031931  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:26.905952  8041 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:27.037147  5557 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:27.037032  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:27.037750  5705 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:27.037415  8041 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.038051  5561 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:27.041116  8041 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:27.042342  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:27.042797  5599 connection.cc:256] RPC call timeout handler was delayed by -1.1165923369999291s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1769s for a response
I20260709 12:25:27.043536  5710 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:27.044519  8206 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.045466  7858 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.045974  8119 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.046137  7858 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.046887  8119 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.045460  8206 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.049753  8092 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.050076  7996 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.050489  8092 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.051985  8206 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:27.052891  8124 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:27.053517  7858 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:26.966822  6275 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:27.059092  8092 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:26.993741  8089 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260709 12:25:27.046888  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:27.061806  8092 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.062767  8092 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.063354  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:27.064563  8078 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:27.065191  8088 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:27.068288  5456 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:27.069387  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:27.070268  5630 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:27.071061  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:27.072913  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.073724  5702 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:27.074734  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.075114  8238 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:27.076879  5707 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:27.077205  8238 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:25:27.078256  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:27.079501  5636 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:27.081012  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.082314  5703 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 3: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:27.083911  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:27.084825  5706 raft_consensus.cc:2376] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5. Re-sending same reply.
I20260709 12:25:27.085274  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:27.086719  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:26.675748  8051 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:27.092798  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 457453us
W20260709 12:25:27.093561  6234 connection.cc:256] RPC call timeout handler was delayed by -1.0542243059999237s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16510000000000002s for a response
I20260709 12:25:26.666260  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:27.093631  5456 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:27.083618  8030 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:25:27.037298  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.035315  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:27.101154  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.101429  5557 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:27.102782  5560 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:27.105454  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.106735  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.107493  8078 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.107635  8230 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.106942  8158 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.064575  8124 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:27.108731  8102 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:27.121641  5705 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:25:27.121403  8102 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.123548  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:27.126971  8053 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:27.127789  8216 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:27.108314  8078 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.128950  8053 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:27.129331  8195 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:25:27.108314  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.906185  8200 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:27.031618  5552 rpcz_store.cc:276] 0709 12:25:25.043112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.927705 (+884593us) service_pool.cc:224] Handling call
0709 12:25:26.767286 (+839581us) spinlock_profiling.cc:243] Waited 6.06 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:27.014579 (+247293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13930880}
I20260709 12:25:27.114748  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:27.115694  5553 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:27.115790  5558 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:25:27.113782  5555 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
W20260709 12:25:27.136739  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.139632  5630 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:27.142951  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:27.148783  5561 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:27.114262  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:27.112727  6271 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:25:27.116196  8229 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:27.128944  8216 consensus_queue.cc:1243] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:25:27.131125  7996 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:27.111459  8030 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:27.130360  8053 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.137501  8092 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:27.164814  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:27.169842  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:27.170637  8210 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:27.170768  5709 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:27.171167  8210 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:27.108876  8158 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:27.173514  7900 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.580s	user 0.007s	sys 0.015s
I20260709 12:25:27.174387  8158 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:27.174675  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:27.172449  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.163508  5556 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:27.184108  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:27.108332  8230 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.189615  8030 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.191644  8230 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:27.112555  8043 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.194900  5554 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:27.201622  5636 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:25:27.199641  5553 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:27.197894  5630 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:27.199028  5558 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:25:27.199841  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:27.197137  6234 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:27.202615  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:27.195518  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:27.197037  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:27.218375  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:27.202078  6271 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:25:27.194765  8043 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:27.192317  8030 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:27.182363  8102 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:27.197789  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:27.232916  5552 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:25:27.237684  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:27.235702  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:27.243856  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1654) took 1072 ms. Trace:
W20260709 12:25:27.230376  5676 leader_election.cc:336] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.012s (SENT)
I20260709 12:25:27.244374  5552 rpcz_store.cc:276] 0709 12:25:26.167603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.135979 (+968376us) service_pool.cc:224] Handling call
0709 12:25:27.203237 (+ 67258us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.239342 (+ 36105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.239618 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:27.232190  6271 pending_rounds.cc:85] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 6
I20260709 12:25:27.248669  5709 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:25:27.239162  8127 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.535s	user 0.012s	sys 0.006s
I20260709 12:25:27.247752  5555 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:27.249814  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:27.257946  5558 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:27.256250  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59382.6us
W20260709 12:25:27.268994  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.259378  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:27.285933  8193 raft_consensus.cc:2764] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:25:27.226432  8200 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.289512  8030 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:27.291815  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:27.292747  5634 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:27.293303  5630 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:25:27.293354  5633 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:27.293910  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:27.294489  5634 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:27.294909  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:27.295820  5633 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
W20260709 12:25:27.296423  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 83739.2us
I20260709 12:25:27.297299  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:27.298884  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:27.299094  8195 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:27.299968  8195 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.300671  8229 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:27.301839  8207 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:27.302260  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:27.304724  5552 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:25:27.309300  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 96481.9us
W20260709 12:25:27.311849  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.294381  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:27.323194  8238 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:27.327705  8078 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:27.286062  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:27.310439  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.329480  5627 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:25:27.250859  6271 pending_rounds.cc:107] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:25:27.256400  6273 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:25:27.253818  8119 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:27.345424  5630 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:25:27.302345  8207 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:27.350628  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.356020  5633 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:27.358028  5702 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:27.343024  6271 raft_consensus.cc:2889] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:27.356715  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:27.365943  5627 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:27.364341  5636 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:27.356554  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:27.300581  8206 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.053s
I20260709 12:25:27.363202  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:27.372691  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:27.376490  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:25:27.377682  7933 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.756s	user 0.001s	sys 0.017s
I20260709 12:25:27.373624  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.381556  8200 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:27.382234  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 185223us
I20260709 12:25:27.386075  5703 raft_consensus.cc:2410] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:25:27.390060  8210 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.342027  6273 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:27.417232  5709 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260709 12:25:27.395385  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55576.9us
I20260709 12:25:27.352057  8207 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:27.353745  8088 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.371181  5635 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:27.405529  5634 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:27.354444  7858 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:27.412933  5558 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:27.383910  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:27.421909  8208 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:27.422582  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:27.419417  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:27.437469  5554 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:27.474418  5628 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:27.353359  8216 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:27.479171  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:27.486346  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:27.490630  5706 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: [4.4-6.5]
I20260709 12:25:27.485834  8216 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.481734  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:27.499217  8216 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:27.499774  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 80803.5us
I20260709 12:25:27.500044  8050 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:27.500211  5702 raft_consensus.cc:2410] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:27.478016  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.501776  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.502293  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.503733  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:27.489842  8200 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:27.505182  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.505409  8273 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260709 12:25:27.505071  8200 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:27.505895  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.506716  8144 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.506917  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:27.507233  8144 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.507694  8200 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:27.508798  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.508998  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 125446us
I20260709 12:25:27.480473  8238 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 12:25:27.502539  5702 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:27.512732  5705 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:27.514225  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.516717  5675 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:27.476959  8078 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:27.477051  8208 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.524309  5705 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:27.526937  5634 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20260709 12:25:27.532428  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.521900  5456 catalog_manager.cc:5697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:27.539608  8270 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.345s	user 0.002s	sys 0.001s
I20260709 12:25:27.523219  8238 consensus_queue.cc:1243] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:25:27.518824  8230 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.547438  5558 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:27.535604  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:27.553085  5707 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:27.554257  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:27.556432  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 180013us
I20260709 12:25:27.561858  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:27.562685  5556 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:27.563177  5631 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:27.489542  8088 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.564581  8230 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.566920  8230 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:27.567101  8047 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.026s
I20260709 12:25:27.553730  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:27.581436  5555 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:27.531960  8208 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:27.584635  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:27.585834  5709 raft_consensus.cc:2410] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:25:27.582688  5702 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:27.581064  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:27.589468  8204 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.590245  8204 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.543870  8078 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:27.549784  8200 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:27.605302  7858 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.611230  5552 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:27.614367  5560 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:27.615075  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1647) took 1184 ms. Trace:
I20260709 12:25:27.615509  5560 rpcz_store.cc:276] 0709 12:25:26.430752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.764169 (+333417us) service_pool.cc:224] Handling call
0709 12:25:27.614998 (+850829us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20736}
I20260709 12:25:27.585484  8193 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.566941  8088 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:27.618997  5635 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
W20260709 12:25:27.615164  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 316371us
I20260709 12:25:27.623225  5559 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:25:27.641650  5706 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 12:25:27.602092  6275 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:27.651536  5706 pending_rounds.cc:85] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 2
I20260709 12:25:27.660563  5706 pending_rounds.cc:107] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 12:25:27.660919  5706 raft_consensus.cc:2889] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:27.662858  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 227366us
W20260709 12:25:27.667603  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.545390  6273 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.659612  8030 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:27.684636  5701 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:25:27.685235  5701 pending_rounds.cc:85] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 2
I20260709 12:25:27.685575  5701 pending_rounds.cc:107] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:25:27.685932  5701 raft_consensus.cc:2889] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:27.687884  8183 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:27.624305  7858 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.682296  5599 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:27.676376  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:27.699592  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:27.688525  8183 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:25:27.703943  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 202326us
I20260709 12:25:27.706727  5560 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
W20260709 12:25:27.709400  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 199528us
I20260709 12:25:27.681838  8200 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:27.643978  8193 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.714444  8053 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:27.714986  8053 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:27.715139  8275 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.311s	user 0.002s	sys 0.000s
I20260709 12:25:27.715386  8053 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:27.717780  8124 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.609s	user 0.008s	sys 0.005s
I20260709 12:25:27.718216  8124 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:25:27.723826  8047 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.681344  8224 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:25:27.712324  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63338.8us
I20260709 12:25:27.714300  8158 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:27.730387  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:27.706311  8183 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:27.707782  8051 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.009s	sys 0.017s
W20260709 12:25:27.733189  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.750083  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:27.753859  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.752492  8053 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.757793  7858 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:27.758291  8053 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:27.760417  8195 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:27.761291  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 530) took 2700 ms. Trace:
I20260709 12:25:27.761686  6272 rpcz_store.cc:276] 0709 12:25:25.061195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.061371 (+   176us) service_pool.cc:224] Handling call
0709 12:25:25.124717 (+ 63346us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:25.185088 (+ 60371us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:25.188288 (+  3200us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:26.636873 (+1448585us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b2c00040818. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:25:26.824104 (+187231us) log.cc:844] Serialized 506 byte log entry
0709 12:25:27.202601 (+378497us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:27.215241 (+ 12640us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.234062 (+ 18821us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:27.725246 (+491184us) raft_consensus.cc:1685] finished
0709 12:25:27.729129 (+  3883us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.761103 (+ 31974us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17346,"raft.queue_time_us":35015,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":8074,"spinlock_wait_cycles":28385024,"thread_start_us":323087,"threads_started":1,"wal-append.queue_time_us":382975}
I20260709 12:25:27.764489  8280 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.027s
I20260709 12:25:27.766125  8285 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
W20260709 12:25:27.739995  8051 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.010s	sys 0.018s
W20260709 12:25:27.767760  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77427.9us
I20260709 12:25:27.739467  6273 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:27.736171  8206 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:27.776193  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:27.792120  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.724689  8047 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.793705  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:27.805907  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:27.807161  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1653) took 1639 ms. Trace:
W20260709 12:25:27.810525  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56378.3us
I20260709 12:25:27.808074  5559 rpcz_store.cc:276] 0709 12:25:26.167311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.125728 (+958417us) service_pool.cc:224] Handling call
0709 12:25:27.622965 (+497237us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.623696 (+   731us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.806912 (+183216us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b54001be750. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:27.807081 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1129740032}
I20260709 12:25:27.776242  8050 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:27.781085  8206 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.818691  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:27.820524  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:27.821755  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:27.821938  8050 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.822618  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:27.823287  5558 raft_consensus.cc:2376] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7. Re-sending same reply.
I20260709 12:25:27.831223  5560 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:27.832870  8289 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:27.833355  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:27.834554  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:27.835117  5558 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:27.736852  8078 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:27.855074  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.788908  8280 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
W20260709 12:25:27.858282  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 321506us
W20260709 12:25:27.858880  8282 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:25:27.859194  8200 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:27.879099  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 81765.2us
I20260709 12:25:27.815851  8030 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.859812  8293 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.860803  8300 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:25:27.861478  8216 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:27.871670  7858 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:27.889475  8216 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:27.889623  7858 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:27.793085  7933 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:25:27.833375  8289 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:27.869547  8047 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:27.870333  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 99347.7us
I20260709 12:25:27.776867  8158 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 12:25:27.872794  5552 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:27.873569  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:27.819362  8206 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:27.885816  8293 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.887197  8300 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:27.893113  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.894085  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.895414  7933 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:27.901827  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.902391  7897 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.895828  7858 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.903275  8144 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:27.903752  8292 raft_consensus.cc:2764] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:25:27.905495  8124 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.906002  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.905606  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:27.920897  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.915285  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 145816us
W20260709 12:25:27.922530  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.925632  5706 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:25:27.910840  8229 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.610s	user 0.011s	sys 0.003s
I20260709 12:25:27.930130  5707 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:27.900130  8285 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:27.933197  7897 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.934305  8193 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:27.935165  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.942586  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:27.943174  5558 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:27.896198  8289 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:27.885131  8200 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.945375  8300 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:27.949314  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:27.912626  8047 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:27.912724  7933 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:27.960026  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.948590  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1655) took 1780 ms. Trace:
I20260709 12:25:27.970479  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:25:27.971237  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.959915  8127 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:25:27.972069  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.980048  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:27.960727  8229 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:27.978513  5557 rpcz_store.cc:276] 0709 12:25:26.167839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.140284 (+972445us) service_pool.cc:224] Handling call
0709 12:25:27.180554 (+ 40270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.239727 (+ 59173us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:25:27.615674 (+375947us) log.cc:844] Serialized 14 byte log entry
0709 12:25:27.922680 (+307006us) log.cc:844] Serialized 14 byte log entry
0709 12:25:27.924343 (+  1663us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:27.924431 (+    88us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:27.931110 (+  6679us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.931472 (+   362us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.948510 (+ 17038us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10715,"prepare.queue_time_us":368133,"prepare.run_cpu_time_us":139,"prepare.run_wall_time_us":6274,"raft.queue_time_us":11853,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":36079,"spinlock_wait_cycles":14564736,"thread_start_us":569425,"threads_started":2}
I20260709 12:25:27.971613  8285 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:27.989270  5558 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 6: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:27.982807  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:27.985355  5705 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:27.996724  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:27.997771  5552 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:27.984800  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:28.002511  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:27.947882  8124 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:27.982079  8088 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:27.970063  8158 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.008005  5703 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:28.010911  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:27.975173  8047 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:27.967798  8216 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260709 12:25:28.014212  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:28.014493  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 56722.5us
W20260709 12:25:28.014739  8286 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:25:28.014982  5631 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:25:28.016763  7897 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:28.015744  5635 raft_consensus.cc:2955] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:27.983639  8127 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:28.017320  8144 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:28.017467  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:28.018536  5706 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:25:28.019073  8124 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:28.019277  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:28.020331  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 138714us
I20260709 12:25:28.015802  5704 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:25:27.985448  8293 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:28.024094  8300 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:28.026999  8207 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.545s	user 0.008s	sys 0.002s
I20260709 12:25:28.028064  8285 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.029842  5627 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:28.030318  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:28.101351  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.102818  5552 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:28.103670  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:28.103731  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:28.103749  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:28.068012  5631 pending_rounds.cc:85] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 6
W20260709 12:25:28.108687  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 88309.8us
I20260709 12:25:28.012768  8193 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.115217  5707 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:28.011313  7933 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.116847  5636 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:28.248835  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:28.261344  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1664) took 1286 ms. Trace:
I20260709 12:25:28.261921  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:28.120911  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:28.274561  5558 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:28.276885  5634 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:28.271839  5561 rpcz_store.cc:276] 0709 12:25:26.975441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.339842 (+364401us) service_pool.cc:224] Handling call
0709 12:25:27.369843 (+ 30001us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:27.475004 (+105161us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.475053 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:27.487712 (+ 12659us) log.cc:844] Serialized 52 byte log entry
0709 12:25:27.777491 (+289779us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:25:27.777667 (+   176us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.777848 (+   181us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.261085 (+483237us) raft_consensus.cc:1685] finished
0709 12:25:28.261107 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.261270 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":38701,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":65133,"thread_start_us":245569,"threads_started":1,"wal-append.queue_time_us":513756}
I20260709 12:25:28.107930  8127 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:28.121153  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 153698us
I20260709 12:25:28.286569  5553 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:28.290196  5552 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:28.278522  5631 pending_rounds.cc:107] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:25:28.273661  5709 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
W20260709 12:25:28.284363  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 171841us
I20260709 12:25:28.291086  5631 raft_consensus.cc:2889] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:28.278187  5557 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:28.295671  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 474016us
I20260709 12:25:28.114192  8299 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:28.306829  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58102us
W20260709 12:25:28.307214  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.272436  6273 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:25:28.309538  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.311405  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.313004  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:28.114944  8088 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:28.256043  8306 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:28.293417  8293 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:28.288403  7897 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.345990  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:28.294485  8312 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:28.310493  7933 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:28.114482  8229 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.237s
I20260709 12:25:28.275084  8158 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.297722  8124 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 12:25:28.312455  8193 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.436077  5557 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:28.387508  5552 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:25:28.390231  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.357946  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.374583  5631 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 12:25:28.377463  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.358467  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.442822  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.390143  5553 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:25:28.411140  5703 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:28.387720  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:28.355980  6236 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:25:28.437903  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 78856.2us
W20260709 12:25:28.372648  8192 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
W20260709 12:25:28.381202  8310 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 12:25:28.372476  8305 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260709 12:25:28.116535  8292 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.311s
I20260709 12:25:28.378870  8053 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:28.371373  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 538) took 1164 ms. Trace:
I20260709 12:25:28.289361  8207 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:28.471063  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 115159us
W20260709 12:25:28.466638  8206 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.543s	user 0.011s	sys 0.000s
W20260709 12:25:28.457301  8208 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.846s	user 0.011s	sys 0.000s
W20260709 12:25:28.454551  8183 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.706s	user 0.012s	sys 0.000s
I20260709 12:25:28.396528  8306 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:25:28.471624  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 117100us
W20260709 12:25:28.491791  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.018s	sys 0.018s
I20260709 12:25:28.495291  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:25:28.501879  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.018s	sys 0.018s
W20260709 12:25:28.502514  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 596978us
I20260709 12:25:28.370049  7897 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:28.470602  8053 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:28.513365  5710 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:25:28.496968  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 187503us
I20260709 12:25:28.468142  8200 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:28.355556  8047 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:28.470932  6273 rpcz_store.cc:276] 0709 12:25:27.193988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.213615 (+ 19627us) service_pool.cc:224] Handling call
0709 12:25:27.248988 (+ 35373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:28.318944 (+1069956us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.354263 (+ 35319us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:28.451623  8144 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:28.524610  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:28.499125  8206 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.525175  8144 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.457182  8311 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.353192  8299 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.493945  8124 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:28.540781  8124 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.541378  8124 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:28.542474  7897 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:28.542315  8311 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:28.543817  6234 connection.cc:256] RPC call timeout handler was delayed by -1.6471792679999453s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17010000000000003s for a response
I20260709 12:25:28.544298  8316 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:25:28.544727  6234 connection.cc:256] RPC call timeout handler was delayed by -1.6444792679999409s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1704s for a response
W20260709 12:25:28.545815  8308 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260709 12:25:28.546203  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 414) took 1450 ms. Trace:
I20260709 12:25:28.546641  8047 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:28.547520  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1649) took 1298 ms. Trace:
I20260709 12:25:28.546707  5554 rpcz_store.cc:276] 0709 12:25:27.095673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.561281 (+465608us) service_pool.cc:224] Handling call
0709 12:25:27.561716 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.607388 (+ 45672us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.607438 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:27.627765 (+ 20327us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.915009 (+287244us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:27.915056 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.915129 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.545969 (+630840us) raft_consensus.cc:1685] finished
0709 12:25:28.545992 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.546139 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30426,"raft.queue_time_us":39652,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":22778,"thread_start_us":237099,"threads_started":1,"wal-append.queue_time_us":343544}
I20260709 12:25:28.530027  8206 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:28.494843  8229 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:25:28.535202  8301 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260709 12:25:28.549180  8204 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:28.549737  8322 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:28.550539  8322 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.550926  8322 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:28.549764  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:28.551185  8301 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.547981  5556 rpcz_store.cc:276] 0709 12:25:27.249316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.564416 (+315100us) service_pool.cc:224] Handling call
0709 12:25:27.565022 (+   606us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:27.619686 (+ 54664us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.619729 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:27.680116 (+ 60387us) log.cc:844] Serialized 506 byte log entry
0709 12:25:28.110036 (+429920us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:28.110123 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.110222 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.547283 (+437061us) raft_consensus.cc:1685] finished
0709 12:25:28.547304 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.547449 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46923,"raft.queue_time_us":47180,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":9153,"thread_start_us":384950,"threads_started":1,"wal-append.queue_time_us":548356}
I20260709 12:25:28.551789  8301 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:28.553151  8309 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
W20260709 12:25:28.553658  6275 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 534) took 1440 ms (client timeout 1611 ms). Trace:
W20260709 12:25:28.563912  6275 rpcz_store.cc:269] 0709 12:25:27.113330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.113899 (+   569us) service_pool.cc:224] Handling call
0709 12:25:28.553451 (+1439552us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b540020c454. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:28.553566 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1007346432}
I20260709 12:25:28.564607  8193 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:28.520520  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:28.564399  8301 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:28.530879  8200 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:25:28.571640  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.575491  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.572764  7897 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:25:28.510380  8208 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:28.584124  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:28.584926  5556 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:28.586241  8208 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:28.587738  8311 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:28.588593  8301 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:25:28.589145  8318 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:25:28.589303  8301 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:28.588231  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.591536  6234 connection.cc:256] RPC call timeout handler was delayed by -1.1656792680000763s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22360000000000002s for a response
W20260709 12:25:28.592787  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 92777.5us
I20260709 12:25:28.510190  8183 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:28.583069  8327 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:25:28.530694  8088 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.599237  6236 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:28.592041  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:28.560233  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:28.601567  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:28.665026  8325 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:28.665571  8325 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.665931  8325 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:28.668466  5524 leader_election.cc:336] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.806s (SENT)
I20260709 12:25:28.668762  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:28.670310  8314 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.209s	user 0.000s	sys 0.000s
W20260709 12:25:28.671432  8302 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.433s	user 0.001s	sys 0.001s
W20260709 12:25:28.672860  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 73893.1us
W20260709 12:25:28.672930  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 74704.6us
I20260709 12:25:28.593061  8316 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:28.678557  5556 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
W20260709 12:25:28.683131  5524 leader_election.cc:336] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.812s (SENT)
W20260709 12:25:28.683418  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 188193us
I20260709 12:25:28.685109  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:25:28.685580  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 84117.6us
I20260709 12:25:28.687543  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:28.689334  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.612932  8200 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:28.695406  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:28.729199  5631 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:28.613653  8216 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.111s
I20260709 12:25:28.683882  8193 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:25:28.744274  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.685886  8238 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:28.732471  8078 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:28.738886  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:28.740105  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59956.7us
I20260709 12:25:28.740983  8316 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.742003  8321 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:28.690265  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:28.605747  8317 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.128s
W20260709 12:25:28.758157  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.758626  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 231665us
I20260709 12:25:28.743392  8193 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.689177  7858 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 12:25:28.742865  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:28.748674  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:28.791254  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 108186us
I20260709 12:25:28.754156  8183 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:28.796633  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 96840.1us
I20260709 12:25:28.746340  8195 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 12:25:28.763572  8208 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260709 12:25:28.806229  8324 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.118s
I20260709 12:25:28.808478  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:28.808552  8329 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 12:25:28.810456  8330 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:28.770581  8193 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:28.765838  8207 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:28.766248  8301 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:28.763978  8321 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:28.790621  8158 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:28.845451  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 88690.8us
I20260709 12:25:28.845664  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:28.841646  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:28.852126  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 57797.3us
I20260709 12:25:28.689816  8306 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.797431  8300 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.863665  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:28.876673  8208 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:28.877170  8208 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:25:28.877271  8127 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.583s	user 0.005s	sys 0.006s
I20260709 12:25:28.877875  8127 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:28.878129  8208 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:28.879388  8208 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:28.881893  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:28.882786  6271 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:28.825662  8301 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.826005  8329 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:28.791038  8078 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:25:28.822968  8307 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:28.843510  8238 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:25:28.886220  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 89986us
W20260709 12:25:28.900681  5524 leader_election.cc:336] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.869s (SENT)
I20260709 12:25:28.884485  6272 raft_consensus.cc:2410] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:25:28.842322  8330 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:28.913498  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:28.907644  8300 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.920511  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:28.922425  8300 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:28.922447  8328 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:28.924149  8328 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:28.925635  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:28.926376  8299 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:28.906890  8194 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.111s
I20260709 12:25:28.925236  8328 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:28.917954  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 175692us
I20260709 12:25:28.935101  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:28.918699  6234 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.701s (SENT) [suppressed 36 similar messages]
I20260709 12:25:28.936802  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:28.913532  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 50613.2us
W20260709 12:25:28.944468  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45341.4us
I20260709 12:25:28.864542  8200 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.925236  8330 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.934643  8124 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.954456  8124 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:28.931950  8238 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:28.955973  8238 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.961839  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:28.974623  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 119959us
I20260709 12:25:28.942324  8127 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:28.978356  8124 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:28.978233  8344 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:28.942749  8338 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:28.930938  8301 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:28.989500  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.989825  5702 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:28.933233  7858 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
W20260709 12:25:28.988332  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 74921.8us
I20260709 12:25:28.993970  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:28.994481  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:28.973511  8194 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.930513  8329 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:28.918542  8307 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:28.973160  8158 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:25:28.999675  5524 leader_election.cc:336] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.131s (SENT)
I20260709 12:25:28.998821  5709 raft_consensus.cc:2376] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5. Re-sending same reply.
W20260709 12:25:29.002331  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.989207  8338 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:28.972082  8321 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:29.053227  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.130951  5709 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260709 12:25:29.104832  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 105894us
I20260709 12:25:29.138283  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:29.142498  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 144663us
I20260709 12:25:28.998649  8127 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:29.144140  8127 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.144184  5707 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [3.5-5.7]
I20260709 12:25:29.145164  8127 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:29.013094  8194 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:29.090363  8158 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:29.046959  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:29.087777  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 169068us
W20260709 12:25:29.020426  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.089378  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:29.108104  8289 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.136s	user 0.011s	sys 0.003s
W20260709 12:25:29.013020  8334 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 12:25:28.996430  8124 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:29.167520  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:29.167361  8158 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:29.168288  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:29.193600  5632 raft_consensus.cc:2410] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:25:29.168403  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:29.171499  8322 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.620s	user 0.002s	sys 0.009s
I20260709 12:25:29.172576  8307 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:29.137877  8338 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:29.147996  8102 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:25:29.176514  8335 log.cc:927] Time spent T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.235s	user 0.000s	sys 0.001s
I20260709 12:25:29.177234  8127 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:29.178135  8301 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:29.180346  6234 leader_election.cc:336] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.704s (SENT)
I20260709 12:25:29.147655  8346 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:29.182986  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:29.175752  8325 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.509s	user 0.009s	sys 0.000s
W20260709 12:25:29.207082  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.013228  8347 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:25:29.211095  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:29.180941  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:29.198364  8336 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.216s	user 0.000s	sys 0.001s
I20260709 12:25:29.232786  5702 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:29.138893  8337 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.251s
W20260709 12:25:29.234472  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 264246us
I20260709 12:25:29.241572  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:29.243978  8088 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:29.254607  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:29.255089  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:29.255671  5630 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:29.259446  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 76660.4us
I20260709 12:25:29.260386  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:29.163726  8289 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:29.234995  8347 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:29.267120  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:29.271615  5707 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:29.278566  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:29.281210  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 290932us
I20260709 12:25:29.301339  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:29.302765  5634 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:25:29.304023  8348 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:25:29.304613  8342 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.263s	user 0.001s	sys 0.000s
W20260709 12:25:29.305594  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 125295us
I20260709 12:25:29.247527  8301 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:25:29.210728  8322 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:29.239125  8325 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:29.273418  8347 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:25:29.329226  8366 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:29.302544  8346 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:29.337791  5707 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 2. Current term is 3. Ops: []
I20260709 12:25:29.340919  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:29.316004  7858 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:25:29.426805  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 237348us
W20260709 12:25:29.431780  8047 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.884s	user 0.014s	sys 0.000s
W20260709 12:25:29.424834  8335 log.cc:927] Time spent T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:25:29.425426  7858 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.479738  7858 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:29.476940  5524 leader_election.cc:336] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.207s (SENT)
I20260709 12:25:29.432389  8329 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:29.478705  8357 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:25:29.475719  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:29.403585  8356 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.482064  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:29.478065  8350 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.484170  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:29.487731  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 419) took 1715 ms. Trace:
W20260709 12:25:29.488593  8359 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
W20260709 12:25:29.489445  8351 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.164s	user 0.000s	sys 0.000s
W20260709 12:25:29.491897  8291 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.558s	user 0.007s	sys 0.002s
I20260709 12:25:29.487800  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:29.486284  5706 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:29.499611  8356 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:29.494864  5630 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:29.493383  8124 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:25:29.494871  5558 rpcz_store.cc:276] 0709 12:25:27.772335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.328929 (+556594us) service_pool.cc:224] Handling call
0709 12:25:28.448061 (+119132us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:28.531354 (+ 83293us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:28.531476 (+   122us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:28.574942 (+ 43466us) log.cc:844] Serialized 52 byte log entry
0709 12:25:28.732721 (+157779us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:28.738886 (+  6165us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.742943 (+  4057us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.743048 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:29.487484 (+744436us) raft_consensus.cc:1685] finished
0709 12:25:29.487509 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:29.487645 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6469,"prepare.queue_time_us":12289,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":3034,"raft.queue_time_us":114096,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":7596,"spinlock_wait_cycles":4300032,"thread_start_us":209822,"threads_started":2,"wal-append.queue_time_us":179392}
I20260709 12:25:29.501680  8350 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:29.526731  8119 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:25:29.505767  8358 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260709 12:25:29.529970  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 458) took 1518 ms. Trace:
I20260709 12:25:29.530426  5628 rpcz_store.cc:276] 0709 12:25:28.011181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.011992 (+   811us) service_pool.cc:224] Handling call
0709 12:25:29.529722 (+1517730us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54001f2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:29.529876 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2468877568}
I20260709 12:25:29.533728  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:29.534413  5707 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:29.534118  7858 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:29.508030  8325 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:29.483914  8127 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:29.508985  8289 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:29.535408  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:29.716697  7858 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:29.724167  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 445695us
I20260709 12:25:29.700460  5702 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:29.583312  5709 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:25:29.609778  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:29.726121  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.691581  5706 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:29.728967  5706 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:25:29.699884  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 238420us
W20260709 12:25:29.403096  8355 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
W20260709 12:25:29.566018  7933 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.180s	user 0.013s	sys 0.005s
I20260709 12:25:29.715943  5707 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:29.714733  8301 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:25:29.747432  8338 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.509s	user 0.007s	sys 0.000s
W20260709 12:25:29.727967  8208 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.844s	user 0.010s	sys 0.005s
I20260709 12:25:29.368780  8346 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:29.725061  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:29.749414  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:29.770812  8304 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.652s	user 0.003s	sys 0.002s
I20260709 12:25:29.771353  8301 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:29.775686  8193 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.930s	user 0.008s	sys 0.002s
I20260709 12:25:29.779036  8356 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:29.779495  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.425985  8348 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:29.780474  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 540930us
I20260709 12:25:29.789353  6275 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:29.769441  5706 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:29.804267  8206 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.803215  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 541) took 1116 ms. Trace:
W20260709 12:25:29.836460  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.742319  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:30.208137  6280 rpcz_store.cc:276] 0709 12:25:28.686422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.740916 (+ 54494us) service_pool.cc:224] Handling call
0709 12:25:29.580609 (+839693us) spinlock_profiling.cc:243] Waited 2.46 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:29.803143 (+222534us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5668224}
W20260709 12:25:29.795758  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.852217  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:30.213478  5709 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:29.786528  8346 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:29.837152  8350 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:30.290416  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:30.291162  5705 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:30.291309  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.290447  8206 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:30.301955  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:30.293113  5706 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:30.304752  5709 raft_consensus.cc:2376] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8. Re-sending same reply.
I20260709 12:25:29.794996  8193 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:30.311769  8102 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.080s	user 0.001s	sys 0.009s
I20260709 12:25:30.312381  8102 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:30.314769  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:30.316345  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 591303us
I20260709 12:25:30.317652  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:30.318051  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:30.319936  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.321843  5709 raft_consensus.cc:2410] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20260709 12:25:30.328125  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.341440  5702 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:29.613314  6273 raft_consensus.cc:2955] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:25:30.343761  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60580.6us
I20260709 12:25:29.786199  8338 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:30.351998  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:30.338716  8158 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.863s	user 0.038s	sys 0.089s
I20260709 12:25:30.354084  8102 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:25:30.356449  8158 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.039s	sys 0.089s
I20260709 12:25:30.356833  8193 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:30.355195  5709 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:30.338387  8292 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.356951  8102 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:30.315008  8356 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.339867  8229 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.360443  5709 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:25:30.319304  8088 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:25:30.365864  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.090s	user 0.092s	sys 0.149s
W20260709 12:25:30.366319  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.094s	user 0.092s	sys 0.149s
I20260709 12:25:30.367054  5705 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
W20260709 12:25:30.377854  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 60328.6us
I20260709 12:25:30.379113  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:30.380766  5602 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:30.382433  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:30.410873  5707 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.622s	user 0.010s	sys 0.000s
I20260709 12:25:30.411458  5707 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:30.424091  5706 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260709 12:25:30.427119  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:30.427528  5706 pending_rounds.cc:85] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 7
I20260709 12:25:30.428102  5706 pending_rounds.cc:107] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 6.8
W20260709 12:25:30.388010  8347 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.938s	user 0.010s	sys 0.001s
I20260709 12:25:30.428431  5706 raft_consensus.cc:2889] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:30.387588  8158 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:30.387741  8124 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:30.388386  8088 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:30.388271  8301 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:30.433344  8365 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.028s	user 0.010s	sys 0.018s
I20260709 12:25:30.435111  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:30.434728  8321 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.955s	user 0.024s	sys 0.103s
I20260709 12:25:30.364991  8356 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:30.362703  8292 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:30.367147  8229 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:30.423784  8333 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.940s	user 0.017s	sys 0.042s
W20260709 12:25:30.446549  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58469.8us
W20260709 12:25:30.454017  5633 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.113s	user 0.037s	sys 0.014s
W20260709 12:25:30.446619  8365 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.041s	user 0.010s	sys 0.019s
W20260709 12:25:30.457055  5559 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.204s	user 0.072s	sys 0.102s
I20260709 12:25:30.439893  8124 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:30.367100  8348 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:30.440268  8158 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:25:30.448588  8321 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.969s	user 0.025s	sys 0.103s
I20260709 12:25:30.438519  8347 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:30.460417  8170 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.400s	user 0.009s	sys 0.039s
I20260709 12:25:30.477116  8158 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:30.469734  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:30.456578  8371 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:25:30.471968  5559 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.219s	user 0.072s	sys 0.102s
I20260709 12:25:30.497457  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 429) took 1685 ms. Trace:
W20260709 12:25:30.473388  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.980s	user 0.009s	sys 0.041s
I20260709 12:25:30.469288  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:30.469488  5633 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.129s	user 0.038s	sys 0.015s
I20260709 12:25:30.493692  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:30.501459  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.009s	user 0.009s	sys 0.041s
I20260709 12:25:30.473688  8124 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:30.471247  8333 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.987s	user 0.017s	sys 0.043s
I20260709 12:25:30.457445  8301 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260709 12:25:30.506724  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:30.489488  8170 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.562s	user 0.009s	sys 0.043s
W20260709 12:25:30.467799  8306 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.035s	user 0.008s	sys 0.022s
I20260709 12:25:30.508586  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:30.508738  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:30.519111  5526 connection.cc:256] RPC call timeout handler was delayed by -1.2133274610000626s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21150000000000002s for a response
I20260709 12:25:30.448913  8102 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:30.498634  7858 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:30.524150  5705 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
W20260709 12:25:30.522460  5627 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.030s	user 0.034s	sys 0.042s
I20260709 12:25:30.517488  5709 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
W20260709 12:25:30.530640  5627 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.038s	user 0.034s	sys 0.042s
I20260709 12:25:30.497888  5559 rpcz_store.cc:276] 0709 12:25:28.814387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.814593 (+   206us) service_pool.cc:224] Handling call
0709 12:25:28.846290 (+ 31697us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:28.846532 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:25:29.004766 (+158234us) log.cc:844] Serialized 14 byte log entry
0709 12:25:29.252810 (+248044us) log.cc:844] Serialized 14 byte log entry
0709 12:25:30.497072 (+1244262us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:30.497137 (+    65us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:30.497183 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.497257 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:30.497387 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27266,"prepare.queue_time_us":86156,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":97,"raft.queue_time_us":405341,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":5586,"spinlock_wait_cycles":71354496,"thread_start_us":1573438,"threads_started":3}
I20260709 12:25:30.498903  8371 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:30.499141  8367 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:25:30.521945  8306 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.096s	user 0.008s	sys 0.022s
I20260709 12:25:30.545194  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:30.521636  8365 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:30.529964  8350 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:30.530431  7858 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:30.532680  8295 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.573813  8295 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:30.576143  6276 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.805s	user 0.013s	sys 0.017s
W20260709 12:25:30.576610  6276 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.880s	user 0.014s	sys 0.017s
W20260709 12:25:30.577896  8379 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:25:30.579926  8372 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:25:30.581043  8371 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:30.582252  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:30.591210  8312 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:30.592178  8312 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:30.592828  8312 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:30.595422  8312 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:30.598333  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1937) took 1431 ms. Trace:
I20260709 12:25:30.571602  8350 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:30.608130  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.611128  8350 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:30.563304  8338 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:30.572652  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 471) took 1589 ms. Trace:
I20260709 12:25:30.560473  8210 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:30.568015  7858 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:30.598869  5633 rpcz_store.cc:276] 0709 12:25:29.166938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.248341 (+ 81403us) service_pool.cc:224] Handling call
0709 12:25:29.248701 (+   360us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:29.278224 (+ 29523us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:29.278257 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:29.307647 (+ 29390us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.500204 (+1192557us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:30.500249 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.500323 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:30.598047 (+ 97724us) raft_consensus.cc:1685] finished
0709 12:25:30.598076 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:30.598254 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8149,"raft.queue_time_us":14648,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":29882,"thread_start_us":1159329,"threads_started":1,"wal-append.queue_time_us":1227784}
I20260709 12:25:30.639355  8356 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:30.620949  8295 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:30.641287  8347 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:30.612443  5636 rpcz_store.cc:276] 0709 12:25:28.983436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.983539 (+   103us) service_pool.cc:224] Handling call
0709 12:25:28.988452 (+  4913us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:28.996479 (+  8027us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:28.996511 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:29.010662 (+ 14151us) log.cc:844] Serialized 26 byte log entry
0709 12:25:29.425873 (+415211us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:29.425932 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:29.426031 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:30.572374 (+1146343us) raft_consensus.cc:1685] finished
0709 12:25:30.572397 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:30.572582 (+   185us) inbound_call.cc:177] Queueing success response
0709 12:25:30.590696 (+ 18114us) spinlock_profiling.cc:243] Waited 4.73 ms on lock 0x7b48002727d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
Metrics: {"mutex_wait_us":1809,"raft.queue_time_us":13534,"raft.run_cpu_time_us":13851,"raft.run_wall_time_us":1580025,"spinlock_wait_cycles":10873216,"thread_start_us":311620,"threads_started":1,"wal-append.queue_time_us":1357510}
I20260709 12:25:30.653534  5707 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
W20260709 12:25:30.673899  5601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.209s	user 0.015s	sys 0.017s
W20260709 12:25:30.675076  5601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.211s	user 0.015s	sys 0.017s
I20260709 12:25:30.656534  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:30.685182  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 319777us
W20260709 12:25:30.688336  5524 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.846s	user 0.009s	sys 0.027s
W20260709 12:25:30.688880  5524 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.847s	user 0.009s	sys 0.027s
W20260709 12:25:30.689272  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 892748us
I20260709 12:25:30.596109  8312 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:25:30.704663  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:30.704572  5526 leader_election.cc:336] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.563s (SENT)
I20260709 12:25:30.706511  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:30.707067  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:30.697273  8312 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.698997  8347 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:30.708951  8380 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260709 12:25:30.769768  8338 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:30.771430  8380 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:30.768316  8356 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:25:30.773568  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 69011.7us
I20260709 12:25:30.725240  8312 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:30.772715  8347 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:30.784927  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.706095  8229 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:30.808079  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.699304  8383 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:30.775275  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 71221.6us
I20260709 12:25:30.722826  8292 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:30.728277  8333 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.789682  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:30.788561  8380 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:30.811225  8229 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.814571  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:30.815146  8229 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:30.815358  8380 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:30.815819  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1939) took 1636 ms. Trace:
I20260709 12:25:30.816565  8380 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:30.817412  8229 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:30.820530  8229 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.794898  8350 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:30.821026  8229 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:30.829548  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:30.830470  8229 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:30.831935  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:30.834317  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 327649us
I20260709 12:25:30.816232  5627 rpcz_store.cc:276] 0709 12:25:29.179535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.491120 (+311585us) service_pool.cc:224] Handling call
0709 12:25:29.491517 (+   397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:29.491892 (+   375us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:29.491920 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:29.492427 (+   507us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.531015 (+1038588us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:30.531065 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.531132 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:30.815582 (+284450us) raft_consensus.cc:1685] finished
0709 12:25:30.815602 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:30.815754 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":225594,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":1038161,"threads_started":1,"wal-append.queue_time_us":1203596}
W20260709 12:25:30.846251  5601 leader_election.cc:336] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.683s (SENT)
I20260709 12:25:30.853497  8356 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:25:30.825438  8333 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:30.877462  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 95504.3us
W20260709 12:25:30.873034  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 58524.7us
I20260709 12:25:30.872289  5629 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:25:30.878875  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:30.860496  8338 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:25:30.813946  8383 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:30.877067  8346 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.564s	user 0.007s	sys 0.007s
I20260709 12:25:30.853971  8088 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:30.913434  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:30.888144  8030 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:30.915952  8030 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.916303  8030 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:30.923734  8030 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:30.924208  8333 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:30.926149  8030 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:30.874732  8392 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:30.932156  8124 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:30.839972  8350 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:30.950202  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 111530us
I20260709 12:25:30.954924  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:30.956143  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:30.957024  5706 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:30.868976  8386 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20260709 12:25:30.909889  8369 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260709 12:25:30.880456  8348 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260709 12:25:30.915056  8346 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:30.963776  6280 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:30.963920  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:30.964341  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:30.965960  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:30.961740  6273 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.533s	user 0.013s	sys 0.000s
I20260709 12:25:30.952641  8393 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:30.954468  8344 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:30.973315  6272 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:30.988168  8365 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:30.989231  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:30.989305  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:30.989532  8365 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:30.990140  8124 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:30.990411  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:30.990589  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:30.991842  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:30.992496  8398 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.993170  8398 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:30.994732  8158 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.504s	user 0.008s	sys 0.003s
I20260709 12:25:30.995319  8398 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:30.970031  8371 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.027s
I20260709 12:25:30.995985  8319 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:25:30.975764  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:30.996156  8170 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:25:30.978057  8388 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 12:25:30.997159  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:30.997787  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:30.969903  8384 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 12:25:30.998317  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 548) took 1432 ms. Trace:
I20260709 12:25:30.998914  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:30.981860  6274 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260709 12:25:30.983367  8312 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:30.999619  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 551) took 1418 ms. Trace:
I20260709 12:25:30.999843  6274 pending_rounds.cc:85] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 7
I20260709 12:25:31.000171  6274 pending_rounds.cc:107] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 12:25:31.000661  6274 raft_consensus.cc:2889] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:30.986171  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:31.001371  8324 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.970160  8380 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:31.001852  8324 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.002494  8338 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:31.003248  5702 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:31.003091  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:31.004559  8383 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:31.005045  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:30.979318  8393 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.974050  6271 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:31.016616  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:31.017302  7858 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.017829  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:31.018807  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:30.990631  8347 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:31.000162  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:31.005313  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1818) took 3131 ms. Trace:
I20260709 12:25:31.005880  8346 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:31.020210  5524 connection.cc:256] RPC call timeout handler was delayed by -1.6298975070003507s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1778s for a response
W20260709 12:25:30.983032  5675 connection.cc:256] RPC call timeout handler was delayed by -1.134476493999955s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16410000000000002s for a response
I20260709 12:25:30.982419  8392 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
W20260709 12:25:30.986698  8375 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.509s	user 0.001s	sys 0.001s
I20260709 12:25:31.001394  8312 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:31.000046  6276 rpcz_store.cc:276] 0709 12:25:29.580861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.691622 (+110761us) service_pool.cc:224] Handling call
0709 12:25:29.692118 (+   496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:29.695024 (+  2906us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:29.695052 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:29.696554 (+  1502us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.577062 (+880508us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:30.577118 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.577205 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:30.999440 (+422235us) raft_consensus.cc:1685] finished
0709 12:25:30.999457 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:30.999567 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":813,"raft.queue_time_us":2566,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":135,"thread_start_us":880033,"threads_started":1,"wal-append.queue_time_us":991370}
I20260709 12:25:31.041824  6276 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:25:31.043715  8350 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:31.020597  5701 rpcz_store.cc:276] 0709 12:25:27.873437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.878569 (+  5132us) service_pool.cc:224] Handling call
0709 12:25:27.893834 (+ 15265us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.949114 (+ 55280us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:25:27.982932 (+ 33818us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.147106 (+164174us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:28.955768 (+808662us) spinlock_profiling.cc:243] Waited 84.6 ms on lock 0x7b2c0008bd48. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:25:28.956134 (+   366us) log.cc:844] Serialized 26 byte log entry
0709 12:25:28.956297 (+   163us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:28.995584 (+ 39287us) log.cc:844] Serialized 14 byte log entry
0709 12:25:30.520600 (+1525016us) spinlock_profiling.cc:243] Waited 3.53 ms on lock 0x7b48001701d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:25:31.004896 (+484296us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b54000472a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5ed733 00007f4ffa672dbf 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c
0709 12:25:31.004955 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.005060 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.005106 (+    46us) raft_consensus.cc:1685] finished
0709 12:25:31.005118 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.005239 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34746,"prepare.queue_time_us":209088,"prepare.run_cpu_time_us":135,"prepare.run_wall_time_us":10112,"raft.queue_time_us":126442,"raft.run_cpu_time_us":14948,"raft.run_wall_time_us":2613676,"spinlock_wait_cycles":233651072,"thread_start_us":90704,"threads_started":2,"wal-append.queue_time_us":380799,"wal-append.run_cpu_time_us":5032,"wal-append.run_wall_time_us":1792825}
W20260709 12:25:31.045255  5710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1790) took 3716 ms (client timeout 1616 ms). Trace:
I20260709 12:25:31.045910  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.047047  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.047147  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.047945  5704 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:25:30.995146  8158 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.006827  8292 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:31.009599  8319 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:31.056555  6276 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:31.056986  5706 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:31.055240  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.014339  8393 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:30.998255  8344 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:31.067358  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48849.7us
I20260709 12:25:30.998872  6279 rpcz_store.cc:276] 0709 12:25:29.566316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.566853 (+   537us) service_pool.cc:224] Handling call
0709 12:25:29.723886 (+157033us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:29.764041 (+ 40155us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:29.764088 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:29.788238 (+ 24150us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.669230 (+880992us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:30.669403 (+   173us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.669518 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:30.998134 (+328616us) raft_consensus.cc:1685] finished
0709 12:25:30.998155 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:30.998265 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":59049,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":5307,"thread_start_us":349898,"threads_started":1,"wal-append.queue_time_us":607658}
I20260709 12:25:31.022022  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:31.023303  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:31.069037  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:31.070426  6278 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 4: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:25:31.020823  5524 connection.cc:256] RPC call timeout handler was delayed by -1.5371975070002009s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18810000000000002s for a response
I20260709 12:25:31.025955  7858 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:31.073357  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:31.027066  5634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1933) took 1867 ms (client timeout 2350 ms). Trace:
I20260709 12:25:31.029317  8399 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:25:31.026005  8230 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:25:31.045648  5710 rpcz_store.cc:269] 0709 12:25:27.330531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.354741 (+ 24210us) service_pool.cc:224] Handling call
0709 12:25:31.045003 (+3690262us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400210554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:31.045129 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2543893888}
I20260709 12:25:31.011054  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.047358  6272 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:31.078883  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.079468  5632 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
I20260709 12:25:31.058866  8306 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:31.076678  8346 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:31.083668  6280 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:31.053952  8311 raft_consensus.cc:2955] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:31.083791  8346 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.077203  8230 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:31.076956  8312 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:31.085487  8312 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.085850  8230 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:31.087457  8230 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:31.087602  8393 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:31.069623  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 47690.6us
I20260709 12:25:31.074051  5630 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:31.059011  8158 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:31.074000  5634 rpcz_store.cc:269] 0709 12:25:29.160011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.160261 (+   250us) service_pool.cc:224] Handling call
0709 12:25:31.026845 (+1866584us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54000c1f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:31.026989 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":257508224}
I20260709 12:25:31.076812  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:31.105018  5631 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:31.067691  8292 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.082000  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.082270  8210 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:31.082980  5628 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:31.082741  8344 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:31.084507  8346 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:31.065918  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:31.116453  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:31.116793  5633 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:25:31.117055  8293 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 12:25:31.088229  8312 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:31.117555  8393 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:31.119141  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 444) took 4012 ms. Trace:
W20260709 12:25:31.088586  5524 connection.cc:256] RPC call timeout handler was delayed by -1.3265975070003151s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21150000000000002s for a response
I20260709 12:25:31.119504  5629 rpcz_store.cc:276] 0709 12:25:27.107062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.182940 (+ 75878us) service_pool.cc:224] Handling call
0709 12:25:27.499475 (+316535us) spinlock_profiling.cc:243] Waited 5.08 ms on lock 0x7b54001cf128. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff616ebc 00007f4fff64c08c 00007f4fff63693b 00007f4fff6363d2 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c
0709 12:25:27.499602 (+   127us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.878853 (+3379251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.118866 (+240013us) spinlock_profiling.cc:243] Waited 3.37 s on lock 0x7b54001cf2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:31.119061 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7767374592}
I20260709 12:25:31.121738  5634 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:31.088968  8338 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:31.089198  8380 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.090030  6276 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:31.091536  5635 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:31.088893  5707 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:31.094769  5706 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 12:25:31.097446  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.105387  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:31.109661  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 157724us
I20260709 12:25:31.110325  5704 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:25:31.071422  8386 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:31.064277  5701 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:31.113269  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:31.128178  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:31.128931  5632 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:31.129184  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.130461  5628 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:31.106091  8350 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.135053  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.078519  8348 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:31.110214  8210 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.137667  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.077478  8330 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 12:25:31.139484  5710 raft_consensus.cc:2368] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6.
I20260709 12:25:31.150951  5556 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:31.151611  5628 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:25:31.152340  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.152751  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:31.153035  5556 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:31.150974  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:31.153847  5559 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:31.154911  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:31.155707  5561 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 8: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:31.156378  5554 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:25:31.157320  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.157740  5560 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:31.158514  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:31.159097  5558 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:31.108425  8333 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 12:25:31.148787  6271 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260709 12:25:31.165077  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 96102.9us
I20260709 12:25:31.162902  8348 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.160619  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:31.166501  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:31.167450  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.169066  5632 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:31.169128  5555 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:31.169361  5710 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:31.153726  8210 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:31.172578  5634 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:31.174167  5628 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 12:25:31.170845  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:31.178957  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:31.179829  5635 raft_consensus.cc:2410] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:25:31.174692  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.182905  5710 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:31.131800  8380 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:31.192801  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.193331  5561 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:31.193483  5710 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
W20260709 12:25:31.194131  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.194389  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:31.187008  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.137331  8386 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:31.195080  5630 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:31.195317  5556 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:31.139063  8338 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.165777  8405 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.196270  5629 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:31.196328  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:31.199123  5558 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:31.197193  5631 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:31.150995  8350 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.170862  8347 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.197865  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:31.202186  5552 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:31.202199  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:31.203326  5560 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:31.200074  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:31.174676  8333 consensus_queue.cc:1243] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:25:31.175100  8088 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:31.203933  5554 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:31.201692  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 120803us
I20260709 12:25:31.205085  5524 leader_election.cc:400] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6.
I20260709 12:25:31.205456  5524 leader_election.cc:403] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:25:31.205408  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.207783  8088 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:31.210476  8088 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:31.210677  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.211558  5629 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:31.206387  8347 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:31.214762  8347 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:31.218973  5555 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:31.219388  5554 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:31.227478  5629 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:31.229951  8322 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:31.232869  8332 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6.
I20260709 12:25:31.221590  8405 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:31.238042  5560 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:31.215873  8346 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:31.244230  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52080us
I20260709 12:25:31.244702  5552 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:31.245357  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:31.250708  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.251802  5634 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:31.210693  8338 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:31.228226  8380 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:31.275995  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:31.278614  6275 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 603) took 2603 ms (client timeout 2115 ms). Trace:
I20260709 12:25:31.279039  8346 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:31.280283  8348 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:31.281876  8386 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:31.282555  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.283327  5705 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:31.284546  8210 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.284216  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.287748  5705 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:31.295626  5632 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:31.291915  5710 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:31.297480  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.298458  5710 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:31.301366  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:31.302287  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 120564us
I20260709 12:25:31.273288  8332 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:31.303287  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:31.305734  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.306464  5705 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:31.307698  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:31.308379  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.309007  5632 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:31.309156  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:31.309284  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:31.309865  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:31.310892  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.312481  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:31.312812  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:31.313436  8369 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:31.315392  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:31.305027  8405 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:31.317782  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:31.279057  6275 rpcz_store.cc:269] 0709 12:25:28.675028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.677742 (+  2714us) service_pool.cc:224] Handling call
0709 12:25:31.278404 (+2600662us) spinlock_profiling.cc:243] Waited 993 ms on lock 0x7b5400343ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:31.278516 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2284560256}
I20260709 12:25:31.332962  5636 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 4: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:31.314451  8380 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:31.315071  8338 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:31.358240  5634 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 12:25:31.359547  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.360080  5629 raft_consensus.cc:1217] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Deduplicated request from leader. Original: 3.6->[5.7-6.8]   Dedup: 6.8->[]
I20260709 12:25:31.360679  5629 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: [5.7-6.8]
I20260709 12:25:31.361863  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.365032  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.365115  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.366453  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.367242  5635 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:31.371634  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.362982  5630 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:25:31.346632  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:31.376580  8369 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:31.377144  8369 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.377568  8369 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:31.333127  8210 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:31.380875  8350 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:31.386046  8326 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 2.140s	user 0.003s	sys 0.012s
I20260709 12:25:31.345165  8338 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.387101  8338 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:31.386207  8053 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.383569  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.393671  5706 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:31.397055  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.397869  5636 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:31.344942  8380 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:31.407617  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.347168  8293 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.410581  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:31.412631  8311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:35612 (request call id 87) took 17249 ms. Trace:
I20260709 12:25:31.413453  8194 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:31.413970  8206 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:31.450377  8311 rpcz_store.cc:276] 0709 12:25:14.163239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.163397 (+   158us) service_pool.cc:224] Handling call
0709 12:25:14.225431 (+ 62034us) log.cc:844] Serialized 480 byte log entry
0709 12:25:16.242269 (+2016838us) spinlock_profiling.cc:243] Waited 73.9 ms on lock 0x7b5400180220. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e7dbe 00007f4ffa5f2930 00007f4ffa5f28d8 00007f4ffa5f2858 00007f4ffa5f2813 00007f4ffa5f1a9c 00007f50004dd2f6 00007f50004d7aa8 00007f4ffa62e7cb 00007f4ffa6312c0
0709 12:25:31.412417 (+15170148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9592,"raft.queue_time_us":48621,"raft.run_cpu_time_us":1128,"raft.run_wall_time_us":1128,"spinlock_wait_cycles":169893760,"thread_start_us":260919,"threads_started":2,"wal-append.queue_time_us":299849,"wal-append.run_cpu_time_us":33110,"wal-append.run_wall_time_us":2728129}
I20260709 12:25:31.442433  8293 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:31.453008  5445 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2acd6f5df0204b2295547e021cd8cb9f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 11)
I20260709 12:25:31.455369  8293 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:31.472127  5632 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:25:31.472806  5632 pending_rounds.cc:85] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 2
I20260709 12:25:31.473052  5632 pending_rounds.cc:107] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:25:31.473294  5632 raft_consensus.cc:2889] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:31.474601  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.475692  5635 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:31.476881  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:31.478062  5628 raft_consensus.cc:2410] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:25:31.388127  8053 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:31.481559  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 550) took 1901 ms. Trace:
I20260709 12:25:31.471864  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:31.483286  5559 raft_consensus.cc:2410] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:25:31.484602  6273 rpcz_store.cc:276] 0709 12:25:29.580489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.612435 (+ 31946us) service_pool.cc:224] Handling call
0709 12:25:29.612889 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:29.613168 (+   279us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:25:31.008813 (+1395645us) log.cc:844] Serialized 14 byte log entry
0709 12:25:31.161515 (+152702us) log.cc:844] Serialized 14 byte log entry
0709 12:25:31.480220 (+318705us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b18003e7f40. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:25:31.480398 (+   178us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:31.480459 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:31.480532 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.480627 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.481463 (+   836us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3036,"prepare.queue_time_us":16112,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":4400,"spinlock_wait_cycles":34615168,"thread_start_us":128370,"threads_started":1,"wal-append.queue_time_us":155067}
I20260709 12:25:31.488692  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:31.488759  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.414443  8380 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:31.492193  8404 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:25:31.493201  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 118554us
I20260709 12:25:31.496899  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:31.495077  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:31.507674  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:31.512193  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:31.512449  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:31.513249  5630 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:31.513798  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:31.484409  8344 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:31.492314  8320 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:31.492731  8404 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.509052  8311 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:25:31.546093  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 375289us
W20260709 12:25:31.546149  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 298217us
I20260709 12:25:31.547019  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:31.515724  8230 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:31.518831  8348 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:31.518636  8386 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:31.556738  5630 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:31.538309  8347 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:31.540534  8194 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:31.539584  8320 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:31.567816  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:31.568578  5554 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:31.568989  5675 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:31.569880  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:31.571132  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:31.545549  8311 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:31.544461  8192 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:31.544919  8404 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:31.522429  8402 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.047s
W20260709 12:25:31.573761  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 76900.6us
I20260709 12:25:31.591805  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1834) took 1081 ms. Trace:
I20260709 12:25:31.592233  5702 rpcz_store.cc:276] 0709 12:25:30.510083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.581880 (+ 71797us) service_pool.cc:224] Handling call
0709 12:25:31.591623 (+1009743us) spinlock_profiling.cc:243] Waited 420 ms on lock 0x7b54000e01d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:31.591744 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":966017408}
I20260709 12:25:31.581974  8311 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:31.593289  8194 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:25:31.594693  8350 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:31.595043  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.595744  5561 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:31.595077  8350 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:31.597345  8350 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.597906  8399 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:31.598588  8350 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:31.601085  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.601749  5561 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:31.602207  8295 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:31.602568  8295 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:31.603303  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.603427  8295 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.603933  5561 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
W20260709 12:25:31.606896  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:31.607851  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60924.4us
I20260709 12:25:31.566040  5560 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:31.521292  8338 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:31.552379  8230 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:31.572453  8224 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:25:31.590071  5703 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1799) took 4206 ms (client timeout 1663 ms). Trace:
W20260709 12:25:31.607300  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60057.1us
I20260709 12:25:31.540738  8344 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.596118  8292 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:31.617167  8210 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:31.618119  5558 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:31.567235  8348 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:31.621539  5557 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 12:25:31.617882  5703 rpcz_store.cc:269] 0709 12:25:27.383247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.422024 (+ 38777us) service_pool.cc:224] Handling call
0709 12:25:31.589869 (+4167845us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b54004529d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:31.589983 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1220599680}
I20260709 12:25:31.553359  8170 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 12:25:31.614442  8295 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:31.574409  8346 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260709 12:25:31.579833  8398 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:31.579450  8386 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:31.633340  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.634047  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.563226  8333 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260709 12:25:31.653347  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.639029  5554 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:31.664850  5554 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:25:31.566707  8229 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260709 12:25:31.617394  8230 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:31.553011  8293 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260709 12:25:31.642020  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:25:31.650722  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.655993  5557 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:31.665980  5556 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:31.635226  8210 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.669736  5456 catalog_manager.cc:5697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 3, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 3 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:31.681391  8407 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:31.708329  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.711076  5553 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:31.712370  5558 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
W20260709 12:25:31.716744  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 74856.5us
W20260709 12:25:31.730415  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 88508.1us
W20260709 12:25:31.725886  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 154803us
I20260709 12:25:31.698187  8210 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:31.734987  5557 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:31.732115  6236 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:31.738478  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:31.697472  8306 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.038s
I20260709 12:25:31.698051  8238 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 12:25:31.746337  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.713294  8206 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 12:25:31.683638  8320 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.740506  8293 consensus_queue.cc:1243] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:25:31.740499  8144 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:31.765151  8204 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:25:31.763842  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:31.756542  8294 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 3.854s	user 0.038s	sys 0.081s
I20260709 12:25:31.737699  8347 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:31.775473  5553 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:31.732581  8330 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.017s
I20260709 12:25:31.777899  8404 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:31.784876  5554 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:31.785203  8338 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:31.786608  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.787220  8144 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:31.787518  5557 raft_consensus.cc:2410] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:25:31.788133  8144 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.788535  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.789165  5558 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:31.790139  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:31.791280  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.792682  5553 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: [4.4-5.5]
I20260709 12:25:31.742899  8170 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 12:25:31.796996  8428 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:25:31.796182  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 64159.1us
W20260709 12:25:31.805019  8424 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 12:25:31.808128  8421 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:25:31.816690  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:31.817468  5630 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 4: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:31.747614  8348 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.747840  8386 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:31.818845  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 86189.5us
I20260709 12:25:31.808772  8170 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:31.808869  8428 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:31.821534  6277 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:31.821847  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:31.824772  8437 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 12:25:31.797842  8404 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:31.825066  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:31.843009  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:31.825320  6276 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:25:31.795184  8369 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.852732  8399 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:31.853157  8399 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:31.853816  8369 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:31.853988  8399 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:31.855288  8399 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:31.855930  8332 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.549s	user 0.012s	sys 0.000s
I20260709 12:25:31.826920  8293 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:25:31.856458  8332 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:31.856858  8195 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:31.857256  8195 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:31.844372  8051 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.858215  8051 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.858327  8195 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.859890  8195 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:31.845103  5628 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:31.860580  5628 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:31.819828  8422 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.103s	user 0.001s	sys 0.001s
I20260709 12:25:31.846356  8347 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:31.846817  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:31.842685  8333 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 12:25:31.848680  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:31.848578  8428 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.878098  6275 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 4: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:31.879627  5628 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.849108  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:31.851567  8324 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:31.857285  5559 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:31.884557  8144 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:31.885109  8144 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:31.866299  8230 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.885957  8144 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.887358  8230 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:31.888546  8053 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:31.827461  8170 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:31.860966  5555 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:31.884039  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:31.846390  8436 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:31.827528  6273 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:31.827885  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:31.838642  8340 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 12:25:31.863425  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:31.865969  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:31.899992  5630 raft_consensus.cc:2410] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:25:31.838389  8238 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:25:31.874382  8311 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:31.875376  8293 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:25:31.880674  5628 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:31.903836  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:31.904718  8333 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:31.906456  8301 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.898427  8324 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:31.908653  5636 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:31.907740  8436 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:31.910075  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 599) took 3312 ms. Trace:
I20260709 12:25:31.911755  8410 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:31.876629  8295 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:31.902688  8398 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:25:31.916579  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 88753.3us
I20260709 12:25:31.921054  8380 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.923418  8311 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:31.901597  8438 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:25:32.051563  8410 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:32.081669  8410 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.933055  8301 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:32.083585  8410 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:31.931891  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:31.882108  8210 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:32.085000  5554 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:32.085400  8301 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:32.086298  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:32.086560  8210 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:32.066628  8333 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.087430  8333 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:31.982092  8293 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:32.089102  8293 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.089538  8293 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:32.089884  8333 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:32.100934  5561 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:32.103284  8051 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:32.104969  8440 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:25:32.108093  8380 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:32.087900  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:32.114037  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:32.114711  5554 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:32.115989  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:32.116811  5554 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:32.063457  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:31.950052  6274 rpcz_store.cc:276] 0709 12:25:28.597432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.614077 (+ 16645us) service_pool.cc:224] Handling call
0709 12:25:28.732486 (+118409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.001151 (+2268665us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.909794 (+908643us) spinlock_profiling.cc:243] Waited 2.23 s on lock 0x7b5400055854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:31.910012 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5119257856}
W20260709 12:25:32.075376  5458 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 271) took 21901 ms (client timeout 15000 ms). Trace:
W20260709 12:25:32.118312  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54981.4us
I20260709 12:25:32.120514  8398 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:25:32.118343  5458 rpcz_store.cc:269] 0709 12:25:10.173605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.179608 (+  6003us) service_pool.cc:224] Handling call
0709 12:25:32.075104 (+21895496us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:20.576380 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:20.831120 (+254740us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:20.831188 (+    68us) write_op.cc:435] Acquired schema lock
0709 12:25:20.831207 (+    19us) tablet.cc:662] Decoding operations
0709 12:25:20.929675 (+ 98468us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:20.929776 (+   101us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:20.929811 (+    35us) tablet.cc:685] Acquiring locks for 16 operations
0709 12:25:20.931306 (+  1495us) tablet.cc:701] Row locks acquired
0709 12:25:20.931389 (+    83us) write_op.cc:260] PREPARE: finished
0709 12:25:20.931607 (+   218us) write_op.cc:270] Start()
0709 12:25:20.931793 (+   186us) write_op.cc:276] Timestamp: P: 1783599920931523 usec, L: 0
0709 12:25:20.931821 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:25:21.051935 (+120114us) log.cc:844] Serialized 6193 byte log entry
0709 12:25:22.309837 (+1257902us) op_driver.cc:464] REPLICATION: finished
0709 12:25:22.514270 (+204433us) write_op.cc:301] APPLY: starting
0709 12:25:22.538380 (+ 24110us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:22.637518 (+ 99138us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:22.637691 (+   173us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:22.644541 (+  6850us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:22.644958 (+   417us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0709 12:25:22.645019 (+    61us) write_op.cc:312] APPLY: finished
0709 12:25:22.722230 (+ 77211us) log.cc:844] Serialized 113 byte log entry
0709 12:25:22.724945 (+  2715us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:22.725104 (+   159us) write_op.cc:454] Released schema lock
0709 12:25:22.725177 (+    73us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:22.725674 (+   497us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:22.725710 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":30856,"reported_tablets":39,"child_traces":[["op",{"apply.queue_time_us":164847,"apply.run_cpu_time_us":5743,"apply.run_wall_time_us":245066,"num_ops":16,"prepare.queue_time_us":306282,"prepare.run_cpu_time_us":57802,"prepare.run_wall_time_us":1195855,"raft.queue_time_us":393951,"raft.run_cpu_time_us":12965,"raft.run_wall_time_us":94522,"replication_time_us":1377821,"thread_start_us":1058154,"threads_started":4,"wal-append.queue_time_us":803381,"wal-append.run_cpu_time_us":10216,"wal-append.run_wall_time_us":1823668}]]}
I20260709 12:25:32.137912  5628 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 12:25:32.139253  5628 pending_rounds.cc:85] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 7
W20260709 12:25:32.140005  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.140575  5628 pending_rounds.cc:107] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:25:32.145963  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:32.146735  5628 raft_consensus.cc:2889] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:32.148579  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:32.152213  5710 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:32.186229  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:32.213282  5703 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:32.164412  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.152613  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:32.152251  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:32.219761  5707 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:32.125793  8295 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:32.204203  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:32.221027  5705 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:32.188133  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.222579  5709 raft_consensus.cc:2376] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6. Re-sending same reply.
I20260709 12:25:32.226356  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:32.147176  8369 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.147727  8333 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.149591  8399 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:32.237151  5707 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:32.240379  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:32.241737  5705 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
W20260709 12:25:32.250679  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 346951us
I20260709 12:25:32.252393  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:32.258149  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:32.187604  8293 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:32.237056  8369 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:32.274592  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.274785  8293 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:32.282522  8369 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:32.283814  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:32.285087  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:32.286216  8194 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:32.286907  8194 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:32.287592  8411 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:32.288049  8411 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:32.288213  8194 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:32.289173  8411 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:32.290454  8411 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:32.291039  8332 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:32.261051  5703 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:32.291761  8332 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:32.292845  8332 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:32.294286  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:32.295672  5630 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:25:32.296311  5634 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 12:25:32.297150  5628 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:25:32.297482  5634 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:32.300371  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:32.301024  5629 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:32.301270  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:32.295675  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:32.302078  5557 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:32.302000  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:32.303026  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:32.303593  5557 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:32.304917  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:32.306285  8347 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:32.306452  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:32.306676  8347 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.307049  8347 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:32.307524  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:32.261442  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:32.308369  5554 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:32.308264  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:32.308864  5636 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:32.229362  8328 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:25:32.238649  8333 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:32.310351  8328 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:32.310705  8328 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:32.311393  8333 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:32.311507  8328 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:32.309473  5554 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 12:25:32.312906  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:25:32.313834  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.314633  5554 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:32.263401  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:32.316563  8311 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:32.316653  5556 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:32.317214  8311 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:32.317487  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:32.317979  5556 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:32.318102  8311 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:32.319242  5556 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260709 12:25:32.268528  8224 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:25:32.320089  8224 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:32.320565  8224 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:32.321446  8224 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:32.321861  5619 tablet_service.cc:1558] Processing DeleteTablet for tablet f0f7b2d2a7e34d629491c96dbfbc7fdc with delete_type TABLET_DATA_TOMBSTONED (TS 4a1c5e6dd8e249bf82d38f90fa37b43d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35612
I20260709 12:25:32.322135  8328 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:32.267654  8295 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:32.324676  8295 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:32.258364  8399 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:25:32.325635  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 592489us
I20260709 12:25:32.326053  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:32.326402  8399 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:25:32.326856  8405 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:25:32.249811  8380 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:32.338665  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:32.328207  8399 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:32.342046  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:32.343505  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:32.345008  8398 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:25:32.317612  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 595) took 1342 ms. Trace:
I20260709 12:25:32.268539  8436 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.349555  6280 rpcz_store.cc:276] 0709 12:25:30.975539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.990174 (+ 14635us) service_pool.cc:224] Handling call
0709 12:25:32.317460 (+1327286us) spinlock_profiling.cc:243] Waited 92.6 ms on lock 0x7b54005eef54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:32.317559 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":213004160}
I20260709 12:25:32.336660  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:32.337497  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:32.345093  8428 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:25:32.351670  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 93615.1us
I20260709 12:25:32.352468  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:32.347528  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:32.356174  8347 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:32.359162  8347 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:32.354099  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:32.360442  5557 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:32.348623  8230 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:32.377321  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.378263  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:32.382290  5556 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:32.307663  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:32.346657  8399 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:25:32.387452  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 79862us
I20260709 12:25:32.363914  8210 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:32.363425  8369 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:32.416428  5710 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:32.418200  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:32.399137  5636 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:32.346374  8380 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:32.346132  8195 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 12:25:32.399369  8398 raft_consensus.cc:2764] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:25:32.402153  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:32.404063  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:32.350592  8333 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 12:25:32.406308  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:32.354357  8410 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 12:25:32.411610  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:32.363695  8347 raft_consensus.cc:2764] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260709 12:25:32.442462  5709 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
W20260709 12:25:32.452477  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44140.4us
I20260709 12:25:32.398116  8371 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:25:32.402000  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:32.453370  6273 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:32.445995  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:32.454312  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:32.454667  8441 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:25:32.455673  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:32.457027  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:32.458640  8295 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:32.459213  8295 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.459669  8295 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:32.446907  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:32.448763  8369 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:32.461674  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:32.461372  8369 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:32.400023  8444 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20260709 12:25:32.415227  8210 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.443087  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:32.417152  8293 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260709 12:25:32.468034  5555 raft_consensus.cc:1813] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Handling vote request from an unknown peer f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:32.468922  5555 raft_consensus.cc:2410] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 6 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:25:32.465202  8444 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:32.359998  8436 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:32.465929  8210 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:32.491858  5556 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:32.449836  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:32.501822  5710 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:32.453261  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:32.503038  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49849.9us
I20260709 12:25:32.506806  5556 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:25:32.508060  5556 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260709 12:25:32.509143  5556 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:32.505157  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:32.513065  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:32.515084  7897 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.466835  8333 consensus_queue.cc:1243] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:25:32.515653  7897 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:32.518599  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.520690  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:32.496600  8405 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:32.524418  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:32.522531  5557 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:32.521541  8436 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:32.506417  8404 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:32.543969  8404 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.545336  8404 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:32.544584  7897 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:32.522352  8347 raft_consensus.cc:2764] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:25:32.544840  6280 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:32.545688  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:32.565171  6275 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:32.565681  8380 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:32.566452  8452 tablet_replica.cc:333] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:25:32.524559  8402 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 12:25:32.569006  8402 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:32.569392  8402 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:32.570220  8402 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:32.571377  8402 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:32.572898  6278 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:32.529917  8405 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:32.580107  6274 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
W20260709 12:25:32.581070  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.583333  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:32.465222  8399 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:32.587555  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:32.524501  8371 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 12:25:32.539496  5702 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
W20260709 12:25:32.612942  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:32.628153  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 175177us
I20260709 12:25:32.631194  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:32.631999  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:32.632697  6274 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
W20260709 12:25:32.637620  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 301011us
I20260709 12:25:32.588639  8405 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:32.584190  8295 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:32.640815  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:32.641016  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 288621us
I20260709 12:25:32.644692  6280 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:32.644930  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:32.645684  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:32.641868  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.645376  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:32.592692  8206 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:32.663039  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:32.638535  5675 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:32.666482  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:32.665791  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 146480us
W20260709 12:25:32.667771  5526 connection.cc:256] RPC call timeout handler was delayed by -1.4081611309998152s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21389999999999998s for a response
W20260709 12:25:32.668092  5526 connection.cc:256] RPC call timeout handler was delayed by -1.2182611309999629s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23500000000000001s for a response
I20260709 12:25:32.668128  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:32.665560  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:32.669575  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:32.670132  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:32.670753  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:32.638384  8158 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:32.675053  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:32.642916  8347 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:32.679778  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:32.680039  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:32.676905  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:32.680665  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.686254  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:32.663481  8344 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:32.662309  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:32.692111  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:32.687063  6278 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:32.667290  8295 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:32.666335  8431 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:32.704859  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:32.714166  5627 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:32.673683  8333 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:32.674131  8436 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:32.671172  8446 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:32.721503  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:32.721762  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:32.683463  8399 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:32.727311  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:32.722905  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:32.679246  8444 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:32.681504  8194 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.708982  8380 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:32.671701  8206 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:32.713250  8295 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.714561  8224 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:32.726532  8446 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.688387  8158 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:32.693288  8347 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:32.730547  8457 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:25:32.751045  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.748247  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:32.737140  8350 raft_consensus.cc:2764] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:25:32.735703  7897 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:32.740247  8301 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:32.740096  8295 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:32.721766  8431 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:32.734874  8328 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20260709 12:25:32.734966  8194 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:32.749645  8446 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:32.892210  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:32.888441  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:32.804490  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:32.883603  6277 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 544) took 4041 ms (client timeout 1683 ms). Trace:
I20260709 12:25:32.737838  8454 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:32.889204  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:32.930238  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 170543us
I20260709 12:25:32.900110  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:32.853050  8450 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.165s	user 0.000s	sys 0.000s
I20260709 12:25:32.871548  5704 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:32.973973  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1681) took 1189 ms. Trace:
W20260709 12:25:32.963327  6277 rpcz_store.cc:269] 0709 12:25:28.842091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.842223 (+   132us) service_pool.cc:224] Handling call
0709 12:25:32.883360 (+4041137us) spinlock_profiling.cc:243] Waited 2.93 s on lock 0x7b54002e2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:32.883509 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6734593920}
I20260709 12:25:32.917937  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:32.904749  8158 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:32.856760  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:32.914490  8461 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
I20260709 12:25:32.902645  8386 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.958662  8350 raft_consensus.cc:2764] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:25:32.760821  8301 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:33.004076  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:33.001971  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 248377us
W20260709 12:25:32.975358  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 221585us
I20260709 12:25:32.881929  8230 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:32.996305  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 450) took 1083 ms. Trace:
I20260709 12:25:32.959822  8144 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:32.975746  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:33.006875  5558 rpcz_store.cc:276] 0709 12:25:31.784335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.888068 (+1103733us) service_pool.cc:224] Handling call
0709 12:25:32.973901 (+ 85833us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:32.966858  8469 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.064s	user 0.001s	sys 0.002s
I20260709 12:25:32.850410  8431 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:32.823308  8449 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:25:32.849450  8206 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:32.853804  8347 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:33.019924  5557 rpcz_store.cc:276] 0709 12:25:31.924012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.529118 (+605106us) service_pool.cc:224] Handling call
0709 12:25:32.542288 (+ 13170us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:32.542769 (+   481us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:32.542811 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:32.557859 (+ 15048us) log.cc:844] Serialized 52 byte log entry
0709 12:25:32.720195 (+162336us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:32.972111 (+251916us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.995816 (+ 23705us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.995927 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:32.995990 (+    63us) raft_consensus.cc:1685] finished
0709 12:25:32.996007 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.996224 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32429,"prepare.run_cpu_time_us":8899,"prepare.run_wall_time_us":133735,"raft.queue_time_us":2901,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":11324,"thread_start_us":307908,"threads_started":2,"wal-append.queue_time_us":251574}
I20260709 12:25:33.026575  8230 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:25:33.000226  8454 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:32.877533  8457 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.141s
I20260709 12:25:32.935922  8170 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.018445  8301 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:33.027287  8194 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:33.030210  8473 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:25:33.032975  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:32.844398  8224 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.177s
W20260709 12:25:33.042579  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 66895.9us
I20260709 12:25:33.030658  8472 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:32.878391  8311 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.145s
W20260709 12:25:33.045670  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.024927  8144 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:25:33.047291  5556 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.013s	sys 0.016s
W20260709 12:25:33.058988  5556 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.014s	sys 0.017s
I20260709 12:25:33.038363  5558 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:25:33.051203  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:33.060042  5709 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
W20260709 12:25:33.050742  8404 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.505s	user 0.015s	sys 0.000s
I20260709 12:25:33.018901  8386 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:33.071692  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 171725us
W20260709 12:25:33.051306  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 141128us
I20260709 12:25:33.052876  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:33.072638  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:33.084667  8454 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:33.085292  8405 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:33.080091  8144 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.088958  8144 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:33.089219  8482 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:33.089677  8482 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:33.090039  8482 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:33.096820  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:33.088958  5601 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:33.098465  8481 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:33.098929  8481 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:33.099295  8481 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:33.104063  5599 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:33.104916  8194 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:33.107757  8194 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:33.063709  8170 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:33.115646  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:33.112552  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:33.117319  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 487) took 1294 ms. Trace:
I20260709 12:25:33.118307  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:33.119386  5702 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:33.137405  7897 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:33.137487  8194 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:33.059695  8472 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.117843  5632 rpcz_store.cc:276] 0709 12:25:31.823370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.823856 (+   486us) service_pool.cc:224] Handling call
0709 12:25:31.844223 (+ 20367us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:32.298052 (+453829us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:32.298081 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:32.298681 (+   600us) log.cc:844] Serialized 52 byte log entry
0709 12:25:32.593538 (+294857us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:32.593578 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.593645 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.117007 (+523362us) raft_consensus.cc:1685] finished
0709 12:25:33.117037 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.117244 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":453282,"raft.queue_time_us":453812,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":110,"thread_start_us":294467,"threads_started":1,"wal-append.queue_time_us":372784}
I20260709 12:25:33.049597  8333 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 12:25:33.143859  8472 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:33.151726  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47766.7us
W20260709 12:25:33.160705  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:33.161805  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.162495  8472 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.109720  8311 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:33.158820  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:33.168498  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 146418us
I20260709 12:25:33.169363  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:33.170347  8483 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:33.171847  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:33.070221  8404 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:33.172564  5702 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:33.173287  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:33.176764  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:33.177726  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:33.158236  8371 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:33.184383  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:33.163241  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:33.189149  5633 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:25:33.167335  8311 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:33.151510  8158 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.029s
W20260709 12:25:33.195125  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.174034  8478 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:33.174120  8483 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:33.163206  8472 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:33.203347  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:33.198565  8210 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.685s	user 0.012s	sys 0.000s
I20260709 12:25:33.191943  8371 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:33.194710  8399 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:25:33.205250  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 89727.8us
I20260709 12:25:33.220045  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:33.193925  8477 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:33.194473  8311 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:33.223341  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:33.206315  8402 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.635s	user 0.015s	sys 0.000s
I20260709 12:25:33.230985  8346 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:33.231606  8346 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:33.237123  8346 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:33.238557  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 602) took 1415 ms. Trace:
I20260709 12:25:33.238837  8346 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:33.241268  8088 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260709 12:25:33.201828  8483 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:33.243721  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:33.246280  5705 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:33.239164  6276 rpcz_store.cc:276] 0709 12:25:31.823518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.823780 (+   262us) service_pool.cc:224] Handling call
0709 12:25:31.825071 (+  1291us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:31.843835 (+ 18764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.238049 (+1394214us) spinlock_profiling.cc:243] Waited 80.4 ms on lock 0x7b5400637ca0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa675a02 00007f4ffa671dd6 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:25:33.238433 (+   384us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":184917888}
I20260709 12:25:33.225293  8350 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:33.260865  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:33.224846  8486 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:33.245754  8380 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:33.269146  8350 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:33.273720  8350 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:33.269703  8489 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:33.250862  8480 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:33.224030  8210 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.294701  5705 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:33.298950  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:33.300453  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:33.301285  5705 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 3.
I20260709 12:25:33.302867  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:33.295292  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 70203.2us
I20260709 12:25:33.306672  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:33.296657  7897 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:25:33.292567  8411 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.002s	user 0.014s	sys 0.011s
I20260709 12:25:33.272703  8486 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:33.312295  6273 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:33.308970  8386 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:33.268523  8482 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:33.317240  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 56505.4us
I20260709 12:25:33.323768  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:33.313122  8158 consensus_queue.cc:1243] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:25:33.311179  8371 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.312896  8486 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:33.307595  8301 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:33.331645  8301 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:33.333793  8482 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:33.344345  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:33.346771  7897 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:33.349538  8392 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:33.350025  8392 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:33.350903  8392 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:33.322641  8404 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:33.367347  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:33.363101  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:33.376765  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 199066us
I20260709 12:25:33.398470  5704 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
W20260709 12:25:33.406023  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.407084  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:33.360842  8425 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.624s	user 0.003s	sys 0.003s
I20260709 12:25:33.352334  8392 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:33.365079  8488 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
W20260709 12:25:33.368050  8484 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
I20260709 12:25:33.446566  5634 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:25:33.447606  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 288874us
I20260709 12:25:33.400619  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1694) took 1069 ms. Trace:
I20260709 12:25:33.375196  8324 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.358682  8405 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:33.427325  8399 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:33.431026  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:33.401674  8346 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:33.432679  8452 raft_consensus.cc:2243] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:33.434769  5633 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:25:33.470247  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:33.471067  5628 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
W20260709 12:25:33.435504  8436 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.708s	user 0.009s	sys 0.000s
I20260709 12:25:33.435663  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 431) took 1872 ms. Trace:
I20260709 12:25:33.472452  5559 rpcz_store.cc:276] 0709 12:25:31.564630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.565542 (+   912us) service_pool.cc:224] Handling call
0709 12:25:31.582873 (+ 17331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.882349 (+299476us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.435346 (+1552997us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b54003a93d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:33.435574 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":630623104}
I20260709 12:25:33.437094  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:33.438521  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:33.368804  8472 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:33.378139  8498 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:33.439131  8446 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:33.439466  8295 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:33.440232  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:33.360657  8477 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:33.423866  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:33.444633  5704 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260709 12:25:33.529348  5704 pending_rounds.cc:85] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 4
I20260709 12:25:33.529398  6276 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 3.
I20260709 12:25:33.529742  5704 pending_rounds.cc:107] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260709 12:25:33.530056  5704 raft_consensus.cc:2889] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:33.446209  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:33.378222  8194 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:33.450090  5560 rpcz_store.cc:276] 0709 12:25:32.330821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.063238 (+732417us) service_pool.cc:224] Handling call
0709 12:25:33.075129 (+ 11891us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:33.101677 (+ 26548us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:33.101725 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:33.107477 (+  5752us) log.cc:844] Serialized 52 byte log entry
0709 12:25:33.168666 (+ 61189us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:33.345829 (+177163us) log.cc:844] Serialized 14 byte log entry
0709 12:25:33.400187 (+ 54358us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.400293 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.400367 (+    74us) raft_consensus.cc:1685] finished
0709 12:25:33.400385 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.400550 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":185550,"prepare.queue_time_us":123969,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":1607,"raft.queue_time_us":81899,"raft.run_cpu_time_us":725,"raft.run_wall_time_us":42117,"thread_start_us":127638,"threads_started":2,"wal-append.queue_time_us":61421}
I20260709 12:25:33.352703  8210 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:33.375356  8428 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:33.452889  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:33.356170  6272 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
W20260709 12:25:33.455806  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54220.3us
W20260709 12:25:33.538081  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.538240  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1827) took 2383 ms. Trace:
I20260709 12:25:33.527035  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:33.463526  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:33.465114  5552 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:33.538954  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:33.463526  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:33.464001  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:33.526242  8488 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:25:33.473821  5635 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:33.474493  5632 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:33.481691  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:33.543797  5631 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 6: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:33.545804  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:33.546447  5560 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:33.447006  8431 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.358027  8486 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.475099  8346 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:33.455307  8324 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:33.482101  8436 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:33.410619  8293 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260709 12:25:33.526690  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:33.366053  8481 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.464830  8399 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.613281  8399 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:33.532095  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:33.616796  5559 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:33.618069  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:33.619478  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:33.455891  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:33.538789  5706 rpcz_store.cc:276] 0709 12:25:31.154780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.396464 (+241684us) service_pool.cc:224] Handling call
0709 12:25:31.396791 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:31.397184 (+   393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.397227 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:31.397749 (+   522us) log.cc:844] Serialized 26 byte log entry
0709 12:25:31.559727 (+161978us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:31.559783 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.559854 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.359954 (+1800100us) spinlock_profiling.cc:243] Waited 2.33 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:33.534473 (+174519us) raft_consensus.cc:1685] finished
0709 12:25:33.534492 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.538163 (+  3671us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1391,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":10817,"spinlock_wait_cycles":5359488,"thread_start_us":155567,"threads_started":1,"wal-append.queue_time_us":292016}
I20260709 12:25:33.544483  8428 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.541610  5708 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:33.526881  8472 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.549571  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:33.625142  8472 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.627534  8472 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:33.628228  8295 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:33.556581  5705 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:33.558724  5704 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
W20260709 12:25:33.629207  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 174304us
I20260709 12:25:33.630147  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:33.631669  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:33.533437  8380 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.052s
I20260709 12:25:33.486727  8350 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:33.562682  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:33.524017  8473 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.024s
I20260709 12:25:33.564954  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:33.641520  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 76740.2us
I20260709 12:25:33.589521  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:33.603940  8511 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.605405  8311 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.597832  8452 raft_consensus.cc:2272] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:33.596619  8456 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.611788  8293 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:33.550392  8194 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:33.559448  8229 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:33.612535  8481 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:33.548619  8210 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.080s
I20260709 12:25:33.551692  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:33.559448  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:33.635308  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:33.637637  8504 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
W20260709 12:25:33.639374  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 76881.5us
I20260709 12:25:33.527287  8498 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.653973  8503 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:33.682899  8498 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:33.683264  8503 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:33.584707  8346 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:33.684696  8238 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:33.685099  8238 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:33.685866  8238 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.686789  8238 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:33.649742  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.657303  8446 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:33.690161  5635 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:33.690996  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1677) took 1907 ms. Trace:
I20260709 12:25:33.661461  8293 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:33.691866  8293 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:33.691357  5553 rpcz_store.cc:276] 0709 12:25:31.783409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.690294 (+1906885us) service_pool.cc:224] Handling call
0709 12:25:33.690570 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.690726 (+   156us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:33.690750 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:33.690787 (+    37us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:33.690817 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.690862 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.690941 (+    79us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:33.692937  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 440) took 2087 ms. Trace:
I20260709 12:25:33.693374  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:33.693421  5561 rpcz_store.cc:276] 0709 12:25:31.605940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.636690 (+ 30750us) service_pool.cc:224] Handling call
0709 12:25:31.659279 (+ 22589us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:32.101664 (+442385us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.692678 (+1591014us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b54001c43d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:33.692876 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1015007616}
I20260709 12:25:33.661818  8456 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:33.695112  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:33.697696  8456 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:33.609670  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:33.653940  8506 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:25:33.650874  8170 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.500s	user 0.008s	sys 0.004s
I20260709 12:25:33.651974  8486 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:33.663812  8350 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:33.660037  8501 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:33.669094  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 449) took 1774 ms. Trace:
I20260709 12:25:33.737555  5556 rpcz_store.cc:276] 0709 12:25:31.894477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.515261 (+620784us) service_pool.cc:224] Handling call
0709 12:25:32.515580 (+   319us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:32.516199 (+   619us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:25:33.059428 (+543229us) log.cc:844] Serialized 14 byte log entry
0709 12:25:33.170127 (+110699us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:33.172075 (+  1948us) log.cc:844] Serialized 52 byte log entry
0709 12:25:33.172269 (+   194us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:33.278483 (+106214us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b180045d4e0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:25:33.278742 (+   259us) log.cc:844] Serialized 14 byte log entry
0709 12:25:33.295805 (+ 17063us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.295922 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.668801 (+372879us) raft_consensus.cc:1685] finished
0709 12:25:33.668831 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.669008 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44888,"prepare.queue_time_us":353819,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":84,"raft.queue_time_us":17344,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":12149,"spinlock_wait_cycles":6966400,"thread_start_us":599249,"threads_started":2,"wal-append.queue_time_us":125253}
I20260709 12:25:33.739032  8501 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.739542  8501 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:33.739758  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:33.639190  8158 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:33.654778  8511 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.740530  8158 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:33.740962  8158 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:33.741437  8320 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:33.652395  8489 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:25:33.669544  8507 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260709 12:25:33.750746  5552 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:33.751389  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:33.751405  5552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1679) took 1967 ms (client timeout 2382 ms). Trace:
I20260709 12:25:33.677938  8452 ts_tablet_manager.cc:1916] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:25:33.736577  8456 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.678505  8428 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:33.679566  8194 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:33.670497  8229 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:33.761008  8229 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:33.762005  8229 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:33.694218  5553 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:33.698192  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 451) took 1571 ms. Trace:
I20260709 12:25:33.767148  5554 rpcz_store.cc:276] 0709 12:25:32.126426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.685693 (+559267us) service_pool.cc:224] Handling call
0709 12:25:32.686198 (+   505us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:32.707555 (+ 21357us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:32.707603 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:32.715350 (+  7747us) log.cc:844] Serialized 52 byte log entry
0709 12:25:33.342281 (+626931us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b180045cdc0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:25:33.426177 (+ 83896us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:33.426222 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.426296 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.697897 (+271601us) raft_consensus.cc:1685] finished
0709 12:25:33.697916 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.698137 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":836,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":23171,"spinlock_wait_cycles":9870208,"thread_start_us":83643,"threads_started":1,"wal-append.queue_time_us":868482}
W20260709 12:25:33.752256  5552 rpcz_store.cc:269] 0709 12:25:31.783875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.430688 (+1646813us) service_pool.cc:224] Handling call
0709 12:25:33.751347 (+320659us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:33.769822  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:33.770454  5554 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:33.699790  8311 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:33.700256  5631 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:33.566684  8431 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.691373  5635 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:33.584591  8448 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:33.707450  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:33.709074  5707 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:25:33.684290  8503 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.710393  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:33.791235  5561 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:33.711985  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:33.791380  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:33.792538  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 80615.2us
I20260709 12:25:33.793309  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:33.713407  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.633523  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:33.794946  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 161730us
I20260709 12:25:33.715446  8346 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:33.715998  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:33.716174  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 177301us
W20260709 12:25:33.801967  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.718287  8338 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:33.656404  8380 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.735455  5629 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:25:33.743391  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:33.747076  5559 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:33.820446  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:33.590296  8483 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.748054  8489 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:33.759294  8511 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:33.780844  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.770035  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:33.803802  5707 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:25:33.817351  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 109979us
I20260709 12:25:33.821645  5556 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:33.776144  6271 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:33.826287  5634 raft_consensus.cc:2410] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:33.791158  8448 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.834789  5628 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:33.828264  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:33.834216  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:33.827507  5552 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:33.837891  5560 raft_consensus.cc:2410] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:33.809763  8481 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:33.843235  5702 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:33.817198  8338 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:33.843463  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:33.758658  8456 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.823050  8311 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.079s
I20260709 12:25:33.816112  8230 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.079s
I20260709 12:25:33.843243  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:33.991581  8483 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:34.221654  8516 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.019s	sys 0.041s
I20260709 12:25:34.514101  6274 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:33.938468  5558 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:33.994118  8399 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:34.515537  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1696) took 2178 ms. Trace:
I20260709 12:25:34.515874  5558 rpcz_store.cc:276] 0709 12:25:32.336850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.065331 (+728481us) service_pool.cc:224] Handling call
0709 12:25:33.065736 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.938289 (+872553us) spinlock_profiling.cc:243] Waited 4.61 ms on lock 0x7b54002fbfa0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa6741c0 00007f4ffa675231 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c
0709 12:25:34.515266 (+576977us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.515476 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10593664}
I20260709 12:25:33.910722  5628 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260709 12:25:34.516824  5628 pending_rounds.cc:85] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 8
I20260709 12:25:34.517119  5628 pending_rounds.cc:107] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260709 12:25:33.841073  8448 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:34.517602  5628 raft_consensus.cc:2889] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:33.845366  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:34.520545  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.985164  8324 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:33.825664  8515 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:25:34.522141  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:34.011880  5554 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:25:34.522778  8344 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:34.509665  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 716372us
W20260709 12:25:34.537002  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.522949  8324 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:34.523442  8407 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:25:34.201539  8293 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.509s	user 0.006s	sys 0.006s
I20260709 12:25:33.826848  8380 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:33.989427  8431 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:34.697233  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:33.913682  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:34.702502  5707 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:34.273808  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 565) took 1743 ms. Trace:
I20260709 12:25:34.181857  5561 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:34.704731  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1705) took 1324 ms. Trace:
I20260709 12:25:34.705154  5561 rpcz_store.cc:276] 0709 12:25:33.380232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.696196 (+315964us) service_pool.cc:224] Handling call
0709 12:25:34.704647 (+1008451us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:33.944761  5553 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:34.706550  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1717) took 1058 ms. Trace:
I20260709 12:25:34.707001  5553 rpcz_store.cc:276] 0709 12:25:33.648081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.693120 (+ 45039us) service_pool.cc:224] Handling call
0709 12:25:34.706479 (+1013359us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:34.708104  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:33.905282  5559 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:34.522790  6280 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:34.470129  8521 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:25:34.407370  8464 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.550s	user 0.006s	sys 0.000s
W20260709 12:25:34.595814  8320 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.854s	user 0.004s	sys 0.006s
W20260709 12:25:34.599613  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.275022  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:34.732787  5709 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:34.275826  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1853) took 1422 ms. Trace:
I20260709 12:25:34.733546  5701 rpcz_store.cc:276] 0709 12:25:32.853017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.856437 (+  3420us) service_pool.cc:224] Handling call
0709 12:25:34.275538 (+1419101us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b5400451ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:34.275728 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":667841792}
I20260709 12:25:34.604949  6278 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:25:34.268981  8088 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:34.615501  8192 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:34.744663  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.745596  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:34.745391  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:34.746488  5628 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:34.271639  8332 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:34.255751  5633 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: []
W20260709 12:25:34.207914  6236 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.117s (SENT) [suppressed 23 similar messages]
I20260709 12:25:33.939765  8292 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:33.985661  8506 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:34.514299  8516 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.803s	user 0.024s	sys 0.056s
I20260709 12:25:34.045133  5706 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:34.043455  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:33.861819  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:34.710000  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1697) took 2372 ms. Trace:
I20260709 12:25:34.703831  6273 rpcz_store.cc:276] 0709 12:25:32.530201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.530321 (+   120us) service_pool.cc:224] Handling call
0709 12:25:32.890965 (+360644us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.312801 (+421836us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.272569 (+959768us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b54002e2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:34.273490 (+   921us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b54002e2f50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:34.273718 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1763717632}
I20260709 12:25:34.187258  8501 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:34.714578  5558 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: [4.4-5.5]
I20260709 12:25:34.637570  8481 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:34.716172  8483 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:34.513053  8498 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.862411  7897 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:33.937440  8346 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:34.716949  8454 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.938601  8462 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.717495  5552 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:25:33.832942  8489 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:34.190399  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 574) took 1027 ms. Trace:
I20260709 12:25:34.698892  8293 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:34.265658  8350 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:34.718282  8429 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.807269  8429 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:34.768505  8483 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:34.809231  8429 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:34.809768  8429 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:25:34.716984  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:34.810690  8483 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:34.764920  8457 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.646965  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:34.736704  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:34.765467  8229 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.738875  6276 raft_consensus.cc:2410] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
W20260709 12:25:34.608932  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.739169  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:33.983250  8158 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:34.744725  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:34.730993  5527 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.031s	sys 0.030s
I20260709 12:25:33.825846  8347 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.899s
W20260709 12:25:34.752569  5556 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.767s	user 0.019s	sys 0.018s
I20260709 12:25:34.753304  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:34.756551  5553 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:34.816829  6280 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:34.821182  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1714) took 1174 ms. Trace:
I20260709 12:25:34.821528  5553 rpcz_store.cc:276] 0709 12:25:33.646697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.756021 (+1109324us) service_pool.cc:224] Handling call
0709 12:25:34.756380 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:34.820942 (+ 64562us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.821130 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:34.823426  5556 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.019s	sys 0.018s
I20260709 12:25:34.720886  8398 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:34.726228  8333 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.731492  8480 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.759899  5559 rpcz_store.cc:276] 0709 12:25:32.337184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.836982 (+1499798us) service_pool.cc:224] Handling call
0709 12:25:33.905100 (+ 68118us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:34.709686 (+804586us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.709930 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:33.989099  6272 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 560) took 1544 ms (client timeout 1842 ms). Trace:
W20260709 12:25:34.748703  6275 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.743s	user 0.020s	sys 0.047s
I20260709 12:25:34.763227  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1709) took 1381 ms. Trace:
I20260709 12:25:34.873337  5558 rpcz_store.cc:276] 0709 12:25:33.381867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.713986 (+1332119us) service_pool.cc:224] Handling call
0709 12:25:34.714347 (+   361us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:34.762875 (+ 48528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.763157 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:34.760767  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:34.766551  5628 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:25:34.738379  8088 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:34.773665  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1704) took 1393 ms. Trace:
W20260709 12:25:34.777050  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.770980  8498 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:34.755352  8292 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:34.895710  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1718) took 1247 ms. Trace:
I20260709 12:25:34.896215  5558 rpcz_store.cc:276] 0709 12:25:33.648539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.894839 (+1246300us) service_pool.cc:224] Handling call
0709 12:25:34.895197 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:34.895381 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:34.895403 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:34.895442 (+    39us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:34.895479 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.895533 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:34.895642 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:34.899513  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:34.900075  5707 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:34.900561  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1869) took 1102 ms. Trace:
I20260709 12:25:34.763196  8506 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:34.755316  8192 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:34.767184  8521 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.802902  8293 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:34.774936  8454 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:34.805038  8513 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.805119  8346 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:34.806727  8501 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:34.810886  5631 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:34.645563  8344 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:34.812294  5704 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:34.719791  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:34.723230  6274 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:34.816927  5561 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:34.925287  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 609) took 1097 ms. Trace:
I20260709 12:25:34.818166  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:34.925801  6274 rpcz_store.cc:276] 0709 12:25:33.828064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.828593 (+   529us) service_pool.cc:224] Handling call
0709 12:25:34.925204 (+1096611us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:34.926416  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 108382us
W20260709 12:25:34.820062  5527 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.031s	sys 0.030s
W20260709 12:25:34.726082  8194 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.019s	sys 0.038s
I20260709 12:25:34.826992  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1870) took 1028 ms. Trace:
I20260709 12:25:34.845103  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1851) took 2108 ms. Trace:
W20260709 12:25:34.890908  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.700512  8407 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:34.816483  8483 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:34.890537  5552 rpcz_store.cc:276] 0709 12:25:33.380071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.509680 (+1129609us) service_pool.cc:224] Handling call
0709 12:25:34.717276 (+207596us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:34.772978 (+ 55702us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.773551 (+   573us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:34.849289  5709 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:34.961264  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1871) took 1162 ms. Trace:
I20260709 12:25:34.759042  8332 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:34.803995  8462 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:34.900892  5707 rpcz_store.cc:276] 0709 12:25:33.798455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.842302 (+1043847us) service_pool.cc:224] Handling call
0709 12:25:34.900499 (+ 58197us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:34.872893  6275 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.878s	user 0.020s	sys 0.048s
I20260709 12:25:34.825208  8398 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:34.824772  8144 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.763350  8338 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:34.820250  8482 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:34.845032  8320 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.957986  8293 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.914891  8078 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.778450  6271 rpcz_store.cc:276] 0709 12:25:33.162601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.162782 (+   181us) service_pool.cc:224] Handling call
0709 12:25:34.190126 (+1027344us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b54005ee554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:34.190306 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1252052352}
I20260709 12:25:34.842669  8229 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:34.825392  8350 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:34.958748  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 239078us
I20260709 12:25:35.006407  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:34.958592  5701 rpcz_store.cc:276] 0709 12:25:33.798802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.744343 (+945541us) service_pool.cc:224] Handling call
0709 12:25:34.826927 (+ 82584us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:34.875814  8333 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:34.826122  8457 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:34.961701  5709 rpcz_store.cc:276] 0709 12:25:33.799106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.274643 (+475537us) service_pool.cc:224] Handling call
0709 12:25:34.961192 (+686549us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:34.959998  5708 rpcz_store.cc:276] 0709 12:25:32.736824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.737436 (+   612us) service_pool.cc:224] Handling call
0709 12:25:32.749068 (+ 11632us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.623352 (+874284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.844801 (+1221449us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b5400074c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:34.845030 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1822206336}
W20260709 12:25:34.962841  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:34.958475  8194 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.019s	sys 0.038s
I20260709 12:25:34.896607  8480 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:34.983958  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.032617  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.844372  8404 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:34.990168  5560 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:25:34.992075  5552 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:25:35.035358  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:34.872400  8481 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:34.960464  8483 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:34.970914  8501 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.998327  6279 raft_consensus.cc:2376] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7. Re-sending same reply.
I20260709 12:25:34.963455  8088 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:35.015100  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:34.981629  8523 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:34.985627  8347 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:35.027769  5555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1692) took 3209 ms (client timeout 3121 ms). Trace:
I20260709 12:25:34.999143  8320 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:34.980010  8292 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:35.114089  8292 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:35.035718  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1700) took 1329 ms. Trace:
I20260709 12:25:35.116611  5552 rpcz_store.cc:276] 0709 12:25:33.706227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.960921 (+1254694us) service_pool.cc:224] Handling call
0709 12:25:34.986388 (+ 25467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.035341 (+ 48953us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.035641 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:35.036659  5708 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:25:35.117965  5708 pending_rounds.cc:85] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 6
I20260709 12:25:35.118273  5708 pending_rounds.cc:107] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:25:35.118557  5708 raft_consensus.cc:2889] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:34.992161  5561 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:35.119863  5561 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:35.039528  5705 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:35.006079  8380 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:34.996673  8078 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:35.032696  8448 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.515s	user 0.008s	sys 0.002s
I20260709 12:25:34.990051  8144 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:34.994361  8482 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:34.890412  6272 rpcz_store.cc:269] 0709 12:25:32.450468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.496057 (+ 45589us) service_pool.cc:224] Handling call
0709 12:25:33.988914 (+1492857us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b5400534dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:33.989030 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1823657856}
I20260709 12:25:35.058913  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1839) took 1235 ms. Trace:
I20260709 12:25:35.152591  5707 rpcz_store.cc:276] 0709 12:25:33.823740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.998937 (+1175197us) service_pool.cc:224] Handling call
0709 12:25:35.058824 (+ 59887us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:35.012180  5557 raft_consensus.cc:2955] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:35.034742  8530 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:35.075222  8528 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 12:25:35.011852  8452 ts_tablet_manager.cc:1929] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:25:35.081709  8332 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:35.082193  8088 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:35.089919  8229 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:35.090844  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1702) took 1716 ms. Trace:
I20260709 12:25:35.092191  8350 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:35.092797  8407 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:35.037948  8404 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:35.001463  8293 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:35.094594  8498 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:35.095536  8192 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:35.048413  8398 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:35.097261  8333 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:35.097803  8347 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:35.098794  8405 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:25:35.099347  8506 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:35.100584  8344 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:35.101329  8454 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:35.040606  8456 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:35.071650  5555 rpcz_store.cc:269] 0709 12:25:31.802553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.307317 (+504764us) service_pool.cc:224] Handling call
0709 12:25:35.011571 (+2704254us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54002279d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:35.011799 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":367756544}
W20260709 12:25:35.110162  6234 connection.cc:256] RPC call timeout handler was delayed by -1.2965910249999979s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1643s for a response
W20260709 12:25:35.259262  6234 connection.cc:256] RPC call timeout handler was delayed by -1.1012910250001369s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18600000000000003s for a response
I20260709 12:25:35.261458  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:35.113157  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1954) took 1667 ms (client timeout 2093 ms). Trace:
I20260709 12:25:35.123302  5632 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:25:35.124063  5559 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:25:34.919677  8521 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:35.134166  5635 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:25:35.136550  5552 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:25:35.139429  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 454) took 1979 ms. Trace:
I20260709 12:25:35.149127  8534 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:34.930042  8513 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:35.103664  8158 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:35.101300  8480 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:35.198916  5556 rpcz_store.cc:276] 0709 12:25:33.379521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.830807 (+451286us) service_pool.cc:224] Handling call
0709 12:25:33.831113 (+   306us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:33.831457 (+   344us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:33.831485 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:33.841094 (+  9609us) log.cc:844] Serialized 52 byte log entry
0709 12:25:34.824616 (+983522us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:34.824689 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.824783 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:35.090537 (+265754us) raft_consensus.cc:1685] finished
0709 12:25:35.090565 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:35.090741 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":358078,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":134,"thread_start_us":881751,"threads_started":1,"wal-append.queue_time_us":1085533}
I20260709 12:25:35.143541  8346 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:35.073951  8501 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:35.231360  5707 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:25:35.090869  8483 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:35.208273  5705 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:35.103076  8523 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:35.143774  8448 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:35.243913  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 208627us
I20260709 12:25:35.247095  8230 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.248337  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 610) took 1420 ms. Trace:
W20260709 12:25:35.249778  5525 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.003s	sys 0.005s
I20260709 12:25:35.144187  8503 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.091s
W20260709 12:25:35.253664  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.446s	user 0.009s	sys 0.024s
I20260709 12:25:35.253801  5709 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 12:25:35.235111  8452 log.cc:1199] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/wals/f0f7b2d2a7e34d629491c96dbfbc7fdc
W20260709 12:25:35.330269  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 63992.5us
I20260709 12:25:35.255009  8398 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:35.262522  5629 rpcz_store.cc:269] 0709 12:25:33.445288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.445409 (+   121us) service_pool.cc:224] Handling call
0709 12:25:35.112871 (+1667462us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b54001f2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:35.113078 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":470024576}
I20260709 12:25:35.197064  8530 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:35.268453  6271 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:35.248888  8229 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:25:35.258489  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.518062  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.518991  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.491411  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.453783  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.646s	user 0.009s	sys 0.028s
W20260709 12:25:35.422153  5558 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.000s	sys 0.014s
W20260709 12:25:35.520085  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 1.71255e+06us
I20260709 12:25:35.269999  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:35.520684  5558 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.003s	sys 0.017s
I20260709 12:25:35.335615  5710 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
W20260709 12:25:35.454638  5601 leader_election.cc:336] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.842s (SENT)
W20260709 12:25:35.453564  5525 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.004s	sys 0.010s
W20260709 12:25:35.522292  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 67691.1us
I20260709 12:25:35.452948  8078 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:35.327052  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.516981  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 255596us
I20260709 12:25:35.253430  8333 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:35.422362  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 494505us
I20260709 12:25:35.269608  6277 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: []
W20260709 12:25:35.330536  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 301810us
I20260709 12:25:35.537480  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:35.304308  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.008s	sys 0.001s
W20260709 12:25:35.538837  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.012s	sys 0.007s
W20260709 12:25:35.422686  5526 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.006s	sys 0.008s
W20260709 12:25:35.542640  5526 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.009s	sys 0.010s
W20260709 12:25:35.543069  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 644379us
W20260709 12:25:35.422592  6236 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.003s	sys 0.011s
I20260709 12:25:35.519271  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:35.546217  6236 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.795s	user 0.003s	sys 0.017s
W20260709 12:25:35.546630  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 1.33867e+06us
W20260709 12:25:35.548060  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.553108  5556 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:25:35.516724  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 459) took 2354 ms. Trace:
I20260709 12:25:35.521590  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 460) took 2358 ms. Trace:
W20260709 12:25:35.439023  8532 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.165s	user 0.000s	sys 0.000s
W20260709 12:25:35.571795  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.526180  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:35.573127  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47039.4us
I20260709 12:25:35.574695  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:35.514171  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1704) took 1806 ms. Trace:
I20260709 12:25:35.576305  5552 rpcz_store.cc:276] 0709 12:25:33.707608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.117773 (+1410165us) service_pool.cc:224] Handling call
0709 12:25:35.125384 (+  7611us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.452246 (+326862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.514085 (+ 61839us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:35.422487  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.001s	sys 0.013s
W20260709 12:25:35.578724  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.793s	user 0.004s	sys 0.015s
I20260709 12:25:35.553696  5707 rpcz_store.cc:276] 0709 12:25:33.162110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.196955 (+2034845us) service_pool.cc:224] Handling call
0709 12:25:35.231132 (+ 34177us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.516233 (+285101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.516641 (+   408us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:35.579669  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:35.580417  5707 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:35.582250  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:35.516049  6271 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:35.584494  6271 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:35.515574  5708 rpcz_store.cc:276] 0709 12:25:33.160115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.023885 (+1863770us) service_pool.cc:224] Handling call
0709 12:25:35.036267 (+ 12382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.135750 (+ 99483us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.139359 (+  3609us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:35.416944  8457 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:35.334904  8498 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.439997  8546 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.330447  8344 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:35.274366  7897 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:35.590206  8344 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.590842  8344 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:35.588439  8498 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:35.553141  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:35.421962  5636 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.001s	sys 0.013s
W20260709 12:25:35.607954  5636 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.860s	user 0.002s	sys 0.018s
I20260709 12:25:35.589510  5554 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:25:35.577611  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:35.555491  8369 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:25:35.587767  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 811385us
I20260709 12:25:35.562626  5710 rpcz_store.cc:276] 0709 12:25:33.163637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.256625 (+2092988us) service_pool.cc:224] Handling call
0709 12:25:35.335344 (+ 78719us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.521345 (+186001us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.521540 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:35.597656  5701 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:35.264514  8522 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.457510  8538 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.273394  8454 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:35.699903  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1698) took 1994 ms. Trace:
I20260709 12:25:35.606876  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:35.269146  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:35.269114  8482 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:35.551054  8550 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.554929  8229 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:35.549165  8346 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:35.526443  8534 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:35.270220  8293 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:35.610116  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 72789.6us
I20260709 12:25:35.685576  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:35.586912  5552 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:25:35.570128  8332 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.686908  5561 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:25:35.518461  8230 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:35.526115  8158 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.150s
I20260709 12:25:35.595402  8320 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:35.694217  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 453) took 2535 ms (client timeout 1643 ms). Trace:
W20260709 12:25:35.754949  5702 rpcz_store.cc:269] 0709 12:25:33.159271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.184028 (+ 24757us) service_pool.cc:224] Handling call
0709 12:25:35.694027 (+2509999us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b5400324454. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:35.694149 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1512249216}
I20260709 12:25:35.756390  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:35.757234  5702 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:35.758190  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:35.758746  5702 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:35.759603  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:35.695101  6279 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 561) took 3244 ms (client timeout 1570 ms). Trace:
W20260709 12:25:35.760541  6279 rpcz_store.cc:269] 0709 12:25:32.451125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.507112 (+ 55987us) service_pool.cc:224] Handling call
0709 12:25:34.646757 (+2139645us) spinlock_profiling.cc:243] Waited 6.75 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:35.694878 (+1048121us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b54005ee554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:35.695011 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":434323072}
I20260709 12:25:35.695720  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 608) took 1868 ms. Trace:
I20260709 12:25:35.761507  6276 rpcz_store.cc:276] 0709 12:25:33.827773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.828522 (+   749us) service_pool.cc:224] Handling call
0709 12:25:35.695483 (+1866961us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b54002eed54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:35.695638 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":517128960}
I20260709 12:25:35.605688  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:35.608042  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 68476us
I20260709 12:25:35.600942  8407 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.105s
I20260709 12:25:35.762674  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:35.556172  8333 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:35.700331  5559 rpcz_store.cc:276] 0709 12:25:33.705728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.896684 (+1190956us) service_pool.cc:224] Handling call
0709 12:25:35.123848 (+227164us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.264098 (+140250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.692706 (+428608us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b540023f350. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:35.699835 (+  7129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":474246400}
I20260709 12:25:35.707166  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1858) took 2839 ms. Trace:
W20260709 12:25:35.708751  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 134149us
I20260709 12:25:35.675288  8433 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.719103  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:35.678097  8553 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:35.690407  7897 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.764365  5706 rpcz_store.cc:276] 0709 12:25:32.854510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.621267 (+766757us) service_pool.cc:224] Handling call
0709 12:25:34.044874 (+423607us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:34.759355 (+714481us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.693296 (+933941us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x7b54001987d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:35.693416 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":973454080}
W20260709 12:25:35.768841  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 215766us
I20260709 12:25:35.769009  5556 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:35.696813  8078 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:35.775051  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:35.789636  5710 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:35.789873  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 184249us
I20260709 12:25:35.790773  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:35.791916  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:35.700815  8516 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.080s
I20260709 12:25:35.715258  6272 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:35.609748  8546 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:35.717159  8534 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:35.796851  5706 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:35.711519  8454 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:35.706048  8369 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:35.708228  8538 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:35.708312  8522 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:35.716882  8550 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:35.800179  8347 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:35.981223  8535 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:35.980679  8552 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.204s	user 0.001s	sys 0.000s
I20260709 12:25:36.110020  8535 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.110647  8454 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:35.924459  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.111738  5555 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
W20260709 12:25:35.981194  8548 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.452s	user 0.001s	sys 0.000s
I20260709 12:25:36.009550  8192 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:25:35.975322  8533 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.299s	user 0.000s	sys 0.000s
I20260709 12:25:36.112577  8521 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:35.893785  8333 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:36.113188  8369 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:35.965626  8560 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.155s
I20260709 12:25:35.974215  8452 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35612 (request call id 88) took 3653 ms. Trace:
I20260709 12:25:35.893610  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:35.814576  8506 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:36.008966  8498 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:36.116835  8538 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:35.815553  8503 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:35.895358  8501 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:35.795388  8332 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:35.890872  8230 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:35.818293  8380 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:35.800661  8554 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:25:35.977073  8551 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.208s	user 0.000s	sys 0.000s
I20260709 12:25:36.120797  8535 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:36.121107  8546 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:25:35.819738  8328 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.929041  8346 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.155506  8333 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:35.923595  8387 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:35.955474  8540 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.167s
I20260709 12:25:35.813865  8553 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:35.821269  8344 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:35.826273  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.797191  7897 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:35.779860  8433 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:35.453415  6275 rpcz_store.cc:276] 0709 12:25:33.828334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.839240 (+ 10906us) service_pool.cc:224] Handling call
0709 12:25:33.919385 (+ 80145us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.994576 (+ 75191us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:25:34.994790 (+1000214us) log.cc:844] Serialized 14 byte log entry
0709 12:25:35.247823 (+253033us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:35.247900 (+    77us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:35.247975 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.248083 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:35.248260 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1186,"prepare.queue_time_us":966761,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":60,"raft.queue_time_us":11128,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"thread_start_us":1206166,"threads_started":2,"wal-append.queue_time_us":412101}
I20260709 12:25:35.763203  8559 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:35.813514  8320 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:25:35.797269  8531 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.537s	user 0.001s	sys 0.000s
I20260709 12:25:35.797698  8482 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:25:35.980010  8088 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.776s	user 0.011s	sys 0.000s
W20260709 12:25:35.928792  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 106311us
I20260709 12:25:36.239388  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:35.777335  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:35.889485  8292 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:35.974520  8530 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:36.127375  5601 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.007s	sys 0.023s
W20260709 12:25:36.244709  5601 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.007s	sys 0.023s
I20260709 12:25:36.243898  8292 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:36.245146  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 722049us
I20260709 12:25:36.246333  6271 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:35.894025  8229 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:35.788138  8404 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:36.131094  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 367307us
I20260709 12:25:35.913136  8078 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:25:36.132087  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.248673  8399 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:36.249902  8557 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
W20260709 12:25:36.130086  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 168105us
I20260709 12:25:33.429836  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:36.129813  5556 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:36.050776  5710 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:36.254710  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 451) took 3096 ms. Trace:
W20260709 12:25:35.940860  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.255116  5703 rpcz_store.cc:276] 0709 12:25:33.158745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.005301 (+1846556us) service_pool.cc:224] Handling call
0709 12:25:35.031068 (+ 25767us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.234622 (+203554us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:35.234655 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:35.257583 (+ 22928us) log.cc:844] Serialized 26 byte log entry
0709 12:25:35.538746 (+281163us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:36.097136 (+558390us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.128616 (+ 31480us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.128726 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:36.254485 (+125759us) raft_consensus.cc:1685] finished
0709 12:25:36.254506 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.254646 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32697,"prepare.queue_time_us":623372,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":67,"raft.queue_time_us":203753,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":11899,"thread_start_us":825328,"threads_started":3,"wal-append.queue_time_us":686253}
I20260709 12:25:36.143281  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 463) took 2978 ms. Trace:
W20260709 12:25:36.151417  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 257832us
I20260709 12:25:36.118060  8550 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:36.163551  5559 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:25:36.139469  8448 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:36.150835  8489 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.339s	user 0.007s	sys 0.007s
I20260709 12:25:36.140446  8192 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:25:36.167622  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 402697us
I20260709 12:25:36.154924  8542 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.155597  8503 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.322278  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1715) took 2675 ms. Trace:
I20260709 12:25:36.344959  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:36.345278  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:36.346259  5706 rpcz_store.cc:276] 0709 12:25:33.164984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.796087 (+2631103us) service_pool.cc:224] Handling call
0709 12:25:35.796475 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:36.142938 (+346463us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.143207 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:36.345332  5553 rpcz_store.cc:276] 0709 12:25:33.647216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.822535 (+1175319us) service_pool.cc:224] Handling call
0709 12:25:34.822971 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:34.841934 (+ 18963us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:34.841982 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:34.890858 (+ 48876us) log.cc:844] Serialized 26 byte log entry
0709 12:25:35.199330 (+308472us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:35.199374 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.199446 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:36.321976 (+1122530us) raft_consensus.cc:1685] finished
0709 12:25:36.321997 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.322136 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18518,"raft.queue_time_us":217512,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":2371,"thread_start_us":276843,"threads_started":1,"wal-append.queue_time_us":617535}
I20260709 12:25:36.289637  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.347476  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.256521  5552 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:25:36.347725  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1861) took 3492 ms. Trace:
W20260709 12:25:36.186041  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 581199us
W20260709 12:25:36.187222  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.154140  8506 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:36.350679  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:36.348273  5704 rpcz_store.cc:276] 0709 12:25:32.855315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.630579 (+775264us) service_pool.cc:224] Handling call
0709 12:25:33.630824 (+   245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:34.924772 (+1293948us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.347379 (+1422607us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540033fc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:36.347655 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2716599168}
I20260709 12:25:36.171437  8523 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:36.172138  8380 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.215724  8559 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:36.187242  8566 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.219806  8320 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:36.347771  5556 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:36.219022  8540 consensus_queue.cc:1243] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260709 12:25:36.350528  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 101971us
I20260709 12:25:36.356518  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:36.188884  8333 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:36.176597  8328 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.186666  8498 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:36.177016  8554 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:25:36.362989  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.364864  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:36.151424  8452 rpcz_store.cc:276] 0709 12:25:32.321218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.321574 (+   356us) service_pool.cc:224] Handling call
0709 12:25:32.566017 (+244443us) ts_tablet_manager.cc:1151] Deleting tablet f0f7b2d2a7e34d629491c96dbfbc7fdc
0709 12:25:32.566126 (+   109us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:25:34.093059 (+1526933us) tablet_metadata.cc:665] Metadata flushed
0709 12:25:35.001301 (+908242us) tablet_metadata.cc:665] Metadata flushed
0709 12:25:35.974081 (+972780us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":766717,"tablet-delete.queue_time_us":224075,"thread_start_us":127039,"threads_started":1}
W20260709 12:25:36.468114  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 103392us
I20260709 12:25:36.403920  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:36.468436  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.404247  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:36.469616  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.470356  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:36.470778  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:36.471310  5555 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:36.396379  8550 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:36.404726  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:36.207633  8513 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:36.350081  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 572766us
W20260709 12:25:36.409339  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 58777.8us
I20260709 12:25:36.282284  8230 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:36.412744  8320 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:36.415727  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:36.485598  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 69971.9us
I20260709 12:25:36.486842  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:36.271690  8088 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:36.354679  8456 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.354965  8503 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:36.288412  8573 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.408030  8566 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:36.414299  8523 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.077s
I20260709 12:25:36.361781  8501 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.362051  8311 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:36.428316  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 71970us
I20260709 12:25:36.500538  8407 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:25:36.281332  8078 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:36.448508  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.403911  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:36.522441  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:36.324216  8399 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.472826  5633 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:25:36.475042  5703 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
W20260709 12:25:36.404902  8555 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260709 12:25:36.361402  8542 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.476807  8565 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:36.477315  8556 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:36.477798  8541 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:36.398840  8559 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:36.405442  8454 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.247218  8292 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:36.483407  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:36.483407  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:36.482028  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.525945  8456 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:36.489046  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:36.489658  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:36.498106  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:36.498584  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:36.427063  8333 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:36.504302  8561 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.207s	user 0.000s	sys 0.000s
I20260709 12:25:36.505914  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:36.473219  8538 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:36.513247  8562 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.306s	user 0.000s	sys 0.000s
I20260709 12:25:36.505188  8554 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:36.514014  8332 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:36.497907  8540 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 12:25:36.478646  8448 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20260709 12:25:36.489725  8498 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:36.515766  8324 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:36.590444  8565 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:36.521385  5702 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: [6.7-8.8]
I20260709 12:25:36.592638  8556 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:36.474319  8306 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:36.317461  7897 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:36.483330  8230 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.616864  8230 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:36.618536  8498 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:36.619149  5559 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:36.620010  8534 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.505s	user 0.004s	sys 0.004s
W20260709 12:25:36.620493  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.620666  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1699) took 2914 ms. Trace:
I20260709 12:25:36.625161  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:36.621915  5558 rpcz_store.cc:276] 0709 12:25:33.705983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.897854 (+1191871us) service_pool.cc:224] Handling call
0709 12:25:34.898148 (+   294us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.521165 (+623017us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:35.521197 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:35.538297 (+ 17100us) log.cc:844] Serialized 26 byte log entry
0709 12:25:35.818979 (+280682us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:35.819018 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.819080 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:36.620428 (+801348us) raft_consensus.cc:1685] finished
0709 12:25:36.620449 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.620604 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":175550,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":110,"thread_start_us":887059,"threads_started":2,"wal-append.queue_time_us":607115}
I20260709 12:25:36.626634  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:36.380920  8489 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.628209  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:36.631091  6272 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:36.626644  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.632388  5558 raft_consensus.cc:2410] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:25:36.320947  8433 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:36.428668  8340 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.082s
I20260709 12:25:36.428215  8344 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.083s
I20260709 12:25:36.541862  6280 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260709 12:25:36.461835  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54197.4us
I20260709 12:25:36.555778  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 503) took 2748 ms. Trace:
I20260709 12:25:36.640447  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:36.641178  5633 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:36.491122  8550 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:36.548000  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:36.558706  5556 raft_consensus.cc:2410] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:25:36.560528  5555 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:36.650084  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:36.650583  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.506314  8482 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 12:25:36.651681  7897 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:36.579636  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:36.590849  8454 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:36.652974  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:36.653482  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.584321  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:36.654197  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:36.507236  8387 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
I20260709 12:25:36.654668  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.655256  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:36.557483  8566 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:36.409224  8371 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.558456  8564 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:36.659160  8371 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:36.659459  8564 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:36.661468  8371 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:36.661940  8564 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:36.473732  8553 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:36.662341  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 458) took 3500 ms. Trace:
I20260709 12:25:36.662720  5701 rpcz_store.cc:276] 0709 12:25:33.161465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.038813 (+1877348us) service_pool.cc:224] Handling call
0709 12:25:35.039153 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.694159 (+655006us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.662062 (+967903us) spinlock_profiling.cc:243] Waited 558 ms on lock 0x7b5400323054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:36.662272 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1283997568}
I20260709 12:25:36.663959  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:36.664644  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 452) took 3505 ms. Trace:
W20260709 12:25:36.592245  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 102668us
I20260709 12:25:36.665027  5709 rpcz_store.cc:276] 0709 12:25:33.159013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.023784 (+1864771us) service_pool.cc:224] Handling call
0709 12:25:35.024126 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.453213 (+429087us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.664327 (+1211114us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b54002f5b54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:36.664582 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":520294656}
I20260709 12:25:36.666329  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:36.664855  8565 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.580917  8328 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:36.556900  8432 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.103s
I20260709 12:25:36.545825  8501 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:36.678972  8432 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:25:36.596534  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:36.552271  8311 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.681497  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:36.682729  8311 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:36.522725  8192 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.534317  8572 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.070s
I20260709 12:25:36.598235  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:36.695977  5710 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:36.602293  8568 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.602341  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:36.577368  8399 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:36.520318  5705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1874) took 2720 ms (client timeout 2247 ms). Trace:
I20260709 12:25:36.610692  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:36.703296  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.703684  8542 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:36.703482  8550 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.703977  5561 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
W20260709 12:25:36.523607  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.960s	user 0.029s	sys 0.046s
W20260709 12:25:36.704968  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.141s	user 0.029s	sys 0.046s
W20260709 12:25:36.705296  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 1.14157e+06us
W20260709 12:25:36.614238  8398 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.085s	user 0.009s	sys 0.003s
I20260709 12:25:36.614554  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1965) took 2074 ms. Trace:
W20260709 12:25:36.641852  5526 connection.cc:256] RPC call timeout handler was delayed by -1.0296628719997898s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24740000000000004s for a response
I20260709 12:25:36.642762  5559 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:36.643640  8529 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:36.644200  8230 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.644598  8489 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:36.644935  8324 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:36.726879  8568 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:36.727257  8324 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.729825  8529 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:36.645915  8344 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:36.597976  8575 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:36.647229  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:36.535423  8573 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.635411  5634 rpcz_store.cc:276] 0709 12:25:33.807044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.025345 (+218301us) service_pool.cc:224] Handling call
0709 12:25:34.029521 (+  4176us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:34.034191 (+  4670us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:25:34.078500 (+ 44309us) log.cc:844] Serialized 14 byte log entry
0709 12:25:34.393445 (+314945us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b1800425a40. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:25:36.270948 (+1877503us) spinlock_profiling.cc:243] Waited 7.07 ms on lock 0x7b1800425a40. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:25:36.478548 (+207600us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:36.479070 (+   522us) log.cc:844] Serialized 26 byte log entry
0709 12:25:36.479210 (+   140us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:36.479241 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.479313 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:36.555464 (+ 76151us) raft_consensus.cc:1685] finished
0709 12:25:36.555490 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.555692 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63848,"prepare.queue_time_us":50500,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":64837,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":26280,"spinlock_wait_cycles":18648064,"thread_start_us":137717,"threads_started":1,"wal-append.queue_time_us":2474812}
I20260709 12:25:36.647578  8556 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:36.581792  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 101619us
I20260709 12:25:36.657506  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:36.590536  5631 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:36.666992  5709 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:36.771171  6280 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:36.668635  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.572834  8522 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:36.670392  5560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1721) took 2159 ms (client timeout 1604 ms). Trace:
W20260709 12:25:36.813067  5560 rpcz_store.cc:269] 0709 12:25:34.511503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.644999 (+133496us) service_pool.cc:224] Handling call
0709 12:25:36.667641 (+2022642us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b54001be254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:36.670321 (+  2680us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":398955776}
I20260709 12:25:36.686196  5627 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:36.687595  5633 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:36.685223  8483 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:36.692986  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.815364  5555 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:36.816221  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.816798  5555 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:36.817515  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.818243  5555 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:25:36.693934  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:36.818825  5703 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 6: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:36.597613  8538 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:36.701769  5705 rpcz_store.cc:269] 0709 12:25:33.800460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.760315 (+959855us) service_pool.cc:224] Handling call
0709 12:25:36.520036 (+1759721us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54002d0fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:36.520234 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":379168000}
I20260709 12:25:36.707137  5636 rpcz_store.cc:276] 0709 12:25:34.540182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.541164 (+   982us) service_pool.cc:224] Handling call
0709 12:25:34.697429 (+156265us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:35.668030 (+970601us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:35.668073 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:35.668828 (+   755us) log.cc:844] Serialized 52 byte log entry
0709 12:25:35.801014 (+132186us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:35.801062 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.801157 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:36.614293 (+813136us) raft_consensus.cc:1685] finished
0709 12:25:36.614314 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.614484 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49703,"raft.queue_time_us":98798,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":135,"thread_start_us":976481,"threads_started":2,"wal-append.queue_time_us":487230}
I20260709 12:25:36.710147  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:36.820706  6274 raft_consensus.cc:2410] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:25:36.656329  8498 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 12:25:36.667572  8078 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.707695  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:36.685658  8387 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:36.759747  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.706542  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:36.796579  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:36.833923  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 127446us
W20260709 12:25:36.809298  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 118582us
W20260709 12:25:36.650770  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 102899us
W20260709 12:25:36.837738  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 130152us
W20260709 12:25:36.822918  6271 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.576s	user 0.006s	sys 0.006s
I20260709 12:25:36.823633  5630 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:36.649232  8534 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:36.827363  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:36.829326  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.830631  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.831920  5708 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:36.771052  8489 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.730875  8529 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:36.840238  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:36.841435  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:36.770972  8575 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:36.842101  5561 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:36.855614  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:36.842464  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.847682  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:36.856267  5705 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:36.839895  6271 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260709 12:25:36.845736  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:36.858112  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:36.858978  6280 raft_consensus.cc:2410] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:25:36.845901  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.846808  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:36.769553  8344 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:36.841656  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:36.731686  8547 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.117s
I20260709 12:25:36.848389  8407 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:36.849011  8229 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.851384  8320 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.853452  5703 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:36.841758  5555 raft_consensus.cc:1813] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Handling vote request from an unknown peer f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:36.860327  6271 pending_rounds.cc:85] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 7
I20260709 12:25:36.830276  8311 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.099s
I20260709 12:25:36.830382  8371 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.104s
W20260709 12:25:36.921957  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 80406.5us
W20260709 12:25:36.887976  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.846417  8522 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:37.187678  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.321471  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:36.929638  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.331694  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:37.041515  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:36.875003  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.322161  5555 raft_consensus.cc:2410] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 6 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:25:37.322683  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:37.318570  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 471000us
I20260709 12:25:37.323571  6271 pending_rounds.cc:107] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 12:25:37.110128  5558 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:36.847697  8306 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:37.264806  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:37.335374  5555 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:25:37.336504  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1710) took 1135 ms. Trace:
W20260709 12:25:37.351931  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.336504  5705 raft_consensus.cc:2410] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:37.335413  6271 raft_consensus.cc:2889] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:37.342983  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1967) took 2780 ms. Trace:
I20260709 12:25:37.347121  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1705) took 1148 ms. Trace:
I20260709 12:25:37.356899  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 618) took 2096 ms. Trace:
I20260709 12:25:37.356767  5558 rpcz_store.cc:276] 0709 12:25:36.199223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.859821 (+660598us) service_pool.cc:224] Handling call
0709 12:25:37.109856 (+250035us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.346825 (+236969us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.347061 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.356719  5632 rpcz_store.cc:276] 0709 12:25:34.562827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.563028 (+   201us) service_pool.cc:224] Handling call
0709 12:25:35.122990 (+559962us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.263753 (+140763us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.331769 (+2068016us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x7b54001f2f50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:37.342755 (+ 10986us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1286986496}
I20260709 12:25:37.348902  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:37.357861  6271 rpcz_store.cc:276] 0709 12:25:35.260218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.267752 (+  7534us) service_pool.cc:224] Handling call
0709 12:25:35.268275 (+   523us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:36.246105 (+977830us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b58004a8dd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5ce0aa 00007f4ffa650447 00007f4ffa668fe7 00007f4ffa66a7de 00007f4ffa674b49 00007f4ffa675416 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4
0709 12:25:37.356379 (+1110274us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.356818 (+   439us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":53072256}
I20260709 12:25:37.358716  5703 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:37.382565  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:36.863131  8575 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:37.358620  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.392693  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:37.393565  6271 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:36.855074  8567 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.512s
I20260709 12:25:36.852398  8568 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:37.370262  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.006s	sys 0.035s
W20260709 12:25:37.398463  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.006s	sys 0.035s
W20260709 12:25:37.398873  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 555159us
I20260709 12:25:36.991838  8564 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:25:37.160285  8559 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.569s	user 0.006s	sys 0.002s
I20260709 12:25:37.371356  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:36.854769  8489 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.407564  6280 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:37.112778  8498 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.377353  6278 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:25:37.351154  8585 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.527s	user 0.001s	sys 0.000s
W20260709 12:25:37.319002  8503 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.784s	user 0.010s	sys 0.000s
I20260709 12:25:37.346055  8328 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:37.346594  8448 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:37.394762  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:36.929797  8529 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:36.921339  8206 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.512s
I20260709 12:25:37.359953  8566 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:36.855036  7897 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.520s
I20260709 12:25:36.846673  8387 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.863080  8346 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.386467  8230 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:37.339871  8229 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.497164  8320 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:37.498046  8144 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:37.462302  8498 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:37.499037  8229 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:37.406956  8564 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.499862  8564 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:37.500661  8206 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:37.501192  8206 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:37.480954  8398 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.494323  8538 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:37.494755  8582 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260709 12:25:37.495666  8387 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:37.489792  8566 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.503507  8534 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:37.422236  8559 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.492530  8431 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:37.472496  8503 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:37.504317  8573 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:37.511083  8566 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:37.523108  8559 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:37.525913  8564 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:37.527009  8575 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.527653  8564 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:37.528636  8575 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:37.558678  5560 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:37.363755  5555 rpcz_store.cc:276] 0709 12:25:36.200944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.334292 (+1133348us) service_pool.cc:224] Handling call
0709 12:25:37.334832 (+   540us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.336106 (+  1274us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.336417 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:37.573800  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:37.574734  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.575013  5629 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:37.383446  5703 raft_consensus.cc:2376] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9. Re-sending same reply.
I20260709 12:25:37.412242  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:37.405867  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:37.578190  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1893) took 1050 ms. Trace:
I20260709 12:25:37.573648  5628 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:37.578588  5705 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:37.578838  5703 rpcz_store.cc:276] 0709 12:25:36.527630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.382213 (+854583us) service_pool.cc:224] Handling call
0709 12:25:37.577864 (+195651us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.578689  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.580108  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:37.580665  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:37.581394  5706 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:25:37.581760  5703 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:25:37.581504  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.581993  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1853) took 1014 ms. Trace:
I20260709 12:25:37.582626  5632 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 6: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:37.582913  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:37.386987  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.584828  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:37.585453  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.586313  5634 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:37.586546  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:37.587350  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.587694  5632 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:25:37.588981  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:37.589800  5632 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:37.582485  5706 rpcz_store.cc:276] 0709 12:25:36.567178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.579352 (+1012174us) service_pool.cc:224] Handling call
0709 12:25:37.581898 (+  2546us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.593425  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:37.594915  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:37.595757  5703 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:37.588223  5634 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:37.598181  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:37.599848  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:37.600787  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:37.601660  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.602540  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:37.603223  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:37.603499  5706 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:37.604025  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1889) took 1246 ms. Trace:
I20260709 12:25:37.604738  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:37.606200  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:37.613467  5703 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:37.614117  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1890) took 1248 ms. Trace:
I20260709 12:25:37.614530  5703 rpcz_store.cc:276] 0709 12:25:36.365783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.605803 (+1240020us) service_pool.cc:224] Handling call
0709 12:25:37.614053 (+  8250us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:37.615176  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.615286  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.615921  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:37.606775  5632 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
W20260709 12:25:37.618018  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.618427  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:37.621336  6272 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 6: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:37.622138  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.621336  6271 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 4: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:37.622473  6274 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:37.623560  5632 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:37.627485  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.629978  5634 raft_consensus.cc:2410] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:37.621187  5703 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:37.631695  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.631995  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:37.641597  5602 connection.cc:256] RPC call timeout handler was delayed by -1.8354440190000787s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1612s for a response
W20260709 12:25:37.642234  5602 connection.cc:256] RPC call timeout handler was delayed by -1.7877440190000016s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
W20260709 12:25:37.642616  5602 connection.cc:256] RPC call timeout handler was delayed by -1.3908440190002693s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2106s for a response
W20260709 12:25:37.642973  5602 connection.cc:256] RPC call timeout handler was delayed by -1.3593440190002184s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2141s for a response
W20260709 12:25:37.643826  5602 connection.cc:256] RPC call timeout handler was delayed by -1.2531440190000467s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2259s for a response
I20260709 12:25:37.645326  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:37.643271  5706 rpcz_store.cc:276] 0709 12:25:36.357346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.602098 (+1244752us) service_pool.cc:224] Handling call
0709 12:25:37.603976 (+  1878us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.632074  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.467701  8328 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:37.654619  8206 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.496076  8346 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:37.506481  8398 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.659806  8347 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:37.669505  8347 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:37.670763  8398 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:37.673898  5706 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:25:37.675036  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 466) took 4508 ms. Trace:
I20260709 12:25:37.675503  5706 rpcz_store.cc:276] 0709 12:25:33.166240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.672571 (+4506331us) service_pool.cc:224] Handling call
0709 12:25:37.673266 (+   695us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.674516 (+  1250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.674899 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.655951  5705 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:37.667248  8503 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:37.676898  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1856) took 1093 ms. Trace:
I20260709 12:25:37.677297  5705 rpcz_store.cc:276] 0709 12:25:36.583941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.378209 (+794268us) service_pool.cc:224] Handling call
0709 12:25:37.676792 (+298583us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.660530  8206 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:37.660235  8392 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.512167  8431 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:37.633436  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.668692  5703 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:37.672040  8347 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:37.658497  5634 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:37.639827  6280 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:37.667711  8346 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:37.659379  8328 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:37.679760  8230 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:25:37.705806  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58430.1us
I20260709 12:25:37.710616  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:37.715441  5559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1706) took 1515 ms (client timeout 1612 ms). Trace:
I20260709 12:25:37.716804  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:37.718809  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 593) took 1170 ms. Trace:
I20260709 12:25:37.716475  5632 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
W20260709 12:25:37.716210  5559 rpcz_store.cc:269] 0709 12:25:36.199560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.345565 (+146005us) service_pool.cc:224] Handling call
0709 12:25:37.714837 (+1369272us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b540044eb54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:37.715110 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":345264000}
I20260709 12:25:37.719528  6280 rpcz_store.cc:276] 0709 12:25:36.526889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.370749 (+843860us) service_pool.cc:224] Handling call
0709 12:25:37.694518 (+323769us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:37.723954  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.725859  8431 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:37.728210  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:37.728909  5705 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:37.729326  5556 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:37.729817  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1836) took 4331 ms. Trace:
I20260709 12:25:37.730787  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:37.730326  5705 rpcz_store.cc:276] 0709 12:25:33.398870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.701263 (+4302393us) service_pool.cc:224] Handling call
0709 12:25:37.728525 (+ 27262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.729469 (+   944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.729710 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.735112  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.735808  5632 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:37.739774  5561 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:37.749882  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 496) took 4271 ms. Trace:
I20260709 12:25:37.750356  5635 rpcz_store.cc:276] 0709 12:25:33.478775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.788702 (+309927us) service_pool.cc:224] Handling call
0709 12:25:33.789149 (+   447us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.525480 (+1736331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.749246 (+2223766us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b5400134f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:37.749692 (+   446us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3092310912}
I20260709 12:25:37.751591  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:37.752559  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 397228us
I20260709 12:25:37.754149  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:37.757033  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 425263us
I20260709 12:25:37.757479  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1972) took 1120 ms. Trace:
I20260709 12:25:37.757958  8566 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.758972  8328 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.760325  5555 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:37.748116  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 493) took 4270 ms. Trace:
I20260709 12:25:37.738055  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:37.761153  5630 rpcz_store.cc:276] 0709 12:25:33.477562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.684706 (+207144us) service_pool.cc:224] Handling call
0709 12:25:33.685119 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:36.839064 (+3153945us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.747210 (+908146us) spinlock_profiling.cc:243] Waited 3.14 s on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:37.747798 (+   588us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7217528576}
I20260709 12:25:37.762760  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:37.762900  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:37.763409  5701 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:37.763991  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:37.764441  8431 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:37.722376  8392 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:37.765846  8328 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:37.766655  8565 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:37.777724  8565 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.766503  6273 raft_consensus.cc:1217] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Deduplicated request from leader. Original: 3.5->[4.6-6.7]   Dedup: 4.6->[6.7-6.7]
I20260709 12:25:37.763991  8586 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:37.773469  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.780014  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:37.778378  6273 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260709 12:25:37.781812  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:37.782011  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 622) took 1193 ms. Trace:
I20260709 12:25:37.778574  8586 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.782699  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 583) took 1278 ms. Trace:
I20260709 12:25:37.783021  8586 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:37.783659  8392 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:37.784161  6276 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:37.783118  6273 rpcz_store.cc:276] 0709 12:25:36.504891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.505430 (+   539us) service_pool.cc:224] Handling call
0709 12:25:36.514112 (+  8682us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:37.781682 (+1267570us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.782381 (+   699us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54005910d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:37.782627 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2879030016}
I20260709 12:25:37.784878  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 582) took 1287 ms. Trace:
I20260709 12:25:37.785245  6276 rpcz_store.cc:276] 0709 12:25:36.497991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.498145 (+   154us) service_pool.cc:224] Handling call
0709 12:25:37.783391 (+1285246us) spinlock_profiling.cc:243] Waited 4.16 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:25:37.783738 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.784626 (+   888us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.784830 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9567488}
I20260709 12:25:37.787400  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 628) took 1130 ms. Trace:
I20260709 12:25:37.787849  6272 rpcz_store.cc:276] 0709 12:25:36.656731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.657252 (+   521us) service_pool.cc:224] Handling call
0709 12:25:37.787107 (+1129855us) spinlock_profiling.cc:243] Waited 806 ms on lock 0x7b5400773154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:37.787324 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1853732096}
I20260709 12:25:37.772393  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:37.789999  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:37.791288  8328 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:37.791782  8328 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.792148  8328 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:37.792477  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 580) took 1313 ms. Trace:
I20260709 12:25:37.792614  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:37.792897  6277 rpcz_store.cc:276] 0709 12:25:36.478667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.479236 (+   569us) service_pool.cc:224] Handling call
0709 12:25:37.791243 (+1312007us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:25:37.791553 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.791968 (+   415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:37.791999 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:37.792061 (+    62us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:37.792089 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.792145 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:37.792290 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8790400}
I20260709 12:25:37.794006  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:37.794929  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.795370  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.797991  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1867) took 4000 ms. Trace:
I20260709 12:25:37.798509  5705 rpcz_store.cc:276] 0709 12:25:33.797822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.756464 (+3958642us) service_pool.cc:224] Handling call
0709 12:25:37.796995 (+ 40531us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.797525 (+   530us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:37.797554 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:37.797643 (+    89us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:37.797696 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.797769 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:37.797900 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.769163  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 1970) took 1297 ms. Trace:
I20260709 12:25:37.800339  5628 rpcz_store.cc:276] 0709 12:25:36.471722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.471862 (+   140us) service_pool.cc:224] Handling call
0709 12:25:36.489056 (+ 17194us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.579067 (+1090011us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.768830 (+189763us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540018bd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:37.769091 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2492424320}
I20260709 12:25:37.801074  5705 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:25:37.801900  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.802412  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1842) took 3975 ms. Trace:
I20260709 12:25:37.802752  5628 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:37.803115  5705 rpcz_store.cc:276] 0709 12:25:33.826660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.800016 (+3973356us) service_pool.cc:224] Handling call
0709 12:25:37.800519 (+   503us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.801788 (+  1269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.802231 (+   443us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.757931  5629 rpcz_store.cc:276] 0709 12:25:36.637247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.642088 (+  4841us) service_pool.cc:224] Handling call
0709 12:25:36.658761 (+ 16673us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.575792 (+917031us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.757212 (+181420us) spinlock_profiling.cc:243] Waited 915 ms on lock 0x7b540032cdd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:37.757407 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2105458432}
I20260709 12:25:37.806107  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:37.806854  5629 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:37.808151  8598 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.772768  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 465) took 4607 ms. Trace:
I20260709 12:25:37.808820  8598 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:37.809486  5708 rpcz_store.cc:276] 0709 12:25:33.165821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.822548 (+2656727us) service_pool.cc:224] Handling call
0709 12:25:36.105473 (+282925us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:36.842277 (+736804us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.772179 (+929902us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b5400210554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:37.772571 (+   392us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1670185344}
I20260709 12:25:37.811511  5708 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:37.811878  8598 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:37.766083  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 462) took 4601 ms. Trace:
W20260709 12:25:37.771164  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1881) took 2228 ms (client timeout 1866 ms). Trace:
I20260709 12:25:37.768051  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.778205  8565 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:37.772825  8596 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:37.823423  8522 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:37.823462  8481 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:37.840832  8481 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:37.825429  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.826773  8501 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:37.845458  5630 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:37.846716  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.764189  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1868) took 3966 ms. Trace:
I20260709 12:25:37.776038  5710 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:37.796265  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1984) took 1126 ms. Trace:
I20260709 12:25:37.776966  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:37.781869  5632 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:25:37.804162  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.835767  5702 rpcz_store.cc:276] 0709 12:25:33.164413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.795710 (+2631297us) service_pool.cc:224] Handling call
0709 12:25:36.520940 (+725230us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:36.610701 (+ 89761us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.765622 (+1154921us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b54002d0fd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:37.765855 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1612676864}
I20260709 12:25:37.815917  8462 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:37.819082  8380 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:37.835961  5707 rpcz_store.cc:269] 0709 12:25:35.542080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.581914 (+ 39834us) service_pool.cc:224] Handling call
0709 12:25:37.770625 (+2188711us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54001982d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:37.770885 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2870632320}
I20260709 12:25:37.838582  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:37.813570  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.821390  8566 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:37.831013  8344 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.853701  5636 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:37.782419  6278 rpcz_store.cc:276] 0709 12:25:36.588845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.681177 (+ 92332us) service_pool.cc:224] Handling call
0709 12:25:37.781531 (+1100354us) spinlock_profiling.cc:243] Waited 695 ms on lock 0x7b54005f8f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:37.781802 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1598187264}
I20260709 12:25:37.830922  5705 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:37.855793  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.825898  8338 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:37.822666  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:37.765597  8592 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:37.812180  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1876) took 3065 ms. Trace:
I20260709 12:25:37.858680  5708 rpcz_store.cc:276] 0709 12:25:34.746731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.810671 (+3063940us) service_pool.cc:224] Handling call
0709 12:25:37.811174 (+   503us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.811921 (+   747us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.812121 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.851913  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.856204  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1875) took 3109 ms. Trace:
I20260709 12:25:37.848594  5634 rpcz_store.cc:276] 0709 12:25:36.669475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.795036 (+1125561us) service_pool.cc:224] Handling call
0709 12:25:37.796179 (+  1143us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.859788  5705 rpcz_store.cc:276] 0709 12:25:34.746448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.829763 (+3083315us) service_pool.cc:224] Handling call
0709 12:25:37.830336 (+   573us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.855884 (+ 25548us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.856137 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.857178  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1840) took 4028 ms. Trace:
I20260709 12:25:37.848134  5701 rpcz_store.cc:276] 0709 12:25:33.798113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.762559 (+3964446us) service_pool.cc:224] Handling call
0709 12:25:37.762998 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.763899 (+   901us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.764112 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.860836  5710 rpcz_store.cc:276] 0709 12:25:33.823977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.774847 (+3950870us) service_pool.cc:224] Handling call
0709 12:25:37.775365 (+   518us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.848245 (+ 72880us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.852458 (+  4213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.860555  5628 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 3.
I20260709 12:25:37.864580  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:37.865231  5707 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:37.865439  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.865865  6273 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:37.866446  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 589) took 1352 ms. Trace:
I20260709 12:25:37.866873  6273 rpcz_store.cc:276] 0709 12:25:36.514354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.822017 (+1307663us) service_pool.cc:224] Handling call
0709 12:25:37.866354 (+ 44337us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.860538  8592 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.868865  8592 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:37.869869  8483 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.039s	user 0.007s	sys 0.013s
I20260709 12:25:37.870283  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1690) took 5540 ms. Trace:
I20260709 12:25:37.870767  8344 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:37.871040  5704 raft_consensus.cc:2410] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:25:37.871815  8346 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.872449  8346 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.868865  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1708) took 4487 ms. Trace:
I20260709 12:25:37.842717  8481 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.884307  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:37.883409  5554 rpcz_store.cc:276] 0709 12:25:33.381554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.531787 (+1150233us) service_pool.cc:224] Handling call
0709 12:25:34.532336 (+   549us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.689130 (+1156794us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.868238 (+2179108us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400026cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:37.868600 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2430487296}
W20260709 12:25:37.890031  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 113295us
I20260709 12:25:37.890357  8481 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:37.890795  8432 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:25:37.891153  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:37.891672  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.891793  8328 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.890239  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.892858  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:37.870802  5557 rpcz_store.cc:276] 0709 12:25:32.329902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.032613 (+702711us) service_pool.cc:224] Handling call
0709 12:25:35.011752 (+1979139us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.012021 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:25:35.242766 (+230745us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.204255 (+961489us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.397884 (+193629us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:36.397948 (+    64us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:36.398008 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.398099 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:37.869862 (+1471763us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b54002279d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:37.870066 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":265472,"prepare.queue_time_us":345749,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":473372,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":62080,"spinlock_wait_cycles":4522188672,"thread_start_us":756737,"threads_started":3,"wal-append.queue_time_us":553944,"wal-append.run_cpu_time_us":7806,"wal-append.run_wall_time_us":1864310}
I20260709 12:25:37.894189  5630 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:37.894371  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:37.895013  5710 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:37.895993  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.897531  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:37.898649  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 616) took 1335 ms. Trace:
I20260709 12:25:37.899050  6278 rpcz_store.cc:276] 0709 12:25:36.563520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.897107 (+1333587us) service_pool.cc:224] Handling call
0709 12:25:37.898540 (+  1433us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.900988  5630 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:25:37.901983  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:37.902523  5630 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:37.903275  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:37.903985  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.904991  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:37.906670  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:37.908640  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:37.909596  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1983) took 1241 ms. Trace:
I20260709 12:25:37.909982  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 615) took 1364 ms. Trace:
I20260709 12:25:37.909914  5636 rpcz_store.cc:276] 0709 12:25:36.667836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.862925 (+1195089us) service_pool.cc:224] Handling call
0709 12:25:37.909527 (+ 46602us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.906965  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 465) took 1213 ms. Trace:
I20260709 12:25:37.907508  8078 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:37.894919  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:37.883690  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:37.911332  5628 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:25:37.900279  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:37.910413  6271 rpcz_store.cc:276] 0709 12:25:36.545386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.394464 (+849078us) service_pool.cc:224] Handling call
0709 12:25:37.909782 (+515318us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54006383d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:37.909927 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":337815936}
I20260709 12:25:37.910928  5632 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:37.910749  5555 rpcz_store.cc:276] 0709 12:25:36.693973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.561636 (+867663us) service_pool.cc:224] Handling call
0709 12:25:37.562405 (+   769us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.760721 (+198316us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.906701 (+145980us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b54001be254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:37.906891 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":454380800}
I20260709 12:25:37.921553  5631 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:25:37.921712  5634 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 6. Ops: []
I20260709 12:25:37.922376  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1642) took 1105 ms. Trace:
I20260709 12:25:37.923023  5631 rpcz_store.cc:276] 0709 12:25:36.816698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.904256 (+1087558us) service_pool.cc:224] Handling call
0709 12:25:37.921301 (+ 17045us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:37.922004 (+   703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.922286 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.926541  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:37.927592  6271 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:37.922473  5556 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:37.928761  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:37.929430  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:37.929906  8593 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:25:37.930418  5556 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:37.930147  8495 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:25:37.930663  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:37.930954  8593 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:37.931407  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:37.931859  8495 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:37.932533  5556 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:37.932765  8593 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:37.934197  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:37.935088  8192 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:37.934510  8593 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:37.932296  8495 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:37.938114  8495 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:37.939132  5702 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:37.939224  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1742) took 1435 ms. Trace:
I20260709 12:25:37.939571  8495 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:37.941241  5631 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:37.941901  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1649) took 1097 ms. Trace:
I20260709 12:25:37.922382  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1646) took 1104 ms. Trace:
I20260709 12:25:37.942911  5635 raft_consensus.cc:2410] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 9 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 12:25:37.942216  5631 rpcz_store.cc:276] 0709 12:25:36.847149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.940474 (+1093325us) service_pool.cc:224] Handling call
0709 12:25:37.940844 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.941617 (+   773us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.941845 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.928149  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 617) took 1363 ms. Trace:
I20260709 12:25:37.944173  6271 rpcz_store.cc:276] 0709 12:25:36.564414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.925876 (+1361462us) service_pool.cc:224] Handling call
0709 12:25:37.928052 (+  2176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.942713  5634 rpcz_store.cc:276] 0709 12:25:36.817667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.921194 (+1103527us) service_pool.cc:224] Handling call
0709 12:25:37.921537 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.922124 (+   587us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.922288 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.946607  5630 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:37.949723  5601 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:37.951987  8230 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:25:37.952664  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.951503  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1644) took 1134 ms. Trace:
I20260709 12:25:37.953686  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:37.917007  8078 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.955091  8078 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:37.954582  5629 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:37.962694  5629 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:25:37.952615  8230 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:37.918953  8432 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:25:37.919481  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61502us
I20260709 12:25:37.965061  8432 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:37.965265  8230 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.966172  8432 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.966521  8206 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:25:37.967494  8206 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:37.967850  8206 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:37.968945  8206 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:37.970178  8503 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:37.971057  5635 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:37.971585  8206 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:37.973090  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:37.974011  8596 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.974598  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:37.975935  5557 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:37.975960  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:37.975962  8456 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:37.977591  8595 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:37.978011  8595 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:37.980892  5635 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:25:37.981256  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:37.978179  5710 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:25:37.990672  8530 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:37.991497  8328 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:37.991941  8454 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:37.981264  8498 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:37.993429  8595 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:38.024207  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:37.994102  8542 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:37.978649  8369 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 12:25:37.980041  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:37.977075  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:37.953317  5632 rpcz_store.cc:276] 0709 12:25:36.817201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.950491 (+1133290us) service_pool.cc:224] Handling call
0709 12:25:37.950892 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.951112 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:37.951136 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:37.951202 (+    66us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:37.951238 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.951292 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:37.951418 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:37.992866  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:37.997423  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:37.999032  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:37.999641  5553 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:38.000082  8392 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:38.001719  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:38.006803  8578 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:37.992861  5554 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:37.935992  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:38.025853  5702 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:38.055989  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:37.980705  8348 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:25:38.033764  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 52639us
I20260709 12:25:38.035145  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:38.057204  5710 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:38.013797  8530 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:38.056291  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:38.062109  5632 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:38.060599  6276 raft_consensus.cc:2410] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 12:25:38.058595  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 78653.1us
I20260709 12:25:38.063783  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:38.064898  5557 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:37.979931  8399 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:38.070178  5631 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:25:38.071549  5632 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260709 12:25:38.072057  5632 pending_rounds.cc:85] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 6
I20260709 12:25:38.069594  5706 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:38.074313  6277 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:38.077842  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1650) took 1223 ms. Trace:
I20260709 12:25:38.082387  5631 rpcz_store.cc:276] 0709 12:25:36.847504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.009730 (+1162226us) service_pool.cc:224] Handling call
0709 12:25:38.068041 (+ 58311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.070599 (+  2558us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.070783 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.075835  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 631) took 1390 ms. Trace:
I20260709 12:25:38.083606  6276 rpcz_store.cc:276] 0709 12:25:36.677896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.000221 (+1322325us) service_pool.cc:224] Handling call
0709 12:25:38.067908 (+ 67687us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.055995  8578 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:38.084906  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:38.084937  5709 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:37.974648  8596 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.085686  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:38.058794  8348 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:38.081918  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 93067.2us
I20260709 12:25:38.087524  6272 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:25:38.089550  5636 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:38.089852  5557 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:38.077376  5632 pending_rounds.cc:107] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:25:38.091174  5632 raft_consensus.cc:2889] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:38.091885  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.092444  5556 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 3.
I20260709 12:25:38.093357  5702 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:38.094624  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:38.095338  5702 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
W20260709 12:25:38.096977  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 104235us
I20260709 12:25:38.097218  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:38.097886  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:38.098834  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:38.098884  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:38.099772  5557 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 8: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:38.074079  8498 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:38.069442  8591 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:38.096315  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:38.114260  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:38.115839  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.075503  8586 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.085151  8347 raft_consensus.cc:2764] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:25:38.087138  8431 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:38.139118  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:38.105656  8591 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.100419  8462 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.091720  8564 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:38.115253  8338 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:38.152890  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:38.154392  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 85462.2us
I20260709 12:25:38.134750  8328 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:38.155164  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:38.134572  8344 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.159029  6277 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:38.152379  5702 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:38.139701  8431 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.110359  8565 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.143574  8498 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:25:38.167456  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 165932us
I20260709 12:25:38.153095  8432 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:38.151468  8591 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:38.157207  8564 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.172115  5632 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:38.156231  8338 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.174026  5707 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
W20260709 12:25:38.174894  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.176750  8586 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:38.163537  8521 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.143852  8542 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
I20260709 12:25:38.192900  5710 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:38.180624  8578 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.165431  8566 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
W20260709 12:25:38.181945  8603 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:25:38.181473  8530 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.183602  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:38.184736  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:38.164631  8431 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:38.182643  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 467) took 1477 ms. Trace:
I20260709 12:25:38.186003  8338 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:38.203765  5629 raft_consensus.cc:2376] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10. Re-sending same reply.
I20260709 12:25:38.176569  8348 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.190971  6278 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:38.180768  8596 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:38.194231  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:38.159070  8462 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.198644  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1727) took 2242 ms. Trace:
I20260709 12:25:38.200083  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:38.186074  8506 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:38.204139  5557 rpcz_store.cc:276] 0709 12:25:36.694505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.116999 (+1422494us) service_pool.cc:224] Handling call
0709 12:25:38.156769 (+ 39770us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.171378 (+ 14609us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:38.171412 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:38.171453 (+    41us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:38.171487 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.171544 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:38.171668 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.182524  8328 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:38.163509  8344 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.195184  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:38.205499  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 594) took 1664 ms. Trace:
I20260709 12:25:38.210522  6278 rpcz_store.cc:276] 0709 12:25:36.541066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.899913 (+1358847us) service_pool.cc:224] Handling call
0709 12:25:38.205427 (+305514us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.204869  5552 rpcz_store.cc:276] 0709 12:25:35.956565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.097026 (+140461us) service_pool.cc:224] Handling call
0709 12:25:36.256235 (+159209us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:36.348220 (+ 91985us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.171119 (+1822899us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b5400082850. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:38.198459 (+ 27340us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":365134080}
I20260709 12:25:38.188861  8564 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:38.191869  8537 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.217667  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:38.198684  8542 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:38.223516  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:38.223879  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:38.226735  5629 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:38.230342  5707 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:25:38.231796  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:38.233326  8503 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:38.233420  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:38.233950  8503 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:38.233757  8380 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:25:38.234709  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:38.235955  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:38.237680  5702 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:38.235697  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:38.236660  8578 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:25:38.245774  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.220570  8596 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:38.253703  8595 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.254973  5556 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:38.262425  5710 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:25:38.263141  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1877) took 2723 ms. Trace:
I20260709 12:25:38.263470  5710 rpcz_store.cc:276] 0709 12:25:35.540755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.197983 (+2657228us) service_pool.cc:224] Handling call
0709 12:25:38.226859 (+ 28876us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.262882 (+ 36023us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.263074 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.264510  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:38.265317  5710 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:38.268637  8595 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:38.269389  8078 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.270706  5634 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 6. Ops: []
I20260709 12:25:38.271840  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:38.272395  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:38.272584  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 87441.6us
I20260709 12:25:38.236168  8348 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:38.223260  8542 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:38.256623  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:38.274065  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:38.277767  8429 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:38.277912  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:38.281080  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:38.281677  5635 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:25:38.283907  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.203181  8521 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.289556  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:38.290813  8521 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:38.290848  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:37.939736  5558 rpcz_store.cc:276] 0709 12:25:36.423527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.358341 (+934814us) service_pool.cc:224] Handling call
0709 12:25:37.558870 (+200529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.559484 (+   614us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:25:37.705210 (+145726us) log.cc:844] Serialized 14 byte log entry
0709 12:25:37.857322 (+152112us) log.cc:844] Serialized 14 byte log entry
0709 12:25:37.857522 (+   200us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:37.857563 (+    41us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:37.857608 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.857662 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:37.858417 (+   755us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62345,"prepare.queue_time_us":179426,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":101,"raft.queue_time_us":96575,"raft.run_cpu_time_us":390,"raft.run_wall_time_us":14119,"thread_start_us":175845,"threads_started":2,"wal-append.queue_time_us":207429}
I20260709 12:25:38.296931  8432 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:38.297402  8432 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:38.194952  8498 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.302155  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:38.307611  6280 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:38.313724  5702 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:38.281904  8566 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:38.326639  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1878) took 2785 ms. Trace:
I20260709 12:25:38.328899  8344 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:38.321907  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.328014  5707 rpcz_store.cc:276] 0709 12:25:35.541132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.241426 (+2700294us) service_pool.cc:224] Handling call
0709 12:25:38.315454 (+ 74028us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.326247 (+ 10793us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:38.326284 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:38.326353 (+    69us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:38.326397 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.326471 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:38.326584 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1218,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":158}
I20260709 12:25:38.329303  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:38.323503  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.330545  8399 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:38.330845  8489 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:38.330961  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1885) took 2787 ms. Trace:
I20260709 12:25:38.331290  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:38.280512  8429 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.331507  5707 rpcz_store.cc:276] 0709 12:25:35.543319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.329889 (+2786570us) service_pool.cc:224] Handling call
0709 12:25:38.330305 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.330589 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:38.330633 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:38.330685 (+    52us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:38.330739 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.330789 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:38.330903 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.332418  8429 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:38.333292  5707 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:38.333740  8542 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.335116  8542 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:38.246686  8537 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.333909  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1843) took 2634 ms. Trace:
I20260709 12:25:38.338958  5707 rpcz_store.cc:276] 0709 12:25:35.699747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.332651 (+2632904us) service_pool.cc:224] Handling call
0709 12:25:38.333055 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.333691 (+   636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.333861 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.339524  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:38.340034  8448 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:25:38.340559  6279 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 579) took 1867 ms (client timeout 1665 ms). Trace:
I20260709 12:25:38.340921  8448 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:38.340988  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1844) took 2633 ms. Trace:
I20260709 12:25:38.341421  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1732) took 2192 ms. Trace:
I20260709 12:25:38.341799  5561 rpcz_store.cc:276] 0709 12:25:36.148895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.855823 (+706928us) service_pool.cc:224] Handling call
0709 12:25:37.726333 (+870510us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:37.740322 (+ 13989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.341204 (+600882us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b540044eb50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:38.341330 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1979171584}
I20260709 12:25:38.342384  5701 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:25:38.323256  8564 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.328622  8340 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:38.342967  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1845) took 2627 ms. Trace:
I20260709 12:25:38.343380  8340 consensus_queue.cc:1243] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:25:38.343346  5701 rpcz_store.cc:276] 0709 12:25:35.715815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.341693 (+2625878us) service_pool.cc:224] Handling call
0709 12:25:38.342048 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.342735 (+   687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.342905 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.344152  8340 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:38.344525  8340 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:38.344584  8537 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:38.233340  8328 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.345522  8340 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.346844  8340 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:38.347352  8473 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:38.347800  8473 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:38.348744  8473 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:36.534827  5445 catalog_manager.cc:5028] TS 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): tablet f0f7b2d2a7e34d629491c96dbfbc7fdc (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
W20260709 12:25:38.351740  5445 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 1.8167e+06us
I20260709 12:25:38.277882  8546 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:38.352592  8546 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:38.353761  5553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1738) took 2203 ms (client timeout 1680 ms). Trace:
I20260709 12:25:38.353507  8546 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.354542  8546 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:38.315953  8595 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:38.331033  8399 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.357126  8399 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:38.358059  8564 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:38.357702  8592 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.358645  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:38.358922  6275 raft_consensus.cc:2368] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7.
I20260709 12:25:38.359849  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:38.361101  5635 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:38.361413  5635 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:38.363157  8473 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:38.365085  5555 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:38.365211  8592 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:38.366641  5555 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
W20260709 12:25:38.340991  6279 rpcz_store.cc:269] 0709 12:25:36.473492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.474276 (+   784us) service_pool.cc:224] Handling call
0709 12:25:38.339235 (+1864959us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:38.340491 (+  1256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30865280}
I20260709 12:25:38.369095  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:38.369879  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 468) took 1675 ms. Trace:
I20260709 12:25:38.370260  6279 raft_consensus.cc:2410] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:38.370342  5556 rpcz_store.cc:276] 0709 12:25:36.694686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.124219 (+1429533us) service_pool.cc:224] Handling call
0709 12:25:38.168287 (+ 44068us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.255375 (+ 87088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.369634 (+114259us) spinlock_profiling.cc:243] Waited 86.5 ms on lock 0x7b54000ebad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:38.369816 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":198869248}
I20260709 12:25:38.371475  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:38.324721  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1837) took 4921 ms. Trace:
I20260709 12:25:38.362769  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:38.372318  5556 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:38.380134  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1643) took 1563 ms. Trace:
I20260709 12:25:38.380667  5636 rpcz_store.cc:276] 0709 12:25:36.816960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.963122 (+1146162us) service_pool.cc:224] Handling call
0709 12:25:37.963465 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.092556 (+129091us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.379874 (+287318us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b5400438d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:38.380063 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":289265024}
I20260709 12:25:38.381911  5556 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:25:38.383899  8462 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:38.276961  8078 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:38.373492  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:38.386209  6271 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:38.387063  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:38.387715  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:38.388461  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:38.376822  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:38.389063  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.362020  8346 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:38.342968  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:38.390779  6277 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:38.390861  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:38.391563  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 586) took 1878 ms. Trace:
I20260709 12:25:38.393113  8311 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:25:38.393705  6274 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 630) took 1716 ms (client timeout 1890 ms). Trace:
I20260709 12:25:38.394451  8371 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:25:38.394995  6275 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 575) took 1926 ms (client timeout 1842 ms). Trace:
I20260709 12:25:38.361956  5631 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:25:38.394961  8371 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:38.395579  6275 rpcz_store.cc:269] 0709 12:25:36.469034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.474030 (+  4996us) service_pool.cc:224] Handling call
0709 12:25:37.789593 (+1315563us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:38.394809 (+605216us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b5400534dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:38.394930 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1346592256}
I20260709 12:25:38.396992  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1651) took 1549 ms. Trace:
I20260709 12:25:38.397063  8192 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:38.397418  6275 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:25:38.397504  5634 rpcz_store.cc:276] 0709 12:25:36.853109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.947223 (+1094114us) service_pool.cc:224] Handling call
0709 12:25:38.061331 (+114108us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.271109 (+209778us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.396780 (+125671us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b540008bbd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:38.396935 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":480898432}
I20260709 12:25:38.398207  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 587) took 1884 ms. Trace:
I20260709 12:25:38.406194  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1873) took 4606 ms. Trace:
I20260709 12:25:38.399734  8192 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:25:38.406728  5676 connection.cc:256] RPC call timeout handler was delayed by -1.2063810310000918s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
I20260709 12:25:38.406675  5709 rpcz_store.cc:276] 0709 12:25:33.800053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.773577 (+3973524us) service_pool.cc:224] Handling call
0709 12:25:37.773982 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.087667 (+313685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.405978 (+318311us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b54002f4254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:38.406132 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":651445632}
I20260709 12:25:38.407120  8192 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.367106  8591 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.409536  8192 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:38.399401  8371 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:38.354195  5553 rpcz_store.cc:269] 0709 12:25:36.150640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.347214 (+196574us) service_pool.cc:224] Handling call
0709 12:25:38.353544 (+2006330us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b54000271d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:38.353688 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3797446528}
I20260709 12:25:38.410605  8371 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:38.409668  5559 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:38.372623  5706 rpcz_store.cc:276] 0709 12:25:33.403381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.705052 (+4301671us) service_pool.cc:224] Handling call
0709 12:25:37.731311 (+ 26259us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.083139 (+351828us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.324458 (+241319us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b5400324454. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:38.324641 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":762472704}
I20260709 12:25:38.409639  8591 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:38.341424  5707 rpcz_store.cc:276] 0709 12:25:35.707886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.339742 (+2631856us) service_pool.cc:224] Handling call
0709 12:25:38.340338 (+   596us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.340661 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:38.340698 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:38.340739 (+    41us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:38.340788 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.340841 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:38.340939 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.413000  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:38.413986  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:38.413854  5706 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:38.414829  5707 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 4: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:38.415207  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:38.415649  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:38.415886  5706 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
W20260709 12:25:38.400292  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.363884  5635 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.372287  6279 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:38.391973  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:38.331028  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:38.400533  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.400599  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:38.402194  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1860) took 1796 ms. Trace:
W20260709 12:25:38.403007  5627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 1975) took 1764 ms (client timeout 1520 ms). Trace:
I20260709 12:25:38.403184  8586 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:25:38.403781  5560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 462) took 1710 ms (client timeout 2055 ms). Trace:
I20260709 12:25:38.405141  8556 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:25:38.394158  6274 rpcz_store.cc:269] 0709 12:25:36.677277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.832747 (+155470us) service_pool.cc:224] Handling call
0709 12:25:38.393473 (+1560726us) spinlock_profiling.cc:243] Waited 769 ms on lock 0x7b54002e2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:38.393650 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1767646976}
I20260709 12:25:38.405794  8078 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:25:38.376521  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.406677  8346 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.412124  5559 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:38.417379  8538 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20260709 12:25:38.426072  8538 consensus_queue.cc:1243] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:25:38.426157  5633 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:38.406245  6275 rpcz_store.cc:276] 0709 12:25:36.513801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.396679 (+1882878us) service_pool.cc:224] Handling call
0709 12:25:38.397083 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.397954 (+   871us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.398157 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.394168  8311 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:38.427760  8311 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:38.428194  8311 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:38.428242  6275 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:25:38.428653  6275 pending_rounds.cc:85] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 6
I20260709 12:25:38.417547  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:38.431267  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:38.431887  5706 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:38.418323  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 618) took 1849 ms. Trace:
I20260709 12:25:38.418531  8481 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:38.433343  6279 rpcz_store.cc:276] 0709 12:25:36.568805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.371160 (+1802355us) service_pool.cc:224] Handling call
0709 12:25:38.418251 (+ 47091us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.434048  8481 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.435302  8481 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:38.435876  6279 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:38.416303  8432 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.441401  5707 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:25:38.419248  8333 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:38.420074  8554 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260709 12:25:38.421914  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:38.418867  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:38.444468  8566 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:38.420136  8431 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.445329  8595 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:38.391932  6277 rpcz_store.cc:276] 0709 12:25:36.513530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.390122 (+1876592us) service_pool.cc:224] Handling call
0709 12:25:38.390547 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.391284 (+   737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.391471 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.420791  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:38.417630  5555 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:25:38.423862  6274 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:25:38.474377  6274 pending_rounds.cc:85] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 6
I20260709 12:25:38.474730  6274 pending_rounds.cc:107] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:25:38.474970  6274 raft_consensus.cc:2889] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:38.424207  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:38.419319  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 88387.7us
I20260709 12:25:38.424666  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:38.420894  6273 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 624) took 1903 ms (client timeout 2244 ms). Trace:
W20260709 12:25:38.476845  6273 rpcz_store.cc:269] 0709 12:25:36.517622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.400247 (+1882625us) service_pool.cc:224] Handling call
0709 12:25:38.420824 (+ 20577us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:38.405187  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 73999.6us
I20260709 12:25:38.422693  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:38.421030  5703 rpcz_store.cc:276] 0709 12:25:36.606292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.631554 (+1025262us) service_pool.cc:224] Handling call
0709 12:25:38.401975 (+770421us) spinlock_profiling.cc:243] Waited 31.7 ms on lock 0x7b54002d23d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:38.402107 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":72970624}
I20260709 12:25:38.479089  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:38.480134  6278 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20260709 12:25:38.424290  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.428875  6275 pending_rounds.cc:107] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:25:38.481401  6275 raft_consensus.cc:2889] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:38.451597  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:38.452589  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1892) took 1946 ms. Trace:
I20260709 12:25:38.482098  5702 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:38.482038  5707 rpcz_store.cc:276] 0709 12:25:36.506269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.438816 (+1932547us) service_pool.cc:224] Handling call
0709 12:25:38.441002 (+  2186us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.452250 (+ 11248us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.452464 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:38.421167  5627 rpcz_store.cc:269] 0709 12:25:36.638109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.653959 (+ 15850us) service_pool.cc:224] Handling call
0709 12:25:38.402669 (+1748710us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b5400134554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:38.402808 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":72165248}
I20260709 12:25:38.377453  5701 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260709 12:25:38.422435  5560 rpcz_store.cc:269] 0709 12:25:36.693256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.814248 (+120992us) service_pool.cc:224] Handling call
0709 12:25:38.403612 (+1589364us) spinlock_profiling.cc:243] Waited 698 ms on lock 0x7b54003a93d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:38.403728 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1604376832}
I20260709 12:25:38.424738  5559 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:38.455833  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.456789  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 614) took 1906 ms. Trace:
I20260709 12:25:38.378278  5629 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:38.485482  6279 rpcz_store.cc:276] 0709 12:25:36.544476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.434711 (+1890235us) service_pool.cc:224] Handling call
0709 12:25:38.435416 (+   705us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.450565 (+ 15149us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.450798 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.486568  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:38.421001  8596 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:38.489682  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 620) took 1895 ms. Trace:
I20260709 12:25:38.490140  6278 rpcz_store.cc:276] 0709 12:25:36.585531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.282318 (+1696787us) service_pool.cc:224] Handling call
0709 12:25:38.480495 (+198177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.490176  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 592) took 1965 ms. Trace:
I20260709 12:25:38.491317  6274 rpcz_store.cc:276] 0709 12:25:36.515195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.423132 (+1907937us) service_pool.cc:224] Handling call
0709 12:25:38.423442 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.475502 (+ 52060us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.480267 (+  4765us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.422765  8556 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:38.491107  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 625) took 1973 ms. Trace:
I20260709 12:25:38.495616  6275 rpcz_store.cc:276] 0709 12:25:36.517868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.427543 (+1909675us) service_pool.cc:224] Handling call
0709 12:25:38.427886 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.490870 (+ 62984us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.491064 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.493921  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 468) took 2677 ms. Trace:
W20260709 12:25:38.488039  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 69287.9us
I20260709 12:25:38.496608  5701 rpcz_store.cc:276] 0709 12:25:35.806505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.344304 (+2537799us) service_pool.cc:224] Handling call
0709 12:25:38.377127 (+ 32823us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:38.483229 (+106102us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.483509 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.453714  8333 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:38.498389  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:38.499692  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:38.454067  8338 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.453307  8481 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.506681  6275 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:25:38.506551  8306 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260709 12:25:38.507403  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 599) took 1668 ms. Trace:
I20260709 12:25:38.507817  8306 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:38.508289  8306 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:38.508558  8431 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:38.508891  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 88176.7us
I20260709 12:25:38.509236  8431 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:38.509481  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:38.509251  8306 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.510309  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:38.510994  8306 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:38.512141  8230 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:38.512377  8566 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.514067  8230 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:38.514406  8554 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:38.514570  8230 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:38.515658  5703 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:25:38.511667  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:38.507982  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:38.516278  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1851) took 1950 ms. Trace:
I20260709 12:25:38.509799  8431 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:38.507761  6275 rpcz_store.cc:276] 0709 12:25:36.839398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.504432 (+1665034us) service_pool.cc:224] Handling call
0709 12:25:38.506453 (+  2021us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:38.507120 (+   667us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.507328 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.516693  5703 rpcz_store.cc:276] 0709 12:25:36.566401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.478611 (+1912210us) service_pool.cc:224] Handling call
0709 12:25:38.515332 (+ 36721us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.516044 (+   712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.516230 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.456931  8380 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.424054  8346 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.522495  5701 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260709 12:25:38.523041  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.526643  5636 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:38.523674  5560 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:25:38.515573  8230 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.492720  8456 raft_consensus.cc:2764] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:25:38.495380  8078 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:38.494853  8506 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:38.517287  8462 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
W20260709 12:25:38.534610  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 361932us
I20260709 12:25:38.587395  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 596) took 1911 ms. Trace:
I20260709 12:25:38.646328  6274 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:38.645949  6273 rpcz_store.cc:276] 0709 12:25:36.633511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.477768 (+1844257us) service_pool.cc:224] Handling call
0709 12:25:38.519978 (+ 42210us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.528874 (+  8896us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:38.528916 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:38.528974 (+    58us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:38.544980 (+ 16006us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.545089 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:38.545274 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.629585  5703 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:38.647126  5559 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:38.517124  8333 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:38.581940  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:38.630209  6277 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:25:38.584728  8340 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:25:38.573535  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50068.1us
I20260709 12:25:38.507397  8481 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:38.507397  8338 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:38.525416  8586 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:38.645164  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.537158  8078 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.647917  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 470) took 1952 ms. Trace:
I20260709 12:25:38.649276  6279 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:38.649390  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1674) took 1071 ms. Trace:
I20260709 12:25:38.661104  5628 rpcz_store.cc:276] 0709 12:25:37.563821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.865189 (+301368us) service_pool.cc:224] Handling call
0709 12:25:38.634852 (+769663us) spinlock_profiling.cc:243] Waited 44.8 ms on lock 0x7b54003afa54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:38.635017 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103085824}
I20260709 12:25:38.655546  6275 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:38.662190  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:38.648535  5706 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:38.567361  8454 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:38.659799  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 597) took 2015 ms. Trace:
I20260709 12:25:38.668735  6274 rpcz_store.cc:276] 0709 12:25:36.633781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.492096 (+1858315us) service_pool.cc:224] Handling call
0709 12:25:38.630537 (+138441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.646888 (+ 16351us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.649533 (+  2645us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.660640  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 595) took 2065 ms. Trace:
W20260709 12:25:38.663723  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 169878us
I20260709 12:25:38.666993  5702 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:38.670156  6277 rpcz_store.cc:276] 0709 12:25:36.595126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.486387 (+1891261us) service_pool.cc:224] Handling call
0709 12:25:38.629987 (+143600us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:38.660308 (+ 30321us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.660576 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:38.672147  5557 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
W20260709 12:25:38.673781  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 162239us
I20260709 12:25:38.660508  5552 rpcz_store.cc:276] 0709 12:25:36.695164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.291848 (+1596684us) service_pool.cc:224] Handling call
0709 12:25:38.292216 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.313960 (+ 21744us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:25:38.322532 (+  8572us) log.cc:844] Serialized 14 byte log entry
0709 12:25:38.647226 (+324694us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:38.647339 (+   113us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:38.647422 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.647591 (+   169us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:38.647801 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":183,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":63531,"thread_start_us":263965,"threads_started":1}
I20260709 12:25:38.676100  6273 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: []
W20260709 12:25:38.686825  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:38.687615  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.690848  5702 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 12:25:38.636751  8506 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:25:38.694680  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 183533us
I20260709 12:25:38.655846  8593 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:38.700930  5703 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:25:38.664587  8078 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:38.670051  8432 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:38.703074  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:38.707273  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45157.6us
I20260709 12:25:38.704164  6274 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:38.705858  5706 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:25:38.695246  8566 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:38.714924  5703 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:38.702077  8506 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:38.649753  8380 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.717962  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:38.659596  8399 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.661751  8586 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.758966  5561 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
W20260709 12:25:38.817610  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 58318.9us
I20260709 12:25:38.711227  8462 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:38.802866  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:38.759377  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 127980us
I20260709 12:25:38.830732  8454 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:38.831503  8454 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:38.831727  5552 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260709 12:25:38.833173  8617 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 12:25:38.832661  8454 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.714864  8566 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:38.862145  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 139571us
I20260709 12:25:38.674140  8595 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:38.868538  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52785.6us
W20260709 12:25:38.871119  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.872673  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:38.876444  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:38.877652  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:38.878243  8503 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.644s	user 0.011s	sys 0.008s
I20260709 12:25:38.878790  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:38.879549  6277 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:38.879199  8503 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.880769  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:38.884034  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 166186us
I20260709 12:25:38.886631  6277 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:38.887429  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.893810  6279 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:38.865852  8566 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:38.902235  8348 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.623s	user 0.009s	sys 0.001s
I20260709 12:25:38.896309  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:38.908980  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:38.916482  5630 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
W20260709 12:25:38.917822  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 220144us
W20260709 12:25:38.890628  8619 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:25:38.925426  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1852) took 2345 ms. Trace:
W20260709 12:25:38.893146  8473 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.529s	user 0.010s	sys 0.000s
W20260709 12:25:38.926826  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:38.930398  8625 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
W20260709 12:25:38.935963  8605 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.761s	user 0.003s	sys 0.000s
I20260709 12:25:38.843233  8462 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:38.924836  8546 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:38.941942  8399 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:38.758406  8431 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:38.954113  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 471) took 1325 ms. Trace:
W20260709 12:25:38.955150  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.926681  5707 rpcz_store.cc:276] 0709 12:25:36.566724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.494691 (+1927967us) service_pool.cc:224] Handling call
0709 12:25:38.505826 (+ 11135us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.506032 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:25:38.670843 (+164811us) log.cc:844] Serialized 14 byte log entry
0709 12:25:38.895249 (+224406us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:38.895377 (+   128us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:38.911580 (+ 16203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.911714 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:38.911922 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":358769,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":13335,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":336806,"threads_started":2,"wal-append.queue_time_us":224175}
I20260709 12:25:38.955191  5558 rpcz_store.cc:276] 0709 12:25:37.620813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.295788 (+674975us) service_pool.cc:224] Handling call
0709 12:25:38.296123 (+   335us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:38.444405 (+148282us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:38.444436 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:38.445145 (+   709us) log.cc:844] Serialized 52 byte log entry
0709 12:25:38.649710 (+204565us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:38.863545 (+213835us) log.cc:844] Serialized 14 byte log entry
0709 12:25:38.863849 (+   304us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.863953 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:38.945303 (+ 81350us) raft_consensus.cc:1685] finished
0709 12:25:38.945324 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:38.945794 (+   470us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7376,"prepare.queue_time_us":45327,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":1709,"raft.queue_time_us":138726,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":15736,"thread_start_us":377659,"threads_started":2,"wal-append.queue_time_us":258557}
I20260709 12:25:38.976531  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1728) took 1168 ms. Trace:
I20260709 12:25:38.977136  5559 rpcz_store.cc:276] 0709 12:25:37.808449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.409219 (+600770us) service_pool.cc:224] Handling call
0709 12:25:38.409506 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:38.975982 (+566476us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:38.976041 (+    59us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:38.976100 (+    59us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:38.976159 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.976253 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:38.976452 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:38.988018  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 79101.8us
I20260709 12:25:38.989703  8346 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:38.967803  8591 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.954411  8546 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:38.954470  8392 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:25:38.999187  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:39.006568  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 132503us
W20260709 12:25:39.007319  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:39.020257  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:38.991523  8591 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.033442  8591 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:39.016294  8346 raft_consensus.cc:2764] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:25:39.023090  8503 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:39.036875  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.038205  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.039911  5555 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:25:39.044246  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:39.044982  8626 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.029s
I20260709 12:25:39.046983  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:39.047879  5676 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:39.048655  8399 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:39.049471  8399 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:39.049880  8399 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:39.050297  8399 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:39.052572  8554 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:39.054273  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:39.055228  8591 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:39.001135  8546 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:39.067842  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 636) took 1115 ms. Trace:
I20260709 12:25:39.068511  6280 rpcz_store.cc:276] 0709 12:25:37.951955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.974269 (+ 22314us) service_pool.cc:224] Handling call
0709 12:25:39.067627 (+1093358us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b54006379d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:39.067773 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":734019456}
W20260709 12:25:39.070885  8429 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.684s	user 0.008s	sys 0.004s
I20260709 12:25:39.071373  8429 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.072630  5708 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:39.075186  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1850) took 3182 ms. Trace:
I20260709 12:25:39.075749  5708 rpcz_store.cc:276] 0709 12:25:35.893080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.355797 (+2462717us) service_pool.cc:224] Handling call
0709 12:25:38.356188 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:39.073269 (+717081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.074709 (+  1440us) spinlock_profiling.cc:243] Waited 716 ms on lock 0x7b54002d0ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:39.075077 (+   368us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1646761728}
W20260709 12:25:39.077879  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:39.078608  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.081112  5706 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:25:39.025138  8565 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:39.026981  8462 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:39.089452  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:39.027825  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.029927  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:39.030350  8311 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.602s	user 0.008s	sys 0.001s
I20260709 12:25:38.998831  8348 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 12:25:39.001524  8392 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:39.002674  8371 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:25:39.031163  8586 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:39.003044  8380 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:39.039697  8566 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.081489  5706 pending_rounds.cc:85] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 6
I20260709 12:25:39.115329  5706 pending_rounds.cc:107] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:25:39.115700  5706 raft_consensus.cc:2889] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:39.080240  5704 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 469) took 3273 ms (client timeout 1920 ms). Trace:
I20260709 12:25:39.064124  8192 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:39.094655  8473 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:39.119589  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 89749.1us
W20260709 12:25:39.116372  5704 rpcz_store.cc:269] 0709 12:25:35.806813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.558910 (+752097us) service_pool.cc:224] Handling call
0709 12:25:39.079933 (+2521023us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400452ed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:39.080159 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2800258944}
I20260709 12:25:39.112869  8591 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:39.122259  8591 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:39.122617  8448 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.781s	user 0.011s	sys 0.002s
I20260709 12:25:39.123039  8448 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.100855  8392 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:39.118000  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:39.140123  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:39.141422  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.122658  8591 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:39.118002  8192 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:39.099418  8311 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:39.151198  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.150590  8586 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:39.157363  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:39.159142  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:39.161065  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:39.161656  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:39.162144  5634 raft_consensus.cc:2410] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:39.162390  8566 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:39.162607  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 623) took 2570 ms. Trace:
I20260709 12:25:39.162925  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:39.163753  5631 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:39.163892  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:39.164674  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:39.164954  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:39.165246  5631 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:39.166033  8462 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:39.166740  8462 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:39.169220  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 469) took 2474 ms. Trace:
I20260709 12:25:39.169638  5557 rpcz_store.cc:276] 0709 12:25:36.694919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.213881 (+1518962us) service_pool.cc:224] Handling call
0709 12:25:38.273722 (+ 59841us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:38.672804 (+399082us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.168941 (+496137us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x7b540041a0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:39.169156 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":915404032}
I20260709 12:25:39.167996  8462 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:39.173249  5554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1727) took 1365 ms (client timeout 1631 ms). Trace:
W20260709 12:25:39.173692  5554 rpcz_store.cc:269] 0709 12:25:37.808081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.890873 (+ 82792us) service_pool.cc:224] Handling call
0709 12:25:39.173049 (+1282176us) spinlock_profiling.cc:243] Waited 89.9 ms on lock 0x7b540044dc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:39.173187 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":206823168}
I20260709 12:25:39.163329  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:39.176079  5634 raft_consensus.cc:2376] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8. Re-sending same reply.
I20260709 12:25:39.176952  8448 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:39.178408  8192 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.181896  8311 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:39.182185  5635 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:39.183130  8429 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:39.185168  8333 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.521s	user 0.009s	sys 0.001s
I20260709 12:25:39.185681  8333 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.188659  8392 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.192443  8530 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:39.193190  8506 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.195631  8506 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:39.199905  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1894) took 2570 ms. Trace:
I20260709 12:25:39.200327  5701 rpcz_store.cc:276] 0709 12:25:36.629547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.504344 (+1874797us) service_pool.cc:224] Handling call
0709 12:25:38.505998 (+  1654us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:38.526046 (+ 20048us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.199638 (+673592us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b5400074c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:39.199838 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37347584}
I20260709 12:25:39.163259  8586 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:25:39.202049  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.210073  8347 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:39.210477  8347 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:39.211391  8347 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.214063  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.215320  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.215442  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.215839  5554 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:39.214255  5559 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
W20260709 12:25:39.218400  8607 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.041s	user 0.000s	sys 0.003s
I20260709 12:25:39.154527  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:39.162992  6271 rpcz_store.cc:276] 0709 12:25:36.592019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.388001 (+1795982us) service_pool.cc:224] Handling call
0709 12:25:39.161570 (+773569us) spinlock_profiling.cc:243] Waited 61.5 ms on lock 0x7b5c0044010c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffdfdc161 00007f4ffe17da79 00007f4fff6379b6 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:39.162472 (+   902us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":141511936}
I20260709 12:25:39.162407  5636 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:39.227010  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:39.220057  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.227943  5557 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:39.229409  6275 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.540s	user 0.012s	sys 0.000s
I20260709 12:25:39.229755  6275 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:39.180186  8192 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:39.231989  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 64692.7us
W20260709 12:25:39.192466  8632 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 12:25:39.244222  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 89786.7us
W20260709 12:25:39.246968  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 140171us
I20260709 12:25:39.188266  8333 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:39.204257  8392 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:39.233824  8429 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:25:39.264281  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 108504us
I20260709 12:25:39.204567  8595 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 12:25:39.250895  8565 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:39.261581  8448 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:25:39.271708  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 136345us
I20260709 12:25:39.281714  5675 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:39.265898  8347 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:39.285238  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1888) took 2992 ms. Trace:
I20260709 12:25:39.273615  8565 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.265218  8399 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.266160  8586 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.019s
W20260709 12:25:39.294777  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.295156  6277 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
W20260709 12:25:39.305065  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.322366  5554 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:39.322544  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:39.300544  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 94584.1us
I20260709 12:25:39.276818  8595 consensus_queue.cc:1243] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:25:39.316766  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:39.318370  5701 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:39.293952  8454 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:39.286809  5709 rpcz_store.cc:276] 0709 12:25:36.293275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.407908 (+2114633us) service_pool.cc:224] Handling call
0709 12:25:38.408273 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:38.425318 (+ 17045us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:25:38.671183 (+245865us) log.cc:844] Serialized 14 byte log entry
0709 12:25:39.098620 (+427437us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:39.118694 (+ 20074us) log.cc:844] Serialized 26 byte log entry
0709 12:25:39.118907 (+   213us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:39.118952 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.119015 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:39.285035 (+166020us) raft_consensus.cc:1685] finished
0709 12:25:39.285054 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:39.285182 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51773,"prepare.queue_time_us":289453,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":45,"raft.queue_time_us":12270,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":16452,"thread_start_us":625684,"threads_started":3,"wal-append.queue_time_us":201618}
W20260709 12:25:39.308235  8078 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.589s	user 0.001s	sys 0.010s
I20260709 12:25:39.294625  8565 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:39.334136  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:39.335111  5704 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:39.336426  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:39.337462  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:39.341801  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:39.342316  8629 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.356s	user 0.001s	sys 0.001s
I20260709 12:25:39.343750  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:39.344182  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1729) took 1535 ms. Trace:
I20260709 12:25:39.344280  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:39.344534  8566 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:39.344576  5553 rpcz_store.cc:276] 0709 12:25:37.808775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.410788 (+602013us) service_pool.cc:224] Handling call
0709 12:25:38.411143 (+   355us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:38.488417 (+ 77274us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:38.488448 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:38.502247 (+ 13799us) log.cc:844] Serialized 52 byte log entry
0709 12:25:38.657073 (+154826us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:38.657114 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.658536 (+  1422us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:39.341644 (+683108us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:39.343850 (+  2206us) raft_consensus.cc:1685] finished
0709 12:25:39.343870 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:39.344105 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45537,"raft.queue_time_us":77441,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":5419,"spinlock_wait_cycles":29595520,"thread_start_us":70093,"threads_started":1,"wal-append.queue_time_us":363695}
I20260709 12:25:39.346208  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:39.346958  8498 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:39.347455  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:39.349273  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:39.350286  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:39.350968  8618 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:39.351606  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:39.352954  8542 raft_consensus.cc:2764] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:25:39.352837  8380 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:39.353269  8328 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:39.353729  8328 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.353874  8554 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:39.354214  8328 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:39.354331  8630 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.365s	user 0.001s	sys 0.000s
I20260709 12:25:39.354079  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:39.357089  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:39.357189  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1725) took 1568 ms. Trace:
W20260709 12:25:39.357914  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.357674  5556 rpcz_store.cc:276] 0709 12:25:37.789138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.384463 (+595325us) service_pool.cc:224] Handling call
0709 12:25:38.384885 (+   422us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:38.495039 (+110154us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:25:38.530527 (+ 35488us) log.cc:844] Serialized 14 byte log entry
0709 12:25:38.706144 (+175617us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:38.758363 (+ 52219us) log.cc:844] Serialized 78 byte log entry
0709 12:25:38.758575 (+   212us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:38.864660 (+106085us) log.cc:844] Serialized 14 byte log entry
0709 12:25:38.866645 (+  1985us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.866778 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:39.353680 (+486902us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:39.356970 (+  3290us) raft_consensus.cc:1685] finished
0709 12:25:39.356990 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:39.357126 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":128525,"prepare.queue_time_us":161785,"prepare.run_cpu_time_us":3307,"prepare.run_wall_time_us":48759,"raft.queue_time_us":107251,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":11803,"spinlock_wait_cycles":9329408,"thread_start_us":149365,"threads_started":2,"wal-append.queue_time_us":382609}
I20260709 12:25:39.360396  8448 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:39.353740  8542 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:39.361819  8542 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:25:39.363013  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.351701  8618 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:39.345201  5708 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:39.369099  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:39.340616  5635 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 12:25:39.371716  8641 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:25:39.373040  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1686) took 1116 ms. Trace:
I20260709 12:25:39.373539  5633 rpcz_store.cc:276] 0709 12:25:38.256125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.301779 (+ 45654us) service_pool.cc:224] Handling call
0709 12:25:39.372818 (+1071039us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54001ce8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:39.372964 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":283496192}
I20260709 12:25:39.374135  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:39.375329  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:39.375640  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.578s (SENT) [suppressed 30 similar messages]
W20260709 12:25:39.377365  8627 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.388s	user 0.000s	sys 0.001s
I20260709 12:25:39.377398  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:39.379767  5702 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:39.380803  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:39.377820  8546 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.378551  8392 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260709 12:25:39.384778  8556 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.385939  8556 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:39.379154  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:39.355926  8554 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:25:39.391855  5705 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:39.393765  8311 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:25:39.394618  8495 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:39.395170  8495 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:39.396189  8495 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.362979  8542 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.398804  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1688) took 1142 ms. Trace:
I20260709 12:25:39.399389  5630 rpcz_store.cc:276] 0709 12:25:38.256730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.273291 (+ 16561us) service_pool.cc:224] Handling call
0709 12:25:39.398637 (+1125346us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b5400438354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:39.398748 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1436303232}
I20260709 12:25:39.400195  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1682) took 1162 ms. Trace:
I20260709 12:25:39.400777  5635 rpcz_store.cc:276] 0709 12:25:38.208548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.360681 (+152133us) service_pool.cc:224] Handling call
0709 12:25:38.360982 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:39.181768 (+820786us) spinlock_profiling.cc:243] Waited 9.32 ms on lock 0x7b54003f00a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e7a1d 00007f4ffa5ce1a4 00007f4ffa650447 00007f4ffa668fe7 00007f4ffa66a7de 00007f4ffa674b49 00007f4ffa675416 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419
0709 12:25:39.370451 (+188683us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.370703 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21433728}
W20260709 12:25:39.406617  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53556.7us
I20260709 12:25:39.407492  6275 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:39.406647  5557 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:39.368486  8618 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.404666  8371 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260709 12:25:39.409642  6277 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:25:39.409807  6276 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:25:39.410094  8432 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:25:39.410323  8348 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:25:39.410684  6274 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:25:39.415118  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:39.410874  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:39.412981  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:39.335381  8078 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.411373  8538 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260709 12:25:39.420228  8230 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:25:39.421352  5628 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: [5.7-7.8]
I20260709 12:25:39.409958  8473 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:39.422384  5702 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:39.429859  5703 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:39.434069  8432 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.435015  5627 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:39.434746  8432 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.413904  8618 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:39.437137  8586 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:25:39.421437  8078 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:39.441856  8628 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.451s	user 0.001s	sys 0.001s
I20260709 12:25:39.442514  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:39.403721  8392 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.025s
I20260709 12:25:39.448777  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:39.451556  5703 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:25:39.421813  8431 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:25:39.424309  8503 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.467224  8503 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.469448  8503 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:39.473349  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:39.474045  5705 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:39.475085  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:39.422916  8626 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:25:39.477322  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:39.462092  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:39.459702  8542 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:39.462710  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54700.9us
I20260709 12:25:39.435948  8591 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.465186  5703 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:25:39.466543  8454 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:25:39.497139  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:39.501071  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 79031us
W20260709 12:25:39.501497  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52815.3us
I20260709 12:25:39.502703  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:39.505631  8432 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:39.506872  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:39.508785  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.509546  5558 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:25:39.508946  8647 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:25:39.508059  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:39.514748  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.519356  5703 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:39.519565  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.519806  5703 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:39.520197  5556 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:39.500043  8566 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:39.524150  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.462282  8230 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:39.524960  5554 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:39.526324  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.526784  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:39.527103  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.529023  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.529580  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.534008  5558 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 8: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:39.521723  8348 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:39.536697  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:39.523483  8347 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:39.530974  5703 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.541532  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.541265  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.542411  5554 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:39.542613  5560 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 6: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:39.526407  8651 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:25:39.532282  8192 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:39.533299  8554 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.573907  5701 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:25:39.573684  8554 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:39.575976  5456 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 174) took 25488 ms (client timeout 15000 ms). Trace:
I20260709 12:25:39.577455  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:39.579113  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:39.554700  5703 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:39.584596  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48229.8us
W20260709 12:25:39.557217  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:39.587975  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:39.588312  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.559876  8319 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:39.560459  8328 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.560882  8592 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:25:39.562937  8635 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.572s	user 0.000s	sys 0.003s
I20260709 12:25:39.563969  8586 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20260709 12:25:39.567322  8546 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:39.569259  8591 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:39.569380  5552 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:25:39.578859  8565 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.581028  8462 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:39.533412  8431 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:39.536504  8432 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 12:25:39.541082  8454 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:39.541914  8347 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:39.596971  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 88958.5us
I20260709 12:25:39.576149  8554 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:39.576452  5456 rpcz_store.cc:269] 0709 12:25:14.087864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.104503 (+ 16639us) service_pool.cc:224] Handling call
0709 12:25:39.575747 (+25471244us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:32.390462 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:32.461830 (+ 71368us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:32.461879 (+    49us) write_op.cc:435] Acquired schema lock
0709 12:25:32.461898 (+    19us) tablet.cc:662] Decoding operations
0709 12:25:32.462622 (+   724us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:32.462737 (+   115us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:32.462774 (+    37us) tablet.cc:685] Acquiring locks for 5 operations
0709 12:25:32.463873 (+  1099us) tablet.cc:701] Row locks acquired
0709 12:25:32.463929 (+    56us) write_op.cc:260] PREPARE: finished
0709 12:25:32.464160 (+   231us) write_op.cc:270] Start()
0709 12:25:32.464337 (+   177us) write_op.cc:276] Timestamp: P: 1783599932464069 usec, L: 0
0709 12:25:32.497414 (+ 33077us) op_driver.cc:348] REPLICATION: starting
0709 12:25:32.545414 (+ 48000us) log.cc:844] Serialized 2020 byte log entry
0709 12:25:33.169339 (+623925us) op_driver.cc:464] REPLICATION: finished
0709 12:25:33.295417 (+126078us) write_op.cc:301] APPLY: starting
0709 12:25:33.308907 (+ 13490us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:33.324937 (+ 16030us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:33.333504 (+  8567us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:33.388608 (+ 55104us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:33.388957 (+   349us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0709 12:25:33.389018 (+    61us) write_op.cc:312] APPLY: finished
0709 12:25:33.389397 (+   379us) log.cc:844] Serialized 47 byte log entry
0709 12:25:33.389610 (+   213us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:33.389725 (+   115us) write_op.cc:454] Released schema lock
0709 12:25:33.389786 (+    61us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:33.390233 (+   447us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:33.390279 (+    46us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":9121533,"reported_tablets":13,"child_traces":[["op",{"apply.queue_time_us":82185,"apply.run_cpu_time_us":4076,"apply.run_wall_time_us":122207,"num_ops":5,"prepare.queue_time_us":293405,"prepare.run_cpu_time_us":30649,"prepare.run_wall_time_us":357503,"raft.queue_time_us":89756,"raft.run_cpu_time_us":5256,"raft.run_wall_time_us":23581,"replication_time_us":671721,"thread_start_us":524969,"threads_started":4,"wal-append.queue_time_us":477896,"wal-append.run_cpu_time_us":22512,"wal-append.run_wall_time_us":1402584}]]}
I20260709 12:25:39.628342  5703 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:25:39.552073  8538 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:39.592370  8348 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:39.621232  8591 raft_consensus.cc:2764] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:25:39.621829  8431 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.613951  8462 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:39.612594  8495 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.600791  8328 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:39.623584  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1737) took 1311 ms. Trace:
I20260709 12:25:39.654436  5561 rpcz_store.cc:276] 0709 12:25:38.312564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.342689 (+ 30125us) service_pool.cc:224] Handling call
0709 12:25:39.623399 (+1280710us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x7b54001bec54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:39.623513 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":773405568}
I20260709 12:25:39.629395  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:39.634583  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.648933  8586 consensus_queue.cc:1243] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:25:39.600399  8654 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:39.656116  8654 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:39.657059  8654 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.658365  8654 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:39.659219  8462 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:39.659291  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:39.659942  5702 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:39.663444  8495 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.653584  8230 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.665649  8565 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:39.668152  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 311135us
I20260709 12:25:39.667856  8347 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.669202  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:39.670156  8473 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:39.670331  8595 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:39.670994  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.670806  8595 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:39.671429  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.671983  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.674059  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:39.676527  5557 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:39.672473  8380 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:39.672950  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 481) took 1262 ms. Trace:
I20260709 12:25:39.683182  5706 rpcz_store.cc:276] 0709 12:25:38.410718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.709292 (+298574us) service_pool.cc:224] Handling call
0709 12:25:39.080722 (+371430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:39.129899 (+ 49177us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.653905 (+524006us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b5400452ed0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:39.672885 (+ 18980us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":853494656}
I20260709 12:25:39.669268  8448 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:39.673689  8456 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:25:39.674057  5553 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
I20260709 12:25:39.674458  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:39.676831  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:39.681134  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 178734us
I20260709 12:25:39.693226  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:39.680568  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 121444us
I20260709 12:25:39.682637  5703 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:39.695482  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.695528  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:39.696206  5560 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:39.671496  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:39.697013  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:39.689999  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:39.700403  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.703264  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.665640  8495 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:39.703691  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:39.702715  8448 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:39.711252  8554 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:39.711862  8348 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:39.712550  8568 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:39.712980  8568 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.713359  8568 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:39.720649  5557 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:39.721236  5561 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:39.722294  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.722976  5561 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:39.723963  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.724848  8319 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:25:39.708041  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:39.725315  5554 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:39.725835  5627 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:39.726795  5554 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:39.724759  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:39.728159  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.728950  5560 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260709 12:25:39.678433  8538 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:39.688169  5702 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:39.682637  8595 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:39.708384  5553 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:39.682480  8473 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:39.712083  8448 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.731084  5561 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:25:39.696764  8662 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:39.733670  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:39.706821  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:39.746663  5557 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
W20260709 12:25:39.737679  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 280333us
I20260709 12:25:39.749293  8473 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:39.757696  8546 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:25:39.760236  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:39.760296  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.761350  5557 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:39.767189  5627 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:39.736023  8456 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:39.768680  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:25:39.770378  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63629.4us
I20260709 12:25:39.736079  8554 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:39.752794  8319 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:39.776409  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:39.777767  8591 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:39.779306  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.780404  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.781015  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.794797  5558 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:39.750730  8530 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:25:39.784791  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48126us
I20260709 12:25:39.795554  8530 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.795934  8530 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:39.799269  5526 leader_election.cc:336] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.890s (SENT)
I20260709 12:25:39.745455  8662 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:39.751904  8348 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:39.801259  8348 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.801630  8662 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.803073  8662 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:39.803717  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:39.804788  8542 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:39.805683  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:39.806511  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:39.806814  5635 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:25:39.807518  5631 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:39.807839  5627 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:25:39.805862  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:39.808568  5630 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:39.808851  5634 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:39.810729  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:39.811408  5632 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:39.814414  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:39.816766  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:39.818328  8596 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:25:39.824581  5560 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:25:39.859401  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:39.859915  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 113501us
I20260709 12:25:39.786574  8565 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:39.831143  8338 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.034s
I20260709 12:25:39.865835  8338 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:39.866223  8338 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:39.866501  8568 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.867033  8338 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.868185  8338 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:39.872716  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:39.873390  5561 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:39.874302  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.875684  8347 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:39.876494  8538 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:39.877158  8669 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.226s	user 0.001s	sys 0.000s
W20260709 12:25:39.870189  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 93915.2us
I20260709 12:25:39.871495  8652 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:25:39.871455  8333 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:39.871641  8546 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.870201  8556 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:39.880129  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:39.880690  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.881433  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:39.881865  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.794127  8591 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:39.902607  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:39.903970  5632 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
W20260709 12:25:39.904621  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 325578us
I20260709 12:25:39.829329  8542 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:25:39.829313  8495 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:25:39.916918  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 51881.1us
W20260709 12:25:39.908762  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 80845.5us
I20260709 12:25:39.787041  8626 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.108s
W20260709 12:25:39.920228  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.923048  8591 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.924541  8591 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:39.924754  8673 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:25:39.892726  8556 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:39.932056  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:39.933660  8556 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:39.934319  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1879) took 4392 ms. Trace:
W20260709 12:25:39.934931  8659 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
I20260709 12:25:39.936614  8328 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:25:39.938930  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:39.784965  8462 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:39.947765  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:39.934732  5710 rpcz_store.cc:276] 0709 12:25:35.541427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.265824 (+2724397us) service_pool.cc:224] Handling call
0709 12:25:38.409736 (+143912us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:38.410172 (+   436us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:25:38.498435 (+ 88263us) log.cc:844] Serialized 14 byte log entry
0709 12:25:38.696830 (+198395us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:38.715744 (+ 18914us) log.cc:844] Serialized 52 byte log entry
0709 12:25:38.715984 (+   240us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:38.716024 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.716203 (+   179us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:39.441177 (+724974us) spinlock_profiling.cc:243] Waited 7.08 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:39.444231 (+  3054us) raft_consensus.cc:1685] finished
0709 12:25:39.452989 (+  8758us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:39.934137 (+481148us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54002d23d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:39.934255 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":253951,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":1588,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":5751,"spinlock_wait_cycles":346345216,"thread_start_us":201514,"threads_started":2,"wal-append.queue_time_us":464531}
I20260709 12:25:39.912982  8565 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:39.948760  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:39.958364  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:39.984015  5709 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:39.959331  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:39.915951  8568 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:39.916150  8495 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:39.879901  8546 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:40.007795  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 60347.2us
W20260709 12:25:40.009871  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 50690us
W20260709 12:25:40.009863  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 100280us
I20260709 12:25:40.011878  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:40.013459  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:39.921924  8340 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.086s
I20260709 12:25:40.010546  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:39.987818  8346 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:25:40.017490  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 68914.5us
W20260709 12:25:40.023170  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:40.022537  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 108589us
W20260709 12:25:40.017297  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77619.6us
I20260709 12:25:39.915933  8554 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:40.018129  8230 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:40.010692  8495 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:40.009488  8392 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.033160  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:40.038547  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:40.051636  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:40.034289  8348 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:40.058861  8348 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.059384  8348 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:40.062363  8311 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.062683  8338 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:40.063181  8338 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:40.062929  8311 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:40.043267  8653 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:40.069300  8311 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:40.046936  8230 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.072837  8230 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:40.072937  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:40.074323  8666 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
W20260709 12:25:40.076331  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.006349  8498 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
W20260709 12:25:40.030944  8618 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.594s	user 0.013s	sys 0.002s
I20260709 12:25:40.023213  8346 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.048547  5561 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:25:40.029759  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:40.091591  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.097872  5632 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:40.105319  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.034447  8530 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:40.095986  8665 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:40.021148  8565 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.101512  8346 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:40.064035  8338 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:40.132285  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 107852us
I20260709 12:25:40.136749  5632 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:40.090595  8392 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.133816  8626 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:40.188953  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.193099  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:40.102404  8618 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:40.196909  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:40.198693  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:40.148716  8328 consensus_queue.cc:1243] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:25:40.147099  5636 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: [6.5-8.6]
W20260709 12:25:40.207024  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 155493us
W20260709 12:25:40.207486  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 180252us
I20260709 12:25:40.198693  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:40.196413  8503 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.148522  8679 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:40.226760  5556 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:40.199761  8538 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.234073  8538 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:40.221084  8665 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:40.232995  8546 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:40.219436  8664 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.582s	user 0.000s	sys 0.004s
W20260709 12:25:40.241392  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48454.8us
I20260709 12:25:40.243831  5554 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:25:40.247416  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:40.248859  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:40.227846  8673 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:40.266672  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:40.267577  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:40.270308  5710 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:40.263046  8546 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:40.271143  8670 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.317s	user 0.001s	sys 0.000s
I20260709 12:25:40.271324  8546 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.273761  8546 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:40.274290  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.241884  8554 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:40.279389  5633 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:40.280890  8693 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:40.281888  8677 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 12:25:40.282841  8678 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:25:40.261929  8462 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:40.306813  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:40.309993  8657 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.771s	user 0.003s	sys 0.000s
W20260709 12:25:40.311651  8672 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.282s	user 0.001s	sys 0.000s
I20260709 12:25:40.311933  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:40.312592  5559 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:40.236902  8503 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.303915  8653 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:40.318763  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:40.319595  5557 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:40.322359  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1744) took 1284 ms. Trace:
I20260709 12:25:40.322968  5552 rpcz_store.cc:276] 0709 12:25:39.038122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.043490 (+  5368us) service_pool.cc:224] Handling call
0709 12:25:39.043828 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:39.609070 (+565242us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.322054 (+712984us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b54002283d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:40.322281 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1207400704}
I20260709 12:25:40.324093  8328 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.324803  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:40.258978  8456 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:40.325047  8692 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 12:25:40.325995  8456 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:25:40.327149  8456 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:40.294420  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:40.295655  8687 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:40.297169  8538 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:40.297385  8392 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:40.297588  8689 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:40.291680  8679 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:40.298374  5710 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:40.212647  8626 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.324657  8328 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.331364  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:40.280954  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:40.358307  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 112129us
W20260709 12:25:40.365090  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 84276.8us
I20260709 12:25:40.360716  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:40.360987  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 138772us
I20260709 12:25:40.351915  8687 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.366900  5557 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:40.322424  8429 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:40.366679  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:40.301172  8473 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:25:40.359736  8662 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.556s	user 0.011s	sys 0.004s
I20260709 12:25:40.364095  5710 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:40.355098  8554 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:25:40.367290  8687 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:40.360255  8230 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:40.328605  8333 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.042s
I20260709 12:25:40.360605  8689 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:40.379853  8689 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.381449  5552 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260709 12:25:40.381870  5552 pending_rounds.cc:85] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 2
I20260709 12:25:40.382107  5552 pending_rounds.cc:107] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 12:25:40.382366  5552 raft_consensus.cc:2889] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:40.383821  5552 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:25:40.384073  8333 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:25:40.386653  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:40.358552  8399 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:40.369204  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:40.364344  8679 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.393777  8679 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:40.384532  8684 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.381438  8568 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:40.395694  5710 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.425858  8328 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:40.428150  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:40.403034  5559 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:40.434144  5558 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 8: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:25:40.422919  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.444062  5710 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:40.436216  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:40.434231  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:40.445480  5634 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:40.435834  5702 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:25:40.397583  8429 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:40.459300  8503 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:40.460965  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:40.462402  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:40.464324  5627 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:40.438524  5705 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:25:40.447800  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:40.467053  5632 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:40.467069  6275 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:40.397857  8662 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:40.371973  8554 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:40.397065  8694 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:25:40.414635  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.470201  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:40.473253  6274 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:25:40.493539  6274 pending_rounds.cc:85] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 5
I20260709 12:25:40.408556  8568 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:40.408707  8333 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.039s
I20260709 12:25:40.476542  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:40.476855  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:40.477151  8700 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.009s
I20260709 12:25:40.463184  5552 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:40.396114  8462 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:40.430446  8591 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.505s	user 0.004s	sys 0.004s
I20260709 12:25:40.493829  6274 pending_rounds.cc:107] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:25:40.503710  6274 raft_consensus.cc:2889] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:40.475526  8346 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.428755  8626 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:40.372114  8473 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.496760  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:40.431800  8530 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:40.501435  6273 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
W20260709 12:25:40.510370  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.464987  8596 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:40.426044  8697 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.436229  8665 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.488229  8662 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.529517  8662 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:40.529558  8679 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:40.530198  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:40.531385  5560 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:40.528918  8665 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.531821  5552 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:25:40.531967  8679 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:40.433632  8348 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:40.514609  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:40.517046  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:40.533591  5558 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:40.533833  5630 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:40.413846  8684 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:40.537166  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:40.532927  8626 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.538559  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:40.538962  5631 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:40.516960  8568 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:40.517663  8695 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.019s
I20260709 12:25:40.517760  6277 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:40.517046  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:40.518110  8701 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.521692  8686 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:25:40.522599  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:40.462759  8495 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:40.508997  8319 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.683s	user 0.008s	sys 0.004s
W20260709 12:25:40.529099  8347 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.653s	user 0.011s	sys 0.002s
I20260709 12:25:40.492931  8694 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.435899  8399 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260709 12:25:40.490378  8429 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:25:40.550499  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:40.556193  5710 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:25:40.500085  8328 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260709 12:25:40.591588  6277 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 12:25:40.705796  8649 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:25:40.539498  8697 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.549408  8565 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:25:40.566370  8591 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.104s
I20260709 12:25:40.591049  8593 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.567219  8568 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:40.543598  8596 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:40.597286  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:40.591223  8554 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:40.626893  8694 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:40.552127  8626 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.693588  8699 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.627686  8319 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:40.669953  8665 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:40.695343  8702 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:25:40.567137  8701 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.694087  8704 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.151s
I20260709 12:25:40.590971  8687 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:40.844663  8348 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:40.846666  6272 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:25:40.960886  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:41.235543  6274 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
W20260709 12:25:40.847886  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:40.903239  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.927862  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:40.963557  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.239171  5710 pending_rounds.cc:85] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 5
I20260709 12:25:41.239665  5710 pending_rounds.cc:107] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:25:41.240034  5710 raft_consensus.cc:2889] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:40.847527  5558 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:40.856914  5634 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
W20260709 12:25:41.234239  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 764207us
I20260709 12:25:41.242802  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:41.243435  5634 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:25:41.232575  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 709945us
I20260709 12:25:40.845180  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:41.250428  5630 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:41.237529  6279 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:41.267835  5634 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260709 12:25:40.925838  8693 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.619s	user 0.032s	sys 0.023s
I20260709 12:25:40.927088  8338 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.303755  5635 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:25:41.303877  8338 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:40.955998  8685 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.306442  8685 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:41.307775  5558 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:25:41.308284  8399 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:41.309072  8399 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:41.310250  8703 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.382s	user 0.000s	sys 0.000s
I20260709 12:25:41.311630  6271 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:25:41.204380  8692 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.845585  8347 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.845726  8679 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:41.316838  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:41.338435  6278 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:41.204311  8684 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:41.349699  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70096.5us
I20260709 12:25:41.351048  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1746) took 2221 ms. Trace:
I20260709 12:25:41.351502  5555 rpcz_store.cc:276] 0709 12:25:39.129903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.133187 (+  3284us) service_pool.cc:224] Handling call
0709 12:25:39.133547 (+   360us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:39.226285 (+ 92738us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:39.226317 (+    32us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:39.262142 (+ 35825us) log.cc:844] Serialized 78 byte log entry
0709 12:25:39.327404 (+ 65262us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:39.364967 (+ 37563us) log.cc:844] Serialized 14 byte log entry
0709 12:25:39.667865 (+302898us) spinlock_profiling.cc:243] Waited 7.89 ms on lock 0x7b1800432160. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:25:39.668060 (+   195us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.668141 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:40.439633 (+771492us) spinlock_profiling.cc:243] Waited 4.62 ms on lock 0x7b1800432160. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:25:41.350778 (+911145us) raft_consensus.cc:1685] finished
0709 12:25:41.350805 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:41.350983 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18529,"prepare.queue_time_us":40997,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":345675,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":129,"spinlock_wait_cycles":28780032,"thread_start_us":46104,"threads_started":1,"wal-append.queue_time_us":105326}
I20260709 12:25:41.277258  8665 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.356166  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:41.376731  6277 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:41.277638  8704 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:41.313287  8346 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:40.958663  8662 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:41.259261  8593 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:41.279408  8699 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:41.664228  5556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1761) took 1871 ms (client timeout 2291 ms). Trace:
I20260709 12:25:41.444199  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:41.414256  6275 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.017s	sys 0.023s
I20260709 12:25:41.590080  8626 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:41.804973  5673 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.026s	sys 0.029s
I20260709 12:25:41.762737  5630 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260709 12:25:42.016445  5673 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.891s	user 0.035s	sys 0.030s
I20260709 12:25:41.376906  8692 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.016999  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1694) took 1479 ms. Trace:
I20260709 12:25:42.017513  8695 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:41.686410  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.319072  8693 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:42.018078  8695 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.019045  8692 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:41.394098  8429 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:41.499485  8473 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.019918  8429 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:42.020179  8695 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:41.620486  8700 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:41.354141  8319 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:42.021034  8700 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:42.020825  8429 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.021420  8319 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.021876  8700 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:42.022423  6275 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.177s	user 0.017s	sys 0.068s
I20260709 12:25:42.023161  8700 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:41.414963  8665 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:41.583604  8708 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.328s	user 0.000s	sys 0.000s
W20260709 12:25:41.955088  8712 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
W20260709 12:25:41.870195  5601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.003s	sys 0.019s
I20260709 12:25:41.498974  8596 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:41.844290  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.009s	sys 0.005s
I20260709 12:25:42.025692  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2012) took 1591 ms. Trace:
I20260709 12:25:42.025699  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 491) took 1776 ms. Trace:
W20260709 12:25:42.026266  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.011s	sys 0.005s
W20260709 12:25:41.588256  8714 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:25:41.926229  8399 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:42.026510  5559 rpcz_store.cc:276] 0709 12:25:40.249299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.434722 (+185423us) service_pool.cc:224] Handling call
0709 12:25:40.435022 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:40.458444 (+ 23422us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:40.458474 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:40.479208 (+ 20734us) log.cc:844] Serialized 26 byte log entry
0709 12:25:40.864098 (+384890us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:40.864155 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.864247 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:42.025500 (+1161253us) raft_consensus.cc:1685] finished
0709 12:25:42.025519 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.025633 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":126,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":15328,"thread_start_us":349059,"threads_started":1,"wal-append.queue_time_us":1276912}
I20260709 12:25:42.028146  8732 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.028258  8192 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.028993  8192 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.028836  5628 rpcz_store.cc:276] 0709 12:25:40.434004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.449311 (+ 15307us) service_pool.cc:224] Handling call
0709 12:25:40.474756 (+ 25445us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:40.475125 (+   369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:40.475152 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:40.475611 (+   459us) log.cc:844] Serialized 26 byte log entry
0709 12:25:40.586834 (+111223us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:40.586877 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.586971 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:42.025500 (+1438529us) raft_consensus.cc:1685] finished
0709 12:25:42.025519 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.025633 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7002,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":6058,"thread_start_us":110766,"threads_started":1,"wal-append.queue_time_us":454948}
I20260709 12:25:42.031669  8192 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.032737  8732 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.032204  8192 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:41.591892  8687 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:41.843982  6235 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.008s	sys 0.001s
I20260709 12:25:41.556098  8718 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.037842  8718 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:42.038290  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:42.015211  5557 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.090s	user 0.000s	sys 0.015s
W20260709 12:25:42.040829  5557 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.116s	user 0.000s	sys 0.015s
I20260709 12:25:42.037014  8192 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.041512  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 488) took 1793 ms. Trace:
I20260709 12:25:42.042389  8593 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.043001  8192 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:42.043221  5710 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
W20260709 12:25:41.678112  8462 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.171s	user 0.014s	sys 0.005s
W20260709 12:25:42.015200  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.964s	user 0.005s	sys 0.008s
I20260709 12:25:42.044597  8462 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:25:42.045197  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.994s	user 0.005s	sys 0.008s
I20260709 12:25:42.045842  6273 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:42.045938  6280 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:25:41.862082  8704 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.047879  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:42.014424  8694 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.731s	user 0.009s	sys 0.001s
I20260709 12:25:42.049199  8694 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.049650  5628 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:41.779400  8328 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:42.050683  8328 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.051021  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:42.051913  8694 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.052284  5628 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:42.053413  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:41.834749  5556 rpcz_store.cc:269] 0709 12:25:39.793223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.881157 (+ 87934us) service_pool.cc:224] Handling call
0709 12:25:41.663908 (+1782751us) spinlock_profiling.cc:243] Waited 344 ms on lock 0x7b54000271d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:41.664137 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":791776512}
I20260709 12:25:42.055003  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:42.055178  8473 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:41.546343  8591 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:42.025383  5601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.005s	sys 0.022s
I20260709 12:25:42.058465  8591 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.059688  8626 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.060176  8626 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:41.439980  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:42.060703  8593 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.062145  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 647) took 1104 ms. Trace:
W20260709 12:25:41.865990  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.007s	sys 0.007s
W20260709 12:25:42.063611  6234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.083s	user 0.008s	sys 0.010s
W20260709 12:25:42.078006  6234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.098s	user 0.009s	sys 0.010s
I20260709 12:25:42.077785  6277 rpcz_store.cc:276] 0709 12:25:40.957364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:41.377419 (+420055us) service_pool.cc:224] Handling call
0709 12:25:42.062075 (+684656us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:42.078882  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.913s	user 0.008s	sys 0.008s
W20260709 12:25:42.014696  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 561084us
I20260709 12:25:41.722548  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2007) took 1916 ms. Trace:
I20260709 12:25:42.081182  5636 rpcz_store.cc:276] 0709 12:25:39.806361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.809759 (+  3398us) service_pool.cc:224] Handling call
0709 12:25:39.810171 (+   412us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:40.223861 (+413690us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:41.706309 (+1482448us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b54001f2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:41.722437 (+ 16128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":774064384}
I20260709 12:25:41.439003  8495 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:42.037293  6235 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.757s	user 0.009s	sys 0.002s
I20260709 12:25:42.085985  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 650) took 1127 ms. Trace:
W20260709 12:25:42.086272  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.355540  8347 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:42.017596  5630 rpcz_store.cc:276] 0709 12:25:40.537362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:41.251251 (+713889us) service_pool.cc:224] Handling call
0709 12:25:41.390804 (+139553us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:42.016605 (+625801us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.016930 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:42.098280  5524 connection.cc:256] RPC call timeout handler was delayed by -1.4606099369998446s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1625s for a response
I20260709 12:25:42.046916  8662 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:41.337551  8710 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:42.060775  8346 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:25:42.086392  6274 rpcz_store.cc:276] 0709 12:25:40.958101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:41.349113 (+391012us) service_pool.cc:224] Handling call
0709 12:25:42.085927 (+736814us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:42.022110  8429 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:41.869809  8697 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:42.098814  5560 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.171s	user 0.013s	sys 0.005s
W20260709 12:25:42.140641  5560 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.213s	user 0.013s	sys 0.005s
I20260709 12:25:42.140578  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:42.141392  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1764) took 1697 ms. Trace:
W20260709 12:25:42.098893  5524 connection.cc:256] RPC call timeout handler was delayed by -1.3724099370001568s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
W20260709 12:25:42.142714  5524 connection.cc:256] RPC call timeout handler was delayed by -1.2967381909998039s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15780000000000002s for a response
W20260709 12:25:42.099794  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.051s	user 0.005s	sys 0.013s
I20260709 12:25:42.036330  8732 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:41.869453  8701 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.039981  8718 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:42.077936  8568 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.819s	user 0.010s	sys 0.002s
I20260709 12:25:42.154640  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 632) took 1560 ms. Trace:
I20260709 12:25:42.161702  6280 rpcz_store.cc:276] 0709 12:25:40.594073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.846884 (+252811us) service_pool.cc:224] Handling call
0709 12:25:40.847194 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:42.046680 (+1199486us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.154231 (+107551us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54002ce7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:42.154543 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2756348160}
I20260709 12:25:42.041868  5557 rpcz_store.cc:276] 0709 12:25:40.248377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.389374 (+140997us) service_pool.cc:224] Handling call
0709 12:25:40.458126 (+ 68752us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.458475 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:25:40.539682 (+ 81207us) log.cc:844] Serialized 14 byte log entry
0709 12:25:42.041185 (+1501503us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:42.041240 (+    55us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:25:42.041287 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.041350 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.041459 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63787,"prepare.queue_time_us":1319979,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":454289,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":264,"thread_start_us":1518147,"threads_started":3,"wal-append.queue_time_us":836224}
I20260709 12:25:42.024413  8319 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.049441  8704 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.139624  8495 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.044992  8699 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.137552  5705 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:25:42.137526  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.197191  6277 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 4.
I20260709 12:25:42.197711  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 623) took 1241 ms. Trace:
I20260709 12:25:42.197880  5561 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260709 12:25:42.198035  6277 rpcz_store.cc:276] 0709 12:25:40.956793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.096904 (+1140111us) service_pool.cc:224] Handling call
0709 12:25:42.197646 (+100742us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:42.198348  5561 pending_rounds.cc:85] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 4
I20260709 12:25:42.136852  8693 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:42.147804  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.202142  5702 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:42.097110  8692 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.148492  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:42.149152  5704 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 12:25:42.153873  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.154354  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.203465  5701 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:42.203727  6274 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:42.204237  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 654) took 1245 ms. Trace:
I20260709 12:25:42.140473  8694 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.155164  8715 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.147930  8328 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.096778  8593 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.128228  8687 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.121994  8695 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.141728  5560 rpcz_store.cc:276] 0709 12:25:40.444268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.532302 (+ 88034us) service_pool.cc:224] Handling call
0709 12:25:40.532648 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.532831 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:25:40.755170 (+222339us) log.cc:844] Serialized 14 byte log entry
0709 12:25:42.141065 (+1385895us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:42.141119 (+    54us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:42.141160 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.141229 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.141339 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80095,"prepare.queue_time_us":1260469,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":86,"raft.queue_time_us":468043,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":72,"thread_start_us":1434998,"threads_started":2}
I20260709 12:25:42.014219  8554 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:25:42.198125  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.062268  8626 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:42.143534  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.095s	user 0.005s	sys 0.013s
I20260709 12:25:42.199191  5561 pending_rounds.cc:107] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260709 12:25:42.204569  6274 rpcz_store.cc:276] 0709 12:25:40.959085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.154054 (+1194969us) service_pool.cc:224] Handling call
0709 12:25:42.204177 (+ 50123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:42.206750  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:42.205549  8693 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:42.183100  8568 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.215865  6280 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 6: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:25:42.216694  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.216389  5561 raft_consensus.cc:2889] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:42.231993  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:42.232570  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.233747  5705 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:42.229012  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:42.228571  5707 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:42.229084  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.238654  8665 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:42.239173  8371 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:42.239593  8371 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:42.240553  8371 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.241653  8371 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:42.244568  8626 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:42.229848  8695 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.246694  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1699) took 1701 ms. Trace:
I20260709 12:25:42.247171  6280 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:25:42.247843  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 649) took 1290 ms. Trace:
I20260709 12:25:42.245534  8568 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.247071  5635 rpcz_store.cc:276] 0709 12:25:40.545704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:41.305165 (+759461us) service_pool.cc:224] Handling call
0709 12:25:41.305596 (+   431us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:41.306009 (+   413us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:25:41.306048 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:41.306890 (+   842us) log.cc:844] Serialized 26 byte log entry
0709 12:25:41.730465 (+423575us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:41.730519 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:41.730620 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:42.246462 (+515842us) raft_consensus.cc:1685] finished
0709 12:25:42.246481 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.246624 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":710371,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"thread_start_us":422980,"threads_started":1,"wal-append.queue_time_us":937677}
I20260709 12:25:42.248199  6280 rpcz_store.cc:276] 0709 12:25:40.957849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.246533 (+1288684us) service_pool.cc:224] Handling call
0709 12:25:42.246845 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:42.247596 (+   751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.247785 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:42.249167  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:42.249802  6280 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:25:42.249938  5635 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:25:42.249948  8593 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.250973  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:42.251617  5635 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:42.252920  5635 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:25:42.253181  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 616) took 1758 ms. Trace:
W20260709 12:25:42.256516  5524 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.578s (SENT)
I20260709 12:25:42.256043  5701 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:42.254549  6275 rpcz_store.cc:276] 0709 12:25:40.494998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.499507 (+  4509us) service_pool.cc:224] Handling call
0709 12:25:40.499829 (+   322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:40.514120 (+ 14291us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:25:40.514152 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:40.538805 (+ 24653us) log.cc:844] Serialized 26 byte log entry
0709 12:25:42.023103 (+1484298us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:42.023146 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.023213 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:42.252955 (+229742us) raft_consensus.cc:1685] finished
0709 12:25:42.252977 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.253111 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7955,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":10731,"thread_start_us":1178057,"threads_started":1,"wal-append.queue_time_us":1660408}
I20260709 12:25:42.252856  5702 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:42.254128  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 651) took 1295 ms. Trace:
I20260709 12:25:42.247958  8695 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.213559  8473 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.234632  8591 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.214179  8731 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:42.264950  6280 rpcz_store.cc:276] 0709 12:25:40.958365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.249281 (+1290916us) service_pool.cc:224] Handling call
0709 12:25:42.249602 (+   321us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:42.253895 (+  4293us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.254053 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:42.300781  8462 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.301779  8462 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.213989  8692 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.230243  8687 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.214701  8694 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:42.270612  8720 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:42.270438  8729 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.269932  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.281011  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.282922  8328 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.358127  5709 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:42.267131  8734 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:42.271142  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.281754  8726 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.282922  5706 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:42.272811  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:42.362102  5630 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:42.284603  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.277295  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.285326  8568 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.288456  8722 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.217344  8715 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.290700  5455 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 6 to 7, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 7 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:42.307428  8473 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.139259  8710 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.232653  8596 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:42.199707  8495 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.339337  6280 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 12:25:42.225152  8078 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260709 12:25:42.268734  5702 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:25:42.402398  6280 pending_rounds.cc:85] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 8
I20260709 12:25:42.404695  6280 pending_rounds.cc:107] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 12:25:42.405023  6280 raft_consensus.cc:2889] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:42.281064  8693 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:42.409746  8715 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.411857  8473 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:42.360824  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 161603us
I20260709 12:25:42.374565  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.412756  5706 raft_consensus.cc:2376] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8. Re-sending same reply.
I20260709 12:25:42.412806  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:42.413410  8618 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.413718  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.413969  5710 raft_consensus.cc:606] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:25:42.414259  8710 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.414374  5706 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 5.
I20260709 12:25:42.414788  8596 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.415333  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.416101  5706 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:25:42.396709  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.417742  5455 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 2 to 5, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 5 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:42.418642  8495 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.418812  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.377241  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.397425  5705 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:42.402582  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.397684  8568 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.420862  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.420778  8568 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:42.397924  5701 raft_consensus.cc:2410] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:42.406445  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.421960  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:42.408826  5708 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 4.
I20260709 12:25:42.422596  5556 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:42.419672  5455 catalog_manager.cc:5697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 3 to 4, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 4 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:42.419553  5706 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:42.423187  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:42.401875  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 142877us
I20260709 12:25:42.414378  5710 raft_consensus.cc:614] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:42.409576  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 653) took 1450 ms. Trace:
I20260709 12:25:42.417109  8596 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.424255  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.361987  8328 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:42.360800  8729 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:42.357308  8731 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.563340  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 618) took 2024 ms. Trace:
I20260709 12:25:42.518117  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1759) took 2793 ms. Trace:
W20260709 12:25:42.572757  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.592088  5554 rpcz_store.cc:276] 0709 12:25:39.724755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.727550 (+  2795us) service_pool.cc:224] Handling call
0709 12:25:39.727809 (+   259us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:40.247052 (+519243us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.517827 (+2270775us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b540044e654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:42.518059 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1185982848}
I20260709 12:25:42.593413  5556 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:42.593731  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:42.514343  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.491907  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 622) took 1535 ms. Trace:
I20260709 12:25:42.594681  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:42.518064  5634 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 3. Current term is 4. Ops: []
W20260709 12:25:42.589197  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 163512us
I20260709 12:25:42.583813  5702 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:42.481380  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.590727  6271 rpcz_store.cc:276] 0709 12:25:40.539295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.539456 (+   161us) service_pool.cc:224] Handling call
0709 12:25:40.565890 (+ 26434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:41.312108 (+746218us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.544120 (+1232012us) spinlock_profiling.cc:243] Waited 745 ms on lock 0x7b54004f9cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:42.563251 (+ 19131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1714174848}
I20260709 12:25:42.472800  5455 catalog_manager.cc:5697] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 1 to 3, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 3 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:25:42.549638  5675 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:42.518185  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 614) took 2051 ms. Trace:
I20260709 12:25:42.597779  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.596787  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:42.602099  5706 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:42.603096  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:42.594504  5554 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:42.475709  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:42.604154  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.605031  5702 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:42.506618  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:42.419602  8078 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:42.605803  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:25:42.537451  8706 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.534s	user 0.010s	sys 0.008s
W20260709 12:25:42.528092  8700 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.505s	user 0.010s	sys 0.002s
W20260709 12:25:42.522867  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 120410us
I20260709 12:25:42.607409  5705 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:25:42.624521  8666 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:25:42.598426  8192 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.625247  8668 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:42.600461  6280 rpcz_store.cc:276] 0709 12:25:40.958860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.301138 (+1342278us) service_pool.cc:224] Handling call
0709 12:25:42.338841 (+ 37703us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:42.409352 (+ 70511us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.409521 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:42.625710  8668 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:42.626475  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:42.626729  8668 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.601627  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 652) took 1643 ms. Trace:
I20260709 12:25:42.608345  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:42.608117  5524 leader_election.cc:336] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.673s (SENT)
I20260709 12:25:42.600958  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.627696  8586 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:25:42.598799  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49250.4us
I20260709 12:25:42.595436  6274 rpcz_store.cc:276] 0709 12:25:40.956473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.217550 (+1261077us) service_pool.cc:224] Handling call
0709 12:25:42.217931 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:42.218174 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:25:42.359280 (+141106us) log.cc:844] Serialized 14 byte log entry
0709 12:25:42.491457 (+132177us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:42.491543 (+    86us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:42.491602 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.491684 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.491831 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1278,"prepare.queue_time_us":181496,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":2179,"raft.queue_time_us":447,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":9251,"thread_start_us":266785,"threads_started":2}
I20260709 12:25:42.610608  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:42.628293  5701 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:42.629843  8340 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:42.630148  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.627799  8668 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:42.605903  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 99418.9us
I20260709 12:25:42.631222  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1922) took 3038 ms. Trace:
I20260709 12:25:42.631721  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:42.631623  5703 rpcz_store.cc:276] 0709 12:25:39.592440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.696671 (+104231us) service_pool.cc:224] Handling call
0709 12:25:39.704555 (+  7884us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:39.710098 (+  5543us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:39.710156 (+    58us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:39.710985 (+   829us) log.cc:844] Serialized 52 byte log entry
0709 12:25:40.131209 (+420224us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:40.192453 (+ 61244us) log.cc:844] Serialized 14 byte log entry
0709 12:25:40.224557 (+ 32104us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.224678 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:42.628901 (+2404223us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b18003809e0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:25:42.630889 (+  1988us) raft_consensus.cc:1685] finished
0709 12:25:42.630982 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.631142 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":64785,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":54,"raft.queue_time_us":75095,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":11257,"spinlock_wait_cycles":5175040,"thread_start_us":369703,"threads_started":2,"wal-append.queue_time_us":433272}
I20260709 12:25:42.633170  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 613) took 2166 ms. Trace:
I20260709 12:25:42.633265  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:42.612112  8546 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:42.611513  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:42.633659  6278 rpcz_store.cc:276] 0709 12:25:40.466760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.467061 (+   301us) service_pool.cc:224] Handling call
0709 12:25:40.493483 (+ 26422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:41.341463 (+847980us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.632939 (+1291476us) spinlock_profiling.cc:243] Waited 845 ms on lock 0x7b5400772754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:42.633087 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1942671104}
I20260709 12:25:42.472122  8371 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:42.628711  8586 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:42.635197  8738 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:42.635725  8738 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.636207  8738 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:42.636368  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.637390  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.637269  8626 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:25:42.634063  8546 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.638662  8546 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:42.638831  8705 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.434s	user 0.011s	sys 0.005s
I20260709 12:25:42.408174  8722 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.640682  5709 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:42.618578  8596 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:42.642441  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:42.627714  6275 rpcz_store.cc:276] 0709 12:25:40.958625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.258115 (+1299490us) service_pool.cc:224] Handling call
0709 12:25:42.258429 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:42.258643 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:25:42.387959 (+129316us) log.cc:844] Serialized 14 byte log entry
0709 12:25:42.601249 (+213290us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:42.601309 (+    60us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:42.601365 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.601449 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.601569 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":162170,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":1425,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":12928,"thread_start_us":330722,"threads_started":2,"wal-append.queue_time_us":245755}
I20260709 12:25:42.613391  8731 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.476627  8692 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:42.481437  8568 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.617880  5706 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:25:42.619685  8495 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:42.620045  8740 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260709 12:25:42.423520  8347 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:25:42.375913  8734 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:42.464649  8591 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.610729  8078 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:42.625303  8192 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.607138  5455 catalog_manager.cc:5697] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 2 to 4, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 4 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:42.627247  8694 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:42.630280  8340 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.604974  8431 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:42.635299  8586 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.631237  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.565312  8230 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.643560  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:42.644618  5703 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:42.537734  8462 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:42.698074  5701 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:42.599404  6276 rpcz_store.cc:276] 0709 12:25:40.467028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.467134 (+   106us) service_pool.cc:224] Handling call
0709 12:25:40.477550 (+ 10416us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:41.743989 (+1266439us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:41.744089 (+   100us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:41.785954 (+ 41865us) log.cc:844] Serialized 26 byte log entry
0709 12:25:42.263202 (+477248us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:42.263260 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.417261 (+154001us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b5400591ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:42.417306 (+    45us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:42.487046 (+ 69740us) raft_consensus.cc:1685] finished
0709 12:25:42.487076 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.518097 (+ 31021us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":256764,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"spinlock_wait_cycles":1977266944,"thread_start_us":849127,"threads_started":2,"wal-append.queue_time_us":558882}
I20260709 12:25:42.644981  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:42.407138  8726 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:42.645648  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:42.616808  8729 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.615155  8328 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.549913  8456 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
I20260709 12:25:42.648731  8710 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:42.646601  8715 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:42.634763  8371 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.640909  8722 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.597265  5556 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:42.650828  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:42.730227  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 79486.6us
I20260709 12:25:42.730304  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:42.658650  5702 raft_consensus.cc:2410] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:42.662271  5709 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:25:42.362764  8720 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:42.732652  8722 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:25:42.733722  8722 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:25:42.734265  8722 consensus_queue.cc:1243] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:25:42.734537  5559 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:25:42.735616  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:42.409790  8687 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.699052  8586 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:25:42.661377  8429 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.516s	user 0.011s	sys 0.000s
I20260709 12:25:42.465303  8473 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:42.695813  8431 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:42.701921  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57051.8us
I20260709 12:25:42.745040  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:42.660902  8495 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.665450  8596 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:42.702422  5554 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:42.749675  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:42.749845  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.750572  6278 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:42.750849  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:42.751600  5704 raft_consensus.cc:2410] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:42.751789  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:42.712224  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 68720us
I20260709 12:25:42.753269  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:42.661203  8454 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:42.760113  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:42.735635  8695 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:42.719148  8726 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.720196  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:42.772138  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.772663  5705 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:25:42.729344  8371 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:42.737237  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:42.789316  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52176.1us
I20260709 12:25:42.789599  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.738889  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.791380  6274 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:42.696585  8340 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:42.694437  8591 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.758139  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.737541  8328 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:42.660984  8693 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:42.759454  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.764333  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:42.802678  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.772770  5560 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:42.741196  8642 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:42.807420  5701 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:42.808418  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.809080  5701 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:42.810099  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.810901  5701 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:42.802948  6272 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:42.757673  8380 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:42.666499  5706 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:25:42.809911  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:42.765080  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.815716  5560 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:42.760862  5559 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:42.759876  8592 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:42.750167  8456 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.775354  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.775480  8720 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.763996  8666 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:42.821671  8666 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:42.821823  8748 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:25:42.822144  8666 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:42.822710  8530 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:42.823482  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:42.824442  8592 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:42.824870  8592 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:42.749437  8495 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:42.825906  8456 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:42.777954  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:42.827065  5557 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:42.779656  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.828117  5709 raft_consensus.cc:2376] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7. Re-sending same reply.
I20260709 12:25:42.828445  5705 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:42.779004  8685 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.783505  8734 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.751124  8473 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:42.764407  8695 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:42.764817  8725 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:42.830315  8725 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:42.830694  8725 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:42.749538  8431 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:42.722903  8689 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:42.738809  6275 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:42.833171  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.834368  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.714437  8727 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:42.798036  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:42.799253  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.793511  5455 catalog_manager.cc:5697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 3 to 4, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 4 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:42.835228  8727 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:42.836628  8720 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:42.836194  8727 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.836139  5455 catalog_manager.cc:5697] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 1 to 3, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 3 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:25:42.837023  8720 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.837409  8720 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:42.838102  8542 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:25:42.838465  5455 catalog_manager.cc:5697] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 2 to 5, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 5 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:42.839624  8542 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:25:42.772903  6276 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:42.840497  8734 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:25:42.841117  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.842394  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.842756  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:42.765759  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:42.845621  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.846468  6271 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 7: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:42.847354  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.843770  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.848496  6276 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:42.848610  5601 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:42.849817  8230 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:42.840145  5455 catalog_manager.cc:5697] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 4 to 6, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 6 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:42.850186  8230 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.850564  8230 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:42.851027  5455 catalog_manager.cc:5697] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 1 to 3, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 3 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:42.851709  8495 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.816565  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:42.818347  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:42.822085  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.781018  8546 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.828637  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.761261  8078 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.814429  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1757) took 3036 ms. Trace:
I20260709 12:25:42.797189  8687 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 12:25:42.798164  8692 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:42.857749  8711 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:42.807117  8328 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.863376  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.843847  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:42.851943  8738 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:42.872565  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.795701  8726 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.852564  5455 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:42.918480  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.919327  5703 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:42.919662  5455 catalog_manager.cc:5697] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:42.872989  8498 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:42.874523  8554 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.631s	user 0.007s	sys 0.004s
I20260709 12:25:42.894348  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:42.941109  5560 raft_consensus.cc:2410] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:25:42.897907  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:42.905318  5702 raft_consensus.cc:2410] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:42.896247  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.757854  8651 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.105s
I20260709 12:25:42.911657  5559 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:42.864948  6275 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:25:42.913096  5708 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:42.764955  8696 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.109s
I20260709 12:25:42.907065  5553 rpcz_store.cc:276] 0709 12:25:39.778083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.329086 (+551003us) service_pool.cc:224] Handling call
0709 12:25:40.329503 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.329750 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:25:40.351653 (+ 21903us) log.cc:844] Serialized 14 byte log entry
0709 12:25:42.536910 (+2185257us) spinlock_profiling.cc:243] Waited 6.03 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:42.640942 (+104032us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b180003f2e0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:25:42.803639 (+162697us) log.cc:844] Serialized 14 byte log entry
0709 12:25:42.812835 (+  9196us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:42.814088 (+  1253us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:42.814146 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.814238 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.814368 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9002,"prepare.queue_time_us":1286502,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":108,"raft.queue_time_us":10332,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":232,"spinlock_wait_cycles":19185664,"thread_start_us":309406,"threads_started":2,"wal-append.queue_time_us":292612}
I20260709 12:25:42.911056  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:42.815151  8380 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:42.908412  8668 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:42.927851  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.929365  8542 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:25:42.930558  8371 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.930958  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1698) took 2385 ms. Trace:
I20260709 12:25:42.957214  8668 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.967463  5634 rpcz_store.cc:276] 0709 12:25:40.545415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:41.274789 (+729374us) service_pool.cc:224] Handling call
0709 12:25:41.275224 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:42.596118 (+1320894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.930688 (+334570us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b5400242054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:42.930891 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2857838976}
W20260709 12:25:42.931700  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1929) took 2861 ms (client timeout 2236 ms). Trace:
W20260709 12:25:42.968756  5707 rpcz_store.cc:269] 0709 12:25:40.070603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.072552 (+  1949us) service_pool.cc:224] Handling call
0709 12:25:42.931516 (+2858964us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b5400451fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:42.931622 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4955134848}
I20260709 12:25:42.969909  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.970187  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:42.970463  5707 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:42.971319  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:42.972050  5707 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:25:42.933270  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 630) took 2397 ms. Trace:
I20260709 12:25:42.916484  8328 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:42.972483  6273 rpcz_store.cc:276] 0709 12:25:40.535858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.536074 (+   216us) service_pool.cc:224] Handling call
0709 12:25:40.549716 (+ 13642us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:42.046576 (+1496860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.932232 (+885656us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b54005eef54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:42.933222 (+   990us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3440342912}
I20260709 12:25:42.973795  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:42.910588  8711 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:42.975538  8711 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.911722  8692 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:42.976619  8711 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:42.976694  8692 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:42.977301  8546 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.944675  6272 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:42.981271  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:42.937125  8595 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.020s
I20260709 12:25:42.981297  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.982412  8750 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.982885  8750 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:42.983767  5707 raft_consensus.cc:2410] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:25:42.932964  8498 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:42.957224  8380 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:42.987030  8498 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:42.985126  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.958020  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:42.987782  5560 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:42.988672  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.958159  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.990172  5708 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: [4.4-8.6]
I20260709 12:25:42.873481  8596 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:42.959720  8693 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:42.964969  8666 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:42.944998  8685 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:42.967974  8668 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:42.939636  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:43.014485  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 74928us
W20260709 12:25:42.947202  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 49386.2us
W20260709 12:25:42.874037  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55817.6us
I20260709 12:25:42.985069  8750 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:43.055428  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:42.977547  8738 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:42.962561  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:43.058396  5702 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 5.
I20260709 12:25:43.059386  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:43.059703  5702 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:25:43.061026  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:42.992837  5703 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:42.993767  5707 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:25:42.924562  8726 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:43.005203  5633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2005) took 3185 ms (client timeout 1706 ms). Trace:
W20260709 12:25:43.062942  5633 rpcz_store.cc:269] 0709 12:25:39.805811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.806292 (+   481us) service_pool.cc:224] Handling call
0709 12:25:42.983125 (+3176833us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x7b54001cedd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:42.983239 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":83468032}
I20260709 12:25:43.063342  5702 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:25:42.990219  8684 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:42.941949  8568 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
W20260709 12:25:43.021077  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:43.041612  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.052623  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:43.054971  5708 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260709 12:25:42.990358  6273 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:42.994138  8696 consensus_queue.cc:1243] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:25:43.065331  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:43.066635  6271 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:43.005883  8697 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.006183  8693 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.067775  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:43.070976  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:43.069849  5708 pending_rounds.cc:85] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 2
I20260709 12:25:43.076881  5708 pending_rounds.cc:107] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:25:43.077188  5708 raft_consensus.cc:2889] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:43.077338  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:43.078241  5557 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:43.070629  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:43.067757  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:43.074868  5702 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:43.075291  8542 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:43.085007  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:43.085392  8734 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:43.121258  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:43.068866  8673 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:43.086899  8192 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:43.084064  8495 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:43.092423  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:43.148221  5560 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:43.093106  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1762) took 2952 ms. Trace:
I20260709 12:25:43.149442  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:43.149412  5561 rpcz_store.cc:276] 0709 12:25:40.140929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.383314 (+242385us) service_pool.cc:224] Handling call
0709 12:25:40.383696 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:42.232439 (+1848743us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.092800 (+860361us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b540044f054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:43.093036 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4112466688}
I20260709 12:25:43.151422  5561 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:43.095618  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:43.095813  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:43.096235  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.098909  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:43.153477  6277 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:43.057184  8729 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:43.097908  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:43.065878  8684 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:43.068718  8651 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.054s
I20260709 12:25:43.055042  8685 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:43.120554  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:43276 (request call id 9) took 1056 ms. Trace:
W20260709 12:25:43.132906  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:43.182775  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.086004  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:43.183463  5702 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:25:43.183629  6279 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:43.132486  5553 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
W20260709 12:25:43.133869  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63341us
I20260709 12:25:43.133919  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:43.135092  5703 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20260709 12:25:43.255048  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:43.254993  5709 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:43.080578  8693 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:43.135921  8734 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:43.166414  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:43.166654  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.402262  5552 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:43.402360  5627 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:43.250273  8192 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:25:43.403596  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.093847  8595 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:43.212796  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.404759  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.405323  5636 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:43.406143  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:43.406628  5636 raft_consensus.cc:2376] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7. Re-sending same reply.
I20260709 12:25:43.407090  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.407485  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.222527  5628 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260709 12:25:43.408313  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.408704  8371 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:43.408895  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.409418  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.409960  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.410579  5628 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:43.074692  8431 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.182402  5710 rpcz_store.cc:276] 0709 12:25:42.042941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.043100 (+   159us) service_pool.cc:224] Handling call
0709 12:25:43.098959 (+1055859us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b54002f5b54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa66c853 00007f4fff63a60a 00007f4ff8bc07d4 00007f4ff8bc0762 00007f4ff8bc0691 00007f4ff8bc060c 00007f4ff8bbf8c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:43.099073 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":850882816}
I20260709 12:25:43.096022  8347 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:43.094317  8730 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:43.155254  8692 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.115371  8448 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:43.308631  8592 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.140780  8673 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.423686  8673 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:43.423875  8592 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:43.424325  6272 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:43.415717  8347 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:43.425422  8347 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.426187  8347 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:43.099489  8697 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.418151  8730 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:43.218711  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.253082  5635 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:43.252406  5632 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:25:43.222833  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.255403  5703 pending_rounds.cc:85] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 2
I20260709 12:25:43.429417  8730 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.430095  5703 pending_rounds.cc:107] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 12:25:43.156711  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.430397  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:43.158638  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.431146  5709 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:43.123896  8340 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:43.259719  5599 connection.cc:256] RPC call timeout handler was delayed by -1.8239258339999651s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
I20260709 12:25:43.262780  6277 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:43.432430  8695 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:43.433166  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:43.433441  8340 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:43.433550  8695 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:43.140056  8586 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.434428  6274 raft_consensus.cc:2368] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10.
I20260709 12:25:43.435568  8586 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:43.389081  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 291236us
I20260709 12:25:43.435932  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:43.436597  6274 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:43.436838  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:43.438594  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:43.439597  6278 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: []
W20260709 12:25:43.439862  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.440639  6280 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:43.440737  8699 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.441301  8699 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.442420  6280 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:25:43.443732  8699 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:43.423875  8692 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:43.444483  8565 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 12:25:43.432125  5599 connection.cc:256] RPC call timeout handler was delayed by -1.8088160980000794s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1524s for a response
W20260709 12:25:43.446405  5599 connection.cc:256] RPC call timeout handler was delayed by -1.6340258340001128s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18330000000000002s for a response
W20260709 12:25:43.446763  5599 connection.cc:256] RPC call timeout handler was delayed by -1.4486258340002678s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20390000000000003s for a response
W20260709 12:25:43.447067  5599 connection.cc:256] RPC call timeout handler was delayed by -1.3087083990003521s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
W20260709 12:25:43.447379  5599 connection.cc:256] RPC call timeout handler was delayed by -1.1624784290002026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15910000000000002s for a response
I20260709 12:25:43.430531  5703 raft_consensus.cc:2889] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:43.447718  5599 connection.cc:256] RPC call timeout handler was delayed by -1.00776752100046s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1738s for a response
I20260709 12:25:43.118820  8230 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.450358  8230 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:43.429435  5630 raft_consensus.cc:2410] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:25:43.410219  8456 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:43.455468  8685 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:43.458051  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:43.440109  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:43.300400  8498 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:43.255002  5708 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:43.396200  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:43.129189  8666 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:43.404254  8595 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:43.469795  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:43.469854  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:43.080770  8720 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.430011  5629 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:25:43.147046  8542 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:43.470845  8595 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.471874  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:43.472478  8720 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:43.467555  8586 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:43.430526  8730 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:43.433849  6277 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:43.472110  8595 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:43.475118  8380 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:43.477203  8761 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.255s	user 0.000s	sys 0.001s
I20260709 12:25:43.471906  5557 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:43.477738  8431 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:43.479712  8328 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.507s	user 0.006s	sys 0.005s
I20260709 12:25:43.479863  8230 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:43.480427  8230 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:43.480162  8328 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:43.481529  5556 raft_consensus.cc:2368] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8.
I20260709 12:25:43.481362  8230 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.484607  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:43.487830  6279 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:43.490888  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:43.395459  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.464776  5636 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:43.465792  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:43.466265  5560 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:43.494088  5635 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:43.494513  5558 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:43.495241  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 504) took 1064 ms. Trace:
I20260709 12:25:43.495471  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:43.495615  5560 rpcz_store.cc:276] 0709 12:25:42.431036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.092063 (+661027us) service_pool.cc:224] Handling call
0709 12:25:43.495183 (+403120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:43.496055  6279 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:43.496856  5560 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 5. Current term is 7. Ops: []
I20260709 12:25:43.148619  8568 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:43.489634  8498 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:43.504987  8568 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.505429  8568 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:43.505861  8498 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.313160  8726 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:43.490356  8506 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
W20260709 12:25:43.499239  8711 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.522s	user 0.008s	sys 0.002s
I20260709 12:25:43.452095  8693 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.300478  8668 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.483016  8592 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.509068  8592 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.509936  8693 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:43.510403  8668 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:43.511116  8592 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:43.398551  8596 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.099s
I20260709 12:25:43.514863  8697 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:43.515079  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.541405  8568 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:43.517083  8734 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.543171  6272 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:25:43.543322  5705 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:25:43.518826  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.545257  5634 raft_consensus.cc:2410] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:25:43.545766  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:43.546145  6272 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:43.525112  8530 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:43.547257  8530 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:43.547482  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:43.520478  8725 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.689s	user 0.012s	sys 0.000s
I20260709 12:25:43.527186  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:43.548731  5559 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:25:43.548636  8725 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.549952  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:43.527040  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:43.528681  6276 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:25:43.529803  8328 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:43.530305  6279 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:43.564918  6275 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
W20260709 12:25:43.531277  8763 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:25:43.489682  8546 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 12:25:43.313716  8684 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:43.537038  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.537889  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 510) took 1055 ms. Trace:
I20260709 12:25:43.490207  8192 raft_consensus.cc:2764] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:25:43.516135  8503 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:43.545141  8734 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:43.546499  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1773) took 1127 ms. Trace:
I20260709 12:25:43.546782  8592 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:43.550828  6271 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:43.520722  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:25:43.583359  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62716.1us
I20260709 12:25:43.528297  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1772) took 1109 ms. Trace:
I20260709 12:25:43.583442  5557 rpcz_store.cc:276] 0709 12:25:42.482872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.399489 (+916617us) service_pool.cc:224] Handling call
0709 12:25:43.416959 (+ 17470us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:43.477792 (+ 60833us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.537644 (+ 59852us) spinlock_profiling.cc:243] Waited 54.8 ms on lock 0x7b5400082854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:43.537831 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":125964672}
I20260709 12:25:43.551914  6278 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:43.553777  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.585794  6277 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:43.555509  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:43.554128  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 63339.9us
I20260709 12:25:43.589298  5601 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:43.556037  6273 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:43.560873  5635 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:25:43.564385  5526 leader_election.cc:400] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10.
I20260709 12:25:43.570358  5559 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:25:43.579667  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:43.591253  5526 leader_election.cc:403] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:43.579418  5558 rpcz_store.cc:276] 0709 12:25:42.419151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.134917 (+715766us) service_pool.cc:224] Handling call
0709 12:25:43.494299 (+359382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:43.494927 (+   628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.546321 (+ 51394us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b54001c5cd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:43.546442 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":714912512}
I20260709 12:25:43.581029  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:43.585897  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.586668  5629 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:43.555825  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.594975  6236 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:43.598922  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:43.599783  5557 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:43.605988  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:43.582594  8592 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.576655  8730 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:43.577248  8626 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:25:43.654366  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 48481.7us
I20260709 12:25:43.655332  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:43.609877  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:43.584117  5555 rpcz_store.cc:276] 0709 12:25:42.418872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.423687 (+  4815us) service_pool.cc:224] Handling call
0709 12:25:42.424016 (+   329us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:42.717941 (+293925us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:42.717976 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:42.718751 (+   775us) log.cc:844] Serialized 52 byte log entry
0709 12:25:43.001488 (+282737us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:43.001531 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.527888 (+526357us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b540023f354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:43.527937 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:43.527987 (+    50us) raft_consensus.cc:1685] finished
0709 12:25:43.527999 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:43.528236 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1475,"raft.queue_time_us":1604,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":666876800,"thread_start_us":274475,"threads_started":1,"wal-append.queue_time_us":532121}
W20260709 12:25:43.657006  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47239.1us
I20260709 12:25:43.657644  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:43.655695  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.658182  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:43.659260  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:43.579802  8596 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:43.705272  5555 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:43.662839  5629 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
W20260709 12:25:43.682839  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:43.673043  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.587545  8692 raft_consensus.cc:2764] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20260709 12:25:43.728085  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51934.5us
I20260709 12:25:43.729092  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:43.730496  6274 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:43.731477  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:43.732532  5558 raft_consensus.cc:2410] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:25:43.596611  8726 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10.
I20260709 12:25:43.608044  8347 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:43.601460  8503 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:43.739143  5557 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:43.727375  8693 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:43.753314  8726 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:25:43.653209  8592 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:43.740891  8542 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:25:43.744148  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:43.609158  8328 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.136s
I20260709 12:25:43.661263  8697 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:25:43.744908  8711 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:43.733935  8720 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:43.745548  8338 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:43.660960  8348 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:43.652783  8733 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.095s
I20260709 12:25:43.764636  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:43.765157  8685 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:43.727662  8725 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:43.708333  8730 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:43.734025  8724 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:43.607851  8431 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.137s
I20260709 12:25:43.706019  8596 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.753607  8693 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:43.786518  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 251619us
I20260709 12:25:43.763651  8595 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:43.785629  8720 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:43.789554  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 263551us
W20260709 12:25:43.791234  5554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1775) took 1371 ms (client timeout 1625 ms). Trace:
W20260709 12:25:43.811908  5554 rpcz_store.cc:269] 0709 12:25:42.419651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.604660 (+185009us) service_pool.cc:224] Handling call
0709 12:25:43.791024 (+1186364us) spinlock_profiling.cc:243] Waited 74.4 ms on lock 0x7b54003a98d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:43.791164 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":171206528}
I20260709 12:25:43.801712  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:43.813448  5706 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:43.802975  5701 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: []
W20260709 12:25:43.815212  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.821152  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:43.822032  5706 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:43.787855  8697 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:43.788833  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:43.809633  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:43.788606  8685 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.823098  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:43.831666  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:43.793004  8724 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:25:43.805480  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.806073  8328 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:43.837798  5709 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:43.802982  8596 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:43.842618  5708 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:43.837751  6274 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:43.868948  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:43.816884  8720 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:43.787487  8711 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.869784  5629 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:43.881605  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.882123  5627 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:25:43.881999  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.882937  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.883111  5629 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:43.884267  8568 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:43.884902  8568 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:43.885062  8554 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:43.885284  8568 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:43.885596  8554 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:43.886314  8568 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.886690  8554 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:43.887764  8568 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:43.867619  8456 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.870949  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.859179  8685 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:43.859171  8448 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:43.871820  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:43.895457  6272 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 7: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:43.871793  8730 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.900710  8711 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:43.900628  8456 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.889068  8380 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:43.901497  5629 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:43.902673  8456 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:43.865156  8699 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.903420  8699 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:43.904103  6272 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:43.904775  5629 pending_rounds.cc:85] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 3
I20260709 12:25:43.905086  5629 pending_rounds.cc:107] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:25:43.905306  5629 raft_consensus.cc:2889] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:43.905450  8699 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:43.870399  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:43.884117  5627 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:43.861266  8722 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.865883  8724 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:43.898479  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.899573  8448 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.878893  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.917622  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:43.926175  5630 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:25:43.927691  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:43.928118  5708 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
W20260709 12:25:43.892056  8759 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:25:43.982151  5635 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:25:43.983843  5627 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:43.988066  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:43.991786  6274 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:43.911532  8695 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:43.951079  6271 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
W20260709 12:25:44.011677  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 351866us
I20260709 12:25:43.991912  8725 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.908320  8380 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:43.908577  8751 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.995230  8695 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.992281  8626 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:44.028087  5702 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.008s	sys 0.025s
I20260709 12:25:44.009392  8767 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.992619  8697 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:43.978899  8722 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:43.939018  8699 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
W20260709 12:25:44.032995  5702 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.008s	sys 0.025s
I20260709 12:25:43.996397  8347 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:44.034490  5556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1772) took 1404 ms (client timeout 1663 ms). Trace:
W20260709 12:25:44.347842  5556 rpcz_store.cc:269] 0709 12:25:42.630560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.730031 (+ 99471us) service_pool.cc:224] Handling call
0709 12:25:44.034294 (+1304263us) spinlock_profiling.cc:243] Waited 750 ms on lock 0x7b54001bec54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:44.034430 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1725894400}
W20260709 12:25:44.128690  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:44.179651  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:44.246922  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.354805  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:44.291098  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.025660  8769 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:44.182511  8230 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:44.033958  8695 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:44.365839  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:44.362861  8592 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.581s	user 0.009s	sys 0.001s
I20260709 12:25:44.191855  8456 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.029526  8725 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:44.421988  5629 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.028s	sys 0.036s
W20260709 12:25:44.423043  5629 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.028s	sys 0.036s
I20260709 12:25:44.358624  8711 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.407572  8769 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:44.424275  8711 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:44.426358  5703 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.033s	sys 0.030s
W20260709 12:25:44.427078  5703 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.034s	sys 0.030s
I20260709 12:25:44.191100  8380 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:44.415359  8730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.015s	sys 0.046s
W20260709 12:25:44.434116  5674 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.000s	sys 0.046s
W20260709 12:25:44.453703  5674 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.000s	sys 0.047s
W20260709 12:25:44.454241  5552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.017s	sys 0.038s
W20260709 12:25:44.454576  5552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.017s	sys 0.038s
I20260709 12:25:44.455601  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:44.436784  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.404601  8230 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:44.418156  8546 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.009s	sys 0.027s
I20260709 12:25:44.191277  8751 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:44.450263  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.015s	sys 0.005s
W20260709 12:25:44.465351  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.016s	sys 0.005s
I20260709 12:25:44.466331  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:44.468390  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 628) took 1713 ms. Trace:
I20260709 12:25:44.468941  6276 rpcz_store.cc:276] 0709 12:25:42.755412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.849188 (+ 93776us) service_pool.cc:224] Handling call
0709 12:25:43.528434 (+679246us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:43.577471 (+ 49037us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.468108 (+890637us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b54006121d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:44.468295 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1561357184}
I20260709 12:25:44.472723  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:44.475307  5706 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:25:44.477010  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:44.359901  8733 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.009s	sys 0.018s
I20260709 12:25:44.411759  8699 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:44.459231  5710 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:25:44.250296  8767 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:44.505882  8767 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:44.481405  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:44.482203  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:44.503296  8699 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:44.506832  5701 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
W20260709 12:25:44.465832  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 540680us
I20260709 12:25:44.419687  8592 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:44.510615  8592 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:44.483412  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.000s	sys 0.023s
W20260709 12:25:44.457964  8673 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.034s	user 0.008s	sys 0.005s
W20260709 12:25:44.511391  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.000s	sys 0.024s
I20260709 12:25:44.511549  8673 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:44.513247  5707 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:44.514431  8673 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:44.515244  8673 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:44.461732  8546 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.009s	sys 0.027s
W20260709 12:25:44.459882  8498 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.019s	sys 0.041s
W20260709 12:25:44.517632  8498 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.020s	sys 0.041s
I20260709 12:25:44.452332  8591 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:44.462352  8503 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.708s	user 0.077s	sys 0.129s
W20260709 12:25:44.487838  5676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.022s	sys 0.033s
I20260709 12:25:44.515764  8673 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:44.518330  5553 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: []
W20260709 12:25:44.519093  5676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.022s	sys 0.033s
W20260709 12:25:44.519589  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 696569us
W20260709 12:25:44.518553  8503 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.765s	user 0.078s	sys 0.130s
I20260709 12:25:44.520964  8503 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:44.521873  8673 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:44.522328  8503 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:44.525338  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:44.551774  5709 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.687s	user 0.008s	sys 0.004s
I20260709 12:25:44.532403  5602 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:44.534895  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:44.562760  5709 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
W20260709 12:25:44.547904  5675 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.776s (SENT) [suppressed 26 similar messages]
I20260709 12:25:44.565551  6274 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:44.458106  8693 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:44.567132  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:44.578081  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:44.427592  8456 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:44.502453  8733 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.010s	sys 0.018s
W20260709 12:25:44.487107  8730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.015s	sys 0.046s
I20260709 12:25:44.603758  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:44.604010  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:44.604363  5559 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:44.586398  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:44.605504  5709 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
W20260709 12:25:44.588472  5526 leader_election.cc:336] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.723s (SENT)
I20260709 12:25:44.591256  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:44.591853  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:44.593250  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:44.596019  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:44.598591  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 66294.8us
I20260709 12:25:44.608564  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:44.598958  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:44.610762  5704 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:44.456835  8760 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20260709 12:25:44.500464  8782 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.119s
I20260709 12:25:44.617554  6272 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:44.529430  8473 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:44.457222  8783 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.149s
W20260709 12:25:44.612483  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.009s	sys 0.053s
W20260709 12:25:44.641780  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.009s	sys 0.053s
W20260709 12:25:44.642133  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 621322us
I20260709 12:25:44.533849  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:44.644088  6273 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:44.603323  8787 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.585544  8498 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.460716  8347 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:44.612721  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 87497.7us
I20260709 12:25:44.669749  5676 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:44.670953  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:44.486230  8769 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:44.566450  8592 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:25:44.629122  8766 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.194s	user 0.000s	sys 0.000s
I20260709 12:25:44.629793  8785 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:25:44.634126  8768 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.389s	user 0.001s	sys 0.000s
I20260709 12:25:44.616744  8673 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:44.635056  8230 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:44.637346  8782 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:25:44.640529  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47366.9us
I20260709 12:25:44.704895  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:44.646596  5559 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:44.437211  8626 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:44.650012  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 631) took 1670 ms. Trace:
I20260709 12:25:44.716670  6278 rpcz_store.cc:276] 0709 12:25:42.979444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.454006 (+474562us) service_pool.cc:224] Handling call
0709 12:25:43.551682 (+ 97676us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:43.586756 (+ 35074us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.649758 (+1063002us) spinlock_profiling.cc:243] Waited 97.3 ms on lock 0x7b5400055850. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:44.649937 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":223766912}
I20260709 12:25:44.618700  5556 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:25:44.639088  8773 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.628s	user 0.001s	sys 0.000s
W20260709 12:25:44.686070  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 331326us
I20260709 12:25:44.722272  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:44.708202  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:44.586071  8778 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.115s
I20260709 12:25:44.717901  5556 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:44.614677  8729 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.072s
I20260709 12:25:44.742205  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 499) took 2321 ms. Trace:
I20260709 12:25:44.743503  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 554) took 1276 ms. Trace:
I20260709 12:25:44.743839  5628 rpcz_store.cc:276] 0709 12:25:43.466794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.466930 (+   136us) service_pool.cc:224] Handling call
0709 12:25:43.480406 (+ 13476us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:43.481057 (+   651us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:43.481087 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:43.481586 (+   499us) log.cc:844] Serialized 26 byte log entry
0709 12:25:43.811293 (+329707us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:43.811331 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.811395 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:44.743353 (+931958us) raft_consensus.cc:1685] finished
0709 12:25:44.743366 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:44.743459 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5726,"raft.queue_time_us":1363,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"thread_start_us":329365,"threads_started":1,"wal-append.queue_time_us":880208}
I20260709 12:25:44.744730  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1780) took 2147 ms. Trace:
I20260709 12:25:44.744876  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:44.745246  5552 rpcz_store.cc:276] 0709 12:25:42.597148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.403009 (+805861us) service_pool.cc:224] Handling call
0709 12:25:43.403368 (+   359us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:43.792488 (+389120us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:43.792558 (+    70us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:43.798165 (+  5607us) log.cc:844] Serialized 52 byte log entry
0709 12:25:44.454949 (+656784us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:44.454991 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.455057 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:44.744465 (+289408us) raft_consensus.cc:1685] finished
0709 12:25:44.744493 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:44.744662 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6580,"raft.queue_time_us":45717,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":101,"thread_start_us":951436,"threads_started":2,"wal-append.queue_time_us":848434}
I20260709 12:25:44.729404  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:44.730772  8791 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:25:44.731719  8473 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:25:44.731841  8730 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:25:44.732023  8784 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
W20260709 12:25:44.778072  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 107215us
I20260709 12:25:44.733338  8778 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:44.735564  8078 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:44.739849  5556 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:44.675400  8498 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:44.748423  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:44.703866  8506 raft_consensus.cc:2764] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:25:44.742573  5561 rpcz_store.cc:276] 0709 12:25:42.427987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.166291 (+738304us) service_pool.cc:224] Handling call
0709 12:25:43.397433 (+231142us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:43.398112 (+   679us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:43.398142 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:43.418133 (+ 19991us) log.cc:844] Serialized 26 byte log entry
0709 12:25:43.804739 (+386606us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:43.804781 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.804843 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:44.741977 (+937134us) raft_consensus.cc:1685] finished
0709 12:25:44.741999 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:44.742130 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":113,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"spinlock_wait_cycles":120832,"thread_start_us":382774,"threads_started":1,"wal-append.queue_time_us":577674}
I20260709 12:25:44.720378  8782 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:44.776393  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.814146  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 514) took 1752 ms. Trace:
I20260709 12:25:44.779219  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1922) took 1366 ms. Trace:
I20260709 12:25:44.782124  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:25:44.787467  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.733217  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:44.799163  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 189028us
I20260709 12:25:44.723083  8769 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:25:44.747880  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 104271us
I20260709 12:25:44.722102  8780 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
W20260709 12:25:44.803815  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 236789us
I20260709 12:25:44.748121  8722 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:44.806504  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:44.809343  5556 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:44.757488  8230 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:44.769999  8595 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:44.704660  8673 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:44.830291  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:44.840595  5635 raft_consensus.cc:2368] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7.
I20260709 12:25:44.841925  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:44.842506  5627 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:25:44.721943  8779 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.080s
W20260709 12:25:44.813323  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 65019.8us
I20260709 12:25:44.666688  8787 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:44.775725  8711 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:44.851739  5634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:25:44.847693  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:44.852689  5634 raft_consensus.cc:606] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:25:44.852833  5635 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:44.852917  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:44.823275  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:44.834723  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.854019  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:44.828130  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:44.835597  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:44.776496  8800 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:44.814644  5560 rpcz_store.cc:276] 0709 12:25:43.062042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.520559 (+458517us) service_pool.cc:224] Handling call
0709 12:25:43.520881 (+   322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:43.521213 (+   332us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:43.521240 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:43.521691 (+   451us) log.cc:844] Serialized 26 byte log entry
0709 12:25:43.788851 (+267160us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:43.788897 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.789063 (+   166us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:44.813888 (+1024825us) raft_consensus.cc:1685] finished
0709 12:25:44.813916 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:44.814077 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1023,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":83,"thread_start_us":266768,"threads_started":1,"wal-append.queue_time_us":464818}
I20260709 12:25:44.824047  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:44.846552  5526 leader_election.cc:400] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7.
I20260709 12:25:44.856549  5526 leader_election.cc:403] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260709 12:25:44.818236  8720 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.934s	user 0.009s	sys 0.004s
I20260709 12:25:44.854992  5628 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:44.814870  5705 rpcz_store.cc:276] 0709 12:25:43.412651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.413263 (+   612us) service_pool.cc:224] Handling call
0709 12:25:43.413554 (+   291us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:43.543865 (+130311us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.778955 (+1235090us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b5400197dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:44.779159 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":297743104}
W20260709 12:25:44.856707  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.856472  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:44.864208  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:44.865280  5635 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:44.809391  8730 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:44.866343  8722 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:44.867141  8711 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:44.866725  8730 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:44.867471  8722 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:25:44.819875  8782 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:44.868418  8722 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:44.835676  8380 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:44.812656  8348 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.016s	user 0.012s	sys 0.000s
I20260709 12:25:44.872717  8348 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:25:44.874745  8711 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:44.876163  8788 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:25:44.864347  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:44.848402  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:44.879433  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:44.880142  8798 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.896896  8448 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:44.897576  8448 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:44.851171  8694 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:44.902268  8448 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:44.903509  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:44.827469  8530 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.280s	user 0.006s	sys 0.006s
I20260709 12:25:44.904628  8530 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:44.889974  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 66778.3us
I20260709 12:25:44.924806  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:44.891206  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44786.7us
I20260709 12:25:44.815685  8725 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:44.832651  8473 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:25:44.900097  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.897289  8798 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:44.902329  8694 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:44.882362  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:44.881038  5630 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:44.925978  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:44.880816  8713 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:44.959664  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:44.855455  8783 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:44.929729  6234 connection.cc:256] RPC call timeout handler was delayed by -1.178932652000185s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1804s for a response
W20260709 12:25:44.964551  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:44.965049  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:44.965476  6234 connection.cc:256] RPC call timeout handler was delayed by -1.019162917000358s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1875s for a response
I20260709 12:25:44.967453  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:44.968331  6277 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:44.931321  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:44.969827  5628 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:44.928247  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 501) took 2498 ms. Trace:
I20260709 12:25:44.970849  5553 rpcz_store.cc:276] 0709 12:25:42.430174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.255935 (+825761us) service_pool.cc:224] Handling call
0709 12:25:43.540651 (+284716us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:25:43.540856 (+   205us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:44.519341 (+978485us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.927938 (+408597us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x7b54000ebad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:44.928158 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2251532032}
W20260709 12:25:44.937005  5636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 548) took 1619 ms (client timeout 2023 ms). Trace:
I20260709 12:25:44.861148  8769 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:44.860903  8800 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.938354  5556 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 12:25:44.979656  5556 pending_rounds.cc:85] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 5
I20260709 12:25:44.980054  5556 pending_rounds.cc:107] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 12:25:44.980428  5556 raft_consensus.cc:2889] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:44.939930  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:44.879952  8722 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:25:44.962888  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.892419  8733 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7.
W20260709 12:25:44.972334  5636 rpcz_store.cc:269] 0709 12:25:43.317731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.407251 (+ 89520us) service_pool.cc:224] Handling call
0709 12:25:44.936793 (+1529542us) spinlock_profiling.cc:243] Waited 44.4 ms on lock 0x7b5400241154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:44.936943 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":102084480}
I20260709 12:25:44.896427  8767 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:44.975977  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1924) took 1555 ms. Trace:
W20260709 12:25:44.984072  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44279us
I20260709 12:25:44.990654  5702 rpcz_store.cc:276] 0709 12:25:43.420922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.421220 (+   298us) service_pool.cc:224] Handling call
0709 12:25:43.421734 (+   514us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:43.422186 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:43.422215 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:43.422928 (+   713us) log.cc:844] Serialized 52 byte log entry
0709 12:25:44.342950 (+920022us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:44.343006 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.343102 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:44.975743 (+632641us) raft_consensus.cc:1685] finished
0709 12:25:44.975763 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:44.975913 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18546,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":919507,"threads_started":1,"wal-append.queue_time_us":1013548}
I20260709 12:25:44.938989  8348 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:44.963995  8713 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:44.991364  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:44.940563  8785 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:25:44.996116  5705 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:44.881294  8720 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:44.998581  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:44.958647  8751 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:44.974388  8546 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:45.009810  5704 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:25:44.974191  8777 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:45.004838  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1928) took 1197 ms. Trace:
I20260709 12:25:44.999694  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:25:45.014489  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 111064us
I20260709 12:25:45.015471  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:45.015779  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:44.908089  8595 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260709 12:25:45.008023  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:45.000227  8566 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 12:25:45.013980  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:45.017668  5524 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:44.936537  8380 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20260709 12:25:45.017663  5709 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:45.023798  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.988899  8769 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:25:45.010118  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 228083us
I20260709 12:25:44.995206  8713 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:44.982035  8800 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:44.996507  8767 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.965629  8618 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.043s
I20260709 12:25:45.014623  5703 rpcz_store.cc:276] 0709 12:25:43.807635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.807793 (+   158us) service_pool.cc:224] Handling call
0709 12:25:43.808155 (+   362us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:43.817945 (+  9790us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:43.817991 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:43.818729 (+   738us) log.cc:844] Serialized 52 byte log entry
0709 12:25:44.427524 (+608795us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:44.772420 (+344896us) log.cc:844] Serialized 14 byte log entry
0709 12:25:45.003901 (+231481us) log.cc:844] Serialized 14 byte log entry
0709 12:25:45.004349 (+   448us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:45.004493 (+   144us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:45.004571 (+    78us) raft_consensus.cc:1685] finished
0709 12:25:45.004583 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:45.004767 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2217,"prepare.queue_time_us":477165,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":108024,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":16231,"thread_start_us":1103087,"threads_started":3,"wal-append.queue_time_us":812243}
I20260709 12:25:45.024031  8790 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:44.995812  8473 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:44.997607  8348 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:45.004657  8693 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:45.032160  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:45.010790  8795 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:44.979173  8783 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.039848  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:25:45.018327  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 151530us
I20260709 12:25:44.995541  8554 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.027169  8778 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:45.028103  8785 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:45.053833  5703 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260709 12:25:45.042237  8802 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
W20260709 12:25:45.049055  8596 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.159s	user 0.009s	sys 0.005s
I20260709 12:25:45.020974  8733 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
W20260709 12:25:45.085450  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45740us
I20260709 12:25:45.086117  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:45.081182  5709 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:45.027588  8760 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 12:25:45.043164  8807 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:45.045619  8767 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:45.023794  8806 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 12:25:45.021587  8694 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.073638  8348 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:45.079283  8456 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:45.076157  8790 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:45.097143  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 630) took 2317 ms. Trace:
I20260709 12:25:45.112114  6275 rpcz_store.cc:276] 0709 12:25:42.779556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.418862 (+639306us) service_pool.cc:224] Handling call
0709 12:25:43.419133 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:43.565303 (+146170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:45.096884 (+1531581us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b5400591ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:45.097068 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":302239104}
I20260709 12:25:45.086616  8795 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:45.113090  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:45.115787  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:45.098892  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 99244.8us
I20260709 12:25:45.079824  8778 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:45.090597  8546 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:45.094554  8804 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:25:45.105854  8793 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.265s	user 0.001s	sys 0.000s
I20260709 12:25:45.119024  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:45.106086  8506 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:45.105319  8498 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:45.104737  8473 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:45.088253  8554 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:45.117133  8778 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:45.112689  8812 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:45.090766  8596 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.118517  8694 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:45.163915  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 85522.6us
I20260709 12:25:45.118243  8530 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:45.172713  6234 leader_election.cc:336] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.185s (SENT)
I20260709 12:25:45.173327  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:45.095413  8782 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.106412  8808 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:25:45.105036  8811 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
W20260709 12:25:45.166428  8794 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
I20260709 12:25:45.138407  8506 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:45.207000  8813 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:25:45.207180  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.206844  8506 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.141399  8595 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:45.208575  8595 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:45.209072  8595 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:45.108888  8814 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20260709 12:25:45.210729  8814 consensus_queue.cc:1243] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:25:45.211220  8814 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:45.211541  8814 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:25:45.212391  8809 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.137s	user 0.002s	sys 0.000s
I20260709 12:25:45.212366  8814 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:45.213280  8777 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:45.108318  8807 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.184156  5559 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 10: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:45.187961  8666 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:45.100116  8769 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:45.191329  8825 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:45.192727  8821 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 6, Time since last communication: 0.017s
I20260709 12:25:45.193823  8554 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:45.200673  8810 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 12:25:45.209658  8795 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:45.215932  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 97010.2us
I20260709 12:25:45.208280  8506 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:45.246580  5707 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:45.213596  8814 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:45.255957  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 223863us
W20260709 12:25:45.189365  8805 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:25:45.129935  8817 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
W20260709 12:25:45.245919  8685 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.344s	user 0.013s	sys 0.001s
I20260709 12:25:45.186378  8812 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:45.213672  8777 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:45.176497  8787 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:45.214066  8684 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:45.266628  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:45.256127  8734 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.677s	user 0.010s	sys 0.006s
I20260709 12:25:45.253607  8697 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.275230  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:45.255609  8720 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:45.281179  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:45.282794  5631 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:45.284009  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:45.176030  8546 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.286594  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1712) took 1384 ms. Trace:
I20260709 12:25:45.287046  5629 rpcz_store.cc:276] 0709 12:25:43.902669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.904255 (+  1586us) service_pool.cc:224] Handling call
0709 12:25:43.904584 (+   329us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:43.906378 (+  1794us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:43.906407 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:43.906872 (+   465us) log.cc:844] Serialized 52 byte log entry
0709 12:25:44.426179 (+519307us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:44.426216 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.426278 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:45.286375 (+860097us) raft_consensus.cc:1685] finished
0709 12:25:45.286395 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:45.286536 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29775,"raft.queue_time_us":498171,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":4410,"thread_start_us":518945,"threads_started":1,"wal-append.queue_time_us":924477}
I20260709 12:25:45.287979  8697 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.291451  8697 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:45.292604  8803 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.230s	user 0.001s	sys 0.000s
W20260709 12:25:45.293712  8818 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260709 12:25:45.286000  8530 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:45.297338  8807 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:45.297747  8782 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:45.286033  8785 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:45.298709  8785 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:45.299052  8785 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:45.297886  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.300086  5709 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "addbcf2919194d2e866b5e8826422851"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:25:45.300716  5709 raft_consensus.cc:606] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:25:45.301051  5709 raft_consensus.cc:614] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:45.278007  8812 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:45.302527  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:45.304474  5635 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:25:45.306769  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:45.307329  6277 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:45.308947  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:45.310297  5627 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:25:45.310137  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:45.314049  5630 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:25:45.280890  8798 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:45.322346  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:45.328529  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:45.285038  8751 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.328912  6277 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
W20260709 12:25:45.332731  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 217019us
I20260709 12:25:45.334676  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:45.335389  6279 raft_consensus.cc:2368] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7.
I20260709 12:25:45.336445  5526 leader_election.cc:400] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7.
I20260709 12:25:45.277875  8685 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:45.334124  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:45.328500  8825 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:45.340078  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:45.340257  8825 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:45.335650  8777 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.342510  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:45.340061  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.340183  6278 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:45.286424  8733 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260709 12:25:45.342031  8825 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:45.344599  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:45.347955  6272 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:45.316668  8812 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:45.349107  5705 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: [3.5-7.8]
I20260709 12:25:45.358770  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:45.244697  8790 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.358839  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:45.349748  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:45.321835  8530 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:45.337877  8720 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:25:45.396534  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.424620  8720 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:45.425056  8720 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:45.427434  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:45.314958  8821 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.027s
I20260709 12:25:45.428572  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.428864  8821 consensus_queue.cc:1243] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:25:45.429435  5705 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:45.430171  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:45.432680  8695 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.017s	user 0.013s	sys 0.000s
I20260709 12:25:45.423684  8530 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.433245  8695 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.432557  5702 raft_consensus.cc:2376] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7. Re-sending same reply.
I20260709 12:25:45.420717  8827 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:45.432468  6279 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:45.420341  8830 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.431754  8713 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:45.339505  8596 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:45.341058  8751 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.497155  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:45.346952  8787 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.497498  5707 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 12:25:45.478267  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:45.462188  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 128124us
I20260709 12:25:45.603091  5702 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:45.498394  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:45.629091  5630 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:45.550815  5705 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:45.462646  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:45.630308  5627 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:45.630698  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:45.628989  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:45.632315  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.633488  5705 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:25:45.526687  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.633924  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:45.635648  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:45.594471  8800 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.526s	user 0.010s	sys 0.000s
W20260709 12:25:45.601122  8837 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.268s	user 0.002s	sys 0.000s
I20260709 12:25:45.643075  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:45.564011  8568 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.664s	user 0.011s	sys 0.007s
I20260709 12:25:45.525892  8456 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:45.637245  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.638398  5702 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:45.462185  8783 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:45.555900  8230 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.717s	user 0.009s	sys 0.002s
W20260709 12:25:45.640544  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.640947  8826 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:25:45.626259  8328 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.738s	user 0.014s	sys 0.000s
I20260709 12:25:45.603248  8827 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:45.697564  8328 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:45.697719  8827 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:45.698483  5558 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:45.699371  8618 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:45.699839  8618 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:45.643919  5707 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:45.651257  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:45.358151  8767 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:45.661449  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:45.712738  6279 raft_consensus.cc:1813] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Handling vote request from an unknown peer f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:45.629719  8821 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:45.672685  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.605568  8595 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:45.673827  8823 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:25:45.677369  8825 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:45.678557  8831 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:25:45.632158  8830 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:45.740569  8830 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:45.744896  8685 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:45.748862  8751 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:45.749434  8713 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:45.752805  8724 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.748s	user 0.004s	sys 0.010s
I20260709 12:25:45.753294  8724 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:45.755446  8724 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:45.756747  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.757412  5709 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:45.758343  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:45.758862  5709 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:45.698522  8695 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:45.682298  8781 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.216s	user 0.002s	sys 0.000s
I20260709 12:25:45.733129  8821 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:45.760632  8618 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:45.762581  8618 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:45.762804  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.763181  8821 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:45.764004  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.688560  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 550) took 2225 ms. Trace:
I20260709 12:25:45.688320  5627 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:25:45.777529  5631 rpcz_store.cc:276] 0709 12:25:43.463052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.463187 (+   135us) service_pool.cc:224] Handling call
0709 12:25:43.463465 (+   278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:45.283144 (+1819679us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:45.688292 (+405148us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b54000c0b54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:45.688500 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4183355136}
I20260709 12:25:45.778471  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:45.778748  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:45.779027  5627 raft_consensus.cc:1813] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Handling vote request from an unknown peer f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:45.779491  5627 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:45.690960  5630 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
W20260709 12:25:45.691772  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 516754us
I20260709 12:25:45.628332  8787 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:45.657704  8456 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:45.740108  8448 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.837s	user 0.004s	sys 0.005s
I20260709 12:25:45.713670  6279 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:45.794036  5703 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:45.663159  6277 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.8-7.8]
W20260709 12:25:45.722944  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.548000  8833 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.111s
I20260709 12:25:45.541950  8817 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.117s
I20260709 12:25:45.636617  8808 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:45.731526  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:45.788450  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:45.805570  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.684777  8699 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.766639  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.767364  8595 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:45.811916  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:45.812891  5555 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:45.814288  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:45.815047  5555 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:45.760521  8778 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.589s	user 0.005s	sys 0.005s
I20260709 12:25:45.770987  8825 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:45.771308  8767 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:45.648077  8800 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.726502  8328 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:45.776986  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.787292  5709 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:45.788300  5558 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:45.779357  5631 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:45.712615  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:45.790966  8737 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:45.793058  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:45.797387  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:45.798600  5553 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:45.799616  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:45.801651  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:45.802512  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:45.803630  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:45.790413  8380 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.808219  5627 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e20e319f46504838a20c558a66692da6"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:25:45.808382  5704 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 4.
W20260709 12:25:45.810084  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.792227  8456 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:45.820674  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:45.827616  5706 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.009s	sys 0.017s
W20260709 12:25:45.865772  5706 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.010s	sys 0.017s
W20260709 12:25:45.818385  8348 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.698s	user 0.011s	sys 0.000s
I20260709 12:25:45.848394  5559 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:45.847971  5703 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:45.848006  5557 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 4.
I20260709 12:25:45.849151  5556 raft_consensus.cc:2410] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:45.851198  5631 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20260709 12:25:45.852439  5707 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:45.853705  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.854856  8726 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:45.855655  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:45.867448  5555 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:45.869949  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.865010  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:25:45.857653  8785 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.558s	user 0.011s	sys 0.000s
I20260709 12:25:45.858173  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:45.857003  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:45.818626  8795 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:45.870674  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:45.906286  8795 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.906903  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:45.907287  5627 raft_consensus.cc:606] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:25:45.907579  5561 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:45.907945  5627 raft_consensus.cc:614] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:45.907281  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:45.799906  8448 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:45.908766  5555 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:45.898448  5628 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:45.910058  5561 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:25:45.910964  5705 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:45.899888  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:45.909127  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:45.891196  5634 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.015s	sys 0.034s
W20260709 12:25:45.914777  5634 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.017s	sys 0.034s
I20260709 12:25:45.914106  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:45.867563  5631 pending_rounds.cc:85] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 2
I20260709 12:25:45.915618  5561 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:25:45.915975  5631 pending_rounds.cc:107] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 12:25:45.916298  5631 raft_consensus.cc:2889] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:45.871060  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:45.914211  5559 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:45.917560  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:45.870925  5709 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:45.929384  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.930550  5704 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
W20260709 12:25:45.932888  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 67990.9us
I20260709 12:25:45.936971  5707 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:45.938190  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.939046  5703 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:45.939311  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.940130  5553 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:45.940744  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.941470  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.942126  5703 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:45.943403  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.943964  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:45.944423  5703 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:25:45.847424  8694 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.652s	user 0.005s	sys 0.005s
I20260709 12:25:45.946719  5704 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:45.917945  8787 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:45.832340  8699 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:45.869870  8726 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:45.899047  8380 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.035665  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:46.074677  5703 raft_consensus.cc:2410] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:25:45.947219  5707 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:45.826655  8778 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.910189  8230 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260709 12:25:45.897267  8506 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260709 12:25:45.942431  8503 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:25:45.961673  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.900978  8720 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:45.913596  8785 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:46.079844  5707 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:45.917584  8348 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:45.920961  8737 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:46.087119  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 172191us
I20260709 12:25:45.941854  8821 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.133s
W20260709 12:25:46.088160  8812 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.562s	user 0.005s	sys 0.005s
I20260709 12:25:46.088505  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:45.863273  8328 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:45.947811  8431 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.134s
I20260709 12:25:46.109004  5635 raft_consensus.cc:2368] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9.
I20260709 12:25:46.125502  5707 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:25:46.075374  8783 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260709 12:25:46.092839  8503 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:46.136868  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.095584  8795 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:46.138860  5554 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
W20260709 12:25:46.150857  8846 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
I20260709 12:25:46.123109  8726 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.153401  8726 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:46.159713  5556 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20260709 12:25:46.165254  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 89889.9us
I20260709 12:25:46.143004  8503 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:46.184659  8448 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:46.186846  8770 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.827s	user 0.006s	sys 0.007s
I20260709 12:25:46.187310  8380 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:46.127943  8812 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.189489  8720 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:46.189771  8673 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:46.190974  8785 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:46.192698  8843 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.391s	user 0.000s	sys 0.003s
I20260709 12:25:46.192803  8778 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:46.148295  8827 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:46.224398  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 163504us
I20260709 12:25:46.213030  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 666) took 3346 ms. Trace:
I20260709 12:25:46.213347  8812 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:46.219525  8854 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.209836  8666 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:46.215338  8684 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.927s	user 0.004s	sys 0.008s
I20260709 12:25:46.225063  6280 rpcz_store.cc:276] 0709 12:25:42.866733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.443090 (+576357us) service_pool.cc:224] Handling call
0709 12:25:43.443358 (+   268us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:43.446126 (+  2768us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:43.446155 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:43.446596 (+   441us) log.cc:844] Serialized 26 byte log entry
0709 12:25:43.836067 (+389471us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:43.836109 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.836198 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:46.080183 (+2243985us) spinlock_profiling.cc:243] Waited 39.2 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:46.212737 (+132554us) raft_consensus.cc:1685] finished
0709 12:25:46.212758 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:46.212969 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2134,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":90200960,"thread_start_us":389069,"threads_started":1,"wal-append.queue_time_us":562839}
I20260709 12:25:46.196408  8856 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.231941  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:46.217465  8800 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:46.240728  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:46.248451  5629 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:46.252715  5629 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 4. Current term is 5. Ops: []
I20260709 12:25:46.260206  5704 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:46.231132  8666 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:46.253964  8849 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260709 12:25:46.270149  5704 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:25:46.252442  8724 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:46.232277  8854 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.289433  8853 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:25:46.236827  8856 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.276876  8866 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.278505  8724 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.287151  8827 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:46.288782  8348 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:46.288009  8737 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:46.298332  8328 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:46.298635  8724 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:46.328161  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:46.329499  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:46.330868  5629 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 10: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:46.331688  8854 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:46.333400  8666 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:46.335153  6236 connection.cc:256] RPC call timeout handler was delayed by -1.3741901659996074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1567s for a response
I20260709 12:25:46.341495  5628 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:25:46.342417  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:46.339291  8853 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.346112  8868 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:25:46.349504  8868 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:25:46.352967  8861 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
W20260709 12:25:46.361294  8863 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:25:46.374374  8856 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:46.384742  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:46.387095  8868 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:46.388027  8868 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:25:46.339402  8854 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.400835  8853 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.397073  8854 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.407756  8854 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:46.415087  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2050) took 1105 ms. Trace:
I20260709 12:25:46.415570  5635 rpcz_store.cc:276] 0709 12:25:45.309268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.309832 (+   564us) service_pool.cc:224] Handling call
0709 12:25:46.414895 (+1105063us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:46.415023 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1834937088}
I20260709 12:25:46.428236  5555 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:25:46.429388  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1955) took 1436 ms. Trace:
W20260709 12:25:46.438994  5558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1783) took 1847 ms (client timeout 1980 ms). Trace:
W20260709 12:25:46.439428  5558 rpcz_store.cc:269] 0709 12:25:44.591460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.591602 (+   142us) service_pool.cc:224] Handling call
0709 12:25:46.438843 (+1847241us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54000ebfd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:46.438940 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2510290816}
I20260709 12:25:46.438903  5706 rpcz_store.cc:276] 0709 12:25:44.992966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.027422 (+ 34456us) service_pool.cc:224] Handling call
0709 12:25:45.046311 (+ 18889us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:45.046623 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:25:45.276279 (+229656us) log.cc:844] Serialized 14 byte log entry
0709 12:25:46.428873 (+1152594us) log.cc:844] Serialized 14 byte log entry
0709 12:25:46.429042 (+   169us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:46.429082 (+    40us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:46.429132 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.429209 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:46.429329 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":621911,"prepare.queue_time_us":46547,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":81,"raft.queue_time_us":567056,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":128,"thread_start_us":792581,"threads_started":2,"wal-append.queue_time_us":667065}
W20260709 12:25:46.437891  8456 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.527s	user 0.010s	sys 0.002s
I20260709 12:25:46.306309  8866 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.441674  8456 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:46.443352  8866 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:46.443403  5704 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:25:46.444623  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:46.430857  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 531225us
W20260709 12:25:46.445755  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 213855us
I20260709 12:25:46.446305  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:46.446646  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:46.445794  5705 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:46.435071  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:46.447878  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:46.452596  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:46.459028  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 514) took 1111 ms. Trace:
I20260709 12:25:46.460768  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:46.462268  5702 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 9: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:46.463406  5599 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:46.464515  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:46.470048  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:46.473001  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:46.434405  8851 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.310s	user 0.002s	sys 0.000s
I20260709 12:25:46.459620  5701 rpcz_store.cc:276] 0709 12:25:45.348024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.348137 (+   113us) service_pool.cc:224] Handling call
0709 12:25:45.348468 (+   331us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:45.526436 (+177968us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:45.526465 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:45.636036 (+109571us) log.cc:844] Serialized 52 byte log entry
0709 12:25:45.827972 (+191936us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:45.860573 (+ 32601us) log.cc:844] Serialized 14 byte log entry
0709 12:25:46.379779 (+519206us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x7b1800046480. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:25:46.423933 (+ 44154us) spinlock_profiling.cc:243] Waited 30.4 ms on lock 0x7b1800046480. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:25:46.424476 (+   543us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.424639 (+   163us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:46.458703 (+ 34064us) raft_consensus.cc:1685] finished
0709 12:25:46.458732 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:46.458945 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":187029,"prepare.queue_time_us":41256,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":748254,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":242,"spinlock_wait_cycles":137018112,"thread_start_us":165881,"threads_started":1,"wal-append.queue_time_us":225537}
I20260709 12:25:46.479902  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:46.485720  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:46.488714  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:46.449259  8866 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:46.431396  8788 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:46.491201  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:46.433363  8693 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.494371  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:46.435578  8811 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:46.530725  5627 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:25:46.477895  8783 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:46.480801  8787 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:25:46.480890  8854 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:46.490378  8853 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:46.531817  8348 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:25:46.532758  6280 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:25:46.533314  5627 raft_consensus.cc:606] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:25:46.534631  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:46.532327  8866 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:46.540315  6275 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
I20260709 12:25:46.490549  8751 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:46.478235  8695 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:46.521644  8868 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:46.592830  6276 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:46.584162  6278 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:25:46.595947  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:46.605032  6277 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:46.606881  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:46.607586  6274 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:46.521694  8780 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:25:46.565107  5602 leader_election.cc:336] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.738s (SENT)
I20260709 12:25:46.619117  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:46.492424  8836 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:25:46.587100  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47679.2us
W20260709 12:25:46.604772  8841 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:25:46.553630  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:46.642485  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 153866us
W20260709 12:25:46.643003  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 77881.3us
W20260709 12:25:46.644027  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.540091  8431 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:46.491390  8673 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:46.604826  5627 raft_consensus.cc:614] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:46.497735  8800 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:46.482796  8856 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:46.525209  8788 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:46.606109  6273 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:25:46.545727  8347 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20260709 12:25:46.628562  8503 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:46.629092  6280 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:25:46.546977  8860 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.083s
W20260709 12:25:46.667321  8848 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.584s	user 0.000s	sys 0.002s
I20260709 12:25:46.606231  8868 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.668583  8868 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:46.670281  8503 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:46.656544  8800 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.671901  8800 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:46.632205  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:46.636976  8821 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:46.640756  8506 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:46.651201  8858 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:25:46.641471  8699 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.567s	user 0.008s	sys 0.004s
W20260709 12:25:46.673805  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.674572  5709 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 12:25:46.674952  5709 pending_rounds.cc:85] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 7
I20260709 12:25:46.675170  5709 pending_rounds.cc:107] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:25:46.675390  5709 raft_consensus.cc:2889] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:46.611733  6277 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:25:46.625175  8836 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.498566  8711 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:25:46.686376  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 54245.8us
W20260709 12:25:46.687327  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 61275.7us
I20260709 12:25:46.653157  8804 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20260709 12:25:46.688850  8858 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:46.689291  8858 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.689649  8858 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:46.680202  8812 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.693264  8812 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.654187  6274 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:46.653378  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2045) took 1352 ms. Trace:
I20260709 12:25:46.618993  8780 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.655110  8751 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:46.537621  8693 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:46.661830  8431 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:46.660097  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 317764us
I20260709 12:25:46.665203  8456 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:46.666025  8760 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:25:46.666275  8724 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.672310  8230 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:46.664038  8788 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:46.725536  8693 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:46.726449  5704 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:46.726758  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1953) took 1734 ms. Trace:
I20260709 12:25:46.644245  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:46.696663  5634 rpcz_store.cc:276] 0709 12:25:45.301034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.302261 (+  1227us) service_pool.cc:224] Handling call
0709 12:25:45.302551 (+   290us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:45.304771 (+  2220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:45.304798 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:45.305242 (+   444us) log.cc:844] Serialized 26 byte log entry
0709 12:25:45.915157 (+609915us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:45.915218 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:45.915291 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:46.433862 (+518571us) spinlock_profiling.cc:243] Waited 5.94 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:46.642074 (+208212us) raft_consensus.cc:1685] finished
0709 12:25:46.642096 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:46.653311 (+ 11215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1922,"raft.queue_time_us":2615,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":20346,"spinlock_wait_cycles":13672448,"thread_start_us":570133,"threads_started":1,"wal-append.queue_time_us":721267}
I20260709 12:25:46.703390  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:46.728586  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 84451.5us
I20260709 12:25:46.686319  8836 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:46.698679  8860 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:46.706535  8431 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:46.705430  8503 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.700134  8780 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:46.712283  5707 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:25:46.712644  8760 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:46.713532  8711 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:46.713351  8868 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:46.716924  5602 leader_election.cc:336] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.886s (SENT)
I20260709 12:25:46.719066  8858 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.729481  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:46.727113  5708 rpcz_store.cc:276] 0709 12:25:44.992394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.002255 (+  9861us) service_pool.cc:224] Handling call
0709 12:25:45.020692 (+ 18437us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:45.021083 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:45.021111 (+    28us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:45.028089 (+  6978us) log.cc:844] Serialized 78 byte log entry
0709 12:25:46.234275 (+1206186us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b18004689a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:25:46.627177 (+392902us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:25:46.662979 (+ 35802us) log.cc:844] Serialized 14 byte log entry
0709 12:25:46.663381 (+   402us) log.cc:844] Serialized 14 byte log entry
0709 12:25:46.663858 (+   477us) log.cc:844] Serialized 14 byte log entry
0709 12:25:46.710454 (+ 46596us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b18004689a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:25:46.710677 (+   223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.710762 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:46.712322 (+  1560us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b18004689a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:25:46.715234 (+  2912us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b18004689a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:25:46.718735 (+  3501us) spinlock_profiling.cc:243] Waited 8.98 ms on lock 0x7b18004689a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:25:46.726549 (+  7814us) raft_consensus.cc:1685] finished
0709 12:25:46.726568 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:46.726703 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8566,"prepare.queue_time_us":37537,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":80,"raft.queue_time_us":74242,"raft.run_cpu_time_us":452,"raft.run_wall_time_us":34954,"spinlock_wait_cycles":121771008,"thread_start_us":392686,"threads_started":1,"wal-append.queue_time_us":1616567}
I20260709 12:25:46.739064  5704 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:46.750432  5705 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
I20260709 12:25:46.732487  8693 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:46.732812  8860 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:46.760922  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:46.719851  8751 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.762689  5631 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:46.781335  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:46.782588  5631 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:46.763336  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:46.783664  5628 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:25:46.763826  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:46.766466  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:46.765244  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:46.785974  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:46.767475  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:46.770083  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.773674  8868 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:46.776849  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:46.777400  8854 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:46.773818  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1936) took 1130 ms. Trace:
I20260709 12:25:46.763386  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:25:46.796939  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:46.789173  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:46.797657  5629 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:46.797744  5705 raft_consensus.cc:2376] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6. Re-sending same reply.
I20260709 12:25:46.798223  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:46.798583  5710 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 12:25:46.799111  5710 pending_rounds.cc:85] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 7
I20260709 12:25:46.799608  5710 pending_rounds.cc:107] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:25:46.799882  5710 raft_consensus.cc:2889] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:46.742897  8503 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:46.794337  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:46.805503  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:46.806334  5705 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:25:46.795051  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:46.795814  5703 rpcz_store.cc:276] 0709 12:25:45.643824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.119630 (+475806us) service_pool.cc:224] Handling call
0709 12:25:46.141633 (+ 22003us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:46.431990 (+290357us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:46.432039 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:46.432699 (+   660us) log.cc:844] Serialized 26 byte log entry
0709 12:25:46.626596 (+193897us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:25:46.626649 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.626740 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:46.773590 (+146850us) raft_consensus.cc:1685] finished
0709 12:25:46.773611 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:46.773752 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":289772,"raft.queue_time_us":291427,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"thread_start_us":193435,"threads_started":1,"wal-append.queue_time_us":313287}
I20260709 12:25:46.780269  8693 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.801550  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:46.779940  8787 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:46.739820  8878 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:25:46.757331  8760 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.806573  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:46.791230  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 74344.6us
I20260709 12:25:46.822026  5708 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:25:46.822877  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:46.779153  8779 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.825145  8779 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:46.827435  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1797) took 1018 ms. Trace:
I20260709 12:25:46.805243  5629 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 12:25:46.808962  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:46.828938  5705 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:46.827869  5560 rpcz_store.cc:276] 0709 12:25:45.809426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.908540 (+ 99114us) service_pool.cc:224] Handling call
0709 12:25:45.908802 (+   262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:45.909004 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:25:46.109886 (+200882us) log.cc:844] Serialized 14 byte log entry
0709 12:25:46.224404 (+114518us) log.cc:844] Serialized 14 byte log entry
0709 12:25:46.826920 (+602516us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b18002940a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:25:46.827060 (+   140us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:46.827112 (+    52us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:46.827166 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.827248 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:46.827375 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":602892,"prepare.queue_time_us":238791,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":1360,"raft.queue_time_us":153,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":3913,"spinlock_wait_cycles":3409664,"thread_start_us":282478,"threads_started":2,"wal-append.queue_time_us":127037}
I20260709 12:25:46.810424  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:46.812701  5631 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 12:25:46.814101  5706 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:25:46.743481  8871 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.059s
I20260709 12:25:46.794210  8347 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:46.794323  8864 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:46.817198  8800 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:46.817721  8503 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:46.812795  8348 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 12:25:46.812708  8693 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:46.820060  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:46.851223  5703 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:46.839741  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:46.844193  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:46.849918  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:46.855976  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.870162  5705 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:46.871632  8699 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 12:25:46.853720  8855 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.003s	sys 0.009s
I20260709 12:25:46.873109  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:46.876310  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:46.872239  8868 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.007s
I20260709 12:25:46.877501  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:46.811960  8711 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.892310  5629 raft_consensus.cc:2410] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:25:46.872619  8699 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:25:46.852702  8827 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.856120  8737 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:46.878721  8855 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.003s	sys 0.010s
W20260709 12:25:46.896231  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:46.901973  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:46.902426  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.905161  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:46.905786  5558 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:25:46.906284  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:46.906936  5554 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:46.906802  5555 raft_consensus.cc:2376] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7. Re-sending same reply.
I20260709 12:25:46.906834  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:46.907968  5558 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:25:46.908224  5561 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 12:25:46.908300  5558 pending_rounds.cc:85] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 6
I20260709 12:25:46.908699  5558 pending_rounds.cc:107] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:25:46.908964  5558 raft_consensus.cc:2889] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:46.909477  5556 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:25:46.908838  5554 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:25:46.880406  8804 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:46.912273  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:46.912954  5560 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:25:46.913542  5559 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:46.911847  5558 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:46.914160  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:46.915892  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:46.916246  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:46.916610  5558 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:46.914985  5555 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:46.918354  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:46.919091  5560 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:46.878396  8854 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:25:46.921552  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:46.922793  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:46.923555  5552 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:46.896893  8699 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:46.903046  8696 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.910296  8720 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.828682  8860 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:46.933596  8696 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.912103  8380 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.940595  5561 raft_consensus.cc:1217] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Deduplicated request from leader. Original: 3.3->[4.4-9.6]   Dedup: 4.4->[7.5-9.6]
I20260709 12:25:46.947032  5561 pending_rounds.cc:85] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 4
I20260709 12:25:46.947470  5561 pending_rounds.cc:107] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 12:25:46.947826  5561 raft_consensus.cc:2889] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:46.951686  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:46.952862  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:46.953167  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:46.903705  8737 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:46.955981  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:46.954268  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:46.958629  5558 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:46.954509  8684 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:25:46.903913  8827 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.955909  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:46.960877  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:46.961326  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:46.961552  6279 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:46.962432  6277 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:46.961355  6274 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:46.972100  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:46.972293  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:46.973219  5556 raft_consensus.cc:2410] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 8 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:46.944310  8686 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:46.945999  8858 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:46.985467  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:46.924265  8804 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.975677  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:46.974911  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:46.950014  8785 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.988224  8686 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.998971  8785 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.999580  8827 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:47.000466  8804 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:47.000725  8686 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:46.980194  8699 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:47.002640  8699 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:46.985600  8506 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:46.963881  8328 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:46.986058  5556 raft_consensus.cc:2376] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8. Re-sending same reply.
I20260709 12:25:47.004859  8506 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:47.006018  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:47.007144  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:47.007833  5556 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:46.985267  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:46.955550  8729 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:25:46.997574  5560 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:46.952991  8380 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:46.958722  8876 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.008888  6280 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:46.940613  8720 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:46.988925  5558 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
W20260709 12:25:47.011202  5602 leader_election.cc:336] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.101s (SENT)
I20260709 12:25:47.031752  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:46.995494  6274 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:46.960973  8860 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:46.972638  8673 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.005942  8737 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:47.036805  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:47.034533  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:47.049674  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:47.054148  5631 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:47.045498  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:47.033255  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 79422.3us
I20260709 12:25:47.027940  8858 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:47.061167  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 96381.9us
I20260709 12:25:47.074522  5526 leader_election.cc:400] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9.
I20260709 12:25:47.080672  5526 leader_election.cc:403] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:47.084822  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:47.086091  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:47.087563  5556 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:47.088291  5675 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:47.090001  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:47.062186  6279 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:47.021533  8328 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:47.090219  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:47.033819  8729 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:47.128253  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:47.128664  5633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 557) took 3573 ms (client timeout 1567 ms). Trace:
W20260709 12:25:47.129272  5633 rpcz_store.cc:269] 0709 12:25:43.555413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.555622 (+   209us) service_pool.cc:224] Handling call
0709 12:25:47.104866 (+3549244us) spinlock_profiling.cc:243] Waited 2.32 s on lock 0x7b54001cf2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:47.128561 (+ 23695us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5330512256}
I20260709 12:25:47.164098  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:47.165197  5706 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 6: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:47.163996  5631 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:47.050630  8800 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:47.176160  5632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1707) took 3296 ms (client timeout 2991 ms). Trace:
I20260709 12:25:47.066908  8695 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.093995  8456 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.082996  8804 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.094238  8828 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.099931  5556 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:47.108646  8686 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 9
W20260709 12:25:47.180083  5632 rpcz_store.cc:269] 0709 12:25:43.879977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.882694 (+  2717us) service_pool.cc:224] Handling call
0709 12:25:47.175833 (+3293139us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54001ce8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:47.176038 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3194754304}
I20260709 12:25:47.106985  8506 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:47.106508  8779 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:47.034456  8876 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:47.199482  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:47.227986  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.056061  8673 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:47.083663  8448 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.104s
I20260709 12:25:47.211493  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:47.205181  8800 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:47.098906  8858 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.181602  8338 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260709 12:25:47.194978  8684 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260709 12:25:47.205505  8804 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.205401  8456 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.197602  8695 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.215628  8828 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.364622  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:47.365538  5556 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:25:47.245429  8881 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.306s	user 0.001s	sys 0.001s
W20260709 12:25:47.318065  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 229843us
W20260709 12:25:47.296447  8880 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.352s	user 0.001s	sys 0.000s
W20260709 12:25:47.370105  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 281410us
I20260709 12:25:47.368901  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:47.348927  8890 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
W20260709 12:25:47.325099  8866 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.705s	user 0.008s	sys 0.005s
I20260709 12:25:47.370415  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:47.339284  8884 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260709 12:25:47.383971  5703 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
W20260709 12:25:47.378295  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.378859  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:47.347318  8811 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:47.339958  8696 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:47.396096  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:47.396301  5632 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:25:47.397126  5631 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 8 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:25:47.380789  5602 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.236s (SENT)
I20260709 12:25:47.257558  8785 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:47.406885  5602 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.296s (SENT)
I20260709 12:25:47.412108  8866 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:47.409983  8811 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:47.316781  8729 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.457507  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:47.421960  8684 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:47.253888  8328 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:47.421114  8338 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:25:47.472632  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 382486us
W20260709 12:25:47.493549  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 86709.4us
W20260709 12:25:47.494421  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.508841  5706 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:25:47.435441  8785 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.055s
I20260709 12:25:47.470989  8684 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:47.484341  8338 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:47.457892  8903 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:47.513166  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:47.524971  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:47.516618  8731 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.007s	sys 0.014s
I20260709 12:25:47.550431  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:47.551359  5555 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:47.491767  8380 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:47.516558  8673 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:47.562942  8804 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:47.563483  8699 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.560s	user 0.013s	sys 0.000s
I20260709 12:25:47.565913  8858 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:47.537542  8876 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:47.568527  8779 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:25:47.568831  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:47.569902  8078 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:25:47.570663  8821 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:47.571326  8821 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:47.572599  8904 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:47.573179  8904 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:47.572492  8821 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.574432  8673 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.574932  8673 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:47.533398  8788 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.811s	user 0.012s	sys 0.001s
I20260709 12:25:47.576138  8788 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.578328  8788 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:47.577396  8904 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.580186  8876 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:47.579015  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:47.536959  8902 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
W20260709 12:25:47.558403  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.584296  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:47.585996  5636 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: [6.5-8.6]
I20260709 12:25:47.586472  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:47.586851  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2046) took 2285 ms. Trace:
I20260709 12:25:47.587534  8854 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:25:47.588544  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:47.589300  5630 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
W20260709 12:25:47.589888  5630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1727) took 1949 ms (client timeout 2212 ms). Trace:
I20260709 12:25:47.589973  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:47.590324  5630 rpcz_store.cc:269] 0709 12:25:45.641664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.781010 (+139346us) service_pool.cc:224] Handling call
0709 12:25:47.588320 (+1807310us) spinlock_profiling.cc:243] Waited 7.45 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:47.589825 (+  1505us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17138048}
I20260709 12:25:47.538151  8896 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.592656  8828 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:47.592919  8896 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.593567  8338 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.594822  8338 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:47.561249  8456 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:47.584386  8866 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:47.629055  8854 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:25:47.629575  8836 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.888s	user 0.013s	sys 0.000s
I20260709 12:25:47.630191  8836 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:47.632962  8897 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.146s	user 0.000s	sys 0.000s
I20260709 12:25:47.634564  8673 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:47.636533  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:47.638613  8780 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.883s	user 0.011s	sys 0.000s
I20260709 12:25:47.595079  8896 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:47.639163  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:47.537046  8807 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:47.588107  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:47.641964  5701 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:25:47.642975  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:47.656265  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:47.587245  5636 rpcz_store.cc:276] 0709 12:25:45.301311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.302761 (+  1450us) service_pool.cc:224] Handling call
0709 12:25:45.321200 (+ 18439us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:47.585746 (+2264546us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54001d9820. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa6741c0 00007f4ffa675231 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c
0709 12:25:47.586446 (+   700us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:47.586774 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":274013952}
I20260709 12:25:47.585228  5558 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:47.658708  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:47.659603  5558 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 6: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:25:47.549890  8731 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.008s	sys 0.014s
I20260709 12:25:47.607189  5628 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
W20260709 12:25:47.556082  5602 leader_election.cc:336] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.663s (SENT)
I20260709 12:25:47.657160  5703 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:47.664791  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:47.665959  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 109735us
I20260709 12:25:47.663292  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:47.689054  8896 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:47.690030  8858 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:25:47.693308  5630 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
W20260709 12:25:47.698526  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 319720us
I20260709 12:25:47.648588  8866 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.652825  8854 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:47.724790  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.705996  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:47.731756  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:25:47.733145  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:47.709894  8880 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:25:47.701228  8566 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.003s	sys 0.028s
I20260709 12:25:47.745415  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:47.754199  5558 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:47.745412  8912 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.737497  8380 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.759922  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:47.754534  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:47.739718  8673 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:47.648874  8684 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:47.749846  8566 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.004s	sys 0.028s
I20260709 12:25:47.800559  8506 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:47.655874  8780 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:47.717696  8795 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.045s
I20260709 12:25:47.679085  8807 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:47.804782  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49329.5us
I20260709 12:25:47.739307  8896 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20260709 12:25:47.762161  8779 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:47.748783  8876 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.084s
I20260709 12:25:47.792572  8912 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.737972  8866 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.844266  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:47.839900  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1986) took 1139 ms. Trace:
I20260709 12:25:47.749914  8078 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.092s
I20260709 12:25:47.793330  8891 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:47.739336  8854 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.746713  8858 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.093s
I20260709 12:25:47.803194  8726 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:47.803210  8911 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:47.858667  5709 rpcz_store.cc:276] 0709 12:25:46.700130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.700301 (+   171us) service_pool.cc:224] Handling call
0709 12:25:46.700727 (+   426us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:46.714247 (+ 13520us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:46.714279 (+    32us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:46.715023 (+   744us) log.cc:844] Serialized 78 byte log entry
0709 12:25:46.841555 (+126532us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:46.880327 (+ 38772us) log.cc:844] Serialized 14 byte log entry
0709 12:25:46.892159 (+ 11832us) log.cc:844] Serialized 14 byte log entry
0709 12:25:46.892416 (+   257us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.892490 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:47.567099 (+674609us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b48001701d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:25:47.839543 (+272444us) raft_consensus.cc:1685] finished
0709 12:25:47.839571 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:47.839812 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41300,"prepare.queue_time_us":33522,"prepare.run_wall_time_us":44,"raft.queue_time_us":27277,"raft.run_cpu_time_us":4898,"raft.run_wall_time_us":675219,"spinlock_wait_cycles":25221760,"thread_start_us":125873,"threads_started":1,"wal-append.queue_time_us":187897}
I20260709 12:25:47.790944  8856 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
I20260709 12:25:47.856653  8836 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:47.790213  8922 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.112s
I20260709 12:25:47.852487  8779 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:47.861392  8684 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:47.835503  8554 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 12:25:47.804240  8380 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:47.873566  8911 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:47.871312  8896 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
W20260709 12:25:47.897585  8899 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.344s	user 0.001s	sys 0.000s
I20260709 12:25:47.875753  8726 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:47.892590  8828 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:47.881561  8903 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:25:47.917032  8921 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:25:47.916945  8912 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:47.911820  8911 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:47.902149  8779 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:25:47.911286  8864 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.068s	user 0.012s	sys 0.002s
W20260709 12:25:47.914887  8686 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.699s	user 0.008s	sys 0.004s
I20260709 12:25:47.914472  8878 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:47.969300  8858 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:47.970139  8858 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:47.964640  8693 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.102s	user 0.014s	sys 0.000s
W20260709 12:25:47.973263  8915 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:25:47.973383  8693 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:47.975664  8693 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:47.966792  8878 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:47.976888  8760 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:47.977054  8878 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:47.978307  8878 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:47.967796  8686 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9.
I20260709 12:25:47.968099  8864 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:25:47.969244  8800 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.601s	user 0.011s	sys 0.000s
I20260709 12:25:47.971354  8858 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:48.011584  8933 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:48.012192  8933 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:48.019434  8933 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:48.008101  8934 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:48.074558  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:47.792061  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:48.087383  5553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1803) took 2266 ms (client timeout 2089 ms). Trace:
W20260709 12:25:48.089138  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:48.092749  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 300636us
I20260709 12:25:48.094976  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:48.095953  5559 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.005s	sys 0.022s
W20260709 12:25:48.111826  5559 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.005s	sys 0.023s
I20260709 12:25:48.097512  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1803) took 1166 ms. Trace:
I20260709 12:25:48.098912  5557 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:48.112834  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1793) took 1205 ms. Trace:
I20260709 12:25:48.113343  5557 rpcz_store.cc:276] 0709 12:25:46.907470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.912736 (+  5266us) service_pool.cc:224] Handling call
0709 12:25:48.098308 (+1185572us) spinlock_profiling.cc:243] Waited 3.63 ms on lock 0x7b54002fa6a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa675e18 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:48.112738 (+ 14430us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8350080}
I20260709 12:25:48.113806  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 532) took 1144 ms. Trace:
I20260709 12:25:48.101866  5558 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:48.114257  5559 rpcz_store.cc:276] 0709 12:25:46.969279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:47.465659 (+496380us) service_pool.cc:224] Handling call
0709 12:25:47.471652 (+  5993us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:47.472004 (+   352us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:47.472034 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:47.475895 (+  3861us) log.cc:844] Serialized 26 byte log entry
0709 12:25:48.113271 (+637376us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:25:48.113370 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:48.113460 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:48.113524 (+    64us) raft_consensus.cc:1685] finished
0709 12:25:48.113535 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:48.113730 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":24570,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":12624,"thread_start_us":635988,"threads_started":1,"wal-append.queue_time_us":441751}
W20260709 12:25:48.088476  5553 rpcz_store.cc:269] 0709 12:25:45.820647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.857919 (+ 37272us) service_pool.cc:224] Handling call
0709 12:25:48.086336 (+2228417us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x7b54003a98d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:48.086844 (+   508us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":63476992}
I20260709 12:25:48.112556  5561 rpcz_store.cc:276] 0709 12:25:46.931171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.931348 (+   177us) service_pool.cc:224] Handling call
0709 12:25:46.940296 (+  8948us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:47.077965 (+137669us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:25:47.369758 (+291793us) log.cc:844] Serialized 14 byte log entry
0709 12:25:47.793246 (+423488us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:47.793955 (+   709us) log.cc:844] Serialized 52 byte log entry
0709 12:25:47.794111 (+   156us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:25:47.794141 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:47.794219 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:48.036910 (+242691us) spinlock_profiling.cc:243] Waited 6.22 ms on lock 0x7b18000fbec0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:25:48.096718 (+ 59808us) raft_consensus.cc:1685] finished
0709 12:25:48.096955 (+   237us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:48.097310 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5712,"prepare.queue_time_us":224482,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":23316,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":11831,"spinlock_wait_cycles":14301056,"thread_start_us":652824,"threads_started":3,"wal-append.queue_time_us":510624}
I20260709 12:25:48.105684  5560 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:48.096444  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:48.119024  5524 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:48.119318  5560 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:48.122108  5560 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:48.124946  5560 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:48.126744  5556 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:48.124949  5552 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
W20260709 12:25:48.135113  5633 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.012s	sys 0.053s
W20260709 12:25:48.135661  5633 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.012s	sys 0.054s
I20260709 12:25:48.144330  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1735) took 1107 ms. Trace:
I20260709 12:25:48.144997  5628 rpcz_store.cc:276] 0709 12:25:47.036284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:47.036505 (+   221us) service_pool.cc:224] Handling call
0709 12:25:48.143770 (+1107265us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b540009e7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:48.143982 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":753358592}
I20260709 12:25:47.887570  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:48.153151  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 265530us
W20260709 12:25:48.153268  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.154605  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:48.156657  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.157968  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:47.889065  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:48.165695  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 276564us
I20260709 12:25:48.166658  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 1984) took 1722 ms. Trace:
I20260709 12:25:48.167629  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:48.167150  5701 rpcz_store.cc:276] 0709 12:25:46.444292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.495784 (+ 51492us) service_pool.cc:224] Handling call
0709 12:25:46.496159 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:47.655538 (+1159379us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:48.165731 (+510193us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400210f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:48.166363 (+   632us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2634312576}
I20260709 12:25:48.171284  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1938) took 2526 ms. Trace:
I20260709 12:25:48.171782  5707 rpcz_store.cc:276] 0709 12:25:45.644412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.126819 (+482407us) service_pool.cc:224] Handling call
0709 12:25:46.712030 (+585211us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:46.740861 (+ 28831us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:48.170731 (+1429870us) spinlock_profiling.cc:243] Waited 585 ms on lock 0x7b54000e01d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:48.171046 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1344776576}
W20260709 12:25:47.893039  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 250178us
W20260709 12:25:48.174836  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:48.177034  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.008s	sys 0.008s
W20260709 12:25:48.177559  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.009s	sys 0.008s
W20260709 12:25:48.178030  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 525453us
W20260709 12:25:48.181725  5704 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 1981) took 1737 ms (client timeout 1829 ms). Trace:
W20260709 12:25:48.182587  5704 rpcz_store.cc:269] 0709 12:25:46.443602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.444402 (+   800us) service_pool.cc:224] Handling call
0709 12:25:48.180950 (+1736548us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b5400451fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:48.181311 (+   361us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":646810368}
I20260709 12:25:48.183754  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:48.184741  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.185458  5559 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:48.193003  8448 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:25:48.193924  8328 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:48.194608  8328 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:48.194442  8448 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:25:48.194969  8380 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:48.195436  8448 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.196130  8448 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:25:48.195863  8380 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:25:48.200759  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:48.204557  8328 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:48.182603  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:48.206666  6273 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:48.207402  8503 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:25:48.207855  8503 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:48.225757  8941 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.180s	user 0.000s	sys 0.002s
W20260709 12:25:48.237128  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 61389.6us
I20260709 12:25:48.021729  8866 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:48.246151  8866 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:25:48.249128  8693 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.998562  8800 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:48.253326  8693 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:48.261545  5554 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.003s	sys 0.010s
W20260709 12:25:48.262418  5554 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.004s	sys 0.010s
I20260709 12:25:48.264936  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:48.266862  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:48.267848  5553 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:48.012346  8854 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:48.269695  8780 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:48.274425  8711 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:25:48.275715  8864 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:48.276511  8864 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:48.278155  8864 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:48.280726  8903 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:48.285389  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:48.286646  8812 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:48.264200  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:48.288841  8854 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:48.289285  8854 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.289621  8854 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:48.293354  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:48.295132  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:48.296182  5561 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:48.235141  8737 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:48.257004  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 76663.8us
I20260709 12:25:48.304334  5560 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:25:48.308769  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:48.309203  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:48.310204  8904 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:48.310761  8904 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.311125  8699 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:25:48.313056  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 81092.9us
I20260709 12:25:48.314208  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:48.316327  8780 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:48.317020  8780 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:48.317427  8724 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:48.317943  8554 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:48.318795  8554 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:48.320793  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68182.9us
W20260709 12:25:48.323055  8786 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 12:25:48.323815  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:48.309038  8804 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:48.324276  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:48.325944  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:48.326751  5709 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:48.322489  8737 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:48.327582  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:48.328971  5706 raft_consensus.cc:2410] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:25:48.217912  8503 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:48.330675  8503 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:48.331190  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:48.332705  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 536) took 1362 ms. Trace:
I20260709 12:25:48.334049  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:48.334378  8937 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:25:48.335839  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:48.336504  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:48.337232  5558 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:48.337742  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1810) took 1166 ms. Trace:
I20260709 12:25:48.338117  5556 rpcz_store.cc:276] 0709 12:25:47.171682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:47.370172 (+198490us) service_pool.cc:224] Handling call
0709 12:25:48.337444 (+967272us) spinlock_profiling.cc:243] Waited 702 ms on lock 0x7b54000267d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:48.337598 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1615690624}
I20260709 12:25:48.339141  8693 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:48.334076  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:48.322824  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:48.316569  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.005s	sys 0.013s
I20260709 12:25:48.333091  5554 rpcz_store.cc:276] 0709 12:25:46.970487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:47.677649 (+707162us) service_pool.cc:224] Handling call
0709 12:25:47.677959 (+   310us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:47.678302 (+   343us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:47.678337 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:47.678895 (+   558us) log.cc:844] Serialized 52 byte log entry
0709 12:25:48.263243 (+584348us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:48.263415 (+   172us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:48.263569 (+   154us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:48.332419 (+ 68850us) raft_consensus.cc:1685] finished
0709 12:25:48.332436 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:48.332637 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10489,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":77,"thread_start_us":576357,"threads_started":1,"wal-append.queue_time_us":642566}
W20260709 12:25:48.345891  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.005s	sys 0.013s
I20260709 12:25:48.310204  6273 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:48.342090  5702 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:48.351440  5705 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:48.317426  8780 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:48.352294  8807 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:48.353467  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:48.355247  8777 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:48.311154  8904 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:48.373071  8711 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:25:48.375742  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 575755us
W20260709 12:25:48.385743  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 97751.3us
I20260709 12:25:48.390441  8760 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.390787  8800 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.391193  5710 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:48.391162  8760 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:48.391507  8800 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:48.394273  8760 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:48.394577  8800 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:48.398363  8788 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:48.400871  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:48.402963  8751 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:48.411908  8751 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.412396  8751 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:48.404572  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:48.404847  8711 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:48.405829  8860 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.219s	user 0.014s	sys 0.000s
I20260709 12:25:48.373072  8911 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:48.390568  8787 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:48.385337  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:48.408653  5556 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:25:48.396170  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:48.443194  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:48.444976  8711 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:48.446498  8338 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.851s	user 0.010s	sys 0.015s
I20260709 12:25:48.447275  8787 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:48.452860  8787 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:48.448637  8866 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:48.453150  8554 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:48.448906  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:48.454290  8811 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.965s	user 0.015s	sys 0.002s
I20260709 12:25:48.454495  8911 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:48.455009  8811 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:48.455878  5555 raft_consensus.cc:2410] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 12:25:48.449137  5707 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
I20260709 12:25:48.454223  5706 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:25:48.458606  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 72044.8us
I20260709 12:25:48.449997  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:48.452957  8726 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:48.458081  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:48.466121  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:48.457864  8506 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.614s	user 0.010s	sys 0.002s
I20260709 12:25:48.473912  8506 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:48.474740  8726 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:25:48.474175  8902 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
W20260709 12:25:48.470569  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 74515.9us
W20260709 12:25:48.475934  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 57481.3us
I20260709 12:25:48.476438  6274 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:48.496871  8811 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:48.497480  8726 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:48.502468  8726 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:48.505824  6280 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:48.508643  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:48.510215  8695 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:48.481633  8866 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.514459  8866 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:48.517436  8878 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.539s	user 0.009s	sys 0.009s
I20260709 12:25:48.517562  8933 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:48.523892  8566 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:25:48.497982  8934 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:25:48.533603  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 155132us
I20260709 12:25:48.487215  8911 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:25:48.540444  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52059us
W20260709 12:25:48.516192  8779 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.547s	user 0.008s	sys 0.006s
I20260709 12:25:48.552230  5707 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:25:48.531424  8566 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:48.548955  8726 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:48.552755  8911 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.580775  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:48.636904  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:48.658602  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.587484  8911 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:48.745152  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:48.742635  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 172804us
I20260709 12:25:48.750993  5707 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:48.751639  5705 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:48.752683  5559 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:48.753093  5559 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:48.750660  5553 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:48.756246  5708 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:48.579032  8448 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:48.586706  8726 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:48.737646  8347 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:48.785980  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 277518us
I20260709 12:25:48.753513  8811 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:48.769073  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.751127  8787 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:48.786085  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:48.579175  8779 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:48.795441  5561 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 5: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:48.805207  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:48.818221  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:48.790105  5705 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
W20260709 12:25:48.880492  5455 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 274) took 29950 ms (client timeout 15000 ms). Trace:
W20260709 12:25:48.923307  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 134058us
I20260709 12:25:48.826850  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:48.821106  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:48.792318  8860 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:48.846812  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:48.940284  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 381933us
I20260709 12:25:48.953871  5557 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:48.869508  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:49.033649  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:48.999007  5558 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.661s	user 0.010s	sys 0.003s
I20260709 12:25:49.071769  5558 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
W20260709 12:25:48.996443  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45445us
W20260709 12:25:48.955457  8956 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
W20260709 12:25:48.878954  8949 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:25:48.954057  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:49.000588  5707 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
W20260709 12:25:49.113086  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.991448  5553 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:48.797853  8347 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.023685  5559 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:48.879961  8954 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:25:48.874315  8078 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:48.878023  8448 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:49.038774  8338 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:48.987493  8788 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.588s	user 0.015s	sys 0.000s
W20260709 12:25:48.880846  8950 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260709 12:25:49.123587  5705 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:48.826606  8943 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:49.152803  8855 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:49.130484  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 178254us
I20260709 12:25:49.145738  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:49.135380  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 317240us
I20260709 12:25:49.103176  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:49.038601  8695 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:48.829605  8866 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:49.146509  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:49.108485  8854 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.813s	user 0.003s	sys 0.014s
W20260709 12:25:48.956737  8828 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.997s	user 0.010s	sys 0.009s
I20260709 12:25:49.148869  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:49.001204  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:48.888262  8751 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:49.173132  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:48.953064  8760 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.197s
I20260709 12:25:48.932794  8787 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.183316  6280 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:49.191651  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:48.846045  8811 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:49.198361  5559 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:49.141444  8078 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:49.191596  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 91931.4us
W20260709 12:25:49.207088  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.793367  8934 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:49.212316  8935 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 12:25:49.163043  8788 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:49.124439  8711 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.077s
I20260709 12:25:49.215409  5552 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:49.221661  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:49.223490  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:49.226953  8904 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.837s	user 0.010s	sys 0.000s
I20260709 12:25:49.228204  8967 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:25:49.232492  6280 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:48.875103  8566 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:49.237883  6274 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:25:49.250995  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:49.240530  8904 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.251874  6277 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:49.252282  8078 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.253407  8078 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:49.253564  8904 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:49.239494  5707 raft_consensus.cc:2376] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7. Re-sending same reply.
I20260709 12:25:49.220163  8828 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:49.241616  8566 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:49.185537  8854 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:49.305717  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 211477us
I20260709 12:25:49.287719  6279 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:49.257808  8864 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:49.342796  8864 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.345885  8864 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:49.331317  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:49.331676  8856 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.332700  8737 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:49.230381  8695 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:49.332978  6278 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:25:49.334435  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2004) took 1983 ms. Trace:
I20260709 12:25:49.365032  5708 rpcz_store.cc:276] 0709 12:25:47.350562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:47.415438 (+ 64876us) service_pool.cc:224] Handling call
0709 12:25:47.441699 (+ 26261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:48.761002 (+1319303us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:49.334170 (+573168us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002d23d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:49.334371 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2342582528}
I20260709 12:25:49.366370  8720 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:49.275359  8554 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:49.366971  8695 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:48.932925  5455 rpcz_store.cc:269] 0709 12:25:18.930044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.970566 (+ 40522us) service_pool.cc:224] Handling call
0709 12:25:48.880076 (+29909510us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:43.912957 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:43.913785 (+   828us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:43.913854 (+    69us) write_op.cc:435] Acquired schema lock
0709 12:25:43.913881 (+    27us) tablet.cc:662] Decoding operations
0709 12:25:43.915322 (+  1441us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:43.915440 (+   118us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:43.915481 (+    41us) tablet.cc:685] Acquiring locks for 12 operations
0709 12:25:44.452210 (+536729us) tablet.cc:701] Row locks acquired
0709 12:25:44.467783 (+ 15573us) write_op.cc:260] PREPARE: finished
0709 12:25:44.508158 (+ 40375us) write_op.cc:270] Start()
0709 12:25:44.508922 (+   764us) write_op.cc:276] Timestamp: P: 1783599944507707 usec, L: 0
0709 12:25:44.509005 (+    83us) op_driver.cc:348] REPLICATION: starting
0709 12:25:44.527475 (+ 18470us) log.cc:844] Serialized 4723 byte log entry
0709 12:25:45.673624 (+1146149us) op_driver.cc:464] REPLICATION: finished
0709 12:25:46.096214 (+422590us) write_op.cc:301] APPLY: starting
0709 12:25:46.096323 (+   109us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:46.145368 (+ 49045us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:46.145565 (+   197us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:46.151269 (+  5704us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:46.151711 (+   442us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0709 12:25:46.151767 (+    56us) write_op.cc:312] APPLY: finished
0709 12:25:46.152257 (+   490us) log.cc:844] Serialized 89 byte log entry
0709 12:25:46.152528 (+   271us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:46.152703 (+   175us) write_op.cc:454] Released schema lock
0709 12:25:46.152779 (+    76us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:46.182522 (+ 29743us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:46.184740 (+  2218us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":41,"child_traces":[["op",{"apply.queue_time_us":402816,"apply.run_cpu_time_us":5011,"apply.run_wall_time_us":65240,"num_ops":12,"prepare.queue_time_us":690888,"prepare.run_cpu_time_us":36692,"prepare.run_wall_time_us":921940,"raft.queue_time_us":426329,"raft.run_cpu_time_us":47128,"raft.run_wall_time_us":185201,"replication_time_us":1164372,"thread_start_us":1009493,"threads_started":4,"wal-append.queue_time_us":470995,"wal-append.run_cpu_time_us":90986,"wal-append.run_wall_time_us":2226506}]]}
W20260709 12:25:49.349622  8933 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.831s	user 0.012s	sys 0.003s
I20260709 12:25:49.372470  8788 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:49.373134  8788 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.374297  8903 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:49.372483  8821 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:49.375092  8821 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:49.375325  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1946) took 1051 ms. Trace:
I20260709 12:25:49.375027  8903 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:49.215128  8338 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:49.348946  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 60770.6us
W20260709 12:25:49.358256  8924 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:25:49.358865  8751 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:49.375751  5702 rpcz_store.cc:276] 0709 12:25:48.323653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.325452 (+  1799us) service_pool.cc:224] Handling call
0709 12:25:49.375151 (+1049699us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b54001a8954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:49.375267 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35070208}
I20260709 12:25:49.287729  8800 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:49.314926  8726 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:49.326239  8804 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:49.411104  5704 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:49.411978  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:25:49.405742  8777 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.050s	user 0.011s	sys 0.007s
I20260709 12:25:49.406358  6276 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:49.375710  8788 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:49.376250  8866 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.389809  8737 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.389479  8554 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:49.449208  5553 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260709 12:25:49.470813  8684 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.563s	user 0.008s	sys 0.008s
I20260709 12:25:49.470702  8800 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.472164  5556 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:49.414575  8864 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:49.376169  8821 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:49.484156  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 72279.6us
I20260709 12:25:49.425529  8779 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:49.486179  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.464550  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:49.396867  8856 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:49.523902  5632 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:49.525262  8856 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:49.477895  8804 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.504101  8800 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:49.527778  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.528245  8804 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:49.528646  5630 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 6: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:49.528964  5635 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:49.525271  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.530222  5628 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:49.530349  8821 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:49.530526  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.425539  8968 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:49.531558  5632 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:49.532002  8968 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:49.533013  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.516484  8348 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.533715  8348 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:49.534271  8968 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:49.535447  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:49.535627  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1820) took 1236 ms. Trace:
I20260709 12:25:49.504011  8903 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:49.535979  5561 rpcz_store.cc:276] 0709 12:25:48.300636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.301374 (+   738us) service_pool.cc:224] Handling call
0709 12:25:49.535442 (+1234068us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b5400082354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:49.535572 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1114030976}
I20260709 12:25:49.522631  8371 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.537096  5630 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 12:25:49.537467  5630 pending_rounds.cc:85] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 4
I20260709 12:25:49.537241  8371 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.537716  5630 pending_rounds.cc:107] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 12:25:49.537994  5630 raft_consensus.cc:2889] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:49.522858  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:49.491050  8780 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.138s	user 0.015s	sys 0.003s
I20260709 12:25:49.517529  8854 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:49.533128  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.517396  8554 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.493046  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:49.525601  8338 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:49.504485  8858 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:49.578926  5632 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
W20260709 12:25:49.561951  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 424997us
W20260709 12:25:49.584139  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.585490  5706 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.106s	user 0.013s	sys 0.000s
I20260709 12:25:49.585834  5706 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:49.586243  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2011) took 1138 ms. Trace:
I20260709 12:25:49.587862  8371 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:49.588953  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.590706  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:49.591243  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 1.35111e+06us
I20260709 12:25:49.591693  5630 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:25:49.593546  8348 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:49.594765  8787 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:49.587754  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.498677  8967 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.596501  5628 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:25:49.596696  8967 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:49.599004  8967 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:49.599519  8967 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:49.458381  8866 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:49.600908  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.601755  5636 raft_consensus.cc:2410] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:25:49.602386  8866 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:49.483006  8737 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:49.606185  8737 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:49.614259  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 539) took 1083 ms. Trace:
I20260709 12:25:49.586488  5706 rpcz_store.cc:276] 0709 12:25:48.447902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.448501 (+   599us) service_pool.cc:224] Handling call
0709 12:25:49.586196 (+1137695us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13568}
I20260709 12:25:49.592029  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:49.490547  8695 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:49.631582  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:49.631728  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:49.614830  5554 rpcz_store.cc:276] 0709 12:25:48.530710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.530831 (+   121us) service_pool.cc:224] Handling call
0709 12:25:48.564337 (+ 33506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:48.758754 (+194417us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:25:49.061018 (+302264us) log.cc:844] Serialized 14 byte log entry
0709 12:25:49.613784 (+552766us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:49.613855 (+    71us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:49.613925 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:49.614015 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:49.614179 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5124,"prepare.queue_time_us":38930,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":4078,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":28188,"thread_start_us":728130,"threads_started":2,"wal-append.queue_time_us":76274}
I20260709 12:25:49.683777  8787 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:25:49.686717  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.689615  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 57963.6us
I20260709 12:25:49.690613  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:49.705215  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:49.656852  8911 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.893s	user 0.009s	sys 0.004s
I20260709 12:25:49.710101  5559 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
W20260709 12:25:49.685420  8855 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.532s	user 0.006s	sys 0.006s
I20260709 12:25:49.710824  5559 pending_rounds.cc:85] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 7
I20260709 12:25:49.722676  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.685274  8779 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:25:49.565845  8780 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.629861  8347 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:49.721844  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.742479  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 249521us
W20260709 12:25:49.744421  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 152459us
I20260709 12:25:49.650684  8854 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.070s
I20260709 12:25:49.706301  8348 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.746379  5559 pending_rounds.cc:107] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:25:49.758591  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2014) took 1279 ms. Trace:
I20260709 12:25:49.798367  5559 raft_consensus.cc:2889] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:49.756027  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.751685  5634 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: []
W20260709 12:25:49.800009  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.761509  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.801580  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.801795  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:25:49.721704  5709 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.833s	user 0.014s	sys 0.030s
W20260709 12:25:49.811316  5709 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.923s	user 0.014s	sys 0.030s
I20260709 12:25:49.751092  8780 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:49.742182  8778 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.006s	sys 0.025s
I20260709 12:25:49.722261  8866 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.721565  8855 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:49.828570  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1824) took 1253 ms. Trace:
I20260709 12:25:49.828848  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:49.760862  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:49.829005  5559 rpcz_store.cc:276] 0709 12:25:48.575659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.575830 (+   171us) service_pool.cc:224] Handling call
0709 12:25:48.738128 (+162298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:49.828270 (+1090142us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:49.828511 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:49.723124  8503 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.391s	user 0.017s	sys 0.004s
I20260709 12:25:49.719509  8967 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:49.719642  8970 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:49.721366  8856 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:25:49.842425  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:49.798583  5701 rpcz_store.cc:276] 0709 12:25:48.478851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.479010 (+   159us) service_pool.cc:224] Handling call
0709 12:25:48.492584 (+ 13574us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:48.499513 (+  6929us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:48.499557 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:48.500238 (+   681us) log.cc:844] Serialized 26 byte log entry
0709 12:25:48.751206 (+250968us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:48.751292 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:48.751476 (+   184us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:49.758363 (+1006887us) raft_consensus.cc:1685] finished
0709 12:25:49.758390 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:49.758521 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11553,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":128,"spinlock_wait_cycles":48000,"thread_start_us":240477,"threads_started":1,"wal-append.queue_time_us":1255871}
W20260709 12:25:49.813596  5555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1818) took 2275 ms (client timeout 1629 ms). Trace:
I20260709 12:25:49.745831  8779 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:49.815925  8695 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:49.831673  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:49.832954  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2012) took 1384 ms. Trace:
W20260709 12:25:49.752485  8946 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.028s	sys 0.053s
W20260709 12:25:49.835419  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.838090  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 77356.4us
I20260709 12:25:49.862270  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:25:49.828246  8778 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.006s	sys 0.035s
I20260709 12:25:49.719127  8911 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:49.761207  8566 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:49.844890  5555 rpcz_store.cc:269] 0709 12:25:47.537866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:47.555844 (+ 17978us) service_pool.cc:224] Handling call
0709 12:25:49.798600 (+2242756us) spinlock_profiling.cc:243] Waited 787 ms on lock 0x7b54002279d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:49.813522 (+ 14922us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1810266496}
I20260709 12:25:49.864952  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:49.859429  8804 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:49.859215  8800 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:25:49.848596  8760 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.017s	sys 0.030s
I20260709 12:25:49.859441  8779 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:49.868942  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:49.813848  8348 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.860682  8695 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:49.847078  5709 rpcz_store.cc:276] 0709 12:25:48.448212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.460171 (+ 11959us) service_pool.cc:224] Handling call
0709 12:25:48.494292 (+ 34121us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:48.738464 (+244172us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:25:48.786250 (+ 47786us) log.cc:844] Serialized 14 byte log entry
0709 12:25:49.832397 (+1046147us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:49.832481 (+    84us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:49.832603 (+   122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:49.832730 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:49.832883 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32774,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":11567,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":5992,"thread_start_us":928229,"threads_started":1}
I20260709 12:25:49.864039  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:49.865015  5599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.522s (SENT) [suppressed 30 similar messages]
I20260709 12:25:49.846455  8856 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:49.836615  8970 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:49.873696  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:49.889295  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:49.861569  8946 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.037s	sys 0.054s
I20260709 12:25:49.865595  8347 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.873229  8787 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:49.889954  8904 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:49.898188  5557 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:49.898757  5561 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:25:49.898643  5674 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.006s	sys 0.036s
W20260709 12:25:49.927960  5674 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.007s	sys 0.036s
I20260709 12:25:49.874853  8804 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:49.889199  8800 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:25:49.900686  5602 connection.cc:256] RPC call timeout handler was delayed by -1.790905023999585s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19210000000000002s for a response
W20260709 12:25:49.932823  5602 connection.cc:256] RPC call timeout handler was delayed by -1.2627619019995109s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16290000000000002s for a response
I20260709 12:25:49.896914  8695 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:49.901736  8944 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.037s	sys 0.043s
W20260709 12:25:49.902082  8783 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.853s	user 0.000s	sys 0.045s
I20260709 12:25:49.834775  8903 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.056s
I20260709 12:25:49.909868  8911 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:49.889372  8828 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:49.875073  8866 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:49.956650  8828 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.958168  8866 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:49.916234  8948 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:49.958838  8828 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:49.958988  8948 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:49.959431  8933 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:49.960108  8948 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:49.960901  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:49.961599  5628 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:49.917903  5710 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:49.964984  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.965338  8821 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.965927  8821 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.966382  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:49.966825  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1828) took 1146 ms. Trace:
I20260709 12:25:49.967248  5556 rpcz_store.cc:276] 0709 12:25:48.820595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.820764 (+   169us) service_pool.cc:224] Handling call
0709 12:25:49.966662 (+1145898us) spinlock_profiling.cc:243] Waited 535 ms on lock 0x7b540044f554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:49.966765 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1230366080}
I20260709 12:25:49.967585  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:49.968685  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.969321  5632 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:49.934332  8800 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:49.970175  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.970772  5632 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:49.971594  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.972625  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:49.968241  5630 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:49.923360  8970 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:49.894724  8760 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.019s	sys 0.030s
W20260709 12:25:49.934798  8944 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.037s	sys 0.044s
I20260709 12:25:49.932655  8804 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:49.919114  8856 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:49.937577  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.942364  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:49.887552  8566 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.926410  8711 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:25:49.936014  8783 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.975s	user 0.000s	sys 0.045s
I20260709 12:25:49.934209  8347 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:49.949458  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:49.961633  8948 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:25:49.920441  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 56479us
I20260709 12:25:49.989799  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:49.990628  8788 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:25:49.934177  5552 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:25:49.924315  8960 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.591s	user 0.005s	sys 0.000s
I20260709 12:25:49.992592  8788 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:49.993032  8788 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:49.993439  8788 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:49.993620  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:49.994019  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:49.994355  5705 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:49.998492  8347 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:50.006572  5557 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
W20260709 12:25:49.980871  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 458157us
I20260709 12:25:49.983091  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:49.985046  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:49.929610  8787 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:49.988133  5702 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:50.011255  5558 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:25:50.011406  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.011806  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:50.012739  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 153250us
I20260709 12:25:50.044498  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.049743  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:50.050552  6279 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:50.052678  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.053072  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:50.053555  6278 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:50.053946  5705 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:50.056510  5706 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:50.060756  8760 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.061416  8760 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:50.063725  8760 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:25:50.064903  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:50.066540  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:50.069540  8348 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:50.072842  8944 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:50.074167  5703 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
W20260709 12:25:50.077250  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.988232  8711 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:50.083082  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:50.087996  5558 raft_consensus.cc:2376] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8. Re-sending same reply.
W20260709 12:25:50.088663  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 43809.4us
I20260709 12:25:50.088686  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:50.043390  8787 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:50.092725  5706 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:25:50.101770  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.103432  5628 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:50.106633  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:50.107381  5632 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:50.087201  8828 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.087790  8711 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:50.120024  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:50.117901  8828 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:50.126529  8828 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:50.126948  8811 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:50.128402  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:50.133713  8980 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.224s	user 0.001s	sys 0.000s
I20260709 12:25:50.133828  8779 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:50.135639  5710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 1956) took 1340 ms (client timeout 1529 ms). Trace:
W20260709 12:25:50.136111  5710 rpcz_store.cc:269] 0709 12:25:48.795319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.846533 (+ 51214us) service_pool.cc:224] Handling call
0709 12:25:50.135412 (+1288879us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b54003ad4d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:50.135571 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1908405504}
I20260709 12:25:50.113121  8855 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:25:50.142225  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:50.143667  5705 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:50.144572  5627 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 11: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:25:50.151355  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.153208  8695 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:50.154423  5554 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:50.155025  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:50.155926  5556 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:25:50.156453  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1833) took 1078 ms. Trace:
I20260709 12:25:50.205886  5632 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:50.205828  5553 rpcz_store.cc:276] 0709 12:25:49.077743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:49.203998 (+126255us) service_pool.cc:224] Handling call
0709 12:25:49.231265 (+ 27267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:49.450138 (+218873us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:50.156218 (+706080us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b54003c4bd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:50.156403 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":463916800}
I20260709 12:25:50.158021  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:50.185256  8821 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:50.121526  8983 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.163641  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:50.168295  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.167565  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.173974  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.174185  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:50.175884  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:50.217561  5559 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:50.168304  8903 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:25:50.176693  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:50.222358  5706 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 6: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:50.176822  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:50.177837  8979 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:25:50.179100  8982 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:25:50.172492  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:50.242563  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 79036.5us
I20260709 12:25:50.186062  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:25:50.188805  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.190415  5561 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:50.245595  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.144726  8904 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:50.194706  8230 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:25:50.250049  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:50.192737  5630 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
W20260709 12:25:50.202493  5524 connection.cc:256] RPC call timeout handler was delayed by -1.2409580290000122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1529s for a response
I20260709 12:25:50.119872  8554 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.207727  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:50.208231  5710 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:50.209002  6274 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:50.205214  5634 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:50.215188  6276 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:50.218114  5554 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:50.250335  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:50.255965  5552 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:50.200850  8856 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:50.220392  8983 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:50.260329  5555 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:50.221138  5557 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:50.248234  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:50.251657  5524 connection.cc:256] RPC call timeout handler was delayed by -1.2202580289999787s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1552s for a response
W20260709 12:25:50.264822  5524 connection.cc:256] RPC call timeout handler was delayed by -1.1878580289999263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
W20260709 12:25:50.265542  5524 connection.cc:256] RPC call timeout handler was delayed by -1.1563580289998754s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1623s for a response
I20260709 12:25:50.265683  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:50.265980  5524 connection.cc:256] RPC call timeout handler was delayed by -1.1248580289998245s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1658s for a response
I20260709 12:25:50.262295  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:50.267562  6277 raft_consensus.cc:2410] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:25:50.219377  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:50.253322  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.268517  5561 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:50.213145  8779 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:50.257423  5705 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 5: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:50.204799  8971 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:50.258886  5556 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:25:50.258764  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.259865  5553 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:50.251991  6279 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
W20260709 12:25:50.269990  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.273030  5554 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:25:50.245556  8811 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:25:50.270512  5602 leader_election.cc:336] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.122s (SENT)
I20260709 12:25:50.249584  8566 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:50.279291  5708 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 12:25:50.249491  8230 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:50.256709  8828 raft_consensus.cc:2764] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
I20260709 12:25:50.283129  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.284929  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:50.263414  8911 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:50.288028  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.289016  5554 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:50.291870  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:50.276207  8811 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:50.274842  8855 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:50.293984  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:50.297209  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.291091  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:50.298987  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:50.294054  8811 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:50.299782  5634 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:50.310685  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.311723  5554 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:25:50.302765  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 116830us
I20260709 12:25:50.278954  8904 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:50.313962  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.267059  8800 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:50.318010  5559 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:25:50.321787  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:50.333230  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62761.8us
I20260709 12:25:50.334342  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.335656  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.279245  8554 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:50.300869  8911 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:50.300534  8855 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:50.365746  8855 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:50.368537  8938 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:25:50.351845  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:50.347749  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.351845  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.353783  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:50.370711  5630 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:50.356979  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.371110  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:50.287796  8971 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:50.359403  8800 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:50.361335  8856 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:50.362506  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:50.364010  8695 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:50.364648  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.375412  5556 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:25:50.392431  5559 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:50.314767  8828 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20260709 12:25:50.402150  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:50.405282  5552 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 12:25:50.369369  8983 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:50.409020  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:50.412258  5628 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:50.412454  6277 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:50.424777  5561 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:25:50.427613  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:50.429225  8737 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:25:50.430135  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:50.430658  8969 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:25:50.432822  8911 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:50.437067  5629 pending_rounds.cc:85] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 5
I20260709 12:25:50.437428  5629 pending_rounds.cc:107] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:25:50.437709  5629 raft_consensus.cc:2889] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:50.439595  6275 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:25:50.439679  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.440194  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:50.440637  6272 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:50.440917  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.441565  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.442126  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:50.442549  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.443166  6271 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:25:50.446080  8971 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:50.446718  8971 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:50.447072  8971 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:50.446549  5456 catalog_manager.cc:5697] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:50.402365  8506 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
W20260709 12:25:50.449205  8986 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:25:50.424866  8975 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260709 12:25:50.452979  8348 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:25:50.455300  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:50.427917  8904 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:50.450301  5456 catalog_manager.cc:5697] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 1 to 7, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 7 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:50.458549  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 96121.4us
I20260709 12:25:50.459824  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:50.442704  6275 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:50.466827  5632 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:50.461098  5456 catalog_manager.cc:5697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.404606  8821 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:50.410409  8566 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.486696  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:50.487682  8983 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:50.324842  8230 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:50.473246  8778 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.007s	sys 0.019s
I20260709 12:25:50.447918  8971 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:50.500491  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:50.493875  5599 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.012s	sys 0.023s
I20260709 12:25:50.483333  8904 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:50.505890  5631 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:25:50.502928  8804 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.517s	user 0.007s	sys 0.004s
I20260709 12:25:50.506340  5553 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
W20260709 12:25:50.506634  5599 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.013s	sys 0.023s
W20260709 12:25:50.514109  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 649068us
I20260709 12:25:50.515285  6274 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:50.506961  5456 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.476869  8821 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:50.490572  8783 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:25:50.503949  8778 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.007s	sys 0.020s
I20260709 12:25:50.523442  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:50.529939  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.522069  8804 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:50.509227  8983 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:50.532859  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:50.537233  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.545997  5705 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:50.537691  6274 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:50.536525  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:50.532725  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:50.543596  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:50.510756  8904 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:50.485263  8566 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:50.496814  8760 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260709 12:25:50.548025  5707 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:50.550622  8554 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:50.551503  8821 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:50.573455  8821 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:50.573587  8804 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:50.574045  8566 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:50.574417  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:50.553198  8695 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:50.557536  6275 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:25:50.548130  5710 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:50.536423  5456 catalog_manager.cc:5697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 3 to 6, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 6 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.569147  5708 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:50.582875  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.583593  6275 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:50.584594  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.590360  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:50.591035  5706 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:50.592859  8778 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:50.593304  8778 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:50.594167  8778 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:50.599742  5633 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.015s	sys 0.022s
I20260709 12:25:50.533111  8903 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.570729  8904 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:50.576999  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:50.601611  5633 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.016s	sys 0.023s
W20260709 12:25:50.606076  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:50.608433  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.596798  5631 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
W20260709 12:25:50.609212  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.609246  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:50.609889  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:50.610677  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:50.580467  8695 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:50.615360  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:50.614961  8804 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:50.602440  8778 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:50.629244  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.629242  5456 catalog_manager.cc:5697] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:50.642570  5701 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.008s	sys 0.030s
W20260709 12:25:50.545816  5636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.022s	sys 0.037s
I20260709 12:25:50.598990  6274 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
W20260709 12:25:50.644974  5636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.022s	sys 0.037s
I20260709 12:25:50.635586  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:50.544566  8711 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:50.648566  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2075) took 1111 ms. Trace:
W20260709 12:25:50.644670  5701 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.009s	sys 0.030s
I20260709 12:25:50.640713  8800 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:50.612807  8554 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:50.635270  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:25:50.662768  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 403405us
I20260709 12:25:50.665118  5553 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:25:50.620918  8856 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:25:50.637176  8695 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:50.678817  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:50.639235  8804 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:50.680166  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.640535  8506 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:50.680645  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:50.683619  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48443.6us
I20260709 12:25:50.704252  5709 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 12:25:50.662257  5456 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.705013  5636 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:25:50.718308  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:50.718864  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:50.719362  5560 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:50.619918  8903 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:50.720170  5632 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 9 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:25:50.707966  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.723825  5561 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:25:50.724387  5631 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:25:50.724511  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.649130  5633 rpcz_store.cc:276] 0709 12:25:49.521237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:49.566735 (+ 45498us) service_pool.cc:224] Handling call
0709 12:25:49.603206 (+ 36471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:49.603499 (+   293us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:25:50.008851 (+405352us) log.cc:844] Serialized 14 byte log entry
0709 12:25:50.621212 (+612361us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:50.621283 (+    71us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:50.621353 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:50.621438 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:50.632578 (+ 11140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2078,"prepare.queue_time_us":158277,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":9601,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":8268,"thread_start_us":954229,"threads_started":2}
I20260709 12:25:50.749084  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:50.726013  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 581) took 1136 ms. Trace:
W20260709 12:25:50.750115  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.749142  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:50.707587  8804 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:50.761497  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 483957us
I20260709 12:25:50.762692  5456 catalog_manager.cc:5697] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: config changed from index 3 to 5, term changed from 4 to 6, VOTER f613ba929b1046d3b5aed909414a95bf (127.4.65.3) evicted. New cstate: current_term: 6 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:50.768266  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:50.786324  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:50.719267  8856 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:50.750082  5635 rpcz_store.cc:276] 0709 12:25:49.589932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:49.607153 (+ 17221us) service_pool.cc:224] Handling call
0709 12:25:49.684731 (+ 77578us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:49.684946 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:25:50.259275 (+574329us) log.cc:844] Serialized 14 byte log entry
0709 12:25:50.725719 (+466444us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:50.725767 (+    48us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:50.725812 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:50.725868 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:50.725965 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22990,"prepare.queue_time_us":63706,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":23387,"thread_start_us":945530,"threads_started":2}
I20260709 12:25:50.787431  5627 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
W20260709 12:25:50.760337  8788 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.766s	user 0.012s	sys 0.000s
I20260709 12:25:50.725572  8779 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:50.796371  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:50.805223  5560 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:50.807142  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:50.807363  6280 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.003s	sys 0.026s
I20260709 12:25:50.796751  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:50.760550  8970 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.771s	user 0.011s	sys 0.000s
W20260709 12:25:50.809582  6280 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.004s	sys 0.026s
I20260709 12:25:50.810535  6275 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:50.810441  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:50.801774  5456 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 4 to 8, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 8 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.813302  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:50.818279  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:50.817921  5456 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:50.807794  8948 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.801s	user 0.004s	sys 0.008s
W20260709 12:25:50.822904  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.753773  8811 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:50.823381  5456 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:50.829452  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.830300  6275 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:50.817844  8347 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:50.843701  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:50.822566  8933 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:50.845033  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:50.839694  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:50.845335  5456 catalog_manager.cc:5697] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.848598  5456 catalog_manager.cc:5697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.849452  8711 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:50.850143  8711 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:50.850600  8711 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:50.850384  5456 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 2 to 6, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 6 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:25:50.851517  8711 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:50.855413  6276 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:50.857695  8811 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:50.858469  8811 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.859007  8811 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:50.861207  8811 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:50.861717  8811 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:50.862207  8811 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:50.812117  8788 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:50.906168  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.907648  6275 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:50.838150  8783 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:50.817631  8970 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:50.904909  5633 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:50.868992  8998 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.844009  8981 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:25:50.839067  8347 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:50.876919  8903 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:50.876029  8983 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:25:50.915867  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 76268.9us
I20260709 12:25:50.864319  8811 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:50.844432  8566 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
W20260709 12:25:50.961254  5555 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.014s	sys 0.028s
W20260709 12:25:50.963117  5555 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.015s	sys 0.028s
I20260709 12:25:50.910413  8938 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.912879  8944 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.953138  5633 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:25:50.954200  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.952502  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:50.966203  5704 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:50.974819  6275 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:50.948990  8347 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:50.978291  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1749) took 1015 ms. Trace:
I20260709 12:25:50.931459  9005 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:25:50.989133  6272 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.006s	sys 0.030s
W20260709 12:25:50.994380  6272 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.006s	sys 0.030s
I20260709 12:25:50.987856  5630 rpcz_store.cc:276] 0709 12:25:49.962995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.412963 (+449968us) service_pool.cc:224] Handling call
0709 12:25:50.460783 (+ 47820us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:50.461026 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:25:50.461385 (+   359us) log.cc:844] Serialized 14 byte log entry
0709 12:25:50.977921 (+516536us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:50.977975 (+    54us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:50.978025 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:50.978110 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:50.978232 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":171,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"thread_start_us":489717,"threads_started":1}
W20260709 12:25:50.991891  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 167419us
I20260709 12:25:50.949090  9000 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:50.965559  8969 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.012235  5633 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
W20260709 12:25:51.003674  8855 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.021s	sys 0.013s
I20260709 12:25:50.994213  9005 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:51.036283  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:50.927929  8856 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:51.036146  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:50.989290  8998 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:51.037518  8855 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.022s	sys 0.014s
I20260709 12:25:50.978741  8938 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:51.013804  8800 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:51.010910  9004 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:25:51.080009  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.995321  8821 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:25:51.097630  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1748) took 1134 ms. Trace:
I20260709 12:25:50.988598  8944 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:51.098977  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62861.5us
I20260709 12:25:51.101992  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:51.050796  9005 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:51.103432  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2022) took 1162 ms. Trace:
I20260709 12:25:51.109537  5701 rpcz_store.cc:276] 0709 12:25:49.941314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:49.948172 (+  6858us) service_pool.cc:224] Handling call
0709 12:25:49.948523 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:49.948778 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:25:50.675644 (+726866us) log.cc:844] Serialized 14 byte log entry
0709 12:25:51.092769 (+417125us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:51.092840 (+    71us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:51.103106 (+ 10266us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:51.103214 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:51.103361 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":885664,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":7038,"thread_start_us":1076215,"threads_started":2}
W20260709 12:25:51.111919  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 296216us
I20260709 12:25:51.088743  8566 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:25:51.103909  8997 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.223s	user 0.001s	sys 0.000s
I20260709 12:25:51.119324  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:51.124599  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:51.119022  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:51.121212  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 111748us
I20260709 12:25:51.123521  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:51.098096  5634 rpcz_store.cc:276] 0709 12:25:49.962714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.394752 (+432038us) service_pool.cc:224] Handling call
0709 12:25:50.424856 (+ 30104us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:50.480754 (+ 55898us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:50.480807 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:50.490419 (+  9612us) log.cc:844] Serialized 26 byte log entry
0709 12:25:50.984034 (+493615us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:25:50.984201 (+   167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:50.984286 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:51.097392 (+113106us) raft_consensus.cc:1685] finished
0709 12:25:51.097412 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:51.097566 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8674,"raft.queue_time_us":8911,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":7942,"thread_start_us":444896,"threads_started":1,"wal-append.queue_time_us":572390}
I20260709 12:25:51.094338  9004 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:51.127251  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:51.130600  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.127154  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:51.133275  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:51.134917  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:51.135896  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:51.053319  9000 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260709 12:25:51.052544  8969 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:51.124787  8993 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.302s	user 0.000s	sys 0.001s
W20260709 12:25:51.154541  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.157400  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:51.122260  8983 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.158279  5634 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:51.115967  8821 consensus_queue.cc:1243] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 12:25:51.159509  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 288224us
I20260709 12:25:51.128015  8981 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:51.108106  8760 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 12:25:51.131947  9004 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:51.172081  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:51.154313  8566 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.180269  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:51.180871  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45044.9us
I20260709 12:25:51.161360  8711 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:51.186131  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:51.159458  8788 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:51.177192  9001 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.267s	user 0.000s	sys 0.001s
I20260709 12:25:51.194727  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:51.195024  9009 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:25:51.195861  8998 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:51.196419  8999 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 12:25:51.196491  8998 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:51.198556  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1828) took 1098 ms. Trace:
I20260709 12:25:51.206251  5558 rpcz_store.cc:276] 0709 12:25:50.100648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.102447 (+  1799us) service_pool.cc:224] Handling call
0709 12:25:50.115850 (+ 13403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:50.150589 (+ 34739us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:50.150634 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:50.183757 (+ 33123us) log.cc:844] Serialized 26 byte log entry
0709 12:25:50.639237 (+455480us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:50.639285 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:50.639356 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:51.198273 (+558917us) raft_consensus.cc:1685] finished
0709 12:25:51.198302 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:51.198477 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18024,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":36790,"thread_start_us":420535,"threads_started":1,"wal-append.queue_time_us":621694}
I20260709 12:25:51.178030  8903 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:25:51.225324  5701 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 12:25:51.210978  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.213040  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:51.167886  8983 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:51.185020  8970 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:51.197603  8821 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:51.192395  8711 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:25:51.301107  9008 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 12:25:51.192071  8566 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:51.303880  8566 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:51.304664  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:51.225495  9003 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
W20260709 12:25:51.297477  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 136455us
W20260709 12:25:51.298807  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 171734us
I20260709 12:25:51.289923  5601 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:51.227622  8760 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:51.304903  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:51.224596  8998 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:25:51.344568  5707 raft_consensus.cc:2376] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6. Re-sending same reply.
I20260709 12:25:51.225106  8506 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:51.334117  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:51.304852  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 164766us
I20260709 12:25:51.333739  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:51.347823  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 676) took 1257 ms. Trace:
I20260709 12:25:51.351526  6278 rpcz_store.cc:276] 0709 12:25:50.076512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.100651 (+ 24139us) service_pool.cc:224] Handling call
0709 12:25:50.111430 (+ 10779us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:50.139175 (+ 27745us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:50.139221 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:50.168778 (+ 29557us) log.cc:844] Serialized 52 byte log entry
0709 12:25:50.648112 (+479334us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:50.648150 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:50.648217 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:51.333565 (+685348us) raft_consensus.cc:1685] finished
0709 12:25:51.333585 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:51.333729 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5010,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":106,"thread_start_us":445900,"threads_started":1,"wal-append.queue_time_us":800206}
W20260709 12:25:51.227082  9014 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:25:51.353837  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:51.355878  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:51.357973  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:51.354982  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:51.358626  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 547) took 1203 ms. Trace:
I20260709 12:25:51.358891  5630 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:51.360190  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:51.360954  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:51.359043  5555 rpcz_store.cc:276] 0709 12:25:50.155701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.344484 (+188783us) service_pool.cc:224] Handling call
0709 12:25:50.381140 (+ 36656us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:50.381527 (+   387us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:25:50.971180 (+589653us) log.cc:844] Serialized 14 byte log entry
0709 12:25:51.358297 (+387117us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:51.358349 (+    52us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:51.358401 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:51.358465 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:51.358569 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22110,"prepare.queue_time_us":733399,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":3300,"thread_start_us":916141,"threads_started":3}
I20260709 12:25:51.362884  6278 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:51.365065  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:51.226303  9006 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:25:51.371470  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:51.198194  9023 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:51.224814  8348 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:51.226404  8788 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.376883  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:51.294863  8903 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.378892  9023 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:51.378933  6277 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:51.366696  6278 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:25:51.380837  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:51.380983  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 171957us
I20260709 12:25:51.382098  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:51.404610  6279 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:51.350812  8760 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.385954  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 656) took 1214 ms. Trace:
I20260709 12:25:51.416682  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:51.406276  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:51.305778  8821 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:51.368105  8506 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.297449  8711 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:51.409451  8788 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:51.410506  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:51.420508  6277 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:51.420859  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:51.422005  6278 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:25:51.400924  8903 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:51.414778  8969 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:51.285669  8804 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:51.415704  9017 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
I20260709 12:25:51.417934  6279 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
W20260709 12:25:51.418716  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61102.3us
W20260709 12:25:51.413545  9018 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.173s	user 0.000s	sys 0.000s
I20260709 12:25:51.408212  9035 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:51.416889  6280 rpcz_store.cc:276] 0709 12:25:50.184572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.190274 (+  5702us) service_pool.cc:224] Handling call
0709 12:25:50.190825 (+   551us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:50.202644 (+ 11819us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:50.202680 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:50.259573 (+ 56893us) log.cc:844] Serialized 52 byte log entry
0709 12:25:50.809999 (+550426us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:50.810048 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:50.812758 (+  2710us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:51.385490 (+572732us) raft_consensus.cc:1685] finished
0709 12:25:51.385514 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:51.385845 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":120,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":10939,"thread_start_us":515225,"threads_started":1,"wal-append.queue_time_us":779768}
I20260709 12:25:51.381989  8230 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:25:51.477350  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 126583us
I20260709 12:25:51.408883  9023 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:51.424189  5704 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
W20260709 12:25:51.512745  8554 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.827s	user 0.006s	sys 0.005s
I20260709 12:25:51.473666  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:51.209960  8800 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:51.419749  8760 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:51.488762  9000 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:25:51.489740  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 374647us
W20260709 12:25:51.470152  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 111842us
W20260709 12:25:51.509238  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 106031us
I20260709 12:25:51.434432  8506 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:51.513237  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:51.401487  8348 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:51.478081  9035 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.471654  8938 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:51.558306  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 100849us
W20260709 12:25:51.571220  9036 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:25:51.462750  9038 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:25:51.564483  8778 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.917s	user 0.011s	sys 0.002s
I20260709 12:25:51.519635  9020 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:51.420322  8981 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:51.546731  8821 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:51.584481  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:51.832845  6280 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:51.706506  6277 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:51.817728  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 672) took 1378 ms. Trace:
W20260709 12:25:51.665722  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 259509us
I20260709 12:25:51.703024  8804 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:51.837020  9035 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:25:51.834120  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:51.837103  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.834538  6271 rpcz_store.cc:276] 0709 12:25:50.439533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.443807 (+  4274us) service_pool.cc:224] Handling call
0709 12:25:50.444196 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:50.466306 (+ 22110us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:50.466347 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:50.495573 (+ 29226us) log.cc:844] Serialized 26 byte log entry
0709 12:25:50.952986 (+457413us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:51.119466 (+166480us) log.cc:844] Serialized 14 byte log entry
0709 12:25:51.119663 (+   197us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:51.119737 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:51.817458 (+697721us) raft_consensus.cc:1685] finished
0709 12:25:51.817490 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:51.817660 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":175298,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":3499,"raft.queue_time_us":48072,"spinlock_wait_cycles":13798784,"thread_start_us":504226,"threads_started":2,"wal-append.queue_time_us":608784}
W20260709 12:25:51.829442  5634 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.662s	user 0.011s	sys 0.000s
W20260709 12:25:51.667142  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.839871  5634 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
W20260709 12:25:51.840862  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:51.841416  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:51.573189  9025 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:25:51.843071  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:51.843979  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:51.726562  8347 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.721s	user 0.010s	sys 0.000s
I20260709 12:25:51.844877  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:51.846134  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 367300us
I20260709 12:25:51.846276  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:51.629278  9043 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:25:51.569159  8944 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:51.858489  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:51.752848  9029 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.589151  8554 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:51.870066  6271 raft_consensus.cc:2376] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8. Re-sending same reply.
W20260709 12:25:51.669401  8787 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.548s	user 0.011s	sys 0.004s
I20260709 12:25:51.815871  8969 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:51.802778  9051 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:25:51.876911  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:51.728889  8983 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:51.725294  9042 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.887676  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:51.838047  9020 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:51.835147  9038 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.874341  8347 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:51.911891  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:51.891783  9029 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:51.912791  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 52499.5us
I20260709 12:25:51.913026  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 677) took 1234 ms. Trace:
I20260709 12:25:51.924242  6277 raft_consensus.cc:2376] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11. Re-sending same reply.
I20260709 12:25:51.881237  8778 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:25:51.923098  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:51.916421  9050 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.084s
I20260709 12:25:51.918838  8787 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:51.961282  5706 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:51.963346  9030 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:25:51.963654  8996 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.134s	user 0.003s	sys 0.001s
I20260709 12:25:51.922569  9000 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:51.965392  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2020) took 2025 ms. Trace:
W20260709 12:25:51.966213  9031 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.393s	user 0.000s	sys 0.001s
I20260709 12:25:51.966120  5709 rpcz_store.cc:276] 0709 12:25:49.940543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:49.974734 (+ 34191us) service_pool.cc:224] Handling call
0709 12:25:49.981953 (+  7219us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:50.720112 (+738159us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:51.965145 (+1245033us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b54002f5b54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:51.965334 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1628176768}
I20260709 12:25:51.968770  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:51.918292  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 73458.2us
I20260709 12:25:51.970831  6271 raft_consensus.cc:2410] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:25:51.971434  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:51.877621  8788 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:51.978579  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:51.822507  9026 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.106s
I20260709 12:25:51.979259  5561 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:51.977790  6274 rpcz_store.cc:276] 0709 12:25:50.678209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.678640 (+   431us) service_pool.cc:224] Handling call
0709 12:25:50.678906 (+   266us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:50.702166 (+ 23260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:50.702213 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:50.722758 (+ 20545us) log.cc:844] Serialized 26 byte log entry
0709 12:25:51.162031 (+439273us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:51.162149 (+   118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:51.162906 (+   757us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:51.912777 (+749871us) raft_consensus.cc:1685] finished
0709 12:25:51.912803 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:51.912961 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2911,"raft.queue_time_us":27812,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":5325,"thread_start_us":366218,"threads_started":1,"wal-append.queue_time_us":533325}
I20260709 12:25:51.968101  9045 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:51.924515  8938 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:52.017315  5709 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:51.997531  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:51.918061  8821 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:25:51.917588  8998 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.043s	sys 0.104s
I20260709 12:25:52.030148  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:52.031037  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:52.031675  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:52.018023  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:52.027906  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:52.032891  5553 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:25:52.033188  5555 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:52.028740  5633 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:25:52.026273  5554 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:52.027519  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:51.963984  9030 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:52.046092  5560 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:52.047550  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:51.924808  8969 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:51.936980  9020 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:52.048439  5707 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 9: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:51.917009  9049 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:52.027892  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:51.937742  9038 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:51.979319  9050 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:52.062377  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:52.066722  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:52.094224  5709 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:52.068593  6274 raft_consensus.cc:2376] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10. Re-sending same reply.
I20260709 12:25:52.036229  8938 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:52.032794  9000 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:51.939245  9042 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:51.961390  8348 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:52.108196  5554 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:52.086321  9030 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:52.087045  8711 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.617s	user 0.009s	sys 0.000s
I20260709 12:25:52.090342  9035 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:25:52.037380  8998 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.044s	sys 0.105s
I20260709 12:25:52.123574  5558 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:52.092134  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:52.099774  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:51.985033  8778 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:52.102944  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:52.081367  8695 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.359s	user 0.020s	sys 0.002s
I20260709 12:25:52.125893  6274 raft_consensus.cc:2410] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20260709 12:25:52.076162  8230 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.563s	user 0.001s	sys 0.009s
I20260709 12:25:52.106504  6280 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:52.017774  9055 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:25:52.096763  8856 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.009s	sys 0.035s
I20260709 12:25:52.049691  8787 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:52.114718  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:52.115376  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:52.115700  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 227024us
I20260709 12:25:52.120157  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:52.122772  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 669) took 1684 ms. Trace:
I20260709 12:25:52.137431  5709 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:52.139981  6280 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:52.140192  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:52.140775  5705 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:52.130450  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:52.130740  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:52.142519  5558 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:52.142768  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:52.145782  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:52.147071  6271 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
W20260709 12:25:52.133793  8856 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.009s	sys 0.035s
I20260709 12:25:52.151965  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 681) took 1459 ms. Trace:
I20260709 12:25:52.118897  9050 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:52.155287  6280 rpcz_store.cc:276] 0709 12:25:50.681747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.139394 (+1457647us) service_pool.cc:224] Handling call
0709 12:25:52.139742 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:52.140411 (+   669us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.140611 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:52.156306  6280 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:52.124099  8998 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:52.157502  6274 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:52.160049  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 678) took 1479 ms. Trace:
I20260709 12:25:52.160485  6274 rpcz_store.cc:276] 0709 12:25:50.680800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.136220 (+1455420us) service_pool.cc:224] Handling call
0709 12:25:52.157114 (+ 20894us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:52.159750 (+  2636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.159984 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:52.161839  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:52.165241  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 676) took 1196 ms. Trace:
I20260709 12:25:52.165638  6277 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: []
I20260709 12:25:52.166733  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 685) took 1314 ms. Trace:
I20260709 12:25:52.137359  6272 rpcz_store.cc:276] 0709 12:25:50.438586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.442518 (+  3932us) service_pool.cc:224] Handling call
0709 12:25:50.442865 (+   347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:50.443271 (+   406us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:25:50.994883 (+551612us) log.cc:844] Serialized 14 byte log entry
0709 12:25:51.172171 (+177288us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:51.178679 (+  6508us) log.cc:844] Serialized 26 byte log entry
0709 12:25:51.179143 (+   464us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:51.183167 (+  4024us) log.cc:844] Serialized 14 byte log entry
0709 12:25:51.298300 (+115133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:51.298384 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:52.122379 (+823995us) raft_consensus.cc:1685] finished
0709 12:25:52.122413 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:52.122672 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":654598,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":104,"raft.queue_time_us":395064,"raft.run_cpu_time_us":366,"raft.run_wall_time_us":23481,"spinlock_wait_cycles":18176,"thread_start_us":802851,"threads_started":3,"wal-append.queue_time_us":369713}
I20260709 12:25:52.167913  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:52.168011  6280 rpcz_store.cc:276] 0709 12:25:50.962006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.155936 (+1193930us) service_pool.cc:224] Handling call
0709 12:25:52.156183 (+   247us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:52.156776 (+   593us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.158698 (+  1922us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:52.121575  8711 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:52.151165  8933 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:52.169983  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:52.173023  6271 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:25:52.167243  6277 rpcz_store.cc:276] 0709 12:25:50.852639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.150816 (+1298177us) service_pool.cc:224] Handling call
0709 12:25:52.165456 (+ 14640us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:52.166364 (+   908us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.166655 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:52.169723  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:52.067636  9045 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:52.128809  8695 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:52.178474  5558 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:52.181793  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:52.107816  9000 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:52.170432  9004 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.985s	user 0.006s	sys 0.006s
I20260709 12:25:52.185303  5553 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:25:52.189603  6277 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: []
W20260709 12:25:52.194947  8904 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.565s	user 0.016s	sys 0.000s
I20260709 12:25:52.195824  9046 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:52.189659  9032 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:25:52.218786  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63505.7us
W20260709 12:25:52.222060  9064 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:25:52.226500  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:52.211557  8760 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.568s	user 0.012s	sys 0.000s
I20260709 12:25:52.171972  8938 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:52.227229  5553 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:52.229310  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:52.178880  8778 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:52.241986  8969 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:52.196419  9004 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:52.242236  9060 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:52.228741  9032 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:52.243741  9000 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:52.252766  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:52.253804  9029 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:52.257027  9042 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:52.258558  9035 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:25:52.259672  8695 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:52.260809  8783 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:52.261184  8711 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:52.261744  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:52.262890  9004 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:52.250751  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:52.271997  9032 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:25:52.273721  8903 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.758s	user 0.014s	sys 0.006s
I20260709 12:25:52.275681  9062 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:25:52.301151  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.284080  9060 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.212646  8904 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:52.296890  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:52.298862  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48314.8us
I20260709 12:25:52.240168  8506 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:52.301805  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:52.238891  8760 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:52.303081  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.277981  5555 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:52.305501  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:52.296482  8783 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:52.313678  5701 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:52.239508  8998 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:52.306972  9023 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.781s	user 0.007s	sys 0.006s
I20260709 12:25:52.303370  9046 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:52.311322  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:52.303100  8695 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:52.340793  5634 raft_consensus.cc:2368] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8.
I20260709 12:25:52.232277  9050 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:52.349751  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:52.357914  9066 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:25:52.303228  8903 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:52.311322  9060 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:52.308892  9062 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:52.400653  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.463806  5709 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:52.359040  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:52.427425  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 127427us
W20260709 12:25:52.559404  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 297774us
I20260709 12:25:52.312913  8711 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:25:52.436136  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.558498  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:52.568246  5526 leader_election.cc:336] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.623s (SENT)
W20260709 12:25:52.570536  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:52.572176  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 213185us
I20260709 12:25:52.414173  8933 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:52.428052  9045 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:52.443401  8904 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:52.574326  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:52.365585  9063 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
I20260709 12:25:52.576824  5632 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:25:52.568992  8711 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:52.575385  9062 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:52.544916  9038 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:52.575677  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:52.578794  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:52.579335  5636 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:52.579640  9038 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:52.577374  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1774) took 1219 ms. Trace:
I20260709 12:25:52.582218  8783 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:52.577720  8711 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:52.584997  8783 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:52.582854  5632 rpcz_store.cc:276] 0709 12:25:51.358116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.574043 (+1215927us) service_pool.cc:224] Handling call
0709 12:25:52.577314 (+  3271us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:52.447633  9005 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.327s	user 0.002s	sys 0.012s
I20260709 12:25:52.599779  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:52.599121  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2038) took 1401 ms. Trace:
I20260709 12:25:52.598603  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:52.358767  8695 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:52.358680  9023 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:52.445461  9020 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:52.789891  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 693) took 1934 ms. Trace:
I20260709 12:25:52.933241  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:52.605875  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:52.368433  8800 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:52.365504  8828 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:52.617707  8566 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:52.935207  8695 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:52.935493  8566 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:52.646011  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:52.654031  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:52.614454  9005 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:52.937914  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1777) took 1577 ms. Trace:
I20260709 12:25:52.938539  5560 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: []
W20260709 12:25:52.940768  5704 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2035) took 2375 ms (client timeout 1885 ms). Trace:
W20260709 12:25:52.941274  5704 rpcz_store.cc:269] 0709 12:25:50.565203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.565375 (+   172us) service_pool.cc:224] Handling call
0709 12:25:52.940491 (+2375116us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b54002d0fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:52.940650 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":899556224}
I20260709 12:25:52.933732  6271 rpcz_store.cc:276] 0709 12:25:50.855013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.154130 (+1299117us) service_pool.cc:224] Handling call
0709 12:25:52.170744 (+ 16614us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:52.178139 (+  7395us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.789294 (+611155us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b54000b54d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:52.789764 (+   470us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4680960}
I20260709 12:25:52.934866  8828 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:52.464962  8506 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:52.611140  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2094) took 1254 ms. Trace:
I20260709 12:25:52.609109  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:52.597894  8787 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:52.649662  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:52.599677  9032 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:52.967324  5627 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:25:52.968518  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 362737us
W20260709 12:25:52.970659  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.605754  9046 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:52.600029  9039 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.347s
I20260709 12:25:52.644387  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:52.938362  5634 rpcz_store.cc:276] 0709 12:25:51.360632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.653693 (+1293061us) service_pool.cc:224] Handling call
0709 12:25:52.937840 (+284147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:52.934494  8800 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:52.967460  5526 leader_election.cc:336] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.961s (SENT)
W20260709 12:25:52.971568  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.981814  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:52.557855  8760 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:52.933815  8903 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:52.947633  9020 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:52.966748  5633 rpcz_store.cc:276] 0709 12:25:51.357024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.361154 (+  4130us) service_pool.cc:224] Handling call
0709 12:25:51.361555 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:52.034004 (+672449us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.610672 (+576668us) spinlock_profiling.cc:243] Waited 656 ms on lock 0x7b540008bbd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:52.611034 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1509802752}
I20260709 12:25:52.992815  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:52.993453  5558 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:25:52.993700  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:52.993948  9032 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:52.993427  5633 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:25:52.994609  6274 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:52.994840  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1776) took 1634 ms. Trace:
I20260709 12:25:52.995321  5633 rpcz_store.cc:276] 0709 12:25:51.360191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.992870 (+1632679us) service_pool.cc:224] Handling call
0709 12:25:52.993145 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:52.994597 (+  1452us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.994790 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:52.975040  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.969703  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2095) took 1612 ms. Trace:
I20260709 12:25:52.978730  8711 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:52.948010  9005 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:52.608510  8904 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.982782  5628 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
W20260709 12:25:52.983376  5526 leader_election.cc:336] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 2.110s (SENT)
I20260709 12:25:52.986915  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:52.934175  5706 rpcz_store.cc:276] 0709 12:25:51.197980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.199296 (+  1316us) service_pool.cc:224] Handling call
0709 12:25:51.228241 (+ 28945us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:51.961807 (+733566us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.598890 (+637083us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b540033fc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:52.599062 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1685366656}
I20260709 12:25:52.963146  8975 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.988471  5634 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:25:53.013942  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1775) took 1655 ms. Trace:
I20260709 12:25:52.988703  9060 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:52.981838  8800 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:52.968940  8971 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:53.021209  8828 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.022830  8711 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:53.016511  8904 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:52.972775  9024 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:53.024616  9024 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.025134  9024 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:53.025568  8904 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:52.968202  8787 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:52.991231  5636 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:25:52.996702  5632 rpcz_store.cc:276] 0709 12:25:51.357457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.646384 (+1288927us) service_pool.cc:224] Handling call
0709 12:25:52.969632 (+323248us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:52.997283  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 433802us
I20260709 12:25:52.994297  5552 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:52.998818  5559 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 9. Current term is 10. Ops: []
W20260709 12:25:52.990607  8779 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.152s	user 0.007s	sys 0.007s
I20260709 12:25:53.002347  5557 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:53.037771  5635 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:53.002902  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:53.039109  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:52.991640  5627 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:25:53.040153  5627 raft_consensus.cc:606] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:25:53.040671  5627 raft_consensus.cc:614] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:25:53.004215  9011 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.776s	user 0.000s	sys 0.017s
I20260709 12:25:53.005142  5558 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:25:53.011495  5561 raft_consensus.cc:2410] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 10 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:25:53.011829  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1781) took 1178 ms. Trace:
I20260709 12:25:53.048316  8904 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.048496  9039 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:53.048686  5628 rpcz_store.cc:276] 0709 12:25:51.833610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.974484 (+1140874us) service_pool.cc:224] Handling call
0709 12:25:53.011746 (+ 37262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:53.049343  8904 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.049300  9039 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.051797  9039 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:53.051860  8904 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:53.036836  8779 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.211s	user 0.007s	sys 0.007s
I20260709 12:25:53.028855  8787 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:53.052809  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 665) took 2615 ms. Trace:
I20260709 12:25:53.041548  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:53.014387  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:53.054764  5555 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:53.014285  5634 rpcz_store.cc:276] 0709 12:25:51.358403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.981376 (+1622973us) service_pool.cc:224] Handling call
0709 12:25:52.988082 (+  6706us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:53.013695 (+ 25613us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.013888 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:52.982456  8998 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.015707  5708 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:53.014469  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1778) took 1652 ms. Trace:
I20260709 12:25:53.056854  5633 rpcz_store.cc:276] 0709 12:25:51.361936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.996339 (+1634403us) service_pool.cc:224] Handling call
0709 12:25:53.013959 (+ 17620us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:53.014170 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:53.014194 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:53.014234 (+    40us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:53.014268 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.014320 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:53.014420 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:53.058173  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 452990us
I20260709 12:25:53.019785  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:53.059032  5553 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:53.059898  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:53.028753  9005 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:52.998143  8566 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.061012  8737 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:53.061295  8566 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:53.042397  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:53.062952  5632 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
W20260709 12:25:53.042766  9078 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:25:53.043118  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:53.070519  5636 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:53.044982  5631 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:52.980212  9046 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.016819  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:53.071399  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1779) took 1238 ms. Trace:
I20260709 12:25:53.066706  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:53.062196  9005 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:25:53.053246  6273 rpcz_store.cc:276] 0709 12:25:50.437097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.440455 (+  3358us) service_pool.cc:224] Handling call
0709 12:25:50.440765 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:50.441183 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:50.441211 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:50.441638 (+   427us) log.cc:844] Serialized 26 byte log entry
0709 12:25:50.820532 (+378894us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:50.820575 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:50.821318 (+   743us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:53.003593 (+2182275us) spinlock_profiling.cc:243] Waited 9.39 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:53.052437 (+ 48844us) raft_consensus.cc:1685] finished
0709 12:25:53.052460 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:53.052729 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":237,"raft.queue_time_us":1948,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":98,"spinlock_wait_cycles":21602048,"thread_start_us":378528,"threads_started":1,"wal-append.queue_time_us":713138}
I20260709 12:25:53.071797  5631 rpcz_store.cc:276] 0709 12:25:51.832983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.044457 (+1211474us) service_pool.cc:224] Handling call
0709 12:25:53.044807 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:53.071138 (+ 26331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.071343 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:53.064885  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:53.073594  6276 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 671) took 2624 ms (client timeout 1673 ms). Trace:
I20260709 12:25:53.083614  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2097) took 1064 ms. Trace:
I20260709 12:25:53.084069  5634 rpcz_store.cc:276] 0709 12:25:52.018886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.063410 (+1044524us) service_pool.cc:224] Handling call
0709 12:25:53.083110 (+ 19700us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:53.083297 (+   187us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:53.083322 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:53.083363 (+    41us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:53.083402 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.083456 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:53.083557 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:53.080664  6276 rpcz_store.cc:269] 0709 12:25:50.439248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.439960 (+   712us) service_pool.cc:224] Handling call
0709 12:25:53.063248 (+2623288us) spinlock_profiling.cc:243] Waited 384 ms on lock 0x7b54005f5d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:53.063363 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":883448192}
I20260709 12:25:53.055574  8933 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:25:53.056527  9060 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:25:53.087267  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:53.052850  8779 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:53.098011  5636 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:53.101198  5553 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:25:53.018721  8975 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.084864  9045 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:25:53.111730  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 128360us
I20260709 12:25:53.113233  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:53.116601  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:53.117372  5552 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:53.121203  9023 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:53.088696  8967 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 12:25:53.125429  8967 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:53.125875  8967 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:53.126735  8967 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.127933  8967 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:53.131309  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:53.132021  6276 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:53.132490  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:53.133421  6273 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:25:53.069159  8737 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
W20260709 12:25:53.137673  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.137768  8737 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:53.138518  8737 raft_consensus.cc:2764] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:25:53.143039  8737 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:53.143407  8737 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.143435  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:53.145321  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:53.146703  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:53.091754  9060 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.143805  8737 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:25:53.151770  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 86956.5us
I20260709 12:25:53.069294  8998 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.163801  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:53.159950  6276 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:53.166173  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:53.168207  5704 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:53.134801  8695 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.165249  5631 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:25:53.173672  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:53.174450  5706 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:53.174650  5631 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 9. Current term is 10. Ops: []
I20260709 12:25:53.176821  8779 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.167181  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:53.169281  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:53.178936  5634 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:25:53.182397  9032 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:53.183176  9032 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:53.187760  9067 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.885s	user 0.004s	sys 0.006s
I20260709 12:25:53.190419  8998 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:53.170388  8904 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.191824  8904 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.192245  9079 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.193068  9079 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.194042  8904 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:53.110929  9045 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:53.154958  9023 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.102283  8855 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 12:25:53.171164  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:53.105177  8944 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.062s
I20260709 12:25:53.233593  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:53.190738  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 677) took 2228 ms. Trace:
I20260709 12:25:53.161293  9060 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:25:53.216668  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49594.1us
I20260709 12:25:53.234259  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:53.242532  6278 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:25:53.167867  8973 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.234431  6274 rpcz_store.cc:276] 0709 12:25:50.962251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.161047 (+1198796us) service_pool.cc:224] Handling call
0709 12:25:52.175204 (+ 14157us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:52.994980 (+819776us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.190268 (+195288us) spinlock_profiling.cc:243] Waited 814 ms on lock 0x7b54004fa6d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:53.190627 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1872548608}
W20260709 12:25:53.248492  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 135380us
I20260709 12:25:53.253401  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:53.254173  5710 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 6: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:53.254742  5709 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:25:53.216202  9045 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:53.258536  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:53.259368  5703 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:25:53.257030  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:53.263411  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:53.266701  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:53.267519  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:53.268142  5710 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:53.268404  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:53.268520  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:53.278702  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:53.279569  5703 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:25:53.252229  9005 raft_consensus.cc:2764] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:25:53.286329  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:53.271382  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:53.283730  9086 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 12:25:53.255216  9080 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:25:53.235527  9023 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:53.293833  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:53.278326  8828 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:53.300968  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:53.301834  6280 raft_consensus.cc:2376] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8. Re-sending same reply.
I20260709 12:25:53.302731  8970 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:53.303429  8970 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:53.303972  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:53.304636  5557 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:53.304674  8933 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:25:53.305315  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:53.306300  6273 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20260709 12:25:53.306501  9084 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:25:53.255486  8973 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.310362  8973 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:53.310958  8973 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:25:53.312775  9002 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:53.312855  9091 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:25:53.314314  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:53.314646  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 545) took 1297 ms. Trace:
I20260709 12:25:53.314908  5561 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:53.315058  5708 rpcz_store.cc:276] 0709 12:25:52.017446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.017626 (+   180us) service_pool.cc:224] Handling call
0709 12:25:52.060770 (+ 43144us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:53.056788 (+996018us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.314418 (+257630us) spinlock_profiling.cc:243] Waited 955 ms on lock 0x7b54002f6054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:53.314586 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2195546496}
I20260709 12:25:53.280550  9046 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:53.276883  9082 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:53.319944  8975 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:53.323140  9023 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:53.324385  9010 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 2.099s	user 0.001s	sys 0.032s
I20260709 12:25:53.325263  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1860) took 1270 ms. Trace:
I20260709 12:25:53.267303  9059 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:25:53.325652  5560 rpcz_store.cc:276] 0709 12:25:52.054527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.054734 (+   207us) service_pool.cc:224] Handling call
0709 12:25:52.087611 (+ 32877us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:52.939119 (+851508us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.324817 (+385698us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b540023f354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:53.325188 (+   371us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1956508032}
I20260709 12:25:53.326423  5709 raft_consensus.cc:2376] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9. Re-sending same reply.
I20260709 12:25:53.345216  5552 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
W20260709 12:25:53.346676  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 51607.6us
I20260709 12:25:53.347533  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:53.349471  5557 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:25:53.352255  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:53.353528  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:53.356647  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 587) took 3136 ms. Trace:
I20260709 12:25:53.351230  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:53.362483  5557 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:53.365628  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1861) took 1255 ms. Trace:
W20260709 12:25:53.371634  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.305365  8933 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:53.357030  5629 rpcz_store.cc:276] 0709 12:25:50.216270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.436436 (+220166us) service_pool.cc:224] Handling call
0709 12:25:50.436790 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:50.438292 (+  1502us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:50.438326 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:50.438753 (+   427us) log.cc:844] Serialized 26 byte log entry
0709 12:25:50.831437 (+392684us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:50.831485 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:50.831559 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:53.323735 (+2492176us) spinlock_profiling.cc:243] Waited 4.28 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:53.329459 (+  5724us) raft_consensus.cc:1685] finished
0709 12:25:53.329479 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:53.352473 (+ 22994us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9841920,"thread_start_us":392319,"threads_started":1,"wal-append.queue_time_us":738703}
I20260709 12:25:53.367297  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:53.369210  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 105920us
W20260709 12:25:53.374321  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.380321  5560 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:53.378484  5561 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:53.371740  5554 rpcz_store.cc:276] 0709 12:25:52.109862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.109995 (+   133us) service_pool.cc:224] Handling call
0709 12:25:52.110315 (+   320us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:52.123230 (+ 12915us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:52.123269 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:52.128481 (+  5212us) log.cc:844] Serialized 26 byte log entry
0709 12:25:52.248171 (+119690us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:52.248211 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.248286 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:53.186918 (+938632us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:53.355203 (+168285us) raft_consensus.cc:1685] finished
0709 12:25:53.355225 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:53.365549 (+ 10324us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":149,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":4420,"spinlock_wait_cycles":4018048,"thread_start_us":70055,"threads_started":1,"wal-append.queue_time_us":149777}
I20260709 12:25:53.383315  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:53.384693  5552 raft_consensus.cc:2410] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:25:53.355728  8971 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:53.390522  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:53.386075  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:53.355353  9002 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:53.395519  5627 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:53.403617  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:25:53.401695  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 69911.9us
I20260709 12:25:53.357484  9082 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:53.399852  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:53.380285  9046 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:25:53.413404  6278 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46486
I20260709 12:25:53.414706  6278 raft_consensus.cc:606] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:53.415213  6278 raft_consensus.cc:614] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:53.416364  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:53.418104  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:53.418973  5631 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:53.421872  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:53.422986  5557 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:25:53.424628  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:53.426465  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:53.428716  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:53.429296  5629 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:53.429541  5634 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:25:53.429699  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:53.430325  5635 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:53.430447  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:53.431113  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:53.431454  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:53.431496  9092 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:53.413024  9005 raft_consensus.cc:2764] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:25:53.432883  8933 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.434624  8904 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:25:53.354854  8800 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.357817  8828 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 12:25:53.438052  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:53.449304  5636 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:53.402727  9039 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:25:53.441032  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:53.364804  9024 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.444624  6276 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:25:53.406296  8903 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:53.430902  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:53.455190  5630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1772) took 2097 ms (client timeout 1707 ms). Trace:
W20260709 12:25:53.460991  5630 rpcz_store.cc:269] 0709 12:25:51.357526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.359891 (+  2365us) service_pool.cc:224] Handling call
0709 12:25:53.454881 (+2094990us) spinlock_profiling.cc:243] Waited 2.06 s on lock 0x7b54001de7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:53.455088 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4743709440}
W20260709 12:25:53.463752  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60218.4us
I20260709 12:25:53.414206  9046 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:25:53.470261  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54230.9us
I20260709 12:25:53.477286  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:53.477294  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:53.441334  8904 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:53.494014  5706 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 8. Ops: []
I20260709 12:25:53.412822  8711 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 12:25:53.506708  5633 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:53.496447  5636 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:53.441573  8971 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:25:53.496847  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 228464us
I20260709 12:25:53.457453  9044 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:25:53.501961  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:53.414511  8970 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.503755  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:53.516116  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:53.471190  9032 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:53.453507  9039 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:53.505774  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 93303.5us
I20260709 12:25:53.478586  8903 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:53.521397  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:53.478785  9023 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260709 12:25:53.507519  9093 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 12:25:53.524029  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:53.557289  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:53.555786  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.552913  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:53.577803  5629 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:25:53.539729  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:53.507835  9091 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:25:53.617385  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 93700.3us
I20260709 12:25:53.548166  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1965) took 2184 ms. Trace:
I20260709 12:25:53.622980  5701 rpcz_store.cc:276] 0709 12:25:51.352939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.355190 (+  2251us) service_pool.cc:224] Handling call
0709 12:25:51.368339 (+ 13149us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:52.314893 (+946554us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.537142 (+1222249us) spinlock_profiling.cc:243] Waited 945 ms on lock 0x7b5400075154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:53.537344 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2173394176}
I20260709 12:25:53.509512  9087 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:25:53.594794  8787 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.542s	user 0.008s	sys 0.004s
I20260709 12:25:53.619621  9032 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:53.539143  9086 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:53.525357  8971 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:25:53.626524  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 86865.6us
W20260709 12:25:53.626343  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 106822us
I20260709 12:25:53.480490  5631 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
W20260709 12:25:53.572108  8566 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.510s	user 0.011s	sys 0.009s
I20260709 12:25:53.631639  9096 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260709 12:25:53.538694  9082 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:53.645318  9102 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:25:53.572901  8800 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:53.571530  8737 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.568116  9060 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.568665  8904 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.628285  9091 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.523066  9002 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.522413  9046 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.523200  9098 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:53.633843  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:25:53.717082  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 83316.5us
I20260709 12:25:53.633874  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:53.649806  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:53.650475  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:53.680008  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 127213us
I20260709 12:25:53.718641  5560 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:53.718927  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:53.719695  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:53.720275  5560 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:53.680891  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:53.724961  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44183.9us
I20260709 12:25:53.725154  9005 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:53.725819  9005 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:53.726238  5524 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:53.524253  9039 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:53.728804  9086 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:25:53.679500  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:53.635221  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:25:53.631891  9105 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:53.702832  8828 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:25:53.730086  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50697.2us
I20260709 12:25:53.730237  8828 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.730481  9105 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:25:53.730675  8828 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:53.731000  9105 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:53.731273  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:53.731884  9105 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.693291  5710 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:53.681425  9080 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:53.726320  9005 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:53.638799  9024 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:53.705276  9044 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:53.716854  9055 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:53.645017  8787 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:53.738564  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 104762us
I20260709 12:25:53.713419  9091 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:53.744524  9045 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.746070  9080 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.747402  9045 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:53.748446  9055 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.749145  9107 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:25:53.742807  9044 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.737737  8970 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:53.758734  8787 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:53.760943  5553 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:25:53.764534  5675 leader_election.cc:400] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8.
I20260709 12:25:53.767477  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:53.769008  5675 leader_election.cc:403] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:25:53.769567  5710 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:53.771615  5560 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:53.772290  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:53.772877  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:53.779580  5705 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:53.779786  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:53.774544  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55708.2us
I20260709 12:25:53.775049  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:53.781399  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:53.775843  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:53.776185  5710 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20260709 12:25:53.778546  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.774364  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:53.784323  5708 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:53.782135  5560 raft_consensus.cc:2368] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9.
I20260709 12:25:53.741745  8566 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:53.794077  9091 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.765630  8737 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:53.827131  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51352.3us
I20260709 12:25:53.643896  9087 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:53.831691  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 100480us
I20260709 12:25:53.766662  8970 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.830942  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:53.832180  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:53.830662  9020 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:53.813500  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.797204  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:53.832959  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:53.891521  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:53.892555  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 59667.6us
I20260709 12:25:53.892616  5560 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:53.787443  9022 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:53.894435  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:53.895066  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:53.795815  9046 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:53.900380  5705 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:25:53.894707  5709 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
I20260709 12:25:53.895318  5560 raft_consensus.cc:2376] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6. Re-sending same reply.
W20260709 12:25:53.898485  5526 leader_election.cc:336] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.588s (SENT)
I20260709 12:25:53.899036  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:53.787118  9024 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:53.795593  8566 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:53.829278  9098 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.905963  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:53.907547  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 75526.9us
I20260709 12:25:53.836298  8970 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:53.910612  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:53.894239  9022 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.834206  9023 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:25:53.766032  8903 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.918522  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:53.916208  9022 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:53.916962  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:53.909293  8566 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:53.940604  5553 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:53.927619  9060 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:53.928653  9104 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:25:53.929015  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 131899us
I20260709 12:25:53.929330  9114 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:25:53.930212  8711 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:53.930069  9109 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:25:53.931138  5710 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:53.931800  5704 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
I20260709 12:25:53.901177  9096 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:53.935670  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:53.918450  9080 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.021s
I20260709 12:25:53.937711  5705 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:25:53.903771  9020 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.916671  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:53.965333  5560 raft_consensus.cc:2376] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9. Re-sending same reply.
I20260709 12:25:53.927929  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:53.941730  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:53.914203  8800 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:53.968024  5553 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:53.948817  9032 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:25:53.949460  9087 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:53.961799  9104 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:53.962306  5708 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:53.963047  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:53.956663  9114 consensus_queue.cc:1243] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:25:53.977803  9106 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:25:53.980307  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:25:53.984120  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 79084.2us
I20260709 12:25:53.998910  5704 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:54.001521  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:53.983848  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:54.003356  9086 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.004683  8828 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:54.006892  9046 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.007345  9046 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:53.991065  9093 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:54.008508  9120 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.009241  9120 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:54.011509  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2096) took 2026 ms. Trace:
I20260709 12:25:54.012372  8737 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:54.013782  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:54.012037  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 796906us
I20260709 12:25:54.014952  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:54.015195  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.015481  5555 raft_consensus.cc:2376] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9. Re-sending same reply.
I20260709 12:25:53.983970  8711 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:25:54.016976  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.017257  8975 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.017144  8711 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:54.017871  8975 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.020754  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:54.021229  8695 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:54.021387  6280 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:54.021847  8695 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:54.022039  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:54.022408  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:53.980497  9096 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:53.916085  9098 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:53.995157  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:54.005187  9115 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:25:54.021494  5629 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:25:53.990016  9117 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:25:54.025036  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:54.026285  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:53.979537  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:54.027237  5704 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:25:54.027546  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:54.011873  5628 rpcz_store.cc:276] 0709 12:25:51.985014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.049484 (+1064470us) service_pool.cc:224] Handling call
0709 12:25:53.049773 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:53.049964 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:25:53.050285 (+   321us) log.cc:844] Serialized 14 byte log entry
0709 12:25:53.204976 (+154691us) log.cc:844] Serialized 14 byte log entry
0709 12:25:53.559668 (+354692us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b18003600a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:25:54.010660 (+450992us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b18003600a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:25:54.011018 (+   358us) log.cc:844] Serialized 14 byte log entry
0709 12:25:54.011168 (+   150us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:54.011210 (+    42us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:54.011254 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:54.011325 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:54.011455 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":770991,"prepare.queue_time_us":19128,"prepare.run_cpu_time_us":130,"prepare.run_wall_time_us":129,"raft.queue_time_us":820451,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":24883,"spinlock_wait_cycles":6300800,"thread_start_us":154115,"threads_started":1,"wal-append.queue_time_us":233540}
I20260709 12:25:54.023655  5560 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:54.022217  8695 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:54.031291  6278 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:54.003825  9086 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:54.020079  8975 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:54.062949  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:54.063866  6275 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:54.055357  5705 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:54.056210  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:54.057683  5636 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:54.081637  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:54.082407  5708 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 11: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:54.083742  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:54.085956  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:54.120566  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:54.121551  5555 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:54.127139  6279 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:25:54.127588  5631 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:25:54.023191  8506 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.129252  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:54.130113  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:54.011888  9120 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:54.141564  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:25:54.131119  9124 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:25:54.131915  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:54.142685  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:54.143555  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:54.132274  9116 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.216s	user 0.001s	sys 0.000s
I20260709 12:25:54.132840  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:54.147312  5561 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:25:54.148717  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:54.150417  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:54.073254  9076 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:25:54.133164  9119 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260709 12:25:54.133189  9032 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:25:54.134137  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:25:54.134172  9121 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:25:54.068943  8779 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.067s
I20260709 12:25:54.054401  8737 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.093s
I20260709 12:25:54.004675  9091 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:54.063980  8695 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:54.145251  5704 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: []
W20260709 12:25:54.152769  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 135236us
I20260709 12:25:53.996855  8800 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.132s
I20260709 12:25:54.016168  9005 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:54.166148  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.167996  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:54.167537  9080 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:54.227491  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:54.134584  9096 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:54.080245  8347 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:54.168574  9076 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:54.293326  9076 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:54.296429  9126 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
W20260709 12:25:54.182209  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 180747us
I20260709 12:25:54.151674  8506 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:54.297829  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:54.299866  8506 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:54.069674  9098 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:54.180308  8566 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:54.137012  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:54.300633  9091 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:54.301245  5557 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:54.301414  9091 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:54.301867  9091 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:54.302189  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:54.302415  8566 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:54.303221  8566 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:25:54.068681  9026 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.100s
I20260709 12:25:54.304318  8566 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:54.304603  9080 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:54.305078  8566 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:25:54.305891  9123 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
I20260709 12:25:54.305951  9080 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.305995  8566 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.308590  8566 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:54.190429  8903 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:54.189335  6275 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
W20260709 12:25:54.279505  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.300806  8800 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:25:54.193007  5636 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:25:54.228729  5561 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 5.
I20260709 12:25:54.136086  9022 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:54.314657  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:54.339279  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:54.174199  8828 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:54.259047  5560 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:54.342850  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:54.284533  9114 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.328218  5555 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:25:54.329806  8347 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:54.330320  5557 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:54.330950  9079 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:54.181038  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:54.334319  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2098) took 2217 ms. Trace:
I20260709 12:25:54.349236  5633 rpcz_store.cc:276] 0709 12:25:52.116527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.058132 (+941605us) service_pool.cc:224] Handling call
0709 12:25:53.058557 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:53.512600 (+454043us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:54.333983 (+821383us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x7b540018c754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:54.334231 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1030206208}
I20260709 12:25:54.335494  9046 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:54.226011  9060 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:54.289765  9116 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:25:54.339795  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:54.361433  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:54.343391  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:54.361032  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 218282us
W20260709 12:25:54.327970  9039 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.601s	user 0.010s	sys 0.000s
I20260709 12:25:54.344141  5602 leader_election.cc:400] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9.
W20260709 12:25:54.345009  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.355190  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:54.368439  5633 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:25:54.356251  5701 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:54.343441  5627 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:54.369653  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:54.371479  9079 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:25:54.372646  9079 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:54.373219  9079 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:54.374392  9079 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.363473  5555 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:25:54.319939  8800 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:54.346558  8737 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:54.368153  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:54.410063  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:54.410550  5557 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:54.361416  5636 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:25:54.378139  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:54.413458  5633 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:54.381183  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:54.367785  5602 leader_election.cc:403] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
W20260709 12:25:54.415087  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 71135.9us
I20260709 12:25:54.416724  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:54.417842  5557 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:25:54.419211  5557 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:25:54.408967  5560 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:54.410622  5627 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:54.423377  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:54.425859  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:54.426908  5633 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:54.428009  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:54.430038  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:54.342715  9129 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:25:54.443924  5560 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
I20260709 12:25:54.445834  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:54.367887  9032 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:54.449954  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:54.454123  5632 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:54.220268  9104 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:54.437951  9120 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:54.439260  9020 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:54.353446  9114 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:54.461752  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:54.377290  9060 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.487382  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:54.451578  9046 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:54.409704  8828 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
W20260709 12:25:54.491904  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 469538us
I20260709 12:25:54.388342  9039 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.496328  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:54.497969  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:54.500212  5631 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:25:54.505741  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:54.506412  6274 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:54.460781  9120 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:54.514209  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1849) took 3822 ms. Trace:
I20260709 12:25:54.512934  8787 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.526746  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:54.508330  5632 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:54.516538  5556 rpcz_store.cc:276] 0709 12:25:50.676475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.676610 (+   135us) service_pool.cc:224] Handling call
0709 12:25:50.680695 (+  4085us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:50.681102 (+   407us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:25:51.054763 (+373661us) log.cc:844] Serialized 14 byte log entry
0709 12:25:53.304810 (+2250047us) spinlock_profiling.cc:243] Waited 5.65 ms on lock 0x7b1800408340. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a
0709 12:25:53.628420 (+323610us) log.cc:844] Serialized 14 byte log entry
0709 12:25:53.713607 (+ 85187us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:53.716543 (+  2936us) log.cc:844] Serialized 26 byte log entry
0709 12:25:53.716701 (+   158us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:53.716737 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.717004 (+   267us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:54.498528 (+781524us) raft_consensus.cc:1685] finished
0709 12:25:54.498556 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:54.498696 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66422,"prepare.queue_time_us":711254,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":106,"raft.queue_time_us":81309,"raft.run_cpu_time_us":555,"raft.run_wall_time_us":59430,"spinlock_wait_cycles":12999040,"thread_start_us":737708,"threads_started":4,"wal-append.queue_time_us":1105234,"wal-append.run_cpu_time_us":1492,"wal-append.run_wall_time_us":1010233}
W20260709 12:25:54.531616  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 162069us
I20260709 12:25:54.541913  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:54.491329  8347 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9.
I20260709 12:25:54.540140  5631 pending_rounds.cc:85] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 4
I20260709 12:25:54.549804  5703 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: []
I20260709 12:25:54.552640  5631 pending_rounds.cc:107] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260709 12:25:54.555958  5631 raft_consensus.cc:2889] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:54.547626  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:54.513614  9086 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:54.519500  9039 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:54.499222  8506 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 12:25:54.521065  9026 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:54.561681  5632 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:25:54.552178  6274 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:25:54.515972  8828 consensus_queue.cc:1243] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:25:54.564759  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:54.515638  9134 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.482968  8800 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:54.549693  9105 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:25:54.630118  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 65433.2us
I20260709 12:25:54.642174  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:25:54.639454  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 258351us
I20260709 12:25:54.645251  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:54.645984  5554 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:54.646361  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:54.602771  8933 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:54.520925  8787 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.595788  9096 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:54.657059  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:54.658681  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:54.616899  9147 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:25:54.660837  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 130600us
I20260709 12:25:54.618259  9005 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:54.624927  8347 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:25:54.643829  9026 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:54.672726  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:54.562476  9032 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.673377  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:54.521472  9076 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:54.700465  9152 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:54.628264  9137 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.633127  9086 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:54.702725  9086 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:54.674525  5556 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:25:54.631194  9022 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:54.556859  9127 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:54.674223  8800 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:54.673460  9138 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:54.700523  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 43588.4us
I20260709 12:25:54.701756  9137 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.675370  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:54.679071  8695 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.685446  9096 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:54.625916  9120 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:54.720080  5708 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:54.724256  5560 raft_consensus.cc:2376] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10. Re-sending same reply.
I20260709 12:25:54.701294  9151 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:54.703123  9086 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:54.702035  9076 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:54.744793  9145 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.090s
I20260709 12:25:54.745982  9143 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:54.631796  9134 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:54.736608  5627 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
W20260709 12:25:54.737861  9156 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:25:54.738142  8787 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:54.730000  9096 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:54.738755  9151 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.008s
I20260709 12:25:54.725246  9138 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.743346  9060 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:54.748523  5633 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260709 12:25:54.750584  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 197930us
I20260709 12:25:54.730768  9114 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:54.694739  9147 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.776582  9149 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:54.777913  9141 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:25:54.778548  9141 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:54.778788  9147 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:54.699827  9026 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.779546  9141 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:54.724272  9127 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.810171  9103 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:54.810647  9103 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:54.817404  9103 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.818284  9164 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:54.818727  9164 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:25:54.819065  9140 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 12:25:54.840415  9127 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:54.841723  9127 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.841075  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1846) took 1064 ms. Trace:
I20260709 12:25:54.842345  5555 rpcz_store.cc:276] 0709 12:25:53.776990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.386656 (+609666us) service_pool.cc:224] Handling call
0709 12:25:54.387014 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:54.550345 (+163331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:54.550389 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:54.638439 (+ 88050us) log.cc:844] Serialized 26 byte log entry
0709 12:25:54.727052 (+ 88613us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:54.727106 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:54.727202 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:54.840736 (+113534us) raft_consensus.cc:1685] finished
0709 12:25:54.840766 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:54.840958 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":54790,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":407,"thread_start_us":208212,"threads_started":2,"wal-append.queue_time_us":99740}
I20260709 12:25:54.819645  9164 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.844328  9127 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.849809  9127 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:54.851055  9091 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.549s	user 0.011s	sys 0.003s
I20260709 12:25:54.851568  9091 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:54.852476  9157 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:25:54.855405  9091 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:54.849706  9134 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:54.845654  9164 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:54.856457  5554 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:25:54.859948  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:54.860527  5560 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 12:25:54.861591  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 172530us
I20260709 12:25:54.873191  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:54.873778  5560 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:54.874537  5601 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:25:54.880216  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 247049us
I20260709 12:25:54.880467  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:54.885773  6271 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.011s	sys 0.015s
W20260709 12:25:54.886180  6271 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.011s	sys 0.015s
I20260709 12:25:54.887737  5556 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:25:54.895105  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:54.902985  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 564) took 1503 ms. Trace:
I20260709 12:25:54.851495  9138 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:54.903613  5707 rpcz_store.cc:276] 0709 12:25:53.399327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.399496 (+   169us) service_pool.cc:224] Handling call
0709 12:25:53.501154 (+101658us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:53.711669 (+210515us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:53.711695 (+    26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:53.735342 (+ 23647us) log.cc:844] Serialized 52 byte log entry
0709 12:25:53.958574 (+223232us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:53.959231 (+   657us) log.cc:844] Serialized 14 byte log entry
0709 12:25:54.085968 (+126737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:54.086067 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:54.127078 (+ 41011us) raft_consensus.cc:1685] finished
0709 12:25:54.127102 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:54.902718 (+775616us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b5400452ed0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:54.902894 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1055,"prepare.queue_time_us":178,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":80568,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":165,"spinlock_wait_cycles":232592768,"thread_start_us":554875,"threads_started":3,"wal-append.queue_time_us":223176}
I20260709 12:25:54.920447  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:54.921159  5559 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
W20260709 12:25:54.921868  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 47422.8us
I20260709 12:25:54.922610  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:54.929332  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:54.931233  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 208904us
I20260709 12:25:54.919004  9127 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.933450  9162 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:54.909967  8695 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:54.943938  5560 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:25:54.947512  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:54.956056  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:54.917941  8933 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 12:25:54.949563  9162 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:54.963693  6273 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:25:54.965864  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:54.948983  9093 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.937s	user 0.011s	sys 0.011s
I20260709 12:25:54.966986  5556 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:54.962247  9129 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.976855  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:54.978138  9159 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:54.981943  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:54.987017  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:54.969029  9093 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.960s	user 0.012s	sys 0.012s
I20260709 12:25:54.973944  8855 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.992265  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:54.996176  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:54.997233  5704 raft_consensus.cc:2376] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11. Re-sending same reply.
I20260709 12:25:54.962347  9127 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:54.976317  9137 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:54.994560  9093 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:54.991802  9129 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:55.015432  9177 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:25:55.015966  9172 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.012s
W20260709 12:25:55.016812  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:55.019826  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:55.020794  5704 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:54.984912  9161 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 12:25:55.000109  8855 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:55.028429  9167 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
W20260709 12:25:55.029402  9171 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 12:25:55.030063  9153 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.223s	user 0.000s	sys 0.001s
I20260709 12:25:55.012295  9162 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:55.031462  9105 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:25:55.047508  9179 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260709 12:25:55.033123  9163 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
W20260709 12:25:55.044090  9158 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:25:55.051112  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:55.052044  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 46505us
I20260709 12:25:55.054226  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:55.061821  9141 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:25:55.048966  9024 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.136s	user 0.012s	sys 0.001s
I20260709 12:25:55.046945  9129 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:55.213021  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:55.067307  9161 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:55.365639  9176 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:55.366429  9176 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:55.369944  9170 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.305s
I20260709 12:25:55.370456  8566 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.371176  8566 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:55.371198  9170 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:55.385300  9170 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:55.386308  8855 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:55.393638  9127 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:55.394937  9093 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:55.071798  9024 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8.
W20260709 12:25:55.249325  9167 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.195s	user 0.000s	sys 0.000s
W20260709 12:25:55.428824  5599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.666s (SENT) [suppressed 45 similar messages]
I20260709 12:25:55.469975  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1848) took 1692 ms. Trace:
W20260709 12:25:55.481257  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:55.481412  5557 rpcz_store.cc:276] 0709 12:25:53.777574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.419875 (+642301us) service_pool.cc:224] Handling call
0709 12:25:54.420172 (+   297us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:54.424512 (+  4340us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:25:54.424536 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:54.425026 (+   490us) log.cc:844] Serialized 26 byte log entry
0709 12:25:54.638744 (+213718us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:54.638800 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:54.638906 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:55.469651 (+830745us) raft_consensus.cc:1685] finished
0709 12:25:55.469682 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:55.469887 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":213283,"threads_started":1,"wal-append.queue_time_us":329423}
I20260709 12:25:55.193878  9105 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:55.491645  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 601) took 1467 ms. Trace:
I20260709 12:25:55.492381  5630 rpcz_store.cc:276] 0709 12:25:54.024396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.025460 (+  1064us) service_pool.cc:224] Handling call
0709 12:25:54.074435 (+ 48975us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:54.499713 (+425278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:54.499758 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:54.521716 (+ 21958us) log.cc:844] Serialized 52 byte log entry
0709 12:25:54.802090 (+280374us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:54.802131 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:54.802232 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:55.491255 (+689023us) raft_consensus.cc:1685] finished
0709 12:25:55.491288 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:55.491549 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":204541,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":26800,"thread_start_us":704733,"threads_started":2,"wal-append.queue_time_us":397025}
I20260709 12:25:55.484212  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:55.249797  9175 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:55.508670  8566 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:55.509485  8566 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:55.510046  8566 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:55.513447  5702 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
W20260709 12:25:55.496259  9096 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.734s	user 0.012s	sys 0.000s
I20260709 12:25:55.528883  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:55.529834  5557 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 11 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
W20260709 12:25:55.556808  8970 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.620s	user 0.015s	sys 0.000s
I20260709 12:25:55.572173  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 564) took 1926 ms. Trace:
W20260709 12:25:55.573680  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:55.572728  5561 rpcz_store.cc:276] 0709 12:25:53.645445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.360463 (+715018us) service_pool.cc:224] Handling call
0709 12:25:54.360834 (+   371us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:54.367768 (+  6934us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:25:54.640196 (+272428us) log.cc:844] Serialized 14 byte log entry
0709 12:25:54.750876 (+110680us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:54.757672 (+  6796us) log.cc:844] Serialized 26 byte log entry
0709 12:25:54.757884 (+   212us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:54.757929 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:54.758015 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:55.563332 (+805317us) raft_consensus.cc:1685] finished
0709 12:25:55.563357 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:55.572094 (+  8737us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6516,"prepare.queue_time_us":384524,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":53469,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":39186,"thread_start_us":267365,"threads_started":2,"wal-append.queue_time_us":182894}
W20260709 12:25:55.576273  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.049s	sys 0.078s
W20260709 12:25:55.577558  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.049s	sys 0.078s
W20260709 12:25:55.579210  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 602417us
I20260709 12:25:55.581948  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:55.586826  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:55.499243  9105 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:55.480187  5555 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:25:55.386475  9170 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:55.524148  9096 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:55.596071  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:55.597518  9127 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:55.594089  9175 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:55.608785  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:55.569134  8566 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:55.601028  9170 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:55.589507  8970 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:55.621598  5561 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
W20260709 12:25:55.603972  8800 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.878s	user 0.008s	sys 0.004s
I20260709 12:25:55.606770  8855 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:55.604672  9176 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:55.630334  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.050s	sys 0.070s
I20260709 12:25:55.613231  9127 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:55.630988  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.050s	sys 0.070s
W20260709 12:25:55.642375  9181 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:25:55.634953  9127 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:55.627358  8566 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:55.691712  9128 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.343s	user 0.000s	sys 0.022s
I20260709 12:25:55.691864  9096 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:55.696149  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1971) took 1611 ms. Trace:
I20260709 12:25:55.696592  5708 rpcz_store.cc:276] 0709 12:25:54.100104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.173468 (+ 73364us) service_pool.cc:224] Handling call
0709 12:25:54.173936 (+   468us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:54.729019 (+555083us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.695903 (+966884us) spinlock_profiling.cc:243] Waited 546 ms on lock 0x7b5400309654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:55.696078 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1255290624}
I20260709 12:25:55.697939  5708 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:55.699069  5708 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:55.702790  9103 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:55.710521  5552 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:55.711854  5560 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:25:55.716264  5557 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 12:25:55.718326  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:55.722121  5704 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.013s	sys 0.043s
W20260709 12:25:55.723171  5704 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.013s	sys 0.043s
I20260709 12:25:55.700160  5708 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:55.679747  8566 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:55.726646  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1974) took 1596 ms. Trace:
I20260709 12:25:55.734858  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 554) took 2491 ms. Trace:
I20260709 12:25:55.719744  9141 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.740674  5708 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:25:55.722608  9087 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.732860  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:55.773705  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:55.749051  5704 rpcz_store.cc:276] 0709 12:25:54.128976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.021691 (+892715us) service_pool.cc:224] Handling call
0709 12:25:55.022047 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:55.022303 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:25:55.022661 (+   358us) log.cc:844] Serialized 14 byte log entry
0709 12:25:55.724765 (+702104us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:55.724848 (+    83us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:25:55.724914 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.725005 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:55.725162 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":386,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":7813,"thread_start_us":701322,"threads_started":1}
I20260709 12:25:55.719182  9170 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.749328  5706 rpcz_store.cc:276] 0709 12:25:53.243714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.243863 (+   149us) service_pool.cc:224] Handling call
0709 12:25:53.273617 (+ 29754us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:53.505245 (+231628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.734624 (+2229379us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b5400067054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:55.734801 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":506215296}
I20260709 12:25:55.774413  5708 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 5. Current term is 6. Ops: []
W20260709 12:25:55.726990  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 298219us
I20260709 12:25:55.723721  9182 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:55.785001  9087 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:55.790728  8566 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:55.774365  8737 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:55.792881  8737 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:55.794057  9055 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:55.794886  8737 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:55.772450  9164 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.881s	user 0.014s	sys 0.000s
I20260709 12:25:55.802672  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:55.773284  9105 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:55.806257  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 230553us
I20260709 12:25:55.808609  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:25:55.797786  9086 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.036s	user 0.011s	sys 0.003s
I20260709 12:25:55.780947  9159 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:55.777120  9185 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.777300  8347 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:55.811009  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:55.821633  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:55.770983  9141 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:55.825495  8347 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:55.826299  8347 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:55.831094  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2052) took 2005 ms. Trace:
I20260709 12:25:55.831585  5703 rpcz_store.cc:276] 0709 12:25:53.825811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.826514 (+   703us) service_pool.cc:224] Handling call
0709 12:25:53.826843 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:54.555223 (+728380us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.830805 (+1275582us) spinlock_profiling.cc:243] Waited 695 ms on lock 0x7b5400075654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:55.831018 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1599017728}
I20260709 12:25:55.833303  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1804) took 1399 ms. Trace:
I20260709 12:25:55.833699  8975 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:25:55.833798  5633 rpcz_store.cc:276] 0709 12:25:54.433654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.434404 (+   750us) service_pool.cc:224] Handling call
0709 12:25:54.748170 (+313766us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:54.761600 (+ 13430us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.833075 (+1071475us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b540008b6d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:55.833234 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":720303488}
I20260709 12:25:55.836575  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:55.838689  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:55.840399  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:55.817401  9164 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:55.857349  9164 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:55.857846  9164 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:55.858229  8348 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:25:55.858667  8348 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:55.857908  8970 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:55.862802  9159 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:25:55.863607  9159 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:55.864948  9159 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:55.865962  8506 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:55.859719  8348 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:55.797287  9170 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:55.816578  9086 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:55.809847  9166 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 12:25:55.868824  8348 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:55.870177  5559 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:25:55.870954  9184 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
I20260709 12:25:55.820209  9142 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:25:55.874650  9142 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:55.875144  9142 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:55.877638  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:55.878634  8904 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:55.879060  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:55.880002  8904 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:25:55.881675  9162 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:55.883548  8737 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.884200  8737 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:55.885278  5701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2045) took 2118 ms (client timeout 1666 ms). Trace:
W20260709 12:25:55.885690  5701 rpcz_store.cc:269] 0709 12:25:53.766936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.767068 (+   132us) service_pool.cc:224] Handling call
0709 12:25:55.885074 (+2118006us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b54001a9354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:55.885206 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1320022784}
I20260709 12:25:55.886907  8737 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:55.841634  8975 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:25:55.887596  6275 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 709) took 1988 ms (client timeout 1680 ms). Trace:
W20260709 12:25:55.888088  6275 rpcz_store.cc:269] 0709 12:25:53.898582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.898763 (+   181us) service_pool.cc:224] Handling call
0709 12:25:55.887160 (+1988397us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b5400772754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:55.887317 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":366948736}
I20260709 12:25:55.889740  8975 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:55.890873  8975 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:55.895622  9095 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:55.870052  9170 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:55.901221  8971 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:55.901909  8971 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:55.903270  8971 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:55.926620  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:55.896147  9095 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:55.928699  9103 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:55.928572  9095 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:55.929749  9095 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:55.930133  5707 raft_consensus.cc:2410] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:25:55.931931  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2061) took 1167 ms. Trace:
W20260709 12:25:55.927434  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 87188.5us
I20260709 12:25:55.933418  8971 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:55.881770  8347 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:55.938526  8347 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:55.800966  9182 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:55.940336  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60011us
I20260709 12:25:55.941416  9182 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:55.941937  9142 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:55.837385  9185 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:55.820537  9143 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:55.933841  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:55.932263  5707 rpcz_store.cc:276] 0709 12:25:54.764213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.929032 (+164819us) service_pool.cc:224] Handling call
0709 12:25:55.931714 (+1002682us) spinlock_profiling.cc:243] Waited 1e+03 ms on lock 0x7b54002d14d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:55.931868 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2299272960}
I20260709 12:25:55.872900  9086 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:55.957633  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:25:55.915606  9093 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.003s	sys 0.013s
I20260709 12:25:55.962836  5557 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:55.968889  9142 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:55.975967  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:55.978230  5561 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260709 12:25:55.978799  5561 pending_rounds.cc:85] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 8
I20260709 12:25:55.979148  5561 pending_rounds.cc:107] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 12:25:55.979460  5561 raft_consensus.cc:2889] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:55.980818  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:55.983359  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:55.983767  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:55.989939  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:55.967502  9093 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.004s	sys 0.014s
I20260709 12:25:55.820784  9026 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.151s
I20260709 12:25:55.996280  5557 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:25:55.950913  9143 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:56.033607  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:56.036594  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:56.037369  6275 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:55.820466  9139 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.167s
I20260709 12:25:55.913257  9127 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.044205  9127 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:55.996789  9142 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 12:25:56.032052  9107 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:56.032872  9002 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.017s
I20260709 12:25:56.032500  9145 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:56.032306  9103 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 12:25:56.050509  9143 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:56.051334  9005 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:56.051997  9005 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:56.055359  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.057054  9005 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:55.998688  9174 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.060s
I20260709 12:25:56.015833  9086 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.058924  9086 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:56.060461  9199 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:25:56.061839  9086 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:55.975162  9196 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:25:56.067946  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:56.076427  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 256387us
W20260709 12:25:56.078480  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.096912  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:56.098624  6276 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:25:56.032303  9152 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.102886  6271 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.8-7.8]
W20260709 12:25:56.103929  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 145979us
I20260709 12:25:56.106109  6280 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:25:56.033794  8998 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:56.107846  6276 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:25:56.107345  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:56.112439  9093 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:56.076864  9166 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:25:56.114596  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:56.078687  9107 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:25:56.099310  9149 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.316s	user 0.022s	sys 0.043s
I20260709 12:25:56.094390  9002 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:56.123668  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:56.126245  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.109638  8998 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:56.128497  6275 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:25:56.125286  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:56.148831  5553 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:56.130427  6271 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:56.132982  9170 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260709 12:25:56.105253  9196 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 12:25:56.115309  9139 consensus_queue.cc:1243] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:25:56.098656  9159 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
W20260709 12:25:56.119460  9149 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.342s	user 0.023s	sys 0.044s
I20260709 12:25:56.141135  9109 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.167399  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2048) took 2394 ms. Trace:
I20260709 12:25:56.167963  5709 rpcz_store.cc:276] 0709 12:25:53.773038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.773319 (+   281us) service_pool.cc:224] Handling call
0709 12:25:53.773636 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:53.902820 (+129184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.167121 (+2264301us) spinlock_profiling.cc:243] Waited 63 ms on lock 0x7b54002d19d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:56.167321 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":144929152}
I20260709 12:25:56.120280  9164 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:55.988337  8347 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:56.134253  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:56.171370  5707 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:56.143224  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:55.971125  9191 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.152217  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.151118  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.178490  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.159245  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.173107  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:56.180033  5706 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:56.113451  9152 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.185109  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.186272  9152 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:56.175586  9191 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.161413  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.130066  9002 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.128729  8998 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.165102  5704 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:25:56.127271  9190 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:25:56.140786  8903 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:56.112892  9093 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:56.168864  5553 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:25:56.140972  8711 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:56.174335  9109 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.177038  5703 raft_consensus.cc:2410] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:25:56.151831  9185 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:56.222126  6274 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:25:56.222311  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:56.117362  9182 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 12:25:56.190052  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.170405  9149 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:56.218331  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:56.223255  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.229144  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:56.213395  9148 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.181243  9026 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.039s
W20260709 12:25:56.232136  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:56.239014  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 169147us
I20260709 12:25:56.242005  5557 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:56.245107  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.246374  5557 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:25:56.248256  5553 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:25:56.249385  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:56.216758  9093 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:56.249284  5561 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: [6.6-8.7]
I20260709 12:25:56.259357  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.260679  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:56.261581  5561 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
W20260709 12:25:56.252236  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.253541  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.221104  9141 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:56.270073  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:56.270890  5553 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:56.278860  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:56.279204  5706 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:25:56.225171  9190 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:56.230620  9127 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:25:56.282167  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.284396  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:56.286590  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:56.293408  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.295574  5627 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:25:56.284801  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:56.215883  9107 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.299919  5633 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:56.303360  5553 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:56.304067  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:56.305215  5629 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:56.305687  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:56.303642  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:56.309041  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 231728us
I20260709 12:25:56.259398  8347 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:56.312990  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:56.219674  8711 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.319444  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.319727  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.321172  5707 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:25:56.322072  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.224689  8903 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.328819  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:56.329782  9005 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:56.324699  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:56.324155  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.331326  5707 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:56.244824  9148 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.306552  9127 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:25:56.309676  9197 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.205s	user 0.000s	sys 0.002s
I20260709 12:25:56.330693  5703 raft_consensus.cc:2376] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9. Re-sending same reply.
I20260709 12:25:56.333534  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:56.297240  9143 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.366659  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:56.335832  9218 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.341023  9087 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:56.342782  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:56.344175  6273 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:25:56.344900  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:56.329321  9214 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:56.315398  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:56.314999  9210 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.326882  9208 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.350822  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.352170  9002 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:56.355401  9164 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:56.356460  9191 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:56.391669  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:56.315980  8347 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.396289  9120 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:56.396763  9148 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:56.397027  8347 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:56.363150  8711 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:56.374053  5701 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:56.385114  9218 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.323632  9207 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 12:25:56.384877  5634 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
W20260709 12:25:56.385619  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 70297.9us
I20260709 12:25:56.315315  9162 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:25:56.314133  9194 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
W20260709 12:25:56.357191  9206 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:25:56.334311  9109 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:56.393733  5631 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:56.332443  9149 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:56.382690  9108 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.818s	user 0.049s	sys 0.159s
I20260709 12:25:56.404011  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:56.387598  9111 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:56.389657  9145 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:56.393957  9214 consensus_queue.cc:1243] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:25:56.419302  9161 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:25:56.412846  9218 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:56.425727  9108 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.868s	user 0.050s	sys 0.160s
I20260709 12:25:56.428961  9145 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:56.438098  9145 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:56.438865  9146 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.659s	user 0.004s	sys 0.009s
I20260709 12:25:56.391793  9208 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.390393  9210 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.441799  9208 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:56.442119  9210 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:56.442541  9214 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:56.443142  9143 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:56.443082  9214 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.444274  8783 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:56.444708  8783 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:56.445338  9214 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:56.445609  8783 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.445535  8903 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:56.445874  9162 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.428695  9159 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:25:56.424458  9093 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.389901  9190 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.434588  9164 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:56.464531  5633 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:56.469015  5633 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:56.468389  5629 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:25:56.470577  5629 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:56.473268  5558 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:25:56.457464  9162 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.457237  9159 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:56.477451  9159 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.479161  9159 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:56.479568  9185 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.484719  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:56.485446  5553 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:56.420085  9127 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.480116  9185 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:56.477470  9205 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.218s	user 0.001s	sys 0.000s
I20260709 12:25:56.509994  9164 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:56.511264  9164 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.472065  5629 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.614483  5636 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:25:56.560478  5552 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:25:56.707489  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:56.640494  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.710980  5628 raft_consensus.cc:2410] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20260709 12:25:56.502499  5555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 569) took 1621 ms (client timeout 1605 ms). Trace:
W20260709 12:25:56.712278  5555 rpcz_store.cc:269] 0709 12:25:54.881243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.894846 (+ 13603us) service_pool.cc:224] Handling call
0709 12:25:56.502209 (+1607363us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b540041a0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:56.502397 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":395683200}
I20260709 12:25:56.560824  5561 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:25:56.713743  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:56.715291  5555 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:56.490346  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:56.506716  5633 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.723385  5633 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:56.759985  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:56.716794  9005 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:25:56.787999  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 431419us
I20260709 12:25:56.790882  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:56.499946  9095 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.570s	user 0.012s	sys 0.000s
I20260709 12:25:56.774284  5553 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
I20260709 12:25:56.499928  8348 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:56.806497  8348 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:56.594808  9086 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.810570  5629 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:25:56.812479  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:56.813143  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 322833us
W20260709 12:25:56.815589  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 429442us
I20260709 12:25:56.816447  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:56.819628  9111 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:56.826785  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:56.828310  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:56.833360  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:56.835832  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:56.836720  5553 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:56.807773  8348 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.815368  9023 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:56.814069  9005 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:25:56.846377  5555 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:25:56.856897  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:56.847864  5561 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:25:56.817060  9162 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:56.867184  5636 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:25:56.857844  5627 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:56.860636  5553 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:56.867868  5636 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:56.866945  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:56.869537  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 86925us
I20260709 12:25:56.809628  9002 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260709 12:25:56.875259  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:56.878711  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:56.879103  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:56.879410  5632 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:25:56.879976  5627 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:56.880970  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:56.816401  9086 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.884254  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:56.885605  9005 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:56.887956  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.887614  9145 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.855365  5629 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 12:25:56.897862  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 81511.1us
I20260709 12:25:56.848778  9207 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:56.868710  5636 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:56.931383  5636 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:56.848942  9210 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:25:56.866883  9149 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:56.960712  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48095.2us
W20260709 12:25:56.961691  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62830.4us
W20260709 12:25:56.905841  8506 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260709 12:25:56.845108  9220 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:25:56.897063  9202 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:56.901225  9050 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:25:56.902762  8348 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:57.042136  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 70440.2us
I20260709 12:25:56.824560  9111 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:56.909235  9207 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:56.906294  9120 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:56.905162  9023 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:56.890157  9208 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.144s
W20260709 12:25:57.064281  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44017.6us
I20260709 12:25:57.066010  5633 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:25:57.038331  9109 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.019474  9086 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:57.078814  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:57.017632  9106 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 12:25:57.102773  9105 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.103018  9109 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:57.106825  9109 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:57.107684  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.108489  9145 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:57.084431  9228 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.192s	user 0.001s	sys 0.000s
I20260709 12:25:57.111114  9023 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:57.114228  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:57.115150  5561 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:25:57.060920  9045 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.070556  9207 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:57.020857  9149 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:57.122594  5553 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:57.114475  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:25:57.084378  9226 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.167s	user 0.000s	sys 0.000s
I20260709 12:25:57.130843  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:57.131940  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:57.141870  5629 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:57.142232  5632 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:57.069867  9002 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.022s
I20260709 12:25:57.141132  5633 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:57.146720  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:57.103677  9105 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:57.062336  9202 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:57.145138  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.143764  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:57.142683  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:57.151233  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 72539.6us
I20260709 12:25:57.154654  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:57.157498  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:57.157557  5561 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:57.142683  5553 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:25:57.162326  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:57.156599  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:57.164610  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.166056  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:57.166698  5705 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:57.168663  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.170640  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.125402  9091 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:57.081630  9120 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:57.173535  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 56623us
I20260709 12:25:57.154707  5636 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:25:57.171458  5701 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:25:57.149169  9050 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.013s
I20260709 12:25:57.280531  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:57.247529  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:57.282714  5553 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:57.247779  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:57.252568  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:57.283660  5555 raft_consensus.cc:2410] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:25:57.283433  5632 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:25:57.253433  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:57.255864  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:57.255884  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2050) took 3482 ms (client timeout 1721 ms). Trace:
I20260709 12:25:57.285715  5707 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
W20260709 12:25:57.285651  5702 rpcz_store.cc:269] 0709 12:25:53.773597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.780791 (+  7194us) service_pool.cc:224] Handling call
0709 12:25:57.255721 (+3474930us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b540026eb54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:57.255822 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3810064128}
W20260709 12:25:57.286422  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 129049us
I20260709 12:25:57.160014  9104 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:57.161615  8711 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.162803  9211 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.135073  9045 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:57.264233  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:57.309659  5705 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:57.197552  5703 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
W20260709 12:25:57.269328  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 327589us
I20260709 12:25:57.310195  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:57.311414  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:57.312085  6279 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:57.274622  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:57.275950  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:57.313611  6276 raft_consensus.cc:2376] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9. Re-sending same reply.
I20260709 12:25:57.276243  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:57.317070  6278 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:57.285476  5555 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:57.309789  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:57.320313  6271 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:25:57.321167  5553 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:25:57.324290  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.328383  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.277805  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:57.257014  5634 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:25:57.330238  5636 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:25:57.298468  5632 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:57.281481  6280 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:25:57.308836  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:57.272250  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:57.310811  6272 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:57.266155  9005 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.308856  8856 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20260709 12:25:57.373760  5706 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 12:25:57.374177  5706 pending_rounds.cc:85] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 7
I20260709 12:25:57.374447  5706 pending_rounds.cc:107] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:25:57.374696  5706 raft_consensus.cc:2889] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:57.374027  8856 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:25:57.421103  5556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 575) took 2472 ms (client timeout 1887 ms). Trace:
W20260709 12:25:57.421553  5556 rpcz_store.cc:269] 0709 12:25:54.948190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.975626 (+ 27436us) service_pool.cc:224] Handling call
0709 12:25:57.417070 (+2441444us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b54001c43d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:57.421032 (+  3962us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4013995264}
I20260709 12:25:57.423210  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1879) took 2777 ms. Trace:
I20260709 12:25:57.423810  5554 rpcz_store.cc:276] 0709 12:25:54.646061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.691123 (+ 45062us) service_pool.cc:224] Handling call
0709 12:25:54.691474 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:54.871187 (+179713us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.415495 (+2544308us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54000267d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:57.423133 (+  7638us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":378805120}
I20260709 12:25:57.421690  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.425889  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:57.426391  5554 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:25:57.426504  6273 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:57.426403  5705 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:25:57.250903  9091 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:57.291069  9174 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 12:25:57.293454  9104 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.456869  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:57.457293  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.333074  9106 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.201s
I20260709 12:25:57.449185  8856 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:57.496536  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.315737  8711 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:57.334259  9211 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:57.314525  9208 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.189s
W20260709 12:25:57.505317  9235 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.190s	user 0.000s	sys 0.001s
I20260709 12:25:57.513840  9105 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:57.422698  9238 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:57.521185  9229 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.260s	user 0.000s	sys 0.000s
W20260709 12:25:57.522254  9230 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
W20260709 12:25:57.523777  9232 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:25:57.528903  9202 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:57.548641  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:57.583935  5710 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:57.303201  9209 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:25:57.292229  9135 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:57.334841  9103 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.199s
I20260709 12:25:57.556213  6274 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:25:57.556610  5705 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:57.554414  6280 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:25:57.492815  5636 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:25:57.350410  9109 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.197s
I20260709 12:25:57.570489  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:57.452860  9134 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.102s
W20260709 12:25:57.597656  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 296131us
I20260709 12:25:57.598616  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.598815  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:57.600174  5705 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:57.577421  5633 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:25:57.580283  5701 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 6: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:57.581238  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.559803  9104 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:57.427572  9151 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:57.589046  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:57.592010  6271 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:25:57.580231  9208 consensus_queue.cc:1243] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 7.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 12:25:57.606081  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 50024.4us
I20260709 12:25:57.614617  6280 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:25:57.560242  9005 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:57.639735  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:57.643596  6272 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: [8.9-10.10]
I20260709 12:25:57.644778  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2071) took 1320 ms. Trace:
I20260709 12:25:57.644887  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:57.645246  6279 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:57.645818  6274 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:25:57.645192  5704 rpcz_store.cc:276] 0709 12:25:56.324686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.372487 (+ 47801us) service_pool.cc:224] Handling call
0709 12:25:56.392821 (+ 20334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:56.794902 (+402081us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:56.794951 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:56.809132 (+ 14181us) log.cc:844] Serialized 26 byte log entry
0709 12:25:57.042431 (+233299us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:25:57.042473 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.042544 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:57.644580 (+602036us) raft_consensus.cc:1685] finished
0709 12:25:57.644600 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.644723 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":449779,"raft.queue_time_us":420810,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":4935,"thread_start_us":232072,"threads_started":1,"wal-append.queue_time_us":456527}
I20260709 12:25:57.651497  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:57.646544  6277 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: [6.6-8.7]
I20260709 12:25:57.656852  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:57.657071  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:57.658039  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:57.650278  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 582) took 1678 ms. Trace:
I20260709 12:25:57.666680  5560 rpcz_store.cc:276] 0709 12:25:55.972164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.997375 (+ 25211us) service_pool.cc:224] Handling call
0709 12:25:56.502372 (+504997us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.541295 (+ 38923us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:56.541343 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:56.541418 (+    75us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:56.541479 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.541606 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.650090 (+1108484us) spinlock_profiling.cc:243] Waited 457 ms on lock 0x7b540041a0d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:57.650216 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38377,"raft.queue_time_us":88836,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":11301,"spinlock_wait_cycles":1051474176}
I20260709 12:25:57.693853  6271 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:57.694756  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1822) took 1257 ms. Trace:
I20260709 12:25:57.695443  5635 rpcz_store.cc:276] 0709 12:25:56.437576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.437711 (+   135us) service_pool.cc:224] Handling call
0709 12:25:56.438122 (+   411us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:56.468271 (+ 30149us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:25:56.468311 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:56.708734 (+240423us) log.cc:844] Serialized 26 byte log entry
0709 12:25:56.970100 (+261366us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:56.970162 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.970282 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:57.694421 (+724139us) raft_consensus.cc:1685] finished
0709 12:25:57.694453 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.694667 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37099,"raft.queue_time_us":22868,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":10077,"thread_start_us":196399,"threads_started":1,"wal-append.queue_time_us":387479}
I20260709 12:25:57.697365  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:57.698452  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:57.700596  5706 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:25:57.702521  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:57.585440  9238 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:57.703707  5553 raft_consensus.cc:2410] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:25:57.703809  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:57.709733  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:57.709877  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:57.650928  9159 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.171s	user 0.017s	sys 0.000s
I20260709 12:25:57.655136  8856 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:57.597843  9202 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.082s
I20260709 12:25:57.717887  5560 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:25:57.719352  6272 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
I20260709 12:25:57.719992  9134 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:57.721158  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 122413us
I20260709 12:25:57.721714  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:57.597008  9135 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:57.729605  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.717468  9149 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:57.715998  9249 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:25:57.745860  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:57.722849  9105 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:25:57.728144  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.737035  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:57.695250  9060 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.738732  5554 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:25:57.740430  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.616014  9151 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:57.729053  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:57.759936  6277 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:25:57.721704  9246 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:57.740859  9120 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:57.739221  9207 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.591s	user 0.006s	sys 0.006s
I20260709 12:25:57.756094  6275 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:57.753389  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:57.765735  9091 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:57.789218  9091 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.789867  9005 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:57.790002  9091 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:57.765766  9237 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 12:25:57.767480  9238 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:57.769622  8711 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:57.792268  9091 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:57.772215  9104 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:57.775003  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:57.748528  9149 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:57.796384  9149 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:57.748265  6273 raft_consensus.cc:2410] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:25:57.777087  9151 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:57.749291  8856 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:25:57.801164  6276 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
I20260709 12:25:57.805253  9103 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:25:57.807076  9103 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:57.808421  5693 tablet_service.cc:1558] Processing DeleteTablet for tablet 2acd6f5df0204b2295547e021cd8cb9f with delete_type TABLET_DATA_TOMBSTONED (TS f613ba929b1046d3b5aed909414a95bf not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:38456
I20260709 12:25:57.781383  5552 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 12:25:57.811663  5552 pending_rounds.cc:85] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 7
I20260709 12:25:57.811981  5552 pending_rounds.cc:107] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:25:57.812304  5552 raft_consensus.cc:2889] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:57.814642  5552 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:25:57.792493  9207 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:57.796814  9166 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260709 12:25:57.750627  9105 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.818439  5708 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 6: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:57.825260  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:57.826345  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 73061us
I20260709 12:25:57.762399  9246 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:57.834697  9246 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:57.827064  9105 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:57.836501  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.837347  5706 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:25:57.819885  9120 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:57.838435  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.740239  9134 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:57.839864  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.838691  9120 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.761157  9249 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:57.840811  5706 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:57.796861  9149 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:57.855410  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:57.846374  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:57.858693  6276 raft_consensus.cc:1813] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Handling vote request from an unknown peer f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:57.859279  6276 raft_consensus.cc:2368] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9.
I20260709 12:25:57.848680  6273 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:25:57.851462  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:57.826005  9108 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:25:57.762156  9060 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:57.875293  5633 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:25:57.889338  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.898043  5706 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
W20260709 12:25:57.891678  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 181866us
I20260709 12:25:57.832927  9185 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:57.902699  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:57.894757  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:57.857769  9120 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:57.896341  9202 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:25:57.853427  8904 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.940373  9104 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:57.872692  9209 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:57.941010  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.941777  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.872586  8711 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.946537  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.854044  9045 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:57.947314  5456 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 175) took 14842 ms (client timeout 15000 ms). Trace:
I20260709 12:25:57.955819  5706 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
W20260709 12:25:57.958938  5559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1890) took 3097 ms (client timeout 3091 ms). Trace:
I20260709 12:25:57.890878  9108 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:57.890381  9211 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:57.921754  9134 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:25:57.970579  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 75909.6us
W20260709 12:25:57.960575  5559 rpcz_store.cc:269] 0709 12:25:54.861775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.934538 (+ 72763us) service_pool.cc:224] Handling call
0709 12:25:57.958622 (+3024084us) spinlock_profiling.cc:243] Waited 908 ms on lock 0x7b540044eb54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:57.958797 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2088874880}
I20260709 12:25:57.946471  9202 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:57.979660  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:57.980651  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:57.987823  6272 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 5.
I20260709 12:25:57.942205  9151 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:25:57.974273  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:57.982115  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:57.982872  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:57.960256  5456 rpcz_store.cc:269] 0709 12:25:43.091711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.091916 (+   205us) service_pool.cc:224] Handling call
0709 12:25:57.933561 (+14841645us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:52.126876 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:52.365879 (+239003us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:52.365943 (+    64us) write_op.cc:435] Acquired schema lock
0709 12:25:52.365964 (+    21us) tablet.cc:662] Decoding operations
0709 12:25:52.368008 (+  2044us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:52.368171 (+   163us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:52.368205 (+    34us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:25:52.426361 (+ 58156us) tablet.cc:701] Row locks acquired
0709 12:25:52.426424 (+    63us) write_op.cc:260] PREPARE: finished
0709 12:25:52.426597 (+   173us) write_op.cc:270] Start()
0709 12:25:52.426752 (+   155us) write_op.cc:276] Timestamp: P: 1783599952426530 usec, L: 0
0709 12:25:52.426770 (+    18us) op_driver.cc:348] REPLICATION: starting
0709 12:25:52.437292 (+ 10522us) log.cc:844] Serialized 5386 byte log entry
0709 12:25:53.125810 (+688518us) op_driver.cc:464] REPLICATION: finished
0709 12:25:53.433369 (+307559us) write_op.cc:301] APPLY: starting
0709 12:25:53.433494 (+   125us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:53.433916 (+   422us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:53.434115 (+   199us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:53.565888 (+131773us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:53.566361 (+   473us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0709 12:25:53.566426 (+    65us) write_op.cc:312] APPLY: finished
0709 12:25:53.567210 (+   784us) log.cc:844] Serialized 101 byte log entry
0709 12:25:53.567573 (+   363us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:53.567778 (+   205us) write_op.cc:454] Released schema lock
0709 12:25:53.567894 (+   116us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:53.601539 (+ 33645us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:53.620146 (+ 18607us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":9885,"reported_tablets":31,"spinlock_wait_cycles":14080,"child_traces":[["op",{"apply.queue_time_us":253703,"apply.run_cpu_time_us":7130,"apply.run_wall_time_us":166980,"num_ops":14,"prepare.queue_time_us":1161273,"prepare.run_cpu_time_us":44645,"prepare.run_wall_time_us":879958,"raft.queue_time_us":146262,"raft.run_cpu_time_us":12952,"raft.run_wall_time_us":158216,"replication_time_us":698721,"thread_start_us":960010,"threads_started":4,"wal-append.queue_time_us":538254,"wal-append.run_cpu_time_us":10385,"wal-append.run_wall_time_us":1627562}]]}
I20260709 12:25:57.985363  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:57.997251  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:58.012557  5636 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:25:58.013643  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.014258  5636 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:25:58.015368  5707 raft_consensus.cc:2368] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8.
I20260709 12:25:57.967023  8998 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:57.991534  5706 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:25:57.959116  9023 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.948158  9185 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 12:25:57.989715  9202 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:58.007970  9151 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:58.015651  9134 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:57.987046  9254 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:57.985684  5705 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:25:58.022979  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:57.948323  9091 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.043s
I20260709 12:25:57.978868  9108 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:25:58.016376  9249 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:57.966704  8711 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:57.957656  8904 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:58.016248  8348 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.948s	user 0.008s	sys 0.005s
W20260709 12:25:58.049852  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:58.040073  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.021088  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.166160  6274 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:58.026924  9104 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.146s
I20260709 12:25:58.027833  9248 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:58.020494  8695 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.263s
I20260709 12:25:58.389712  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 718) took 1115 ms. Trace:
I20260709 12:25:58.516934  5636 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:25:58.343614  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 324130us
I20260709 12:25:58.245898  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:58.518452  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 272681us
I20260709 12:25:58.079622  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.517035  6275 rpcz_store.cc:276] 0709 12:25:57.274111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.274290 (+   179us) service_pool.cc:224] Handling call
0709 12:25:58.389481 (+1115191us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b54001e87d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:58.389627 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1018229120}
I20260709 12:25:58.521589  6274 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 12:25:58.525576  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.519882  5634 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
W20260709 12:25:58.407860  9259 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
W20260709 12:25:58.414299  9149 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.517s	user 0.008s	sys 0.000s
W20260709 12:25:58.411134  9105 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.576s	user 0.008s	sys 0.000s
I20260709 12:25:58.358328  9263 tablet_replica.cc:333] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:25:58.417848  9134 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:25:58.529803  9098 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
W20260709 12:25:58.044482  9236 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.782s	user 0.002s	sys 0.001s
I20260709 12:25:58.533425  5675 leader_election.cc:400] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9.
I20260709 12:25:58.042541  9246 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.534519  5675 leader_election.cc:403] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:58.535102  9134 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:25:58.530215  9149 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.536036  9093 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:58.046214  8783 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:58.536204  9246 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:58.537722  9149 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:58.527168  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:58.531351  9262 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.045008  9207 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:58.522225  6274 pending_rounds.cc:85] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 7
I20260709 12:25:58.045006  9151 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:25:58.403545  9258 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:25:58.525712  9268 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:25:58.030952  9023 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:58.077227  9107 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.032368  9135 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:58.530532  9105 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.551558  9261 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20260709 12:25:58.166424  9185 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.429250  9111 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.060223  9238 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:58.562206  6274 pending_rounds.cc:107] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:25:58.587610  6274 raft_consensus.cc:2889] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:58.422163  9120 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.033272  9060 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:58.275857  9254 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:58.577287  9023 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:58.582275  8904 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:58.581302  9107 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.582971  9134 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9.
I20260709 12:25:58.583590  9139 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:25:58.548171  8998 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.521s	user 0.003s	sys 0.007s
I20260709 12:25:58.045324  9209 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:58.452493  9202 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:25:58.548695  9248 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.559140  9262 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.594310  9254 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:58.589148  9185 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:58.642582  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:58.648332  5555 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:25:58.702850  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1906) took 1046 ms. Trace:
I20260709 12:25:58.702977  5635 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 12:25:58.328693  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 258491us
W20260709 12:25:58.704453  5526 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.645s (SENT)
I20260709 12:25:58.704522  9108 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.705175  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:58.650998  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.606118  9262 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:58.295548  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.739311  8506 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:25:58.741065  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 222262us
I20260709 12:25:58.740774  8506 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:58.653669  5628 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
I20260709 12:25:58.742570  5628 pending_rounds.cc:85] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 6
I20260709 12:25:58.742975  5628 pending_rounds.cc:107] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260709 12:25:58.743645  5628 raft_consensus.cc:2889] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:58.748696  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:58.530700  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.544497  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:58.753085  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 208650us
I20260709 12:25:58.753631  9106 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:58.754616  9106 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:58.670286  6271 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:25:58.671959  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:58.671959  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:58.683333  5709 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:25:58.592875  9105 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:58.684788  9060 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:58.685442  5710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 572) took 1357 ms (client timeout 1528 ms). Trace:
I20260709 12:25:58.686348  9086 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:25:58.592303  9109 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.960s	user 0.031s	sys 0.021s
W20260709 12:25:58.780972  9109 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.150s	user 0.031s	sys 0.022s
I20260709 12:25:58.580523  9093 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:58.784195  9093 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:58.784837  9093 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:58.788836  9283 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.789505  5556 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:25:58.789834  9283 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.689518  9107 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:58.589318  9111 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.596606  9139 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:58.598521  9120 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:58.695248  9202 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:58.684252  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 570) took 1356 ms. Trace:
W20260709 12:25:58.606106  9230 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.010s	user 0.002s	sys 0.000s
I20260709 12:25:58.698014  9254 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:58.610579  9248 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:58.591933  9238 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.831841  9238 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:58.706898  5708 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2075) took 1550 ms (client timeout 1867 ms). Trace:
W20260709 12:25:58.835390  5708 rpcz_store.cc:269] 0709 12:25:57.156133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.156305 (+   172us) service_pool.cc:224] Handling call
0709 12:25:58.706079 (+1549774us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b5400210f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:58.706368 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1485095680}
I20260709 12:25:58.703408  5554 rpcz_store.cc:276] 0709 12:25:57.655919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.657600 (+  1681us) service_pool.cc:224] Handling call
0709 12:25:57.657850 (+   250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.755246 (+ 97396us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.702150 (+946904us) spinlock_profiling.cc:243] Waited 80.6 ms on lock 0x7b5400025dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:58.702577 (+   427us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":185282944}
I20260709 12:25:58.707160  9241 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:25:58.708109  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.710754  9108 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:58.713456  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:58.713346  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 1.59869e+06us
I20260709 12:25:58.716960  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1823) took 2257 ms. Trace:
I20260709 12:25:58.725893  9046 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:58.729601  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.732263  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:58.734692  8347 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:58.766732  9250 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.759166  9210 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:58.753795  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.755033  5553 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:58.767242  9060 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.767994  8711 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:58.768901  9185 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:58.769500  9086 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.770983  6274 raft_consensus.cc:2368] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8.
I20260709 12:25:58.754696  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:58.771220  6276 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:25:58.765092  5710 rpcz_store.cc:269] 0709 12:25:57.327853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.421453 (+ 93600us) service_pool.cc:224] Handling call
0709 12:25:58.685034 (+1263581us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b5400066154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:58.685236 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":363555072}
I20260709 12:25:58.755069  9106 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:58.782929  6278 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.864s	user 0.031s	sys 0.101s
I20260709 12:25:58.792858  9283 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:58.792695  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:58.861743  9250 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:58.862725  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 70309us
I20260709 12:25:58.795511  9164 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:58.864877  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.864763  9164 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.867650  9164 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:58.062812  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45636.2us
I20260709 12:25:58.850703  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:58.856674  5631 raft_consensus.cc:2376] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7. Re-sending same reply.
I20260709 12:25:58.857380  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:58.857331  8856 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:25:58.857802  9060 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:58.857843  8347 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:58.867874  9271 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:25:58.868503  6278 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.028s	user 0.031s	sys 0.102s
I20260709 12:25:58.869514  9210 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:58.870733  5636 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:25:58.803388  5701 rpcz_store.cc:276] 0709 12:25:57.327243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.328026 (+   783us) service_pool.cc:224] Handling call
0709 12:25:58.683629 (+1355603us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b540033fc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:58.683951 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":577014144}
I20260709 12:25:58.871302  9086 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:58.872954  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:25:58.892853  5635 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
W20260709 12:25:58.895670  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 141242us
I20260709 12:25:58.893205  9164 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:58.902968  9190 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:25:58.903594  9190 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:58.903720  9196 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.904181  9190 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:58.841809  5630 rpcz_store.cc:276] 0709 12:25:56.451802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.452048 (+   246us) service_pool.cc:224] Handling call
0709 12:25:56.459411 (+  7363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.459714 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:25:56.716705 (+256991us) log.cc:844] Serialized 14 byte log entry
0709 12:25:56.871424 (+154719us) log.cc:844] Serialized 14 byte log entry
0709 12:25:57.418991 (+547567us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b18001c8580. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:25:58.605898 (+1186907us) spinlock_profiling.cc:243] Waited 8.61 ms on lock 0x7b240029c0a8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff740b9c4 00007f4ff7409acb 00007f4ff75c4dc9 00007f4ff75c4a55 00007f4ff75c409d 00007f4ffa1f610e 00007f4ffa20d88c 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221
0709 12:25:58.707849 (+101951us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b18001c8580. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:25:58.708198 (+   349us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:58.708287 (+    89us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:25:58.708473 (+   186us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.708590 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:58.708812 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1849601,"prepare.queue_time_us":352985,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":81,"raft.queue_time_us":1866533,"raft.run_cpu_time_us":492,"raft.run_wall_time_us":493,"spinlock_wait_cycles":83117568,"thread_start_us":389757,"threads_started":2,"wal-append.queue_time_us":430481}
I20260709 12:25:58.904917  9196 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.906234  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.907348  9196 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:58.907593  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:25:58.909153  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.909394  8783 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:58.909852  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.910215  8783 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:58.914647  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.915210  9151 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.915789  5631 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
W20260709 12:25:58.916883  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52852.9us
I20260709 12:25:58.917652  9151 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:58.918604  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:58.918558  6236 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:58.899853  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:58.902814  9164 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.909188  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:58.920288  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.922948  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:58.926568  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:58.927469  9046 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:58.925325  5636 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:25:58.923066  9262 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.014s
I20260709 12:25:58.942368  5553 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:58.938172  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:58.952469  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 65123.7us
I20260709 12:25:58.939633  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:58.936570  6276 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:25:58.939669  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:58.957020  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:58.959061  6279 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:25:58.919924  9139 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:58.930401  9254 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:58.963471  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.959213  6273 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:58.964888  5553 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:25:58.959498  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.937156  9262 consensus_queue.cc:1243] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:25:58.967726  8711 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:25:58.968120  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49723.2us
I20260709 12:25:58.968295  8711 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:58.968775  8711 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:25:58.973100  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 115946us
I20260709 12:25:58.975273  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:58.962944  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:58.936805  8783 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:58.936299  9284 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:25:58.982461  8506 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:58.982729  9139 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:25:58.983312  9104 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:59.013654  9104 consensus_queue.cc:1243] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:25:58.984884  9046 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:59.015574  9283 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:25:58.986351  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.987150  5631 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:59.018110  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1899) took 1844 ms. Trace:
I20260709 12:25:59.018559  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 1983) took 2847 ms. Trace:
I20260709 12:25:59.018772  5561 rpcz_store.cc:276] 0709 12:25:57.173115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.174207 (+  1092us) service_pool.cc:224] Handling call
0709 12:25:57.174505 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.572700 (+398195us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:57.572730 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:57.572773 (+    43us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:25:57.572810 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.017578 (+1444768us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x7b54001be754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:59.017645 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.017859 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":915225344}
I20260709 12:25:59.019016  5709 rpcz_store.cc:276] 0709 12:25:56.171239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.186176 (+ 14937us) service_pool.cc:224] Handling call
0709 12:25:56.186461 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.760289 (+2573828us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.018211 (+257922us) spinlock_profiling.cc:243] Waited 2.5 s on lock 0x7b54002f4254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:59.018445 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5740539648}
I20260709 12:25:59.021010  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:59.022032  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2129) took 1065 ms. Trace:
I20260709 12:25:59.022389  5632 rpcz_store.cc:276] 0709 12:25:57.956181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.956333 (+   152us) service_pool.cc:224] Handling call
0709 12:25:59.020683 (+1064350us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:59.021944 (+  1261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":40149248}
I20260709 12:25:59.024807  9183 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:25:59.031967  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 691) took 1398 ms. Trace:
I20260709 12:25:58.987341  9245 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:25:59.023739  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:58.991569  8903 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:58.993012  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:58.993840  9211 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:25:58.995961  9281 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.123s	user 0.002s	sys 0.000s
I20260709 12:25:58.956879  8856 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:58.997076  8347 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:58.960314  9107 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.998622  9111 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:59.000775  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:58.997071  5558 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1876) took 4351 ms (client timeout 2767 ms). Trace:
I20260709 12:25:58.939538  9207 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.009596  9284 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:25:59.010782  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:25:59.012473  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.012624  8506 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:25:59.018066  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:58.990818  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:59.008064  9060 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:25:59.048228  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:59.038771  8903 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:59.006063  9250 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:59.032490  6278 rpcz_store.cc:276] 0709 12:25:57.633275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.633407 (+   132us) service_pool.cc:224] Handling call
0709 12:25:57.633694 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.633879 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:25:57.840558 (+206679us) log.cc:844] Serialized 14 byte log entry
0709 12:25:59.030536 (+1189978us) log.cc:844] Serialized 14 byte log entry
0709 12:25:59.031105 (+   569us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:59.031184 (+    79us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:59.031259 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.031387 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.031888 (+   501us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4612,"prepare.queue_time_us":339994,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":28268,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":6006,"thread_start_us":1276141,"threads_started":3,"wal-append.queue_time_us":768108}
W20260709 12:25:59.053663  5558 rpcz_store.cc:269] 0709 12:25:54.645205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.645403 (+   198us) service_pool.cc:224] Handling call
0709 12:25:58.996586 (+4351183us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54000ea1d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:25:58.996861 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4088941952}
I20260709 12:25:59.061744  5636 raft_consensus.cc:2410] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:25:59.062505  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:59.039160  9245 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:25:59.066869  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:59.070881  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1907) took 1414 ms. Trace:
I20260709 12:25:59.061863  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:59.138933  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:59.135352  6277 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
W20260709 12:25:59.138816  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 88856.2us
I20260709 12:25:59.060485  5526 leader_election.cc:400] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8.
I20260709 12:25:59.146675  5526 leader_election.cc:403] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:25:59.147878  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 89923.3us
I20260709 12:25:59.151489  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:59.134820  5555 rpcz_store.cc:276] 0709 12:25:57.656185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.656297 (+   112us) service_pool.cc:224] Handling call
0709 12:25:57.709333 (+ 53036us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.703282 (+993949us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.070526 (+367244us) spinlock_profiling.cc:243] Waited 938 ms on lock 0x7b54000ea6d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:59.070805 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2157368576}
I20260709 12:25:59.041219  9093 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.130388  9060 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:59.063696  9283 consensus_queue.cc:1243] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 10.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:25:59.059674  9120 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.157467  6279 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:25:59.052423  8711 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.059670  9151 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.059023  9046 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:25:59.177132  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 166503us
I20260709 12:25:59.177846  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 611) took 2048 ms. Trace:
I20260709 12:25:59.177541  9046 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:59.178995  6271 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:25:59.148629  9086 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:59.058856  9263 raft_consensus.cc:2243] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Raft consensus shutting down.
W20260709 12:25:59.163827  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 103608us
I20260709 12:25:59.171170  8856 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:25:59.171438  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 171275us
I20260709 12:25:59.174253  9220 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:25:59.169312  9196 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:59.158476  9143 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:25:59.070086  9170 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.164s
I20260709 12:25:59.061020  9207 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.062008  9107 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.184934  6276 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:59.178272  5627 rpcz_store.cc:276] 0709 12:25:57.129284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.132566 (+  3282us) service_pool.cc:224] Handling call
0709 12:25:57.136546 (+  3980us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:58.710186 (+1573640us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:58.710231 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:58.711021 (+   790us) log.cc:844] Serialized 26 byte log entry
0709 12:25:58.731809 (+ 20788us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:58.731847 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.731954 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:58.995145 (+263191us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:25:59.177388 (+182243us) raft_consensus.cc:1685] finished
0709 12:25:59.177414 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.177677 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1571876,"raft.queue_time_us":1596902,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":167,"spinlock_wait_cycles":28699392,"thread_start_us":20152,"threads_started":1,"wal-append.queue_time_us":20734}
I20260709 12:25:59.198618  5631 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:25:59.266806  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 718) took 1245 ms. Trace:
I20260709 12:25:59.244113  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:59.185424  5554 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:25:59.187422  6273 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:25:59.201390  5632 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:25:59.253157  5553 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
W20260709 12:25:59.191612  5602 leader_election.cc:336] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.767s (SENT)
I20260709 12:25:59.060752  9174 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.120s
I20260709 12:25:59.255365  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:25:59.060153  9185 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.126s
I20260709 12:25:59.254904  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:25:59.305403  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 139622us
I20260709 12:25:59.195407  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:59.246604  5559 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:25:59.271425  5630 pending_rounds.cc:85] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 7
I20260709 12:25:59.311834  5630 pending_rounds.cc:107] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 12:25:59.312242  5630 raft_consensus.cc:2889] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:59.269937  9284 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:25:59.313346  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.191140  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:59.254120  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:59.203631  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:59.318455  6274 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 12:25:59.293040  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:59.191810  9005 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:25:59.192641  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:59.195362  9164 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:59.297096  9143 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:59.287617  8856 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:59.350417  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 96590us
I20260709 12:25:59.257125  9250 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:25:59.190517  9282 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:59.199285  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1831) took 2346 ms. Trace:
I20260709 12:25:59.288295  9220 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:59.314929  9283 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:25:59.338732  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2127) took 1420 ms. Trace:
I20260709 12:25:59.338696  9086 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.339370  9260 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.331234  9196 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:25:59.339931  9209 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:25:59.341194  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 621) took 1899 ms. Trace:
I20260709 12:25:59.255287  9142 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:25:59.342046  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 349392us
I20260709 12:25:59.343041  9107 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:59.343705  5561 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:59.344594  9183 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:25:59.345173  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 149945us
I20260709 12:25:59.204080  9285 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 6, Time since last communication: 0.142s
I20260709 12:25:59.356755  8856 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:25:59.346719  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 356122us
I20260709 12:25:59.320214  9151 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:59.355019  5636 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:25:59.293550  9106 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.348655  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:59.356768  5633 rpcz_store.cc:276] 0709 12:25:57.918221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.918427 (+   206us) service_pool.cc:224] Handling call
0709 12:25:59.161431 (+1243004us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:25:59.338664 (+177233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35064704}
I20260709 12:25:59.313434  9284 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:59.315169  9120 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:59.375461  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.403136  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:59.407770  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44878.2us
I20260709 12:25:59.246726  9060 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.177183  8903 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:59.356525  9143 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:59.183677  9245 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.418527  9263 raft_consensus.cc:2272] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:59.386972  9183 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:59.423475  5555 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:59.426705  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:59.427601  8347 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:25:59.300458  6272 rpcz_store.cc:276] 0709 12:25:58.021430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.023433 (+  2003us) service_pool.cc:224] Handling call
0709 12:25:58.060736 (+ 37303us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:58.061108 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:58.061148 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:58.061657 (+   509us) log.cc:844] Serialized 26 byte log entry
0709 12:25:59.265787 (+1204130us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x7b18003c77e0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1
0709 12:25:59.266131 (+   344us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:25:59.266180 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.266317 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:59.266365 (+    48us) raft_consensus.cc:1685] finished
0709 12:25:59.266377 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.266615 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":106798,"raft.run_cpu_time_us":544,"raft.run_wall_time_us":68636,"spinlock_wait_cycles":66043264,"thread_start_us":112605,"threads_started":1,"wal-append.queue_time_us":341017}
I20260709 12:25:59.345189  9207 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:59.347915  5635 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:25:59.355964  9283 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:59.353214  5628 rpcz_store.cc:276] 0709 12:25:56.853111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.853244 (+   133us) service_pool.cc:224] Handling call
0709 12:25:56.853519 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.744788 (+1891269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.198957 (+454169us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b540018d154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:25:59.199223 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4137106816}
I20260709 12:25:59.375460  5629 rpcz_store.cc:276] 0709 12:25:57.263451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.263606 (+   155us) service_pool.cc:224] Handling call
0709 12:25:57.263964 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.264256 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:25:57.289935 (+ 25679us) log.cc:844] Serialized 14 byte log entry
0709 12:25:58.953639 (+1663704us) log.cc:844] Serialized 14 byte log entry
0709 12:25:59.056577 (+102938us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:59.092680 (+ 36103us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:25:59.133889 (+ 41209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.145846 (+ 11957us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.162958 (+ 17112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1214767,"prepare.queue_time_us":166019,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":95,"raft.queue_time_us":1152270,"raft.run_cpu_time_us":474,"raft.run_wall_time_us":4672,"thread_start_us":450847,"threads_started":4,"wal-append.queue_time_us":719056,"wal-append.run_cpu_time_us":1521,"wal-append.run_wall_time_us":1071340}
I20260709 12:25:59.383695  9238 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:59.436302  9238 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:59.436722  9238 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:25:59.410385  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.440541  9282 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.441494  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:59.443164  9220 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.444562  9107 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:25:59.445665  9220 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:59.446111  9046 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:59.448549  9105 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:59.448945  9105 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:59.449271  9105 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:59.411119  9260 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.428963  9284 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.454389  9284 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:59.455780  9093 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:59.456746  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 253252us
I20260709 12:25:59.457626  9151 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:59.459323  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.461009  9120 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:59.428120  8347 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:59.423995  9183 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:59.432225  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:25:59.475839  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 284265us
I20260709 12:25:59.478849  9211 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:25:59.479089  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55911.9us
I20260709 12:25:59.403625  9142 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:59.465569  9245 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:25:59.516947  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 134953us
W20260709 12:25:59.517702  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 114777us
I20260709 12:25:59.526787  9045 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:59.527030  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:59.526104  9190 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.621s	user 0.007s	sys 0.007s
W20260709 12:25:59.559666  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 138476us
I20260709 12:25:59.523478  9060 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:25:59.560884  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 130024us
I20260709 12:25:59.562332  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:59.525315  8711 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:59.564815  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:25:59.567606  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 143282us
W20260709 12:25:59.568545  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53544.2us
I20260709 12:25:59.568674  5555 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:25:59.541471  9207 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.559130  5526 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:59.555943  9196 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.577546  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:59.579241  5632 raft_consensus.cc:1813] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Handling vote request from an unknown peer f613ba929b1046d3b5aed909414a95bf
I20260709 12:25:59.578008  6272 raft_consensus.cc:1217] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 3.5->[4.6-9.8]   Dedup: 4.6->[8.7-9.8]
I20260709 12:25:59.581324  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:59.579774  6272 pending_rounds.cc:85] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 6
I20260709 12:25:59.569447  9045 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:59.584954  5632 raft_consensus.cc:2368] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9.
I20260709 12:25:59.585114  6272 pending_rounds.cc:107] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:25:59.582021  5629 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 6.
I20260709 12:25:59.592585  6272 raft_consensus.cc:2889] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:59.589380  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:59.570201  9260 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:59.596660  9174 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:25:59.597158  5636 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: []
I20260709 12:25:59.597400  9283 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.593377  5633 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:25:59.559999  9190 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:25:59.611362  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.612020  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:59.616999  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:25:59.608132  9106 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:59.593832  9207 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.597208  9304 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:25:59.639117  9045 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.639881  9304 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:59.640403  9304 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:59.609193  9174 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:25:59.642019  9283 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:59.643328  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51197.1us
I20260709 12:25:59.643432  9190 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:25:59.635998  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:25:59.618296  9260 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 12:25:59.638695  9196 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:59.628051  9086 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:59.608413  9185 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:25:59.667403  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49338.3us
I20260709 12:25:59.644160  9190 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:25:59.683523  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:25:59.690187  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45236.5us
I20260709 12:25:59.694340  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:25:59.642046  9174 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:25:59.707463  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 98413.1us
I20260709 12:25:59.707661  8506 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.708845  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:59.709704  9210 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:25:59.709897  9207 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:59.710322  9210 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:25:59.692795  9139 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.711423  9210 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:59.712728  5709 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 12:25:59.713109  9247 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:59.713613  9247 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:59.713991  9247 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:59.714998  9104 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:25:59.715888  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:25:59.685572  9285 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.716729  9285 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.717247  9104 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:25:59.717640  9104 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:59.718004  9104 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:25:59.719338  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:59.719512  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:59.715231  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:25:59.722455  5561 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
W20260709 12:25:59.725452  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 201435us
I20260709 12:25:59.689633  9185 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:59.594416  8347 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.693434  9190 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:25:59.737295  5632 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 12:25:59.743103  9106 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:25:59.720916  9164 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:25:59.747046  9164 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:25:59.747476  9164 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:59.748817  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:59.749857  5710 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:25:59.740067  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:59.751049  9127 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:59.685700  9111 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.053s
I20260709 12:25:59.754379  9263 ts_tablet_manager.cc:1916] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:25:59.757104  9190 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:25:59.740756  5632 pending_rounds.cc:85] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 4
I20260709 12:25:59.770732  5632 pending_rounds.cc:107] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:25:59.771324  5632 raft_consensus.cc:2889] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:59.772109  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 57004us
W20260709 12:25:59.778440  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.780112  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:25:59.738802  9185 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:25:59.784030  9305 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:25:59.712780  9139 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:59.786419  5710 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
W20260709 12:25:59.815478  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.779682  9285 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:59.824412  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:59.822898  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:25:59.833914  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:59.841645  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 246642us
I20260709 12:25:59.781065  8506 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.844527  5524 leader_election.cc:400] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8.
I20260709 12:25:59.812781  9151 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:25:59.858244  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:25:59.859210  5631 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:25:59.869953  9139 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:59.858956  9170 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:59.873466  8506 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:25:59.875715  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.876803  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:25:59.878121  9282 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:25:59.878674  9282 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.879098  9282 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:25:59.880498  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 65646.3us
I20260709 12:25:59.809832  9120 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.071s
I20260709 12:25:59.873131  9170 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:25:59.867537  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:25:59.884666  6279 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.715s	user 0.004s	sys 0.007s
I20260709 12:25:59.885025  6279 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
W20260709 12:25:59.887463  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 207386us
I20260709 12:25:59.891742  9039 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:25:59.892711  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:59.893210  9135 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:25:59.894312  9282 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:59.894548  9306 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.114s
I20260709 12:25:59.919706  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:59.902347  9170 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:59.862546  9143 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:25:59.903312  5703 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
W20260709 12:25:59.904958  9307 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 12:25:59.926189  5708 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: []
I20260709 12:25:59.861097  8347 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:25:59.911000  5631 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:25:59.859760  9142 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:25:59.917662  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:59.900032  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 83216us
I20260709 12:25:59.883317  9285 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:59.908543  9139 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.909173  9220 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:25:59.929654  5632 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
W20260709 12:25:59.948446  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.954701  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:25:59.916370  9135 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:25:59.958207  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:59.958207  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:25:59.924649  9005 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.601s	user 0.013s	sys 0.004s
I20260709 12:25:59.962508  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:25:59.966521  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.963239  6277 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 6.
W20260709 12:25:59.975351  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45903.7us
I20260709 12:25:59.975114  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:59.980762  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:59.982877  5709 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:25:59.984184  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:25:59.979401  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:25:59.925300  9105 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:00.001746  6277 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:26:00.002265  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:59.966792  9005 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8.
W20260709 12:26:00.006325  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 129575us
I20260709 12:25:59.991016  9106 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:26:00.029086  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 206274us
W20260709 12:26:00.030670  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.045404  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:00.028517  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 147255us
I20260709 12:25:59.992936  9151 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:00.052772  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:25:59.996117  9196 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:25:59.980273  9220 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:00.061379  5701 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:00.005451  9093 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:00.062706  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:25:59.975108  9285 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:00.064781  9309 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:00.065056  9220 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:00.065286  9309 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:00.065712  9309 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:00.067157  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:00.068034  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:00.068949  6277 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:00.069079  9220 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:26:00.069547  9220 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:00.069352  9151 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:00.069900  9220 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:26:00.069962  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:00.070968  9252 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.009s	sys 0.010s
W20260709 12:26:00.071313  9252 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.009s	sys 0.010s
I20260709 12:26:00.072484  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:00.074527  8711 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:26:00.076922  9310 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 12:26:00.079299  9104 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:00.081288  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.081579  9104 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:00.084851  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:00.086552  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:00.087275  9220 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:00.088716  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:00.089376  5705 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:26:00.090929  9220 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:00.091349  5702 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:26:00.091776  5702 pending_rounds.cc:85] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 5
I20260709 12:26:00.092288  5702 pending_rounds.cc:107] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:26:00.092581  5702 raft_consensus.cc:2889] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:00.091710  9220 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 6, Time since last communication: 0.002s
W20260709 12:26:00.094738  9238 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.658s	user 0.012s	sys 0.000s
I20260709 12:26:00.095171  9238 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:00.096232  9185 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:00.098623  9185 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:00.100546  9093 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:00.101476  9105 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:00.048718  9306 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:26:00.107380  8856 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.686s	user 0.008s	sys 0.000s
I20260709 12:26:00.122726  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:25:59.994524  9139 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:00.113534  5601 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:26:00.108464  9183 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.585s	user 0.017s	sys 0.009s
W20260709 12:26:00.125869  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 139477us
I20260709 12:26:00.127240  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:00.130291  5707 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:26:00.121248  5703 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:26:00.099814  9190 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:00.134752  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77840us
I20260709 12:26:00.136770  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:00.141099  6272 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:26:00.143428  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:00.132054  9282 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:00.153494  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:00.154356  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:00.161175  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:00.163620  6273 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:26:00.057407  9135 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:00.138797  9105 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:26:00.169346  9135 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:00.169838  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:00.170173  5628 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 12:26:00.170666  5628 pending_rounds.cc:85] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 8
I20260709 12:26:00.170882  5628 pending_rounds.cc:107] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 12:26:00.126724  9185 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:26:00.171150  5628 raft_consensus.cc:2889] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:00.171631  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:00.173377  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:00.130241  9170 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.026s
W20260709 12:26:00.188764  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44217.4us
W20260709 12:26:00.199798  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.177367  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:00.129523  8856 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:00.157404  9282 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:00.217798  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:00.128599  9320 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.053s
I20260709 12:26:00.218412  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:00.218470  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46858.4us
I20260709 12:26:00.131197  9183 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:00.206867  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:00.550604  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.245985  9143 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:00.386653  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:00.526623  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.156637  9104 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
W20260709 12:26:00.529906  9247 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.815s	user 0.011s	sys 0.010s
I20260709 12:26:00.532655  6272 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:26:00.533198  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:00.246246  9312 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:00.217700  5633 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 11: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:26:00.533836  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.537190  6273 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:26:00.538432  9005 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:00.541920  5703 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:00.543917  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:00.246124  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:00.559777  9261 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 12:26:00.562502  6278 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:00.571585  9183 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:00.832438  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 699642us
W20260709 12:26:00.832500  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 270203us
W20260709 12:26:00.779395  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:00.837015  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:00.572167  6234 leader_election.cc:336] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.528s (SENT)
I20260709 12:26:00.730150  6274 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:00.849682  5710 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
W20260709 12:26:00.550002  9060 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.974s	user 0.018s	sys 0.009s
I20260709 12:26:00.546926  9282 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:00.723212  9304 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.082s	user 0.010s	sys 0.021s
W20260709 12:26:00.864975  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 331748us
W20260709 12:26:00.873083  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 326733us
I20260709 12:26:00.873342  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:00.867147  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 624) took 2037 ms. Trace:
I20260709 12:26:00.869621  6279 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
W20260709 12:26:00.831423  9313 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.000s	sys 0.022s
I20260709 12:26:00.794390  9238 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:00.786237  9170 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:00.875517  5635 rpcz_store.cc:276] 0709 12:25:58.830036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.985875 (+155839us) service_pool.cc:224] Handling call
0709 12:26:00.866908 (+1881033us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b54001cfcd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:00.867063 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":761183616}
I20260709 12:26:00.794003  9337 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:00.894542  9210 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.563548  9309 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:00.895202  9210 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:00.883834  9304 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:00.898659  9304 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:00.898985  9120 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:00.899413  9120 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:00.900904  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:00.902274  9284 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.447s	user 0.011s	sys 0.020s
W20260709 12:26:00.903442  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.904325  9252 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:00.904767  9105 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:26:00.905100  9252 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:00.905529  9105 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:00.906075  9105 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:00.906913  5634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2139) took 2483 ms (client timeout 2087 ms). Trace:
I20260709 12:26:00.906952  9105 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:26:00.907426  5634 rpcz_store.cc:269] 0709 12:25:58.422992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.528155 (+105163us) service_pool.cc:224] Handling call
0709 12:26:00.906695 (+2378540us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b54001dd8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:00.906827 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1489960192}
I20260709 12:26:00.905241  8347 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.717516  9045 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:00.909646  8347 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:00.882993  9313 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.774s	user 0.000s	sys 0.024s
W20260709 12:26:00.917181  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.917062  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:00.866161  9320 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:26:00.926458  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:26:00.921669  6236 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.805s (SENT) [suppressed 28 similar messages]
I20260709 12:26:00.850752  9312 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:00.909981  9139 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:00.969727  9332 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 12:26:00.928256  9005 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.962822  9315 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:26:00.977106  9308 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.246s	user 0.000s	sys 0.005s
I20260709 12:26:00.933264  9285 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:00.954936  9284 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:26:00.937166  9190 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:00.848217  9247 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:01.004247  9304 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:26:01.006148  9284 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:01.007602  9284 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:01.006187  9247 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:00.962456  9143 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:26:01.013293  9164 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.265s	user 0.011s	sys 0.011s
I20260709 12:26:00.918419  9120 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:01.014052  8903 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:26:01.016245  9312 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:00.918598  9337 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:00.918304  9045 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:00.922969  9170 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:01.022397  6234 leader_election.cc:336] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.820s (SENT)
W20260709 12:26:01.035281  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 175559us
I20260709 12:26:00.975072  9263 ts_tablet_manager.cc:1929] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.4
W20260709 12:26:01.030910  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.988600  9315 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:01.035137  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:01.026284  9245 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.493s	user 0.020s	sys 0.012s
I20260709 12:26:01.026674  9143 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:01.029911  9174 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.290s	user 0.017s	sys 0.004s
I20260709 12:26:01.024564  8856 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:00.997081  9005 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:01.043655  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 499807us
I20260709 12:26:01.046454  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:01.028077  9164 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:01.041775  9120 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:01.049856  5675 connection.cc:256] RPC call timeout handler was delayed by -1.1178600639996148s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2315s for a response
I20260709 12:26:01.042132  9143 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:26:01.068833  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 147182us
I20260709 12:26:01.070658  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2087) took 1080 ms. Trace:
I20260709 12:26:01.044574  8711 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:01.072230  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49886.6us
I20260709 12:26:01.047858  9210 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:01.073565  6279 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:01.065675  9263 log.cc:1199] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-2-root/wals/2acd6f5df0204b2295547e021cd8cb9f
I20260709 12:26:01.056416  8783 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:01.077373  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:01.080673  9337 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:01.081875  9045 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:01.077404  6272 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:26:01.086066  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:01.086647  6275 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:01.003907  9252 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:01.074029  9304 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:26:01.073850  5675 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.538s (SENT)
I20260709 12:26:01.084326  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:01.093325  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:01.071655  9320 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:01.097795  6277 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 7: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:01.096233  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:01.087924  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:01.097151  6273 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:01.071553  5708 rpcz_store.cc:276] 0709 12:25:59.986604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.993648 (+  7044us) service_pool.cc:224] Handling call
0709 12:25:59.994064 (+   416us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:00.041544 (+ 47480us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:00.041591 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:00.108789 (+ 67198us) log.cc:844] Serialized 52 byte log entry
0709 12:26:00.335835 (+227046us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:26:00.779828 (+443993us) log.cc:844] Serialized 14 byte log entry
0709 12:26:01.065497 (+285669us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:01.065827 (+   330us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:01.065877 (+    50us) raft_consensus.cc:1685] finished
0709 12:26:01.065893 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:01.066697 (+   804us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":219553,"prepare.queue_time_us":441024,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":117471,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":16488,"thread_start_us":479254,"threads_started":2,"wal-append.queue_time_us":652253}
I20260709 12:26:01.044121  9174 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:01.113207  9261 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:26:01.114343  9263 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38456 (request call id 79) took 3306 ms. Trace:
I20260709 12:26:01.114823  9263 rpcz_store.cc:276] 0709 12:25:57.807834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.808048 (+   214us) service_pool.cc:224] Handling call
0709 12:25:58.349258 (+541210us) ts_tablet_manager.cc:1151] Deleting tablet 2acd6f5df0204b2295547e021cd8cb9f
0709 12:25:58.357940 (+  8682us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:26:00.087847 (+1729907us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:00.963864 (+876017us) tablet_metadata.cc:665] Metadata flushed
0709 12:26:01.114215 (+150351us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":535478,"tablet-delete.queue_time_us":454277,"thread_start_us":99556,"threads_started":1}
W20260709 12:26:01.115116  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.109390  9320 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:01.118170  9320 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:01.120684  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:01.122815  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:01.123431  5710 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:01.125058  5710 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:26:01.125586  9164 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:01.126518  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:01.127786  9170 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:01.082283  8711 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:01.128862  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:01.129701  5710 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:01.096191  9315 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:01.133006  9315 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:01.095448  9210 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.134145  9210 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:01.137518  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:01.138191  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:01.144784  6272 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:01.144747  5708 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:26:01.152315  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:01.153316  5708 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:01.154381  5710 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:01.136221  9210 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:01.091809  8783 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:01.162350  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:01.168928  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 100180us
I20260709 12:26:01.169756  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:01.173808  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 86021.9us
I20260709 12:26:01.150609  9164 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.181913  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:01.183444  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 114666us
I20260709 12:26:01.185022  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:01.185691  9343 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.007s
I20260709 12:26:01.187881  8783 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:01.189066  8783 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:01.193193  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:01.194460  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:01.186960  6234 leader_election.cc:336] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.009s (SENT)
I20260709 12:26:01.096971  9321 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.082s
I20260709 12:26:01.179345  9261 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:26:01.181649  9304 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:26:01.230286  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51756.5us
W20260709 12:26:01.237051  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 50124.6us
I20260709 12:26:01.227413  9238 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:01.248659  9238 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.249385  9238 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:01.179646  9164 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:01.254832  9164 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:01.228540  9185 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:01.255545  9185 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:01.255961  9185 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:01.243744  9261 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:01.229841  9347 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:26:01.257340  9261 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:01.261128  5709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2019) took 1371 ms (client timeout 1642 ms). Trace:
W20260709 12:26:01.261582  5709 rpcz_store.cc:269] 0709 12:25:59.889884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.890112 (+   228us) service_pool.cc:224] Handling call
0709 12:26:01.260942 (+1370830us) spinlock_profiling.cc:243] Waited 38.9 ms on lock 0x7b5400452ed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:01.261065 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":89382656}
I20260709 12:26:01.263043  6237 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:01.265332  9312 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:01.266223  9284 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.266793  9284 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:01.229635  9086 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:26:01.275621  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 81263.7us
I20260709 12:26:01.232801  9304 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.281242  5631 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:01.282413  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:26:01.283461  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.233793  9105 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:01.274710  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:01.288702  5633 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:01.225951  9046 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 12:26:01.268141  9086 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.290180  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:01.292007  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:01.296147  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:01.298764  6275 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:01.293464  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:01.300405  5635 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:01.295536  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:01.276806  9164 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.296846  5629 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:26:01.266266  8506 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:01.310984  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:26:01.313185  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 85622.3us
I20260709 12:26:01.276969  9174 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:01.285892  9304 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:01.322567  8506 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:01.329561  9093 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:01.329681  9185 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:01.330960  5707 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:26:01.332023  9185 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:01.332334  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2102) took 1246 ms. Trace:
I20260709 12:26:01.333503  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 604) took 1288 ms. Trace:
I20260709 12:26:01.414441  5561 rpcz_store.cc:276] 0709 12:26:00.045614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:00.045950 (+   336us) service_pool.cc:224] Handling call
0709 12:26:00.046254 (+   304us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:00.135072 (+ 88818us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:00.135107 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:00.152649 (+ 17542us) log.cc:844] Serialized 26 byte log entry
0709 12:26:00.362297 (+209648us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:00.362354 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:01.333142 (+970788us) spinlock_profiling.cc:243] Waited 56 ms on lock 0x7b54001af254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:01.333185 (+    43us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:01.333230 (+    45us) raft_consensus.cc:1685] finished
0709 12:26:01.333243 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:01.333444 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":98828,"raft.queue_time_us":29442,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":11918,"spinlock_wait_cycles":128904576,"thread_start_us":184812,"threads_started":1,"wal-append.queue_time_us":365123}
I20260709 12:26:01.333552  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1850) took 1412 ms. Trace:
I20260709 12:26:01.336809  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2016) took 1567 ms. Trace:
I20260709 12:26:01.339408  9353 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.433601  5635 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:01.342947  9309 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:01.437601  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:01.344076  9005 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:01.295686  9086 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:01.347105  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:01.443161  5552 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 10
W20260709 12:26:01.349772  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 104694us
W20260709 12:26:01.350054  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.355113  8347 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:01.417191  5701 rpcz_store.cc:276] 0709 12:25:59.769287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.774336 (+  5049us) service_pool.cc:224] Handling call
0709 12:25:59.810232 (+ 35896us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:00.061879 (+251647us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:01.334020 (+1272141us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b54004529d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:01.334169 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":577064704}
I20260709 12:26:01.424650  5632 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 6.
I20260709 12:26:01.413414  5707 rpcz_store.cc:276] 0709 12:26:00.086260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:00.241477 (+155217us) service_pool.cc:224] Handling call
0709 12:26:01.332092 (+1090615us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b54000e0bd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:01.332260 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1144385792}
I20260709 12:26:01.434136  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:01.339865  9284 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:01.437598  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 121560us
I20260709 12:26:01.346153  9143 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:01.244853  9282 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:01.443048  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 91409.5us
I20260709 12:26:01.462900  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:01.293445  9338 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:01.417080  5636 rpcz_store.cc:276] 0709 12:25:59.924910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.930570 (+  5660us) service_pool.cc:224] Handling call
0709 12:25:59.972039 (+ 41469us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:00.162910 (+190871us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:00.162943 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:00.168838 (+  5895us) log.cc:844] Serialized 26 byte log entry
0709 12:26:00.237281 (+ 68443us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:00.237341 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:01.333066 (+1095725us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54001f2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:01.333141 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:01.333204 (+    63us) raft_consensus.cc:1685] finished
0709 12:26:01.333224 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:01.333481 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8356,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"spinlock_wait_cycles":384778496,"thread_start_us":67827,"threads_started":1,"wal-append.queue_time_us":356321}
I20260709 12:26:01.448107  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:01.449220  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:01.359530  9344 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:26:01.450546  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:01.468940  5632 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:26:01.444782  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:01.451740  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:01.471021  5631 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:01.452421  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:01.443120  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:01.449765  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1852) took 1418 ms. Trace:
I20260709 12:26:01.472136  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:01.472433  5630 rpcz_store.cc:276] 0709 12:26:00.031328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.433843 (+1402515us) service_pool.cc:224] Handling call
0709 12:26:01.449697 (+ 15854us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:01.456401  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:01.323798  8506 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:26:01.484462  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.457954  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:01.486114  6276 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 12:26:01.460608  6271 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:01.302960  9209 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.147s
I20260709 12:26:01.473405  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 705) took 1504 ms. Trace:
I20260709 12:26:01.493490  6278 rpcz_store.cc:276] 0709 12:25:59.955594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.963788 (+  8194us) service_pool.cc:224] Handling call
0709 12:26:01.460166 (+1496378us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b54005f8f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:01.460306 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1334895872}
I20260709 12:26:01.481398  5705 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:01.494760  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 732) took 1568 ms. Trace:
I20260709 12:26:01.439726  9327 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.486143  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:01.495160  6274 rpcz_store.cc:276] 0709 12:25:59.926230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.926892 (+   662us) service_pool.cc:224] Handling call
0709 12:26:01.494561 (+1567669us) spinlock_profiling.cc:243] Waited 774 ms on lock 0x7b54002909d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:01.494697 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1780891392}
I20260709 12:26:01.490813  5556 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:01.490680  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:01.481597  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:01.489215  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:01.451618  9093 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.017s
I20260709 12:26:01.505442  5558 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:01.505534  5635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:01.502157  5707 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:01.506361  5635 raft_consensus.cc:606] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:01.506759  5635 raft_consensus.cc:614] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:01.507903  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:01.508424  9209 consensus_queue.cc:1243] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:26:01.508920  9327 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:01.510426  9238 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:01.511121  9143 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:01.445735  9353 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:01.504019  8783 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:01.513306  9353 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:01.513533  8506 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:01.441002  9170 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 12:26:01.453028  9207 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:01.356410  9164 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:01.517298  5554 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:26:01.466152  9338 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.518296  9338 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:01.518708  9164 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:01.519403  5635 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:01.517379  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:01.521358  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:01.523326  5636 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:26:01.524540  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:01.499312  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:01.527129  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:01.527240  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:01.528990  8856 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:01.529618  8856 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:01.530198  8856 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:01.533394  5709 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:26:01.533267  9093 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:01.535063  9120 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:26:01.516374  9207 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:01.535835  9120 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:01.536260  9120 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:01.536692  9120 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:01.538116  9207 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:01.434819  9320 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:01.541437  9320 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:01.529805  5559 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 8: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:01.542900  5560 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:01.543349  6280 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:01.544894  9282 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:01.545760  5443 catalog_manager.cc:5028] TS f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): tablet 2acd6f5df0204b2295547e021cd8cb9f (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:26:01.548460  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:01.553189  6275 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:26:01.499150  6276 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 709) took 1516 ms (client timeout 1989 ms). Trace:
I20260709 12:26:01.544958  9190 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:01.556967  5701 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:26:01.569657  5710 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:01.571168  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 66581.9us
I20260709 12:26:01.572861  5635 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:01.551460  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:01.559079  5633 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:01.489825  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:01.503559  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:01.574734  5708 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:01.575084  5628 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:01.546638  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:01.548597  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:01.576253  6277 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:26:01.554318  6276 rpcz_store.cc:269] 0709 12:25:59.982907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.441896 (+1458989us) service_pool.cc:224] Handling call
0709 12:26:01.499087 (+ 57191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:01.576316  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:01.577044  5628 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:26:01.577497  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 77928.3us
I20260709 12:26:01.580296  6275 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:26:01.584535  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:01.550098  9260 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:01.587575  5555 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:01.453270  9174 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.099s
I20260709 12:26:01.594352  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:01.606575  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:26:01.612378  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 66567.8us
I20260709 12:26:01.613955  5636 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:26:01.615201  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:01.615872  5636 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:01.610299  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:01.630084  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:01.621786  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:01.511123  9327 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:01.645116  5559 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 6: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:01.567871  9238 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:01.641801  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54430.1us
I20260709 12:26:01.649276  5710 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
W20260709 12:26:01.649439  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.656293  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:01.657380  5556 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:26:01.660840  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:01.610512  9343 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:01.627707  9105 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:01.667837  9343 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:01.641245  9260 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:01.668996  9105 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:01.671422  9260 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:01.620023  9127 raft_consensus.cc:2764] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260709 12:26:01.674001  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:01.677012  9352 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:26:01.677225  9046 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:26:01.679464  5704 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260709 12:26:01.680473  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.677039  9360 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:26:01.681245  9360 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:01.681627  9360 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:26:01.685344  9351 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:26:01.676020  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:01.620291  9353 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:26:01.668030  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:01.692791  5702 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:01.577865  9320 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.078s
I20260709 12:26:01.705721  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:01.706961  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:01.695896  6275 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46486
I20260709 12:26:01.696480  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:01.689868  9086 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:01.711815  6275 raft_consensus.cc:606] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:01.733970  5628 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:26:01.746884  6275 raft_consensus.cc:614] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:01.655395  9190 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:01.753914  5628 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:01.756896  5632 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 12:26:01.757287  5632 pending_rounds.cc:85] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 4
I20260709 12:26:01.757507  5632 pending_rounds.cc:107] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 12:26:01.757750  5632 raft_consensus.cc:2889] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:01.758903  5631 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
W20260709 12:26:01.763049  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 209163us
W20260709 12:26:01.763922  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 87986.6us
I20260709 12:26:01.655819  8783 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.106s
W20260709 12:26:01.764763  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.655192  9164 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.099s
W20260709 12:26:01.775385  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.753940  9093 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:01.753700  9252 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.776288  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:01.783737  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:01.758527  9127 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:01.775133  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:01.787571  5628 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:26:01.804625  5631 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:26:01.751576  9327 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:26:01.844018  5552 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:01.789711  9127 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:01.871877  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:01.890717  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 107053us
W20260709 12:26:01.891706  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56524.5us
I20260709 12:26:01.826870  9093 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:01.821460  9252 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:01.855347  9127 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:01.845899  9361 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:26:02.002877  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 157553us
I20260709 12:26:01.853384  9046 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:01.820076  9143 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:26:01.995575  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.853546  9190 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:01.998271  5628 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:02.008864  6275 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:26:02.016707  9093 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.017360  9261 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.017940  9261 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.018615  9086 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:02.019418  9143 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:02.019830  9261 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:02.020505  9143 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.023168  6275 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46486
I20260709 12:26:02.023767  6275 raft_consensus.cc:606] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:02.024086  6275 raft_consensus.cc:614] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:02.019544  9120 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:02.026149  9120 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:02.026813  9343 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:02.028558  9174 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:01.998660  9170 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:02.075505  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:02.011919  9331 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.455s	user 0.009s	sys 0.010s
I20260709 12:26:02.077111  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:02.078519  9174 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:02.078675  9282 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:02.079600  6277 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46486
I20260709 12:26:02.080158  6277 raft_consensus.cc:606] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:02.080507  6277 raft_consensus.cc:614] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:26:02.032239  9304 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.711s	user 0.005s	sys 0.006s
I20260709 12:26:02.070645  5702 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
W20260709 12:26:02.081189  8506 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.567s	user 0.006s	sys 0.010s
I20260709 12:26:02.096724  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:02.011934  9185 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20260709 12:26:02.013569  9320 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.080s
I20260709 12:26:02.075438  9170 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:02.118827  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:02.120664  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:02.118866  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:02.122373  8711 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:02.123229  9170 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:02.124529  9170 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:02.091173  9046 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:02.097414  9105 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:02.171952  5708 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.597s	user 0.011s	sys 0.000s
I20260709 12:26:02.172703  5708 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:26:02.090876  9127 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.092595  9304 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:02.150417  9320 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:02.190130  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.196631  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:02.211030  9104 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:02.211736  9104 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:02.213256  5557 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:26:02.166608  9046 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.221159  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:02.198022  9105 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.240779  6274 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:02.202013  9320 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.283424  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:02.284322  6275 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:02.286559  6274 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20260709 12:26:02.288731  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.289162  5708 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 12:26:02.289587  5708 pending_rounds.cc:85] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 8
I20260709 12:26:02.289806  5708 pending_rounds.cc:107] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 12:26:02.290298  5708 raft_consensus.cc:2889] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:02.299031  5708 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 12:26:02.286607  9371 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:26:02.304070  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:02.307886  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 230826us
I20260709 12:26:02.308746  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:02.317941  6275 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:02.319504  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:02.321769  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.324493  5702 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:02.320443  5526 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:02.327459  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:02.330331  6275 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20260709 12:26:02.332962  9373 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260709 12:26:02.343824  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.344897  5703 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
W20260709 12:26:02.321585  9359 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.225s	user 0.001s	sys 0.001s
I20260709 12:26:02.268105  9104 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.378189  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.320117  9364 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:02.338553  9086 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:02.382941  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:02.388590  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:02.389590  5559 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:02.393353  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.325484  9353 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.059s
I20260709 12:26:02.394407  5703 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:02.380769  9315 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.247s	user 0.016s	sys 0.000s
I20260709 12:26:02.394352  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:02.399421  5554 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:02.395746  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:02.400142  5560 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:02.363317  9127 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:02.325583  9120 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.059s
I20260709 12:26:02.399531  5556 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:26:02.398149  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:02.406457  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.402290  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:02.408779  5555 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:02.409843  5552 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: []
W20260709 12:26:02.401230  9372 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.209s	user 0.000s	sys 0.002s
W20260709 12:26:02.404699  9360 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.723s	user 0.010s	sys 0.004s
I20260709 12:26:02.402176  9086 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:02.422850  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 742) took 1195 ms. Trace:
I20260709 12:26:02.424814  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.421801  5554 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:26:02.423252  6273 rpcz_store.cc:276] 0709 12:26:01.227218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.227376 (+   158us) service_pool.cc:224] Handling call
0709 12:26:01.232120 (+  4744us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:01.239026 (+  6906us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:01.239073 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:01.276265 (+ 37192us) log.cc:844] Serialized 52 byte log entry
0709 12:26:01.749700 (+473435us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:01.749754 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:01.749900 (+   146us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:02.422634 (+672734us) raft_consensus.cc:1685] finished
0709 12:26:02.422653 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:02.422790 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10686,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":31789,"thread_start_us":385235,"threads_started":1,"wal-append.queue_time_us":551799}
I20260709 12:26:02.428722  5556 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:26:02.435767  9086 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.446236  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:02.447548  9375 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 12:26:02.453009  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:02.488746  6275 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
W20260709 12:26:02.484611  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 980156us
I20260709 12:26:02.496905  5556 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:26:02.496198  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:02.468994  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 818725us
I20260709 12:26:02.505913  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:02.435803  9377 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:26:02.508127  5707 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:02.506222  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:02.509332  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 735) took 2516 ms. Trace:
I20260709 12:26:02.509951  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:02.434059  9320 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:02.485373  9151 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:02.507889  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:02.514060  6271 rpcz_store.cc:276] 0709 12:25:59.990098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:00.029377 (+ 39279us) service_pool.cc:224] Handling call
0709 12:26:01.460430 (+1431053us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:01.487243 (+ 26813us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:02.505829 (+1018586us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54005f8f50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:02.506002 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3234513664}
I20260709 12:26:02.485179  9105 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:02.421465  9360 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:02.512490  9312 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:02.518282  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:02.531680  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.539867  5708 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:26:02.537106  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:02.531723  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.534325  5703 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:26:02.535276  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:02.535936  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:02.532948  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:02.543304  5705 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:02.544265  5704 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: []
I20260709 12:26:02.547798  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.548635  5701 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:26:02.555663  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:02.556514  6271 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:26:02.557032  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:02.564046  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45857.6us
I20260709 12:26:02.568146  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:02.527989  9341 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:02.570784  9341 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:02.571386  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:02.572640  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:02.572827  9338 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.054s	user 0.014s	sys 0.000s
I20260709 12:26:02.573339  9338 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.573882  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.574751  5556 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:02.575520  9338 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:02.571766  9341 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:02.576426  8856 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.046s	user 0.012s	sys 0.004s
I20260709 12:26:02.576866  8856 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.577553  9360 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:02.578496  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.575726  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:02.578994  8856 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:02.579398  5556 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:02.579654  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.586699  5552 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 12:26:02.519317  9151 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:02.580279  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:02.582397  9005 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:02.584456  5709 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 6.
I20260709 12:26:02.584980  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:02.585942  6274 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:02.575551  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1916) took 1254 ms. Trace:
I20260709 12:26:02.598498  5557 rpcz_store.cc:276] 0709 12:26:01.321525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.321686 (+   161us) service_pool.cc:224] Handling call
0709 12:26:01.322018 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:02.214098 (+892080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:02.575318 (+361220us) spinlock_profiling.cc:243] Waited 886 ms on lock 0x7b54000eb5d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:02.575494 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2037972224}
I20260709 12:26:02.600098  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:02.604434  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.605114  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:02.605788  6279 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:02.609376  6273 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:02.611028  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:02.612782  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44690.7us
I20260709 12:26:02.595930  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:02.515570  9135 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260709 12:26:02.627735  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:02.629855  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:02.635288  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:02.635957  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.597612  9005 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:02.606753  9321 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:02.640758  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.676848  5710 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 7: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:02.643642  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.644634  5709 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:02.656277  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 60484.9us
I20260709 12:26:02.680858  6280 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:02.681962  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:02.665055  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:02.683338  6273 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:02.664492  5707 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 9. Current term is 10. Ops: [7.8-9.9]
I20260709 12:26:02.640329  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 741) took 1464 ms. Trace:
I20260709 12:26:02.690019  9388 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.690672  9388 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:02.667443  9387 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:02.692757  9387 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:02.693082  9387 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:02.693277  9304 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:02.696671  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:02.697399  5703 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:26:02.699438  9005 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.700894  9005 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:02.702077  9365 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.951s	user 0.010s	sys 0.012s
I20260709 12:26:02.705413  5556 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:02.708088  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1853) took 1361 ms. Trace:
I20260709 12:26:02.627698  9190 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.078s
I20260709 12:26:02.713254  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:02.683158  9376 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.709106  5634 rpcz_store.cc:276] 0709 12:26:01.346578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.346709 (+   131us) service_pool.cc:224] Handling call
0709 12:26:01.347053 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:01.347445 (+   392us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:01.347472 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:01.347959 (+   487us) log.cc:844] Serialized 26 byte log entry
0709 12:26:01.629001 (+281042us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:26:01.629044 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:01.629127 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:02.701783 (+1072656us) spinlock_profiling.cc:243] Waited 7.39 ms on lock 0x7b240029c018. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff740b9c4 00007f4ff7409acb 00007f4ff75c4dc9 00007f4ff75c4a55 00007f4ff75c409d 00007f4ffa1f610e 00007f4ffa20d88c 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221
0709 12:26:02.705496 (+  3713us) raft_consensus.cc:1685] finished
0709 12:26:02.705514 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:02.708023 (+  2509us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7068,"raft.queue_time_us":90887,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":7458,"spinlock_wait_cycles":17006464,"thread_start_us":280592,"threads_started":1,"wal-append.queue_time_us":402215}
I20260709 12:26:02.707264  9321 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:02.720963  9046 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:02.725132  5708 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:26:02.726541  5705 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:02.729853  9260 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:02.730391  9260 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:02.730881  9321 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.718978  9376 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:02.731456  9260 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.736817  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:02.737848  5556 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:26:02.738473  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.738858  9376 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:02.688025  6272 rpcz_store.cc:276] 0709 12:26:01.176074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.176218 (+   144us) service_pool.cc:224] Handling call
0709 12:26:01.181643 (+  5425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:01.181860 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:26:01.184970 (+  3110us) log.cc:844] Serialized 14 byte log entry
0709 12:26:02.418578 (+1233608us) spinlock_profiling.cc:243] Waited 990 us on lock 0x7b18003c75a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:26:02.639883 (+221305us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:02.639953 (+    70us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:02.640013 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:02.640097 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:02.640255 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1907,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":108,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":20507,"spinlock_wait_cycles":2277248,"thread_start_us":220928,"threads_started":1,"wal-append.queue_time_us":1466026}
I20260709 12:26:02.739612  9260 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:02.748551  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1918) took 1184 ms. Trace:
I20260709 12:26:02.749194  5553 rpcz_store.cc:276] 0709 12:26:01.558477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.560265 (+  1788us) service_pool.cc:224] Handling call
0709 12:26:02.742511 (+1182246us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b54000ea1d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:02.742709 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1015239936}
W20260709 12:26:02.752285  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 443592us
I20260709 12:26:02.752856  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:02.753168  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:02.754520  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:02.750869  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1859) took 1402 ms. Trace:
W20260709 12:26:02.758170  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 168668us
I20260709 12:26:02.759370  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:02.756047  5630 rpcz_store.cc:276] 0709 12:26:01.351799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.491833 (+140034us) service_pool.cc:224] Handling call
0709 12:26:01.498622 (+  6789us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:01.552862 (+ 54240us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:26:01.760737 (+207875us) log.cc:844] Serialized 14 byte log entry
0709 12:26:02.099346 (+338609us) log.cc:844] Serialized 14 byte log entry
0709 12:26:02.452193 (+352847us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b18003c5140. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1
0709 12:26:02.750240 (+298047us) spinlock_profiling.cc:243] Waited 5.44 ms on lock 0x7b18003c5140. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:26:02.750446 (+   206us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:02.750507 (+    61us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:26:02.750571 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:02.750649 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:02.750799 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":594317,"prepare.queue_time_us":459921,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"raft.queue_time_us":655402,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":18840,"spinlock_wait_cycles":105223424,"thread_start_us":444513,"threads_started":2,"wal-append.queue_time_us":410345}
I20260709 12:26:02.776057  6276 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 12:26:02.776878  8903 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:02.779551  9135 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:02.780323  9312 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:26:02.777837  9379 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.573s	user 0.004s	sys 0.011s
I20260709 12:26:02.795212  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:02.795610  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 349560us
I20260709 12:26:02.805195  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:02.802227  9404 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:02.805672  9304 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.803149  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:02.771832  9143 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:26:02.771562  8783 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:02.771147  9046 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:02.815860  5556 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 6. Current term is 7. Ops: []
I20260709 12:26:02.819907  5553 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:02.821044  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:02.822698  5559 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: []
I20260709 12:26:02.782924  9315 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 12:26:02.847033  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:02.806701  9170 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:02.838469  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1864) took 1308 ms. Trace:
I20260709 12:26:02.854872  5553 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:02.854147  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:02.852905  5636 rpcz_store.cc:276] 0709 12:26:01.530339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.633742 (+103403us) service_pool.cc:224] Handling call
0709 12:26:01.634511 (+   769us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:01.688439 (+ 53928us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:26:01.992901 (+304462us) log.cc:844] Serialized 14 byte log entry
0709 12:26:02.697399 (+704498us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:02.773139 (+ 75740us) log.cc:844] Serialized 14 byte log entry
0709 12:26:02.837590 (+ 64451us) log.cc:844] Serialized 14 byte log entry
0709 12:26:02.837784 (+   194us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:02.837838 (+    54us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:02.838001 (+   163us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:02.838133 (+   132us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:02.838392 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":616635,"prepare.queue_time_us":348989,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":11091,"raft.queue_time_us":619655,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":9850,"spinlock_wait_cycles":51115008,"thread_start_us":450002,"threads_started":3,"wal-append.queue_time_us":196860}
I20260709 12:26:02.803159  9190 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 12:26:02.889216  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:02.891798  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 86746.7us
I20260709 12:26:02.826566  9046 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:02.893098  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:02.889983  5636 raft_consensus.cc:2376] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12. Re-sending same reply.
I20260709 12:26:02.806342  9304 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:02.893252  9394 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:26:02.844017  9321 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:02.904326  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:02.905774  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:02.904331  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:02.844529  9388 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:02.931152  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:02.936887  5630 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:26:02.939919  9315 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:02.940392  9315 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:02.940547  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:26:02.940806  9315 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:02.942561  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:02.853755  9143 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:02.891076  9312 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:02.932127  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:02.883268  9135 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:02.934074  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:02.951871  5633 raft_consensus.cc:2376] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7. Re-sending same reply.
I20260709 12:26:02.934283  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:02.953642  9120 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:02.954061  9120 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:02.954977  9120 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.955983  9120 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:02.957330  9143 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:02.959239  9252 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:02.960583  9252 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:02.961128  9252 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:02.963197  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:02.964222  5631 raft_consensus.cc:2376] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9. Re-sending same reply.
W20260709 12:26:02.965315  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 162339us
I20260709 12:26:02.933398  5632 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:02.951897  5628 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:02.897022  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:02.933245  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:02.947209  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:02.969520  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72551.7us
W20260709 12:26:02.969940  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.970543  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:02.971866  5559 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:26:02.883880  8783 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:02.891528  9220 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.897503  9046 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:02.898277  9338 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:02.901069  9285 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:26:02.905074  9379 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:26:02.985371  9336 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:02.993099  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:02.995752  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:02.996654  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.999725  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:02.997012  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.002151  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.004551  5705 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:03.005241  5553 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:03.005671  5559 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 8: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:03.011147  5705 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:26:03.011345  6274 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:26:03.012173  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:03.012874  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 187812us
W20260709 12:26:03.013759  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.015931  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:03.016315  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:03.019657  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:03.019769  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:03.019820  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:03.020874  5633 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:03.016784  5706 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:02.998383  9285 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:03.025565  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.026788  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.029069  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:03.043426  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72983.1us
I20260709 12:26:03.003993  9135 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:03.044744  5703 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 9 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260709 12:26:03.046401  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.006604  9411 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.992480  9220 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:03.060864  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.062203  5559 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:26:03.028262  5629 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:26:03.056777  5675 leader_election.cc:400] T 2acd6f5df0204b2295547e021cd8cb9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9.
W20260709 12:26:03.071897  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.075928  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:03.042848  9285 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:03.080135  6274 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:03.080461  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.084785  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.050251  9410 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.056123  9364 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.090345  5553 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:26:03.091280  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:03.091912  6278 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:03.091507  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 59429.9us
I20260709 12:26:03.041210  9338 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 12:26:03.093127  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:03.044896  9207 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260709 12:26:03.095732  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.097472  5561 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:03.101963  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 647) took 1604 ms. Trace:
I20260709 12:26:03.102337  5635 rpcz_store.cc:276] 0709 12:26:01.498005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.507642 (+  9637us) service_pool.cc:224] Handling call
0709 12:26:03.098823 (+1591181us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b5400241154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:03.101904 (+  3081us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25074048}
I20260709 12:26:03.083370  9127 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.074122  8903 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.022s
I20260709 12:26:03.165207  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:03.167788  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 111284us
I20260709 12:26:03.165639  5553 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 11. Current term is 12. Ops: []
W20260709 12:26:03.166416  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 62102.4us
I20260709 12:26:03.084750  9378 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.079s
I20260709 12:26:03.095249  9364 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.182237  9304 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:03.183818  9364 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:03.185482  9220 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:03.186491  9220 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:03.073596  9411 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:03.082000  9105 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.082s
W20260709 12:26:03.186123  9407 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.241s	user 0.001s	sys 0.000s
I20260709 12:26:03.170804  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.176179  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:03.077873  8783 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:03.172951  9086 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:03.174279  9399 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.223s	user 0.001s	sys 0.000s
I20260709 12:26:03.175781  5561 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
W20260709 12:26:03.220887  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 127863us
I20260709 12:26:03.168594  9127 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:03.211330  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:03.233842  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 59134.1us
I20260709 12:26:03.234984  5633 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:03.096855  9410 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:03.240288  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.243396  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:03.247258  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 393498us
I20260709 12:26:03.249024  5632 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:26:03.256034  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:03.258888  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:03.260042  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:03.260629  5631 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:26:03.262907  9422 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:26:03.248164  9416 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:26:03.265596  9127 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:03.267730  9411 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:03.268580  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:03.274670  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:03.278574  9252 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:03.282160  9420 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:26:03.282994  9420 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:03.252379  9260 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.507s	user 0.011s	sys 0.002s
I20260709 12:26:03.286213  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.283921  9420 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:03.261621  9315 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.290081  5709 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:03.291006  9315 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:03.291131  9405 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.242s	user 0.003s	sys 0.000s
I20260709 12:26:03.291574  8903 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:26:03.259541  5632 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:26:03.296269  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.296375  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:03.298219  5633 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:03.261229  9086 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:03.299461  9416 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:26:03.301297  9416 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:26:03.301764  9416 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:03.302882  9416 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:03.305080  5634 raft_consensus.cc:2410] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:26:03.305639  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:03.310129  6278 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
W20260709 12:26:03.312986  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 138364us
I20260709 12:26:03.313426  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:03.315100  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.316426  5560 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 7: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:03.314792  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:03.287854  9285 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:03.321135  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 59213.2us
I20260709 12:26:03.193217  9387 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.322815  9414 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 12:26:03.333724  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:03.335659  5635 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 11: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:03.335969  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:03.338985  5557 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:03.336593  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.336719  5634 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:26:03.333925  9414 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.005s
I20260709 12:26:03.339721  5560 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:26:03.352660  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:03.353039  5556 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:26:03.353399  5633 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:03.355095  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:03.359865  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:03.360730  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.361462  5553 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:03.362514  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:03.362808  9336 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:03.365197  5552 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:26:03.364249  5553 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:26:03.330777  9315 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 12:26:03.372390  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.377194  5631 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260709 12:26:03.376034  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:03.349607  9046 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.378276  9414 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:03.429106  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:03.423468  5560 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:26:03.379565  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.378690  6272 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:03.381173  5559 raft_consensus.cc:2410] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:26:03.379635  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.435737  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:03.462502  5556 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:03.462131  5558 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:03.456318  9416 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:26:03.371345  9005 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:26:03.367267  9426 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:26:03.483709  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 160906us
I20260709 12:26:03.496973  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:03.498317  5631 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:26:03.422055  9086 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:03.483245  5555 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:03.462875  9312 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:26:03.504652  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 115571us
I20260709 12:26:03.371526  9428 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.137s
I20260709 12:26:03.513229  9434 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.513823  9434 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.426384  9418 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.137s
I20260709 12:26:03.517541  9418 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:03.517975  9418 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:03.518963  9418 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:03.521652  9386 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.522420  9386 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.506076  9411 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 12:26:03.525811  9411 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:03.526074  9425 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.526268  9411 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:03.526774  9425 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.534358  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:03.379834  9105 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.128s
I20260709 12:26:03.499163  9005 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:03.547830  9426 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:03.572407  9429 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:26:03.571868  9426 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.379695  9304 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.131s
I20260709 12:26:03.428459  9252 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
I20260709 12:26:03.527323  8903 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:03.595458  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:03.599184  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:03.599689  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 220997us
W20260709 12:26:03.601719  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 225718us
I20260709 12:26:03.605027  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:03.605126  5524 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:03.606747  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.606698  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:03.540529  9086 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:03.594448  9418 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:03.574419  9005 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:03.625324  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:03.630493  5631 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:26:03.594746  9312 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.623008  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.635766  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:03.636859  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:03.600555  9434 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:03.633901  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:03.615669  9127 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:03.646477  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:03.647611  5635 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:03.527436  9411 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.634047  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:03.598943  9334 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:03.632161  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 78033.6us
I20260709 12:26:03.633796  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:03.638674  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:03.639488  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:03.599462  8903 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:03.635924  5556 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:03.652881  5702 raft_consensus.cc:2376] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10. Re-sending same reply.
I20260709 12:26:03.653625  6276 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:26:03.650751  6278 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:03.606000  9414 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.042s
I20260709 12:26:03.656191  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.663389  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:03.628758  9285 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:03.668147  5558 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
W20260709 12:26:03.573957  9408 consensus_queue.cc:1175] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" responder_term: 11 status { last_received { term: 11 index: 7 } last_committed_idx: 6 last_received_current_leader { term: 11 index: 7 } }
I20260709 12:26:03.685174  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:03.688231  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:03.686841  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:03.689157  5703 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:03.689663  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.690620  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:03.652954  9260 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:03.656780  9086 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:26:03.691461  5702 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:03.694551  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:03.692941  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:03.698462  5556 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
W20260709 12:26:03.693533  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 88580.1us
I20260709 12:26:03.700057  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:03.651269  9105 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:26:03.657853  9387 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:03.695645  5526 leader_election.cc:336] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.653s (SENT)
I20260709 12:26:03.715993  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:03.709364  5702 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:26:03.718477  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:03.644462  9312 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.694269  9411 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:03.692931  9260 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:03.697767  9086 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:03.709424  9127 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:26:03.691087  9438 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:03.726207  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:03.737840  5635 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:26:03.708938  9105 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:03.724442  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 117826us
I20260709 12:26:03.720258  9434 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:03.665107  9334 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.723397  5702 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:03.721876  9046 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:03.743440  5702 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:03.746017  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44257.6us
I20260709 12:26:03.742278  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:03.745555  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:03.730438  9260 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:03.733641  9086 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:26:03.731262  9304 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:26:03.752677  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:03.758481  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:03.758615  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:03.760818  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 97553.1us
I20260709 12:26:03.762441  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:03.763036  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:03.765507  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:03.746073  9438 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:03.748032  9434 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:03.776589  9437 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.777200  9437 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.777427  9434 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.779703  9437 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:03.783360  9105 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:03.786204  5709 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:26:03.756445  9170 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:03.787569  9386 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:03.768159  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 72545.3us
I20260709 12:26:03.732079  9252 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
W20260709 12:26:03.754153  9374 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.408s	user 0.000s	sys 0.005s
I20260709 12:26:03.770740  5633 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: []
I20260709 12:26:03.720607  8903 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.774734  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2038) took 1239 ms. Trace:
I20260709 12:26:03.775218  9440 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:26:03.776270  9441 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:26:03.786047  9105 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:26:03.788079  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.788282  5709 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:26:03.802026  5560 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:03.766873  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:03.850189  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 83428.5us
I20260709 12:26:03.850925  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:03.759665  9285 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:26:03.836423  5526 leader_election.cc:336] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.658s (SENT)
I20260709 12:26:03.769789  9143 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:03.848261  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 96949us
I20260709 12:26:03.752120  9420 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
I20260709 12:26:03.767182  9424 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:26:03.852773  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:03.879182  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 77446.5us
I20260709 12:26:03.882160  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:03.855139  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:03.866279  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:03.885062  5633 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:03.844970  5708 rpcz_store.cc:276] 0709 12:26:02.535504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:02.540614 (+  5110us) service_pool.cc:224] Handling call
0709 12:26:02.569551 (+ 28937us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:02.725740 (+156189us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:03.774326 (+1048586us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b5400324454. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:03.774646 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":356852992}
W20260709 12:26:03.874461  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.852773  9170 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:03.872815  8783 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:03.880200  9321 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:26:03.905480  5526 leader_election.cc:336] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.806s (SENT)
I20260709 12:26:03.909009  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:03.885564  8903 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:03.886015  9386 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:03.923164  5633 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:26:03.922009  9436 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 12:26:03.937150  9447 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 12:26:03.939183  5635 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:26:03.942013  9151 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:03.942158  9438 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.943234  9151 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:03.906288  9127 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:26:03.946110  9383 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:03.895989  9450 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 12:26:03.948911  9334 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:03.949774  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:03.951035  9312 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:03.900665  9376 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:03.959821  9376 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:03.958961  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 103917us
I20260709 12:26:03.962711  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:03.968290  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:03.970141  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:03.972716  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 90683.2us
I20260709 12:26:03.973519  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:03.931679  9304 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:03.922881  9448 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:03.982627  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 77191us
I20260709 12:26:03.953114  8783 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.958534  9170 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:26:03.992058  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:03.993460  5635 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 12:26:03.994383  9104 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:26:03.995117  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45433.7us
I20260709 12:26:03.996424  5635 pending_rounds.cc:85] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 5
I20260709 12:26:03.985576  5703 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:26:03.990273  9448 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.996868  9420 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:04.006875  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:04.007517  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:04.009037  5634 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: [7.7-9.8]
W20260709 12:26:04.031288  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.011776  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:04.013854  6237 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.013386  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.034386  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:04.003419  5635 pending_rounds.cc:107] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 10.6
W20260709 12:26:04.037288  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 67411.2us
I20260709 12:26:04.035712  5635 raft_consensus.cc:2889] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:04.038900  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:03.996296  9450 consensus_queue.cc:1243] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:26:04.043257  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.006150  9448 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:04.021355  9420 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:04.109521  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:04.106597  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 66320.3us
I20260709 12:26:04.010600  9252 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:03.963692  9376 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:04.093528  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 80202.4us
W20260709 12:26:04.114825  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 71688.3us
W20260709 12:26:04.108572  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54639.1us
I20260709 12:26:04.115901  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.116293  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:04.117421  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.042737  9321 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:03.992619  8783 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:03.990532  9304 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:04.118454  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:04.002317  9463 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:26:04.107501  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 85972.4us
I20260709 12:26:04.010512  9447 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:26:04.124066  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:04.125792  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:04.091706  9434 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:04.015569  9260 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:04.008792  9453 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:26:04.075884  9386 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:04.038749  9341 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.105s
I20260709 12:26:04.151346  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.016705  9086 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:04.076987  9360 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:04.155306  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:04.114249  9383 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.193841  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:04.194568  5634 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:04.145695  9093 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:04.148111  9463 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:04.199241  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44036.6us
I20260709 12:26:04.153347  9387 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:26:04.204200  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52952.1us
I20260709 12:26:04.205222  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.208803  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:04.182963  9170 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:26:04.152112  9447 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:04.155309  9321 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:04.214919  5628 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: []
W20260709 12:26:04.220042  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63343.6us
I20260709 12:26:04.222184  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:04.199441  9438 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:26:04.223882  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 108047us
I20260709 12:26:04.240041  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.225247  9312 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:26:04.227015  9452 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:04.246404  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:04.247838  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:04.193625  8856 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 12:26:04.198586  9093 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:04.207007  9252 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:04.200498  8903 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:26:04.207953  9463 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:04.238108  6279 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:04.238842  6277 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:04.250605  6280 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:04.218402  9447 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:26:04.251228  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.249528  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.248847  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.220721  9386 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:04.198905  9360 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:04.255208  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.243557  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:04.268227  6271 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:04.222671  9120 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:26:04.253141  9220 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260709 12:26:04.241513  9341 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:04.238463  9453 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:26:04.241641  9438 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:04.251492  9338 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:04.274191  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.275295  6275 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:04.272312  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.197559  9383 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:04.253868  9282 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:04.250286  9457 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:04.288645  9363 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:26:04.304016  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 155380us
I20260709 12:26:04.274619  9453 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:04.259348  9378 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:26:04.328945  5702 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:04.330338  5702 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:04.320312  9461 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:26:04.331662  9461 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:04.286217  9338 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:04.267802  9386 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:04.277946  9387 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:04.276213  9438 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:26:04.344216  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 72011.6us
W20260709 12:26:04.346216  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.325727  9282 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:04.288905  9151 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:26:04.337611  9338 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:04.258530  9321 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:04.290136  9434 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.046s
I20260709 12:26:04.346287  9466 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.378710  9466 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:04.320524  9460 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.226s
I20260709 12:26:04.343393  9387 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:04.338353  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:04.415206  5634 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:26:04.352399  5702 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:26:04.372619  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 754) took 1362 ms. Trace:
W20260709 12:26:04.389506  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 140736us
I20260709 12:26:04.394954  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.416879  6272 rpcz_store.cc:276] 0709 12:26:03.010571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.010709 (+   138us) service_pool.cc:224] Handling call
0709 12:26:03.016050 (+  5341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:03.460603 (+444553us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:04.372305 (+911702us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b5400773154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:04.372548 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":833521920}
W20260709 12:26:04.373667  9464 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
W20260709 12:26:04.377961  9470 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
W20260709 12:26:04.377168  9468 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
W20260709 12:26:04.374125  9451 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.251s	user 0.000s	sys 0.001s
I20260709 12:26:04.426337  9151 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:04.430225  9005 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.786s	user 0.008s	sys 0.005s
W20260709 12:26:04.369413  9411 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.639s	user 0.012s	sys 0.000s
I20260709 12:26:04.343129  9386 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:04.368309  9086 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:04.358116  9364 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:04.374948  9260 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:04.418998  9387 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:04.450364  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.376233  9304 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:04.428180  8783 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:04.424839  9108 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
W20260709 12:26:04.460238  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 138917us
I20260709 12:26:04.421722  9434 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:04.374863  9472 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:26:04.426398  9363 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:26:04.452802  9462 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.270s	user 0.001s	sys 0.001s
I20260709 12:26:04.374918  9444 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.245s
I20260709 12:26:04.442447  9252 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:04.446312  9460 consensus_queue.cc:1243] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:26:04.459112  9151 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:04.481205  9427 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.233s	user 0.002s	sys 0.002s
I20260709 12:26:04.424431  8903 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260709 12:26:04.374755  9469 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.241s
I20260709 12:26:04.459746  9364 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:04.500833  5704 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:26:04.477674  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:04.420539  9093 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:04.485378  9086 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:26:04.506932  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 232832us
I20260709 12:26:04.451395  9005 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:04.365460  9453 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:04.493093  9340 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:04.519618  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:04.540656  9340 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:04.541811  9340 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:04.542840  9340 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:04.506855  9466 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:04.528903  9093 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:04.544478  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:04.545121  5702 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:04.501644  9383 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:04.529944  8903 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.547271  8903 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:04.547657  9453 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:26:04.549400  8903 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:04.549966  8903 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:04.550800  9489 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:04.531482  9005 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:04.530992  9364 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:04.491500  9378 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
W20260709 12:26:04.528769  9465 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.274s	user 0.001s	sys 0.001s
I20260709 12:26:04.535881  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:04.520180  9445 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.537743  5553 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:26:04.471186  9481 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260709 12:26:04.540241  9252 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:04.538331  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:04.529181  5561 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
W20260709 12:26:04.667222  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 227818us
I20260709 12:26:04.566250  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:04.655072  5556 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: []
W20260709 12:26:04.657694  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.659381  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:04.668561  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:04.660882  5560 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:04.538331  9480 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:26:04.673065  5552 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:26:04.646991  9360 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:04.677630  9151 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:04.678639  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 201051us
I20260709 12:26:04.679428  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:04.680657  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:04.681456  5703 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:26:04.672058  9480 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:04.652966  8783 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.679322  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:04.655215  9105 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.783s	user 0.011s	sys 0.002s
I20260709 12:26:04.660032  9312 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.509222  9434 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:04.707321  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:04.708484  5710 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:04.696003  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:04.719515  5628 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:26:04.725648  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46422us
I20260709 12:26:04.726374  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:04.681304  9466 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.697891  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:04.711216  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2135) took 1087 ms. Trace:
I20260709 12:26:04.711722  5553 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:26:04.670929  9364 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:04.712466  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:04.749127  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.743324  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:04.744637  5708 rpcz_store.cc:276] 0709 12:26:03.624059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.932686 (+308627us) service_pool.cc:224] Handling call
0709 12:26:03.933405 (+   719us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:03.933870 (+   465us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:26:04.451923 (+518053us) log.cc:844] Serialized 14 byte log entry
0709 12:26:04.710750 (+258827us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:04.710826 (+    76us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:04.710897 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:04.710995 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:04.711144 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45240,"prepare.queue_time_us":62033,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":45446,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":16227,"thread_start_us":710512,"threads_started":2}
I20260709 12:26:04.750545  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:04.754685  5628 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 7.
I20260709 12:26:04.751160  5710 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:26:04.756167  5703 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:26:04.757280  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:04.724439  9455 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:04.758354  5708 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:04.764401  9456 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.765046  9456 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:04.767374  9456 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:04.701007  9127 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 12:26:04.672313  9445 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:04.769677  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:04.736742  9418 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.102s	user 0.014s	sys 0.000s
I20260709 12:26:04.771901  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:04.772619  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:04.736112  9486 raft_consensus.cc:2764] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:26:04.777009  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 610) took 1154 ms. Trace:
I20260709 12:26:04.712412  8783 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:04.718267  9312 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:04.790660  9457 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:26:04.795334  9475 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:26:04.798775  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:04.789177  9143 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.912s	user 0.012s	sys 0.003s
I20260709 12:26:04.786347  5706 rpcz_store.cc:276] 0709 12:26:03.622237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.719261 (+ 97024us) service_pool.cc:224] Handling call
0709 12:26:03.733222 (+ 13961us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:03.740935 (+  7713us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:26:03.851792 (+110857us) log.cc:844] Serialized 14 byte log entry
0709 12:26:04.011424 (+159632us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:04.011928 (+   504us) log.cc:844] Serialized 26 byte log entry
0709 12:26:04.012103 (+   175us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:04.012301 (+   198us) log.cc:844] Serialized 14 byte log entry
0709 12:26:04.012483 (+   182us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:04.012579 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:04.776728 (+764149us) raft_consensus.cc:1685] finished
0709 12:26:04.776760 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:04.776930 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58176,"prepare.queue_time_us":232075,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":85,"raft.queue_time_us":156038,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":26496,"thread_start_us":159595,"threads_started":2,"wal-append.queue_time_us":226511}
I20260709 12:26:04.736219  9466 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:04.751847  9408 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:26:04.736275  9438 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:04.802546  9170 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.580s	user 0.011s	sys 0.000s
I20260709 12:26:04.769980  9338 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:04.800652  9475 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:04.769697  9447 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:04.827536  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 377244us
I20260709 12:26:04.809756  9143 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:04.869560  5708 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:26:04.871847  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:04.825875  9170 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:26:04.872714  5630 raft_consensus.cc:2376] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7. Re-sending same reply.
I20260709 12:26:04.828153  9475 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:04.879616  8783 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:26:04.881237  6271 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:26:04.881908  6273 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:26:04.882623  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:04.883507  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:04.886080  9438 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:04.888391  9466 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:04.770565  9480 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:04.894450  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:04.895052  6272 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:26:04.895206  5628 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:04.800725  9456 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 12:26:04.911197  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:04.917062  6273 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:26:04.917505  9404 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:04.917932  9404 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:04.897328  6279 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:26:04.826843  9501 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:04.911219  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:04.922582  5630 raft_consensus.cc:2410] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 12 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:26:04.911603  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:04.903666  6274 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
W20260709 12:26:04.913631  9341 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.635s	user 0.007s	sys 0.004s
I20260709 12:26:04.915673  9387 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:04.919308  5633 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:04.929967  9341 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:04.909235  5448 maintenance_manager.cc:419] P b3df7d767dc641b7bb01c3435d842dd5: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033353
I20260709 12:26:04.936640  9434 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:04.937230  9475 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:04.938102  9338 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:04.939159  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:04.939401  9475 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:04.939718  9420 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.820s	user 0.011s	sys 0.000s
I20260709 12:26:04.939994  6279 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:04.941771  5702 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: []
I20260709 12:26:04.943104  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:04.954579  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:04.940189  9420 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:04.955293  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:04.969276  6271 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:26:04.970846  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:04.971603  6271 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:04.972740  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:04.973351  9312 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:04.973752  6274 raft_consensus.cc:2368] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12.
I20260709 12:26:04.974020  9387 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:04.975381  6274 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:04.976696  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:04.977785  6273 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:26:04.978600  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:04.969463  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:04.981619  9463 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.707s	user 0.007s	sys 0.005s
I20260709 12:26:04.982190  9463 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:04.985216  9447 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:04.986183  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:04.987087  5627 raft_consensus.cc:2410] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:26:04.988121  6271 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: []
W20260709 12:26:04.988219  5627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2146) took 1744 ms (client timeout 1658 ms). Trace:
W20260709 12:26:04.988888  5627 rpcz_store.cc:269] 0709 12:26:03.244263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.244425 (+   162us) service_pool.cc:224] Handling call
0709 12:26:04.985935 (+1741510us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:26:04.988148 (+  2213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4954880}
I20260709 12:26:04.968298  9455 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:04.918830  9404 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:04.993827  9404 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:26:04.992101  5559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1923) took 1933 ms (client timeout 2448 ms). Trace:
I20260709 12:26:04.995643  5632 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:26:04.994230  9285 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:26:04.996445  9285 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:04.997465  9463 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:04.979395  9386 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.516s	user 0.008s	sys 0.000s
I20260709 12:26:05.008661  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:05.009696  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:05.010213  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:05.010880  5630 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:05.011129  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:05.011847  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:05.014777  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.019138  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:05.021780  9425 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:05.017149  9386 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.023797  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:05.025063  6271 raft_consensus.cc:2410] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
W20260709 12:26:04.995338  5559 rpcz_store.cc:269] 0709 12:26:03.058935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.075666 (+ 16731us) service_pool.cc:224] Handling call
0709 12:26:04.991896 (+1916230us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b54001c57d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:04.992043 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":697928448}
I20260709 12:26:05.001194  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:05.024816  9386 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:05.000020  5458 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 2 to 6, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 6 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:05.030671  5458 catalog_manager.cc:5697] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.033111  5458 catalog_manager.cc:5697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 3 to 6, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 6 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.012135  5635 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:05.035879  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:05.040407  5630 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:05.049420  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:05.043274  5458 catalog_manager.cc:5697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 2 to 5, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 5 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:05.053345  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:05.054946  5458 catalog_manager.cc:5697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:05.055676  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.056756  9340 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.057276  9340 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.057044  5458 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 4 to 5, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 5 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.057431  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:05.059417  9340 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:05.054260  5627 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:05.060952  5630 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:05.060092  5458 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 5 to 6, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 6 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:05.044452  9336 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.104478  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:05.106125  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:05.106494  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:05.118844  5635 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:26:05.112778  5560 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:05.114537  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:05.119266  5708 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:26:05.120668  5559 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:05.118180  5458 catalog_manager.cc:5697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 2 to 5, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 5 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.060312  9340 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.128515  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:05.128382  9340 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:05.129089  5635 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:05.130666  9340 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:05.133188  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:05.133970  5556 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:05.132158  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:05.134629  5706 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:05.136199  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:05.138120  9433 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.655s	user 0.005s	sys 0.000s
I20260709 12:26:05.125844  9336 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.139622  9282 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:05.140029  9282 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:05.140375  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 655) took 2035 ms. Trace:
I20260709 12:26:05.140986  9336 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:05.126000  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:05.133222  5708 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 11. Current term is 12. Ops: [9.9-11.10]
I20260709 12:26:05.140743  5629 rpcz_store.cc:276] 0709 12:26:03.104774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.105195 (+   421us) service_pool.cc:224] Handling call
0709 12:26:03.166749 (+ 61554us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:03.255804 (+ 89055us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:03.255834 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:03.293048 (+ 37214us) log.cc:844] Serialized 26 byte log entry
0709 12:26:03.373891 (+ 80843us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:03.374289 (+   398us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:03.374420 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:05.048162 (+1673742us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:05.140087 (+ 91925us) raft_consensus.cc:1685] finished
0709 12:26:05.140108 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:05.140296 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11032,"raft.queue_time_us":72928,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":8165,"spinlock_wait_cycles":39172992,"thread_start_us":126562,"threads_started":2,"wal-append.queue_time_us":131178}
I20260709 12:26:05.126319  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:05.142967  5702 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 9: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:05.140870  9282 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:05.148376  5629 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:26:05.149179  9282 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:05.151952  5629 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:26:05.156561  9505 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.157343  9505 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:05.161536  5635 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
W20260709 12:26:05.165045  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47663.8us
I20260709 12:26:05.173168  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:05.170754  5458 catalog_manager.cc:5697] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 3 to 6, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 6 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.118818  9386 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.185613  5458 catalog_manager.cc:5697] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 2 to 3, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 3 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.177352  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:05.127015  9460 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.160202  9505 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:05.195114  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:05.197155  5630 raft_consensus.cc:1217] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Deduplicated request from leader. Original: 7.8->[8.9-11.10]   Dedup: 8.9->[11.10-11.10]
I20260709 12:26:05.197860  5630 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: [8.9-11.10]
I20260709 12:26:05.198107  5635 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:05.200039  5458 catalog_manager.cc:5697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 5, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 5 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:05.203657  9336 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:05.204432  9336 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.205024  9336 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:05.212460  9485 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.213606  9485 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:05.229852  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:05.230844  5552 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:05.190716  9428 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 12:26:05.239773  5556 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
W20260709 12:26:05.244365  6276 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.922s	user 0.129s	sys 0.199s
I20260709 12:26:05.192864  9386 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:05.246752  6276 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.924s	user 0.129s	sys 0.199s
I20260709 12:26:05.223101  9413 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:05.253767  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 755) took 1046 ms. Trace:
I20260709 12:26:05.197321  9460 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.251758  9413 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:05.286777  5557 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1943) took 2617 ms (client timeout 3377 ms). Trace:
W20260709 12:26:05.287233  5557 rpcz_store.cc:269] 0709 12:26:02.668129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:02.754200 (+ 86071us) service_pool.cc:224] Handling call
0709 12:26:05.285422 (+2531222us) spinlock_profiling.cc:243] Waited 584 ms on lock 0x7b54001c4dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:05.285619 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1342237952}
W20260709 12:26:05.291648  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.028s	sys 0.091s
W20260709 12:26:05.291999  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.028s	sys 0.091s
I20260709 12:26:05.277285  9413 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:05.296321  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.026s	sys 0.083s
I20260709 12:26:05.290153  9501 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:26:05.302258  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.026s	sys 0.083s
I20260709 12:26:05.258616  9285 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.280514  9485 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:05.286013  9386 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:05.305457  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 623306us
W20260709 12:26:05.314038  5674 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.008s	sys 0.008s
W20260709 12:26:05.335461  5674 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.008s	sys 0.008s
W20260709 12:26:05.326170  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 628220us
I20260709 12:26:05.309551  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:05.337399  5709 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:05.337224  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:05.334270  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:05.344177  5703 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.009s	sys 0.023s
I20260709 12:26:05.335638  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:26:05.349601  5703 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.009s	sys 0.023s
I20260709 12:26:05.348269  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:05.351111  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:05.351621  5557 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:05.360296  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:05.343235  8903 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.348423  5458 catalog_manager.cc:5697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 3 to 7, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 7 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:26:05.358036  9473 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:05.348865  9190 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:05.389055  5458 catalog_manager.cc:5697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 2 to 4, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 4 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:05.363183  9285 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:05.361470  9456 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:05.389643  8903 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.386684  9460 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:05.408507  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.415302  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2040) took 2879 ms. Trace:
I20260709 12:26:05.415647  8903 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:05.365188  9336 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.419792  9445 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:05.399307  9456 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.410631  9410 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:05.437307  5559 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:05.432289  5704 rpcz_store.cc:276] 0709 12:26:02.536112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:02.545036 (+  8924us) service_pool.cc:224] Handling call
0709 12:26:02.545296 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:04.517057 (+1971761us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.414926 (+897869us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b5400323054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:05.415217 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4496283392}
I20260709 12:26:05.391669  9190 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:05.400127  9093 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:26:05.394207  9473 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:05.463914  9315 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.459671  9453 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:05.447727  9456 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:05.454807  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 117688us
I20260709 12:26:05.466001  9093 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:05.471455  5602 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.006s	sys 0.035s
W20260709 12:26:05.462242  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:05.474069  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:05.480921  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 146773us
W20260709 12:26:05.488869  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:05.491163  5602 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.006s	sys 0.035s
W20260709 12:26:05.491603  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 592537us
I20260709 12:26:05.469012  9190 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:05.462358  9469 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:05.518776  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:05.519447  9340 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:26:05.521102  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:05.522518  9452 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:26:05.471144  9461 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:05.472904  9453 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.529702  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:05.479391  9093 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:05.480734  9410 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:05.537555  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.541844  5458 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 3 to 5, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 5 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:05.543104  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:05.543503  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.544003  5559 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:05.536692  9461 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:05.545800  9461 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:05.515352  9336 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:05.524709  9479 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:05.477195  9315 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.551338  6275 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:26:05.525012  9469 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:05.548889  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:05.529795  8903 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:26:05.566653  5561 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 12:26:05.567085  5561 pending_rounds.cc:85] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 8
I20260709 12:26:05.567318  5561 pending_rounds.cc:107] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 12:26:05.537478  9509 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:05.568560  9509 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.569028  9469 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:05.549970  9447 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:05.570384  9469 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:05.568274  9463 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.572992  9463 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:05.574255  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:05.574978  9463 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:05.575539  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:05.569059  9509 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:26:05.577567  5633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2170) took 1596 ms (client timeout 1538 ms). Trace:
W20260709 12:26:05.549520  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.539268  9453 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:05.562630  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:05.577996  5633 rpcz_store.cc:269] 0709 12:26:03.987573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.991709 (+  4136us) service_pool.cc:224] Handling call
0709 12:26:05.577393 (+1585684us) spinlock_profiling.cc:243] Waited 912 ms on lock 0x7b54000c1054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:05.577506 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2096461184}
I20260709 12:26:05.567548  5561 raft_consensus.cc:2889] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:05.536790  9486 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.008s	sys 0.033s
I20260709 12:26:05.579718  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:05.583523  5557 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:05.602128  5561 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:05.556655  9479 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.605302  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:05.607333  5631 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.013s	sys 0.038s
I20260709 12:26:05.607744  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:05.608702  5633 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:05.609911  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:05.611816  5633 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:05.614293  5633 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:26:05.618654  5553 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: []
I20260709 12:26:05.621681  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:05.583648  9321 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.017s	sys 0.060s
I20260709 12:26:05.632223  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:05.624549  5561 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:26:05.625916  5633 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:05.584244  9447 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:05.607772  5631 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.689s	user 0.013s	sys 0.038s
W20260709 12:26:05.605847  9486 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.008s	sys 0.034s
I20260709 12:26:05.622021  9479 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:05.633747  9321 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.021s	sys 0.061s
I20260709 12:26:05.633843  5557 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:26:05.633019  8506 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.631774  8903 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:05.550072  9410 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:05.650466  5557 pending_rounds.cc:85] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 4
I20260709 12:26:05.657210  5557 pending_rounds.cc:107] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:26:05.657589  5557 raft_consensus.cc:2889] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:05.634020  9507 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:05.649382  9340 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260709 12:26:05.565665  9473 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:05.660125  9408 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.017s	sys 0.067s
I20260709 12:26:05.631363  9445 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260709 12:26:05.636569  9416 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.024s
I20260709 12:26:05.658926  9507 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.656951  8903 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:05.677654  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:05.659538  8506 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:05.678979  5558 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.014s	sys 0.018s
W20260709 12:26:05.681057  5558 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.014s	sys 0.018s
W20260709 12:26:05.671522  9408 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.017s	sys 0.068s
I20260709 12:26:05.657851  9404 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 12:26:05.671439  9486 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.711892  6277 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:26:05.712365  8506 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:05.698923  9340 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:26:05.698665  9447 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:05.689702  8903 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:05.725322  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2133) took 2101 ms. Trace:
I20260709 12:26:05.726006  5709 rpcz_store.cc:276] 0709 12:26:03.623437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.891202 (+267765us) service_pool.cc:224] Handling call
0709 12:26:05.139046 (+1247844us) spinlock_profiling.cc:243] Waited 5.81 ms on lock 0x7b5800053198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:05.139240 (+   194us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:05.337823 (+198583us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.724947 (+387124us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b540026eb54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:05.725210 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":468650240}
I20260709 12:26:05.728318  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1889) took 1076 ms. Trace:
I20260709 12:26:05.728873  5628 rpcz_store.cc:276] 0709 12:26:04.652247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.755682 (+103435us) service_pool.cc:224] Handling call
0709 12:26:05.728109 (+972427us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b540008c0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:05.728247 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":267100544}
W20260709 12:26:05.732571  5555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1928) took 2486 ms (client timeout 2038 ms). Trace:
W20260709 12:26:05.733031  5555 rpcz_store.cc:269] 0709 12:26:03.246336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.246472 (+   136us) service_pool.cc:224] Handling call
0709 12:26:05.732297 (+2485825us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b54003a89d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:05.732452 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":427261696}
I20260709 12:26:05.736342  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1946) took 1181 ms. Trace:
I20260709 12:26:05.736862  5554 rpcz_store.cc:276] 0709 12:26:04.554854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.555059 (+   205us) service_pool.cc:224] Handling call
0709 12:26:04.658520 (+103461us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:04.659010 (+   490us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:26:04.751864 (+ 92854us) log.cc:844] Serialized 14 byte log entry
0709 12:26:05.161668 (+409804us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:05.202696 (+ 41028us) log.cc:844] Serialized 52 byte log entry
0709 12:26:05.202924 (+   228us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:26:05.208948 (+  6024us) log.cc:844] Serialized 14 byte log entry
0709 12:26:05.306242 (+ 97294us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.306531 (+   289us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:05.736100 (+429569us) raft_consensus.cc:1685] finished
0709 12:26:05.736122 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:05.736275 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":113716,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":91,"raft.queue_time_us":16106,"raft.run_cpu_time_us":528,"raft.run_wall_time_us":29169,"thread_start_us":457138,"threads_started":2,"wal-append.queue_time_us":873697}
I20260709 12:26:05.748142  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:05.748911  9472 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.749253  5707 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:05.749743  9472 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.752538  9472 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:05.753362  9413 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.753520  9472 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:05.756057  9486 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.758234  9486 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:05.679388  9507 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:05.771483  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:05.772415  5704 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:05.749001  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:26:05.776503  5630 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.032s	sys 0.088s
W20260709 12:26:05.776912  5630 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.032s	sys 0.088s
I20260709 12:26:05.774070  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:05.788208  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:05.790635  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:05.790836  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:05.791381  5555 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:05.798921  5707 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:05.799335  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:05.799901  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:05.800978  6279 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:05.799008  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:05.802481  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:05.800118  6280 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:05.802258  6277 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:05.810179  6274 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 12:26:05.812831  9463 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:05.813468  9463 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.815771  9463 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:05.803776  9220 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.015s
I20260709 12:26:05.819216  9453 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:05.821059  9452 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:26:05.821647  9452 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:05.824478  9386 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:05.825096  9386 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.827237  9386 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:05.833114  5704 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:26:05.787981  8506 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:26:05.789496  9321 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:05.837738  8506 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.845840  8506 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:05.845558  9336 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.845695  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:05.849009  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 488770us
I20260709 12:26:05.852123  9336 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.838056  9321 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.875618  9321 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:05.876945  9336 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:05.879217  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:05.879899  5707 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:05.882299  9479 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:05.853709  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 660) took 1826 ms. Trace:
I20260709 12:26:05.885504  5634 rpcz_store.cc:276] 0709 12:26:04.027699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.198698 (+170999us) service_pool.cc:224] Handling call
0709 12:26:04.215658 (+ 16960us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:04.415704 (+200046us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.853464 (+1437760us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b540018b854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:05.853641 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":380364032}
I20260709 12:26:05.861326  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:05.842018  9463 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260709 12:26:05.887710  6273 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
W20260709 12:26:05.869035  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.849110  9413 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:05.892371  9463 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:05.898756  9340 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.900822  9463 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:05.901062  9340 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:05.901933  9463 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.903226  9463 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:05.904016  9340 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:05.905596  9479 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:05.255916  6276 rpcz_store.cc:276] 0709 12:26:04.206215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.207569 (+  1354us) service_pool.cc:224] Handling call
0709 12:26:04.213261 (+  5692us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:04.213511 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:26:04.321801 (+108290us) log.cc:844] Serialized 14 byte log entry
0709 12:26:05.251375 (+929574us) log.cc:844] Serialized 14 byte log entry
0709 12:26:05.251833 (+   458us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:05.251895 (+    62us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:05.251961 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.252044 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:05.252203 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":126741,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":3416,"raft.queue_time_us":1094,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":6673,"spinlock_wait_cycles":1764736,"thread_start_us":1007799,"threads_started":2}
I20260709 12:26:05.915138  9321 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:05.917049  9495 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.375s	user 0.000s	sys 0.003s
I20260709 12:26:05.918265  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1888) took 1266 ms. Trace:
I20260709 12:26:05.918675  5631 rpcz_store.cc:276] 0709 12:26:04.651999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.912753 (+260754us) service_pool.cc:224] Handling call
0709 12:26:04.913074 (+   321us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:04.917910 (+  4836us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:26:04.917940 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:04.918504 (+   564us) log.cc:844] Serialized 26 byte log entry
0709 12:26:05.640431 (+721927us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:26:05.640493 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.640620 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:05.918037 (+277417us) raft_consensus.cc:1685] finished
0709 12:26:05.918057 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:05.918205 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4376,"raft.queue_time_us":5577,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":5777,"thread_start_us":721467,"threads_started":1,"wal-append.queue_time_us":935078}
I20260709 12:26:05.920840  9410 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:05.930958  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:05.931630  5702 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:05.932245  6280 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:26:05.934626  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:05.921672  9321 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:05.937745  9336 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:26:05.936287  9499 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:05.939062  9499 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:05.955091  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:05.952440  5560 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.088s	sys 0.113s
W20260709 12:26:05.962530  5560 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.088s	sys 0.113s
I20260709 12:26:05.937275  5555 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:05.959753  5702 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
W20260709 12:26:05.936292  5526 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.981s	user 0.021s	sys 0.021s
W20260709 12:26:05.967500  5526 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.013s	user 0.021s	sys 0.021s
W20260709 12:26:05.967940  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 1.01328e+06us
W20260709 12:26:05.969090  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.970837  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:05.973318  9340 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:26:05.947947  9509 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:05.977001  9093 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.972288  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:05.978426  9509 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:05.979154  9093 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:05.979198  9340 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:26:05.980605  9404 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:05.981040  9404 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:05.981434  9499 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:05.982052  9404 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:05.984270  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:05.930706  9456 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:05.990985  9456 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:05.991241  9469 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:26:05.991920  6271 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:26:05.993665  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:05.994275  5552 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.003s	sys 0.031s
W20260709 12:26:05.994580  5552 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.003s	sys 0.031s
I20260709 12:26:06.009296  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:06.012285  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:06.013622  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:06.014385  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:06.017536  5708 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 6. Current term is 7. Ops: [4.4-6.5]
I20260709 12:26:06.026098  5707 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 12:26:06.026791  5707 pending_rounds.cc:85] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 4
I20260709 12:26:06.043242  5707 pending_rounds.cc:107] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 12:26:06.043535  5707 raft_consensus.cc:2889] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:05.997056  9426 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.017s	sys 0.017s
I20260709 12:26:06.042812  9445 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:26:06.046989  9520 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:26:06.055894  5708 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 12:26:06.047983  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:06.071699  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:06.075315  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:06.090790  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:06.030934  9185 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:26:06.126597  9190 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.598s	user 0.010s	sys 0.002s
I20260709 12:26:06.127069  9190 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:06.127173  9452 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:06.128789  6272 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:26:06.130267  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1890) took 1477 ms. Trace:
I20260709 12:26:06.130678  5636 rpcz_store.cc:276] 0709 12:26:04.652560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.140898 (+488338us) service_pool.cc:224] Handling call
0709 12:26:05.141198 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:05.169596 (+ 28398us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:05.169645 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:05.170291 (+   646us) log.cc:844] Serialized 26 byte log entry
0709 12:26:05.673764 (+503473us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:26:05.675285 (+  1521us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.675427 (+   142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:06.130007 (+454580us) raft_consensus.cc:1685] finished
0709 12:26:06.130031 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:06.130199 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45599,"raft.queue_time_us":35383,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":2049,"thread_start_us":480768,"threads_started":1,"wal-append.queue_time_us":908032}
I20260709 12:26:06.108865  9207 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
W20260709 12:26:06.098524  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 82182.9us
I20260709 12:26:06.146113  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:06.107221  5703 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.012s	sys 0.025s
I20260709 12:26:06.107721  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:06.149976  5703 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.012s	sys 0.025s
I20260709 12:26:06.024124  9386 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.074s
I20260709 12:26:06.115638  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:06.159453  5708 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:06.116325  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:06.160516  5706 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:26:06.101809  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:06.161703  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:06.167479  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 61249.2us
I20260709 12:26:06.177693  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:06.226686  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:06.061868  9426 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.017s	sys 0.017s
W20260709 12:26:06.185084  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 119083us
I20260709 12:26:06.105163  9463 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:06.176236  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:06.244784  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:06.133242  9285 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:26:06.279733  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 103613us
I20260709 12:26:06.282857  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:06.284229  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:06.183004  9376 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:06.290249  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:06.168383  9127 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.193401  9452 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:06.175659  9364 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:06.294145  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:06.297324  5708 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
W20260709 12:26:06.278442  9519 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:26:06.319703  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:06.321252  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:06.281575  9185 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:26:06.338842  9489 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260709 12:26:06.344827  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:06.345844  6277 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:06.310971  9127 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:06.302489  8506 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:06.293751  9190 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:06.349538  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:06.348500  9523 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:26:06.326350  9386 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:26:06.373675  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:06.377112  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:06.354429  9408 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:06.291221  9170 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:26:06.334486  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:06.328884  5710 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:06.170928  9207 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.131s
I20260709 12:26:06.371197  9386 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:06.390377  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:06.282222  9285 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:06.432674  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:06.433358  5710 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 12:26:06.390074  5556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.005s	sys 0.016s
W20260709 12:26:06.390326  6275 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.013s	sys 0.004s
I20260709 12:26:06.355885  9185 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:26:06.389863  9473 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.006s	sys 0.010s
W20260709 12:26:06.435480  6275 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.013s	sys 0.004s
I20260709 12:26:06.448820  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:06.449851  6280 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 8: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:06.436869  8506 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:06.440184  9489 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:26:06.390786  9447 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.031s	sys 0.079s
W20260709 12:26:06.463863  9473 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.011s	sys 0.010s
I20260709 12:26:06.386307  9463 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.062s
I20260709 12:26:06.441674  5710 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: []
W20260709 12:26:06.432761  9507 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.643s	user 0.012s	sys 0.000s
I20260709 12:26:06.380570  9408 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:06.435299  5556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.005s	sys 0.016s
W20260709 12:26:06.464028  5554 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.021s	sys 0.017s
I20260709 12:26:06.395854  9386 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:06.445302  9503 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:06.543862  9503 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:06.544251  9503 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:06.546020  9525 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260709 12:26:06.431758  9471 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
W20260709 12:26:06.463055  9524 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.181s	user 0.001s	sys 0.001s
W20260709 12:26:06.470207  9447 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.031s	sys 0.080s
I20260709 12:26:06.551384  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1898) took 1861 ms. Trace:
I20260709 12:26:06.551780  5630 rpcz_store.cc:276] 0709 12:26:04.690358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.198783 (+508425us) service_pool.cc:224] Handling call
0709 12:26:05.218015 (+ 19232us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:05.223421 (+  5406us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:05.223469 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:05.237575 (+ 14106us) log.cc:844] Serialized 26 byte log entry
0709 12:26:05.777372 (+539797us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:05.777433 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.777527 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:06.479320 (+701793us) spinlock_profiling.cc:243] Waited 5.01 ms on lock 0x7b2c00055d78. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8922291 00007f4ff893d0ca 00007f4ffa66b4ad 00007f4ffa672f8e 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:26:06.551071 (+ 71751us) raft_consensus.cc:1685] finished
0709 12:26:06.551092 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:06.551322 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4932,"raft.queue_time_us":5562,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":6972,"spinlock_wait_cycles":11514240,"thread_start_us":539389,"threads_started":1,"wal-append.queue_time_us":942903}
W20260709 12:26:06.467012  8903 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.749s	user 0.012s	sys 0.000s
I20260709 12:26:06.553797  8903 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:06.560894  9408 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:06.562379  9408 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:06.564958  5704 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:06.568815  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:06.570020  8903 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:06.570268  9505 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:06.570801  9505 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:06.571470  9505 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:06.576090  5554 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.022s	sys 0.017s
W20260709 12:26:06.435597  5458 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.695s	user 0.012s	sys 0.009s
W20260709 12:26:06.534878  5675 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.027s (SENT) [suppressed 36 similar messages]
W20260709 12:26:06.590021  5458 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.849s	user 0.012s	sys 0.009s
I20260709 12:26:06.537267  5708 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:06.539335  9507 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:06.540031  9528 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:26:06.591888  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:06.596244  6277 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:06.538024  9489 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.597539  5708 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:06.602916  9489 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:06.605329  9489 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:06.609210  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:06.609556  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:06.610674  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:06.611325  5706 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:06.611953  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:06.615748  5600 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.817s	user 0.007s	sys 0.022s
W20260709 12:26:06.616114  5600 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.818s	user 0.007s	sys 0.022s
W20260709 12:26:06.624143  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:06.627740  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:06.630815  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:06.631417  6278 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.015s	sys 0.026s
W20260709 12:26:06.631990  6278 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.015s	sys 0.026s
I20260709 12:26:06.632045  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:06.634929  5561 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:06.640908  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:06.589725  9127 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:06.643394  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:06.643885  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:06.645104  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:06.646054  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:06.650815  5631 raft_consensus.cc:2410] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 12:26:06.608345  9404 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:06.653196  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:06.649933  5561 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:26:06.684015  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:06.685071  5557 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:06.686008  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:06.640081  9447 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:06.687119  9447 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:06.687659  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:06.688397  9447 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:06.690232  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:06.690698  9388 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:06.690975  5628 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:06.691356  9388 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:06.640652  9170 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:06.692276  9170 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:06.692914  9170 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:06.650292  9426 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:06.696834  9426 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:06.697403  9426 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:06.698459  9461 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:06.698958  5628 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:06.659514  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:06.701310  5631 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:06.702431  5629 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:06.702662  9437 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:06.703145  9437 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:06.704540  5629 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:26:06.709730  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:06.711159  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:06.711709  5704 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 7.
I20260709 12:26:06.714288  9480 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:06.715545  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:06.717103  9473 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:06.717365  9526 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:26:06.717830  9473 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:06.718890  9473 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:06.719599  9315 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:06.721190  5630 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:26:06.725787  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:06.726625  5628 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:06.676345  5708 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
W20260709 12:26:06.749215  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.974s	user 0.008s	sys 0.028s
W20260709 12:26:06.750113  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.975s	user 0.008s	sys 0.028s
I20260709 12:26:06.756026  5559 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:26:06.757534  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:06.758371  5633 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:26:06.759898  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:06.766848  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:06.775993  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:06.618706  9207 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:06.513437  9285 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:06.773741  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:06.776192  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:06.816438  5553 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
W20260709 12:26:06.631165  9108 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.023s	sys 0.024s
I20260709 12:26:06.781831  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:06.779500  5561 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:06.822806  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:06.647310  9423 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.783514  5708 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
W20260709 12:26:06.761976  9450 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.483s	user 0.014s	sys 0.025s
I20260709 12:26:06.637319  8856 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.143s
I20260709 12:26:06.838872  9437 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:06.770653  5702 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.722s	user 0.003s	sys 0.072s
W20260709 12:26:06.840468  5702 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.791s	user 0.004s	sys 0.072s
I20260709 12:26:06.831925  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:06.838176  9450 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.554s	user 0.015s	sys 0.026s
I20260709 12:26:06.841777  5553 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:06.845983  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:06.839506  9437 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:06.853200  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:06.776950  9338 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:06.653990  9386 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:06.852873  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:06.853837  5557 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:06.782179  9526 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.059s
I20260709 12:26:06.770685  9127 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.078s
W20260709 12:26:06.824282  9108 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.921s	user 0.023s	sys 0.024s
I20260709 12:26:06.870759  5561 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 12:26:06.871572  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:06.882262  5552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.879s	user 0.018s	sys 0.065s
W20260709 12:26:06.882762  5552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.888s	user 0.018s	sys 0.065s
I20260709 12:26:06.873698  9386 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:06.874542  5559 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:06.875226  5555 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:26:06.883589  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1968) took 1567 ms. Trace:
I20260709 12:26:06.883941  5552 rpcz_store.cc:276] 0709 12:26:05.316417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.316864 (+   447us) service_pool.cc:224] Handling call
0709 12:26:05.349978 (+ 33114us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:05.350163 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:26:05.994915 (+644752us) log.cc:844] Serialized 14 byte log entry
0709 12:26:06.883184 (+888269us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:06.883235 (+    51us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:26:06.883288 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:06.883431 (+   143us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:06.883539 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7812,"prepare.queue_time_us":42427,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"thread_start_us":1515493,"threads_started":2}
I20260709 12:26:06.899281  9533 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:06.863677  9285 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:26:06.818189  9207 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:06.820962  9503 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:06.840157  9426 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:06.888504  9364 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:06.900835  9386 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.847375  9423 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:06.907557  5703 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.005s	sys 0.018s
W20260709 12:26:06.968812  5703 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.765s	user 0.005s	sys 0.019s
I20260709 12:26:06.969420  5553 raft_consensus.cc:2410] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:26:07.003798  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:07.008930  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:07.009709  5561 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:07.008037  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:07.010911  5553 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:07.012140  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:07.013067  5559 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:26:07.017264  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2056) took 2462 ms. Trace:
I20260709 12:26:06.866653  9437 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:06.877233  9338 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:07.018960  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:07.036530  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:07.089277  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52859.9us
I20260709 12:26:07.091145  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:07.037648  5633 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:06.899924  9533 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:07.019383  9509 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:06.966570  9539 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:07.121534  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:06.949245  9435 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.777s	user 0.009s	sys 0.018s
I20260709 12:26:07.101054  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:06.953794  9507 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:07.087319  5552 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:26:07.019721  9456 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:07.049916  9426 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:07.042276  8856 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:26:07.126513  9435 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.958s	user 0.009s	sys 0.018s
I20260709 12:26:07.148765  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:07.111891  9388 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:07.122339  9526 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:07.125694  5553 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:07.130605  5561 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:07.054467  9503 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:06.970889  9386 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:06.947981  5627 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.100s	user 0.012s	sys 0.016s
I20260709 12:26:07.140638  5635 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:26:07.110708  9315 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.143671  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:07.078120  5703 rpcz_store.cc:276] 0709 12:26:04.554359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.768815 (+214456us) service_pool.cc:224] Handling call
0709 12:26:04.769428 (+   613us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:05.349985 (+580557us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:05.350014 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:05.350550 (+   536us) log.cc:844] Serialized 52 byte log entry
0709 12:26:06.150403 (+799853us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:26:06.969416 (+819013us) log.cc:844] Serialized 14 byte log entry
0709 12:26:07.016840 (+ 47424us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.016953 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:07.017001 (+    48us) raft_consensus.cc:1685] finished
0709 12:26:07.017010 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:07.017193 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41426,"prepare.queue_time_us":387890,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":77686,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":25384,"thread_start_us":2020311,"threads_started":3,"wal-append.queue_time_us":1026878}
I20260709 12:26:07.173604  5552 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:26:07.176126  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.831s	user 0.000s	sys 0.033s
W20260709 12:26:07.176631  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.831s	user 0.000s	sys 0.033s
W20260709 12:26:07.182679  5701 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.447s	user 0.031s	sys 0.078s
W20260709 12:26:07.183564  5701 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.448s	user 0.031s	sys 0.078s
I20260709 12:26:07.185542  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:07.186205  5561 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:26:07.189270  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.206256  5561 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:07.207378  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:07.156471  5709 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.004s	sys 0.014s
W20260709 12:26:07.161448  5627 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.316s	user 0.013s	sys 0.017s
I20260709 12:26:07.155869  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:07.219121  5709 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.004s	sys 0.014s
I20260709 12:26:07.219233  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:07.157764  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:07.136031  9453 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:07.125140  9539 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:07.154657  9526 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:07.220789  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63098.9us
I20260709 12:26:07.237871  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:07.222896  9321 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.824s	user 0.020s	sys 0.018s
W20260709 12:26:07.168665  9452 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.855s	user 0.009s	sys 0.005s
W20260709 12:26:07.226763  6275 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.772s	user 0.013s	sys 0.029s
I20260709 12:26:07.229188  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1892) took 2566 ms. Trace:
W20260709 12:26:07.233036  6280 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.005s	sys 0.019s
I20260709 12:26:07.208091  5561 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:07.284603  6280 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.006s	sys 0.019s
I20260709 12:26:07.147945  9456 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:07.285979  5627 rpcz_store.cc:276] 0709 12:26:04.662728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.148034 (+485306us) service_pool.cc:224] Handling call
0709 12:26:05.148388 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:05.190592 (+ 42204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:05.190634 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:05.199079 (+  8445us) log.cc:844] Serialized 26 byte log entry
0709 12:26:05.707215 (+508136us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:07.227607 (+1520392us) log.cc:844] Serialized 14 byte log entry
0709 12:26:07.228252 (+   645us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.228683 (+   431us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:07.228770 (+    87us) raft_consensus.cc:1685] finished
0709 12:26:07.228811 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:07.229063 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37170,"raft.queue_time_us":45522,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":15907,"thread_start_us":1863348,"threads_started":2,"wal-append.queue_time_us":951746}
W20260709 12:26:07.299072  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.306151  5561 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:07.307957  5561 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:07.232913  9127 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:07.159982  9388 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:07.172670  9479 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:26:07.286005  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48354.8us
I20260709 12:26:07.233290  9463 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:07.331385  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:07.332756  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:07.322028  6275 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.013s	sys 0.029s
I20260709 12:26:07.225310  9488 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:07.237282  9539 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:07.238590  9530 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.111s	user 0.001s	sys 0.001s
I20260709 12:26:07.294766  9452 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:07.203701  9315 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:07.238416  9503 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:26:07.241494  9526 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:07.324326  9321 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.972s	user 0.021s	sys 0.019s
I20260709 12:26:07.327214  5552 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
W20260709 12:26:07.327574  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.005s	user 0.012s	sys 0.030s
W20260709 12:26:07.369583  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.047s	user 0.012s	sys 0.030s
W20260709 12:26:07.370496  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 1.04926e+06us
I20260709 12:26:07.371611  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:07.362720  5675 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.006s	sys 0.012s
I20260709 12:26:07.316960  9543 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.355197  9485 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:26:07.402552  5556 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.825s	user 0.007s	sys 0.041s
I20260709 12:26:07.362530  9507 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:07.354446  9523 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:26:07.337890  9479 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:07.360337  5602 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.007s	sys 0.013s
W20260709 12:26:07.412015  5602 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.785s	user 0.008s	sys 0.013s
W20260709 12:26:07.413283  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 786267us
W20260709 12:26:07.403126  5556 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.825s	user 0.007s	sys 0.041s
W20260709 12:26:07.375334  5675 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.007s	sys 0.012s
W20260709 12:26:07.415694  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 880785us
W20260709 12:26:07.416126  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:07.378122  5560 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.018s	sys 0.001s
W20260709 12:26:07.377738  5674 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.008s	sys 0.011s
W20260709 12:26:07.421193  5554 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.844s	user 0.034s	sys 0.016s
W20260709 12:26:07.585659  5674 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.947s	user 0.009s	sys 0.011s
W20260709 12:26:07.435416  5601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.012s	sys 0.018s
W20260709 12:26:07.586287  5601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.013s	sys 0.018s
W20260709 12:26:07.587221  5560 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.974s	user 0.019s	sys 0.001s
W20260709 12:26:07.441917  5527 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.022s	sys 0.067s
W20260709 12:26:07.588088  5527 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.022s	sys 0.067s
I20260709 12:26:07.367725  9503 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:07.393100  9452 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:07.480748  5557 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.612s	user 0.006s	sys 0.005s
I20260709 12:26:07.589646  5557 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:07.373900  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:07.397239  9453 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:07.418625  9426 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.419298  9185 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:26:07.398089  9321 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:26:07.471170  9381 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:07.511596  5553 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:26:07.400987  9543 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:07.497195  9533 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:07.442049  9376 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.137s	user 0.009s	sys 0.005s
I20260709 12:26:07.425105  9479 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:26:07.585731  5554 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.008s	user 0.034s	sys 0.017s
I20260709 12:26:07.355454  9463 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:07.596796  9545 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.239s	user 0.000s	sys 0.001s
I20260709 12:26:07.615620  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2141) took 1602 ms. Trace:
I20260709 12:26:07.623284  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 624) took 1876 ms. Trace:
W20260709 12:26:07.674599  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 300743us
W20260709 12:26:07.507288  9534 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.349s	user 0.001s	sys 0.000s
I20260709 12:26:07.493137  9437 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:07.611995  9489 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.782718  9489 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:07.786547  9489 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:07.739073  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.923s	user 0.032s	sys 0.035s
W20260709 12:26:07.788986  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.973s	user 0.032s	sys 0.035s
W20260709 12:26:07.789621  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 1.01344e+06us
I20260709 12:26:07.773761  5702 rpcz_store.cc:276] 0709 12:26:06.013043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.013422 (+   379us) service_pool.cc:224] Handling call
0709 12:26:06.013773 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:06.016830 (+  3057us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:06.016863 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:06.017397 (+   534us) log.cc:844] Serialized 26 byte log entry
0709 12:26:06.840918 (+823521us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:06.890066 (+ 49148us) log.cc:844] Serialized 14 byte log entry
0709 12:26:06.890544 (+   478us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:06.890715 (+   171us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:07.611522 (+720807us) spinlock_profiling.cc:243] Waited 5.14 ms on lock 0x7b180043ac20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:07.615027 (+  3505us) raft_consensus.cc:1685] finished
0709 12:26:07.615227 (+   200us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:07.615503 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11448,"prepare.queue_time_us":35025,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":14455,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":49169,"spinlock_wait_cycles":11831552,"thread_start_us":791780,"threads_started":1,"wal-append.queue_time_us":1424412}
I20260709 12:26:07.790546  5524 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:07.746253  5632 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.781s	user 0.040s	sys 0.088s
W20260709 12:26:07.791165  5632 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.040s	sys 0.088s
I20260709 12:26:07.496006  9423 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:07.562237  5552 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:26:07.437839  9507 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:07.797447  5524 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:07.800483  5705 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 12:26:07.722963  9108 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.770s	user 0.009s	sys 0.017s
I20260709 12:26:07.501734  9549 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:07.726716  9503 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:26:07.790755  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:07.808499  5704 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.968s	user 0.031s	sys 0.028s
I20260709 12:26:07.736651  9453 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:07.748766  9381 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:07.817468  5526 connection.cc:256] RPC call timeout handler was delayed by -1.39250451099997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
W20260709 12:26:07.843818  5526 connection.cc:256] RPC call timeout handler was delayed by -1.384404510999957s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.169s for a response
W20260709 12:26:07.844278  5526 connection.cc:256] RPC call timeout handler was delayed by -1.3691045109999322s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17070000000000002s for a response
W20260709 12:26:07.844681  5526 connection.cc:256] RPC call timeout handler was delayed by -1.3268045110003186s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1754s for a response
W20260709 12:26:07.844991  5526 connection.cc:256] RPC call timeout handler was delayed by -1.3187045110003055s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1763s for a response
W20260709 12:26:07.845256  5526 connection.cc:256] RPC call timeout handler was delayed by -1.208904511000128s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1885s for a response
W20260709 12:26:07.828977  5636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.046s	user 0.007s	sys 0.027s
I20260709 12:26:07.775803  5709 rpcz_store.cc:276] 0709 12:26:05.746524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.782097 (+ 35573us) service_pool.cc:224] Handling call
0709 12:26:05.804369 (+ 22272us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:05.804851 (+   482us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:05.804897 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:05.805507 (+   610us) log.cc:844] Serialized 26 byte log entry
0709 12:26:06.485544 (+680037us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b1800432040. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:26:07.219851 (+734307us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:07.219908 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.220003 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:07.622774 (+402771us) raft_consensus.cc:1685] finished
0709 12:26:07.622811 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:07.623175 (+   364us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":102,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":9576,"spinlock_wait_cycles":2913664,"thread_start_us":734073,"threads_started":1,"wal-append.queue_time_us":1815341}
W20260709 12:26:07.806012  9473 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.048s	user 0.046s	sys 0.085s
I20260709 12:26:07.772080  9388 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:26:07.771078  9386 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:07.787672  9447 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.948s	user 0.022s	sys 0.052s
W20260709 12:26:07.846786  5636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.064s	user 0.007s	sys 0.027s
W20260709 12:26:07.837510  5704 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.997s	user 0.031s	sys 0.028s
I20260709 12:26:07.736936  9426 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:07.802008  9285 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.869s	user 0.013s	sys 0.000s
W20260709 12:26:07.849503  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.689s	user 0.013s	sys 0.017s
W20260709 12:26:07.883687  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.013s	sys 0.017s
W20260709 12:26:07.828713  9108 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.892s	user 0.010s	sys 0.018s
I20260709 12:26:07.801522  9376 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.812875  9486 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.813503  9437 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:26:07.764914  5600 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.978s	user 0.030s	sys 0.040s
W20260709 12:26:07.893458  5600 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.107s	user 0.031s	sys 0.040s
W20260709 12:26:07.863396  5706 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.624s	user 0.012s	sys 0.000s
I20260709 12:26:07.895018  5706 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
W20260709 12:26:07.852375  9473 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.130s	user 0.046s	sys 0.086s
W20260709 12:26:07.852574  9408 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.284s	user 0.010s	sys 0.006s
W20260709 12:26:07.859035  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.207s	user 0.023s	sys 0.039s
W20260709 12:26:07.914872  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.262s	user 0.023s	sys 0.039s
W20260709 12:26:07.915627  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 1.27219e+06us
I20260709 12:26:07.916713  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:07.840582  9549 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:07.861112  9447 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.021s	user 0.023s	sys 0.053s
W20260709 12:26:07.888906  6235 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.000s	sys 0.017s
W20260709 12:26:07.919723  6235 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.000s	sys 0.018s
I20260709 12:26:07.902110  9315 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:07.883358  8506 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:07.902719  9376 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:07.903174  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2129) took 4282 ms. Trace:
I20260709 12:26:07.903702  9437 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:26:07.905107  9108 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:26:07.905465  9486 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:07.942548  9108 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:07.863010  9452 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:07.946422  9108 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:07.906507  9561 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.948832  9561 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:07.952152  9561 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:07.953711  9561 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.906872  9453 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:07.908069  9564 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.901095  9563 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:07.863893  8856 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.011s	sys 0.019s
I20260709 12:26:07.924132  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:07.979151  6272 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
W20260709 12:26:07.926818  6275 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.012s	sys 0.007s
W20260709 12:26:07.980698  6275 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.012s	sys 0.007s
I20260709 12:26:07.981782  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 763) took 2280 ms. Trace:
W20260709 12:26:07.930534  6280 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.003s	sys 0.010s
W20260709 12:26:07.989599  6280 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.003s	sys 0.010s
I20260709 12:26:07.860069  9456 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:07.942014  9437 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:07.894533  9559 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.887228  9388 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:07.912365  9285 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.937553  5701 rpcz_store.cc:276] 0709 12:26:03.621203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.621317 (+   114us) service_pool.cc:224] Handling call
0709 12:26:03.635127 (+ 13810us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:05.734816 (+2099689us) spinlock_profiling.cc:243] Waited 3.24 ms on lock 0x7b180002de20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffa66a060 00007f4ffa66abfb 00007f4ffa67564b
0709 12:26:05.734971 (+   155us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:05.735000 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:05.735562 (+   562us) log.cc:844] Serialized 26 byte log entry
0709 12:26:07.184256 (+1448694us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:26:07.184321 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.184442 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:07.902727 (+718285us) raft_consensus.cc:1685] finished
0709 12:26:07.902759 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:07.903059 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8816,"raft.queue_time_us":89566,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":3123,"spinlock_wait_cycles":7441664,"thread_start_us":1531697,"threads_started":2,"wal-append.queue_time_us":2111487}
I20260709 12:26:07.865928  9386 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.989951  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:07.991469  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:07.923548  9549 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:07.982268  6275 rpcz_store.cc:276] 0709 12:26:05.701399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.702218 (+   819us) service_pool.cc:224] Handling call
0709 12:26:05.702525 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:05.702728 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:26:05.702973 (+   245us) log.cc:844] Serialized 14 byte log entry
0709 12:26:07.334007 (+1631034us) log.cc:844] Serialized 14 byte log entry
0709 12:26:07.981270 (+647263us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:07.981349 (+    79us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:07.981428 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.981532 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:07.981707 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1169779,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":89,"raft.queue_time_us":10446,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":5620,"spinlock_wait_cycles":26417408,"thread_start_us":2270112,"threads_started":3,"wal-append.queue_time_us":1235642}
I20260709 12:26:08.038177  6277 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:08.020370  5673 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.012s	sys 0.012s
I20260709 12:26:07.960302  9423 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.961397  9108 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:07.940768  9473 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:07.955914  9561 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.029417  9388 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.031364  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:08.117357  6279 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:07.997378  8856 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.795s	user 0.012s	sys 0.020s
W20260709 12:26:08.038692  5673 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.012s	sys 0.012s
W20260709 12:26:08.124905  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:08.126096  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.058032  9386 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:08.037357  9127 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.024s	sys 0.033s
I20260709 12:26:08.071941  9473 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.053378  9562 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:08.005764  9563 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.140142  9386 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.141556  9563 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.124694  8856 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:26:08.056881  9435 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.738s	user 0.007s	sys 0.020s
W20260709 12:26:08.144590  9435 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.008s	sys 0.020s
I20260709 12:26:08.144991  8856 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.145653  9435 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:08.146556  9435 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:08.149081  9543 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:08.150417  9480 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.436s	user 0.006s	sys 0.006s
I20260709 12:26:08.150753  9426 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.118129  9093 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:08.151669  9426 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:26:08.062948  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2052) took 3519 ms. Trace:
I20260709 12:26:08.096057  9488 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:08.080924  9507 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:08.174093  9376 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:08.151933  9093 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:08.083827  9170 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.390s	user 0.012s	sys 0.003s
I20260709 12:26:08.025050  9548 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:08.030406  9207 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:08.090732  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 670) took 1445 ms. Trace:
I20260709 12:26:08.030395  9285 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.093834  9437 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.177521  9093 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.176330  6279 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
W20260709 12:26:08.181109  5674 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.000s	sys 0.010s
W20260709 12:26:08.190083  5674 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.000s	sys 0.010s
W20260709 12:26:08.190757  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.192076  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:07.936966  9533 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.087s
W20260709 12:26:08.131830  9127 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.774s	user 0.024s	sys 0.040s
I20260709 12:26:08.139243  9473 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.210934  9479 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:08.199579  9505 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.628s	user 0.012s	sys 0.005s
I20260709 12:26:08.143786  9561 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.212255  5701 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:26:08.212471  9505 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.064185  9423 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.077855  6277 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
W20260709 12:26:08.079610  9461 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.381s	user 0.009s	sys 0.007s
I20260709 12:26:08.157961  5705 rpcz_store.cc:276] 0709 12:26:04.543735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.546537 (+  2802us) service_pool.cc:224] Handling call
0709 12:26:05.576004 (+1029467us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b5800053198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:05.576207 (+   203us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:07.800933 (+2224726us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:08.062573 (+261640us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b54002e81d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:08.062853 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5110421120}
W20260709 12:26:08.256980  9541 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.104s	user 0.009s	sys 0.004s
W20260709 12:26:08.258349  9535 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.212s	user 0.005s	sys 0.009s
I20260709 12:26:08.259938  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:08.165306  9555 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260709 12:26:08.157204  9489 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.030561  9559 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:08.275250  5706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2064) took 1835 ms (client timeout 2061 ms). Trace:
W20260709 12:26:08.276033  5706 rpcz_store.cc:269] 0709 12:26:06.439314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.615643 (+176329us) service_pool.cc:224] Handling call
0709 12:26:08.274961 (+1659318us) spinlock_profiling.cc:243] Waited 598 ms on lock 0x7b54002f5154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:08.275143 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1374502144}
W20260709 12:26:08.181164  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.946s	user 0.003s	sys 0.006s
W20260709 12:26:08.277154  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.042s	user 0.004s	sys 0.006s
I20260709 12:26:08.184144  5636 rpcz_store.cc:276] 0709 12:26:06.645251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.645532 (+   281us) service_pool.cc:224] Handling call
0709 12:26:06.645850 (+   318us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:06.646194 (+   344us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:06.646237 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:06.647159 (+   922us) log.cc:844] Serialized 52 byte log entry
0709 12:26:07.862482 (+1215323us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:07.862570 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.862739 (+   169us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:08.090472 (+227733us) raft_consensus.cc:1685] finished
0709 12:26:08.090491 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:08.090656 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":116133,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":47903,"thread_start_us":1079003,"threads_started":1,"wal-append.queue_time_us":1338614}
I20260709 12:26:08.184005  9207 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 12:26:08.279717  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:08.279824  5676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.902s	user 0.016s	sys 0.039s
W20260709 12:26:08.280329  5676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.903s	user 0.016s	sys 0.039s
I20260709 12:26:08.183624  9548 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.182934  9488 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:08.183297  9170 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.192261  9486 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:08.147612  9435 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:07.997323  9564 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.205084  9426 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:08.208494  9376 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.210769  9127 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:07.925542  9381 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.214241  9479 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:08.217913  9526 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.823s	user 0.011s	sys 0.000s
W20260709 12:26:08.224539  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 306120us
I20260709 12:26:08.314261  9509 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:08.363312  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2061) took 1924 ms. Trace:
I20260709 12:26:08.233566  9586 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.235160  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 684) took 1463 ms. Trace:
I20260709 12:26:08.186160  9285 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:08.238135  6234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.110s	user 0.038s	sys 0.036s
W20260709 12:26:08.376473  6234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.248s	user 0.039s	sys 0.036s
W20260709 12:26:08.376852  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 1.24874e+06us
I20260709 12:26:08.377493  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:08.238312  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:08.238893  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:08.379137  5630 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:08.239560  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 773) took 1723 ms. Trace:
I20260709 12:26:08.232427  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:08.243278  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 445877us
I20260709 12:26:08.380716  5635 raft_consensus.cc:2410] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
W20260709 12:26:08.182654  9364 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.216s	user 0.002s	sys 0.011s
I20260709 12:26:08.245412  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 673) took 1597 ms. Trace:
I20260709 12:26:08.184885  5710 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
W20260709 12:26:08.261173  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.840s	user 0.012s	sys 0.011s
W20260709 12:26:08.394373  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.974s	user 0.012s	sys 0.011s
W20260709 12:26:08.152830  9557 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.169s	user 0.001s	sys 0.001s
W20260709 12:26:08.162310  9554 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.084s	user 0.001s	sys 0.003s
I20260709 12:26:08.286163  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:08.290647  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:08.403900  5553 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:08.293475  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:08.404951  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:08.297043  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.328411  9426 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:08.251794  9543 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.242233  9321 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:08.444046  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:08.315011  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:08.318610  5558 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:08.445474  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:08.318696  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:08.322443  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:08.446864  5555 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:08.323215  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:08.322448  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:08.289072  9485 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.845s	user 0.003s	sys 0.054s
W20260709 12:26:08.451392  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 132799us
I20260709 12:26:08.452759  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:08.453259  6237 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:08.329277  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:08.319422  9404 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.539s	user 0.014s	sys 0.000s
I20260709 12:26:08.288785  9537 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.459852  9537 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.339649  9581 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:08.341845  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:08.482430  5636 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:08.341472  9376 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.313959  9450 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:08.492705  9450 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:08.365347  5527 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.040s	sys 0.037s
W20260709 12:26:08.496104  5527 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.040s	sys 0.037s
I20260709 12:26:08.320288  9507 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:08.330118  9539 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.969s	user 0.014s	sys 0.000s
I20260709 12:26:08.232331  5633 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:26:08.497439  9564 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.497339  9507 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.369007  5628 rpcz_store.cc:276] 0709 12:26:06.773133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.773277 (+   144us) service_pool.cc:224] Handling call
0709 12:26:06.773592 (+   315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:06.773977 (+   385us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:06.774016 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:06.783660 (+  9644us) log.cc:844] Serialized 26 byte log entry
0709 12:26:07.723858 (+940198us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b180040e8e0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:26:07.958115 (+234257us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:07.958164 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.958244 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:08.234700 (+276456us) raft_consensus.cc:1685] finished
0709 12:26:08.234734 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:08.235074 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":69269,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":16561,"spinlock_wait_cycles":4480384,"thread_start_us":233986,"threads_started":1,"wal-append.queue_time_us":1389626}
I20260709 12:26:08.498870  9550 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:26:08.499662  9507 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:08.365692  5704 rpcz_store.cc:276] 0709 12:26:06.438580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.712226 (+273646us) service_pool.cc:224] Handling call
0709 12:26:06.742570 (+ 30344us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:06.743136 (+   566us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:06.743181 (+    45us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:26:06.766946 (+ 23765us) log.cc:844] Serialized 78 byte log entry
0709 12:26:07.861532 (+1094586us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:26:08.338806 (+477274us) log.cc:844] Serialized 14 byte log entry
0709 12:26:08.339372 (+   566us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:08.362852 (+ 23480us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:08.362947 (+    95us) raft_consensus.cc:1685] finished
0709 12:26:08.362966 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:08.363223 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":941,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":11479,"thread_start_us":1445169,"threads_started":2,"wal-append.queue_time_us":1165873}
I20260709 12:26:08.379225  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1979) took 2620 ms. Trace:
I20260709 12:26:08.500692  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:08.500640  5556 rpcz_store.cc:276] 0709 12:26:05.746596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.746921 (+   325us) service_pool.cc:224] Handling call
0709 12:26:05.747269 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:06.538524 (+791255us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:06.538552 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:06.539021 (+   469us) log.cc:844] Serialized 26 byte log entry
0709 12:26:07.415163 (+876142us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:26:07.415218 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.415323 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:08.366561 (+951238us) raft_consensus.cc:1685] finished
0709 12:26:08.366586 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:08.366766 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15301,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":25286,"thread_start_us":1628168,"threads_started":2,"wal-append.queue_time_us":1366366}
I20260709 12:26:08.245100  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:08.502486  9463 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:08.383683  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:08.503062  9463 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.321975  9585 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.106s
I20260709 12:26:08.497476  9539 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.389037  9548 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.332819  9456 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:08.504432  9207 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:08.504761  9456 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:08.505211  9456 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:08.504967  9207 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.505970  9539 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:08.388108  5635 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:26:08.507113  9207 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:08.507563  5559 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:08.368932  9526 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:08.508673  5559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 1991) took 1446 ms (client timeout 1529 ms). Trace:
W20260709 12:26:08.508944  9542 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.326s	user 0.005s	sys 0.009s
W20260709 12:26:08.509088  5559 rpcz_store.cc:269] 0709 12:26:07.062248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:07.067903 (+  5655us) service_pool.cc:224] Handling call
0709 12:26:08.508458 (+1440555us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54001c43d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:08.508601 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3185587072}
I20260709 12:26:08.318439  9461 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:26:08.510056  9526 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.510284  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 744) took 2677 ms. Trace:
I20260709 12:26:08.401309  9185 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.087s
I20260709 12:26:08.511204  9185 consensus_queue.cc:1243] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 8.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:26:08.510833  6278 rpcz_store.cc:276] 0709 12:26:05.833256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.854474 (+ 21218us) service_pool.cc:224] Handling call
0709 12:26:05.893066 (+ 38592us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:05.893426 (+   360us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:05.893465 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:05.894121 (+   656us) log.cc:844] Serialized 26 byte log entry
0709 12:26:06.633021 (+738900us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:06.633075 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:06.633165 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:08.377414 (+1744249us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:08.510004 (+132590us) raft_consensus.cc:1685] finished
0709 12:26:08.510027 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:08.510224 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":61548928,"thread_start_us":684482,"threads_started":1,"wal-append.queue_time_us":1192728}
I20260709 12:26:08.379967  6280 rpcz_store.cc:276] 0709 12:26:06.515949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.516182 (+   233us) service_pool.cc:224] Handling call
0709 12:26:06.540614 (+ 24432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:06.540843 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:26:07.285528 (+744685us) log.cc:844] Serialized 14 byte log entry
0709 12:26:07.990699 (+705171us) log.cc:844] Serialized 14 byte log entry
0709 12:26:08.238836 (+248137us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:08.238909 (+    73us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:08.238982 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:08.239341 (+   359us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:08.239491 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1088747,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":73,"raft.queue_time_us":296105,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"thread_start_us":1645599,"threads_started":3,"wal-append.queue_time_us":938617}
W20260709 12:26:08.405985  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.014s	sys 0.010s
W20260709 12:26:08.527040  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.015s	sys 0.010s
W20260709 12:26:08.527661  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 681609us
I20260709 12:26:08.310463  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:08.337792  9435 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:08.328709  9566 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.074s
I20260709 12:26:08.405186  5560 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
W20260709 12:26:08.441037  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.442309  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:08.443161  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:08.443312  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:08.455346  5557 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:08.456053  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:08.400388  9388 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.088s
I20260709 12:26:08.539860  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:08.460011  5629 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:08.460158  5707 raft_consensus.cc:2410] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:26:08.471029  5555 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:08.461769  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 635) took 1850 ms. Trace:
I20260709 12:26:08.541780  5558 rpcz_store.cc:276] 0709 12:26:06.610962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.613326 (+  2364us) service_pool.cc:224] Handling call
0709 12:26:06.613604 (+   278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:08.445066 (+1831462us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:08.461435 (+ 16369us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54002283d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:08.461693 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3918628608}
I20260709 12:26:08.454794  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:08.399089  9364 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:26:08.477485  9473 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:08.477710  9447 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:26:08.478307  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 163408us
I20260709 12:26:08.477495  9411 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:08.487879  9338 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.479189  9579 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:08.479579  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:26:08.481415  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 138723us
I20260709 12:26:08.562119  5630 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
W20260709 12:26:08.226496  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 330461us
I20260709 12:26:08.376549  9561 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.080s
W20260709 12:26:08.475595  9485 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.038s	user 0.004s	sys 0.055s
I20260709 12:26:08.503614  9463 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:08.401427  5710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2065) took 1961 ms (client timeout 1597 ms). Trace:
I20260709 12:26:08.289315  9489 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.393376  5632 rpcz_store.cc:276] 0709 12:26:06.648334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.648587 (+   253us) service_pool.cc:224] Handling call
0709 12:26:06.648932 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:06.954492 (+305560us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:26:07.791563 (+837071us) log.cc:844] Serialized 14 byte log entry
0709 12:26:08.236089 (+444526us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:08.236145 (+    56us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:08.236198 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:08.245187 (+  8989us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b54001cf7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:08.245234 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:08.245359 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":420656,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":3191,"raft.queue_time_us":559,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":144,"spinlock_wait_cycles":702187264,"thread_start_us":1269523,"threads_started":2,"wal-append.queue_time_us":443640}
W20260709 12:26:08.595991  5710 rpcz_store.cc:269] 0709 12:26:06.439541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.538209 (+ 98668us) service_pool.cc:224] Handling call
0709 12:26:08.401236 (+1863027us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400451fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:08.401357 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3125251328}
I20260709 12:26:08.467722  9321 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.596658  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:08.597263  5632 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
W20260709 12:26:08.597821  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 118342us
I20260709 12:26:08.544862  9207 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:08.400841  9573 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:08.445098  9577 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.544806  9507 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:08.523141  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:08.530009  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:08.364114  9571 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.159s
W20260709 12:26:08.532836  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 222618us
I20260709 12:26:08.378916  9509 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.545859  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:08.618093  5631 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:08.546864  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:08.619854  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:08.620642  5628 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
W20260709 12:26:08.503437  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 119863us
I20260709 12:26:08.564121  9338 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:08.622197  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:08.489007  9562 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.478549  9590 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:26:08.556520  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 101795us
I20260709 12:26:08.393785  9586 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.559625  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:08.628741  6279 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:08.555751  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:08.630064  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 74397.6us
I20260709 12:26:08.563692  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:08.564705  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:08.456771  9543 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.569124  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:08.632328  5635 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:08.595224  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 1980) took 2830 ms. Trace:
I20260709 12:26:08.633318  5554 rpcz_store.cc:276] 0709 12:26:05.765289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.766156 (+   867us) service_pool.cc:224] Handling call
0709 12:26:05.792419 (+ 26263us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:06.576545 (+784126us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:06.576574 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:06.577025 (+   451us) log.cc:844] Serialized 26 byte log entry
0709 12:26:07.768239 (+1191214us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:07.768307 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.768462 (+   155us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:08.071000 (+302538us) spinlock_profiling.cc:243] Waited 4.57 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:08.564622 (+493622us) raft_consensus.cc:1685] finished
0709 12:26:08.564643 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:08.595141 (+ 30498us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":34915,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"spinlock_wait_cycles":10517760,"thread_start_us":1974331,"threads_started":2,"wal-append.queue_time_us":1354146}
I20260709 12:26:08.598107  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:08.450021  9569 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:26:08.544550  9539 raft_consensus.cc:2764] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:26:08.566955  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:08.552784  9447 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:08.595729  9526 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
W20260709 12:26:08.603979  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2147) took 2531 ms (client timeout 2137 ms). Trace:
I20260709 12:26:08.556576  9185 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:26:08.606477  9507 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.606511  9580 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:08.515173  9581 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:08.614380  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 84486.2us
I20260709 12:26:08.629088  9505 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:08.615789  9577 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.685756  5557 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:08.616729  9571 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:08.617157  6280 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46486
I20260709 12:26:08.547895  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:08.621694  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:08.693405  5628 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:08.495714  9450 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:08.617239  9537 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.562248  9579 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:08.600764  9591 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:08.560124  9583 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.636656  5632 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:26:08.637434  9338 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.603307  9486 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:26:08.617254  9597 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:26:08.594883  9485 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:08.566581  9592 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.094s
I20260709 12:26:08.619472  9423 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.663892  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:08.664788  5707 rpcz_store.cc:269] 0709 12:26:06.068221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.101468 (+ 33247us) service_pool.cc:224] Handling call
0709 12:26:08.567460 (+2465992us) spinlock_profiling.cc:243] Waited 2.3 s on lock 0x7b540006efd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:08.599033 (+ 31573us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5285557504}
I20260709 12:26:08.723884  5630 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:26:08.622378  9573 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:08.669569  5634 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:26:08.666913  9185 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:08.638062  9388 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:08.637820  9562 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.701522  9580 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:08.612273  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:08.770398  5636 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
W20260709 12:26:08.771731  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 159535us
I20260709 12:26:08.772010  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:08.772660  5636 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:26:08.682832  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:08.724694  5635 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
W20260709 12:26:08.669167  9599 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
W20260709 12:26:08.683988  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 74379.8us
I20260709 12:26:08.621652  9595 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 12:26:08.788031  5706 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:08.657444  9447 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:08.791642  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:08.697292  5631 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:08.638100  9600 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:26:08.698721  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 134093us
I20260709 12:26:08.657183  9170 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.666760  9404 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:08.699118  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:08.662827  9559 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:08.543740  9564 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.058s
W20260709 12:26:08.702190  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.702257  5628 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:08.814507  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:08.522421  9127 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.817224  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:08.818394  5709 raft_consensus.cc:2410] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:26:08.706650  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:08.719756  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:08.819442  5633 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:08.713052  9583 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.730219  9321 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.730995  9577 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.731415  9601 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:26:08.740859  9594 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.205s	user 0.001s	sys 0.000s
I20260709 12:26:08.685338  9574 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.056s
I20260709 12:26:08.741976  9507 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:08.743845  9509 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:08.745358  9586 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:08.746914  9456 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.748739  9376 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:08.748437  9485 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.750680  9488 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.751469  9338 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:08.753192  9602 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:26:08.754009  9463 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:08.754004  9093 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:08.713104  9486 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:08.692471  9571 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:08.762141  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:26:08.890889  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 128927us
I20260709 12:26:08.765367  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:08.888890  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 257414us
I20260709 12:26:08.893618  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:08.759790  5629 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260709 12:26:08.766572  5524 connection.cc:256] RPC call timeout handler was delayed by -2.6366600099995594s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1597s for a response
I20260709 12:26:08.681375  9453 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 12:26:08.680776  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:08.705066  9579 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:08.774950  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:08.775220  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:08.780606  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:08.781858  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.784488  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:08.863722  9601 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.786525  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:08.787328  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:08.906474  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:08.916507  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 141625us
I20260709 12:26:08.795507  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:08.809126  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:08.930071  5703 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:26:08.930469  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 135041us
I20260709 12:26:08.814730  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:08.651015  9473 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:26:08.820613  5554 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 8: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:08.819721  9423 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:08.780646  9388 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:08.839053  9604 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260709 12:26:08.857100  6277 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:26:08.859043  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:08.801705  8506 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:08.802182  9452 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.868057  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:08.790323  9582 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.016119  5709 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
W20260709 12:26:08.878844  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 193991us
I20260709 12:26:08.840191  9585 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.927851  9539 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:26:08.840505  9170 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:08.793255  9207 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.076s
I20260709 12:26:08.888413  6280 raft_consensus.cc:606] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:09.026523  6280 raft_consensus.cc:614] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:08.898746  5629 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:08.904503  5707 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:08.784832  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:09.029765  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:09.030593  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:09.031579  5710 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:08.836130  9285 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260709 12:26:08.903375  6275 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:08.918430  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:08.992339  9608 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.148s	user 0.000s	sys 0.000s
I20260709 12:26:08.899726  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:08.932114  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:08.955010  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:08.954888  5703 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:08.801277  9185 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:08.839433  9404 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:09.042963  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:09.042981  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:09.044119  5704 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 9 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:26:09.044744  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:09.044852  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:09.044185  9404 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:09.041728  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:09.045845  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:08.856493  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:08.906426  9611 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:08.995126  9488 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:08.885272  9093 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:08.917869  9463 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:09.000725  9473 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:09.002295  9456 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:08.991720  9607 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.226s	user 0.001s	sys 0.000s
W20260709 12:26:08.991031  9605 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.276s	user 0.000s	sys 0.001s
W20260709 12:26:09.010851  9542 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20260709 12:26:08.920161  5705 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:09.030566  5707 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:08.925102  9590 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:09.008435  9574 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:08.958663  9338 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:09.038295  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:09.007570  9543 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:09.039065  6276 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:09.046833  5708 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:09.052405  9611 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:08.892772  9580 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:09.039857  9503 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.191s	user 0.010s	sys 0.004s
I20260709 12:26:09.020139  9452 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:08.886564  9573 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.076386  6273 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:26:09.055002  9461 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.049090  9609 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:09.057938  9381 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:08.955291  9601 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.059854  9463 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:09.087767  9463 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:09.088604  9590 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.089363  9601 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:09.071843  9574 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:09.038744  9585 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:09.078992  9503 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:09.080117  9598 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.540s	user 0.004s	sys 0.000s
I20260709 12:26:09.079805  9461 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:09.059722  9473 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:09.100041  9503 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:09.101042  9473 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.102439  9473 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:09.103024  9585 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:09.082002  9452 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:09.065306  9614 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 12:26:09.049918  9486 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.127624  9486 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:09.151238  9577 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:09.152081  9577 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:09.157423  9426 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:09.159159  9426 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:09.077132  9611 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:09.167299  9622 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:26:09.171319  9583 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:09.056915  9571 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.184722  9562 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:09.100083  9461 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:09.194044  9571 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:09.093762  9574 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:09.111186  9619 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:09.123159  9463 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.218853  9463 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:09.195677  9489 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:09.196941  9581 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:09.037346  9582 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.222358  9582 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:09.225409  9549 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.161s	user 0.014s	sys 0.000s
I20260709 12:26:09.226372  9549 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:09.206526  9579 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.092669  9609 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:09.232863  9609 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:09.235190  9579 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:09.153435  9577 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.241356  9577 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:08.896791  5524 connection.cc:256] RPC call timeout handler was delayed by -2.2190600099997937s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2061s for a response
W20260709 12:26:08.904522  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 223643us
W20260709 12:26:09.333930  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.334561  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:09.338169  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:09.341640  5526 leader_election.cc:400] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12.
I20260709 12:26:09.344527  5526 leader_election.cc:403] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:26:09.356307  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:09.359649  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:09.359948  5556 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:09.358240  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:09.365572  5557 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:26:09.375478  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:09.053650  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:09.056037  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 124025us
I20260709 12:26:09.382624  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1918) took 1136 ms. Trace:
W20260709 12:26:09.040439  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 140780us
I20260709 12:26:09.050280  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:09.040738  9170 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:09.074160  9388 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.017987  9423 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.568817  9539 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:09.521229  9574 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.570272  9539 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:09.275823  9611 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:09.574005  5701 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:09.517726  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.211194  9447 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.534992  5629 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:09.536904  5705 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:09.482003  5627 rpcz_store.cc:276] 0709 12:26:08.245869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.385850 (+139981us) service_pool.cc:224] Handling call
0709 12:26:08.386510 (+   660us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:08.541492 (+154982us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:08.541533 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:08.557774 (+ 16241us) log.cc:844] Serialized 26 byte log entry
0709 12:26:08.703026 (+145252us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:26:08.703059 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:08.703238 (+   179us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:09.382081 (+678843us) raft_consensus.cc:1685] finished
0709 12:26:09.382122 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:09.382387 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":58628,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":32302,"thread_start_us":244657,"threads_started":2,"wal-append.queue_time_us":243610}
I20260709 12:26:09.558661  9549 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:09.559927  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.568969  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:09.572249  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 517452us
I20260709 12:26:09.522953  9564 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12.
W20260709 12:26:09.516611  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:09.579334  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:09.599920  9622 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:26:09.634464  9574 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:09.571283  9539 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:09.508102  9533 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:09.584776  5709 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:09.597064  9600 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:26:09.724763  9533 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:09.730713  9533 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:09.743026  5556 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
W20260709 12:26:09.745595  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:09.735903  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 360487us
I20260709 12:26:09.760280  5704 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
W20260709 12:26:09.783885  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 172018us
I20260709 12:26:09.791255  9611 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:09.791841  5710 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 11: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:09.793473  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:09.793838  9485 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.794643  9485 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:09.795828  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 751) took 1348 ms. Trace:
I20260709 12:26:09.797511  5709 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: [6.7-9.8]
I20260709 12:26:09.796294  6273 rpcz_store.cc:276] 0709 12:26:08.447014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.447151 (+   137us) service_pool.cc:224] Handling call
0709 12:26:08.462339 (+ 15188us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:09.078516 (+616177us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:09.795056 (+716540us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x7b54004fa6d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:09.795552 (+   496us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1411365760}
I20260709 12:26:09.798062  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:09.796568  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 263223us
I20260709 12:26:09.786597  9364 raft_consensus.cc:2764] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:26:09.897485  9207 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:09.898325  9207 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:09.882333  9539 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:09.883527  9285 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.719s	user 0.005s	sys 0.027s
I20260709 12:26:09.898842  9207 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:09.899189  9285 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.005s	sys 0.028s
I20260709 12:26:09.885720  5557 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:26:09.796172  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:26:09.903852  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 107870us
I20260709 12:26:09.889918  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2072) took 1103 ms. Trace:
I20260709 12:26:09.890298  5708 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:26:09.892107  9611 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.897320  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:09.882072  9634 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 12:26:09.900420  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:09.821827  9609 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.588s	user 0.009s	sys 0.010s
I20260709 12:26:09.911530  9285 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:26:09.914520  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 76846.9us
I20260709 12:26:09.915311  9561 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:09.922797  5635 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:26:09.924486  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:09.929886  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 650) took 1254 ms. Trace:
W20260709 12:26:09.934212  9632 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.181s	user 0.000s	sys 0.000s
W20260709 12:26:09.935523  5634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1914) took 1697 ms (client timeout 1916 ms). Trace:
I20260709 12:26:09.935716  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:09.803907  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:46486 (request call id 8) took 1206 ms. Trace:
I20260709 12:26:09.797838  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 750) took 1351 ms. Trace:
I20260709 12:26:09.796577  9321 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:09.957680  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.959072  9170 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:09.965590  9376 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:09.966687  9580 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.967619  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2199) took 1418 ms. Trace:
W20260709 12:26:09.968715  9426 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.769s	user 0.010s	sys 0.006s
I20260709 12:26:09.969363  5632 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:09.938941  5701 rpcz_store.cc:276] 0709 12:26:08.786252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.786389 (+   137us) service_pool.cc:224] Handling call
0709 12:26:09.889551 (+1103162us) spinlock_profiling.cc:243] Waited 640 ms on lock 0x7b5400451ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:09.889819 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1472797440}
I20260709 12:26:09.799180  9590 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:09.809031  9423 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:09.974778  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 688) took 1312 ms. Trace:
I20260709 12:26:09.980549  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:09.957854  9485 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:09.985555  5629 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
W20260709 12:26:09.986243  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50822.4us
I20260709 12:26:10.017735  5627 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 10: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:09.987452  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 687) took 1362 ms. Trace:
W20260709 12:26:09.979460  5634 rpcz_store.cc:269] 0709 12:26:08.238177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.238299 (+   122us) service_pool.cc:224] Handling call
0709 12:26:09.934998 (+1696699us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b540018d654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:09.935232 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":606989056}
I20260709 12:26:09.979712  6280 rpcz_store.cc:276] 0709 12:26:08.597479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.597654 (+   175us) service_pool.cc:224] Handling call
0709 12:26:09.803140 (+1205486us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54002e2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa66c853 00007f4fff63a60a 00007f4ff8bc07d4 00007f4ff8bc0762 00007f4ff8bc0691 00007f4ff8bc060c 00007f4ff8bbf8c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:09.803542 (+   402us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":449544960}
I20260709 12:26:10.021250  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 697) took 1017 ms. Trace:
I20260709 12:26:10.021317  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:10.021981  5629 rpcz_store.cc:276] 0709 12:26:09.003574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.029559 (+ 25985us) service_pool.cc:224] Handling call
0709 12:26:10.017889 (+988330us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:10.016964  5636 rpcz_store.cc:276] 0709 12:26:08.619923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.930606 (+1310683us) service_pool.cc:224] Handling call
0709 12:26:09.931278 (+   672us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:09.931773 (+   495us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:09.931809 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:09.931888 (+    79us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:09.931958 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:09.932032 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:09.932187 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:10.024366  5629 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:26:10.025233  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 689) took 1401 ms. Trace:
I20260709 12:26:10.025681  5629 rpcz_store.cc:276] 0709 12:26:08.624729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.023257 (+1398528us) service_pool.cc:224] Handling call
0709 12:26:10.023782 (+   525us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:10.024876 (+  1094us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:10.025158 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:10.029160  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:09.975920  5558 rpcz_store.cc:276] 0709 12:26:08.675650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.683248 (+  7598us) service_pool.cc:224] Handling call
0709 12:26:08.705814 (+ 22566us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:08.706217 (+   403us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:26:08.905208 (+198991us) log.cc:844] Serialized 14 byte log entry
0709 12:26:09.359601 (+454393us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:09.360987 (+  1386us) log.cc:844] Serialized 52 byte log entry
0709 12:26:09.361240 (+   253us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:09.361288 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:09.361420 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:09.875003 (+513583us) raft_consensus.cc:1685] finished
0709 12:26:09.875027 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:09.929649 (+ 54622us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":284020,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":125050,"raft.run_cpu_time_us":665,"raft.run_wall_time_us":55579,"thread_start_us":634342,"threads_started":2,"wal-append.queue_time_us":204428}
I20260709 12:26:09.986505  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:10.008826  9285 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:10.014780  9590 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:09.986500  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:10.026156  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 692) took 1370 ms. Trace:
I20260709 12:26:09.975405  9561 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:10.041028  5636 rpcz_store.cc:276] 0709 12:26:08.655994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.024853 (+1368859us) service_pool.cc:224] Handling call
0709 12:26:10.025333 (+   480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:10.025666 (+   333us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:10.025704 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:10.025777 (+    73us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:10.025828 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:10.025890 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:10.026020 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:10.019096  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 73803.9us
I20260709 12:26:09.978572  9609 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:10.047439  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 694) took 1364 ms. Trace:
I20260709 12:26:10.048471  5707 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:26:10.047861  5629 rpcz_store.cc:276] 0709 12:26:08.683478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.026785 (+1343307us) service_pool.cc:224] Handling call
0709 12:26:10.027285 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:10.046979 (+ 19694us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:10.047005 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:10.047078 (+    73us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:10.047130 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:10.047191 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:10.047340 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:10.046490  5560 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:10.049304  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2168) took 1237 ms. Trace:
I20260709 12:26:10.049680  5707 rpcz_store.cc:276] 0709 12:26:08.811770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.029466 (+217696us) service_pool.cc:224] Handling call
0709 12:26:10.049228 (+1019762us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:10.051581  9533 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.047729  9561 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:10.054212  9561 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:10.042881  9590 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:10.056921  5703 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:10.061982  5627 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
I20260709 12:26:10.062433  5627 pending_rounds.cc:85] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 6
I20260709 12:26:10.062729  5627 pending_rounds.cc:107] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260709 12:26:10.063015  5627 raft_consensus.cc:2889] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:10.063920  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 696) took 1136 ms. Trace:
I20260709 12:26:10.066130  9185 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:10.046463  9376 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:26:10.070127  9571 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.852s	user 0.015s	sys 0.005s
I20260709 12:26:10.071303  9573 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:10.071432  9376 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:10.071934  9573 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:10.072453  9573 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:10.074856  9611 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:10.076280  5709 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 7.
I20260709 12:26:10.077533  9611 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:10.078327  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:10.054526  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2078) took 1041 ms. Trace:
I20260709 12:26:10.079137  5705 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:10.079324  5710 rpcz_store.cc:276] 0709 12:26:09.012810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.037164 (+ 24354us) service_pool.cc:224] Handling call
0709 12:26:10.054206 (+1017042us) spinlock_profiling.cc:243] Waited 707 ms on lock 0x7b54001a8954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:10.054442 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1627054848}
I20260709 12:26:10.081003  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:26:10.081955  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51969.4us
I20260709 12:26:10.083443  8856 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:10.084120  8856 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:10.085845  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2163) took 1275 ms. Trace:
I20260709 12:26:10.085563  8856 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.086320  5708 rpcz_store.cc:276] 0709 12:26:08.810608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.492821 (+682213us) service_pool.cc:224] Handling call
0709 12:26:09.889788 (+396967us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:09.950936 (+ 61148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:10.085536 (+134600us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b5400451ad0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:10.085719 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":710637440}
I20260709 12:26:10.088086  9580 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.090039  5560 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:10.090708  9580 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:10.092108  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 405375us
I20260709 12:26:10.092983  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:10.093225  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:10.094918  9574 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:10.095347  9574 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:10.095741  9574 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:10.095544  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:10.025753  9093 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.101426  9093 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:10.101935  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:10.090729  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72658.6us
I20260709 12:26:10.108084  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:09.979873  6276 rpcz_store.cc:276] 0709 12:26:08.445974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.446169 (+   195us) service_pool.cc:224] Handling call
0709 12:26:09.797443 (+1351274us) spinlock_profiling.cc:243] Waited 488 ms on lock 0x7b54002e2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:09.797642 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1121841024}
I20260709 12:26:10.064280  5627 rpcz_store.cc:276] 0709 12:26:08.927057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.060969 (+1133912us) service_pool.cc:224] Handling call
0709 12:26:10.061427 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:10.063625 (+  2198us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:10.063834 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:10.123947  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 325478us
W20260709 12:26:10.145499  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 361482us
I20260709 12:26:10.054416  9533 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:10.009486  5628 rpcz_store.cc:276] 0709 12:26:08.548759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.384084 (+835325us) service_pool.cc:224] Handling call
0709 12:26:09.384687 (+   603us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:09.385716 (+  1029us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:26:09.386283 (+   567us) log.cc:844] Serialized 14 byte log entry
0709 12:26:09.769904 (+383621us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:09.934901 (+164997us) log.cc:844] Serialized 26 byte log entry
0709 12:26:09.935179 (+   278us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:26:09.935229 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:09.935334 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:09.967154 (+ 31820us) raft_consensus.cc:1685] finished
0709 12:26:09.967180 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:09.967411 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61979,"prepare.queue_time_us":41982,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":3854,"raft.queue_time_us":155721,"raft.run_cpu_time_us":745,"raft.run_wall_time_us":44871,"thread_start_us":311795,"threads_started":1,"wal-append.queue_time_us":575845}
W20260709 12:26:10.116222  9486 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.914s	user 0.005s	sys 0.023s
I20260709 12:26:10.161907  8506 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:26:10.175254  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.139029  9423 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:26:10.199436  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.221002  8506 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.227532  9641 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:26:10.228637  5702 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:10.234727  5632 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
W20260709 12:26:10.237354  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 69091.7us
W20260709 12:26:10.240721  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.019809  5635 rpcz_store.cc:276] 0709 12:26:08.619601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.921601 (+1302000us) service_pool.cc:224] Handling call
0709 12:26:09.922161 (+   560us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:09.981528 (+ 59367us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:09.981832 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:10.247548  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61834.2us
I20260709 12:26:10.217113  9485 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:26:10.239655  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 131704us
W20260709 12:26:10.261642  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 166287us
I20260709 12:26:10.227592  9537 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.156734  9321 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.185719  9571 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.063s
I20260709 12:26:10.206256  9574 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.228156  8506 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.287148  9644 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.288093  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2198) took 1731 ms. Trace:
I20260709 12:26:10.287998  9644 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:10.289279  8506 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:10.290159  9619 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:10.288569  5631 rpcz_store.cc:276] 0709 12:26:08.548498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.383720 (+835222us) service_pool.cc:224] Handling call
0709 12:26:09.384687 (+   967us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:09.537737 (+153050us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:26:09.538232 (+   495us) log.cc:844] Serialized 14 byte log entry
0709 12:26:10.096794 (+558562us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b18001b8b00. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a
0709 12:26:10.131852 (+ 35058us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:10.131931 (+    79us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:10.131985 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:10.279347 (+147362us) spinlock_profiling.cc:243] Waited 142 ms on lock 0x7b5400438354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:10.279403 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:10.279552 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37486,"prepare.queue_time_us":1647,"prepare.run_cpu_time_us":4306,"prepare.run_wall_time_us":95480,"raft.queue_time_us":54211,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":6374,"spinlock_wait_cycles":359132416,"thread_start_us":210701,"threads_started":1,"wal-append.queue_time_us":744812}
I20260709 12:26:10.292459  9644 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:10.293452  9604 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:26:10.293955  9604 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:10.295020  9604 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:10.293493  9577 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.051s	user 0.012s	sys 0.014s
W20260709 12:26:10.305123  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 293782us
W20260709 12:26:10.308260  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 68945.4us
I20260709 12:26:10.311930  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:10.311779  9486 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:10.313270  9571 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:10.320515  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55379.7us
W20260709 12:26:10.321938  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:10.328517  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 50428.2us
I20260709 12:26:10.329840  9571 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.334239  9571 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:10.336966  9533 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:10.339591  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 685) took 1789 ms. Trace:
I20260709 12:26:10.340057  5633 rpcz_store.cc:276] 0709 12:26:08.550491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.385894 (+835403us) service_pool.cc:224] Handling call
0709 12:26:09.386541 (+   647us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:09.387470 (+   929us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:09.387498 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:09.388181 (+   683us) log.cc:844] Serialized 26 byte log entry
0709 12:26:09.782903 (+394722us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:09.887063 (+104160us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b48002727d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:26:09.924918 (+ 37855us) log.cc:844] Serialized 14 byte log entry
0709 12:26:09.925417 (+   499us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:09.925499 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:10.339058 (+413559us) raft_consensus.cc:1685] finished
0709 12:26:10.339295 (+   237us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:10.339433 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":141432,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70,"raft.queue_time_us":93147,"raft.run_cpu_time_us":3296,"raft.run_wall_time_us":397280,"spinlock_wait_cycles":33148160,"thread_start_us":394165,"threads_started":1,"wal-append.queue_time_us":809600}
I20260709 12:26:10.342147  9604 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:10.352838  9590 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:10.354280  5706 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:10.355465  9590 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:10.356658  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2167) took 1545 ms. Trace:
I20260709 12:26:10.284461  9485 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:10.339708  9285 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.357049  5703 rpcz_store.cc:276] 0709 12:26:08.811549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.044500 (+232951us) service_pool.cc:224] Handling call
0709 12:26:10.356460 (+1311960us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540026eb54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:10.356596 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2325253504}
I20260709 12:26:10.360818  9590 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:10.367630  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:10.370731  9590 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:10.380769  9590 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:10.381942  9574 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:10.391435  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:10.391232  9590 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.284835  9537 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:10.392231  5559 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:10.404930  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:10.409750  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 512622us
I20260709 12:26:10.359772  9207 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:10.412858  5628 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:10.416940  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:10.419410  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:10.420003  5558 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:10.404414  9561 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:10.424458  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:10.432432  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 120609us
W20260709 12:26:10.450766  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 92565.4us
I20260709 12:26:10.455183  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2157) took 1643 ms. Trace:
I20260709 12:26:10.429958  9561 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:10.458789  5702 rpcz_store.cc:276] 0709 12:26:08.807615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.808440 (+   825us) service_pool.cc:224] Handling call
0709 12:26:08.808732 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:10.256148 (+1447416us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:10.424846 (+168698us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54002f4c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:10.450641 (+ 25795us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3059916160}
W20260709 12:26:10.455837  9381 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.232s	user 0.016s	sys 0.001s
I20260709 12:26:10.466135  5631 raft_consensus.cc:2410] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:26:10.464394  5633 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
W20260709 12:26:10.470364  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:10.471292  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.460317  9533 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.468210  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:10.467154  9561 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:10.486349  9190 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.094s
I20260709 12:26:10.488559  9609 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:10.488722  9207 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:10.489804  9447 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:10.490007  9537 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:10.497695  5627 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260709 12:26:10.498245  5627 pending_rounds.cc:85] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 9
I20260709 12:26:10.506346  5627 pending_rounds.cc:107] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 12:26:10.506807  5627 raft_consensus.cc:2889] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:10.450736  9485 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:10.515230  9533 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:10.521018  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54151us
I20260709 12:26:10.522617  9600 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:10.534493  9577 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:10.535137  9631 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.535880  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:10.506151  9590 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:10.567345  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:10.572611  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 346639us
W20260709 12:26:10.574662  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.549928  9600 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:10.561568  9631 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.562291  9321 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:10.582168  5631 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 10 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260709 12:26:10.560757  9364 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.548949  9381 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.033s
I20260709 12:26:10.559409  9388 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.159s
I20260709 12:26:10.566643  9577 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.578121  9600 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:10.603511  5558 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:10.601565  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:10.611553  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:10.612697  5554 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:26:10.611639  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:10.620723  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:10.621387  5558 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:10.618372  9321 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:26:10.627223  5526 connection.cc:256] RPC call timeout handler was delayed by -1.2122599429999354s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1555s for a response
W20260709 12:26:10.627727  5526 connection.cc:256] RPC call timeout handler was delayed by -1.2095599429999311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
W20260709 12:26:10.628041  5526 connection.cc:256] RPC call timeout handler was delayed by -1.1744599429998743s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1597s for a response
W20260709 12:26:10.632332  5526 connection.cc:256] RPC call timeout handler was delayed by -1.1492599429998336s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1625s for a response
W20260709 12:26:10.634093  5630 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.014s	sys 0.019s
W20260709 12:26:10.634534  5630 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.015s	sys 0.019s
I20260709 12:26:10.624034  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:10.599946  9364 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:10.636982  5636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.060s	sys 0.083s
I20260709 12:26:10.637506  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:10.643647  5636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.060s	sys 0.083s
I20260709 12:26:10.648205  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:10.644411  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:10.644519  9388 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:10.663048  9388 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:10.663180  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:10.667649  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47536.4us
I20260709 12:26:10.668830  5627 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:10.670117  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:10.663904  9364 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:10.674926  9641 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:10.652693  9381 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:26:10.679101  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:10.673154  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:10.653329  9533 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:10.680285  9641 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:10.666229  9185 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:10.685845  5627 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:10.668845  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:10.683748  5631 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:10.649870  9537 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:10.693877  5559 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:10.694265  6280 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:10.680899  9381 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:10.686061  9641 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:10.686908  9388 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:10.704480  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:10.692871  9185 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:10.709462  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:10.718116  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:10.696291  9644 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:10.699947  9364 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.726338  5557 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:10.702558  9381 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:10.730522  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:10.731750  5629 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:10.736011  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:10.721010  9185 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:10.721992  9644 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:10.741462  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.725219  9577 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:10.756186  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:10.757118  9644 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:10.782918  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:10.791205  5557 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:26:10.783712  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 76344.2us
I20260709 12:26:10.785873  5632 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:26:10.757296  9170 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:26:10.755944  9364 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:10.787868  9561 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.788689  5554 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:10.778556  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:10.794373  5634 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:10.781263  6236 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:10.732790  9537 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:10.775859  9619 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:26:10.778162  9388 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.812251  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:10.813429  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:10.814476  5558 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:26:10.818953  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:10.822162  6280 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:26:10.822556  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:10.826043  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44950.3us
I20260709 12:26:10.823235  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:10.828806  5633 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:10.826624  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:10.832237  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.832731  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:10.834479  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 98577.8us
I20260709 12:26:10.835796  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 695) took 1909 ms. Trace:
I20260709 12:26:10.819444  9321 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:10.837105  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:10.841427  9631 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:10.842021  9561 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:10.842264  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:10.852056  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 43805us
I20260709 12:26:10.842552  9388 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:10.857209  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:10.806036  6276 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46486
I20260709 12:26:10.857052  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:10.862452  6276 raft_consensus.cc:606] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:10.862849  6276 raft_consensus.cc:614] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:10.853230  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:10.862849  5559 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:10.867700  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:26:10.854404  9653 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.168s	user 0.000s	sys 0.001s
I20260709 12:26:10.869091  5631 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:10.836172  5636 rpcz_store.cc:276] 0709 12:26:08.926548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.057344 (+1130796us) service_pool.cc:224] Handling call
0709 12:26:10.057804 (+   460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:10.058360 (+   556us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:26:10.059259 (+   899us) log.cc:844] Serialized 14 byte log entry
0709 12:26:10.835266 (+776007us) log.cc:844] Serialized 14 byte log entry
0709 12:26:10.835453 (+   187us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:10.835498 (+    45us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:10.835540 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:10.835606 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:10.835724 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4673,"prepare.queue_time_us":241,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":66,"raft.queue_time_us":3437,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":5924,"thread_start_us":768871,"threads_started":2,"wal-append.queue_time_us":738339}
I20260709 12:26:10.846408  9661 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:10.852941  9533 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:10.895217  9207 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:10.895534  9533 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:10.895687  9207 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:10.896070  9533 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:10.896772  9207 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:10.897253  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:10.898108  9207 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:10.898202  8506 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:10.898613  8506 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:10.899536  8506 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:10.894317  9573 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.821s	user 0.016s	sys 0.000s
I20260709 12:26:10.901708  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 699) took 1108 ms. Trace:
I20260709 12:26:10.901562  9573 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.902114  5634 rpcz_store.cc:276] 0709 12:26:09.792869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.124919 (+332050us) service_pool.cc:224] Handling call
0709 12:26:10.155939 (+ 31020us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:10.799153 (+643214us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:10.901456 (+102303us) spinlock_profiling.cc:243] Waited 634 ms on lock 0x7b5400134f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:10.901645 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1458085248}
I20260709 12:26:10.876318  9537 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:10.878244  9486 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.878412  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:10.871850  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:10.880304  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:10.885797  6275 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: [6.7-9.8]
I20260709 12:26:10.886288  6277 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:26:10.886684  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:10.888872  9577 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:10.889349  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:10.890365  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:10.891121  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:10.891762  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:10.926064  5557 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:10.926712  5628 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:10.892740  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:10.894309  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:10.894630  9321 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:10.886785  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:10.928634  6272 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:10.903621  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:10.920456  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:10.921020  5553 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:10.919076  5558 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:10.921823  5633 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:10.924784  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:10.929165  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:10.930670  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.933459  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:10.933928  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 686) took 2335 ms. Trace:
I20260709 12:26:10.937054  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:10.923362  5636 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
W20260709 12:26:10.939867  5675 connection.cc:256] RPC call timeout handler was delayed by -1.491313554000044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18530000000000002s for a response
I20260709 12:26:10.942695  5561 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:10.943759  5634 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:10.944401  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:10.934293  5630 rpcz_store.cc:276] 0709 12:26:08.598196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.387865 (+789669us) service_pool.cc:224] Handling call
0709 12:26:09.388562 (+   697us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:09.529412 (+140850us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:09.529458 (+    46us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:26:09.783531 (+254073us) log.cc:844] Serialized 78 byte log entry
0709 12:26:09.892737 (+109206us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b48002727d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:26:10.634972 (+742235us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:10.635029 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:10.635128 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:10.933601 (+298473us) raft_consensus.cc:1685] finished
0709 12:26:10.933623 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:10.933846 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19912,"raft.run_cpu_time_us":1705,"raft.run_wall_time_us":363143,"spinlock_wait_cycles":32806272,"thread_start_us":628847,"threads_started":1,"wal-append.queue_time_us":720826}
I20260709 12:26:10.948104  5552 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
W20260709 12:26:10.955533  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.956017  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:10.962865  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:10.996975  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:10.924479  9661 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:10.997709  6271 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:11.001098  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:11.001370  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.072878  5633 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:10.964075  5628 raft_consensus.cc:2376] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13. Re-sending same reply.
I20260709 12:26:11.002306  6274 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:11.004590  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:11.004878  6275 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260709 12:26:11.005005  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 84190.2us
I20260709 12:26:11.002914  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:11.001649  9170 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.005620  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.009325  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.012894  5554 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:10.924707  9486 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:11.016549  9537 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260709 12:26:11.017660  9388 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:11.086011  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:11.017776  9573 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:11.018148  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.018842  5560 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:11.019683  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:11.021113  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:11.022869  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:11.027618  9604 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.685s	user 0.012s	sys 0.000s
W20260709 12:26:11.089486  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 46940us
I20260709 12:26:11.028566  8506 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:11.030246  9364 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:11.033581  9600 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:11.036966  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:11.096377  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:11.097016  5628 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:11.097221  5627 raft_consensus.cc:2410] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:26:11.038585  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:11.039449  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:11.039620  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.040196  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.004590  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:11.041160  9093 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.909s	user 0.014s	sys 0.001s
I20260709 12:26:11.041256  6277 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 12:26:11.042883  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.017568  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:11.098372  5708 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:11.047624  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:11.100329  5635 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:11.053061  9644 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:11.054044  9381 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:11.059458  5634 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:11.061866  9652 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.063585  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.064708  9609 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.065996  5701 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:11.067293  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:11.070603  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.072942  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:11.073256  9533 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:11.074337  6272 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:11.074816  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.126171  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:11.126435  5709 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:26:11.126751  5556 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
W20260709 12:26:11.008823  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.078569  5630 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 11: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:11.079550  5552 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:11.085069  6280 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:11.078649  5561 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:11.090744  5553 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:11.091670  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:11.093211  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:11.037727  5557 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:11.098340  5703 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:26:11.094326  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.049669  9185 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:11.103065  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.137775  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:11.138281  5560 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
W20260709 12:26:11.108073  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 246665us
I20260709 12:26:11.139833  5627 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:11.125137  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.121456  5702 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:11.137158  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.122305  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:11.144156  5634 raft_consensus.cc:2410] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 12 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:26:11.135196  5630 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 12:26:11.136165  5554 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:11.132902  5706 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:26:11.125324  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:11.103415  9571 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.137137  5559 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:11.132771  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:11.149539  5628 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 8: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:11.145005  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:11.125417  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:11.145998  5633 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:11.104343  9537 consensus_queue.cc:1243] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 9.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:26:11.167768  9609 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:11.168743  9486 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:11.169121  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.170047  5560 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:11.151816  9669 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:26:11.171103  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.171283  9533 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:11.171950  5701 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:26:11.148308  9641 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:11.176821  9641 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:11.177371  5627 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:26:11.177630  5629 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:11.177997  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:11.178643  5634 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:11.178848  5627 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: []
I20260709 12:26:11.179425  5555 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:11.133955  5710 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:11.155562  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.156307  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.180287  9561 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:11.156052  9661 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:11.182583  9561 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:11.183107  9561 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:11.184659  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:11.185305  5707 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:11.156975  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:11.183979  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:11.187335  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:11.097913  9170 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:11.156639  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.188041  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:11.188793  5561 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:11.189020  9537 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:11.189378  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.190183  9170 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:11.190752  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.191004  5631 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:26:11.191591  5553 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 9: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:11.161191  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.162039  9600 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:11.119717  9388 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.162750  9577 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.163277  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:11.164340  6279 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:11.165665  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.167137  9666 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:11.168011  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 43856.5us
I20260709 12:26:11.170104  9381 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:11.178637  9571 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:11.182094  5557 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:11.154889  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:11.192649  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.127892  9652 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:11.171321  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:11.154160  9185 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:11.193850  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:11.194960  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 119678us
I20260709 12:26:11.195875  6274 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:11.196316  5558 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:11.188089  5554 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:11.196791  5632 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:11.198618  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.195973  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:11.199777  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 108187us
I20260709 12:26:11.200528  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:11.212869  5707 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:11.219420  5556 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:11.222065  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.222498  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.223443  5560 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:11.225507  5636 raft_consensus.cc:2410] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:26:11.148828  9619 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.233264  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.234133  5553 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:11.177696  9644 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:11.170233  9609 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:11.236564  5710 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:11.237082  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.199398  9539 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:11.340915  5553 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:11.328864  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.242241  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:11.347493  6271 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:26:11.318358  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:11.242650  5628 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:26:11.350168  5705 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:11.323179  5552 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:26:11.339502  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.352162  5709 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:11.353435  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:26:11.321746  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 62661.9us
I20260709 12:26:11.354604  8506 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:11.347529  5707 raft_consensus.cc:2410] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:26:11.355194  8506 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:11.356082  8506 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:11.358598  5635 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:11.340893  5561 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:11.360004  5703 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 9. Current term is 10. Ops: []
W20260709 12:26:11.370306  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.360440  5554 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:11.373844  5635 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:11.376175  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:11.376958  5706 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:11.258493  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:11.379895  5561 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: [8.4-8.4]
W20260709 12:26:11.295322  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53893.4us
I20260709 12:26:11.261444  5704 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:11.383332  5560 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:11.281723  5634 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:11.385241  5627 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:11.385188  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:11.258728  9573 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:11.390367  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:11.391906  5710 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:11.392927  5557 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:11.401583  5558 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:11.406426  5553 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:26:11.409377  5552 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:26:11.319136  9381 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:11.410681  5709 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:11.452967  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.454099  5709 raft_consensus.cc:2410] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20260709 12:26:11.409399  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 213512us
I20260709 12:26:11.413311  5708 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:26:11.475018  5704 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
W20260709 12:26:11.417757  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 271134us
I20260709 12:26:11.213363  9577 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:11.431874  6272 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46486
I20260709 12:26:11.486851  6272 raft_consensus.cc:606] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:11.487475  6272 raft_consensus.cc:614] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:11.235997  9619 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:11.492726  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.389062  9539 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:11.437824  9207 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:11.505259  5707 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:26:11.438583  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 53642us
I20260709 12:26:11.443082  9652 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:11.447661  5560 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:26:11.225518  9666 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:11.448781  5561 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:11.449584  9676 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:11.225706  9388 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:11.715839  9676 consensus_queue.cc:1243] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:26:11.716468  5558 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: [6.3-8.4]
I20260709 12:26:11.717535  9388 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:11.416554  9644 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:26:11.433313  9678 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:26:11.718734  9644 consensus_queue.cc:1243] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 10.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 12:26:11.500005  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:11.720506  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:11.566103  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 101655us
I20260709 12:26:11.721210  8506 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:11.618757  5561 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:11.721902  8506 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:11.723330  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.724119  5561 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:11.725107  8506 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:11.725356  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.726383  5561 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:11.727236  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.727928  5561 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:11.615983  5710 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:11.599552  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:11.603572  5706 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:11.729456  5560 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:11.729674  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.729864  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.730811  5560 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:11.730980  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.731307  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:11.616937  6234 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.698s (SENT) [suppressed 18 similar messages]
I20260709 12:26:11.525952  9539 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:11.734596  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.738204  5710 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 11: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:11.739972  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.648289  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.606570  9652 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:26:11.603016  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.744282  5707 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 8.
I20260709 12:26:11.605821  9577 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:11.610278  9666 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:11.526106  9207 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:11.606567  9619 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:11.648823  5628 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:11.749794  5628 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:11.600740  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:11.525734  9170 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.086s
I20260709 12:26:11.651485  5705 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
W20260709 12:26:11.606383  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 365765us
I20260709 12:26:11.598258  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:11.689213  5634 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:26:11.621826  9561 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:11.736949  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:11.737023  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:11.750689  5708 raft_consensus.cc:2376] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13. Re-sending same reply.
I20260709 12:26:11.752298  8506 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:11.604291  9185 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:11.646744  5633 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:11.774806  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:11.788932  5560 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:26:11.749253  9207 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:11.798236  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:11.842528  9539 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:11.795310  9381 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:11.857839  9539 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:12.083252  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 304544us
W20260709 12:26:12.007001  5701 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.023s	sys 0.053s
W20260709 12:26:11.953608  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 231788us
W20260709 12:26:12.084407  5701 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.026s	sys 0.055s
W20260709 12:26:12.082968  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 596693us
W20260709 12:26:11.891973  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 275016us
I20260709 12:26:12.065495  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:12.086683  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:12.035781  5628 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:11.796805  9619 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:26:12.095465  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:12.010090  9665 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:26:12.097738  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:12.098524  5710 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:12.101263  8506 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:12.101789  8506 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:12.102210  8506 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:11.858657  9381 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:11.860893  9561 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:12.104467  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:11.861434  9185 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:12.097522  9364 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:12.125000  9364 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:12.128101  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.120203  8506 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.134243  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:12.101071  9619 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:12.140407  9619 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.144120  5628 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:12.136961  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:12.137542  9577 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:26:12.146064  8506 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:12.139184  9694 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:12.152843  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.138991  9381 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.148979  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:12.150571  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:12.124015  9185 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:12.157275  5710 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:12.160724  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:12.168206  8506 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:12.147107  9561 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:12.143743  9207 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:12.151614  9674 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:12.158664  9185 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:12.170361  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.159557  9571 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:12.176784  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1982) took 1149 ms. Trace:
I20260709 12:26:12.172155  8506 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:12.158982  9539 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.180090  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.154215  9364 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:12.182581  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:12.158486  9666 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.013s
W20260709 12:26:12.190101  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 138154us
I20260709 12:26:12.166674  9285 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:26:12.189739  9692 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:12.209458  9692 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:12.209841  9692 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:12.172482  9388 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:26:12.216635  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55995.9us
I20260709 12:26:12.178226  9674 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:12.227113  8506 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.189085  5552 rpcz_store.cc:276] 0709 12:26:11.027178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.410427 (+383249us) service_pool.cc:224] Handling call
0709 12:26:11.460706 (+ 50279us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:11.460972 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:11.461177 (+   205us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:11.504790 (+ 43613us) log.cc:844] Serialized 26 byte log entry
0709 12:26:11.890983 (+386193us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:12.064570 (+173587us) log.cc:844] Serialized 14 byte log entry
0709 12:26:12.082620 (+ 18050us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:12.082736 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:12.172459 (+ 89723us) raft_consensus.cc:1685] finished
0709 12:26:12.172481 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:12.176721 (+  4240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17608,"prepare.queue_time_us":137987,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":20538,"raft.queue_time_us":17779,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":3101,"spinlock_wait_cycles":1409280,"thread_start_us":286105,"threads_started":1,"wal-append.queue_time_us":488398}
I20260709 12:26:12.177750  9577 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
W20260709 12:26:12.235261  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.235503  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:12.240446  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 644) took 1170 ms. Trace:
I20260709 12:26:12.241705  5701 rpcz_store.cc:276] 0709 12:26:11.069710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.173842 (+104132us) service_pool.cc:224] Handling call
0709 12:26:11.174121 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:11.174372 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:26:11.310473 (+136101us) log.cc:844] Serialized 14 byte log entry
0709 12:26:12.240084 (+929611us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:12.240132 (+    48us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:12.240176 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:12.240239 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:12.240389 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":249998,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":53588,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":82,"thread_start_us":954860,"threads_started":3,"wal-append.queue_time_us":748098}
I20260709 12:26:12.246351  9388 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:12.223824  9674 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:12.252889  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 158208us
I20260709 12:26:12.223603  9285 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:12.215443  9539 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:12.257390  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1938) took 1430 ms. Trace:
I20260709 12:26:12.264063  5627 rpcz_store.cc:276] 0709 12:26:10.827400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.179588 (+352188us) service_pool.cc:224] Handling call
0709 12:26:11.384930 (+205342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:11.385838 (+   908us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:12.257186 (+871348us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b540032c8d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:12.257326 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":471559040}
I20260709 12:26:12.236059  9694 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.033s
I20260709 12:26:12.249044  9388 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.272234  9700 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:12.285508  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 105529us
I20260709 12:26:12.277562  9700 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.289520  9695 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:12.291594  9539 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:12.293898  9704 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:12.295029  9698 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:26:12.295269  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:12.296305  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:12.297082  9698 raft_consensus.cc:2764] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:26:12.297784  9698 raft_consensus.cc:2764] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:26:12.298085  9700 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:12.298596  9696 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:26:12.292670  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:12.299683  5702 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:12.298008  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:12.301703  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 131397us
I20260709 12:26:12.302371  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:12.302450  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:12.305217  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1939) took 1453 ms. Trace:
I20260709 12:26:12.305609  5632 rpcz_store.cc:276] 0709 12:26:10.851744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.892287 (+ 40543us) service_pool.cc:224] Handling call
0709 12:26:12.305059 (+1412772us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b540018c254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:12.305159 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":558978432}
I20260709 12:26:12.300873  9695 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:12.307847  9695 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:12.314052  9674 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:12.317999  5702 raft_consensus.cc:2410] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:26:12.317883  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:12.324513  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.325774  5628 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20260709 12:26:12.326488  5628 pending_rounds.cc:85] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 3
I20260709 12:26:12.326879  5628 pending_rounds.cc:107] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20260709 12:26:12.327188  5628 raft_consensus.cc:2889] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:12.328910  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 76503.6us
I20260709 12:26:12.330593  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1951) took 1308 ms. Trace:
I20260709 12:26:12.330772  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:12.330948  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:12.330137  9696 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:12.332405  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:12.331073  5628 rpcz_store.cc:276] 0709 12:26:11.022142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.366352 (+344210us) service_pool.cc:224] Handling call
0709 12:26:11.392871 (+ 26519us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:12.330348 (+937477us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:12.330540 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:12.336469  5524 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.597s (SENT)
I20260709 12:26:12.320566  9669 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:12.338938  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:12.336510  9696 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:12.348800  9697 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:12.349355  9697 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:12.350009  9697 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:12.352576  9702 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:12.353513  9702 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:12.354512  9702 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:12.365020  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 62704us
I20260709 12:26:12.324559  9698 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
W20260709 12:26:12.374622  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 43738.7us
I20260709 12:26:12.342852  9285 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:12.354826  9695 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.379987  8506 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.380229  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:12.364123  9609 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:12.383284  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:12.382141  9696 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:12.393095  9698 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:12.393289  9696 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:12.393652  9698 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:12.394083  9696 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:12.394748  9698 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.398376  9695 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:12.399528  9695 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:26:12.400972  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.401472  9573 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:12.402240  9573 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:12.403134  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.404165  9571 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:12.380695  8506 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.400156  9695 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:12.343302  9669 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:12.419884  9695 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.420943  9669 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:12.439404  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:12.441236  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 107693us
W20260709 12:26:12.442384  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 66581.1us
I20260709 12:26:12.443096  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.445359  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.444211  5602 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.404883  9571 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:26:12.456455  8506 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:12.457171  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 670) took 1592 ms. Trace:
I20260709 12:26:12.457657  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.458487  8506 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:12.380473  9388 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.066s
W20260709 12:26:12.471346  5636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1946) took 1575 ms (client timeout 1803 ms). Trace:
W20260709 12:26:12.473026  5636 rpcz_store.cc:269] 0709 12:26:10.895845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.949711 (+ 53866us) service_pool.cc:224] Handling call
0709 12:26:12.463999 (+1514288us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b540009ecd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:12.471250 (+  7251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":330404608}
I20260709 12:26:12.434911  9698 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:12.474855  8506 raft_consensus.cc:2764] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:26:12.457610  5559 rpcz_store.cc:276] 0709 12:26:10.864945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.865108 (+   163us) service_pool.cc:224] Handling call
0709 12:26:10.910227 (+ 45119us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:11.174247 (+264020us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:12.456861 (+1282614us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b54001c48d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:12.457098 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":500699520}
I20260709 12:26:12.441461  9666 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:12.455174  9619 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:26:12.453099  9703 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:12.476270  9695 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.488714  9695 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.491596  9695 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:12.481701  9669 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:26:12.492457  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51567.1us
I20260709 12:26:12.493327  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:26:12.493882  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48597.3us
I20260709 12:26:12.496176  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.486915  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:12.499910  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:12.500675  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.504248  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 706) took 1703 ms. Trace:
I20260709 12:26:12.504689  5631 rpcz_store.cc:276] 0709 12:26:10.801310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.020819 (+219509us) service_pool.cc:224] Handling call
0709 12:26:12.504056 (+1483237us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54001ce8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:12.504186 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":232609664}
W20260709 12:26:12.508474  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.477026  9388 consensus_queue.cc:1243] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:26:12.510710  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:12.472136  9609 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:12.483461  9619 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:12.510152  9571 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:26:12.474277  9692 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.518131  9619 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:12.533011  9713 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.517802  9666 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:12.603685  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1979) took 1577 ms. Trace:
I20260709 12:26:12.608745  5558 rpcz_store.cc:276] 0709 12:26:11.026380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.408082 (+381702us) service_pool.cc:224] Handling call
0709 12:26:11.408541 (+   459us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:11.717022 (+308481us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:12.539031 (+822009us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b540023fd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:12.603609 (+ 64578us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":707230464}
W20260709 12:26:12.609977  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 89821.8us
I20260709 12:26:12.613649  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.617084  9717 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:12.507840  9185 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.520388  9571 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:12.610924  9604 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 12:26:12.595629  9703 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.616207  9666 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:26:12.630435  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 119842us
W20260709 12:26:12.634471  9712 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.101s	user 0.010s	sys 0.007s
I20260709 12:26:12.637800  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:12.638721  9388 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:12.639139  9388 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:12.639489  9388 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:12.606304  9577 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:12.517926  9573 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:12.641263  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:12.645188  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 149093us
W20260709 12:26:12.649114  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 136049us
I20260709 12:26:12.619575  9708 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:12.650837  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.630888  9571 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:12.651652  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.657956  9573 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:12.659566  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:12.662985  9704 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:12.663471  9704 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:12.650372  9577 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:12.660506  9703 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.665313  9692 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:12.666906  9703 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:12.659147  9609 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.668793  9571 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:12.669862  9285 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:12.670837  9571 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:12.616962  5447 maintenance_manager.cc:643] P b3df7d767dc641b7bb01c3435d842dd5: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 7.678s	user 0.176s	sys 0.084s Metrics: {"bytes_written":29588,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":387588,"dirs.run_cpu_time_us":842,"dirs.run_wall_time_us":23719,"drs_written":1,"lbm_read_time_us":734,"lbm_reads_lt_1ms":4,"lbm_write_time_us":178890,"lbm_writes_10-100_ms":7,"lbm_writes_lt_1ms":26,"peak_mem_usage":0,"reinsert_count":0,"rows_written":63,"thread_start_us":161419,"threads_started":1,"update_count":14}
I20260709 12:26:12.677299  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:12.680516  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47663us
I20260709 12:26:12.681777  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:12.682729  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 1973) took 1782 ms. Trace:
I20260709 12:26:12.653602  9702 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:12.711174  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:12.711477  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:12.690654  9722 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:12.691493  9539 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:12.715806  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:12.692735  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:12.652429  9708 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:12.611351  9713 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.683205  5553 rpcz_store.cc:276] 0709 12:26:10.900247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.360032 (+459785us) service_pool.cc:224] Handling call
0709 12:26:11.406081 (+ 46049us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:11.406978 (+   897us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:12.674937 (+1267959us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b540044e650. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:12.682659 (+  7722us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41091968}
I20260709 12:26:12.712559  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:12.710728  5708 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:12.714499  9721 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:26:12.715173  9364 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.735390  9285 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:12.735531  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:12.735924  5448 maintenance_manager.cc:419] P b3df7d767dc641b7bb01c3435d842dd5: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 8497 bytes on disk
W20260709 12:26:12.735998  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.711150  9715 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:12.709095  9702 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:12.736886  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 90341.8us
W20260709 12:26:12.799630  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 148929us
I20260709 12:26:12.711853  5701 raft_consensus.cc:2376] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10. Re-sending same reply.
I20260709 12:26:12.816213  5706 raft_consensus.cc:2376] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9. Re-sending same reply.
I20260709 12:26:12.671983  9697 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:12.828158  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:12.816641  5708 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:12.709851  9666 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:12.815604  9708 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:12.824398  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:12.837922  5705 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:26:12.845268  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:12.820268  9364 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:12.850477  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:12.847081  5555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2016) took 1752 ms (client timeout 1780 ms). Trace:
I20260709 12:26:12.857153  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:12.857743  5706 raft_consensus.cc:2376] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8. Re-sending same reply.
I20260709 12:26:12.859321  5554 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:12.859690  5554 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:12.852613  5555 rpcz_store.cc:269] 0709 12:26:11.090245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.099314 (+  9069us) service_pool.cc:224] Handling call
0709 12:26:12.838799 (+1739485us) spinlock_profiling.cc:243] Waited 36.9 ms on lock 0x7b540044eb54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:12.842446 (+  3647us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":84953216}
I20260709 12:26:12.865773  5447 maintenance_manager.cc:643] P b3df7d767dc641b7bb01c3435d842dd5: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.034s	user 0.002s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":9688,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20260709 12:26:12.830292  9702 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:12.824466  9561 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:12.829934  9571 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:12.884891  6277 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:12.885293  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:12.892895  5706 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:12.894479  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 180980us
I20260709 12:26:12.828729  9722 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:12.742893  9704 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:12.850402  5708 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:12.889261  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44941.7us
I20260709 12:26:12.889398  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 713) took 1425 ms. Trace:
I20260709 12:26:12.897729  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:12.885492  5628 raft_consensus.cc:1217] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Deduplicated request from leader. Original: 7.8->[8.9-12.11]   Dedup: 8.9->[11.10-12.11]
I20260709 12:26:12.898984  5628 pending_rounds.cc:85] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 9
I20260709 12:26:12.899662  5628 pending_rounds.cc:107] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 12:26:12.899989  5628 raft_consensus.cc:2889] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:12.900317  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:12.892105  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:12.897416  5635 rpcz_store.cc:276] 0709 12:26:11.451055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.875399 (+1424344us) service_pool.cc:224] Handling call
0709 12:26:12.875703 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:12.875883 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:12.875908 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:12.875950 (+    42us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:12.875988 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:12.876039 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:12.876140 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:12.903913  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 74063.4us
I20260709 12:26:12.900555  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:12.957482  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 278486us
I20260709 12:26:12.856024  9703 raft_consensus.cc:2764] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:26:12.957693  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:12.960136  5636 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:12.960126  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:12.961025  5635 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:26:12.961944  5701 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 11: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:12.967906  5708 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:12.964694  5554 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:12.975572  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:12.976567  5636 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:12.901707  9722 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:13.020692  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 123059us
W20260709 12:26:13.035614  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 159708us
I20260709 12:26:12.902964  9539 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:12.856885  9715 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.119s
I20260709 12:26:12.896597  9573 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.081s
I20260709 12:26:13.061265  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:12.962486  9185 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.062252  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:13.036860  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:13.075250  5635 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:12.902828  9571 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.035542  9704 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:13.073096  5701 raft_consensus.cc:2376] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10. Re-sending same reply.
I20260709 12:26:13.076375  5554 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:13.079790  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 57542us
I20260709 12:26:13.077613  5706 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
W20260709 12:26:13.061067  9665 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
W20260709 12:26:13.071342  9696 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.676s	user 0.008s	sys 0.006s
I20260709 12:26:13.084534  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:13.086895  9720 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
I20260709 12:26:13.087874  9364 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:13.088321  9185 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:13.091714  6277 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:13.092129  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:13.102483  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:13.102697  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.097546  5554 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 8. (term mismatch)
I20260709 12:26:13.101119  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:13.084012  9631 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:13.104703  5554 pending_rounds.cc:85] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 7
I20260709 12:26:13.104306  5635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:13.105547  5554 pending_rounds.cc:107] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20260709 12:26:13.105684  5635 raft_consensus.cc:606] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:13.105945  5554 raft_consensus.cc:2889] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:13.106145  5635 raft_consensus.cc:614] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:13.112389  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.113406  5706 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:13.126034  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:13.126066  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:13.126996  5706 raft_consensus.cc:2368] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9.
I20260709 12:26:13.127635  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:13.128485  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:13.131867  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:13.129314  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:13.134016  6272 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:26:13.132091  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.098819  9573 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:13.135675  5636 raft_consensus.cc:2410] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:26:13.135772  6234 leader_election.cc:400] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9.
I20260709 12:26:13.136400  6234 leader_election.cc:403] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:26:13.136751  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2027) took 2021 ms. Trace:
I20260709 12:26:13.134822  5701 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:26:13.137252  5554 rpcz_store.cc:276] 0709 12:26:11.115632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.477251 (+361619us) service_pool.cc:224] Handling call
0709 12:26:11.654182 (+176931us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:12.859214 (+1205032us) spinlock_profiling.cc:243] Waited 65.1 ms on lock 0x7b5400233aa0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa6741c0 00007f4ffa675231 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c
0709 12:26:13.136411 (+277197us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:13.136676 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149686656}
I20260709 12:26:13.140455  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:13.125217  9715 consensus_queue.cc:1243] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:26:13.141577  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.141072  6273 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:13.138473  9364 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:13.145196  9644 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260709 12:26:13.114840  9721 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:13.146828  9571 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.152802  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:13.194176  5630 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:13.156757  9723 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.326s	user 0.001s	sys 0.000s
I20260709 12:26:13.158296  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2181) took 2120 ms. Trace:
W20260709 12:26:13.159451  9619 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.543s	user 0.006s	sys 0.001s
I20260709 12:26:13.199795  6274 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:13.159998  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.112463  9700 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:13.201102  9700 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:13.201555  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.202747  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.202118  9700 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.203827  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.199952  9619 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.204605  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.206001  5706 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:13.206336  9619 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.207216  5705 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:13.207449  5706 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:26:13.208266  5705 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
W20260709 12:26:13.210625  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.212591  9716 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:13.215306  9726 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:26:13.215881  9726 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:13.218470  9577 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.156159  5709 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
W20260709 12:26:13.226289  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 189542us
I20260709 12:26:13.138137  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:13.177687  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.108014  9631 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:13.146585  9713 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:13.156100  9388 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.516s	user 0.010s	sys 0.000s
I20260709 12:26:13.189718  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:13.190814  5708 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:26:13.191883  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:13.189724  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:13.113487  9185 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.139706  9093 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:13.103652  9561 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.063s
I20260709 12:26:13.101092  9696 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.198110  5704 rpcz_store.cc:276] 0709 12:26:11.038127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.038259 (+   132us) service_pool.cc:224] Handling call
0709 12:26:13.158088 (+2119829us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54002d19d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:13.158206 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":376073472}
I20260709 12:26:13.227981  5701 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:13.190131  9604 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:26:13.256242  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 118185us
I20260709 12:26:13.232513  6277 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:13.231719  5631 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:26:13.231019  6278 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 762) took 2334 ms (client timeout 1826 ms). Trace:
I20260709 12:26:13.242828  5708 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:13.179437  9573 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:26:13.250650  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58927.1us
I20260709 12:26:13.185138  9364 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.253769  5706 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: [9.9-12.11]
W20260709 12:26:13.263373  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.264242  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:13.264911  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.265556  5704 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 12:26:13.257849  6278 rpcz_store.cc:269] 0709 12:26:10.863208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.878012 (+ 14804us) service_pool.cc:224] Handling call
0709 12:26:13.192809 (+2314797us) spinlock_profiling.cc:243] Waited 727 ms on lock 0x7b5400711954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:13.197633 (+  4824us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1672616320}
I20260709 12:26:13.268406  9730 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:13.268765  5708 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: [6.7-9.8]
I20260709 12:26:13.268850  9736 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:13.270350  5675 connection.cc:256] RPC call timeout handler was delayed by -2.1115751649999766s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22400000000000003s for a response
W20260709 12:26:13.285121  5675 connection.cc:256] RPC call timeout handler was delayed by -1.5905795710000348s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3105s for a response
W20260709 12:26:13.285490  5675 connection.cc:256] RPC call timeout handler was delayed by -1.1128738049997082s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
W20260709 12:26:13.285795  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0057738049999898s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
W20260709 12:26:13.285912  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50851.1us
I20260709 12:26:13.286155  9741 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.286854  9741 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.287300  9742 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:13.258549  9703 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:13.270501  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:13.297672  5632 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:13.292827  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:13.294476  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:13.294771  5561 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:13.299461  5636 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260709 12:26:13.288219  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:13.307165  5554 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:13.182823  9539 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.251657  9577 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:26:13.309533  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.312333  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:13.316586  5561 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:13.313205  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.316998  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.317441  5556 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:13.312789  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:13.310587  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 125582us
I20260709 12:26:13.316002  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.249758  9388 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.317559  5552 raft_consensus.cc:2376] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12. Re-sending same reply.
I20260709 12:26:13.231772  9721 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.330998  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:13.331235  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.331852  5635 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:13.332296  5554 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:13.310545  9713 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:26:13.338603  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51799.4us
W20260709 12:26:13.335930  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 47950.2us
I20260709 12:26:13.341001  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:13.336634  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:13.306759  9619 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.345611  5632 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:13.350549  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.351148  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:13.351663  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1933) took 2605 ms (client timeout 3105 ms). Trace:
I20260709 12:26:13.351917  5557 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:26:13.352064  5629 rpcz_store.cc:269] 0709 12:26:10.733588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.738096 (+  4508us) service_pool.cc:224] Handling call
0709 12:26:13.338708 (+2600612us) spinlock_profiling.cc:243] Waited 420 ms on lock 0x7b54000c2954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:13.338840 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":966027008}
I20260709 12:26:13.316062  9717 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:13.315516  9561 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:13.320938  9716 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.351192  5552 raft_consensus.cc:2376] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10. Re-sending same reply.
W20260709 12:26:13.357846  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 68348us
I20260709 12:26:13.358634  5552 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:26:13.358908  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:13.359910  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:13.360687  5557 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:13.312956  9695 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 12:26:13.365693  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:13.371835  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.322889  9730 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
W20260709 12:26:13.394613  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.296576  9736 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.410377  5632 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:13.413360  5556 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:13.415740  5556 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:26:13.351312  9742 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
W20260709 12:26:13.419855  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 78951.7us
W20260709 12:26:13.421047  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 108619us
I20260709 12:26:13.313872  9726 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.392154  9700 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.393100  9619 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.361701  9717 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.361884  9561 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.435747  5559 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:13.428646  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:13.455899  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:13.457636  5632 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:13.458984  5630 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:13.457931  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:26:13.459767  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 100944us
I20260709 12:26:13.434365  9709 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:13.444566  9700 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.478114  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:13.482345  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:26:13.484197  9736 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.472481  9703 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.485313  9703 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:13.481873  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 184109us
I20260709 12:26:13.370564  9573 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.370545  9716 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.459511  9744 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:13.477691  9734 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.486615  9185 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:13.503422  9695 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:13.519562  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:13.527684  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 205343us
W20260709 12:26:13.527730  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 106870us
W20260709 12:26:13.533175  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.499480  9713 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:13.479658  9718 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.205s
W20260709 12:26:13.536278  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 114540us
I20260709 12:26:13.511332  9696 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:13.511684  9364 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:13.542511  6278 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:26:13.524699  9695 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.540135  9739 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.097s
I20260709 12:26:13.564761  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.551167  9751 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:13.565779  9751 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:13.551971  9750 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:13.566606  9750 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:13.567586  9750 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.569101  9758 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:26:13.570219  9695 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:13.572046  9717 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:13.574928  9093 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9.
I20260709 12:26:13.552721  9748 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:13.576620  9748 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:13.576859  9577 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:13.576974  9748 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:13.577366  9722 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.514s	user 0.008s	sys 0.003s
I20260709 12:26:13.578487  9747 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.555239  9696 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:26:13.527899  9756 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:13.556782  9561 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:13.557350  9721 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:13.560714  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2031) took 1894 ms. Trace:
I20260709 12:26:13.536635  9388 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.537880  9734 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.564306  9364 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:13.614724  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:13.565368  5708 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:13.607865  9741 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:13.616647  5704 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:26:13.617022  9721 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.617486  5556 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:13.617635  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.617942  9364 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:13.617064  5629 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:26:13.618544  5701 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:26:13.618856  9700 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.619241  9754 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:13.618965  9364 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.619776  9754 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.617525  9721 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:13.621246  9704 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.525s	user 0.007s	sys 0.003s
I20260709 12:26:13.621963  9754 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.608143  9716 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.622488  9721 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.608366  5560 rpcz_store.cc:276] 0709 12:26:11.666161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.734716 (+ 68555us) service_pool.cc:224] Handling call
0709 12:26:11.735168 (+   452us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:11.789609 (+ 54441us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:13.560477 (+1770868us) spinlock_profiling.cc:243] Waited 43 ms on lock 0x7b54001bec54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:13.560657 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":98992384}
W20260709 12:26:13.603683  9708 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.747s	user 0.010s	sys 0.000s
I20260709 12:26:13.613442  9760 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:13.623662  9703 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.623900  9760 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.554384  9619 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.613638  5557 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:26:13.614632  9734 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.617738  9388 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.008s
I20260709 12:26:13.577333  9577 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.626237  9577 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:13.628279  5701 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:26:13.628619  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:13.629542  5708 raft_consensus.cc:2368] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9.
I20260709 12:26:13.629593  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.597051  5635 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:13.518013  9744 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.630779  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.623692  9708 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.631561  5708 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:13.632009  9744 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:13.632824  5702 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:13.579238  9747 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.633260  9708 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:13.614063  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:13.578729  9759 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.056s
I20260709 12:26:13.634692  9759 consensus_queue.cc:1243] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:26:13.634843  9747 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.635449  5701 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:26:13.624432  9760 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:13.616932  9741 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:13.539084  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:26:13.637984  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 98987.9us
I20260709 12:26:13.638988  9762 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:26:13.639689  9762 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:13.641609  9762 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:26:13.643968  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:13.646031  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.649919  5631 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:13.641809  9561 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:13.661008  5559 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
W20260709 12:26:13.665753  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 207867us
I20260709 12:26:13.667001  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:13.668680  5599 leader_election.cc:400] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9.
I20260709 12:26:13.669029  5599 leader_election.cc:403] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:26:13.669219  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:13.698143  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.698206  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.700795  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:13.701941  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:13.653013  9756 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:13.662256  9703 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:13.652455  9744 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:13.577960  9722 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:13.577975  9748 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.699470  9734 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:13.705241  9717 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:13.662451  9716 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 12:26:13.705467  9700 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:13.708156  9703 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:13.712457  9702 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.816s	user 0.011s	sys 0.000s
I20260709 12:26:13.715765  9666 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:13.715893  9734 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:13.705420  9713 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.751855  9734 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.722837  9746 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:13.752491  9702 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.753248  9746 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.753711  9746 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:13.754763  9702 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.715997  9744 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.717824  9751 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.725283  9577 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.726052  9757 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:13.758414  9757 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:13.733294  9756 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.759236  9744 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:13.726554  9763 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:13.692211  9754 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.021s
I20260709 12:26:13.721361  9700 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.732327  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.739804  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.733299  5705 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:26:13.718111  9619 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.741223  9703 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:13.776587  9736 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:13.777041  9736 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:13.716693  9631 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.767688  9700 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:13.773885  9751 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:13.774581  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2080) took 1084 ms. Trace:
I20260709 12:26:13.779104  5707 rpcz_store.cc:276] 0709 12:26:12.690055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.690214 (+   159us) service_pool.cc:224] Handling call
0709 12:26:12.690840 (+   626us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:12.714294 (+ 23454us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:26:12.714640 (+   346us) log.cc:844] Serialized 14 byte log entry
0709 12:26:12.825382 (+110742us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:12.825439 (+    57us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:12.825481 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:13.774349 (+948868us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x7b54000e01d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:13.774397 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:13.774511 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1090,"prepare.queue_time_us":1494,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":4991,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":12042,"spinlock_wait_cycles":66508928,"thread_start_us":109179,"threads_started":1,"wal-append.queue_time_us":136017}
I20260709 12:26:13.759312  9757 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.788451  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:13.752113  9666 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 3, Time since last communication: 0.026s
I20260709 12:26:13.762303  9763 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:13.805790  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.807243  6280 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
W20260709 12:26:13.810248  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 141151us
I20260709 12:26:13.810295  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.811831  5560 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:26:13.812150  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 307177us
I20260709 12:26:13.799399  9722 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:13.821421  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.828222  5553 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:13.804728  9715 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:13.829787  9715 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.830224  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.830950  5560 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:13.831794  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:13.831990  9715 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:13.778970  9619 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:13.813573  9381 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.627s	user 0.023s	sys 0.015s
I20260709 12:26:13.841156  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:13.838709  5560 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:13.864815  5631 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:13.811285  9757 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:13.849589  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:13.854012  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 310055us
I20260709 12:26:13.861243  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:13.807410  9702 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.807881  9631 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:13.825929  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:13.865350  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.873091  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:13.841578  9577 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.876873  9702 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.877657  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:13.878641  5560 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20260709 12:26:13.866896  9381 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.678s	user 0.030s	sys 0.016s
I20260709 12:26:13.879545  9702 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.871963  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:13.881165  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:13.881417  9717 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9.
I20260709 12:26:13.882510  9364 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:13.882568  9711 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:13.816823  9736 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:13.890784  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 307101us
I20260709 12:26:13.894369  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:13.895893  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:13.892580  9619 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:13.902155  9702 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:26:13.903093  9768 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:13.903283  9674 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.903627  9768 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.904062  9768 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:13.904557  9692 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:13.905092  9692 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:13.905982  9692 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.832603  9743 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.062s
I20260709 12:26:13.908993  9285 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:13.909425  9285 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:13.890157  9711 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:13.916008  5555 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:26:13.911861  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.913398  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:13.914516  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:13.893901  9713 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:26:13.922344  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.900873  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:13.924499  5558 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:13.924988  5553 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
W20260709 12:26:13.926208  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.926447  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:13.926535  5560 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:26:13.927145  5630 raft_consensus.cc:2376] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8. Re-sending same reply.
I20260709 12:26:13.928951  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.928920  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:13.930970  5629 raft_consensus.cc:2410] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:26:13.931408  5560 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:13.910305  9285 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.933795  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:13.934492  5636 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:13.935037  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:13.936935  5634 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:26:13.939421  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:13.940120  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:13.936218  9748 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:13.921963  9185 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
I20260709 12:26:13.942231  9285 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:13.943492  9746 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.943961  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:13.920650  9695 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.945101  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:13.945746  9746 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.946171  9695 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:13.918371  9364 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:13.949165  5560 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:13.946841  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.953478  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:13.953619  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:13.954209  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:13.954420  5630 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:13.950103  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.956200  5560 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:13.951898  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:13.959611  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:13.962427  9728 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:13.964727  9711 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.969422  9703 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:13.947814  9364 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.972944  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:13.974305  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.970312  9573 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:13.975579  9703 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:13.976867  5634 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:13.996744  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:13.997731  5634 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:13.998719  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:13.999632  5634 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:13.976655  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:14.000481  9767 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.074s
W20260709 12:26:13.980417  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 67182.6us
I20260709 12:26:13.918476  9715 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 12:26:13.911111  9760 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.950587  9736 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:13.987758  9573 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.832162  9763 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:13.989187  9756 raft_consensus.cc:2764] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:26:13.939230  9776 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:13.991732  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:13.976132  9702 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:14.003355  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:14.001325  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:13.988490  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:13.972059  9700 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:14.004663  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:14.011233  5627 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
W20260709 12:26:14.134855  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 131595us
W20260709 12:26:14.113052  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:14.135581  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 131529us
I20260709 12:26:14.136085  5634 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:14.136482  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:14.050096  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 647) took 2974 ms. Trace:
I20260709 12:26:14.137040  5630 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 9: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:14.137696  5703 rpcz_store.cc:276] 0709 12:26:11.075245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.181312 (+106067us) service_pool.cc:224] Handling call
0709 12:26:11.346691 (+165379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:11.373611 (+ 26920us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:12.978010 (+1604399us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b5400278654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:14.049611 (+1071601us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b5400278650. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:14.049899 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":409657984}
I20260709 12:26:14.138614  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:14.099267  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:14.151851  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:14.153621  5632 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:14.004269  9746 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:26:14.166404  5629 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:14.024072  9760 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:14.150064  9604 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:14.167698  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:14.169245  5627 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:14.142011  9776 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:14.143563  9573 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:14.172094  9786 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:14.159718  9619 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:14.172304  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:14.173074  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:14.179664  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2096) took 1438 ms (client timeout 1901 ms). Trace:
I20260709 12:26:14.150323  9715 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:14.181219  5553 raft_consensus.cc:2368] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9.
I20260709 12:26:14.160024  9185 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.183315  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:14.134058  9702 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:14.193617  5702 rpcz_store.cc:269] 0709 12:26:12.741288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.819397 (+ 78109us) service_pool.cc:224] Handling call
0709 12:26:14.179385 (+1359988us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b540006efd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:14.179577 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1843051392}
I20260709 12:26:14.152814  9703 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.196750  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:14.205494  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:14.206512  5634 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:14.207831  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:14.209288  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:14.210829  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:14.211731  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.212255  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:14.202440  9185 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:14.213778  9702 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:14.214668  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 718) took 2751 ms. Trace:
I20260709 12:26:14.215173  9702 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:14.215525  9185 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:14.195559  9669 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.207712  9756 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.215075  5633 rpcz_store.cc:276] 0709 12:26:11.463593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.755466 (+291873us) service_pool.cc:224] Handling call
0709 12:26:11.755720 (+   254us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:11.831443 (+ 75723us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:26:11.889138 (+ 57695us) log.cc:844] Serialized 14 byte log entry
0709 12:26:11.889559 (+   421us) log.cc:844] Serialized 14 byte log entry
0709 12:26:12.336025 (+446466us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b18001c39c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:12.888514 (+552489us) spinlock_profiling.cc:243] Waited 1.34 ms on lock 0x7b18001c39c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ffa228d70 00007f4ffa21c293 00007f4ffa211cea 00007f4ffa21223a 00007f4ffa67ca2a 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:26:12.888789 (+   275us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:12.888836 (+    47us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:12.888882 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:14.214355 (+1325473us) spinlock_profiling.cc:243] Waited 75.2 ms on lock 0x7b5400240754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:14.214422 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:14.214570 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34748,"prepare.queue_time_us":180583,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":114,"raft.queue_time_us":3777,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":13657,"spinlock_wait_cycles":179673984}
I20260709 12:26:14.216550  9669 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:14.216928  9756 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:14.184602  9776 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:14.218947  9669 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:14.219302  9703 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:14.220546  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:14.221145  5627 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:14.222247  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:14.223243  9700 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:14.223619  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:14.224932  9750 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:14.225891  5554 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:26:14.226356  9704 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:14.227099  9728 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:26:14.228039  9777 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:14.231444  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:14.233531  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:14.235710  5627 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:14.201045  9207 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:14.237175  5632 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:14.194989  9715 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:14.224296  5636 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 8.
I20260709 12:26:14.247260  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:14.248018  5634 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:14.241302  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:14.248323  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:14.250304  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:14.265470  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:14.266089  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:14.229202  9721 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:26:14.270943  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.277892  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:14.237979  9703 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:14.238445  9777 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:14.237797  9756 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:14.322552  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 51763.2us
I20260709 12:26:14.238956  9364 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.113s
I20260709 12:26:14.324235  5627 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:14.435917  5627 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260709 12:26:14.437105  9789 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.185s	user 0.000s	sys 0.000s
I20260709 12:26:14.437232  5627 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:14.438397  5627 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:14.439729  5627 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20260709 12:26:14.317812  9743 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:14.354430  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:14.315161  9692 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:14.440981  9692 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:14.441064  5555 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:26:14.441913  9692 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:14.445196  9787 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.232s	user 0.000s	sys 0.000s
W20260709 12:26:14.429283  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:14.429600  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.430393  5632 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 12:26:14.447099  5632 pending_rounds.cc:85] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 4
I20260709 12:26:14.447350  5632 pending_rounds.cc:107] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:26:14.447566  5632 raft_consensus.cc:2889] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:14.324085  9782 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.241s	user 0.001s	sys 0.000s
I20260709 12:26:14.277657  9185 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:14.351737  9669 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:14.453894  9795 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.165s	user 0.000s	sys 0.000s
I20260709 12:26:14.453677  9669 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:14.251109  9715 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:14.250487  9688 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.183s
W20260709 12:26:14.433590  9785 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
I20260709 12:26:14.433699  9777 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:14.456332  9669 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:14.463055  9777 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:14.240485  9728 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:14.467305  9728 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:14.468511  9728 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:14.452811  9185 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:14.512725  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.277920  9631 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:14.277981  9760 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.208s
I20260709 12:26:14.323287  9695 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.177s
I20260709 12:26:14.524951  5632 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:14.470675  9207 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.462529  9688 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
W20260709 12:26:14.532819  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.278075  9757 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.246s
W20260709 12:26:14.531230  9736 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.522s	user 0.013s	sys 0.000s
I20260709 12:26:14.539710  9207 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:14.546875  9207 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:14.513141  9715 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:14.551800  9715 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:14.538122  9768 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.634s	user 0.012s	sys 0.002s
W20260709 12:26:14.559177  9285 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.601s	user 0.013s	sys 0.000s
I20260709 12:26:14.576332  5630 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
W20260709 12:26:14.582742  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 139133us
I20260709 12:26:14.571257  9768 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:14.586139  9743 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:14.594430  9743 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:14.595767  9631 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:14.596230  9631 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:14.597129  9631 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:14.598657  9734 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:14.602355  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:14.603132  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:14.603514  9726 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:14.603746  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:14.604854  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:14.610113  5556 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:14.610731  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:14.611126  5703 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:26:14.611488  5706 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:14.598642  9631 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:14.614800  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:14.626432  6271 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:14.629282  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:14.630021  6274 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
W20260709 12:26:14.632714  9797 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:26:14.614780  9777 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.633657  9777 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:14.639159  9768 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:14.645504  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:14.586378  9185 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:14.636224  9777 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:14.661741  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:14.663035  6271 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:14.669548  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.670449  6274 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:14.671198  5554 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 12:26:14.675127  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.690464  9573 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:14.694137  9573 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:14.659118  9759 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.038s
I20260709 12:26:14.698156  5552 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:26:14.699913  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:14.658103  9735 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.700735  6271 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:14.700968  9735 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:14.703953  9735 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:14.711107  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:14.714219  5561 raft_consensus.cc:2410] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:26:14.666287  9800 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.724203  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:14.731684  9760 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:14.735719  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:14.736510  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:14.737380  6276 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:14.739686  6275 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 7. (term mismatch)
I20260709 12:26:14.739705  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:14.740995  6276 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 9: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:14.740327  6275 pending_rounds.cc:85] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 6
I20260709 12:26:14.742005  6275 pending_rounds.cc:107] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260709 12:26:14.742357  6275 raft_consensus.cc:2889] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:14.724679  9573 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 12:26:14.720679  9776 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:14.754022  6278 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 13: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:14.755249  9776 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:14.722249  9800 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:14.758504  9800 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:14.759125  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:14.766319  9737 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.330s	user 0.004s	sys 0.000s
I20260709 12:26:14.771687  5552 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:26:14.775147  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2039) took 1483 ms. Trace:
I20260709 12:26:14.775684  5559 rpcz_store.cc:276] 0709 12:26:13.291824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.292900 (+  1076us) service_pool.cc:224] Handling call
0709 12:26:14.774936 (+1482036us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b540023f354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:14.775075 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":264303488}
I20260709 12:26:14.778744  9674 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:14.779234  9674 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:14.779990  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 721) took 1916 ms. Trace:
I20260709 12:26:14.780385  5628 rpcz_store.cc:276] 0709 12:26:12.863758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.884915 (+ 21157us) service_pool.cc:224] Handling call
0709 12:26:12.885301 (+   386us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:12.977710 (+ 92409us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:26:13.068252 (+ 90542us) log.cc:844] Serialized 14 byte log entry
0709 12:26:13.281089 (+212837us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:13.294080 (+ 12991us) log.cc:844] Serialized 52 byte log entry
0709 12:26:13.294238 (+   158us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:13.294454 (+   216us) log.cc:844] Serialized 14 byte log entry
0709 12:26:13.370749 (+ 76295us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:13.370844 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:14.765659 (+1394815us) spinlock_profiling.cc:243] Waited 5.02 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:14.779744 (+ 14085us) raft_consensus.cc:1685] finished
0709 12:26:14.779769 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:14.779934 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12590,"prepare.queue_time_us":122539,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":66,"raft.queue_time_us":99375,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":9614,"spinlock_wait_cycles":11554048,"thread_start_us":264732,"threads_started":3,"wal-append.queue_time_us":272659}
W20260709 12:26:14.783788  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 118730us
I20260709 12:26:14.789628  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:14.801898  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:14.806499  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.806227  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:26:14.810376  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 137881us
I20260709 12:26:14.791150  9777 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 12:26:14.803968  9702 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:14.823668  9703 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:14.838356  9688 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.803048  9748 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 12:26:14.803114  9759 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.034s
I20260709 12:26:14.822947  9698 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 12:26:14.842972  9697 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:14.842867  6274 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:14.801759  9802 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:14.809755  9185 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 12:26:14.810031  9726 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:14.854135  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:14.858397  9674 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:14.862315  9688 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:14.869585  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:14.884701  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:14.851495  9702 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:14.893160  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:14.891584  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:14.893842  5709 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:14.852902  9703 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:14.864619  9777 consensus_queue.cc:1243] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:26:14.898469  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:14.895961  9702 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:14.893508  9822 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:26:14.896454  9703 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:14.899672  9726 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:14.911610  9674 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:14.944844  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.948222  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:14.949235  5558 raft_consensus.cc:2376] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9. Re-sending same reply.
I20260709 12:26:14.932168  9822 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:26:14.978920  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:14.980988  9815 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
W20260709 12:26:14.983099  9806 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 12:26:14.985399  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 1986) took 1180 ms. Trace:
I20260709 12:26:14.985800  5561 rpcz_store.cc:276] 0709 12:26:13.805289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.805457 (+   168us) service_pool.cc:224] Handling call
0709 12:26:14.985204 (+1179747us) spinlock_profiling.cc:243] Waited 907 ms on lock 0x7b5400082354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:14.985338 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2085157248}
I20260709 12:26:14.986891  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.002554  5636 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 14 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:26:15.002219  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:15.004707  9715 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.006263  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 732) took 1047 ms. Trace:
I20260709 12:26:15.007210  5560 raft_consensus.cc:2410] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:26:15.005280  9715 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:15.006753  5634 rpcz_store.cc:276] 0709 12:26:13.959058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:14.248744 (+289686us) service_pool.cc:224] Handling call
0709 12:26:14.314830 (+ 66086us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:14.322932 (+  8102us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:26:14.450307 (+127375us) log.cc:844] Serialized 14 byte log entry
0709 12:26:14.667702 (+217395us) log.cc:844] Serialized 14 byte log entry
0709 12:26:14.668297 (+   595us) log.cc:844] Serialized 14 byte log entry
0709 12:26:14.668543 (+   246us) log.cc:844] Serialized 14 byte log entry
0709 12:26:14.668731 (+   188us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:14.706338 (+ 37607us) log.cc:844] Serialized 26 byte log entry
0709 12:26:14.706481 (+   143us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:14.706515 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:14.706585 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:14.984537 (+277952us) spinlock_profiling.cc:243] Waited 5.72 ms on lock 0x7b18003975a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:14.992900 (+  8363us) spinlock_profiling.cc:243] Waited 2.62 ms on lock 0x7b18003975a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:15.006052 (+ 13152us) raft_consensus.cc:1685] finished
0709 12:26:15.006075 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:15.006207 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7357,"prepare.queue_time_us":114077,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":108,"raft.queue_time_us":8044,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":23775,"spinlock_wait_cycles":19191552,"thread_start_us":133776,"threads_started":1,"wal-append.queue_time_us":198686}
I20260709 12:26:15.012145  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 800) took 1846 ms. Trace:
I20260709 12:26:15.012648  6280 rpcz_store.cc:276] 0709 12:26:13.165535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.165984 (+   449us) service_pool.cc:224] Handling call
0709 12:26:13.189882 (+ 23898us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:13.807655 (+617773us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:15.011944 (+1204289us) spinlock_profiling.cc:243] Waited 617 ms on lock 0x7b5400591ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:15.012094 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1419334016}
I20260709 12:26:14.955385  9726 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:15.029486  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:15.070578  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:15.072453  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 188414us
I20260709 12:26:15.057574  9631 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:15.058128  9688 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:15.096374  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 82004.5us
W20260709 12:26:15.096657  9808 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
I20260709 12:26:15.108392  9285 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:26:15.099742  9619 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.015s
I20260709 12:26:15.119459  9715 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:15.120273  9734 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:15.120729  9734 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:15.121749  9734 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:15.123131  9734 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:15.122577  9750 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.123271  9802 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:15.123937  9750 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:15.107795  9703 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:15.126004  9750 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:15.127931  9703 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:15.141603  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.156742  9703 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:15.157076  5554 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:15.157377  9703 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:15.159497  9703 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:15.160140  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.161363  9703 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:15.162088  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.161907  9703 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:15.170512  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.163412  9619 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.181607  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:15.184444  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.164215  9702 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:15.186311  9285 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:15.197223  9285 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:15.200026  9285 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:15.186658  5558 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:15.194996  5559 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:15.205220  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2047) took 1905 ms. Trace:
I20260709 12:26:15.207062  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.176240  9703 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:15.196038  9631 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:15.205585  5557 rpcz_store.cc:276] 0709 12:26:13.299325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.371377 (+ 72052us) service_pool.cc:224] Handling call
0709 12:26:13.401288 (+ 29911us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:13.624831 (+223543us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:15.204961 (+1580130us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b54003a98d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:15.205159 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":487642880}
W20260709 12:26:15.217152  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:15.218811  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.223595  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:15.224480  5558 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:15.216521  9694 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:15.194669  9728 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:15.234607  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:15.194686  9702 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:15.250605  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.251628  5561 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:26:15.253273  5524 leader_election.cc:336] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.558s (SENT)
I20260709 12:26:15.253386  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 683) took 1344 ms. Trace:
I20260709 12:26:15.253736  5560 rpcz_store.cc:276] 0709 12:26:13.908755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.974024 (+ 65269us) service_pool.cc:224] Handling call
0709 12:26:15.253208 (+1279184us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54001af254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:15.253331 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2370370944}
I20260709 12:26:15.255004  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.255807  5560 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:26:15.256610  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.257856  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.258742  5557 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
W20260709 12:26:15.259054  5524 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.618s (SENT)
I20260709 12:26:15.257898  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:15.260237  5524 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.630s (SENT)
I20260709 12:26:15.260689  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:15.212343  9631 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:15.261005  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 685) took 1351 ms. Trace:
I20260709 12:26:15.261168  9801 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:15.261379  5553 rpcz_store.cc:276] 0709 12:26:13.909569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.961646 (+ 52077us) service_pool.cc:224] Handling call
0709 12:26:15.260835 (+1299189us) spinlock_profiling.cc:243] Waited 35.6 ms on lock 0x7b54003a93d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:15.260953 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":81955200}
I20260709 12:26:15.261803  9631 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:15.262631  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.263221  5553 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:15.218573  9285 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.265636  9285 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:15.268181  9285 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:15.269985  9800 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:15.270416  9800 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:26:15.274104  5556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2052) took 1369 ms (client timeout 1647 ms). Trace:
W20260709 12:26:15.274518  5556 rpcz_store.cc:269] 0709 12:26:13.904712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.910879 (+  6167us) service_pool.cc:224] Handling call
0709 12:26:15.273915 (+1363036us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b540016ba54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:15.274048 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1590406016}
I20260709 12:26:15.280723  6237 leader_election.cc:400] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 8. Current term is 9.
W20260709 12:26:15.282613  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.284559  9388 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:15.196266  9619 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:15.288323  9695 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.289284  9619 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:15.289971  9751 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.291011  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 692) took 1349 ms. Trace:
I20260709 12:26:15.290778  9751 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:15.291429  5555 rpcz_store.cc:276] 0709 12:26:13.941145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:14.441860 (+500715us) service_pool.cc:224] Handling call
0709 12:26:14.442139 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:14.544896 (+102757us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:26:14.545205 (+   309us) log.cc:844] Serialized 14 byte log entry
0709 12:26:14.710531 (+165326us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:14.710580 (+    49us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:14.710629 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:15.290798 (+580169us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540041a0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:15.290838 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:15.290956 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3270,"prepare.queue_time_us":557,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":1956,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":4822,"spinlock_wait_cycles":235502976,"thread_start_us":164911,"threads_started":1,"wal-append.queue_time_us":258257}
I20260709 12:26:15.294000  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:15.294994  5704 raft_consensus.cc:2410] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:26:15.296636  5555 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:15.297870  5555 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:15.297989  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2237) took 1134 ms. Trace:
W20260709 12:26:15.298381  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.298429  5636 rpcz_store.cc:276] 0709 12:26:14.163712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:14.269130 (+105418us) service_pool.cc:224] Handling call
0709 12:26:15.297804 (+1028674us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x7b540032d7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:15.297927 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1288727424}
I20260709 12:26:15.298995  5555 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:26:15.299472  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:15.214768  9763 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:15.234093  9728 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:15.250130  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.231613  9829 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.250717  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:26:15.315732  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 65105us
I20260709 12:26:15.316740  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:15.317202  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:15.259528  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.233754  9694 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:15.288949  9695 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:15.301222  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.228971  9703 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:26:15.247759  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.283949  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:15.341073  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:15.341126  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57274.9us
I20260709 12:26:15.302693  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:15.305682  9736 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:15.306442  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.308758  9758 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.309736  5708 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:15.345305  5702 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:15.345577  5708 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:15.271703  9800 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:15.317823  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:15.326479  5558 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:15.337000  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:15.338804  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:15.347429  6275 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:15.354348  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:15.356151  5709 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:26:15.358096  5557 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:15.353025  9801 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:15.359323  9801 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:15.359028  5708 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:15.359747  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:15.360078  5708 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:15.361013  6271 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:15.362160  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:15.362808  6279 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:15.363739  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:15.364094  6280 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:15.359781  9801 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:15.362155  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.367728  5701 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:15.365976  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:15.370617  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1967) took 1420 ms (client timeout 1708 ms). Trace:
I20260709 12:26:15.370908  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:15.371086  5629 rpcz_store.cc:269] 0709 12:26:13.950532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.951460 (+   928us) service_pool.cc:224] Handling call
0709 12:26:15.370439 (+1418979us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b54001de7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:15.370566 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":486152064}
I20260709 12:26:15.371805  5705 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:15.372748  5627 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: []
W20260709 12:26:15.373180  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55422.6us
I20260709 12:26:15.373498  5552 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: [8.7-10.8]
I20260709 12:26:15.373884  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:15.374066  5708 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:26:15.375135  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:15.375253  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:15.375767  5708 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:15.365289  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:15.381518  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:15.382282  6275 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:15.370568  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:15.337415  9822 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:15.337656  9768 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:15.337004  9619 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:26:15.450730  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 269901us
I20260709 12:26:15.454677  5704 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:15.451727  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.355521  9780 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:15.355377  9702 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.459074  5709 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:26:15.471693  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:15.366052  9751 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:15.460053  5701 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:26:15.461380  5554 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:26:15.430826  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:15.388001  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:15.365357  9694 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:15.376541  9698 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:15.492040  5630 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:15.474756  5561 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:15.493479  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:15.485985  5635 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 11: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:15.490228  5705 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:15.370383  9573 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.424760  9644 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:15.502933  5632 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:26:15.419343  9715 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:15.494467  5628 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:15.420810  9800 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:15.514873  5630 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:26:15.419138  9539 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:15.381995  9285 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:26:15.501194  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.499634  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.356453  9758 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:15.347327  9829 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:15.495941  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:15.497439  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 1.84627e+06us
I20260709 12:26:15.354018  9703 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:15.509893  5708 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:15.497705  9822 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:15.542920  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.515774  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:26:15.492502  9836 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:26:15.518039  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:15.519057  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.419435  9207 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.080s
W20260709 12:26:15.523154  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 67993.8us
I20260709 12:26:15.551075  5709 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:15.513163  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.474021  9734 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
I20260709 12:26:15.547389  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:15.548629  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 160727us
I20260709 12:26:15.560075  5630 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:15.523445  9539 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:15.451086  9839 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.075s
I20260709 12:26:15.553793  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:15.511631  9715 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:15.566658  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:15.547390  9822 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:15.517124  9573 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:15.564927  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 197336us
I20260709 12:26:15.533133  9698 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:15.560418  5704 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:15.581208  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:15.573503  9839 consensus_queue.cc:1243] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:26:15.582757  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:15.586374  9844 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:26:15.587201  9844 consensus_queue.cc:1243] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:26:15.587802  9759 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:15.588183  9759 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:15.589061  9759 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:15.593546  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.571545  9539 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:15.523703  9285 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:15.493405  9702 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:15.571975  5631 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 11 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:26:15.566546  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2232) took 1428 ms. Trace:
I20260709 12:26:15.613425  5627 rpcz_store.cc:276] 0709 12:26:14.138039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:14.442296 (+304257us) service_pool.cc:224] Handling call
0709 12:26:15.372587 (+930291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:15.373148 (+   561us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:15.566366 (+193218us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b54001de7d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:15.566493 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2138518912}
I20260709 12:26:15.483144  9780 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:15.578323  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:15.615041  5633 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:15.615914  5709 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:15.623291  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.555001  9619 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:15.600020  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.574550  9715 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:15.649070  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.636971  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:15.602064  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:15.651242  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.570945  5628 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
W20260709 12:26:15.605333  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.578200  9822 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:15.577297  5630 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:26:15.661967  5630 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:15.663553  5629 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:15.631234  5633 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
W20260709 12:26:15.638815  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.650110  5627 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:15.670420  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:15.612689  9285 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:15.679301  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.678879  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.666982  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:15.688696  9846 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:26:15.689548  9840 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:26:15.689962  9666 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:26:15.691023  9841 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:26:15.692310  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:15.695324  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:15.697992  5707 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:15.698918  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.700764  9666 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:15.701226  9666 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:15.701582  9666 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:15.700023  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.705814  5707 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 7. Current term is 8. Ops: []
I20260709 12:26:15.707864  9802 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:26:15.711760  9698 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:15.713912  5707 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:15.715523  5707 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:15.716753  9573 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:15.719561  5707 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:15.728132  5707 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: [9.9-9.9]
W20260709 12:26:15.730618  9805 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:26:15.731779  5707 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:26:15.739787  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 668) took 3062 ms. Trace:
I20260709 12:26:15.740412  5710 rpcz_store.cc:276] 0709 12:26:12.677355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.677480 (+   125us) service_pool.cc:224] Handling call
0709 12:26:12.693958 (+ 16478us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:12.710967 (+ 17009us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:12.711002 (+    35us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:26:12.714476 (+  3474us) log.cc:844] Serialized 78 byte log entry
0709 12:26:12.865356 (+150880us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:12.902382 (+ 37026us) log.cc:844] Serialized 14 byte log entry
0709 12:26:14.175001 (+1272619us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b18000a2dc0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:26:14.175261 (+   260us) log.cc:844] Serialized 14 byte log entry
0709 12:26:14.175503 (+   242us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:14.175582 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:15.109030 (+933448us) spinlock_profiling.cc:243] Waited 71 ms on lock 0x7b18000a2dc0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1
0709 12:26:15.634233 (+525203us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b18000a2dc0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:15.739240 (+105007us) raft_consensus.cc:1685] finished
0709 12:26:15.739272 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:15.739659 (+   387us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1197064,"prepare.queue_time_us":1311469,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":6156,"raft.queue_time_us":1543,"raft.run_cpu_time_us":386,"raft.run_wall_time_us":62645,"spinlock_wait_cycles":174828672,"thread_start_us":150500,"threads_started":1,"wal-append.queue_time_us":249290}
I20260709 12:26:15.743081  9666 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:15.743811  9829 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:15.745682  9829 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.746261  9829 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:15.754338  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.765245  5628 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:26:15.676105  5702 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:15.633265  9619 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:15.782648  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:15.785683  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.802582  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.808903  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:15.674470  9695 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:15.801447  9619 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:15.833117  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:15.834149  5709 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:15.834189  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.848562  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.823369  9743 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
W20260709 12:26:15.874440  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:15.875135  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:15.876163  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:15.876634  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:15.878464  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 324126us
I20260709 12:26:15.848019  9855 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.886137  5705 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:15.840952  9839 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:15.847282  9829 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:15.871243  9703 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:26:15.905019  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:15.908679  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:15.872035  9093 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 12:26:15.913893  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:15.915210  9702 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:15.922542  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:15.899937  9758 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:15.936472  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2067) took 1282 ms. Trace:
I20260709 12:26:15.965703  5552 rpcz_store.cc:276] 0709 12:26:14.654295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:14.988213 (+333918us) service_pool.cc:224] Handling call
0709 12:26:15.369453 (+381240us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:15.452505 (+ 83052us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:15.936168 (+483663us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b54000ea6d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:15.936382 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":875771136}
I20260709 12:26:15.942921  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:15.953027  5708 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:15.956979  5705 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:26:15.951313  9207 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:15.978051  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55627us
I20260709 12:26:15.986370  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:15.950237  9666 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:15.951211  5709 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:15.952319  9860 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:15.991710  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48930.4us
I20260709 12:26:16.000689  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:15.984113  9780 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:15.984966  9758 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:15.953954  9839 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:16.016737  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 200357us
I20260709 12:26:15.990969  9695 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:26:15.997735  9703 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:16.030297  9759 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:16.015705  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:15.953612  9855 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.002882  9666 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.032618  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:16.019534  9698 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260709 12:26:16.016637  9822 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:16.045080  9801 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.679s	user 0.013s	sys 0.002s
I20260709 12:26:16.045645  9801 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.046427  9822 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:16.046929  6276 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:15.990994  9207 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.051791  9743 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:16.052762  9743 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:16.053025  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:26:16.033097  9842 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 12:26:16.002556  9860 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:16.054267  9865 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:16.054157  9743 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.055822  9865 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:26:16.058420  9865 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:16.058254  9666 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.059715  9865 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:16.060282  9865 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:16.061266  9865 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:16.063328  9666 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:16.056474  9860 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:16.009177  9858 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:16.066032  9839 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:15.993083  9748 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260709 12:26:16.067157  9801 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:16.068517  9207 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:16.068739  9728 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.070128  9728 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:16.058254  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.066794  9285 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.077252  5632 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:16.077625  9768 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:16.078598  9768 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:16.079797  9851 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
W20260709 12:26:16.080873  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 650125us
I20260709 12:26:16.082468  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:16.082751  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.083571  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:16.083264  9207 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:16.084167  9207 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:16.084573  9207 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:16.085937  9734 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:26:16.086822  9734 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:26:16.084535  9644 raft_consensus.cc:2764] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:26:16.090833  6237 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:16.093923  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:16.095531  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.052836  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:16.088295  9734 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:26:16.104285  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51541.4us
I20260709 12:26:16.079254  9285 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:16.105173  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:16.106428  5634 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:16.107403  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:16.108119  5628 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 11: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:16.108383  9864 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:16.108897  9864 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:16.110644  9285 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.111737  9864 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:16.091493  9777 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:16.116833  5555 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:26:16.118314  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:16.096493  9093 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:16.097203  9703 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:16.141819  5634 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:26:16.112087  9285 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:16.144733  5555 pending_rounds.cc:85] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 10
I20260709 12:26:16.153510  5555 pending_rounds.cc:107] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 12:26:16.153898  5555 raft_consensus.cc:2889] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:16.103233  9829 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.113045  9734 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:16.098301  9801 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 12:26:16.162364  5701 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
W20260709 12:26:16.166034  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 113079us
I20260709 12:26:16.117019  9777 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:16.116701  9862 raft_consensus.cc:2764] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:26:16.137133  9759 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:16.167645  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:26:16.183681  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 65474us
I20260709 12:26:16.186784  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:16.140586  9695 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:16.187129  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:16.158916  9854 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:26:16.146706  9703 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:16.196722  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:16.137490  9644 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.057s
I20260709 12:26:16.201407  9703 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:16.132651  9539 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:16.140276  9748 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.056s
W20260709 12:26:16.238960  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 148195us
W20260709 12:26:16.240936  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54263.7us
W20260709 12:26:16.219291  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 142196us
I20260709 12:26:16.164412  9829 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:16.240247  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:16.208281  9619 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.267599  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:16.268383  9726 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:26:16.269317  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.202504  9822 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:16.200300  9858 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:16.219528  9698 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:26:16.287869  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 91361.1us
I20260709 12:26:16.214382  9866 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.075s
I20260709 12:26:16.290099  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:16.269835  9702 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.272831  9726 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:16.296942  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55142.4us
I20260709 12:26:16.300704  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:16.317677  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.290738  9860 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:16.300187  9726 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:16.344470  9539 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:16.345180  9539 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.345793  9539 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:16.347528  9715 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.665s	user 0.010s	sys 0.001s
I20260709 12:26:16.348811  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2072) took 1056 ms. Trace:
I20260709 12:26:16.349269  5558 rpcz_store.cc:276] 0709 12:26:15.292437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.569425 (+276988us) service_pool.cc:224] Handling call
0709 12:26:15.583794 (+ 14369us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:16.088595 (+504801us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:16.088621 (+    26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:16.089247 (+   626us) log.cc:844] Serialized 52 byte log entry
0709 12:26:16.258065 (+168818us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:26:16.258109 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.258187 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:16.338064 (+ 79877us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b480027e958. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:26:16.348442 (+ 10378us) raft_consensus.cc:1685] finished
0709 12:26:16.348470 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:16.348738 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":504510,"raft.queue_time_us":509320,"raft.run_cpu_time_us":2871,"raft.run_wall_time_us":241755,"spinlock_wait_cycles":6546816,"thread_start_us":120208,"threads_started":1,"wal-append.queue_time_us":242366}
I20260709 12:26:16.299679  9822 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:16.351644  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.351929  9822 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:16.356619  9802 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:26:16.357805  9802 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:16.358183  9802 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:16.295971  9881 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:16.359058  9802 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.266216  9777 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:16.334509  9870 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:26:16.362205  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2015) took 1133 ms. Trace:
W20260709 12:26:16.363018  9763 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.062s	user 0.003s	sys 0.008s
I20260709 12:26:16.363755  9573 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.364477  9573 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.365679  9839 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:16.366227  9839 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:16.338547  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.334942  9829 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:16.340659  9855 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:16.339555  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 709) took 1047 ms. Trace:
I20260709 12:26:16.367367  9839 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.367563  5554 rpcz_store.cc:276] 0709 12:26:15.291871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.480894 (+189023us) service_pool.cc:224] Handling call
0709 12:26:15.570230 (+ 89336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:15.570479 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:15.570518 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:15.570587 (+    69us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:15.570645 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:15.570724 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:16.339374 (+768650us) spinlock_profiling.cc:243] Waited 62.4 ms on lock 0x7b54001c57d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:16.339494 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":143623552}
I20260709 12:26:16.369143  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.370824  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:16.300586  9702 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.371724  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:16.372701  5560 raft_consensus.cc:2376] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11. Re-sending same reply.
I20260709 12:26:16.373929  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.374238  9702 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:16.375168  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.333263  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:16.366170  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.379118  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.366335  9777 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:16.245385  9734 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:16.379925  9692 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:16.380431  9734 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:16.380813  9692 consensus_queue.cc:1243] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:26:16.381589  9698 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:16.381994  9698 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:16.382997  9698 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.370739  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2073) took 1078 ms. Trace:
I20260709 12:26:16.383992  5559 rpcz_store.cc:276] 0709 12:26:15.292757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.936154 (+643397us) service_pool.cc:224] Handling call
0709 12:26:15.936606 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:16.096984 (+160378us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:16.097026 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:16.148564 (+ 51538us) log.cc:844] Serialized 26 byte log entry
0709 12:26:16.291821 (+143257us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:26:16.291863 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.291947 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:16.370503 (+ 78556us) raft_consensus.cc:1685] finished
0709 12:26:16.370524 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:16.370669 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":165290,"raft.queue_time_us":160394,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":2566,"thread_start_us":93557,"threads_started":1,"wal-append.queue_time_us":204377}
I20260709 12:26:16.385874  9093 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:16.386037  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.387409  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.375661  9856 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.388839  9856 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.376080  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.390419  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.390669  5558 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
W20260709 12:26:16.390936  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 90334.2us
I20260709 12:26:16.391311  5559 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:16.391616  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.391772  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:16.360509  9619 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:16.341311  9285 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:16.379199  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:16.362558  5556 rpcz_store.cc:276] 0709 12:26:15.229363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.302347 (+ 72984us) service_pool.cc:224] Handling call
0709 12:26:15.349738 (+ 47391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:15.356495 (+  6757us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:26:15.899933 (+543438us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b1800279c40. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:26:15.907717 (+  7784us) log.cc:844] Serialized 14 byte log entry
0709 12:26:16.117813 (+210096us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:16.118321 (+   508us) log.cc:844] Serialized 26 byte log entry
0709 12:26:16.118490 (+   169us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:26:16.118526 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.118597 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:16.361913 (+243316us) raft_consensus.cc:1685] finished
0709 12:26:16.362008 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:16.362133 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26997,"prepare.queue_time_us":557862,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":37736,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":27691,"spinlock_wait_cycles":27386624,"thread_start_us":180091,"threads_started":1,"wal-append.queue_time_us":310125}
W20260709 12:26:16.357052  9874 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 12:26:16.376696  9800 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.850s	user 0.014s	sys 0.000s
I20260709 12:26:16.376910  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:16.392493  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 129529us
W20260709 12:26:16.394302  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 97708.2us
W20260709 12:26:16.470744  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:16.477074  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 105476us
W20260709 12:26:16.435154  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.478008  5560 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:26:16.424518  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2071) took 1132 ms. Trace:
W20260709 12:26:16.480675  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 102545us
I20260709 12:26:16.397519  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:16.441457  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:16.482812  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:16.493916  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 96481.4us
I20260709 12:26:16.494199  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.483317  5561 rpcz_store.cc:276] 0709 12:26:15.292135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.569197 (+277062us) service_pool.cc:224] Handling call
0709 12:26:15.569521 (+   324us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:16.088561 (+519040us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:26:16.088825 (+   264us) log.cc:844] Serialized 14 byte log entry
0709 12:26:16.245035 (+156210us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:16.375232 (+130197us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b2c000fb608. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:26:16.375669 (+   437us) log.cc:844] Serialized 26 byte log entry
0709 12:26:16.375846 (+   177us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:26:16.375883 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.375948 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:16.392195 (+ 16247us) raft_consensus.cc:1685] finished
0709 12:26:16.424196 (+ 32001us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:16.424434 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":520908,"prepare.queue_time_us":1035,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":1001,"raft.queue_time_us":522038,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":12059,"spinlock_wait_cycles":28224384,"thread_start_us":155292,"threads_started":1,"wal-append.queue_time_us":249431}
I20260709 12:26:16.348160  9539 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:16.417706  9702 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.080s
I20260709 12:26:16.360276  9802 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:16.470017  9695 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 12:26:16.498359  9697 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:16.499289  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 710) took 1184 ms. Trace:
I20260709 12:26:16.499689  5555 rpcz_store.cc:276] 0709 12:26:15.314978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.144100 (+829122us) service_pool.cc:224] Handling call
0709 12:26:16.144495 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:16.163199 (+ 18704us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:16.163244 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:16.167632 (+  4388us) log.cc:844] Serialized 26 byte log entry
0709 12:26:16.287107 (+119475us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:16.287153 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.287241 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:16.499055 (+211814us) raft_consensus.cc:1685] finished
0709 12:26:16.499080 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:16.499227 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":119158,"threads_started":1,"wal-append.queue_time_us":311888}
I20260709 12:26:16.501488  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:16.502285  5555 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 9: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:16.508852  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.509816  5556 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:16.510946  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.513056  9856 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:16.435532  9862 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:26:16.513696  5556 pending_rounds.cc:85] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 3
I20260709 12:26:16.514050  5556 pending_rounds.cc:107] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260709 12:26:16.435698  9751 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:16.514365  5556 raft_consensus.cc:2889] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:16.516394  5557 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:16.348088  9715 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.518589  9715 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:16.520541  5553 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:16.521410  5557 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:26:16.521739  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.522882  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:16.524324  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.529873  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:16.530874  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:16.541759  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:16.384479  9698 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:16.543403  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:16.544130  9285 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:26:16.552474  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:16.558881  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.555085  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:16.560029  5632 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:16.555879  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:16.557366  5633 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:26:16.554276  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:16.544308  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:16.579347  5560 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:16.581379  5635 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:16.581905  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:16.521468  9750 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:16.537729  9093 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:16.356194  9207 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:16.526319  9697 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:16.542838  9865 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:26:16.598403  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54200.5us
W20260709 12:26:16.599318  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 309398us
I20260709 12:26:16.578101  9388 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:16.465934  9703 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.578269  9702 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:16.601325  9800 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.581938  9285 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:16.591192  9715 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.550544  9185 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 12:26:16.580828  9879 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:16.628021  5635 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:16.595824  9093 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:15.601671  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:16.608299  9858 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:16.582589  9728 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 12:26:16.650765  5636 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:16.656060  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:16.656766  5703 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:16.610452  9388 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:16.658048  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.619907  9702 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:16.604303  9856 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 12:26:16.630859  9285 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:16.674652  5559 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:16.658933  5557 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:16.597944  9750 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:16.585705  9881 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20260709 12:26:16.637356  9093 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:16.610136  9751 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 12:26:16.673992  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:16.652751  9858 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:16.689159  5709 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 12 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:26:16.689724  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:16.690562  5706 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 11: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:16.636056  9800 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.665364  9388 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:16.642452  9715 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:16.716019  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.687218  9763 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.687809  9801 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.692961  9539 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:16.725875  5555 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260709 12:26:16.727077  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.689936  9858 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:16.744535  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 220568us
W20260709 12:26:16.774600  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 279111us
I20260709 12:26:16.708756  9726 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:16.711216  9768 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 12:26:16.728787  9539 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:16.787014  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 199670us
I20260709 12:26:16.792702  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:16.695984  9697 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:16.817530  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 3.291s (SENT) [suppressed 29 similar messages]
I20260709 12:26:16.728987  9763 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.731885  9801 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.798668  9539 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:16.717905  9702 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:16.857529  9899 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.858136  9899 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:16.810555  9758 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 12:26:16.837464  9207 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:16.812865  9856 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.051s
W20260709 12:26:16.893814  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 105793us
I20260709 12:26:16.896637  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:16.906863  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:16.908627  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.835703  9743 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:26:16.854426  9703 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:16.949910  5634 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.001s	sys 0.009s
W20260709 12:26:16.950574  5634 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.001s	sys 0.009s
I20260709 12:26:16.927716  9879 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:16.959610  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 83837.4us
I20260709 12:26:16.872800  9895 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.933177  9901 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260709 12:26:16.955139  9743 consensus_queue.cc:1243] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:26:16.983515  6275 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 12:26:16.989878  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.996143  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:17.036612  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:17.222234  9784 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.029933  9750 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:17.366602  9784 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:17.060652  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:17.100378  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:17.369141  9763 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:17.055931  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.060307  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:26:17.371212  9883 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.266s	user 0.000s	sys 0.001s
W20260709 12:26:17.371227  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.372453  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 748) took 1269 ms. Trace:
I20260709 12:26:17.372881  5636 rpcz_store.cc:276] 0709 12:26:16.104608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.104748 (+   140us) service_pool.cc:224] Handling call
0709 12:26:16.143211 (+ 38463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:16.657476 (+514265us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.372152 (+714676us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b5400240754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:17.372377 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1145851776}
W20260709 12:26:17.374166  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.046693  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:17.375504  6278 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:17.209267  9844 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:26:17.380252  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 780902us
I20260709 12:26:17.381203  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:16.972723  9895 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:17.382560  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 781100us
I20260709 12:26:17.383526  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:16.979060  9800 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:17.393900  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:17.396510  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 762006us
I20260709 12:26:17.397418  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:17.398288  5705 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:17.400715  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:17.047042  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:16.961294  9903 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:17.047840  6277 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:26:17.030071  9715 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:17.411652  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 718) took 1067 ms. Trace:
I20260709 12:26:17.419806  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:17.410164  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:17.085978  9893 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.333s	user 0.001s	sys 0.000s
I20260709 12:26:17.419554  9822 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:17.419855  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:17.420183  5561 rpcz_store.cc:276] 0709 12:26:16.343812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.524743 (+180931us) service_pool.cc:224] Handling call
0709 12:26:16.525977 (+  1234us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:16.784969 (+258992us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:16.785005 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:16.785705 (+   700us) log.cc:844] Serialized 26 byte log entry
0709 12:26:16.877458 (+ 91753us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:16.877512 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.877616 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:17.411381 (+533765us) raft_consensus.cc:1685] finished
0709 12:26:17.411406 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:17.411582 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64725,"raft.queue_time_us":95824,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":9296,"spinlock_wait_cycles":41728,"thread_start_us":293031,"threads_started":2,"wal-append.queue_time_us":266982}
I20260709 12:26:17.436833  5708 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:17.369920  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2079) took 1131 ms. Trace:
I20260709 12:26:17.439460  5559 rpcz_store.cc:276] 0709 12:26:16.238446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.392244 (+153798us) service_pool.cc:224] Handling call
0709 12:26:16.392686 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:16.675126 (+282440us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.369674 (+694548us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b540044f054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:17.369856 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":648011008}
I20260709 12:26:17.431315  6274 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:17.431984  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:17.426813  5703 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.761s	user 0.014s	sys 0.001s
I20260709 12:26:17.449426  5703 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:17.046761  9902 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:17.228922  9897 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.405s	user 0.001s	sys 0.000s
W20260709 12:26:17.213117  9890 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.364s	user 0.001s	sys 0.000s
W20260709 12:26:17.249774  9777 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.865s	user 0.007s	sys 0.004s
I20260709 12:26:17.081352  9904 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:17.326653  9899 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:17.331144  9801 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:17.231309  9891 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.443s	user 0.000s	sys 0.001s
W20260709 12:26:17.208730  9896 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.343s	user 0.001s	sys 0.000s
I20260709 12:26:17.415786  9898 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.478605  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:17.487323  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:17.538867  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 116146us
W20260709 12:26:17.561582  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.007s	sys 0.010s
I20260709 12:26:17.491907  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:17.563261  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.007s	sys 0.010s
I20260709 12:26:17.563598  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:17.564725  5703 raft_consensus.cc:2410] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 11 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:26:17.480257  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:17.546409  6235 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.021s	sys 0.028s
I20260709 12:26:17.243106  9726 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:17.451478  9822 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:17.569828  6235 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.021s	sys 0.028s
W20260709 12:26:17.600819  5557 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.008s	sys 0.012s
W20260709 12:26:17.603284  5557 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.009s	sys 0.012s
I20260709 12:26:17.657279  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:17.460232  9735 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.032s	sys 0.058s
W20260709 12:26:17.661024  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.006s	sys 0.021s
W20260709 12:26:17.696781  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.006s	sys 0.021s
W20260709 12:26:17.697198  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 658992us
I20260709 12:26:17.698360  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:17.699991  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:17.694440  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.005s	sys 0.028s
W20260709 12:26:17.701370  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.006s	sys 0.028s
W20260709 12:26:17.702289  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 805697us
I20260709 12:26:17.528520  9860 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:26:17.711920  5527 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.016s	sys 0.016s
W20260709 12:26:17.723618  5527 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.017s	sys 0.017s
W20260709 12:26:17.724378  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.734321  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:17.735160  6271 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:17.460157  9903 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:17.792232  5675 connection.cc:256] RPC call timeout handler was delayed by -2.7929747759999373s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19430000000000003s for a response
I20260709 12:26:17.791640  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:17.696635  9735 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.032s	sys 0.058s
I20260709 12:26:17.554944  9864 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.693814  9899 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:17.822010  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:17.822949  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:17.859014  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.814236  6277 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:26:17.659057  9822 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:17.828984  6278 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:17.829879  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:17.860683  6271 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:17.861931  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:26:17.830509  5675 connection.cc:256] RPC call timeout handler was delayed by -2.0479063119996681s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22400000000000003s for a response
W20260709 12:26:17.839949  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:17.863180  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.863054  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:17.736081  9726 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:17.868651  9726 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:17.723496  9860 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:17.876185  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:17.722745  9865 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:17.899814  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:17.915100  6275 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.546s	user 0.011s	sys 0.000s
I20260709 12:26:17.737083  9895 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:17.915706  6280 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.869s	user 0.028s	sys 0.069s
W20260709 12:26:17.939507  6280 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.893s	user 0.029s	sys 0.069s
I20260709 12:26:17.926486  6275 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:17.837870  9619 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:17.540091  9902 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:17.949011  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 72959.4us
I20260709 12:26:17.969234  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:17.914914  9748 raft_consensus.cc:2764] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:26:17.869859  9856 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:17.914803  9750 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.922683  9860 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:18.014843  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45726.6us
I20260709 12:26:17.783463  9900 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.245s
I20260709 12:26:17.514608  9879 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:17.573549  9904 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:17.659274  9898 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:17.921033  9784 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:17.972893  9697 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.721s	user 0.044s	sys 0.078s
W20260709 12:26:17.973373  9916 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.731s	user 0.001s	sys 0.001s
I20260709 12:26:18.014679  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:17.898706  9899 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:17.897888  9864 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:17.965341  9910 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:18.043330  6278 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:18.016137  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:18.061674  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45615.4us
I20260709 12:26:18.063236  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:18.063547  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 192562us
I20260709 12:26:18.064172  5710 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 12:26:18.065006  5524 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.711s (SENT)
W20260709 12:26:18.026463  9698 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.474s	user 0.016s	sys 0.000s
I20260709 12:26:18.054503  9865 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:18.006199  9734 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:26:18.097532  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 397634us
I20260709 12:26:18.100273  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:18.101029  5631 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
W20260709 12:26:18.102382  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 380463us
I20260709 12:26:18.103423  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:18.026597  9937 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:18.109403  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:18.065564  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:18.091472  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:18.005829  9925 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:26:18.042341  9697 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.044s	sys 0.078s
I20260709 12:26:17.974174  9619 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:18.050216  9748 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:18.013312  9856 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:18.021497  9750 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:18.122634  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:18.143155  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:18.086467  9844 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:26:18.137913  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72944.3us
I20260709 12:26:18.067272  9921 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:18.049592  9889 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.175068  9929 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.141721  9734 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:18.188606  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:18.187348  9925 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:18.191762  9902 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:18.043314  9903 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:18.174408  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2023) took 1914 ms. Trace:
I20260709 12:26:18.150126  9844 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.234524  5556 rpcz_store.cc:276] 0709 12:26:16.259944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.513048 (+253104us) service_pool.cc:224] Handling call
0709 12:26:16.513452 (+   404us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:16.786282 (+272830us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:16.786331 (+    49us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:26:16.787128 (+   797us) log.cc:844] Serialized 78 byte log entry
0709 12:26:16.951544 (+164416us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:26:16.951597 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.951697 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:18.174132 (+1222435us) raft_consensus.cc:1685] finished
0709 12:26:18.174153 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.174340 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67468,"raft.queue_time_us":146231,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":9316,"thread_start_us":330719,"threads_started":2,"wal-append.queue_time_us":187575}
I20260709 12:26:18.226796  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:18.242986  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:18.244089  5703 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:26:18.087350  9910 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:18.177531  9703 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:18.147040  5707 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
W20260709 12:26:18.234316  9915 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.382s	user 0.000s	sys 0.001s
W20260709 12:26:18.234532  9285 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.547s	user 0.010s	sys 0.005s
I20260709 12:26:18.245563  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:18.285369  9927 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.285630  9726 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:18.286266  9726 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:18.240409  9925 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:18.289434  9912 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
I20260709 12:26:18.289984  9925 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:18.286104  9927 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:18.290791  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.291692  9932 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:18.292541  9927 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:18.294283  5703 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:18.202750  9921 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:18.225018  9858 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.469s	user 0.011s	sys 0.005s
I20260709 12:26:18.295674  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:18.294001  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 702) took 1610 ms. Trace:
I20260709 12:26:18.301704  5702 rpcz_store.cc:276] 0709 12:26:16.683769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.683939 (+   170us) service_pool.cc:224] Handling call
0709 12:26:16.684259 (+   320us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:16.684809 (+   550us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:16.684838 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:16.685294 (+   456us) log.cc:844] Serialized 26 byte log entry
0709 12:26:16.867976 (+182682us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:16.868031 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.868114 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:18.293714 (+1425600us) raft_consensus.cc:1685] finished
0709 12:26:18.293743 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.293926 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":22887,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":5757,"spinlock_wait_cycles":9444864,"thread_start_us":148420,"threads_started":1,"wal-append.queue_time_us":1362256}
I20260709 12:26:18.247031  9844 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:18.310963  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:18.311704  5702 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:18.305431  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 202063us
I20260709 12:26:18.313243  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:18.199698  9748 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:18.305483  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:18.318792  9093 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.611s	user 0.008s	sys 0.008s
I20260709 12:26:18.319391  9093 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:18.320684  9573 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:18.320884  6273 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:18.321386  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:18.322153  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 810) took 1338 ms. Trace:
I20260709 12:26:18.322582  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:18.322978  6275 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:18.323590  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:18.329699  9910 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:18.330513  9910 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.332098  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:18.335384  9931 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:18.335767  5708 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260709 12:26:18.336233  9931 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:18.337245  5708 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:18.337766  5708 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:18.339002  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2084) took 1975 ms. Trace:
I20260709 12:26:18.339841  9904 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:18.340201  9917 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.340906  9898 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:18.338647  5708 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:18.322602  6273 rpcz_store.cc:276] 0709 12:26:16.983324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.983460 (+   136us) service_pool.cc:224] Handling call
0709 12:26:18.321912 (+1338452us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b54002ce7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:18.322085 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2109179264}
I20260709 12:26:18.342314  5708 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:18.280915  9946 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:18.297955  9726 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:26:18.347666  5704 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.007s	sys 0.017s
W20260709 12:26:18.996472  5704 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.288s	user 0.009s	sys 0.019s
I20260709 12:26:18.236369  9734 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:18.306617  9921 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:18.999161  9734 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:18.370858  9946 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.001864  9946 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:18.339792  9931 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:18.397583  9856 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:18.398998  9750 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:18.299839  9285 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:19.005723  5636 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.880s	user 0.019s	sys 0.002s
W20260709 12:26:19.006129  5636 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.890s	user 0.019s	sys 0.002s
I20260709 12:26:19.007244  9285 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:18.402117  9388 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.682s	user 0.008s	sys 0.006s
I20260709 12:26:19.008477  9388 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:19.010798  9388 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:18.374356  9726 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:19.011475  9726 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:18.403997  9539 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.554s	user 0.013s	sys 0.000s
I20260709 12:26:18.340814  9917 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.012696  9539 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:19.014163  9917 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:18.232759  9929 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.014811  9539 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:19.016173  9929 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:18.413817  9093 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:18.413914  9944 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.366155  9839 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:18.415948  9619 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:18.416756  9899 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:18.309695  9858 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:18.279007  9703 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:18.422178  9748 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:18.422370  9864 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:18.355525  9910 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:18.408277  9860 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:18.231997  9889 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:18.448582  9955 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:19.020466  9955 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:18.476292  9954 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:19.021389  9889 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:18.489398  9844 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:18.498267  9822 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.578s	user 0.008s	sys 0.004s
W20260709 12:26:18.580844  9888 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.008s	sys 0.021s
W20260709 12:26:18.589955  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:18.619143  9950 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:26:19.021904  9954 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.022439  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:19.023468  9844 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.023699  5554 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:19.032620  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:18.676210  9862 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:26:18.682206  9861 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 2.464s	user 0.005s	sys 0.015s
W20260709 12:26:18.692338  5634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.010s	sys 0.026s
I20260709 12:26:18.783329  9846 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:18.339386  5560 rpcz_store.cc:276] 0709 12:26:16.363991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.606793 (+242802us) service_pool.cc:224] Handling call
0709 12:26:16.607177 (+   384us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:16.784898 (+177721us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:16.784947 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:16.798054 (+ 13107us) log.cc:844] Serialized 52 byte log entry
0709 12:26:16.954889 (+156835us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:16.955011 (+   122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.955304 (+   293us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:18.338784 (+1383480us) raft_consensus.cc:1685] finished
0709 12:26:18.338805 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.338942 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":105697,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":6326,"thread_start_us":325124,"threads_started":2,"wal-append.queue_time_us":732224}
W20260709 12:26:18.807963  9935 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.612s	user 0.001s	sys 0.001s
W20260709 12:26:18.810009  5552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.014s	sys 0.006s
I20260709 12:26:18.841315  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2093) took 2144 ms. Trace:
W20260709 12:26:18.875247  6276 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 786) took 3531 ms (client timeout 1843 ms). Trace:
I20260709 12:26:18.875451  6272 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 11. Ops: []
I20260709 12:26:18.883195  9702 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:18.895766  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.675s	user 0.016s	sys 0.022s
W20260709 12:26:18.903779  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 590568us
W20260709 12:26:18.920011  5631 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.777s	user 0.021s	sys 0.019s
W20260709 12:26:19.041042  5631 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.899s	user 0.022s	sys 0.019s
W20260709 12:26:18.922523  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.096s	user 0.002s	sys 0.031s
W20260709 12:26:19.041683  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.215s	user 0.002s	sys 0.031s
I20260709 12:26:19.042850  9285 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:19.043283  9946 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.043355  9285 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.043916  9946 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.028736  9839 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.046418  9946 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.046633  9839 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.047680  9285 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.032964  9822 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:19.051685  9822 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:18.952312  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.000s	user 0.020s	sys 0.027s
W20260709 12:26:19.053840  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.102s	user 0.021s	sys 0.027s
I20260709 12:26:19.054337  9858 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:18.928889  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:18.964658  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.968259  9758 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:18.970921  9692 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:26:18.971453  5676 connection.cc:256] RPC call timeout handler was delayed by -1.0063970029996199s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
I20260709 12:26:18.978615  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 707) took 3687 ms. Trace:
I20260709 12:26:18.981663  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:18.986219  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:18.993794  9940 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.799s	user 0.002s	sys 0.002s
W20260709 12:26:18.624122  9963 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:26:18.956156  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 722) took 2156 ms. Trace:
W20260709 12:26:18.928196  9920 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.635s	user 0.002s	sys 0.004s
I20260709 12:26:19.059229  5561 rpcz_store.cc:276] 0709 12:26:16.800118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.486422 (+686304us) service_pool.cc:224] Handling call
0709 12:26:17.486686 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:18.021165 (+534479us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:18.021215 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:18.021281 (+    66us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:18.021338 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.955925 (+934587us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b54000267d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:18.955958 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.956093 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1228619648}
W20260709 12:26:18.625223  9949 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 12:26:19.060904  5559 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.618s	user 0.006s	sys 0.049s
W20260709 12:26:19.062170  5559 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.620s	user 0.006s	sys 0.049s
I20260709 12:26:19.062282  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2236) took 1627 ms. Trace:
I20260709 12:26:19.033454  9802 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:19.062690  5704 rpcz_store.cc:276] 0709 12:26:17.434687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.436142 (+  1455us) service_pool.cc:224] Handling call
0709 12:26:17.486591 (+ 50449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:17.487277 (+   686us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:17.487317 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:17.488075 (+   758us) log.cc:844] Serialized 26 byte log entry
0709 12:26:18.996892 (+1508817us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:26:18.996931 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.996999 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:19.062038 (+ 65039us) raft_consensus.cc:1685] finished
0709 12:26:19.062058 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:19.062223 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8633,"raft.queue_time_us":123,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":143,"thread_start_us":1288695,"threads_started":1,"wal-append.queue_time_us":971642}
W20260709 12:26:19.057408  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 128637us
I20260709 12:26:19.064513  9929 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:19.064586  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:19.064955  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:19.065337  5706 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:19.065237  9929 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:19.065613  9958 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:19.066305  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:19.067457  9864 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:26:19.069121  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2091) took 2544 ms. Trace:
I20260709 12:26:19.069653  5555 rpcz_store.cc:276] 0709 12:26:16.525148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.730799 (+205651us) service_pool.cc:224] Handling call
0709 12:26:16.731184 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:16.752693 (+ 21509us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:16.752725 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:16.788302 (+ 35577us) log.cc:844] Serialized 26 byte log entry
0709 12:26:16.949229 (+160927us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:16.949301 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.949422 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:18.927609 (+1978187us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b240013f4a8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff740b9c4 00007f4ff7409acb 00007f4ff75c4dc9 00007f4ff75c4a55 00007f4ff75c409d 00007f4ffa1f610e 00007f4ffa20d88c 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221
0709 12:26:19.068815 (+141206us) raft_consensus.cc:1685] finished
0709 12:26:19.068836 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:19.069062 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":95780,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":17703,"spinlock_wait_cycles":52900224,"thread_start_us":104031,"threads_started":1,"wal-append.queue_time_us":1467608}
W20260709 12:26:19.071266  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.034152  9862 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:19.071596  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:19.034570  9844 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.072451  9844 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:19.069003  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:19.073951  9964 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:19.074594  9964 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.075001  9844 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:19.075357  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:19.076213  9864 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:19.076889  9964 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.033987  9954 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.077433  9964 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:19.077569  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:19.038832  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 809) took 2055 ms. Trace:
I20260709 12:26:19.036870  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:19.038429  5552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.015s	sys 0.007s
I20260709 12:26:19.028662  9944 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:19.093223  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:19.078796  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:19.061136  5561 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:19.107420  5703 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:19.108639  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:19.109616  5703 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:19.092715  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:19.100756  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2074) took 3807 ms. Trace:
W20260709 12:26:19.039131  6276 rpcz_store.cc:269] 0709 12:26:15.343956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.346249 (+  2293us) service_pool.cc:224] Handling call
0709 12:26:18.875026 (+3528777us) spinlock_profiling.cc:243] Waited 667 ms on lock 0x7b54005910d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:18.875168 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1534230272}
I20260709 12:26:19.101141  5553 rpcz_store.cc:276] 0709 12:26:15.291343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.380018 (+ 88675us) service_pool.cc:224] Handling call
0709 12:26:15.456842 (+ 76824us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:16.526472 (+1069630us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.978276 (+2451804us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540044dc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:18.978510 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2445865856}
W20260709 12:26:19.035126  5634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.918s	user 0.011s	sys 0.026s
W20260709 12:26:19.039775  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.017s	sys 0.022s
I20260709 12:26:19.078394  6280 rpcz_store.cc:276] 0709 12:26:16.983013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.985085 (+  2072us) service_pool.cc:224] Handling call
0709 12:26:17.035699 (+ 50614us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:17.036112 (+   413us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:17.036156 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:17.045888 (+  9732us) log.cc:844] Serialized 26 byte log entry
0709 12:26:17.940086 (+894198us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:17.940162 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.940265 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:18.807259 (+866994us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:19.038490 (+231231us) raft_consensus.cc:1685] finished
0709 12:26:19.038517 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:19.038767 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":205854,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"spinlock_wait_cycles":35308160,"thread_start_us":893656,"threads_started":1,"wal-append.queue_time_us":1050743}
I20260709 12:26:19.039004  5558 rpcz_store.cc:276] 0709 12:26:16.697364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.858608 (+161244us) service_pool.cc:224] Handling call
0709 12:26:16.858907 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:16.859087 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:26:16.953426 (+ 94339us) log.cc:844] Serialized 14 byte log entry
0709 12:26:17.060706 (+107280us) log.cc:844] Serialized 14 byte log entry
0709 12:26:18.820853 (+1760147us) spinlock_profiling.cc:243] Waited 35.6 ms on lock 0x7b18002730a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:18.840879 (+ 20026us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b18002730a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287
0709 12:26:18.841006 (+   127us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:18.841066 (+    60us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:18.841098 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.841149 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.841260 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106303,"prepare.queue_time_us":2542038,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":4935,"raft.queue_time_us":1129585,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":141573,"spinlock_wait_cycles":121935616,"thread_start_us":548649,"threads_started":2}
I20260709 12:26:19.072654  9862 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:19.069144  5556 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 3.
W20260709 12:26:19.125129  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 46404.2us
I20260709 12:26:19.110544  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:19.128585  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:19.129019  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:19.129438  5558 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:26:19.082978  9802 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:19.130733  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:19.131062  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:19.130514  9802 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.130524  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:19.136983  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:19.138214  5558 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 13: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:26:19.138540  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.034322  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:26:19.140859  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 106603us
I20260709 12:26:19.129532  5552 rpcz_store.cc:276] 0709 12:26:15.293033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.967173 (+674140us) service_pool.cc:224] Handling call
0709 12:26:15.967580 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:15.968082 (+   502us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:26:15.984792 (+ 16710us) log.cc:844] Serialized 14 byte log entry
0709 12:26:18.182668 (+2197876us) spinlock_profiling.cc:243] Waited 4.89 ms on lock 0x7b1800074aa0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:26:19.100373 (+917705us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:19.100431 (+    58us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:19.100479 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.100551 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:19.100694 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15239,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":55,"raft.queue_time_us":6607,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":5535,"spinlock_wait_cycles":11258240,"thread_start_us":917421,"threads_started":1,"wal-append.queue_time_us":2546753}
I20260709 12:26:19.146281  5552 pending_rounds.cc:85] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 9
I20260709 12:26:19.146683  5552 pending_rounds.cc:107] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:26:19.147020  5552 raft_consensus.cc:2889] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:19.135615  5560 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:19.151912  5560 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20260709 12:26:19.153322  5560 pending_rounds.cc:85] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 3
I20260709 12:26:19.153875  5560 pending_rounds.cc:107] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20260709 12:26:19.154153  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:19.154212  5560 raft_consensus.cc:2889] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:19.157677  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:19.158624  5558 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:19.022904  9888 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.171s	user 0.009s	sys 0.022s
I20260709 12:26:19.161260  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:19.161823  5627 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:26:19.162006  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:19.155483  5707 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:19.171875  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:19.176489  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47530us
I20260709 12:26:19.178069  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:19.183197  5636 raft_consensus.cc:2368] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13.
I20260709 12:26:19.035988  9703 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.040273  9867 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:19.194840  5561 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 11. (term mismatch)
I20260709 12:26:19.195420  5561 pending_rounds.cc:85] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 10
I20260709 12:26:19.195708  5561 pending_rounds.cc:107] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 10.11
I20260709 12:26:19.195994  5561 raft_consensus.cc:2889] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:19.079488  9822 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:19.187232  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:19.030220  9910 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.161206  9844 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:19.021407  9860 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:19.035267  9846 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:19.151070  9750 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:19.222174  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:19.076892  9864 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:19.288599  9929 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:19.208037  9917 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20260709 12:26:19.291574  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:19.293105  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:19.293990  5636 raft_consensus.cc:2410] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:26:19.294792  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:19.278641  9844 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.297215  9802 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:19.061661  9692 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.298612  9844 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.299249  9692 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.299371  9844 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:19.300542  5703 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:19.301092  9921 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.303895  9921 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.209177  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:19.214737  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:19.212883  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:19.277448  5708 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.935s	user 0.010s	sys 0.003s
I20260709 12:26:19.314807  5708 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:26:19.161077  9957 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:26:19.278491  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60014.7us
I20260709 12:26:19.086978  9866 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.322417  9866 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.326370  9866 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:19.102682  9964 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:19.330252  9964 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:19.061264  9758 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.281032  9750 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.207890  9962 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:19.305788  5555 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 13: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:19.299965  9692 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.295387  5636 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:19.207962  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:19.308823  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:19.189419  9862 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:19.215444  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:19.315583  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2148) took 1094 ms. Trace:
I20260709 12:26:19.040179  9945 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:19.021441  9955 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.313890  5629 raft_consensus.cc:2376] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9. Re-sending same reply.
W20260709 12:26:19.294986  9911 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.396s	user 0.002s	sys 0.002s
I20260709 12:26:19.318845  5627 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 11: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:19.161454  9748 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:19.333279  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:19.212981  9694 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.186597  9889 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.165126  9388 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:19.218426  9759 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.217998  9856 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.171s
I20260709 12:26:19.453816  5632 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 12:26:19.368325  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 56217.3us
I20260709 12:26:19.652827  5561 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:19.653689  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:19.653885  6280 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:19.654830  5708 rpcz_store.cc:276] 0709 12:26:18.220763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.336758 (+115995us) service_pool.cc:224] Handling call
0709 12:26:18.337092 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:19.315260 (+978168us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.315516 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:19.655378  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.420693  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2244) took 1325 ms. Trace:
I20260709 12:26:19.452909  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:19.457036  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 719) took 2657 ms. Trace:
I20260709 12:26:19.657490  5628 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:26:19.440244  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 397120us
W20260709 12:26:19.658550  5601 connection.cc:256] RPC call timeout handler was delayed by -1.0346367840002131s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1837s for a response
W20260709 12:26:19.454457  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 118237us
W20260709 12:26:19.660107  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 327006us
W20260709 12:26:19.387640  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:19.410071  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:19.661438  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.338605  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:19.662204  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.369814  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 760) took 1267 ms. Trace:
I20260709 12:26:19.657521  5559 rpcz_store.cc:276] 0709 12:26:16.799326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.440746 (+641420us) service_pool.cc:224] Handling call
0709 12:26:17.441145 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:17.441403 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:26:19.062508 (+1621105us) log.cc:844] Serialized 14 byte log entry
0709 12:26:19.456552 (+394044us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:19.456636 (+    84us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:19.456699 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.456789 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:19.456955 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":90056,"prepare.queue_time_us":866482,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":98171,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":48248,"thread_start_us":1893584,"threads_started":2,"wal-append.queue_time_us":534515}
I20260709 12:26:19.387768  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:19.664074  5631 rpcz_store.cc:276] 0709 12:26:18.102433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.102563 (+   130us) service_pool.cc:224] Handling call
0709 12:26:18.102890 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:18.103112 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:26:19.041428 (+938316us) log.cc:844] Serialized 14 byte log entry
0709 12:26:19.368037 (+326609us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:19.368115 (+    78us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:26:19.368185 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.368283 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:19.369723 (+  1440us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88146,"prepare.queue_time_us":233846,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":18728,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":54078,"thread_start_us":1225129,"threads_started":2}
I20260709 12:26:19.667737  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:19.668409  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:19.669009  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:19.669950  5558 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.511s	user 0.009s	sys 0.001s
I20260709 12:26:19.670460  5558 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:19.660037  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:19.656431  5705 rpcz_store.cc:276] 0709 12:26:18.094875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.333355 (+238480us) service_pool.cc:224] Handling call
0709 12:26:18.333690 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:18.959218 (+625528us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:26:18.959661 (+   443us) log.cc:844] Serialized 14 byte log entry
0709 12:26:18.991798 (+ 32137us) log.cc:844] Serialized 14 byte log entry
0709 12:26:18.992601 (+   803us) log.cc:844] Serialized 14 byte log entry
0709 12:26:18.993612 (+  1011us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:18.993667 (+    55us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:18.993720 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.369910 (+376190us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b5400210f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:19.369978 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:19.420568 (+ 50590us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1611,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":107,"raft.queue_time_us":245,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":256,"spinlock_wait_cycles":1437677440,"thread_start_us":31019,"threads_started":1,"wal-append.queue_time_us":14530}
I20260709 12:26:19.676272  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:19.668859  5636 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:19.677802  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:19.672006  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:19.669378  5631 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:19.681346  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:19.683986  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.684327  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:19.685467  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:19.684023  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:19.686111  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 465642us
I20260709 12:26:19.288673  9944 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:19.386801  9954 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:26:19.372818  9901 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260709 12:26:19.444818  9957 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:19.694031  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:19.476080  9750 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.696534  5630 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:19.696928  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:19.675818  9888 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:19.707551  5556 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:19.696842  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:19.708600  5631 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:19.658262  9093 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.677029  9858 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:19.710143  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:19.334214  9802 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.368s
I20260709 12:26:19.666821  9692 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.705803  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:19.681612  9829 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:19.634526  9758 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.367826  9907 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:19.476450  9864 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.223s
I20260709 12:26:19.710495  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:19.693346  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:19.727463  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:19.698688  9946 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:19.760311  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 453245us
I20260709 12:26:19.761224  9946 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.762403  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:19.765039  9946 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.765950  5556 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:19.766320  5630 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:19.768028  9899 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:19.708778  5628 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:19.736150  5636 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:19.669641  9759 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.770326  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:19.768451  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:19.670387  9889 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:19.772833  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:19.773454  5630 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:19.669888  9694 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.841564  9987 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 12:26:19.870843  9988 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:19.750810  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 417167us
W20260709 12:26:19.779202  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 68801.2us
I20260709 12:26:19.978188  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:20.594029  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2044) took 1409 ms. Trace:
I20260709 12:26:20.594470  5553 rpcz_store.cc:276] 0709 12:26:19.184408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.977326 (+792918us) service_pool.cc:224] Handling call
0709 12:26:20.593958 (+616632us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:19.980036  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2096) took 1013 ms. Trace:
I20260709 12:26:19.984488  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:20.691872  5555 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:19.766942  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:19.986511  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2088) took 3621 ms. Trace:
I20260709 12:26:19.989687  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:20.709489  5558 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:20.710180  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2045) took 1525 ms. Trace:
I20260709 12:26:20.710561  5558 rpcz_store.cc:276] 0709 12:26:19.184714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.989174 (+804460us) service_pool.cc:224] Handling call
0709 12:26:20.710104 (+720930us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:20.711805  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:19.704909  5561 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
W20260709 12:26:20.714305  5561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2047) took 1528 ms (client timeout 1896 ms). Trace:
W20260709 12:26:20.714730  5561 rpcz_store.cc:269] 0709 12:26:19.232049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.670602 (+438553us) service_pool.cc:224] Handling call
0709 12:26:20.714093 (+1043491us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:19.859445  5582 diagnostics_log.cc:230] Unable to collect stacks to diagnostics log: Incomplete: not collecting stack trace since debugger or strace is attached
I20260709 12:26:19.775429  5628 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:19.774564  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:20.735067  5555 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:26:20.735164  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2000) took 1447 ms. Trace:
I20260709 12:26:20.735720  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2046) took 1550 ms. Trace:
I20260709 12:26:20.735662  5630 rpcz_store.cc:276] 0709 12:26:19.287240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.774265 (+487025us) service_pool.cc:224] Handling call
0709 12:26:20.734940 (+960675us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:19.995198  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:20.737317  5632 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:20.737844  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2280) took 1424 ms. Trace:
I20260709 12:26:20.738150  5632 rpcz_store.cc:276] 0709 12:26:19.313380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.994439 (+681059us) service_pool.cc:224] Handling call
0709 12:26:20.737784 (+743345us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:20.739701  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:19.996272  5635 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.880s	user 0.020s	sys 0.018s
W20260709 12:26:20.743438  5635 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.627s	user 0.024s	sys 0.018s
I20260709 12:26:19.867127  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:20.746363  5636 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:20.746941  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2284) took 1079 ms. Trace:
I20260709 12:26:20.747257  5636 rpcz_store.cc:276] 0709 12:26:19.667844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.866854 (+199010us) service_pool.cc:224] Handling call
0709 12:26:20.746859 (+880005us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:19.852311  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 163466us
W20260709 12:26:19.851341  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 145637us
W20260709 12:26:19.841755  5702 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.529s	user 0.018s	sys 0.000s
I20260709 12:26:20.771852  5702 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
W20260709 12:26:20.772850  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2153) took 2500 ms (client timeout 1772 ms). Trace:
I20260709 12:26:20.000820  5710 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:20.132897 10004 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.161267  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2038) took 1003 ms. Trace:
I20260709 12:26:20.191483 10009 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:26:20.222296  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.804114  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:19.865528  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:20.803730 10001 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:20.804749  6278 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:20.805756  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 818) took 1113 ms. Trace:
I20260709 12:26:20.805099 10001 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:20.806146  6278 rpcz_store.cc:276] 0709 12:26:19.692095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.864983 (+172888us) service_pool.cc:224] Handling call
0709 12:26:20.805538 (+940555us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:19.880997  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:20.808979  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 814) took 1161 ms. Trace:
I20260709 12:26:20.809628  6277 rpcz_store.cc:276] 0709 12:26:19.647817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.880646 (+232829us) service_pool.cc:224] Handling call
0709 12:26:20.808796 (+928150us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:19.850256  6276 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.010s	sys 0.038s
W20260709 12:26:20.815277  6276 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.508s	user 0.011s	sys 0.039s
I20260709 12:26:20.815106  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 817) took 1127 ms. Trace:
I20260709 12:26:20.815721  6280 rpcz_store.cc:276] 0709 12:26:19.687096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.789866 (+102770us) service_pool.cc:224] Handling call
0709 12:26:20.814834 (+1024968us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:19.862824  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 806) took 2149 ms. Trace:
W20260709 12:26:19.776283  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 487264us
I20260709 12:26:20.819105  6272 rpcz_store.cc:276] 0709 12:26:17.713577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.713732 (+   155us) service_pool.cc:224] Handling call
0709 12:26:18.875213 (+1161481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:19.039347 (+164134us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.862568 (+823221us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54005910d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:19.862745 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2670298496}
I20260709 12:26:19.874900  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:19.775352  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:20.823030  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 819) took 1114 ms. Trace:
I20260709 12:26:20.823418  6271 rpcz_store.cc:276] 0709 12:26:19.708667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.775097 (+ 66430us) service_pool.cc:224] Handling call
0709 12:26:20.822847 (+1047750us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:19.669149  9388 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:19.887084  9702 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.847s	user 0.009s	sys 0.002s
I20260709 12:26:19.700850  9703 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:20.835325  9856 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.835937  9703 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:20.836676  9844 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.837361  9844 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:20.839293  9703 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:20.156246  9750 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:20.842319  9750 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:19.772186  9759 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.666812  9748 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:20.242452  9822 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.030s	user 0.014s	sys 0.017s
I20260709 12:26:20.853523  9822 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:19.776404  9829 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:20.238456  9889 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:19.768572  9899 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:20.858776  9899 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:20.170137  9910 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.185577  9539 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.857247  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:19.778450  9907 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:19.825981  9957 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:20.888952  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:20.272816  9986 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.448s	user 0.003s	sys 0.000s
W20260709 12:26:20.340512  9991 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.496s	user 0.000s	sys 0.004s
I20260709 12:26:20.349570  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:20.423835  9881 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.094s	user 0.041s	sys 0.055s
I20260709 12:26:20.877462  9388 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:20.454723  9285 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:26:20.480933  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.483839  9846 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:20.902760  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.511781  9692 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:20.516764  9988 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:20.906340  9988 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:20.521771  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.876955  9910 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:20.515851  9694 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:19.733556  9858 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:20.908841  9910 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:20.531694  9866 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:20.912716  9866 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:20.913543  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 728) took 1845 ms. Trace:
I20260709 12:26:20.547350  9945 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.913977  9866 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:20.914449  9945 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:20.915768  9866 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:20.559396  9862 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.900s	user 0.020s	sys 0.004s
W20260709 12:26:20.916445  9881 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.587s	user 0.043s	sys 0.057s
I20260709 12:26:20.919847  9748 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:20.920922  5631 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
W20260709 12:26:20.923506  5631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2279) took 1639 ms (client timeout 1892 ms). Trace:
I20260709 12:26:20.567972 10000 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.569036  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:20.928087 10000 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:20.929096  5559 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:20.930546 10000 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:20.931951  9829 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:19.739513  9093 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:20.935214  5457 catalog_manager.cc:5697] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 5 to 6, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 6 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:20.937538  9093 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:20.592331  9751 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:20.938382  9751 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:20.937724  5457 catalog_manager.cc:5697] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 4 to 5, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 5 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:20.939570  9751 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:20.611173  5554 rpcz_store.cc:276] 0709 12:26:18.966719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.972479 (+  5760us) service_pool.cc:224] Handling call
0709 12:26:18.972875 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:19.032387 (+ 59512us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.979354 (+946967us) spinlock_profiling.cc:243] Waited 42.7 ms on lock 0x7b54003c4bd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:19.979785 (+   431us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":98167168}
W20260709 12:26:20.623013  9981 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.802s	user 0.002s	sys 0.002s
I20260709 12:26:20.635814  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:19.758162  9709 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.894s
I20260709 12:26:20.703385  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2048) took 1470 ms. Trace:
W20260709 12:26:20.924083  5631 rpcz_store.cc:269] 0709 12:26:19.307895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.709787 (+401892us) service_pool.cc:224] Handling call
0709 12:26:20.923211 (+1213424us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54000c1a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:20.923391 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2746020224}
W20260709 12:26:20.721724  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 90583.2us
I20260709 12:26:20.706677  5557 rpcz_store.cc:276] 0709 12:26:16.365027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.680111 (+315084us) service_pool.cc:224] Handling call
0709 12:26:16.680448 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:16.708156 (+ 27708us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:26:16.723846 (+ 15690us) log.cc:844] Serialized 14 byte log entry
0709 12:26:17.605471 (+881625us) log.cc:844] Serialized 14 byte log entry
0709 12:26:17.661326 (+ 55855us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:17.661384 (+    58us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:17.661441 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.294321 (+1632880us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:19.985952 (+691631us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b540023f854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:19.986040 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:19.986305 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":123217,"prepare.queue_time_us":1284462,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":90,"raft.queue_time_us":569303,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":37041,"spinlock_wait_cycles":114615424,"thread_start_us":813025,"threads_started":2,"wal-append.queue_time_us":995273}
I20260709 12:26:20.736193  5555 rpcz_store.cc:276] 0709 12:26:19.184990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.983938 (+798948us) service_pool.cc:224] Handling call
0709 12:26:20.735651 (+751713us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:20.911808  9846 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:20.752974  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 731) took 1568 ms. Trace:
I20260709 12:26:20.949188  5556 rpcz_store.cc:276] 0709 12:26:19.232676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.766690 (+534014us) service_pool.cc:224] Handling call
0709 12:26:20.703216 (+936526us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:20.874454  9539 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:20.757982  5558 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:20.956964  9709 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:20.957090  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:20.957491  9709 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:20.958374  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:20.758723  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:20.769822  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:19.997707  5701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2125) took 4379 ms (client timeout 2031 ms). Trace:
I20260709 12:26:20.969079  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:20.775856 10004 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:20.969347  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:20.969913  5558 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:20.971038  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:20.971560 10004 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:20.776603  5560 rpcz_store.cc:276] 0709 12:26:19.157688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.166692 (+  9004us) service_pool.cc:224] Handling call
0709 12:26:19.167282 (+   590us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:19.167556 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:26:19.167890 (+   334us) log.cc:844] Serialized 14 byte log entry
0709 12:26:19.992701 (+824811us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b18003ef3e0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:26:20.160473 (+167772us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:20.160552 (+    79us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:20.160696 (+   144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:20.160822 (+   126us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:20.161052 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":147923,"prepare.queue_time_us":174,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":172888,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":190,"spinlock_wait_cycles":10178432,"thread_start_us":17429,"threads_started":1,"wal-append.queue_time_us":844856}
I20260709 12:26:20.777487 10009 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:20.972784  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:20.777689  5702 rpcz_store.cc:269] 0709 12:26:18.272755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.310653 (+ 37898us) service_pool.cc:224] Handling call
0709 12:26:20.772698 (+2462045us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:20.776558  5628 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:20.792238  9958 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:20.794054  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2261) took 2700 ms. Trace:
W20260709 12:26:20.519685  5524 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.901s (SENT)
I20260709 12:26:20.799265  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2138) took 2706 ms. Trace:
I20260709 12:26:20.813633  9962 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.007987  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:20.834596  9758 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:20.839725  9844 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:20.850785  9759 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:19.703258  9932 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:21.045279  9751 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:21.045825  9932 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:21.046430  9932 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:20.982527  9709 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:20.878927  9917 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:20.843726  9750 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:20.884582  9908 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20260709 12:26:20.884594  9907 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:20.836020  9856 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:20.446059  9984 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.630s	user 0.001s	sys 0.004s
I20260709 12:26:20.899828  5627 raft_consensus.cc:2376] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10. Re-sending same reply.
I20260709 12:26:21.052917  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1986) took 1877 ms. Trace:
I20260709 12:26:20.902086  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2260) took 2811 ms. Trace:
W20260709 12:26:20.566717  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.611747  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:26:20.741709  9726 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.730s	user 0.017s	sys 0.012s
I20260709 12:26:20.911718  9858 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:20.951064  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:20.953169  5561 rpcz_store.cc:276] 0709 12:26:19.165557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.715511 (+1549954us) service_pool.cc:224] Handling call
0709 12:26:20.715992 (+   481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:20.716335 (+   343us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:20.732912 (+ 16577us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:20.732979 (+    67us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:20.733046 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:20.733232 (+   186us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:20.733368 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.053257  5627 rpcz_store.cc:276] 0709 12:26:19.175628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.348952 (+1173324us) service_pool.cc:224] Handling call
0709 12:26:21.052852 (+703900us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:20.754896  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2004) took 1462 ms. Trace:
I20260709 12:26:20.951385  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2039) took 1760 ms. Trace:
I20260709 12:26:21.058826  5632 rpcz_store.cc:276] 0709 12:26:19.291478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.739301 (+1447823us) service_pool.cc:224] Handling call
0709 12:26:20.753747 (+ 14446us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:20.949231  6272 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:20.955225  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:20.972184  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:20.940009  5457 catalog_manager.cc:5697] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 5 to 9, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 9 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
W20260709 12:26:20.791334  6301 debug-util.cc:398] Leaking SignalData structure 0x7b08002d4300 after lost signal to thread 4359
I20260709 12:26:20.977344  6273 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:20.978116  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:20.978469  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2272) took 1696 ms. Trace:
I20260709 12:26:21.060895  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2040) took 1889 ms. Trace:
I20260709 12:26:20.978781  9901 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 12:26:20.979769  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:20.979940 10000 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.980742  5629 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:20.978159  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:20.914043  5552 rpcz_store.cc:276] 0709 12:26:19.068324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.145404 (+ 77080us) service_pool.cc:224] Handling call
0709 12:26:19.145970 (+   566us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:19.178540 (+ 32570us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:19.178593 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:19.179421 (+   828us) log.cc:844] Serialized 52 byte log entry
0709 12:26:19.503945 (+324524us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:19.504009 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.661470 (+157461us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:20.271933 (+610463us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:20.912961 (+641028us) raft_consensus.cc:1685] finished
0709 12:26:20.912985 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:20.913332 (+   347us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8509,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":11099,"spinlock_wait_cycles":48106752,"thread_start_us":324055,"threads_started":1,"wal-append.queue_time_us":621194}
W20260709 12:26:20.983976  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.975428  5557 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
W20260709 12:26:20.992017  5628 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2276) took 1708 ms (client timeout 2176 ms). Trace:
I20260709 12:26:20.954375  9910 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:21.063659  5628 rpcz_store.cc:269] 0709 12:26:19.283565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.772507 (+488942us) service_pool.cc:224] Handling call
0709 12:26:20.983264 (+1210757us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:20.969581  5701 rpcz_store.cc:269] 0709 12:26:15.617961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.670033 (+ 52072us) service_pool.cc:224] Handling call
0709 12:26:19.997116 (+4327083us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b5400309654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:19.997395 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":649312000}
I20260709 12:26:20.998077  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:20.975158 10009 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:21.007510  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:21.008508  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:20.981657  5635 rpcz_store.cc:276] 0709 12:26:18.093632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.104833 (+ 11201us) service_pool.cc:224] Handling call
0709 12:26:18.105141 (+   308us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:20.743966 (+2638825us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:20.744024 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:20.755294 (+ 11270us) log.cc:844] Serialized 26 byte log entry
0709 12:26:20.790941 (+ 35647us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:20.790980 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:20.791125 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:20.793578 (+  2453us) raft_consensus.cc:1685] finished
0709 12:26:20.793608 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:20.793934 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10562,"raft.queue_time_us":307927,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":2649635,"threads_started":2,"wal-append.queue_time_us":22614}
I20260709 12:26:21.018746  5710 rpcz_store.cc:276] 0709 12:26:18.090421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.092845 (+  2424us) service_pool.cc:224] Handling call
0709 12:26:19.999914 (+1907069us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:20.775231 (+775317us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:20.789531 (+ 14300us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b5400309650. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:20.796514 (+  6983us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4384716160}
I20260709 12:26:21.025815  9862 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:21.026331  9958 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:26:21.076130  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:21.027365  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 507872us
I20260709 12:26:21.043588  9758 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.037513  9285 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260709 12:26:21.048607  9759 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:21.085588  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:21.052390  6275 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 816) took 1865 ms (client timeout 1663 ms). Trace:
I20260709 12:26:21.061050  5636 rpcz_store.cc:276] 0709 12:26:19.282466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.755159 (+1472693us) service_pool.cc:224] Handling call
0709 12:26:20.978393 (+223234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.049242  9709 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:21.058897  5553 rpcz_store.cc:276] 0709 12:26:19.158070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.595653 (+1437583us) service_pool.cc:224] Handling call
0709 12:26:20.609011 (+ 13358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:20.626731 (+ 17720us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:26:20.757539 (+130808us) log.cc:844] Serialized 14 byte log entry
0709 12:26:20.918135 (+160596us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:20.918187 (+    52us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:20.918249 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:20.918310 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:20.918420 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":135481,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":43,"raft.queue_time_us":34331,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":137,"thread_start_us":235206,"threads_started":2,"wal-append.queue_time_us":179912}
I20260709 12:26:21.037590 10001 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:21.067550  5559 raft_consensus.cc:2410] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:26:21.119877  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 733) took 1926 ms. Trace:
I20260709 12:26:21.050853  9917 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:21.066273  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2269) took 1784 ms (client timeout 1662 ms). Trace:
I20260709 12:26:21.048622  9945 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:21.075147  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 816) took 1418 ms. Trace:
I20260709 12:26:21.120326  5559 rpcz_store.cc:276] 0709 12:26:19.193182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.552316 (+1359134us) service_pool.cc:224] Handling call
0709 12:26:21.119792 (+567476us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.053717  5634 rpcz_store.cc:276] 0709 12:26:18.092412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.100413 (+  8001us) service_pool.cc:224] Handling call
0709 12:26:18.116422 (+ 16009us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:19.128733 (+1012311us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:19.128763 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:19.182432 (+ 53669us) log.cc:844] Serialized 52 byte log entry
0709 12:26:19.728391 (+545959us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:19.855711 (+127320us) log.cc:844] Serialized 14 byte log entry
0709 12:26:19.856019 (+   308us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.856111 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:20.339293 (+483182us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:20.901504 (+562211us) raft_consensus.cc:1685] finished
0709 12:26:20.901643 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:20.901887 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":129154,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":297029,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":1357,"spinlock_wait_cycles":4220160,"thread_start_us":1549865,"threads_started":3,"wal-append.queue_time_us":545447}
I20260709 12:26:21.077497  5556 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
W20260709 12:26:21.121225  5629 rpcz_store.cc:269] 0709 12:26:19.281672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.665409 (+383737us) service_pool.cc:224] Handling call
0709 12:26:20.635497 (+970088us) spinlock_profiling.cc:243] Waited 2.06 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754deee 000000000036eb39 00007f4fff5c11e0 00007f4fff68a0c4 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:26:21.066159 (+430662us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4761600}
W20260709 12:26:21.085572  5458 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35600 (request call id 275) took 42754 ms (client timeout 15000 ms). Trace:
W20260709 12:26:21.105855  6275 rpcz_store.cc:269] 0709 12:26:19.186875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.007054 (+820179us) service_pool.cc:224] Handling call
0709 12:26:21.052273 (+1045219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.085968  6237 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:21.086077  5457 catalog_manager.cc:5697] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 5 to 10, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 10 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:21.088971  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:21.089555  6278 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:21.078358 10000 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:21.090745  9844 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:21.091288  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 820) took 1815 ms. Trace:
I20260709 12:26:21.091770  5557 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:26:21.146227  5557 pending_rounds.cc:85] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 5
I20260709 12:26:21.146710  5557 pending_rounds.cc:107] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:26:21.147053  5557 raft_consensus.cc:2889] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:21.093099  6301 debug-util.cc:398] Leaking SignalData structure 0x7b08002d4360 after lost signal to thread 5493
I20260709 12:26:21.093631  9758 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:21.094381  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:21.094817  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:21.061398  5560 rpcz_store.cc:276] 0709 12:26:19.168615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.979087 (+1810472us) service_pool.cc:224] Handling call
0709 12:26:20.979581 (+   494us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:20.980004 (+   423us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:26:20.980495 (+   491us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.056924 (+ 76429us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.057076 (+   152us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:21.057113 (+    37us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:21.057168 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.057236 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.057342 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2887,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":11049,"raft.queue_time_us":6029,"spinlock_wait_cycles":5812480,"thread_start_us":70592,"threads_started":1}
I20260709 12:26:21.096107  9908 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:21.097404  5676 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:21.098031  9901 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:21.088007  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:21.114773  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:21.115885  5554 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:21.104130  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:21.116398  6272 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:26:21.117307 10009 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.113816  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1987) took 1937 ms. Trace:
I20260709 12:26:21.146140  6273 rpcz_store.cc:276] 0709 12:26:19.276195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.874382 (+598187us) service_pool.cc:224] Handling call
0709 12:26:21.078220 (+1203838us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54002a8d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:21.091200 (+ 12980us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":318865280}
I20260709 12:26:21.155560  5631 rpcz_store.cc:276] 0709 12:26:19.176014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.976745 (+1800731us) service_pool.cc:224] Handling call
0709 12:26:21.113758 (+137013us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.128262  6279 rpcz_store.cc:276] 0709 12:26:19.648396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.611212 (+962816us) service_pool.cc:224] Handling call
0709 12:26:21.066901 (+455689us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.077816  9958 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:21.076635  9862 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:26:21.144944  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52661.1us
I20260709 12:26:21.151124  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2041) took 1982 ms. Trace:
I20260709 12:26:21.160392  5557 rpcz_store.cc:276] 0709 12:26:19.169043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.076745 (+1907702us) service_pool.cc:224] Handling call
0709 12:26:21.091436 (+ 14691us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.150855 (+ 59419us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.151058 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.066633  9726 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:21.128065  9917 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:21.154074  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:21.127980  9748 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.158424  5627 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:21.161796  5630 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:21.167517  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2003) took 1876 ms. Trace:
I20260709 12:26:21.167891  5627 rpcz_store.cc:276] 0709 12:26:19.291056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.058839 (+1767783us) service_pool.cc:224] Handling call
0709 12:26:21.167453 (+108614us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.163182  5560 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:21.164625  5556 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:21.162351  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:21.065773  9962 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:21.166886  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:21.162536  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 167026us
I20260709 12:26:21.165771  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:21.165395  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2006) took 1872 ms. Trace:
I20260709 12:26:21.145658  9844 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:21.174290  5557 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:21.153153  9908 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:21.164297  9862 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:21.175045  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 77832.8us
I20260709 12:26:21.128010  9910 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:21.174926  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 80237.1us
I20260709 12:26:21.174055  5632 rpcz_store.cc:276] 0709 12:26:19.292677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.068166 (+1775489us) service_pool.cc:224] Handling call
0709 12:26:21.165296 (+ 97130us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.220700  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:21.220927  9726 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:21.221377  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.221740  9726 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:21.221978  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:21.222250  9726 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:21.222692  9726 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:21.222399  9844 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.222764  9901 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.223904  9844 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:21.225184  9901 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:21.226488  9908 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.228565  5561 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:26:21.232554  9702 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:21.232702 10025 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:21.233165  9702 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:21.233712  9702 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:21.233505 10025 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:21.234092  5636 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:26:21.234755  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1992) took 1952 ms. Trace:
I20260709 12:26:21.235086  5636 rpcz_store.cc:276] 0709 12:26:19.282755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.233336 (+1950581us) service_pool.cc:224] Handling call
0709 12:26:21.233755 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.234465 (+   710us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.234679 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.235793 10025 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:21.236122 10000 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:21.237533  5628 raft_consensus.cc:2410] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:26:21.174260  5457 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 5 to 6, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 6 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
W20260709 12:26:21.238320  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 65821.1us
I20260709 12:26:21.175635  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:21.232326  9748 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:21.218274  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:21.262704  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:21.255488  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 74012.2us
I20260709 12:26:21.157550 10001 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 12:26:21.255936  9987 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:21.260310  5702 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20260709 12:26:21.249536  5524 connection.cc:256] RPC call timeout handler was delayed by -1.4249875740001698s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
I20260709 12:26:21.256203  5631 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:26:21.259464  5633 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:21.260624  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:21.293372  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:21.297868  9987 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:21.298346  9987 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:21.298794 10009 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:21.299007  5635 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:26:21.299778  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2263) took 2124 ms. Trace:
I20260709 12:26:21.300093  5635 rpcz_store.cc:276] 0709 12:26:19.175382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.162026 (+1986644us) service_pool.cc:224] Handling call
0709 12:26:21.289881 (+127855us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:21.299499 (+  9618us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.299704 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.300642  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2264) took 2120 ms. Trace:
I20260709 12:26:21.301046  5631 rpcz_store.cc:276] 0709 12:26:19.175747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.172290 (+1996543us) service_pool.cc:224] Handling call
0709 12:26:21.255748 (+ 83458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.296250 (+ 40502us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.296463 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.301553  5635 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:21.257720  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:21.259343  5457 catalog_manager.cc:5697] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:21.250240  9388 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:21.304849  5701 raft_consensus.cc:2410] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 10 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:26:21.306516  5636 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: [13.8-13.8]
W20260709 12:26:21.308094  5633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 1978) took 2189 ms (client timeout 1958 ms). Trace:
W20260709 12:26:21.308540  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 152808us
I20260709 12:26:21.170126  9899 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.313284  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1993) took 2030 ms. Trace:
I20260709 12:26:21.313623  5636 rpcz_store.cc:276] 0709 12:26:19.283037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.235869 (+1952832us) service_pool.cc:224] Handling call
0709 12:26:21.236330 (+   461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:21.313052 (+ 76722us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.313239 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.255698  9759 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:21.311309  5633 rpcz_store.cc:269] 0709 12:26:19.118552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.130413 (+ 11861us) service_pool.cc:224] Handling call
0709 12:26:21.304822 (+2174409us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b540009e2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:21.307979 (+  3157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3967153920}
I20260709 12:26:21.294041 10004 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:21.290686  5634 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 9.
I20260709 12:26:21.327404  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1996) took 2032 ms. Trace:
I20260709 12:26:21.327858  5635 rpcz_store.cc:276] 0709 12:26:19.283828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.300908 (+2017080us) service_pool.cc:224] Handling call
0709 12:26:21.301332 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.311355 (+ 10023us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.316254 (+  4899us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.243249 10023 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:21.318173  5627 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: []
W20260709 12:26:21.330435  5731 debug-util.cc:398] Leaking SignalData structure 0x7b08000579a0 after lost signal to thread 4359
I20260709 12:26:21.324852 10004 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:21.330787  5457 catalog_manager.cc:5697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 1 to 7. New cstate: current_term: 7 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:21.332060  5731 debug-util.cc:398] Leaking SignalData structure 0x7b08001e53e0 after lost signal to thread 5493
I20260709 12:26:21.333153  5634 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:21.333773  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2002) took 2044 ms. Trace:
W20260709 12:26:21.334113  5731 debug-util.cc:398] Leaking SignalData structure 0x7b080018c640 after lost signal to thread 5585
I20260709 12:26:21.334079  5634 rpcz_store.cc:276] 0709 12:26:19.290293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.332621 (+2042328us) service_pool.cc:224] Handling call
0709 12:26:21.332982 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.333543 (+   561us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.333721 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:21.334831  5731 debug-util.cc:398] Leaking SignalData structure 0x7b0800195580 after lost signal to thread 5586
I20260709 12:26:21.335807  5634 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:26:21.332201  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:21.337433  5633 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
W20260709 12:26:21.337476  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 81101.5us
I20260709 12:26:21.338270  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:21.331637 10004 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:21.338927  5709 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:21.340437 10022 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
I20260709 12:26:21.340819  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:21.338932  5633 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:26:21.342355  5703 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:26:21.342566  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2281) took 2028 ms. Trace:
I20260709 12:26:21.342904  5633 rpcz_store.cc:276] 0709 12:26:19.313729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.338299 (+2024570us) service_pool.cc:224] Handling call
0709 12:26:21.338617 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.342268 (+  3651us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.342470 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.343484  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:21.344189  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 813) took 2183 ms. Trace:
I20260709 12:26:21.344831  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:21.346941  9858 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.347740  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:21.347657  9858 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:21.348531  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2005) took 2056 ms. Trace:
I20260709 12:26:21.349174  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:21.351426  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:21.351836  9858 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:21.349015  5634 rpcz_store.cc:276] 0709 12:26:19.292096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.335106 (+2043010us) service_pool.cc:224] Handling call
0709 12:26:21.335523 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:21.336180 (+   657us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.348454 (+ 12274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.411482  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:21.356940  5628 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:21.387995  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:21.315042  9908 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:21.390985  9856 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:21.356421  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.450030  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:21.451357  9957 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:21.373594  6274 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:21.355221  9750 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.452569  9750 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:21.453590  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 814) took 2281 ms. Trace:
I20260709 12:26:21.373209  5631 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 10. Ops: []
I20260709 12:26:21.453979  6274 rpcz_store.cc:276] 0709 12:26:19.173483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.173756 (+   273us) service_pool.cc:224] Handling call
0709 12:26:19.174222 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.452425 (+2278203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.453287 (+   862us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b54005eef54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:21.453525 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4604289792}
I20260709 12:26:21.454962  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1997) took 2170 ms. Trace:
I20260709 12:26:21.455358  5631 rpcz_store.cc:276] 0709 12:26:19.284091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.310988 (+2026897us) service_pool.cc:224] Handling call
0709 12:26:21.372977 (+ 61989us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.454705 (+ 81728us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.454906 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.397862  5632 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260709 12:26:21.412314  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 110400us
I20260709 12:26:21.456802  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:21.387224  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:21.315244  9539 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:21.387436  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47892.8us
I20260709 12:26:21.360101  5675 leader_election.cc:400] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13.
W20260709 12:26:21.375483 10033 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
W20260709 12:26:21.355090  5524 connection.cc:256] RPC call timeout handler was delayed by -1.3088875739999821s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1693s for a response
I20260709 12:26:21.371227  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2266) took 2166 ms. Trace:
I20260709 12:26:21.462005  5627 rpcz_store.cc:276] 0709 12:26:19.205227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.173986 (+1968759us) service_pool.cc:224] Handling call
0709 12:26:21.308516 (+134530us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.329271 (+ 20755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.370938 (+ 41667us) spinlock_profiling.cc:243] Waited 64.7 ms on lock 0x7b540009e2d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:21.371144 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":148823808}
W20260709 12:26:21.368749  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 108242us
I20260709 12:26:21.344839  9899 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:21.358701  9932 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.474038  5635 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:21.475045  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2001) took 2187 ms. Trace:
I20260709 12:26:21.475533  5635 rpcz_store.cc:276] 0709 12:26:19.287671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.427535 (+2139864us) service_pool.cc:224] Handling call
0709 12:26:21.473743 (+ 46208us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.474690 (+   947us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.474940 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.476313  9932 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:21.355078  6301 debug-util.cc:398] Leaking SignalData structure 0x7b0800168a20 after lost signal to thread 5585
W20260709 12:26:21.477672  6301 debug-util.cc:398] Leaking SignalData structure 0x7b08000b92c0 after lost signal to thread 5586
W20260709 12:26:21.479316  6301 debug-util.cc:398] Leaking SignalData structure 0x7b080027cc20 after lost signal to thread 5660
W20260709 12:26:21.479596  6301 debug-util.cc:398] Leaking SignalData structure 0x7b080033c340 after lost signal to thread 5661
I20260709 12:26:21.353852  9748 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:21.420405  5704 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:21.311965  9388 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:21.484004  9388 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:21.466228  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:21.362406 10004 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.398921  6271 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:21.337703  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:21.424436  5731 debug-util.cc:398] Leaking SignalData structure 0x7b08002c10c0 after lost signal to thread 5660
I20260709 12:26:21.395444  9702 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.511653  9702 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:21.385321  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:21.407637  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53252.9us
I20260709 12:26:21.425169  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:21.434821  9945 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:21.515602  9945 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.317212 10000 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:21.324926  9901 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.518180  9901 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.520108  5702 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 13: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:21.344633  6276 rpcz_store.cc:276] 0709 12:26:19.168953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.186146 (+ 17193us) service_pool.cc:224] Handling call
0709 12:26:19.217570 (+ 31424us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:20.815896 (+1598326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:20.815941 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:20.817862 (+  1921us) log.cc:844] Serialized 52 byte log entry
0709 12:26:20.985570 (+167708us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:20.985615 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.012912 (+ 27297us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:21.343942 (+331030us) raft_consensus.cc:1685] finished
0709 12:26:21.343959 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.344087 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31780,"raft.queue_time_us":51582,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":115,"thread_start_us":1600377,"threads_started":2,"wal-append.queue_time_us":225599}
W20260709 12:26:21.353989  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:21.524263  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.383312  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:21.525190  6275 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:21.382231  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:21.526237  6272 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:21.440780  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1994) took 2157 ms. Trace:
I20260709 12:26:21.526881  5628 rpcz_store.cc:276] 0709 12:26:19.283290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.249777 (+1966487us) service_pool.cc:224] Handling call
0709 12:26:21.356735 (+106958us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.440565 (+ 83830us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.440734 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.527761  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:21.440745  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 734) took 2165 ms. Trace:
I20260709 12:26:21.528569  5558 rpcz_store.cc:276] 0709 12:26:19.275504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.074893 (+1799389us) service_pool.cc:224] Handling call
0709 12:26:21.089402 (+ 14509us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.089907 (+   505us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:26:21.157290 (+ 67383us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.413548 (+256258us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:21.439019 (+ 25471us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:26:21.439208 (+   189us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.440550 (+  1342us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.440687 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":11125,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":656,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":7409,"thread_start_us":250011,"threads_started":2,"wal-append.queue_time_us":284879}
W20260709 12:26:21.530279  6301 debug-util.cc:398] Leaking SignalData structure 0x7b08003a9c20 after lost signal to thread 5734
I20260709 12:26:21.442776  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:21.532685  9093 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:21.442250  9907 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:21.535495  9901 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:21.535790  9907 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:21.451891  9957 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:21.536535  9957 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:21.537309  9945 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:21.540364  9829 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.443631  5634 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:26:21.540902  9829 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:21.543098  9829 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:21.543486  9987 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.459769  5708 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:21.544754  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:21.467329  9750 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:21.467801  9846 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:21.545644  9846 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:21.546006  9846 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:21.546360  5709 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:21.546617  9862 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.547091  9899 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:21.547500  9899 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:21.547822  9899 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:21.467912  9910 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:21.469627  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 109785us
I20260709 12:26:21.550000  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:21.470100  9285 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:21.552740  9285 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.553789 10000 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:21.554553 10000 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:26:21.554876  9285 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:21.470209  5701 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:26:21.456840 10001 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.014s
I20260709 12:26:21.562269  9945 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.562688 10001 consensus_queue.cc:1243] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:26:21.563288 10001 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:21.562904  9945 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:21.563661 10001 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:21.564733 10001 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:21.565775  9945 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:21.472635  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:21.573150  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 100677us
I20260709 12:26:21.573175  5635 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:26:21.473767  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:26:21.129968  5458 rpcz_store.cc:269] 0709 12:25:38.263270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.263601 (+   331us) service_pool.cc:224] Handling call
0709 12:25:39.202724 (+939123us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5dc090 00007f4ffa67af3a 00007f5000430e9c 00007f500052f04e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc 00007f4ffe9ef392 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:21.009849 (+41807125us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:06.952738 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:07.284406 (+331668us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:07.309172 (+ 24766us) write_op.cc:435] Acquired schema lock
0709 12:26:07.338606 (+ 29434us) tablet.cc:662] Decoding operations
0709 12:26:07.735282 (+396676us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:07.735399 (+   117us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:07.735446 (+    47us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:26:08.076840 (+341394us) tablet.cc:701] Row locks acquired
0709 12:26:08.076900 (+    60us) write_op.cc:260] PREPARE: finished
0709 12:26:08.077131 (+   231us) write_op.cc:270] Start()
0709 12:26:08.077301 (+   170us) write_op.cc:276] Timestamp: P: 1783599968077039 usec, L: 0
0709 12:26:08.077329 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:26:08.078382 (+  1053us) log.cc:844] Serialized 5430 byte log entry
0709 12:26:09.906968 (+1828586us) op_driver.cc:464] REPLICATION: finished
0709 12:26:10.518215 (+611247us) write_op.cc:301] APPLY: starting
0709 12:26:10.518374 (+   159us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:10.553181 (+ 34807us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:10.553298 (+   117us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:11.008815 (+455517us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:11.009267 (+   452us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=28
0709 12:26:11.009323 (+    56us) write_op.cc:312] APPLY: finished
0709 12:26:11.009794 (+   471us) log.cc:844] Serialized 101 byte log entry
0709 12:26:11.321063 (+311269us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:11.321342 (+   279us) write_op.cc:454] Released schema lock
0709 12:26:11.321438 (+    96us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:12.299621 (+978183us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b180034eee0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:12.299750 (+   129us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:12.299788 (+    38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":854793,"reported_tablets":45,"spinlock_wait_cycles":48497408,"child_traces":[["op",{"apply.queue_time_us":610849,"apply.run_cpu_time_us":40800,"apply.run_wall_time_us":810179,"num_ops":14,"prepare.queue_time_us":1160525,"prepare.run_cpu_time_us":72013,"prepare.run_wall_time_us":1400311,"raft.queue_time_us":709737,"raft.run_cpu_time_us":120102,"raft.run_wall_time_us":673708,"replication_time_us":1829319,"spinlock_wait_cycles":25220224,"thread_start_us":2122622,"threads_started":5,"wal-append.queue_time_us":686272,"wal-append.run_cpu_time_us":71372,"wal-append.run_wall_time_us":3883842}]]}
I20260709 12:26:21.575963  9539 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:26:21.576783  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 103109us
I20260709 12:26:21.524040  9962 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:21.530755  6301 debug-util.cc:398] Leaking SignalData structure 0x7b08001b9c60 after lost signal to thread 5735
I20260709 12:26:21.541265  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2285) took 1873 ms. Trace:
I20260709 12:26:21.577883  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:21.578245  5634 rpcz_store.cc:276] 0709 12:26:19.668302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.440058 (+1771756us) service_pool.cc:224] Handling call
0709 12:26:21.443181 (+  3123us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.541019 (+ 97838us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.541215 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.459019  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1995) took 2175 ms. Trace:
I20260709 12:26:21.579315  5632 rpcz_store.cc:276] 0709 12:26:19.283577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.263532 (+1979955us) service_pool.cc:224] Handling call
0709 12:26:21.397543 (+134011us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:21.458707 (+ 61164us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.458962 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.457333  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:21.558665  5703 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: []
W20260709 12:26:21.559404  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 98439.8us
I20260709 12:26:21.481262  5457 catalog_manager.cc:5697] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 1 to 3, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 3 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:21.561965  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:21.566200 10001 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:21.471753  5524 connection.cc:256] RPC call timeout handler was delayed by -1.2620875740003612s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17450000000000002s for a response
W20260709 12:26:21.582994  5524 connection.cc:256] RPC call timeout handler was delayed by -1.2324177750001581s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3226s for a response
W20260709 12:26:21.583388  5524 connection.cc:256] RPC call timeout handler was delayed by -1.0991875740000978s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1926s for a response
I20260709 12:26:21.536289  9907 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:21.567087  9846 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.567973  5704 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:21.569868  9899 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:21.506717  5731 debug-util.cc:398] Leaking SignalData structure 0x7b0800132e00 after lost signal to thread 5661
W20260709 12:26:21.611696  5731 debug-util.cc:398] Leaking SignalData structure 0x7b08003cfee0 after lost signal to thread 5734
I20260709 12:26:21.585232  6273 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:21.586001  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:21.581156  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:26:21.613794  5731 debug-util.cc:398] Leaking SignalData structure 0x7b0800351ba0 after lost signal to thread 5735
I20260709 12:26:21.613945  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:21.618324  5709 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
W20260709 12:26:21.621693  5731 debug-util.cc:398] Leaking SignalData structure 0x7b08003f9b60 after lost signal to thread 6304
W20260709 12:26:21.622112  5731 debug-util.cc:398] Leaking SignalData structure 0x7b08003beda0 after lost signal to thread 6305
W20260709 12:26:21.622465  5731 debug-util.cc:398] Leaking SignalData structure 0x7b08001df880 after lost signal to thread 9388
W20260709 12:26:21.622835  5731 debug-util.cc:398] Leaking SignalData structure 0x7b0800387820 after lost signal to thread 9703
W20260709 12:26:21.623181  5731 debug-util.cc:398] Leaking SignalData structure 0x7b0800373140 after lost signal to thread 9748
W20260709 12:26:21.623492  5731 debug-util.cc:398] Leaking SignalData structure 0x7b08003f8ba0 after lost signal to thread 9759
W20260709 12:26:21.624907  5731 debug-util.cc:398] Leaking SignalData structure 0x7b0800229720 after lost signal to thread 9899
W20260709 12:26:21.625540  5731 debug-util.cc:398] Leaking SignalData structure 0x7b0800397800 after lost signal to thread 9908
W20260709 12:26:21.626900  5731 debug-util.cc:398] Leaking SignalData structure 0x7b0800031b80 after lost signal to thread 9932
I20260709 12:26:21.626513  5457 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 2 to 4, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 4 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:21.639309  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58251.4us
I20260709 12:26:21.609769  9901 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:21.645558  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:21.649791  6278 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:21.647574  5701 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
W20260709 12:26:21.653810  6301 debug-util.cc:398] Leaking SignalData structure 0x7b080033cf80 after lost signal to thread 6304
W20260709 12:26:21.657968  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 96114us
W20260709 12:26:21.656625  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 110571us
W20260709 12:26:21.658564  6301 debug-util.cc:398] Leaking SignalData structure 0x7b08003d0bc0 after lost signal to thread 6305
I20260709 12:26:21.661826  5457 catalog_manager.cc:5697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:21.609531  9932 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.045s
I20260709 12:26:21.627779 10000 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:26:21.692744  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:21.705513  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 60372.5us
I20260709 12:26:21.708809  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:21.711313  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 71078.2us
I20260709 12:26:21.712507  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:21.714795  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53252.3us
I20260709 12:26:21.613027  9957 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.618934  9702 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.089s
I20260709 12:26:21.733063  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:21.718408  6301 debug-util.cc:398] Leaking SignalData structure 0x7b08003e2f00 after lost signal to thread 9777
I20260709 12:26:21.609967  9750 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
I20260709 12:26:21.653544  9709 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:21.734397  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2283) took 2059 ms. Trace:
W20260709 12:26:21.734987  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60056.1us
I20260709 12:26:21.733590  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:21.744753  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:21.749678  6301 debug-util.cc:398] Leaking SignalData structure 0x7b080039b660 after lost signal to thread 9847
I20260709 12:26:21.747835  5633 rpcz_store.cc:276] 0709 12:26:19.653939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.352890 (+1698951us) service_pool.cc:224] Handling call
0709 12:26:21.353262 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.353550 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:26:21.613293 (+259743us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.712319 (+ 99026us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:21.712595 (+   276us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:21.712857 (+   262us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.712931 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.713273 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":198,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"thread_start_us":346232,"threads_started":2}
I20260709 12:26:21.751915  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:21.752564  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:21.752735  5558 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
W20260709 12:26:21.753402  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 127238us
I20260709 12:26:21.755276  5554 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:21.756248  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:21.757529  6301 debug-util.cc:398] Leaking SignalData structure 0x7b0800284ee0 after lost signal to thread 9855
I20260709 12:26:21.752005  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:21.758504  5560 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:21.709498  9862 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:21.709218 10000 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:21.658157  9958 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:21.764508  6277 raft_consensus.cc:2368] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14.
I20260709 12:26:21.764719  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:21.765395  9958 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:21.666489  9388 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:21.768491  5601 leader_election.cc:336] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.843s (SENT)
I20260709 12:26:21.770545 10047 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:21.771137 10047 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:21.771452 10047 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:21.747572  9709 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:21.772765  9709 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:21.760607  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:21.747134  5457 catalog_manager.cc:5697] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 2 to 8. New cstate: current_term: 8 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:26:21.779472  6301 debug-util.cc:398] Leaking SignalData structure 0x7b08003b0c80 after lost signal to thread 9871
I20260709 12:26:21.791369  5457 catalog_manager.cc:5697] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 3 to 6, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 6 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:21.795567  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:21.811479  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1990) took 2508 ms. Trace:
I20260709 12:26:21.813874  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:21.815017  5709 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:26:21.810142  9958 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.850342  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:21.713836  9748 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.063s
I20260709 12:26:21.851238  5555 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:21.718230  9751 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.072s
I20260709 12:26:21.779022  9388 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:21.812156  5630 rpcz_store.cc:276] 0709 12:26:19.282202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.236641 (+1954439us) service_pool.cc:224] Handling call
0709 12:26:21.298067 (+ 61426us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:21.317965 (+ 19898us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:21.317992 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:21.346333 (+ 28341us) log.cc:844] Serialized 26 byte log entry
0709 12:26:21.614321 (+267988us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:26:21.780933 (+166612us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.781477 (+   544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.781550 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:21.781588 (+    38us) raft_consensus.cc:1685] finished
0709 12:26:21.781599 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.790466 (+  8867us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10898,"prepare.queue_time_us":151582,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":32,"raft.queue_time_us":76901,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"thread_start_us":333865,"threads_started":2,"wal-append.queue_time_us":328789}
I20260709 12:26:21.778607 10000 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:21.863153  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:21.771574  9846 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:21.877372  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1998) took 2593 ms. Trace:
W20260709 12:26:21.878935  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 166997us
I20260709 12:26:21.846412  9901 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:26:21.881287  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 163073us
W20260709 12:26:21.876049  5599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.202s (SENT) [suppressed 34 similar messages]
I20260709 12:26:21.887390  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:21.883081  5457 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 6 to 7, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 7 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:21.877950  5636 rpcz_store.cc:276] 0709 12:26:19.284376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.330011 (+2045635us) service_pool.cc:224] Handling call
0709 12:26:21.330745 (+   734us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.331120 (+   375us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:26:21.612102 (+280982us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.876870 (+264768us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:21.876950 (+    80us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:21.877039 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.877132 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.877296 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11946,"prepare.queue_time_us":412738,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"thread_start_us":478312,"threads_started":2}
W20260709 12:26:21.897264  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 179220us
I20260709 12:26:21.941018  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 773) took 1263 ms. Trace:
W20260709 12:26:21.944490  6301 debug-util.cc:398] Leaking SignalData structure 0x7b08003e0ae0 after lost signal to thread 9915
W20260709 12:26:21.951005  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 236458us
I20260709 12:26:21.953065  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:21.954105  5701 raft_consensus.cc:2410] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 10 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20260709 12:26:21.972018  5526 connection.cc:256] RPC call timeout handler was delayed by -1.1118127909999203s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19430000000000003s for a response
W20260709 12:26:21.898115 10009 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.599s	user 0.014s	sys 0.000s
I20260709 12:26:21.891166  9901 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:21.970755  5457 catalog_manager.cc:5697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 5 to 6, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 6 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:21.944941  9987 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:21.879654  9958 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:21.953780 10048 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
I20260709 12:26:22.002651  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.007921  5555 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:21.986657  5457 catalog_manager.cc:5697] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 1 to 6. New cstate: current_term: 6 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:21.985795  9901 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:22.007530  9844 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.770s	user 0.014s	sys 0.000s
I20260709 12:26:21.997555  9899 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:22.006186  9862 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:22.003266  9957 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:22.065786  5457 catalog_manager.cc:5697] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 3 to 6, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 6 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:26:22.071017 10044 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:26:22.072588 10063 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:26:22.075568  9726 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.852s	user 0.008s	sys 0.008s
W20260709 12:26:22.081115  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 81850.1us
I20260709 12:26:22.086849  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:22.088042  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 134533us
I20260709 12:26:22.091652  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.068774  9862 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:22.099870  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.103528  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:22.101936  5706 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:22.117831  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:22.125597  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.130729  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:26:22.132886 10042 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.167s	user 0.000s	sys 0.001s
I20260709 12:26:22.134619  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 1991) took 2852 ms. Trace:
I20260709 12:26:22.135035  5629 rpcz_store.cc:276] 0709 12:26:19.282466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.219505 (+1937039us) service_pool.cc:224] Handling call
0709 12:26:21.220038 (+   533us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:21.305212 (+ 85174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:21.305260 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:21.396689 (+ 91429us) log.cc:844] Serialized 26 byte log entry
0709 12:26:21.668849 (+272160us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:26:21.864172 (+195323us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.940405 (+ 76233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.940478 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:22.134391 (+193913us) raft_consensus.cc:1685] finished
0709 12:26:22.134408 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:22.134552 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":113215,"prepare.queue_time_us":97258,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":81916,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":29930,"spinlock_wait_cycles":35732736,"thread_start_us":422488,"threads_started":2,"wal-append.queue_time_us":457592}
W20260709 12:26:22.138056 10056 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:26:22.145031  9862 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:22.146644  9862 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:22.149514  9907 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.539s	user 0.010s	sys 0.000s
W20260709 12:26:22.151520  9917 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.934s	user 0.008s	sys 0.004s
I20260709 12:26:22.091779  9726 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:22.126835  9539 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.000s	sys 0.029s
I20260709 12:26:22.164496  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.164522  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:22.172596  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.173640  5706 raft_consensus.cc:2410] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 14 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 12:26:22.174489  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.175258  5701 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:22.175746  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.175832 10048 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:26:22.176433  9758 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:22.176592  5705 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:22.151371  9907 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:22.128727 10045 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.240s	user 0.001s	sys 0.000s
I20260709 12:26:22.181452  9907 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:22.182420  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.183802  5705 raft_consensus.cc:2410] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 10 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:26:22.152240  9917 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:22.185168 10052 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:26:22.188371  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:22.189224  5705 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:22.191339  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:22.191988  5701 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:22.195674  9958 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:22.195798  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:22.196458  5702 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:22.199585  9709 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:22.202404  9917 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:22.096249 10060 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
I20260709 12:26:22.208503 10060 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:26:22.210345  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 101782us
I20260709 12:26:22.211369  5702 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:22.212103  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:22.217252  9917 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:22.217677  9917 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:22.217929 10004 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:22.218092  9917 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:22.218900  9759 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:22.213991  9388 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:26:22.212131  6234 connection.cc:256] RPC call timeout handler was delayed by -2.44248338899979s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22240000000000004s for a response
I20260709 12:26:22.276599  5701 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
W20260709 12:26:22.223306 10001 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.623s	user 0.011s	sys 0.000s
I20260709 12:26:22.225332  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:22.213708  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:22.234890  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:22.301805  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:22.302736  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:22.303622  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:22.303768  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 363134us
I20260709 12:26:22.116967  9910 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.120s
I20260709 12:26:22.236972  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:22.237620 10047 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:22.313057  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 99407.8us
I20260709 12:26:22.314288  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:22.314651  5629 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:22.315002 10047 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:22.242908  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:22.315709 10068 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:26:22.315848  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 73033.6us
I20260709 12:26:22.316484 10046 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:26:22.247031  9709 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:22.317363  9917 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:21.941576  5627 rpcz_store.cc:276] 0709 12:26:20.677222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.486609 (+809387us) service_pool.cc:224] Handling call
0709 12:26:21.486942 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.487274 (+   332us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:26:21.646961 (+159687us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.940405 (+293444us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.940561 (+   156us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:21.940622 (+    61us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:21.940710 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.940795 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.940944 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":278158,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":58157,"thread_start_us":258292,"threads_started":2}
I20260709 12:26:22.318261  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:22.170003  9539 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.000s	sys 0.030s
W20260709 12:26:22.244908  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.319024  9539 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:22.319403  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:22.255282  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.291003  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:22.320127  9539 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:22.320662  9758 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:22.291664  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:22.321663  9751 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:22.270229  9759 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:22.322220  9751 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:22.323001  9759 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:22.275563  9388 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:22.235991  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:22.333767  5552 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:22.316799 10068 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:26:22.317860  9709 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:22.320485  9748 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:22.321077  5557 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:26:22.269367  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:22.290270  5705 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:22.291932  9829 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:26:22.319895  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.316018  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:22.374224  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:22.375414  5630 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:22.375433  5554 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:22.339721  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:22.403784  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:22.404191  5628 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:22.368389  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.376757  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:22.323341  9751 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:22.378404  5559 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 11. Current term is 12. Ops: []
W20260709 12:26:22.376732  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 151574us
I20260709 12:26:22.409394  5556 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:22.338431  5561 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:22.376636  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:22.423045  5635 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:22.428769  6234 connection.cc:256] RPC call timeout handler was delayed by -1.6144063629999437s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30960000000000004s for a response
I20260709 12:26:22.319911  9917 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:22.317409 10046 consensus_queue.cc:1243] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 12.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:26:22.433007  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.413550  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:26:22.443929  6234 connection.cc:256] RPC call timeout handler was delayed by -1.3688770549997571s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15500000000000003s for a response
I20260709 12:26:22.444705  5636 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:22.446523  9844 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:26:22.447118  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:22.448259  9844 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:22.449079  9844 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:22.450090  9844 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:22.404855  9989 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:22.412438  9759 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:22.411499  9709 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.378752  9908 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:22.457268  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:22.457118  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.486716  5702 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:26:22.458364  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:22.458850  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45377.4us
I20260709 12:26:22.487615  5709 raft_consensus.cc:2368] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14.
I20260709 12:26:22.186933  9907 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.251s
I20260709 12:26:22.325354  9901 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:22.451768  9957 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.009s
I20260709 12:26:22.300716 10000 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:22.448251 10043 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:22.526719  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52391us
I20260709 12:26:22.479507  9958 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:22.482784  9709 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:22.473053  9285 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.072s
I20260709 12:26:22.484128  9908 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:22.537984  5601 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:22.539366  9703 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 12:26:22.565107  9962 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:22.565707  9962 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:22.565797  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:22.566293  9962 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:22.567478  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:22.548192 10054 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:22.569362 10054 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:22.548103  9388 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:22.572805 10000 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:22.548718 10043 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:22.574434  9846 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:26:22.575812  9846 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:22.576184  9846 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:22.576714  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.576810  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.577450  5706 raft_consensus.cc:2376] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12. Re-sending same reply.
I20260709 12:26:22.577965  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.578758  9901 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:22.579353  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.540858  9858 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:26:22.580526  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.580703  9858 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:22.581171  9858 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:22.582086  9858 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:22.583396  9858 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:22.587903  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:22.588590  5706 raft_consensus.cc:2376] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11. Re-sending same reply.
I20260709 12:26:22.590818  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:22.591490  5553 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:22.600224  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:22.604311 10073 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.050s	user 0.002s	sys 0.002s
W20260709 12:26:22.622753  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 74580.2us
I20260709 12:26:22.534278  9958 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.658450  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.662392  5706 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:22.531809  9750 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.122s
I20260709 12:26:22.665360  5557 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:22.641218  9901 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.670557  5557 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:22.679657  5628 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:22.689174  9093 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:22.689797  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.690485  5710 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:22.577209  9846 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:22.704852  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 390652us
I20260709 12:26:22.702433  5554 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:22.665223  9726 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:22.656698  9958 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:22.710281  5635 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
W20260709 12:26:22.714740  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 235427us
W20260709 12:26:22.710762  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 143440us
I20260709 12:26:22.716661  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:22.667954  9750 consensus_queue.cc:1243] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:26:22.722316  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:22.689849  9093 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:22.649677  9908 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:22.697305 10054 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:22.752147  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:22.753525  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:22.685410  9901 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:22.756479  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:22.734294  9093 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:22.765352 10075 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
W20260709 12:26:22.766165 10072 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20260709 12:26:22.772269  5553 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:22.773250  9709 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:22.775935  9709 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:22.777160  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:22.780548  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:22.779703  5710 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:26:22.781384  5553 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:22.782032  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51614us
I20260709 12:26:22.757076 10043 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:22.763087 10047 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:26:22.773265  9862 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.626s	user 0.019s	sys 0.002s
I20260709 12:26:22.792579  5554 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:22.796752  5552 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:22.782943  9932 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:22.787993  9908 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:22.869803  5601 leader_election.cc:336] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.763s (SENT)
I20260709 12:26:22.779641 10071 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:22.874609  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:22.875531  5553 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
W20260709 12:26:22.874526  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 78399.7us
I20260709 12:26:22.882601  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:22.883378  6277 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:22.885721  5704 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
W20260709 12:26:22.888597  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 136522us
I20260709 12:26:22.890017  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:22.894663  5552 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:22.873858  9858 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:22.897308  9858 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:22.898910  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:22.899854  9858 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:22.903952  9702 raft_consensus.cc:2764] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:26:22.904783 10047 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:26:22.905625  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2122) took 1171 ms. Trace:
I20260709 12:26:22.906234  5561 rpcz_store.cc:276] 0709 12:26:21.734375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.734515 (+   140us) service_pool.cc:224] Handling call
0709 12:26:22.002787 (+268272us) spinlock_profiling.cc:243] Waited 5.43 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:22.002995 (+   208us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:22.417912 (+414917us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:22.905263 (+487351us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b540023fd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:22.905533 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":775016192}
I20260709 12:26:22.910048  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:22.915522  9962 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:22.919900  5630 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:22.921384  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:22.923130  5706 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:22.924167  5553 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:22.925376  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:22.925606  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:22.926870 10055 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:22.927314 10055 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:22.927701 10055 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:22.927883  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:22.928642  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:22.929461  5633 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:22.930292  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:22.931083  6273 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:22.932183  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:22.933385  9702 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:26:22.938009  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:22.938859  9285 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:22.939448  9285 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:22.939682  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:22.940773  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:22.940446  9285 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:22.941824  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:22.945458  9945 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 12:26:22.945636 10088 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 12:26:22.946651  6273 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:22.949563  9945 consensus_queue.cc:1243] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:26:22.950380  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:22.945497 10047 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.951737 10047 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:22.953938 10047 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:22.954439 10047 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:26:22.953174  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:22.955297 10009 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:22.955652 10009 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.955960 10009 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:22.957949  9858 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:22.959844  9962 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:22.962054 10079 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.386s	user 0.006s	sys 0.006s
I20260709 12:26:22.963002  9901 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:22.948975 10091 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:22.964529  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:22.973193 10091 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:22.973459  5632 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:22.965744 10088 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:22.966048  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:22.966996  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:22.974279 10091 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:22.949625  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:22.969311  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:22.974300 10088 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:22.979686 10088 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:22.980530  9846 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:22.981091 10088 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:22.970288  9388 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:22.971182 10058 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:22.987085 10058 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:22.988324 10058 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:22.989825  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:22.976321  5628 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
W20260709 12:26:22.970178  5602 leader_election.cc:336] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.913s (SENT)
I20260709 12:26:22.992439  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:22.994020  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:22.988377  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:22.996825  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:22.997923  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:22.998046  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.999009  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:22.997476 10055 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.001029  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.001693  6277 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 7.
I20260709 12:26:23.001878  6274 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 11: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:23.004928  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:23.005728  6279 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:23.003943 10055 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:23.007268  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:23.058631  5559 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:23.008337  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.009897  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:22.999858 10064 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.020s
I20260709 12:26:23.017275 10047 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:23.017699  9987 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:23.018344 10004 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:23.018735  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:23.019416 10071 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:26:23.017590  9945 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:22.996222 10058 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:23.020735  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:23.022181  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:23.069221  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 50595.5us
I20260709 12:26:23.022215  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.023169  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:23.024010 10046 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:23.024276  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.024772  5553 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:23.026098  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.026134  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:23.027349  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.027822  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:26:23.022285  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:23.053439  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 143499us
W20260709 12:26:23.060032  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 60472.3us
I20260709 12:26:22.995182  9093 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:23.068466  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 127777us
W20260709 12:26:23.071951  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49844.1us
I20260709 12:26:23.053133  9962 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.080188  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:23.081001  6278 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 11: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:23.076946  6276 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
W20260709 12:26:23.086352  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 116240us
I20260709 12:26:23.065907  9899 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:23.087534  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:23.092716  5628 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:26:23.094208  5632 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:26:23.067795  9908 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:23.067354 10047 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:23.101874  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:23.102589  5635 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:23.096235  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:23.103680  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:23.068981  9987 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:23.098021  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:23.105927  5634 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:26:23.092106  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:23.088474  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.104771  5627 raft_consensus.cc:2410] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:26:23.083334  9702 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:23.095539  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:26:23.107759  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 80039.6us
I20260709 12:26:23.083963 10054 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:23.109261  5558 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:23.113660  5558 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:23.113454  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:23.112920  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:23.115321  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.115972  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:23.092438  9899 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:23.098806 10046 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:23.120499  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:23.124507  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:23.128715  5627 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:23.121080  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.096916  9989 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.123831  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:23.112473  9862 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:23.132380  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:23.131860  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:23.133045  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:23.141466  5628 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:23.134703  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:23.109965  9901 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:23.093101  9962 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:23.151067  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:23.147204  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.151544  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:23.152245  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:23.162945  5628 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:23.163625  5635 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:23.164991  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:23.165354  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:23.166492  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:23.166659  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:23.112584 10004 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:23.167716  5635 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:23.167995  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.167972 10004 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.169380  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:23.169930  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:23.170853  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.171007  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.171630  6272 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
W20260709 12:26:23.170867  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.171468  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:23.174508  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.115859  9908 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:23.179869  9908 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.118891  9702 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:23.181768  9702 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.183000  5558 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.184172  5558 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:23.184638  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:23.184857  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.188947  9987 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.123310 10054 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:23.191813 10054 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.112964 10064 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 12:26:23.192664  9899 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.194140  9899 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:23.195487  5631 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:26:23.176126 10046 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:23.175861  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 43580.8us
I20260709 12:26:23.122624  9932 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:23.216377  5636 raft_consensus.cc:1217] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Deduplicated request from leader. Original: 5.4->[6.5-12.8]   Dedup: 6.5->[8.6-12.8]
I20260709 12:26:23.134459  9907 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:23.215890 10023 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:26:23.177616  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45832.7us
I20260709 12:26:23.178256  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:23.134176 10000 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:23.187702  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.197697 10046 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:23.219205  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.264191  5627 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
W20260709 12:26:23.198685  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 64082.1us
I20260709 12:26:23.218560  5636 pending_rounds.cc:85] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 5
I20260709 12:26:23.266417  5636 pending_rounds.cc:107] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:26:23.266741  5636 raft_consensus.cc:2889] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:23.184638  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:26:23.202852  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 97436.7us
I20260709 12:26:23.199178  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:23.175625  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.114583 10055 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.102s
I20260709 12:26:23.272497  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:23.206406  5629 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:23.218346  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:23.293910  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 75633us
I20260709 12:26:23.232450  9932 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:23.243222  9093 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:23.219414  9751 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:23.150610  9759 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20260709 12:26:23.245167  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:23.252274  9962 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:23.253914  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.243880 10009 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.216512  9958 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.323710  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:23.324530  5702 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:23.222524  9866 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:23.263231  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:23.268816  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:23.268927 10088 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:23.295635  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:23.239670 10082 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:23.150328  9989 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.299073  5629 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20260709 12:26:23.354242  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:23.355113  5629 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 11: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:23.303871  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:23.306501  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:23.359228  6280 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:23.317445  5559 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:26:23.318056  5706 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:23.316545  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:23.328655  5628 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:23.329602  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:23.198022 10047 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.333395  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:23.284307 10023 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.335788  5708 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 11. Current term is 12. Ops: []
W20260709 12:26:23.338727  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 75574.5us
I20260709 12:26:23.428819  5704 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:23.335901  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 73666us
I20260709 12:26:23.344745  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:23.305687  9748 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:26:23.430393  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:23.337862  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:23.362283  5629 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:23.401535  5635 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:23.404886  5630 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:23.305207  9751 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.383011  5558 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:23.361814  6279 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 13: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:23.328913 10000 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:23.332543  9866 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:23.350325  9759 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:23.349195  9285 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:23.350472 10082 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:23.353719  9388 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:23.417039  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:23.478969  5705 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 11: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:23.418300  5602 leader_election.cc:336] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.896s (SENT)
I20260709 12:26:23.479822  5602 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:23.420854  5702 raft_consensus.cc:2410] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 10 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:26:23.450960  9751 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:23.421795  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 68353us
I20260709 12:26:23.485076  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:23.426971  5632 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:23.449405  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 743) took 1166 ms. Trace:
I20260709 12:26:23.486812  5558 rpcz_store.cc:276] 0709 12:26:22.283154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:22.283332 (+   178us) service_pool.cc:224] Handling call
0709 12:26:22.302318 (+ 18986us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:23.109163 (+806845us) spinlock_profiling.cc:243] Waited 61.8 ms on lock 0x7b54002aa6a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa6741c0 00007f4ffa675231 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c
0709 12:26:23.448482 (+339319us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:23.449333 (+   851us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":142109952}
I20260709 12:26:23.450311  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:23.489235  5629 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
W20260709 12:26:23.448120  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 370746us
I20260709 12:26:23.493247 10053 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:26:23.493676  6237 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:23.495186 10053 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:23.495590 10053 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:23.454943  5627 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:23.496294 10004 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:23.500933  5709 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:23.503383  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:23.431814 10023 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:23.523663  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:23.432893  9748 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:23.477161  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:23.498222  5627 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:23.525434  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:23.475536  9910 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 12 is higher.
W20260709 12:26:23.513701  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.545140  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:23.538218  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:23.546344  5708 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 11: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:23.546882  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 128631us
I20260709 12:26:23.498265 10004 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.499878  9750 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:26:23.548964  5602 leader_election.cc:336] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.926s (SENT)
I20260709 12:26:23.555860  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:23.475246  9285 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:23.505194  9388 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.499639  9862 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:23.569730  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 242636us
I20260709 12:26:23.548727  6278 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:23.537863  9748 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:23.519387 10068 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:23.525364  9846 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.498221  9751 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.556856  5703 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:23.571247  6280 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:23.496567 10043 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:26:23.554432 10004 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:23.553243 10060 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:23.551004  9759 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.552546  9945 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:23.433104  9932 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:23.569396  9285 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:23.566351  9908 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:23.685619  5710 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:23.644472  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 70642.3us
W20260709 12:26:23.687613  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:23.659471  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 182358us
I20260709 12:26:23.569803  9388 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:23.541142 10000 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:23.499323 10053 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:23.647200  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 121853us
I20260709 12:26:23.576844 10106 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:26:23.736074  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 284351us
I20260709 12:26:23.672900 10096 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:23.574673  9846 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:23.746114  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 105932us
I20260709 12:26:23.685295  9829 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:23.753952  9901 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:23.772125  9957 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:23.706477 10009 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:23.790189  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 303508us
I20260709 12:26:23.592664  9866 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.682082  9759 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:23.750855  9958 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:23.796972  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:23.783004  9709 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.021s
I20260709 12:26:23.780768  9901 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:23.799784  9908 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.661417 10082 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:23.846127  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:23.807852  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:23.830557  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.807127  9945 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.789157 10060 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:26:23.781029  9957 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:26:23.833994  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54834.2us
I20260709 12:26:23.786939 10096 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:23.860905  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.861696 10096 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:23.863986  9989 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:23.863842  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:23.807461 10101 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.398s	user 0.000s	sys 0.002s
I20260709 12:26:23.785156 10055 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 12:26:23.808661  9709 consensus_queue.cc:1243] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:26:23.812448 10108 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:23.833138 10110 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.240s	user 0.001s	sys 0.000s
I20260709 12:26:23.753276  9862 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:23.857313 10068 raft_consensus.cc:2764] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:26:23.860410  5599 leader_election.cc:400] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14.
I20260709 12:26:23.884296  5599 leader_election.cc:403] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:23.860674  9844 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:23.885421  5556 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 9. Current term is 10. Ops: []
W20260709 12:26:23.886351  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 95193.3us
I20260709 12:26:23.887122  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:23.839210  9750 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:23.867918  6274 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
W20260709 12:26:23.876380  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 327400us
I20260709 12:26:23.877909  5561 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 13: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:26:23.879489  5634 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.000s	sys 0.003s
W20260709 12:26:24.001510  5634 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.000s	sys 0.004s
W20260709 12:26:23.881480  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 347497us
W20260709 12:26:23.889741  5709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2154) took 1314 ms (client timeout 1662 ms). Trace:
W20260709 12:26:23.976261  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 116008us
W20260709 12:26:23.976276  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:23.977177  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.874202 10082 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:23.957299  9901 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:23.870765 10090 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:23.852559  9908 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:24.038549 10118 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:24.039186  9388 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:24.039980  5633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:24.040706  5633 raft_consensus.cc:606] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:24.041079  5633 raft_consensus.cc:614] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:26:24.042644  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 235195us
I20260709 12:26:24.045286  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:24.045462  9388 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:24.046555  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:24.047242  5560 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
W20260709 12:26:24.008531 10111 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
W20260709 12:26:24.023523 10110 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
I20260709 12:26:24.008986  9958 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14.
I20260709 12:26:23.885417 10108 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:24.008136 10120 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
I20260709 12:26:24.056428  5633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:24.057116  5633 raft_consensus.cc:606] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:24.057488  5633 raft_consensus.cc:614] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:26:24.004612  5602 leader_election.cc:336] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.971s (SENT)
W20260709 12:26:23.984987 10058 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.904s	user 0.011s	sys 0.005s
W20260709 12:26:24.003803  5709 rpcz_store.cc:269] 0709 12:26:22.574883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:22.599803 (+ 24920us) service_pool.cc:224] Handling call
0709 12:26:23.889544 (+1289741us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b5400066654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:23.889645 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2069281152}
I20260709 12:26:24.022384  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:24.062536  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:24.003974  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:24.007647  9844 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:24.017251  9751 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:24.018664 10104 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.357s	user 0.000s	sys 0.001s
W20260709 12:26:24.019136 10113 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.271s	user 0.000s	sys 0.001s
I20260709 12:26:23.864575  9957 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:23.864768  9945 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:26:24.024025 10112 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
W20260709 12:26:24.024020  9907 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.805s	user 0.012s	sys 0.002s
I20260709 12:26:24.080015  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:24.071384  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 184342us
W20260709 12:26:24.077766  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:24.068676  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 64093.8us
I20260709 12:26:24.116412  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:24.117472  5629 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:24.115398  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52939.2us
I20260709 12:26:24.056471 10000 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:24.068398  9901 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:24.075763  9899 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.881s	user 0.005s	sys 0.008s
I20260709 12:26:24.123481  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:24.129189  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:24.070896 10135 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:26:24.130170  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.128631  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:24.133782  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:24.134513  5709 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:24.134877  5602 leader_election.cc:336] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.166s (SENT)
I20260709 12:26:24.135283  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:24.130618  5702 pending_rounds.cc:85] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 5
I20260709 12:26:24.136541  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2012) took 1215 ms. Trace:
I20260709 12:26:24.136730  5702 pending_rounds.cc:107] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:26:24.137019  5702 raft_consensus.cc:2889] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:24.136919  5630 rpcz_store.cc:276] 0709 12:26:22.920799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:22.921031 (+   232us) service_pool.cc:224] Handling call
0709 12:26:24.136317 (+1215286us) spinlock_profiling.cc:243] Waited 482 ms on lock 0x7b54001ce8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:24.136488 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1108798336}
I20260709 12:26:24.127229  9829 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:24.139252  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:24.139613  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:24.140102  5707 raft_consensus.cc:2410] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 10 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 12:26:24.140417  5704 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 13 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:26:24.140615  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:24.146529 10048 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:24.129348  9957 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:24.037344 10090 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:24.153429  9962 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:24.114640  9388 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.157161  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2028) took 1063 ms. Trace:
I20260709 12:26:24.159337  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:24.122253  9907 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:24.126781 10119 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:24.075649 10063 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20260709 12:26:24.139377 10060 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:24.192399  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:24.184832  5634 rpcz_store.cc:276] 0709 12:26:23.093376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.288699 (+195323us) service_pool.cc:224] Handling call
0709 12:26:23.289119 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:23.289598 (+   479us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:23.289634 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:23.290186 (+   552us) log.cc:844] Serialized 26 byte log entry
0709 12:26:24.001993 (+711807us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:24.054321 (+ 52328us) log.cc:844] Serialized 14 byte log entry
0709 12:26:24.056686 (+  2365us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.056771 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:24.156880 (+100109us) raft_consensus.cc:1685] finished
0709 12:26:24.156908 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:24.157073 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45312,"prepare.queue_time_us":51864,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":8498,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":27635,"thread_start_us":711408,"threads_started":1,"wal-append.queue_time_us":790425}
I20260709 12:26:24.073418  9844 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:24.160550 10000 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:24.218026  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:24.138639 10135 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:24.135306 10058 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.060s
W20260709 12:26:24.232632  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 98332.9us
I20260709 12:26:24.289839  5634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:24.290742  5634 raft_consensus.cc:606] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:24.291209  5634 raft_consensus.cc:614] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:24.230865  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:24.233294  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 110487us
I20260709 12:26:24.237195  5629 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:24.200112 10126 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:26:24.322240  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 91503.5us
I20260709 12:26:24.324327  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:24.325186  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:24.326022  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 191153us
W20260709 12:26:24.327611  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 477812us
W20260709 12:26:24.327898  5602 leader_election.cc:336] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.202s (SENT)
I20260709 12:26:24.190232  9388 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:24.332091  5629 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:26:24.235386 10060 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:24.246410 10118 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:24.242908 10119 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:24.341531  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:24.341619  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 844) took 1248 ms. Trace:
I20260709 12:26:24.248415 10135 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:24.347072  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 845) took 1274 ms. Trace:
I20260709 12:26:24.347645  6277 rpcz_store.cc:276] 0709 12:26:23.073063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.350008 (+276945us) service_pool.cc:224] Handling call
0709 12:26:23.410877 (+ 60869us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:23.423450 (+ 12573us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:26:23.754325 (+330875us) log.cc:844] Serialized 14 byte log entry
0709 12:26:24.346613 (+592288us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:24.346691 (+    78us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:24.346756 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.346848 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:24.346995 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":578942,"prepare.queue_time_us":282511,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":25532,"thread_start_us":311024,"threads_started":2,"wal-append.queue_time_us":284169}
I20260709 12:26:24.353988  5705 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:26:24.323460 10126 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:24.355528  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.333746 10000 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:26:24.356691  5602 leader_election.cc:336] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.014s (SENT)
I20260709 12:26:24.361631  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:24.345494  9751 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:26:24.355640 10023 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.817s	user 0.009s	sys 0.005s
W20260709 12:26:24.371251  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 367324us
I20260709 12:26:24.346264  9829 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:24.381027  5629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:24.359560  9908 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:24.357049 10126 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:24.384274  6236 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:24.365478  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:24.382089  5629 raft_consensus.cc:606] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:24.356549 10108 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:24.382970  9285 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.647s	user 0.012s	sys 0.000s
W20260709 12:26:24.393111  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 62698.1us
I20260709 12:26:24.392644  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:24.394646  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62517.6us
I20260709 12:26:24.395673  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:24.378437  9899 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:26:24.350605 10060 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:24.399628  5526 leader_election.cc:336] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.663s (SENT)
I20260709 12:26:24.395922  9908 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:24.383425 10023 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:24.411630  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:24.422075  6272 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:24.404640 10115 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:26:24.422353  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.428498  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:24.429567  6277 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:24.432613 10058 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.433434 10058 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:24.434623  5560 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:26:24.435572 10127 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:24.436204 10134 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:24.405836 10043 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:24.440708 10143 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:24.441427  5629 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260709 12:26:24.441865  5629 pending_rounds.cc:85] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 9
I20260709 12:26:24.442101  5629 pending_rounds.cc:107] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:26:24.442476  5629 raft_consensus.cc:2889] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:24.445174 10139 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:26:24.440798 10109 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:24.450129 10023 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:24.452672 10127 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:24.454089 10127 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:24.460668  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.462373  5629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bf758b480ecf41619471236195bf1838"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:24.463393  5629 raft_consensus.cc:606] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:24.463919  5629 raft_consensus.cc:614] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:24.413491  9844 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:24.469219  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 74079.8us
W20260709 12:26:24.470088  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.478878 10115 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:24.480037 10115 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:24.483156 10116 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.646s	user 0.012s	sys 0.000s
W20260709 12:26:24.487326  5602 leader_election.cc:336] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.581s (SENT)
I20260709 12:26:24.488088  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:24.431728  9945 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:26:24.438903 10118 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:24.471165 10152 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:26:24.398310  9285 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:24.476559  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52750.1us
I20260709 12:26:24.499032  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:24.490772  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:24.438987  9908 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:24.494050  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 82324.8us
I20260709 12:26:24.475557 10058 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:24.423386 10106 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.070s
I20260709 12:26:24.504043  5628 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:24.506616  5629 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 12:26:24.506685  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 93168.1us
I20260709 12:26:24.511411  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:24.514430 10118 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.515921  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:24.517184 10023 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.517767  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:24.465546  9829 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:24.509290  9388 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:24.515328  5524 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:24.520645  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:24.520972  6276 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:26:24.504230  9751 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:24.524175  5630 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:24.524279  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:24.522428  9899 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.003s
I20260709 12:26:24.525748  9862 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.511641 10046 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:26:24.521677 10004 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.845s	user 0.008s	sys 0.005s
I20260709 12:26:24.523370  9844 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.528606 10130 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.013s
I20260709 12:26:24.544298  9388 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:24.545997  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:24.529596  9285 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:24.531037  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:24.553287  9844 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:24.537978  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:24.515448  9748 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.919s	user 0.009s	sys 0.005s
I20260709 12:26:24.539072  6275 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:24.542313  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:24.553704  5627 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:24.543007  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:24.563431  9702 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.011s
I20260709 12:26:24.575347  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2139) took 1296 ms. Trace:
I20260709 12:26:24.575511  9844 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:24.576285  9748 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:24.575862  5553 rpcz_store.cc:276] 0709 12:26:23.278948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.279127 (+   179us) service_pool.cc:224] Handling call
0709 12:26:23.279438 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:24.077571 (+798133us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:24.077599 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:24.077640 (+    41us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:24.077678 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.575087 (+497409us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b54003c41d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:24.575150 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:24.575280 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16350,"raft.queue_time_us":17012,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":39271,"spinlock_wait_cycles":1796975872}
I20260709 12:26:24.577852  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:24.580183  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:24.579869  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:24.539739 10048 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:24.598539  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 825) took 1512 ms. Trace:
I20260709 12:26:24.521764 10082 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 12:26:24.602216  6280 rpcz_store.cc:276] 0709 12:26:23.075723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.360045 (+284322us) service_pool.cc:224] Handling call
0709 12:26:23.361966 (+  1921us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:23.575321 (+213355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.584656 (+1009335us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b54002a8d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:24.587680 (+  3024us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":390399360}
I20260709 12:26:24.598603  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 786) took 1504 ms. Trace:
I20260709 12:26:24.600988  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:24.578910  9748 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:24.603366  5635 rpcz_store.cc:276] 0709 12:26:23.082245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.173821 (+ 91576us) service_pool.cc:224] Handling call
0709 12:26:23.265119 (+ 91298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:23.444920 (+179801us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.380253 (+935333us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b540018b854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:24.586076 (+205823us) spinlock_profiling.cc:243] Waited 91 ms on lock 0x7b540018b850. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:24.586223 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":522556544}
I20260709 12:26:24.607085  5627 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:24.546162 10127 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.533102 10023 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:24.608027  9388 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:24.588022 10004 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:24.614941 10127 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:24.643378  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63592.8us
I20260709 12:26:24.575527 10046 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:24.580379 10058 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:24.621430  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2305) took 1541 ms. Trace:
I20260709 12:26:24.641861  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:24.641966  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:24.580525  9751 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:26:24.611035  9844 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:26:24.653859  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.661898  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:24.546844  9862 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:24.523618 10118 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:24.647866  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:24.646884  9901 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.506s	user 0.009s	sys 0.004s
W20260709 12:26:24.674693 10136 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.282s	user 0.001s	sys 0.000s
I20260709 12:26:24.648550  9709 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.676440  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 828) took 1685 ms. Trace:
I20260709 12:26:24.676895  9709 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:24.677080  6273 rpcz_store.cc:276] 0709 12:26:22.991342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:22.991499 (+   157us) service_pool.cc:224] Handling call
0709 12:26:24.676139 (+1684640us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54004f9cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:24.676331 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2840444288}
I20260709 12:26:24.678776  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:24.679168  9709 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:24.679787  6273 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:24.681077  9862 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:24.682049  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:24.683250  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:24.684767  5627 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:24.641145  9748 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.584992  9908 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:24.661656  9844 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:24.650360  5636 rpcz_store.cc:276] 0709 12:26:23.080377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.080530 (+   153us) service_pool.cc:224] Handling call
0709 12:26:23.090876 (+ 10346us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:26:23.296049 (+205173us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:26:23.400875 (+104826us) log.cc:844] Serialized 14 byte log entry
0709 12:26:23.555332 (+154457us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:26:23.557277 (+  1945us) log.cc:844] Serialized 78 byte log entry
0709 12:26:23.557533 (+   256us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:24.037841 (+480308us) log.cc:844] Serialized 14 byte log entry
0709 12:26:24.055082 (+ 17241us) spinlock_profiling.cc:243] Waited 28.3 ms on lock 0x7b1800425bc0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:24.058291 (+  3209us) log.cc:844] Serialized 14 byte log entry
0709 12:26:24.058500 (+   209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.295369 (+236869us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54001dd8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:24.295426 (+    57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:24.482076 (+186650us) spinlock_profiling.cc:243] Waited 3.02 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:24.621183 (+139107us) raft_consensus.cc:1685] finished
0709 12:26:24.621208 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:24.621376 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58293,"prepare.queue_time_us":353050,"prepare.run_cpu_time_us":122,"prepare.run_wall_time_us":124,"raft.queue_time_us":26342,"raft.run_cpu_time_us":524,"raft.run_wall_time_us":23344,"spinlock_wait_cycles":359929216,"thread_start_us":569971,"threads_started":2,"wal-append.queue_time_us":320419}
I20260709 12:26:24.649574 10048 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:26:24.692629  9957 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.502s	user 0.009s	sys 0.005s
I20260709 12:26:24.695353  9748 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:24.719697  9846 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:24.721529 10144 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.333s	user 0.001s	sys 0.000s
I20260709 12:26:24.721215 10126 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:24.699296  9844 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:24.725342 10023 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:24.725462  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 728) took 1425 ms. Trace:
I20260709 12:26:24.725934 10135 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:24.726353  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:24.727264 10153 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:26:24.727372 10109 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:26:24.729380  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.729652  5708 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:26:24.728838  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:24.767382  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:24.768003  5631 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:24.769248  9285 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:24.769838  9285 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:24.770211  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 795) took 1408 ms. Trace:
I20260709 12:26:24.770572  5628 rpcz_store.cc:276] 0709 12:26:23.361404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.534035 (+172631us) service_pool.cc:224] Handling call
0709 12:26:23.534448 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:24.512913 (+978465us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.769996 (+257083us) spinlock_profiling.cc:243] Waited 969 ms on lock 0x7b540032c8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:24.770155 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2229376256}
I20260709 12:26:24.771313  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:24.771596 10143 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:24.773082  5630 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:24.736272  5556 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:24.773612  5630 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:24.774529  9846 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:24.774376  5630 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:24.774901 10126 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:24.774957  9846 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:24.344658  6271 rpcz_store.cc:276] 0709 12:26:23.072874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.317243 (+244369us) service_pool.cc:224] Handling call
0709 12:26:23.362825 (+ 45582us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:23.408569 (+ 45744us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:26:23.524418 (+115849us) log.cc:844] Serialized 14 byte log entry
0709 12:26:23.785791 (+261373us) log.cc:844] Serialized 14 byte log entry
0709 12:26:24.319313 (+533522us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b180039eb60. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:26:24.319540 (+   227us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:24.321084 (+  1544us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:24.321167 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.321274 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:24.321448 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31105,"prepare.queue_time_us":163333,"prepare.run_cpu_time_us":5913,"prepare.run_wall_time_us":70967,"raft.queue_time_us":1125937,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":11397,"spinlock_wait_cycles":13092992,"thread_start_us":222727,"threads_started":1,"wal-append.queue_time_us":258866}
I20260709 12:26:24.775558  9846 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:24.740396  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:26:24.741469  6277 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:24.744985  5561 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:26:24.744038 10106 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20260709 12:26:24.755044  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:24.779985  5635 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:24.780149  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:24.757248  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:24.760471  5710 rpcz_store.cc:276] 0709 12:26:23.301045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.317333 (+ 16288us) service_pool.cc:224] Handling call
0709 12:26:24.724938 (+1407605us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54001987d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:24.725275 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":563606400}
I20260709 12:26:24.773144 10083 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:24.790642 10083 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:24.790728 10071 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:26:24.791728 10083 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:24.793185  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:24.793736 10127 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:24.793823  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 833) took 1532 ms. Trace:
I20260709 12:26:24.794816 10151 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:26:24.752444 10004 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:24.796033 10150 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:24.796504 10150 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:24.797336 10150 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:24.772174  9285 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:24.798631 10150 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:24.801395 10118 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:24.801832  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:24.802259  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:24.803398  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:24.717598  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54851.1us
W20260709 12:26:24.804754  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 234309us
I20260709 12:26:24.805087  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:24.782377  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:24.763402 10046 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:24.785939 10060 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:24.779484 10023 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.788658 10135 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:24.789333  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:24.814944  5552 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:24.790148  5627 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
W20260709 12:26:24.792325  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.817529  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:24.794317  6279 rpcz_store.cc:276] 0709 12:26:23.261509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.767246 (+505737us) service_pool.cc:224] Handling call
0709 12:26:23.767677 (+   431us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:24.161325 (+393648us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:24.161372 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:24.196977 (+ 35605us) log.cc:844] Serialized 26 byte log entry
0709 12:26:24.315427 (+118450us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:24.315493 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.315597 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:24.720804 (+405207us) spinlock_profiling.cc:243] Waited 6.02 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:24.793496 (+ 72692us) raft_consensus.cc:1685] finished
0709 12:26:24.793517 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:24.793764 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17539,"raft.queue_time_us":559463,"raft.run_cpu_time_us":3004,"raft.run_wall_time_us":467128,"spinlock_wait_cycles":41496192,"thread_start_us":442512,"threads_started":2,"wal-append.queue_time_us":153725}
I20260709 12:26:24.806557  5635 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:26:24.781723  5630 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:24.809710  5706 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 7. Current term is 8. Ops: []
I20260709 12:26:24.810734  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:24.757117  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:24.812515  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:24.820623  5709 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:24.821591  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:24.820075  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:24.825757  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:24.829208  5710 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:26:24.830888  5628 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:26:24.834470  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:24.835399  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:24.843948  5706 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:26:24.846963  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:24.847615  6271 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:24.850746  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:24.851481  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:24.852305  5709 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:24.813120 10060 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:24.858234  5552 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:26:24.858682  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:24.844964 10126 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:26:24.861726  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 56709.9us
W20260709 12:26:24.862486  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.861726  5559 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:26:24.844029  9750 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:24.868436  5559 pending_rounds.cc:85] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 6
I20260709 12:26:24.856946 10055 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:24.870637  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:24.871665  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:24.874680  5706 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:24.886793  5630 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
W20260709 12:26:24.876377 10141 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.478s	user 0.001s	sys 0.001s
I20260709 12:26:24.876744  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:24.878226  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:24.891614  5552 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:24.879382  5559 pending_rounds.cc:107] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 6.7
W20260709 12:26:24.884225  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.887188  5630 pending_rounds.cc:85] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 8
I20260709 12:26:24.878473  5560 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:24.893409  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:24.892761  5559 raft_consensus.cc:2889] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:24.894272  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:24.892612  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:24.894265  5552 raft_consensus.cc:2410] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 12 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:26:24.896175  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:24.899432  5560 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:24.900527  5630 pending_rounds.cc:107] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 12:26:24.901049  5630 raft_consensus.cc:2889] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:24.903208  5561 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:24.854544 10156 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:24.844563 10023 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:24.854216  9910 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 12:26:24.854411 10096 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 12:26:24.923029  5630 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:26:24.863467 10082 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:26:24.863121 10088 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 12:26:24.930424  5630 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:24.935402  5630 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:24.902055  9285 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 12:26:24.934672  9093 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:26:24.933315  5602 leader_election.cc:336] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.624s (SENT)
I20260709 12:26:24.939687  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:24.940274  5702 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:24.941444  5630 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:24.919770  9750 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:24.939814  5706 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:24.946225  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:24.943459  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 81426.2us
W20260709 12:26:24.947665  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.947021  5561 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:24.954180  5630 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:24.960058  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:24.982151  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 89724.3us
I20260709 12:26:24.920197 10109 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260709 12:26:24.984721  5676 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:24.994171  5702 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:24.942561  9702 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:25.000502  5706 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:26:25.001629  9844 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:25.011611  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55749.8us
I20260709 12:26:25.024653  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:24.938808  9846 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:25.023475  9093 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:25.034291  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:25.036397  5630 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 12:26:25.037957  5630 pending_rounds.cc:85] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 5
I20260709 12:26:25.038416  5630 pending_rounds.cc:107] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:26:25.038726  5630 raft_consensus.cc:2889] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:25.037403  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:25.019923  9901 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:25.022043  9702 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:25.032301  9862 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:25.026651 10151 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:25.054169  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:25.054775  5706 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:25.040577 10108 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:24.955312 10060 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:25.061582 10162 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:26:25.069451  5706 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:25.039240  9751 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.019s
I20260709 12:26:25.056659  9901 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:25.076406  9846 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:25.078527  9844 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:25.084381  5631 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:25.084865  5706 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:26:25.039099 10127 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 12:26:25.048069  9702 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:25.022646  9750 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:25.090242  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:25.059446 10108 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:25.104480  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.082531  9901 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:25.059520 10151 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:25.107095  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:25.107851  5706 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:25.108692 10151 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:25.102097 10108 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:25.100678 10060 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:25.116891  9901 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:26:25.120333 10167 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
W20260709 12:26:25.127379  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 90142.5us
I20260709 12:26:25.128278 10023 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:25.165091  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:25.234867  5558 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:25.235656  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2143) took 1372 ms. Trace:
I20260709 12:26:25.236014  5558 rpcz_store.cc:276] 0709 12:26:23.863241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.879870 (+ 16629us) service_pool.cc:224] Handling call
0709 12:26:25.234507 (+1354637us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:25.234689 (+   182us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:25.235436 (+   747us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.235602 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8553728}
I20260709 12:26:25.238765  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 729) took 1929 ms. Trace:
W20260709 12:26:25.240367  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 101827us
I20260709 12:26:25.241297  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:25.241739  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:25.243459  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:25.244787  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:25.246856  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:25.239238  5703 rpcz_store.cc:276] 0709 12:26:23.304280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.304444 (+   164us) service_pool.cc:224] Handling call
0709 12:26:23.318417 (+ 13973us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:23.579643 (+261226us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.233899 (+1654256us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b5400197dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:25.234065 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":547740928}
I20260709 12:26:25.248924  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:25.253118  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:25.255381  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:25.256172 10126 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:25.256255  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:25.256695 10126 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:25.257100 10126 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:25.257984 10090 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:25.260835  9709 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:25.261309  9709 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:25.261705  9709 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:25.262849 10047 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:25.263376 10047 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:25.265574 10047 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:25.269438  9846 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:25.269897  9846 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:25.270232  9846 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:25.277577  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:25.284075  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.288239  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.291741  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:25.292383  5559 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:25.255621  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:25.294220  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:25.294878  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:25.295475  6279 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:25.298472  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:25.299760  6276 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 13: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:25.300568 10047 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:25.301017  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.301071 10047 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:25.301856 10114 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.561s	user 0.007s	sys 0.008s
I20260709 12:26:25.303380 10047 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:25.306157  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.308665  5555 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:25.310428 10071 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:25.310974  5555 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:26:25.311594  6277 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:25.311955  6277 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:25.288993 10150 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.313406  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.276939 10175 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:26:25.322461  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.334951  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.335963  5558 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:25.326095 10150 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:25.330433  6277 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:25.341439  6277 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:25.341781  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.343149 10055 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:25.323073 10071 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:25.344237  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.346747  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.345978 10071 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:25.361011 10108 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:25.363022  5557 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:25.367764  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:25.369172  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:25.384333  5707 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 11: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:25.369987 10009 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:25.371150  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2288) took 1243 ms. Trace:
I20260709 12:26:25.380551  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.380936  5703 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:25.381023  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.340040 10150 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:25.400864  5557 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:25.387590  6277 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:25.394220  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.391010  5705 rpcz_store.cc:276] 0709 12:26:24.127243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.127356 (+   113us) service_pool.cc:224] Handling call
0709 12:26:24.132994 (+  5638us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:24.354474 (+221480us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.370887 (+1016413us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b5400451fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:25.371072 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":507597184}
I20260709 12:26:25.392961  5559 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:25.403081  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:25.403108  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.403855  5559 raft_consensus.cc:2376] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10. Re-sending same reply.
I20260709 12:26:25.405149  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.406791  5559 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:25.407994  5561 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:25.408977  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:25.411322  9901 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:25.413297  9709 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:25.415952  9709 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:25.417013  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.417157  5561 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:25.417858  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:25.421952  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.425096  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 762) took 1562 ms. Trace:
I20260709 12:26:25.439770  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.425730 10047 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:25.428469  9702 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:25.429401  6279 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:25.439800  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.440932  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.445842  6278 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
W20260709 12:26:25.438025  9987 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 12 is higher.
I20260709 12:26:25.442394  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.443840  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:25.448819  5558 raft_consensus.cc:2376] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10. Re-sending same reply.
I20260709 12:26:25.449674  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.445914  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.450673  5559 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:25.451586  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.440049  5554 rpcz_store.cc:276] 0709 12:26:23.862921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.864935 (+  2014us) service_pool.cc:224] Handling call
0709 12:26:23.869360 (+  4425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:23.869759 (+   399us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:23.869799 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:23.882575 (+ 12776us) log.cc:844] Serialized 26 byte log entry
0709 12:26:25.367516 (+1484941us) spinlock_profiling.cc:243] Waited 3.6 ms on lock 0x7b180020fec0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:26:25.385809 (+ 18293us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:25.385859 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.385940 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:25.424561 (+ 38621us) raft_consensus.cc:1685] finished
0709 12:26:25.424586 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:25.424871 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20263,"raft.queue_time_us":94,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":8959,"spinlock_wait_cycles":8276736,"thread_start_us":18059,"threads_started":1,"wal-append.queue_time_us":1519342}
I20260709 12:26:25.447533  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.454494  5561 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:25.455148  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.452684  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.456221  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.391665 10125 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.056s
I20260709 12:26:25.457808  5552 raft_consensus.cc:2410] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:26:25.458066  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:25.459097  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:25.459887  5559 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:25.459167  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:25.460704  5558 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:26:25.460821  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.461391  5561 raft_consensus.cc:2410] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:26:25.462054  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:25.462801  5559 raft_consensus.cc:2376] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10. Re-sending same reply.
I20260709 12:26:25.464095  5559 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:26:25.442411 10047 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:25.466439  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.467777  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:25.471875  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 538546us
W20260709 12:26:25.472862  5602 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.811s (SENT)
I20260709 12:26:25.473124  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:25.477406  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:25.478473 10060 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:25.485371 10166 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:25.486492  9702 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:25.491549  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:25.492388  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:25.492820  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.493706  5601 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:25.494475 10143 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:25.469792 10068 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 12:26:25.495751  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:25.494508  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:25.497961  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:25.446713 10126 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:25.501477  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:25.503429  9958 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:26:25.505151  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:25.508559  5602 leader_election.cc:336] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 3.015s (SENT)
I20260709 12:26:25.470050  9932 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:26:25.516081  5631 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:26:25.499723  9901 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:25.502224 10090 raft_consensus.cc:2764] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election decision vote started in defunct term 12: won
W20260709 12:26:25.524369  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47013.5us
W20260709 12:26:25.531855 10182 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:26:25.510419 10096 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:25.511212 10060 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:25.513274 10166 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.589329  5705 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:26:25.593127  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:25.597904  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72516us
I20260709 12:26:25.593830  5707 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:25.576902  9829 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 12:26:25.606835  9759 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:25.609131  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2286) took 1484 ms. Trace:
I20260709 12:26:25.609547  5708 rpcz_store.cc:276] 0709 12:26:24.124834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.124976 (+   142us) service_pool.cc:224] Handling call
0709 12:26:24.729363 (+604387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:24.771142 (+ 41779us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.608951 (+837809us) spinlock_profiling.cc:243] Waited 593 ms on lock 0x7b54001987d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:25.609077 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1364154752}
I20260709 12:26:25.611398  5708 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:25.619068 10053 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:25.576766 10166 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:25.628579 10166 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:25.590934 10096 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.629421 10096 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:25.631600 10096 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:25.635844  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.637251 10126 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:25.638427  5707 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 8. Current term is 9. Ops: []
I20260709 12:26:25.641623  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:25.645233  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.646661  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.546383  9846 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:25.519129 10047 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:25.603680 10108 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:25.751601  5703 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:25.755050  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:25.758543  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:25.760131  5553 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:25.760286  5558 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:25.762257  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:25.762945  5559 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:25.765084  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:25.765785  5557 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:25.769955  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:25.770762  5555 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 13: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:25.771554  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:25.773844  9285 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.775031  9285 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:25.778445  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.780207 10068 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:25.780460  6272 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 831) took 2752 ms (client timeout 2226 ms). Trace:
I20260709 12:26:25.780818 10068 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:25.780951  6272 rpcz_store.cc:269] 0709 12:26:23.028457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.272215 (+243758us) service_pool.cc:224] Handling call
0709 12:26:25.780233 (+2508018us) spinlock_profiling.cc:243] Waited 863 ms on lock 0x7b5400055854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:25.780346 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1984815488}
I20260709 12:26:25.782889 10068 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:25.783246  9748 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:25.784180  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2308) took 1716 ms. Trace:
I20260709 12:26:25.784548  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:25.784590  5632 rpcz_store.cc:276] 0709 12:26:24.067618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.068823 (+  1205us) service_pool.cc:224] Handling call
0709 12:26:24.076816 (+  7993us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:24.221983 (+145167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:24.222016 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:24.222055 (+    39us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:24.222099 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.783974 (+1561875us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b5400438d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:25.784017 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:25.784129 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":333319552}
I20260709 12:26:25.785820  9957 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:25.786310  9957 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:25.787129  9957 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:25.787858  6273 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:26:25.792562  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:25.793229  5707 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:25.799438  5558 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:25.802351  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:25.802269  5702 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:25.803040  5560 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:25.803876  5559 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
W20260709 12:26:25.805123  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 275336us
I20260709 12:26:25.802241  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:25.805924  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:25.810022  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:25.811148  5702 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:25.813323  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:25.814119  5708 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 8: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:25.815448  5710 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:25.818193  5557 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:25.820649  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:25.821862  9987 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:25.825479  5708 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:26:25.801616 10166 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.827907  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:25.839761  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:25.841032  5553 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:25.843230  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:25.844923  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:25.852480  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:25.854315  9093 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:25.859131  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:25.848055  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:25.861687  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:25.862797  9285 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:25.864104  9285 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:25.876784  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:25.845134 10150 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:25.856835 10043 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:26:25.861188  9759 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:25.896708  9899 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.897306  9899 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:25.877475 10108 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:26:25.900102 10188 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260709 12:26:25.845412 10187 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:25.861012 10175 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:25.900038 10108 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:25.918123 10108 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:25.880889  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:25.922538  6272 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:25.906975  5560 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:25.904419 10175 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:25.935192  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.847972 10166 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:25.893566  9748 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:25.886226 10150 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:26:25.974434  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.888998 10192 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:25.932909  9759 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:25.963837 10134 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:25.977047 10150 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:26.005952  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 144330us
I20260709 12:26:26.009272  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:25.977793  9945 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 12:26:26.000994  9748 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:26.014308  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:26.015002  5560 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:26.029955  5560 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:26.031941  9285 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:26.015962  9957 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:26.004395 10134 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:26.089959 10196 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:26:26.014313 10175 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:26.093775 10173 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:26:26.117645  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:26.118292  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:26.133226 10203 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.113827  9748 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:26.133857 10203 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.135862 10203 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:26.039880  9759 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:26.137534  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 278529us
I20260709 12:26:26.142972  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:26.167547  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:26.168100  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.125994 10060 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:26.166314  6272 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:26.171057  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:26:26.169056  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.176237  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:26:26.177207  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.143508 10166 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:26.189417  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:26.190485 10091 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:26.190948 10165 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:26:26.194156 10134 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:26.195415  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:26.199594 10165 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:26.203351 10165 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.205168 10165 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:26:26.208556  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.207903  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:26.209757  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 91559.9us
I20260709 12:26:26.183435  9945 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:26.198680 10043 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:26:26.200016 10192 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:26.180939 10060 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:26.225237  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:26.215799 10194 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
I20260709 12:26:26.214558  9945 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:26.216125 10166 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:26.215338 10218 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:26.237769 10218 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:26.239025  9866 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:26.238739 10218 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:26.215723 10221 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
W20260709 12:26:26.236944 10200 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:26:26.250265 10060 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:26.257319 10009 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.259018  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:26.260293  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:26.267817  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:26.268992  5636 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:26:26.258337 10009 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.230656  9945 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:26.274387  9844 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:26.278419  9844 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:26.278906 10140 raft_consensus.cc:993] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:26:26.295753  6274 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:26:26.285007  9844 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:26.297195  9957 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:26:26.297894 10216 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.285015 10140 raft_consensus.cc:1081] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Signalling peer f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:26:26.274613 10060 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:26.263290 10064 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 12:26:26.276618 10130 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:26.314973 10064 consensus_queue.cc:1243] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:26:26.316907 10091 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:26:26.316913  5661 heartbeater.cc:646] Failed to heartbeat to 127.4.65.62:38921 (2 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.4.65.62:38921 timed out after 15.000s (SENT)
W20260709 12:26:26.338922 10189 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.564s	user 0.003s	sys 0.000s
W20260709 12:26:26.340320  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:26.340427 10226 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:26:26.343966  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 805) took 1400 ms. Trace:
I20260709 12:26:26.344377  5628 rpcz_store.cc:276] 0709 12:26:24.943730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.943981 (+   251us) service_pool.cc:224] Handling call
0709 12:26:24.944458 (+   477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:25.473751 (+529293us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:25.473786 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:25.504464 (+ 30678us) log.cc:844] Serialized 26 byte log entry
0709 12:26:25.547948 (+ 43484us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:25.547988 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.548060 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:26.338291 (+790231us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:26.343667 (+  5376us) raft_consensus.cc:1685] finished
0709 12:26:26.343691 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:26.343905 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":528828,"raft.queue_time_us":541701,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":130,"spinlock_wait_cycles":8100224,"thread_start_us":25493,"threads_started":1,"wal-append.queue_time_us":262219}
I20260709 12:26:26.345329  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 835) took 1115 ms. Trace:
I20260709 12:26:26.345852  6278 rpcz_store.cc:276] 0709 12:26:25.230256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.230416 (+   160us) service_pool.cc:224] Handling call
0709 12:26:26.345119 (+1114703us) spinlock_profiling.cc:243] Waited 30.9 ms on lock 0x7b5400591ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:26.345255 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":71052544}
I20260709 12:26:26.348922  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:26.351244  5560 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260709 12:26:26.351634  5560 pending_rounds.cc:85] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 9
I20260709 12:26:26.351838  5560 pending_rounds.cc:107] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:26:26.352041  5560 raft_consensus.cc:2889] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:26.354384 10204 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:26:26.354638 10140 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:26.355201 10140 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:26.356112 10140 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.332216  5702 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:38468
I20260709 12:26:26.361152  5702 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:26.361563  5702 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:26.361824  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:26.364248  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 78856.9us
I20260709 12:26:26.366175  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:26.376945  9945 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.336505 10060 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:26.369086 10166 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:26.366247 10106 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.016s
I20260709 12:26:26.365916 10191 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:26.365583 10091 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.369261 10221 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:26.384023  9285 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:26:26.384840  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:26.314101 10216 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.370455 10192 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:26.369505 10053 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:26.379993  5702 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.393250  9285 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:26.393697  9285 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:26.394418  9285 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:26.395577  9285 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:26.452847  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.520112  5702 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:26.422560 10096 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:26.634037 10175 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.475430 10230 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:26.636921  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.422811 10221 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:26.525959 10228 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260709 12:26:26.524109  9759 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.423249 10053 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:26.645143 10197 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
I20260709 12:26:26.400982 10091 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.390425 10166 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.604039  9932 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:26.425207 10187 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.233s
I20260709 12:26:26.714365 10169 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:26.705502 10202 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260709 12:26:26.735972  9945 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:26.736970  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:26.784794  6271 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:26.748553  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:26.786706  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.654750 10175 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.703871 10230 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.757131 10221 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:26.761125 10216 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:26.401611 10191 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.763283  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2311) took 1999 ms. Trace:
I20260709 12:26:26.797434  5627 rpcz_store.cc:276] 0709 12:26:24.764045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.768716 (+  4671us) service_pool.cc:224] Handling call
0709 12:26:24.769003 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:24.815875 (+ 46872us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.763047 (+1947172us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b540008bbd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:26.763224 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48158080}
I20260709 12:26:26.638440 10096 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:26:26.731943 10150 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.719s	user 0.013s	sys 0.002s
I20260709 12:26:26.860863  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:26.860594 10150 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:26.812773  5552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2061) took 1474 ms (client timeout 1624 ms). Trace:
W20260709 12:26:26.862452  5552 rpcz_store.cc:269] 0709 12:26:25.338208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.341507 (+  3299us) service_pool.cc:224] Handling call
0709 12:26:26.812567 (+1471060us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54003c46d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:26.812709 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":264618240}
I20260709 12:26:26.815160  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 852) took 1997 ms. Trace:
W20260709 12:26:26.816996  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.863922  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2304) took 1285 ms. Trace:
I20260709 12:26:26.864283  5703 rpcz_store.cc:276] 0709 12:26:25.578309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.578912 (+   603us) service_pool.cc:224] Handling call
0709 12:26:25.579206 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:25.760838 (+181632us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.863681 (+1102843us) spinlock_profiling.cc:243] Waited 85.7 ms on lock 0x7b5400323a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:26.863868 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":197183232}
I20260709 12:26:26.866037  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2312) took 2101 ms. Trace:
I20260709 12:26:26.818382 10053 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:26.819931 10091 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:26.787276  9932 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:26.821719 10060 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:26.791278 10169 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.830194  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:26.836520 10047 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:26.838944 10230 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:26.843227 10191 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:26.846935  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.852458 10096 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:26.731315  9759 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:26.854588 10166 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:26.856737  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:26.857359 10009 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:26.857811 10192 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:26:26.862047  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:26.804157 10175 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:26.863590  6273 rpcz_store.cc:276] 0709 12:26:24.817878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.819449 (+  1571us) service_pool.cc:224] Handling call
0709 12:26:25.787517 (+968068us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:25.788203 (+   686us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.814960 (+1026757us) spinlock_profiling.cc:243] Waited 941 ms on lock 0x7b5400055850. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:26.815101 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2164874112}
I20260709 12:26:26.866478  5629 rpcz_store.cc:276] 0709 12:26:24.764407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.768903 (+  4496us) service_pool.cc:224] Handling call
0709 12:26:24.769270 (+   367us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:24.769705 (+   435us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:24.769733 (+    28us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:26:24.770541 (+   808us) log.cc:844] Serialized 78 byte log entry
0709 12:26:24.882487 (+111946us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:26:24.882545 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.882935 (+   390us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:26.862571 (+1979636us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b240029c0a8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff740b9c4 00007f4ff7409acb 00007f4ff75c4dc9 00007f4ff75c4a55 00007f4ff75c3fb0 00007f4ffa23a488 00007f4ffa23abe8 00007f4ffa212398 00007f4ffa20d76a 00007f4ffa20cbb6
0709 12:26:26.865740 (+  3169us) raft_consensus.cc:1685] finished
0709 12:26:26.865761 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:26.865975 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":690,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"spinlock_wait_cycles":6924544,"thread_start_us":111382,"threads_started":1,"wal-append.queue_time_us":157322}
I20260709 12:26:26.866078 10150 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:26.892434  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:26.935241  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:26.936300  5636 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:26:26.895747  9932 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:26.899365  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 358401us
I20260709 12:26:26.900985  6271 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:26.902853  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:26.944031  6278 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:26.915247  5635 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
W20260709 12:26:26.946206  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.641s (SENT) [suppressed 31 similar messages]
I20260709 12:26:26.916810  5634 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:26.918040  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:26.947551  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:26.948400  5631 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:26:26.918972  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:26.949065  5633 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:26.921679  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:26.923776  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:26.950630  6276 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:26.925559  5627 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:26.931466  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:26.927769  6272 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:26.852159 10165 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.646s	user 0.010s	sys 0.001s
I20260709 12:26:26.932297  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:26.932921  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:26.935928  6279 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:26.957585  5630 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:26.898700  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:26.937218  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:26.957942  5630 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:26.958384  5632 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:26.942333  9751 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:26.959187  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:26.959549  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:26.960088  6275 raft_consensus.cc:2410] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:26:26.921909 10023 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.964325  5636 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:26.944934 10217 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:26.968060  9751 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:26.975610 10217 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:26.977131  9751 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:26.977679 10217 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:26.978435  9750 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.978911  9750 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.980808  5630 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.980121  6272 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:26.982052  5630 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:26.977679  9751 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:26.958911 10118 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 12:26:26.985049  9751 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:26.985574  9751 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:26.983886  5632 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:26.971899 10060 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:26.990069 10108 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.990674 10108 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:26.968743  9932 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:26.974670 10023 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.968423 10214 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:26.958658 10175 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:26.998039 10175 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:26.974951  5601 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:27.005870  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:27.016129  6276 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:26.966753  5524 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:26.934607 10115 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.933607 10191 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:26.973201  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:27.022433  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.024s	sys 0.056s
W20260709 12:26:27.026691  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.025s	sys 0.056s
W20260709 12:26:27.026988  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 627290us
I20260709 12:26:27.031306  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:27.021510  5634 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:27.037498  6275 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:27.041361  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:27.021353 10091 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:27.034839 10009 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.017s
I20260709 12:26:27.047570  9829 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.005s
I20260709 12:26:27.052412  9958 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:27.052687 10175 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:27.047206 10088 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:27.065533  9759 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:27.065896  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 837) took 2558 ms. Trace:
I20260709 12:26:27.022425 10115 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.021090  9945 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:27.066484 10151 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.036780 10165 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 12:26:27.086221  9932 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:27.033685 10191 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.075790  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.109082  9750 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:27.081857  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.020464 10166 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:27.092801  5630 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:27.130654 10108 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:27.103731  6280 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:27.131249  6280 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:27.132508 10115 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:27.132184  6280 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.133764  6280 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:27.129511 10023 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:27.126634  9945 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:27.136062  9945 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.137923 10096 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.104885 10169 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:27.140767 10083 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:27.124308 10151 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.107335  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.140139 10096 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:27.110458  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 143812us
W20260709 12:26:27.111215  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:27.111444  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.112339  6274 rpcz_store.cc:276] 0709 12:26:24.507141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.507270 (+   129us) service_pool.cc:224] Handling call
0709 12:26:24.507597 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:26.296172 (+1788575us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.065630 (+769458us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b54002e3954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:27.065824 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4106921216}
I20260709 12:26:27.115191  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:27.086632 10091 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:27.164691  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:27.098259 10090 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:27.069301  9846 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:27.125311  5555 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:27.079219  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 747) took 1270 ms. Trace:
I20260709 12:26:27.173162  5707 rpcz_store.cc:276] 0709 12:26:25.793560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.793881 (+   321us) service_pool.cc:224] Handling call
0709 12:26:25.794129 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:25.794259 (+   130us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:26:26.003952 (+209693us) log.cc:844] Serialized 14 byte log entry
0709 12:26:26.009724 (+  5772us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.063836 (+1054112us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b18003b9080. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287
0709 12:26:27.063969 (+   133us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:27.064033 (+    64us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:27.064088 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.064173 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.064329 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32972,"prepare.queue_time_us":125648,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":996,"raft.queue_time_us":86862,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":12478,"spinlock_wait_cycles":2806656,"thread_start_us":223486,"threads_started":2}
I20260709 12:26:27.174288  9751 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.109426 10088 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.176594 10088 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:27.108166  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.180852  5554 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:27.144992 10166 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.187582  9958 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:26:27.188107  9958 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:27.189500 10166 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:27.190111 10166 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:27.189345  9958 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.192092  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:27.192919  6272 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:26:27.193778  6276 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:26:27.194384 10230 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.194835 10230 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.195595 10103 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.195688  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:27.196189 10103 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.197006 10230 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:27.197868  6274 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:27.198233 10103 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:27.198622  6279 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:27.192914  6275 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:27.199566  5632 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:27.200279  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:27.200750 10191 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:27.202203  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:27.202724  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:27.161533  9748 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.203437  5630 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:27.203394  9748 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.204066  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:27.204489  9844 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.204710  6275 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:27.204990  9844 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.205597  9748 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:27.169246 10091 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:27.206980  9844 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:27.207319  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:27.207832  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:27.208829  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:27.208918  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:27.210253  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.210456  9958 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:27.211298 10088 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.213698  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:27.215371 10151 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:27.217404  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.217456  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.218264  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.218750 10151 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:27.218854  5561 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:27.219038 10060 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:27.219233 10151 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.219607 10151 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:27.219591 10060 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.200788  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:27.222141  6280 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:27.222525  5633 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:27.222793  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 847) took 1936 ms. Trace:
W20260709 12:26:27.223920 10152 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
W20260709 12:26:27.225628  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 266143us
I20260709 12:26:27.226429  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:27.228765  5675 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:27.227026  5633 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:27.232632  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:27.235092  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:27.223163  6280 rpcz_store.cc:276] 0709 12:26:25.285960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.290585 (+  4625us) service_pool.cc:224] Handling call
0709 12:26:25.290899 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:27.103583 (+1812684us) spinlock_profiling.cc:243] Waited 947 ms on lock 0x7b5400290ca0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa6741c0 00007f4ffa675231 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c
0709 12:26:27.222518 (+118935us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.222743 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2177909632}
I20260709 12:26:27.238976  9751 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:27.169165 10090 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:27.242197  5554 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:26:27.225682 10213 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.006s	user 0.009s	sys 0.003s
W20260709 12:26:27.248507  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 83893.1us
I20260709 12:26:27.253901  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.221359 10068 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.255754  5561 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260709 12:26:27.256191  5561 pending_rounds.cc:85] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 8
I20260709 12:26:27.256423  5561 pending_rounds.cc:107] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 12:26:27.256644  5561 raft_consensus.cc:2889] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:27.257270  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:27.258608  5557 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:27.259930  5553 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:26:27.261173  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.247287  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:27.265272  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.272324  5558 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: [10.7-10.7]
W20260709 12:26:27.250425  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 205201us
I20260709 12:26:27.241997 10096 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:27.244467 10130 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:27.283957  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.278332  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:27.242211 10216 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:27.289666  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.286980  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:27.249760  9846 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:27.283063  9751 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:26:27.249121 10142 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 12:26:27.308660  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:27.297961  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54928.1us
I20260709 12:26:27.287815 10130 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.321434  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:27.304069  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.304939  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:27.296121  9901 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:26:27.279561 10096 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:27.305711  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.331439  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.332908  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.313936  5559 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:27.334620  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:27.317710  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:27.336747  5703 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:27.318323  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.319689  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:27.338462  5702 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:27.302614  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.289749 10242 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:27.343168 10060 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:27.312846 10090 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.325006 10130 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:27.280783 10068 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.354460  5709 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:27.359541  5559 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:26:27.357893  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.357671  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.362661  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:27.383994  5706 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:27.298621  9846 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:27.366605  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:27.362718  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.370354  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:27.371114  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:27.372990  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.384449  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.386591  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.396198  5708 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:27.396787  5559 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:27.404314  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:27.404953  5706 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:27.411782  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.356096 10096 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:27.413810  5560 raft_consensus.cc:2410] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:26:27.414700  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:27.410497  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:27.356051 10242 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.419654  5701 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:26:27.306439 10165 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20260709 12:26:27.398377  5561 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:26:27.418216  5702 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:27.344321 10187 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:26:27.418994  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 125303us
I20260709 12:26:27.355579 10088 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:27.434015  5703 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:27.380496  9748 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:27.425694 10242 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:27.420017  9759 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:27.435101 10187 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:27.440769 10165 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:27.513568 10165 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:27.440511 10175 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:26:27.442538  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:27.515738  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 73270.9us
I20260709 12:26:27.389101 10060 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.449847 10068 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:27.450639  9846 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.452776  9899 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:27.388253 10237 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.455965 10023 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:27.420688 10191 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:27.437520 10189 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
W20260709 12:26:27.456278  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.458885  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:27.468669  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:27.458202  5702 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:27.482167  5709 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:26:27.482326  5708 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:27.483569  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.492846  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:27.497174  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 177639us
W20260709 12:26:27.499269  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.460625 10091 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.458801 10151 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:27.508675  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.504449  9901 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:27.516710  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:27.579298  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62706.2us
I20260709 12:26:27.522029  5553 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:26:27.580273  5558 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:26:27.527331  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.580879  5558 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:27.555361  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:27.581729  5558 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.582643  5558 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:27.523131 10043 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:27.558393  5561 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:27.580289  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:27.556245  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 98119.6us
I20260709 12:26:27.585104  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:27.585492  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:27.549839 10237 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.508816  9748 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.590714  9748 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:27.591331  9748 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:26:27.514681 10165 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.593256 10023 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:26:27.576869 10191 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.594707 10023 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:26:27.573346 10108 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:27.517478  9759 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.593580 10165 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:27.583575  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:27.604103  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:27.552309  9932 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:27.605846  5636 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:27.605046 10047 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:27.578429 10009 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:27.598134 10244 consensus_queue.cc:1175] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cea66aa15be3423880843974acdbf50f" responder_term: 11 status { last_received { term: 9 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:26:27.601776  9958 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.608481 10169 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.609529 10255 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:26:27.612002 10243 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.283s	user 0.001s	sys 0.001s
I20260709 12:26:27.595860 10191 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:27.612810  9866 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:27.612002  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.588135 10237 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:27.615744  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:27.526592 10060 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.617982 10178 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.616691 10047 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.598109 10088 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:27.598186 10221 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 12:26:27.599968 10043 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.614596  9932 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:27.618340 10071 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:27.653544  5554 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:27.655418  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:27.640014  5558 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:26:27.640575  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 238193us
I20260709 12:26:27.619958 10143 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:27.636005 10175 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:27.657277 10175 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.659274  5599 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:27.660225 10143 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:27.660631 10143 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:27.660998 10143 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:27.661340 10175 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:27.661506  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.669153  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:27.671427  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:27.642223 10047 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:27.624668 10009 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.627202  9958 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.627389 10169 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.652464 10043 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:27.650065  9987 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:27.640678 10191 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 12:26:27.669155 10143 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.621472 10187 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:27.701133  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 115698us
I20260709 12:26:27.673345 10237 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:27.678515  9945 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:27.671428 10068 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.045s
I20260709 12:26:27.714435  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:27.711024  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:27.719254  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:27.685461 10088 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 12:26:27.723094  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:27.724156  5702 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
W20260709 12:26:27.722893  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 107236us
I20260709 12:26:27.727191  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:27.729113  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:27.708861 10118 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:27.730393  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:27.731105  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 128259us
I20260709 12:26:27.731753  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:27.732805  5703 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:26:27.733505  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 750) took 1615 ms. Trace:
I20260709 12:26:27.738749  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:27.739382  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:27.740048  5710 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:27.739084  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:27.733934  5704 rpcz_store.cc:276] 0709 12:26:26.117951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.118282 (+   331us) service_pool.cc:224] Handling call
0709 12:26:26.130432 (+ 12150us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:26.130992 (+   560us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.131033 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:26.140655 (+  9622us) log.cc:844] Serialized 52 byte log entry
0709 12:26:27.225529 (+1084874us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b180027a660. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:26:27.411876 (+186347us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:27.669837 (+257961us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.687502 (+ 17665us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.708661 (+ 21159us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.713214 (+  4553us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.729778 (+ 16564us) raft_consensus.cc:1685] finished
0709 12:26:27.729805 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.733453 (+  3648us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28391,"prepare.queue_time_us":258079,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":98,"raft.queue_time_us":102991,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":24197,"spinlock_wait_cycles":15379840,"thread_start_us":372791,"threads_started":2,"wal-append.queue_time_us":1315324}
W20260709 12:26:27.741818  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.719393 10255 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:27.720105 10009 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:27.743590  6280 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:26:27.725157 10115 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:27.723675  9945 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:27.747720  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 92391.8us
I20260709 12:26:27.759436  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:27.722458 10237 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:27.761484 10237 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.642432 10178 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.765486  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:27.765697  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:27.766461  5628 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:27.731843  9846 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:27.770193  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:27.771672  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:27.733702 10221 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:27.748525 10255 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.742779 10091 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:27.744253  9866 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:27.771013  6275 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:26:27.781107 10221 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.781988  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:27.744916 10023 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 12:26:27.747030 10108 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:27.752806 10265 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:27.743494 10118 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.743680  9987 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:26:27.795248  6274 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:27.752451 10115 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.795107  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:27.794416  9866 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:27.785071 10255 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:27.781116 10259 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:26:27.800765 10091 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:26:27.800892 10216 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.509s	user 0.010s	sys 0.000s
I20260709 12:26:27.822659 10088 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:26:27.795029 10246 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
W20260709 12:26:27.804842  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 66170.6us
I20260709 12:26:27.728179  9932 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:27.807613  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.880754  9932 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:27.880146  9866 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.883157 10047 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.828755 10242 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.822172  9759 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:27.821872 10265 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:27.801819 10257 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:26:27.828027 10115 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:27.885157  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.831163 10060 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:27.830986 10262 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:27.826084 10254 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.074s
I20260709 12:26:27.876215 10216 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:26:27.902551  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 130714us
I20260709 12:26:27.872676 10088 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:27.907265  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 164125us
I20260709 12:26:27.916143  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:27.898967  9958 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:27.875226  9844 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:26:27.917847  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:27.901944 10130 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:27.884454 10269 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.027s
I20260709 12:26:27.907088  9846 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:27.911103 10143 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:27.909590 10169 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:27.927487  5555 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:27.924285  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:27.941174  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:27.942631 10265 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.878558  9945 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:27.956390  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:27.955348  5629 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:26:27.917800 10088 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:27.912873 10266 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:27.967629  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:27.968685 10142 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:27.969463 10142 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.969947 10142 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:27.991286  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.000638  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:28.002010  5709 raft_consensus.cc:2368] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11.
I20260709 12:26:28.002421  5707 raft_consensus.cc:2410] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:26:27.960263 10216 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:28.011550  5707 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:26:27.962661 10253 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:27.921823 10218 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 12:26:28.015700  5707 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:27.922315 10270 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.090s
I20260709 12:26:28.057579  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:27.957738 10249 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.073s
I20260709 12:26:28.007778 10265 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:28.026584 10047 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:27.957500 10268 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.109s
I20260709 12:26:28.011384 10216 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:28.092025  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 196691us
W20260709 12:26:28.065102 10267 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
I20260709 12:26:28.094636  5709 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:26:28.025297  5707 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:26:28.100412  5706 raft_consensus.cc:1217] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Deduplicated request from leader. Original: 8.6->[9.7-9.7]   Dedup: 9.7->[]
I20260709 12:26:28.101183  5706 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 12:26:28.062897 10221 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:27.979059 10266 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.073324 10216 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:28.143174  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 404127us
I20260709 12:26:28.143639  5707 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: [10.6-10.6]
W20260709 12:26:28.142593  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:28.142987  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.112318  5708 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:28.146924  5709 raft_consensus.cc:1217] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Deduplicated request from leader. Original: 10.9->[11.10-11.10]   Dedup: 11.10->[]
I20260709 12:26:28.147400  5708 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:26:28.147552  5709 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:26:28.144187  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:28.142023  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:28.153049  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 394899us
I20260709 12:26:28.152643  5708 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:28.109571 10253 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:28.158435  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:28.161031 10274 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.203s	user 0.000s	sys 0.001s
I20260709 12:26:28.159284  5708 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:28.248919  5709 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:28.254768  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:28.260854 10096 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.823s	user 0.009s	sys 0.005s
W20260709 12:26:28.261546  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 151493us
W20260709 12:26:28.262387  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:28.267028  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.263233  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:28.072443  9846 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.249702 10242 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:28.263701 10096 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.273087  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:28.150319 10279 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.156s
I20260709 12:26:28.284922 10293 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.289335  5701 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:28.298704 10096 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:28.253222 10253 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.307266  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:28.322037  5708 raft_consensus.cc:2368] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11.
W20260709 12:26:28.316674  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 43675.5us
W20260709 12:26:28.307112 10165 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.698s	user 0.009s	sys 0.002s
I20260709 12:26:28.331758  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:28.326119 10264 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.527s	user 0.004s	sys 0.000s
I20260709 12:26:28.308631 10130 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:28.337626  9748 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:26:28.342211  5707 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:28.344895  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.346276  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.347244  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:28.347600  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.348023  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.349491  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.349959 10175 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:28.350875  5701 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:26:28.351840 10175 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:28.355494  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.322556 10242 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:28.338173 10253 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:28.353427 10175 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:28.374753  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.375590 10090 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:28.377151 10115 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.379261 10115 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:28.380491 10276 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.222s	user 0.002s	sys 0.000s
I20260709 12:26:28.380648 10266 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:28.381073 10271 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
I20260709 12:26:28.381145  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 824) took 1175 ms. Trace:
I20260709 12:26:28.381605  5636 rpcz_store.cc:276] 0709 12:26:27.205612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.207595 (+  1983us) service_pool.cc:224] Handling call
0709 12:26:28.380924 (+1173329us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b5400439c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:28.381089 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":913448192}
W20260709 12:26:28.382578 10284 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
W20260709 12:26:28.384125 10285 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:26:28.385985 10088 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.300297 10142 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.392163 10142 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:28.327713 10293 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:28.394711 10043 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.685s	user 0.011s	sys 0.000s
I20260709 12:26:28.395612 10293 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:28.395252 10043 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.396762  5708 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:28.393872  5630 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:28.399896  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:28.355057 10218 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:28.406059  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:28.392253  9932 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.511s	user 0.009s	sys 0.000s
I20260709 12:26:28.417940  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.405689 10175 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:28.436986  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.438233  5701 raft_consensus.cc:2410] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:26:28.445518 10151 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:28.428821 10088 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:28.461884 10187 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:28.456894 10255 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.585s	user 0.009s	sys 0.004s
I20260709 12:26:28.485253  5558 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:26:28.495549  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:28.501641 10245 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.098s	user 0.003s	sys 0.000s
I20260709 12:26:28.503234  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 784) took 1429 ms. Trace:
I20260709 12:26:28.503661  5552 rpcz_store.cc:276] 0709 12:26:27.074205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.075421 (+  1216us) service_pool.cc:224] Handling call
0709 12:26:27.075744 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:27.093125 (+ 17381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:27.093165 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:27.093844 (+   679us) log.cc:844] Serialized 26 byte log entry
0709 12:26:27.246787 (+152943us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:27.246837 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.246920 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:28.500887 (+1253967us) spinlock_profiling.cc:243] Waited 9.07 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:28.502940 (+  2053us) raft_consensus.cc:1685] finished
0709 12:26:28.502961 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.503173 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1215,"raft.queue_time_us":11020,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":15802,"spinlock_wait_cycles":20872448,"thread_start_us":129492,"threads_started":1,"wal-append.queue_time_us":184468}
I20260709 12:26:28.505935 10043 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:28.508798  9751 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:28.509230  9751 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:28.514029  9751 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.515458  9751 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:28.518296 10118 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:28.519160  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:28.468889 10151 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.526300  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:28.521346 10230 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.007s
I20260709 12:26:28.527845  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:28.531669 10047 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.532263 10047 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.476799 10255 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.542122  5558 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 13: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:28.546151 10218 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:28.546904 10218 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:28.549642 10273 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:28.550243 10273 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:28.551378 10273 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.556434 10151 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.558880  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2102) took 1290 ms. Trace:
I20260709 12:26:28.584199  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:28.560897  5635 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:28.585858  6276 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 12:26:28.521983 10296 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:26:28.564810 10255 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:28.576951  6274 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:28.580256  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:28.587046  6275 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:28.587050  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:28.584623  5557 rpcz_store.cc:276] 0709 12:26:27.268269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.269690 (+  1421us) service_pool.cc:224] Handling call
0709 12:26:27.270126 (+   436us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:27.380324 (+110198us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:27.380373 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:27.403937 (+ 23564us) log.cc:844] Serialized 52 byte log entry
0709 12:26:27.705657 (+301720us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:27.912116 (+206459us) log.cc:844] Serialized 14 byte log entry
0709 12:26:28.470220 (+558104us) spinlock_profiling.cc:243] Waited 8.5 ms on lock 0x7b18003b7b20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ffe54658f 00007f4ff7585571 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd
0709 12:26:28.470839 (+   619us) log.cc:844] Serialized 14 byte log entry
0709 12:26:28.471167 (+   328us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.471274 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:28.518556 (+ 47282us) spinlock_profiling.cc:243] Waited 5.02 ms on lock 0x7b18003b7b20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:28.558562 (+ 40006us) raft_consensus.cc:1685] finished
0709 12:26:28.558605 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.558799 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":573180,"prepare.queue_time_us":570379,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":85,"raft.queue_time_us":76158,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":4919,"spinlock_wait_cycles":31092352,"thread_start_us":534511,"threads_started":3,"wal-append.queue_time_us":278787}
I20260709 12:26:28.590034  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:28.593497  6271 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:28.594887  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:28.594771  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:28.595832  6279 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:26:28.596618 10273 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:28.596848 10175 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:28.642634 10175 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:28.644481  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.652316  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:28.653178 10216 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.654915 10151 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:28.655200  5561 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:26:28.656044 10265 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.585s	user 0.012s	sys 0.000s
I20260709 12:26:28.658900 10178 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:28.664093  5554 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20260709 12:26:28.664696  5554 pending_rounds.cc:85] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 6
I20260709 12:26:28.664971  5554 pending_rounds.cc:107] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20260709 12:26:28.665246  5554 raft_consensus.cc:2889] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:28.668727  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.668985  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.668888  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:28.681290  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.684695  9844 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:28.688246  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:28.689986  9759 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:28.690563  9759 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:28.691090  9759 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:28.725548  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:28.727121  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63124.4us
W20260709 12:26:28.728051  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.728683 10266 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.729200  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:28.729441 10251 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:28.730062 10251 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.730984 10060 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:28.731784 10293 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:28.732570 10068 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:28.732858 10293 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:28.733510 10142 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:28.729202 10266 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:28.733954 10142 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:28.734774 10068 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:26:28.734872 10142 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:28.732661 10295 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:26:28.735952 10068 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:28.736270 10142 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:28.736336 10295 consensus_queue.cc:1243] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:26:28.737092 10249 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:28.737464 10249 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:28.738183  5524 leader_election.cc:400] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11.
I20260709 12:26:28.738502 10295 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:26:28.733369 10293 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:28.739461 10295 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:28.739941 10295 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:28.740293 10293 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.740767 10295 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:28.741683 10293 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:28.742041 10295 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:28.738191 10060 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:26:28.736399 10068 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:28.745282 10068 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:28.746773 10068 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:28.748901  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.738287 10249 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:28.739364  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:28.750048 10249 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:28.750217 10253 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.750720 10253 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:28.752661 10060 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:28.752975 10253 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:28.753933 10251 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:28.754103  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:28.754570 10251 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.754858 10060 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:28.755111 10251 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:28.756116  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:28.735634 10266 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:28.759645 10253 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:28.760982  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:28.763778  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:28.765842  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:28.766650 10266 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:28.768422  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:28.770190  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:28.761744 10253 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:28.761296 10251 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:28.720954 10255 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:28.785584  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:28.786223  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.787452  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:28.769073 10295 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:28.789698  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:28.792254  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:28.777509 10266 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:28.780876 10060 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:28.803889  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:28.804572  6276 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:28.792042 10115 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:28.795248  9846 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:28.793659  9844 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:28.824258  6276 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:28.791854 10300 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.016s
I20260709 12:26:28.826622  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:28.791348 10255 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:28.839296 10303 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:26:28.829627 10216 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:28.885628  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 105934us
I20260709 12:26:28.849197  9846 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:28.883363 10060 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:28.911862  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:28.912602  5710 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:28.911965  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:28.915884  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.884757  9750 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:28.920982  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.922559  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:28.925340  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2337) took 1159 ms. Trace:
I20260709 12:26:28.926211  5629 rpcz_store.cc:276] 0709 12:26:27.766144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.768283 (+  2139us) service_pool.cc:224] Handling call
0709 12:26:27.771865 (+  3582us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:27.959019 (+187154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.924787 (+965768us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b540032cdd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:28.925170 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":411815808}
I20260709 12:26:28.896174 10218 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:28.916682  9846 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:28.950206 10305 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:26:28.952383  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:28.953246  5557 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:28.954159  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:28.954792  5559 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:28.914584 10060 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:28.961454  5706 raft_consensus.cc:2368] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10.
I20260709 12:26:28.956826  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:28.935786 10230 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.973292 10282 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:28.974581  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 830411us
I20260709 12:26:28.981900  5559 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:28.987058  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:29.000530  5559 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:26:29.001818  5557 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:26:29.004168  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:29.007012  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 117773us
I20260709 12:26:29.012892  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:28.980592 10237 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:29.028476  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:28.927361  9750 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:29.091228  5632 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:29.094076  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:29.024607  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:29.094718  5629 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:29.092259  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.095569  5627 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:26:29.098591  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.100590  5627 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:28.975494 10230 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:29.100984  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.107514  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:29.097043 10306 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.201s	user 0.000s	sys 0.001s
I20260709 12:26:29.110368  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:29.136503  5632 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:29.093243 10096 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.988860 10282 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:29.135390  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.225298  5627 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
W20260709 12:26:29.141534  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.100045 10255 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:29.097425  9945 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.095647 10165 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:29.097560 10214 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.223395 10322 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:26:29.227424  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 203107us
W20260709 12:26:29.237512  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 250489us
I20260709 12:26:29.253638  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:29.236320 10242 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:29.248556  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.233237  5629 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:29.265268  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:29.144541 10273 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:29.293239  5627 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
W20260709 12:26:29.251946  5555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2202) took 1872 ms (client timeout 1541 ms). Trace:
W20260709 12:26:29.294569  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 283280us
W20260709 12:26:29.294727  5555 rpcz_store.cc:269] 0709 12:26:27.379358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.384109 (+  4751us) service_pool.cc:224] Handling call
0709 12:26:29.251762 (+1867653us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b54001c4dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:29.251875 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1220751744}
I20260709 12:26:29.099249  9759 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:29.319293  5632 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:29.244928  9958 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.320855  5554 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:29.322222  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.248618  9751 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:29.324301  5629 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
W20260709 12:26:29.245072 10315 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
I20260709 12:26:29.241127 10166 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:29.256585 10288 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:29.089874  9846 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:29.296721 10262 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:29.242280 10165 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:29.267735 10151 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:29.291213 10242 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:29.228524 10096 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:29.291415 10192 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:29.335606 10249 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.582s	user 0.010s	sys 0.001s
I20260709 12:26:29.241457  9945 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:29.245374 10214 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:29.344215  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 96475.7us
W20260709 12:26:29.342078  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 234677us
I20260709 12:26:29.344835  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:29.362233  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:29.344755  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:29.363592  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:29.363966  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:29.337379  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:29.365064  5632 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:29.242534 10143 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:29.331496  9958 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:29.352677  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:29.451625  5629 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 13: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:29.311439 10273 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:29.247299 10322 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:29.459775  5710 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.547s	user 0.010s	sys 0.000s
I20260709 12:26:29.346728 10203 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:29.349558  9751 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:29.473793  5710 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:29.476320  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:29.361809 10218 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:29.340627 10289 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.124s
I20260709 12:26:29.492427  5632 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:29.505666  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:29.504098  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 142624us
I20260709 12:26:29.507218  5632 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:29.508328  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.450762 10249 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.509178  5632 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:29.507736  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:29.461937 10060 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:29.461072  9829 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:29.521449  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.526510  5632 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:29.526765  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.527526  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:29.528429  5632 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:26:29.530217  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.531737  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.483019 10288 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:29.533270  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.360594 10192 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:29.493840 10187 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.493597 10088 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.483388 10237 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:29.539629  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.460749 10143 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:29.497820 10221 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.497982 10327 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.551796  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:29.554814  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.483247 10203 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:29.538659  9750 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:29.563190  5552 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:26:29.572544  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.573518  5629 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:29.576884 10143 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:29.559742 10327 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:29.578336 10282 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:29.559818 10221 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:29.580878 10327 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:29.553865 10088 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:29.586047  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:29.587143 10088 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:26:29.543373  9829 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:29.588652 10096 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:29.529151 10249 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:29.601262  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 248364us
I20260709 12:26:29.602171  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:29.602933  5559 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:29.588847  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:29.604861  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:29.604964  5560 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:29.589430  9759 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:29.554569 10187 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:29.537671 10218 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:29.595134  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:29.512560  9751 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:29.597106  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 89453.7us
I20260709 12:26:29.622262  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:29.641336  5524 leader_election.cc:400] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10.
I20260709 12:26:29.641732  5524 leader_election.cc:403] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:26:29.623651  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:29.629685  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:29.568920 10230 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:29.596184 10165 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:29.628077  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.555693 10237 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:29.550195 10288 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:29.635870 10118 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:29.634589 10328 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.640291  9759 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:29.655455  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.647670 10108 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 12:26:29.658947 10328 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:29.587143  9958 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:29.646659  9750 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:26:29.652837 10068 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.903s	user 0.014s	sys 0.000s
I20260709 12:26:29.648628  9932 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:29.662248  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 364799us
I20260709 12:26:29.649623  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:29.653342  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.651212 10230 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:29.654153 10268 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:29.643110 10218 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:29.689275  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.690045  5552 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 12:26:29.708791  5552 pending_rounds.cc:85] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 5
W20260709 12:26:29.703338  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54205.2us
I20260709 12:26:29.709450  5552 pending_rounds.cc:107] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:26:29.709947  5552 raft_consensus.cc:2889] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:29.711006  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:29.711350  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.714012  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:29.666455  9759 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:29.682215 10340 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:29.727530  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:29.728045  5635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 830) took 2014 ms (client timeout 1584 ms). Trace:
W20260709 12:26:29.733505  5635 rpcz_store.cc:269] 0709 12:26:27.713947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.714195 (+   248us) service_pool.cc:224] Handling call
0709 12:26:29.727770 (+2013575us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b54001cf2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:29.727952 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1945037696}
I20260709 12:26:29.734915  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:29.735548  5635 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:29.687353  9750 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:29.691004 10230 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:29.694957  9945 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:29.697925 10331 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.700692 10118 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:29.739526 10340 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:29.684916 10108 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 12:26:29.699788 10214 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:29.751240  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 389059us
I20260709 12:26:29.752102  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:29.756026 10325 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:26:29.756826 10118 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:29.757552 10334 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:26:29.755932  9759 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:29.738128 10328 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:29.758706  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2228) took 1043 ms. Trace:
I20260709 12:26:29.738897 10068 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.741739 10230 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:29.738329 10203 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:29.761503  5557 rpcz_store.cc:276] 0709 12:26:28.715338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.021989 (+306651us) service_pool.cc:224] Handling call
0709 12:26:29.022366 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:29.022863 (+   497us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:29.022906 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:29.094372 (+ 71466us) log.cc:844] Serialized 26 byte log entry
0709 12:26:29.492778 (+398406us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:29.495029 (+  2251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:29.495462 (+   433us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:29.758469 (+263007us) raft_consensus.cc:1685] finished
0709 12:26:29.758495 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:29.758649 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":306,"raft.queue_time_us":111,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":67890,"thread_start_us":265970,"threads_started":1,"wal-append.queue_time_us":524796}
I20260709 12:26:29.729269 10060 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 12.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.027s
I20260709 12:26:29.701570 10268 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:29.752507 10331 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:29.770970  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.751897 10322 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:29.763288 10175 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:26:29.793663  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 93638.3us
W20260709 12:26:29.797580 10293 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.051s	user 0.016s	sys 0.000s
I20260709 12:26:29.802471 10151 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:29.802843  9829 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:29.772271  9751 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:29.807228  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 96313.7us
I20260709 12:26:29.868593  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:29.869452  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:29.790551  9750 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:29.845033  9945 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:29.871946  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.872651  5632 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
W20260709 12:26:29.811092  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 157823us
I20260709 12:26:29.873564  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:29.810304  5635 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:26:29.849573  9932 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10.
I20260709 12:26:29.790602 10337 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:29.769140 10068 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:29.758597 10346 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
I20260709 12:26:29.796789 10254 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:29.820849  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:29.881603  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.902037  5706 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:29.829680 10331 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:29.830247 10237 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:29.831182 10218 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:29.814628 10312 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.827s	user 0.001s	sys 0.004s
I20260709 12:26:29.805763 10187 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:29.833348 10251 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:29.834156  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 800) took 1340 ms. Trace:
I20260709 12:26:29.879072 10288 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:29.837136 10047 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:29.842157  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 247121us
W20260709 12:26:29.939606  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 179892us
I20260709 12:26:29.940615  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:29.842708 10165 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:29.805565 10249 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:29.844058  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.866623  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.804286 10268 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:29.869766  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:29.880064  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:29.823138  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 96251.2us
I20260709 12:26:29.950119  5632 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:29.905694  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:29.913823  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:29.920812  9932 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.951566  5702 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:29.905997  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:29.920781  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 168790us
I20260709 12:26:29.953429  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:29.953550  5633 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:29.924057  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:29.954255  5557 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:29.955076  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:29.925693  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:29.927930  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:29.881017 10337 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:29.937387 10251 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:29.937489  5558 rpcz_store.cc:276] 0709 12:26:28.493412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.511290 (+ 17878us) service_pool.cc:224] Handling call
0709 12:26:29.833923 (+1322633us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b54000267d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:29.834074 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50672896}
I20260709 12:26:29.834880  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 806) took 1237 ms. Trace:
I20260709 12:26:29.938668  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 810) took 1107 ms. Trace:
I20260709 12:26:29.942109  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:29.973661  5561 rpcz_store.cc:276] 0709 12:26:28.597845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.651565 (+ 53720us) service_pool.cc:224] Handling call
0709 12:26:29.834692 (+1183127us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b5400082854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:29.834820 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3690752}
I20260709 12:26:29.901687 10254 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:30.001418 10254 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:29.945428  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.881125 10214 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 12:26:29.949469  5706 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:29.951413  5707 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
W20260709 12:26:29.957952  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 88259.6us
I20260709 12:26:30.013598  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:29.959673  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:29.960031  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:29.960213  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:29.960322  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:30.013645  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:29.961462  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:30.016155  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56203.3us
I20260709 12:26:30.017938  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:30.018149  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:30.018543  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.019378  5706 raft_consensus.cc:2410] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:26:30.019629  9844 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:30.020254 10270 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:29.961469  5704 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:29.962309  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:30.021157  9751 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.021659  9751 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:30.022125  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:30.022846  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.023211  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:30.019169 10353 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:30.023955  6278 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 12: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:29.949826  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:29.963480  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:30.024832  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.025110  6273 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:29.965636 10165 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:29.966589 10068 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:29.878495  9759 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:29.955613  5557 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:30.033179  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:29.881249  9945 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:29.974936  6236 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:30.039634  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 64774.8us
I20260709 12:26:30.033941  5557 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:29.973728  5553 rpcz_store.cc:276] 0709 12:26:28.831524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.257199 (+425675us) service_pool.cc:224] Handling call
0709 12:26:29.257485 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:29.257692 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:26:29.651563 (+393871us) log.cc:844] Serialized 14 byte log entry
0709 12:26:29.938330 (+286767us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:29.938386 (+    56us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:29.938440 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:29.938503 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:29.938616 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":303686,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":69237,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":116,"thread_start_us":639844,"threads_started":3,"wal-append.queue_time_us":164508}
I20260709 12:26:30.041174  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:30.041218  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:30.042563  5557 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 12:26:30.042645  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:29.977198  5559 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:29.972491 10337 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:29.967725 10354 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:29.943529 10341 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.943303 10237 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:29.987332  5632 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:30.014874  6275 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:29.960808  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:30.024112 10353 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:30.053223  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:30.054373  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:30.055189  6274 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 13: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:30.025548  5630 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 12:26:30.055532  5630 pending_rounds.cc:85] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 5
I20260709 12:26:30.055791  5630 pending_rounds.cc:107] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:26:30.056074  5630 raft_consensus.cc:2889] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:30.026177  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.027220  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:30.057642  5702 raft_consensus.cc:2410] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 15 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 12:26:30.027894  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:30.026724  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:30.029438  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:30.058938  5558 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 8.
I20260709 12:26:30.044909  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:30.044950  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:30.015746 10047 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:29.999958 10230 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:30.054478  5629 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:30.028633  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:29.973127  9932 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:30.071630  5635 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:26:30.069585  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:30.068718 10353 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:30.088692  5553 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:26:30.050195 10351 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:30.051757 10331 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:30.090255  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:30.069494 10341 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:30.119457  5702 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:30.115563 10047 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:30.039117 10165 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:30.121384  6276 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:30.080724 10270 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:30.136539 10047 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:30.139483  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 80014.6us
I20260709 12:26:30.141542  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:30.120730  5704 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:30.133620  6276 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:30.053624 10251 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:30.144269 10218 raft_consensus.cc:2764] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:26:30.116281 10351 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:30.151180  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2342) took 1175 ms. Trace:
I20260709 12:26:30.151651  5634 rpcz_store.cc:276] 0709 12:26:28.975599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.976098 (+   499us) service_pool.cc:224] Handling call
0709 12:26:30.150475 (+1174377us) spinlock_profiling.cc:243] Waited 8.96 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:30.150664 (+   189us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.150841 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:30.150866 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:30.150911 (+    45us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:30.150952 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.151006 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:30.151108 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20614912}
I20260709 12:26:30.154438  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:30.135205 10270 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:30.197597  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 56171.9us
I20260709 12:26:30.198477  6236 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:30.140635 10053 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.052s
W20260709 12:26:30.205495 10363 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:26:30.208458 10370 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:30.209004 10165 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:30.210311  9932 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:30.159116 10351 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:30.204329  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 58800.5us
I20260709 12:26:30.210916  6275 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
W20260709 12:26:30.211678 10366 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:26:30.203068 10251 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:30.202891 10270 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:30.220288 10322 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:30.240736 10341 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:30.240253 10337 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:30.244583 10337 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:30.244962  9945 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:30.245100  6273 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:30.248443  6273 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:26:30.257797  9844 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:30.221457 10369 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:30.258219 10351 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:30.263438 10230 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:30.264331 10369 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:30.265301 10369 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:30.268821  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2214) took 1359 ms. Trace:
I20260709 12:26:30.269268  5703 rpcz_store.cc:276] 0709 12:26:28.909677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.912392 (+  2715us) service_pool.cc:224] Handling call
0709 12:26:28.916190 (+  3798us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:28.924469 (+  8279us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:26:28.936417 (+ 11948us) log.cc:844] Serialized 14 byte log entry
0709 12:26:30.031835 (+1095418us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b18002b4620. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:26:30.124831 (+ 92996us) log.cc:844] Serialized 14 byte log entry
0709 12:26:30.125078 (+   247us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:30.140068 (+ 14990us) log.cc:844] Serialized 26 byte log entry
0709 12:26:30.140241 (+   173us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:30.140484 (+   243us) log.cc:844] Serialized 14 byte log entry
0709 12:26:30.140634 (+   150us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.144114 (+  3480us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:30.255442 (+111328us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b18002b4620. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:30.268468 (+ 13026us) raft_consensus.cc:1685] finished
0709 12:26:30.268492 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:30.268751 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9210,"prepare.queue_time_us":22711,"prepare.run_cpu_time_us":185,"prepare.run_wall_time_us":3105,"raft.queue_time_us":20900,"raft.run_cpu_time_us":475,"raft.run_wall_time_us":7194,"spinlock_wait_cycles":69694208,"thread_start_us":86777,"threads_started":1,"wal-append.queue_time_us":1182355}
I20260709 12:26:30.267872 10353 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:30.272663  6277 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 13. Ops: []
I20260709 12:26:30.273389 10353 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:30.293562 10254 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:30.295499 10339 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.596s	user 0.002s	sys 0.000s
I20260709 12:26:30.298502 10331 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:30.314267 10270 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:30.315322 10192 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:30.304427 10254 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:30.314281 10242 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:30.331156 10351 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:30.337967 10270 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:30.393644  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:30.398288  5630 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:30.422622  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:30.395717  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:30.405803  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2350) took 1304 ms. Trace:
W20260709 12:26:30.407733  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 229830us
I20260709 12:26:30.423878  5701 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:30.410153  5631 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:30.425333  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2344) took 1428 ms. Trace:
I20260709 12:26:30.425865  5631 rpcz_store.cc:276] 0709 12:26:28.996803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.996992 (+   189us) service_pool.cc:224] Handling call
0709 12:26:30.409124 (+1412132us) spinlock_profiling.cc:243] Waited 7.82 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:30.409581 (+   457us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.424740 (+ 15159us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.425214 (+   474us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17978752}
I20260709 12:26:30.414261  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:30.417776  5636 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:30.418785  5628 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:30.435832  5710 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:26:30.436108  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:30.435351  5703 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:30.489454 10060 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.490612 10060 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:30.491313  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:30.492316  5634 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:30.423686  5627 rpcz_store.cc:276] 0709 12:26:29.100759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.236290 (+135531us) service_pool.cc:224] Handling call
0709 12:26:30.405172 (+1168882us) spinlock_profiling.cc:243] Waited 35.9 ms on lock 0x7b54000c1f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:30.405580 (+   408us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":82583424}
I20260709 12:26:30.439333  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2126) took 1085 ms. Trace:
W20260709 12:26:30.493779  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 60404.7us
I20260709 12:26:30.428297  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:30.497118  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2343) took 1512 ms. Trace:
I20260709 12:26:30.435355  5704 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 8. (term mismatch)
I20260709 12:26:30.498770  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2340) took 1536 ms. Trace:
I20260709 12:26:30.440742  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.498634  5704 pending_rounds.cc:85] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 7
I20260709 12:26:30.440160  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:30.497057  5552 rpcz_store.cc:276] 0709 12:26:29.353483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.571449 (+217966us) service_pool.cc:224] Handling call
0709 12:26:29.571888 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:29.710618 (+138730us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.438715 (+728097us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b540023fd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:30.439089 (+   374us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":263887104}
I20260709 12:26:30.498373  5628 rpcz_store.cc:276] 0709 12:26:28.984830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.985019 (+   189us) service_pool.cc:224] Handling call
0709 12:26:30.408014 (+1422995us) spinlock_profiling.cc:243] Waited 6.52 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:30.418248 (+ 10234us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.493320 (+ 75072us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.497043 (+  3723us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15004928}
I20260709 12:26:30.533768  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.534214  5628 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:26:30.535001  5709 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:30.536908  5707 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 9: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:30.538082  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.539080  5707 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:30.539422  9751 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:30.509490  5636 rpcz_store.cc:276] 0709 12:26:28.962645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.963737 (+  1092us) service_pool.cc:224] Handling call
0709 12:26:30.415403 (+1451666us) spinlock_profiling.cc:243] Waited 6.47 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:30.415844 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.484882 (+ 69038us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.498702 (+ 13820us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14887808}
I20260709 12:26:30.534250  6278 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:26:30.527465  5706 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:30.544685  6276 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:30.540016  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.489321 10068 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.583940  5628 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260709 12:26:30.536229  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.526149  5704 pending_rounds.cc:107] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260709 12:26:30.584760  5628 pending_rounds.cc:85] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 6
I20260709 12:26:30.591765  5628 pending_rounds.cc:107] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 12:26:30.589104  5704 raft_consensus.cc:2889] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:30.592339  5628 raft_consensus.cc:2889] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:30.615567  5701 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:30.616894  5627 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:26:30.617429  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.617769  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2086) took 1034 ms. Trace:
I20260709 12:26:30.618626  5701 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:30.618875  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:30.622028  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.623973  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.624881  5707 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 9: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:30.625154  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2089) took 1040 ms. Trace:
I20260709 12:26:30.625619  5628 rpcz_store.cc:276] 0709 12:26:29.584559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.535374 (+950815us) service_pool.cc:224] Handling call
0709 12:26:30.535864 (+   490us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.624874 (+ 89010us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.625097 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:30.627624  5628 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:30.628830  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.620134  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.618291  5627 rpcz_store.cc:276] 0709 12:26:29.583779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.545603 (+961824us) service_pool.cc:224] Handling call
0709 12:26:30.596634 (+ 51031us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.617431 (+ 20797us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.617680 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:30.552696 10281 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 12:26:30.640911  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.647382  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:30.648663  6278 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:30.652933  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2078) took 1097 ms. Trace:
I20260709 12:26:30.654125 10060 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:30.644608  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:30.657651  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:30.658372  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:30.661007  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:30.664562  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 269229us
I20260709 12:26:30.664974 10270 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:30.665786 10270 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:30.666046 10249 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:30.667008 10249 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:30.668609  6275 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:26:30.667575 10270 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:30.664659  5631 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 9: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:30.680902  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2120) took 1333 ms. Trace:
I20260709 12:26:30.689731 10068 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:30.693790 10216 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:30.694548  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:30.695400  5552 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:30.697273 10273 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:30.698045 10273 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:30.641947  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 529725us
I20260709 12:26:30.665815  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:30.710035  5703 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 9.
I20260709 12:26:30.698825 10273 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:30.653659  5632 rpcz_store.cc:276] 0709 12:26:29.555435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.073438 (+518003us) service_pool.cc:224] Handling call
0709 12:26:30.113439 (+ 40001us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:30.113750 (+   311us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:30.113790 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:30.118872 (+  5082us) log.cc:844] Serialized 26 byte log entry
0709 12:26:30.386230 (+267358us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:30.386575 (+   345us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.386761 (+   186us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:30.652274 (+265513us) raft_consensus.cc:1685] finished
0709 12:26:30.652296 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:30.652634 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":467,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":6274,"thread_start_us":240320,"threads_started":1,"wal-append.queue_time_us":517798}
I20260709 12:26:30.681617  5555 rpcz_store.cc:276] 0709 12:26:29.346847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.489833 (+142986us) service_pool.cc:224] Handling call
0709 12:26:29.490250 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:29.490722 (+   472us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:29.490772 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:29.501668 (+ 10896us) log.cc:844] Serialized 26 byte log entry
0709 12:26:29.560315 (+ 58647us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:30.294789 (+734474us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:30.423103 (+128314us) spinlock_profiling.cc:243] Waited 6.16 ms on lock 0x7b180041df40. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf
0709 12:26:30.423647 (+   544us) log.cc:844] Serialized 14 byte log entry
0709 12:26:30.680101 (+256454us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.680265 (+   164us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:30.680373 (+   108us) raft_consensus.cc:1685] finished
0709 12:26:30.680400 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:30.680606 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":256756,"prepare.queue_time_us":20297,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":256112,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":11718,"spinlock_wait_cycles":18787968,"thread_start_us":154350,"threads_started":2,"wal-append.queue_time_us":155462}
W20260709 12:26:30.748427  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 82886.8us
I20260709 12:26:30.761588  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.766170  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.760937  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.769884  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:30.771337 10166 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:26:30.780469  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 866) took 2017 ms. Trace:
W20260709 12:26:30.772698  5633 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.011s	sys 0.038s
I20260709 12:26:30.781198  9750 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:30.841197  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44564.1us
I20260709 12:26:30.842509 10221 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:30.843979  6276 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:30.840955  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:30.770761 10142 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:30.848373  6278 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:26:30.849787  5703 raft_consensus.cc:2410] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:26:30.847142  5555 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:30.854944  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:30.865996 10192 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.857643  5701 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10. Ops: []
W20260709 12:26:30.871610  5633 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.011s	sys 0.040s
I20260709 12:26:30.880393  5706 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:30.866900 10192 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:30.888633 10341 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:30.889353 10330 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.889446 10341 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:30.890062 10330 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:30.893095  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 167888us
W20260709 12:26:30.886791  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 251693us
I20260709 12:26:30.898921  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:30.906037  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 76055.3us
I20260709 12:26:30.907311 10354 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:30.907771 10354 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:30.907882 10023 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:30.908177 10354 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:30.908293 10023 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:30.908778 10023 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:30.884610 10068 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:30.885295 10166 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:30.912415  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 100562us
W20260709 12:26:30.914945  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 67190.3us
I20260709 12:26:30.852084  6271 rpcz_store.cc:276] 0709 12:26:28.763395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.763547 (+   152us) service_pool.cc:224] Handling call
0709 12:26:28.773280 (+  9733us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:28.773492 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:26:28.773799 (+   307us) log.cc:844] Serialized 14 byte log entry
0709 12:26:29.791666 (+1017867us) spinlock_profiling.cc:243] Waited 24.7 ms on lock 0x7b240030d478. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff740b9c4 00007f4ff7409acb 00007f4ff75c4dc9 00007f4ff75c4a55 00007f4ff75c409d 00007f4ffa1f610e 00007f4ffa20d88c 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221
0709 12:26:30.249874 (+458208us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b180043bb80. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:26:30.540969 (+291095us) log.cc:844] Serialized 14 byte log entry
0709 12:26:30.741479 (+200510us) log.cc:844] Serialized 14 byte log entry
0709 12:26:30.779770 (+ 38291us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:30.779852 (+    82us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:30.779944 (+    92us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.780043 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:30.780236 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34079,"prepare.queue_time_us":118426,"prepare.run_cpu_time_us":150,"prepare.run_wall_time_us":1798,"raft.queue_time_us":104732,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":303,"spinlock_wait_cycles":147822464,"thread_start_us":365323,"threads_started":2,"wal-append.queue_time_us":170836}
I20260709 12:26:30.922717  5634 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:30.946273  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:30.948563  5552 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:30.944792  9945 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.953647  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:30.944885 10068 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:31.021021 10068 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:31.021560 10068 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:30.987454  9844 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:26:31.021653 10351 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:30.988512 10218 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:26:31.023231 10378 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.162s	user 0.000s	sys 0.000s
W20260709 12:26:31.023836 10312 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.266s	user 0.000s	sys 0.002s
I20260709 12:26:30.935230  9750 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.029574  9750 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:31.031034  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 135264us
W20260709 12:26:31.021780  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 82113.4us
I20260709 12:26:30.989025 10192 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:30.992509  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2248) took 1033 ms. Trace:
I20260709 12:26:31.042586  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:30.997212 10360 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.009069  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2066) took 1523 ms. Trace:
I20260709 12:26:31.009631 10023 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.043622 10360 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.009980  9945 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.014765  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:31.041319  5557 rpcz_store.cc:276] 0709 12:26:29.959150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.113672 (+154522us) service_pool.cc:224] Handling call
0709 12:26:30.113914 (+   242us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:30.437032 (+323118us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:26:30.437547 (+   515us) log.cc:844] Serialized 14 byte log entry
0709 12:26:30.683642 (+246095us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:30.685047 (+  1405us) log.cc:844] Serialized 26 byte log entry
0709 12:26:30.685241 (+   194us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:30.685501 (+   260us) log.cc:844] Serialized 14 byte log entry
0709 12:26:30.685709 (+   208us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.685829 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:30.992073 (+306244us) raft_consensus.cc:1685] finished
0709 12:26:30.992096 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:30.992277 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":411534,"prepare.queue_time_us":57153,"prepare.run_cpu_time_us":160,"prepare.run_wall_time_us":16436,"raft.queue_time_us":467577,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":212,"spinlock_wait_cycles":8201088,"thread_start_us":149904,"threads_started":1,"wal-append.queue_time_us":511217}
I20260709 12:26:30.936452 10142 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.046721 10023 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:31.048013  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 864) took 1085 ms. Trace:
I20260709 12:26:31.049424  6275 rpcz_store.cc:276] 0709 12:26:29.962109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.213246 (+251137us) service_pool.cc:224] Handling call
0709 12:26:30.213697 (+   451us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:30.669290 (+455593us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.047590 (+378300us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b54000b54d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:31.047916 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":618955648}
I20260709 12:26:31.048200 10142 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.044112  5633 rpcz_store.cc:276] 0709 12:26:29.485581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.970497 (+484916us) service_pool.cc:224] Handling call
0709 12:26:29.971123 (+   626us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:29.971570 (+   447us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:29.971613 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:29.972177 (+   564us) log.cc:844] Serialized 26 byte log entry
0709 12:26:30.141967 (+169790us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:30.916217 (+774250us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.008475 (+ 92258us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.008590 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:31.008679 (+    89us) raft_consensus.cc:1685] finished
0709 12:26:31.008703 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.008969 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12031,"prepare.queue_time_us":80979,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":1200,"raft.queue_time_us":39405,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":6737,"thread_start_us":822668,"threads_started":2,"wal-append.queue_time_us":224380}
I20260709 12:26:30.949846 10273 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:31.058509  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 136581us
I20260709 12:26:30.946801 10341 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:31.065474  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 77670.3us
I20260709 12:26:31.077914 10369 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:31.083146  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2080) took 1475 ms. Trace:
I20260709 12:26:31.084793 10369 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:31.086048 10369 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.083828  5635 rpcz_store.cc:276] 0709 12:26:29.582109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.082174 (+500065us) service_pool.cc:224] Handling call
0709 12:26:30.114400 (+ 32226us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:30.114749 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:26:30.144522 (+ 29773us) log.cc:844] Serialized 14 byte log entry
0709 12:26:30.389750 (+245228us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:30.391556 (+  1806us) log.cc:844] Serialized 26 byte log entry
0709 12:26:30.391887 (+   331us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:30.395166 (+  3279us) log.cc:844] Serialized 14 byte log entry
0709 12:26:30.395473 (+   307us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.395900 (+   427us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:30.926738 (+530838us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b48002727d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:26:31.057640 (+130902us) raft_consensus.cc:1685] finished
0709 12:26:31.057667 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.057841 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":189739,"prepare.queue_time_us":122072,"prepare.run_cpu_time_us":150,"prepare.run_wall_time_us":155,"raft.queue_time_us":312483,"raft.run_cpu_time_us":5494,"raft.run_wall_time_us":849795,"spinlock_wait_cycles":9378048,"thread_start_us":179098,"threads_started":1,"wal-append.queue_time_us":537980}
W20260709 12:26:31.096218  6274 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.620s	user 0.012s	sys 0.016s
I20260709 12:26:31.096527 10281 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:31.096812  6274 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:31.097209 10281 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.098132  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:31.101629  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:31.080914 10142 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:31.079671 10249 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.107040  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.111701  6271 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:31.114096  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.117246  5555 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:26:31.116770  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.103636 10023 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:31.120474  5707 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:31.124056  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.126538  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.126777  5706 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:31.127337  5709 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:31.127861  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.120237  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:31.131105  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 173805us
I20260709 12:26:31.131767  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:31.105705 10142 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:31.133060 10251 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:31.133674  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:31.134958  5701 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:31.129307  5557 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 8.
I20260709 12:26:31.129626  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.136328 10281 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:31.122371 10023 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:31.137923  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:31.138680  5702 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:31.142037 10360 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:31.142280 10340 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.145148 10068 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.148890  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 863) took 1187 ms. Trace:
W20260709 12:26:31.155753  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:31.160743  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.152496 10142 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:31.159873  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:31.158915  6277 rpcz_store.cc:276] 0709 12:26:29.961852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.083342 (+121490us) service_pool.cc:224] Handling call
0709 12:26:30.084425 (+  1083us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.273355 (+188930us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.148467 (+875112us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b54002a8354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:31.148740 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":432528000}
I20260709 12:26:31.167130  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:31.160676 10273 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:31.170680 10273 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.172644  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:31.174253  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:31.174036 10340 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.174063  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.177182 10251 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.159962 10023 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:31.183130  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:31.184904 10192 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.183957 10250 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:31.188089 10341 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:26:31.188755  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2233) took 1251 ms. Trace:
I20260709 12:26:31.189972  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:31.191115 10351 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:31.191645 10351 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:31.189301  5560 rpcz_store.cc:276] 0709 12:26:29.924338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.924509 (+   171us) service_pool.cc:224] Handling call
0709 12:26:29.924852 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:29.925092 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:26:29.943174 (+ 18082us) log.cc:844] Serialized 14 byte log entry
0709 12:26:30.427206 (+484032us) spinlock_profiling.cc:243] Waited 4.55 ms on lock 0x7b180041ff20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:26:30.502579 (+ 75373us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:30.525669 (+ 23090us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:31.175756 (+650087us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b54002fc9a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5ed733 00007f4ffa672dbf 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c
0709 12:26:31.175808 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.175980 (+   172us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.176182 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1847,"prepare.queue_time_us":19342,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":11114,"raft.queue_time_us":545469,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":308,"spinlock_wait_cycles":14075520}
I20260709 12:26:31.192799 10351 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.185484 10192 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.198050  9945 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:31.198310 10192 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.198711  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:31.199451  6275 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:31.201608  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 817) took 1221 ms. Trace:
W20260709 12:26:31.214890  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 64557.9us
W20260709 12:26:31.215839  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:31.217895  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55110.6us
I20260709 12:26:31.219131  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:31.229187  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:31.234323  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:31.202025  5556 rpcz_store.cc:276] 0709 12:26:29.979805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.204340 (+224535us) service_pool.cc:224] Handling call
0709 12:26:30.596060 (+391720us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:30.736387 (+140327us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:30.736434 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:30.770061 (+ 33627us) log.cc:844] Serialized 26 byte log entry
0709 12:26:30.978207 (+208146us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:30.978266 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.978368 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:31.146158 (+167790us) raft_consensus.cc:1685] finished
0709 12:26:31.146177 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.201208 (+ 55031us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b54001bd850. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:31.201445 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":116239,"raft.queue_time_us":137712,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":16466,"spinlock_wait_cycles":470552832,"thread_start_us":74464,"threads_started":1,"wal-append.queue_time_us":265775}
W20260709 12:26:31.238356  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 59159.1us
I20260709 12:26:31.238852 10265 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:31.240097 10250 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:31.240545 10250 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:31.240849 10192 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:31.241576 10250 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.244105 10288 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:31.244637 10192 consensus_queue.cc:1243] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 14.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:26:31.245515 10192 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:31.245980 10192 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:31.247529 10192 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.248582  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:31.249074 10251 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.249188 10192 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:31.249677 10249 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:31.250098 10249 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:31.250447 10249 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:31.252641 10358 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:31.253371 10358 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:31.241892  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:31.254519 10358 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.255301  9958 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:31.214195 10340 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:31.259159  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:31.274735  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:31.274964  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:31.277637  5552 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 13: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:31.230621 10218 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:31.274138  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:31.281369  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:31.292877  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.293596  5710 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:31.281479 10360 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:31.298141 10151 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:31.298681  9945 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:31.298795 10385 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:31.300011  9958 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:31.300750  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.260545 10023 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.291356 10218 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:31.303145 10360 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:31.214425 10166 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.323529  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:31.313189 10385 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:31.316401 10340 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:26:31.325316  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 64644.7us
I20260709 12:26:31.321848 10360 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:31.335088 10043 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:31.342634  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 192182us
I20260709 12:26:31.343204  5635 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:26:31.344523 10242 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:31.345268 10242 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.346398 10151 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:26:31.347968 10242 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:31.348479 10068 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:31.341071  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:31.328466 10340 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:31.320902 10354 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.337910 10273 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.337141 10385 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.366645  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:31.347980 10151 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:31.377256  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.359958 10340 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:31.387372  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:31.388811  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:31.396564  9945 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:31.379664 10270 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 12:26:31.400728 10218 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.397135  9945 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:31.357491 10043 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:31.413146  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55779.4us
I20260709 12:26:31.407274  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:31.377332 10151 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:31.385403 10351 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:26:31.420940 10392 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:26:31.432374 10281 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:26:31.434991 10281 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:31.435308  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:31.437062 10251 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:31.437786 10251 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:26:31.444600 10230 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:26:31.448625  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:31.452678  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:31.455655 10023 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:31.457353  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:31.458526  5627 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:31.459941 10251 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:31.460453 10251 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:31.460889  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:31.461782  5633 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:31.461567 10251 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.463351 10251 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:31.464866  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:31.469586  5710 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:31.471282  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:31.472824 10151 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.409271 10068 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:26:31.475019 10068 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:31.473667 10354 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:31.477705  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 114176us
I20260709 12:26:31.412040  9945 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:31.471807 10142 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.486828  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:31.455072  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:31.488607  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:31.491106  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:31.491375  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:31.492096  5634 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 12:26:31.475641 10068 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:26:31.494513 10273 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:31.477397 10369 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:31.475406 10151 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:31.496824  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 77128.9us
W20260709 12:26:31.497288  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 147551us
I20260709 12:26:31.491276  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:31.507583  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:31.498205 10369 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:31.498404 10354 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:31.509208 10192 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:31.516075  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:31.518348  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:31.522564  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 67648.5us
I20260709 12:26:31.522830  5634 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:26:31.514248 10369 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:31.533349  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 867) took 1570 ms. Trace:
I20260709 12:26:31.533932  6274 rpcz_store.cc:276] 0709 12:26:29.962858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.069341 (+106483us) service_pool.cc:224] Handling call
0709 12:26:31.533028 (+1463687us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b54002922d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:31.533254 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":419429760}
I20260709 12:26:31.535883 10142 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.524684 10288 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:31.536581 10288 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:31.537021 10288 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:31.540534 10354 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.541692  5559 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:31.543543 10354 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:31.525068  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:31.540593  5561 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:31.545967  5561 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:31.547039 10375 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.598s	user 0.003s	sys 0.000s
I20260709 12:26:31.546854  5561 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.551208 10250 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:31.555085  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 845) took 1710 ms (client timeout 1544 ms). Trace:
W20260709 12:26:31.555583  5629 rpcz_store.cc:269] 0709 12:26:29.844165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.846613 (+  2448us) service_pool.cc:224] Handling call
0709 12:26:31.554706 (+1708093us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b54001f2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:31.554901 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":250667904}
I20260709 12:26:31.557931  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:31.558542  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:31.559805 10142 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:31.560586  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:31.560819  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:31.561167 10249 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.560590 10142 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.566985 10330 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:31.548195  5561 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:31.570087  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:31.546782  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:31.580413 10383 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.673s	user 0.005s	sys 0.001s
I20260709 12:26:31.583033  9958 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:31.584088 10166 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:31.584575 10166 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:31.584954 10166 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:31.585067  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 776) took 1174 ms. Trace:
I20260709 12:26:31.587206  5627 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:31.585456  5708 rpcz_store.cc:276] 0709 12:26:30.410779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.436523 (+ 25744us) service_pool.cc:224] Handling call
0709 12:26:30.437257 (+   734us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:30.438237 (+   980us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:30.438310 (+    73us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:30.439352 (+  1042us) log.cc:844] Serialized 52 byte log entry
0709 12:26:30.700412 (+261060us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:30.751536 (+ 51124us) log.cc:844] Serialized 14 byte log entry
0709 12:26:30.751799 (+   263us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.751880 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:31.579562 (+827682us) spinlock_profiling.cc:243] Waited 4.52 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:31.584587 (+  5025us) raft_consensus.cc:1685] finished
0709 12:26:31.584706 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.584880 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":29482,"prepare.run_cpu_time_us":6246,"prepare.run_wall_time_us":88093,"raft.queue_time_us":118077,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":3644,"spinlock_wait_cycles":10388224,"thread_start_us":260430,"threads_started":1,"wal-append.queue_time_us":412203}
W20260709 12:26:31.589352  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:31.591090  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 60892.2us
I20260709 12:26:31.518630 10192 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:31.595146 10192 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:31.608039  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.618736  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:31.620286  5557 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:31.602487 10216 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:31.622129  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.622903  5702 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:31.626051  5706 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:31.633193  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:31.638891  5633 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:31.642983  5701 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:26:31.611400 10358 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.632030 10216 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.652539  6274 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:26:31.653326  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:31.642232  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:31.651072  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.659354  5705 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:31.673750 10360 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.675988 10360 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.676849  5552 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:31.630380 10068 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:31.689503  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:31.691047  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 160532us
W20260709 12:26:31.697312 10259 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.913s	user 0.016s	sys 0.012s
W20260709 12:26:31.697731  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 60389.1us
W20260709 12:26:31.707969 10259 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.988s	user 0.016s	sys 0.012s
I20260709 12:26:31.708469  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:31.709458  5553 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:31.694895 10358 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.710714  9945 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.711727 10068 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.683992 10341 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:31.713137 10358 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:31.713660  9945 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:31.715505  6274 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:31.716293  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:31.716840  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:31.717851  6276 raft_consensus.cc:2410] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 13 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:26:31.718211  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:31.718650 10288 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.717873  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:31.718956  5636 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:31.721694 10216 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:31.723846  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:31.725592 10192 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:31.728682  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 145251us
I20260709 12:26:31.729411 10259 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:31.729997 10259 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:31.731280 10259 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.733064 10336 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:31.733536 10336 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:31.696533 10251 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:31.734520 10336 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.735682 10336 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:31.737088 10288 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.731364  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:31.741818  5632 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:31.742106  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:31.742671 10142 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.743276 10142 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:31.744016 10142 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:31.671047 10242 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:26:31.744947  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49329.2us
I20260709 12:26:31.735294 10251 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.678601 10249 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.713881  9932 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:31.758678 10192 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.767450 10047 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:26:31.768327 10270 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:31.769523  9945 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.769340 10270 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.770284  9945 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.768749 10047 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:31.772596 10047 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:31.773015  9945 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:31.773658 10401 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260709 12:26:31.773947 10047 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.775367 10047 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:31.784837  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:31.791193  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 46377.9us
I20260709 12:26:31.749615 10369 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.750384 10259 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:31.802047  5559 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:26:31.803177  5555 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:26:31.803385  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:31.750059  5561 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 10. (term mismatch)
I20260709 12:26:31.772253 10270 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.806759  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:31.807865  5561 pending_rounds.cc:85] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 9
I20260709 12:26:31.809791  5561 pending_rounds.cc:107] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260709 12:26:31.810138  5561 raft_consensus.cc:2889] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:31.812160  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:31.817622  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:31.794157 10351 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:31.828737  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2352) took 1984 ms. Trace:
I20260709 12:26:31.829252  5636 rpcz_store.cc:276] 0709 12:26:29.846875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.620127 (+773252us) service_pool.cc:224] Handling call
0709 12:26:31.554870 (+934743us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:31.720499 (+165629us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.772004 (+ 51505us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54001f2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:31.828499 (+ 56495us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b54001f2550. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:31.828667 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2444059648}
I20260709 12:26:31.831759  6280 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:31.837271  5709 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:31.839059  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:31.839208  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.840871  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:31.844767  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.844843  5701 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:26:31.853061  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:31.855141  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:31.857687  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:31.844007  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:31.866027  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:31.868429  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 50910.2us
W20260709 12:26:31.868671  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 62279.4us
I20260709 12:26:31.876116  5553 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:31.883257  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:31.838919 10043 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.865921  9945 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:31.873988 10192 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.907207  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:31.902444  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47625.7us
I20260709 12:26:31.899399  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.895635  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:31.872659  9958 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:31.889866 10412 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:31.866083 10354 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:26:31.932152 10417 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:26:31.932600 10337 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:31.933189 10249 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:31.933133 10337 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.933681 10249 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:31.934098 10249 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:31.935873 10337 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:31.937131 10408 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
W20260709 12:26:31.938764 10340 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.532s	user 0.011s	sys 0.001s
I20260709 12:26:31.938659 10192 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.939251 10340 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.941118 10192 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:31.941278 10340 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:31.895033 10414 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:26:31.941772 10416 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:26:31.919667 10412 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:31.942830 10412 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:31.908017  9932 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.945950  9932 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:31.948575  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 53196.9us
I20260709 12:26:31.949497  5706 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:31.950229  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:31.952718 10414 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:31.914220  9945 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.955588  9945 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:31.933202 10417 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:31.957017 10417 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:31.957837 10417 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.959137 10417 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:31.875948 10270 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:26:31.962227 10412 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:31.964763  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 92290.3us
I20260709 12:26:31.967456  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:31.968782  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:31.969549  6276 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:31.970203  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:31.970566  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:31.970957  6277 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:31.948153  9932 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:31.979081 10403 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.384s	user 0.000s	sys 0.004s
I20260709 12:26:31.981603 10251 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:31.894289 10288 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.984714 10288 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.985335  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:31.987210 10270 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:31.988974  5701 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:26:31.989674  5602 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:31.989823 10166 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:31.990662 10192 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:31.991158 10192 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:31.929656  9958 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:31.993278  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:31.991755 10192 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:31.999326  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68029.3us
I20260709 12:26:31.996766  6274 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:32.017953  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:32.018896 10251 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:32.019340 10251 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.019711 10251 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:32.021270  9932 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.022034  9932 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:32.024847  9932 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:32.028947  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:32.039182  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:32.057586  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:32.058449  6278 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 13: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:31.994480 10413 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:32.060784  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:26:32.062705  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 73138.7us
I20260709 12:26:32.068276  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.069057  5561 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:32.070324  5559 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:32.064945  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:32.071074  5701 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:26:32.082939  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.083659  5636 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:32.068033  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:32.087662  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 192102us
I20260709 12:26:32.059846  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:32.094066  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:32.121315  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.121984  5634 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
W20260709 12:26:32.123385  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62847.2us
I20260709 12:26:32.124855  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:32.124778  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:32.062883 10249 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:32.126307  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:32.014748 10043 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.079s
I20260709 12:26:32.067593 10422 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:32.130249 10411 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:32.132714  9958 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:32.133663 10288 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:32.120838  9932 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:32.136946  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:32.040423 10412 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:32.128182 10249 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:32.138550 10391 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:26:32.137683  6279 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:32.142882  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.143564  5627 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:32.142884  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:32.146291  5552 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:32.147830  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.075595 10413 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.157857  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.158532  5634 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:32.159966  5636 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:32.161418  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.162384  5634 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:32.029458 10337 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.092s
I20260709 12:26:32.088994 10351 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.167351  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.138270  5456 catalog_manager.cc:5697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 6 to 8, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 8 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:32.169881  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:32.169831  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:32.170838  6277 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
W20260709 12:26:32.173182  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46979.7us
I20260709 12:26:32.173308  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:32.177382  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 103160us
W20260709 12:26:32.180728  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 111073us
I20260709 12:26:32.065775 10218 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.085s
I20260709 12:26:32.136983  9932 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:32.182925  6271 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:32.184610  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:32.191459  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70970.5us
I20260709 12:26:32.156144 10250 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:32.189464  5701 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:26:32.166253 10413 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:32.208707  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:32.196138  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.215709  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:32.217861  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:32.218739  5627 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:32.227316  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:32.194150  9932 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:32.238306  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 47590.4us
I20260709 12:26:32.232506 10270 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:32.183866 10351 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:32.241421  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:32.196060 10319 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:32.227420 10358 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:32.196342 10385 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:32.245253 10385 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:32.245255 10358 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:32.245795 10385 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:32.248608 10358 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:32.240849 10411 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.008s
I20260709 12:26:32.244045  5552 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:32.241972  9945 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:32.241794 10288 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260709 12:26:32.253362 10416 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:26:32.254650  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61266.9us
I20260709 12:26:32.254760 10416 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:32.260830  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 196009us
I20260709 12:26:32.256296 10273 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:32.266822  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:32.238601 10166 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:32.278028 10423 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:32.278795 10423 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:32.279270 10423 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:32.280232 10413 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:32.282248 10270 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:32.282750 10351 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:32.283331 10270 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:32.285316 10230 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:32.282655 10413 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:32.286993 10230 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:32.287488  6277 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:26:32.288823  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:32.289146  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:32.290544  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:32.291122  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.292599  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:32.293149  6276 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:26:32.293707  5630 raft_consensus.cc:2368] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10.
I20260709 12:26:32.294909  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:32.266271 10273 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:32.296654  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.297493  5633 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:32.301450  5634 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: []
W20260709 12:26:32.306038  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 134534us
I20260709 12:26:32.308048  5553 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
W20260709 12:26:32.305795 10047 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.530s	user 0.013s	sys 0.000s
I20260709 12:26:32.274118 10412 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:32.316483  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:32.317101  5627 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:32.257416 10251 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:32.310871  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:32.324940  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.307612 10265 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:32.355791 10265 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:32.357995 10265 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:32.359081  5555 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:32.360734  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.361431  5627 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:32.361922  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:32.362280  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:32.362496  5630 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:26:32.362825  5627 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:32.365154  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.360757  5631 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:32.318271  5456 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 12, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 12 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:32.345220  6275 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:32.274556 10288 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 12:26:32.367656  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.352648 10251 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:32.353374 10369 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:32.354725  5636 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:32.369444  5634 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:26:32.369784  5628 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 9.
I20260709 12:26:32.306021 10230 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 12:26:32.302887 10358 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
W20260709 12:26:32.370246  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44174.4us
I20260709 12:26:32.297245 10273 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:32.374130  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 144357us
I20260709 12:26:32.259670 10416 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:32.344947 10412 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:32.394570  9932 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:32.383069  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 105549us
I20260709 12:26:32.382216  5627 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:32.320041 10249 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 12:26:32.412148  6277 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:32.416911  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:32.418432  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:26:32.452610  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52024.9us
I20260709 12:26:32.454919  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:32.376554 10214 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260709 12:26:32.457072  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:32.423664 10417 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:32.423309 10251 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.457901  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:32.460739  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.429260 10400 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:32.484004  6278 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:26:32.493413  5636 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:32.456384 10369 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:32.404083 10192 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:32.469854 10416 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:32.542408  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 84558.7us
I20260709 12:26:32.543035  6278 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:26:32.543570  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:32.485447 10440 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:26:32.546478  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:32.470559 10250 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 12:26:32.502902  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:32.456687 10430 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 12:26:32.552042  5629 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:32.496337 10288 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:32.558815  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:32.561645  5636 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:32.497167 10251 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:32.567140  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:32.568099  5708 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:32.568207  5704 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:26:32.569211 10288 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:32.570261 10288 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:32.572912  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:32.573673  5705 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:32.576136 10354 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.576754 10354 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:32.578981  5636 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:32.579049 10354 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:32.487695 10417 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:32.582347  5708 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:32.471252 10412 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:32.588495  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:32.590965  5703 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:32.549515 10440 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.557138 10358 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:32.503724 10369 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:32.594399  5708 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:32.605877  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:32.605830  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:32.608059  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:32.616571  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:32.595535 10358 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.593889 10440 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:32.619500  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:32.605374 10446 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:32.677245  5630 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:32.728734  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:32.796634  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 190906us
I20260709 12:26:32.797991  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:32.620921 10358 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:32.642597 10414 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:32.636831 10446 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:32.821081  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 203625us
I20260709 12:26:32.828452  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:32.830364  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:32.818004 10428 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
I20260709 12:26:32.833261 10433 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:32.812093 10214 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:32.819507 10446 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:32.888793  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:32.901372  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 357888us
I20260709 12:26:32.866122 10414 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:32.811573 10447 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.269s
W20260709 12:26:32.916504 10419 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20260709 12:26:32.918448 10449 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
W20260709 12:26:32.907285 10385 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.661s	user 0.013s	sys 0.000s
I20260709 12:26:32.900913 10250 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:32.910710 10414 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:32.912667 10259 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:32.955942  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:33.002262  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 113591us
I20260709 12:26:33.041566 10433 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:26:33.042505  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 244653us
I20260709 12:26:33.043524  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:32.947449 10414 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:33.057025  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 871) took 1025 ms. Trace:
I20260709 12:26:32.998404 10417 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:33.049741 10369 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:33.066960  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 111188us
I20260709 12:26:33.069655  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:33.057669  6274 rpcz_store.cc:276] 0709 12:26:32.031933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.057936 (+ 26003us) service_pool.cc:224] Handling call
0709 12:26:32.058274 (+   338us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:32.093571 (+ 35297us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:32.093614 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:32.128936 (+ 35322us) log.cc:844] Serialized 52 byte log entry
0709 12:26:32.196785 (+ 67849us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:32.196821 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:32.196897 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:33.056743 (+859846us) raft_consensus.cc:1685] finished
0709 12:26:33.056769 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:33.056947 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43418,"raft.queue_time_us":37282,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":24344,"thread_start_us":67315,"threads_started":1,"wal-append.queue_time_us":169306}
I20260709 12:26:32.926431 10385 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:33.055903 10430 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:32.999388 10250 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:33.073529 10452 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:33.137902 10452 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.139144 10250 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:33.140264 10250 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:33.141289 10250 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:33.067647 10192 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:33.141453 10452 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:33.160712 10459 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:33.136924 10417 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:33.202139 10385 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:33.227752 10423 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.948s	user 0.006s	sys 0.008s
I20260709 12:26:33.234364 10358 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:33.258095 10416 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.753s	user 0.011s	sys 0.001s
I20260709 12:26:33.259260 10437 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.270718 10440 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:33.255530 10446 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:33.302263  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.061s	sys 0.113s
W20260709 12:26:33.319911  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.061s	sys 0.113s
I20260709 12:26:33.321822  5635 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
W20260709 12:26:33.317451  9945 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.001s	sys 0.011s
W20260709 12:26:33.325443  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 708887us
W20260709 12:26:33.320466  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 277003us
I20260709 12:26:33.328637  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:33.328534  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:33.325500  5675 connection.cc:256] RPC call timeout handler was delayed by -1.2569462649998968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1968s for a response
W20260709 12:26:33.322494 10424 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.158s	user 0.022s	sys 0.015s
W20260709 12:26:33.332330  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 262752us
W20260709 12:26:33.331681  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0156304110000747s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
I20260709 12:26:33.318403 10437 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.315861 10423 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.371835  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:33.372398  5632 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:33.373037  5561 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 14: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:33.374109 10423 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:33.321588 10440 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:33.374775 10437 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:33.349179 10439 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.791s	user 0.005s	sys 0.007s
W20260709 12:26:33.327497  9945 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.001s	sys 0.011s
I20260709 12:26:33.323299 10446 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:33.376576  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:33.369912  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:33.343426  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:33.377492  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44773.1us
I20260709 12:26:33.334625 10457 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:33.380642  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52296.1us
I20260709 12:26:33.382037  5707 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:33.382246  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:33.384186  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:33.353796 10452 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:33.388254  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:33.392661  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:33.393013  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:33.393442  5554 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:33.389446  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:33.388895  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:33.387112  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:33.390331  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:33.343526 10358 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:33.393739  5558 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:33.334137 10417 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:33.331552 10453 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:33.331450 10385 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:33.395423  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:33.397192  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:33.397909  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:33.385730 10446 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:33.350123 10454 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.402709  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:33.399711 10440 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:26:33.402562  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:33.419507  5636 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:33.404497  5702 raft_consensus.cc:2368] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10.
I20260709 12:26:33.404340  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:33.404173  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:33.407060 10358 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:33.407126  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:33.421746  5554 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:33.407560 10433 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.425333  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:33.426179  5704 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:33.407629  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:33.427165  5553 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:33.408246 10385 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:33.408640 10336 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:33.388577 10423 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 12:26:33.431927  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:33.409324  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:33.417502  5627 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:33.423303  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:33.433561  5557 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:33.404812 10250 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:33.416800 10446 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:33.435314 10250 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:33.435688 10446 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:33.435338 10423 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:33.436667 10423 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:33.437489 10423 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.438676 10423 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:33.428054  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:33.440816  5634 raft_consensus.cc:2410] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:26:33.428769  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:33.442109  5558 raft_consensus.cc:2368] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12.
I20260709 12:26:33.427716  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:33.444752 10385 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:33.428601  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:33.434369 10336 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:33.434849 10462 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:33.435745 10250 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:33.445768  5707 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:33.384810 10457 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.446707  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:33.466558  5706 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:33.449254  5634 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: [13.8-13.8]
W20260709 12:26:33.467293  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 70185.6us
I20260709 12:26:33.467754 10429 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.419883 10454 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.470655  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:33.419874 10273 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.850s	user 0.017s	sys 0.021s
I20260709 12:26:33.430742 10433 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:33.476044 10273 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.906s	user 0.018s	sys 0.021s
I20260709 12:26:33.469332  5526 leader_election.cc:400] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10.
I20260709 12:26:33.469081  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:33.452919  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:33.426038 10453 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:33.485040 10251 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:33.485538 10453 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.487406  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 881) took 1197 ms. Trace:
I20260709 12:26:33.474221 10454 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:33.490577  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 869) took 1131 ms. Trace:
I20260709 12:26:33.491060  5629 rpcz_store.cc:276] 0709 12:26:32.359349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.359508 (+   159us) service_pool.cc:224] Handling call
0709 12:26:32.359852 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:32.565842 (+205990us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:33.490231 (+924389us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b540018d154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:33.490495 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":439258496}
I20260709 12:26:33.498178  5707 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:33.499233 10414 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:33.500121  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:33.478569  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:33.479238  5526 leader_election.cc:403] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:26:33.477886 10433 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:33.472101 10336 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:33.482936  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:33.526090  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:33.468299 10461 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:33.483137  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:33.468582 10429 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.500377  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2381) took 1141 ms. Trace:
I20260709 12:26:33.479674  5702 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:33.500998  5707 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:33.487926  6277 rpcz_store.cc:276] 0709 12:26:32.290205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.290313 (+   108us) service_pool.cc:224] Handling call
0709 12:26:33.487209 (+1196896us) spinlock_profiling.cc:243] Waited 82.4 ms on lock 0x7b5400534dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:33.487340 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":189412736}
I20260709 12:26:33.508113  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:33.508111  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:33.510262  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:33.511756  5553 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:26:33.512832  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:33.526679  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.507s (SENT) [suppressed 14 similar messages]
I20260709 12:26:33.535372 10273 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 9. Current term is 10.
I20260709 12:26:33.536896  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:33.514214  5631 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:26:33.516840  5557 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:33.517720  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:33.420768 10452 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.524483 10400 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:33.548026 10400 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.518020  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:33.527393  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:33.528342  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:33.543692  5636 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:33.529453  5710 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:33.530918  5633 raft_consensus.cc:2376] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10. Re-sending same reply.
I20260709 12:26:33.533303  5634 raft_consensus.cc:2368] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12.
I20260709 12:26:33.533390  5629 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 11: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:33.533555  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:33.531906  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 62805.1us
I20260709 12:26:33.546767  5556 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:26:33.553200  5627 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:33.547425  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:33.554134  5557 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:33.550242 10400 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:33.555182 10249 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:33.555570  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:33.556488 10249 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:33.556882 10249 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:33.557813 10249 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.558809 10249 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:33.560761  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:33.560918  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 1.06073e+06us
I20260709 12:26:33.561543  5633 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:33.564164  5636 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:33.528995  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:33.540243 10438 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.041s	user 0.022s	sys 0.005s
I20260709 12:26:33.565388  5703 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:33.529400  5635 rpcz_store.cc:276] 0709 12:26:32.358685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.364539 (+  5854us) service_pool.cc:224] Handling call
0709 12:26:32.615733 (+251194us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754df8c 00007f4ff754def6 000000000036eb39 00007f4fff5c11e0 00007f4fff68a0c4 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:32.615870 (+   137us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:33.327026 (+711156us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:33.500088 (+173062us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:33.500287 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1633299584}
I20260709 12:26:33.528707 10412 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:33.567575  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:33.560930  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:33.568035  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:33.570327  6274 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:33.534022  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:33.570410  5702 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:33.574239  5627 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: [7.8-11.9]
I20260709 12:26:33.531579  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:33.575223  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 43745.3us
I20260709 12:26:33.575632  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:33.580148  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:33.575912  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:33.576704  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:33.584754  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:33.579174  5710 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:33.586866  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:33.589725  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:33.602218  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:33.527608 10461 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:33.618149  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:33.631209  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44598.6us
I20260709 12:26:33.609589  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:33.598968  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:33.579399 10047 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.632016  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:33.637458  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:33.622435  6274 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:33.595371 10336 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:33.645123  5553 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:33.533039 10214 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.097s
I20260709 12:26:33.588594  9945 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:33.582794 10251 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:33.651369  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:33.656498  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:33.593437 10281 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:33.635665 10151 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.629769 10429 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:33.595829 10385 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.063s
I20260709 12:26:33.627630 10470 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:33.635504 10319 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:33.634385 10461 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:33.649123 10218 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:26:33.705004  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53767.2us
I20260709 12:26:33.589304 10192 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
W20260709 12:26:33.721915  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.637671 10270 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:33.595108 10358 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:26:33.734406  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 98354.4us
W20260709 12:26:33.736196  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 104270us
I20260709 12:26:33.657943 10417 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:33.755762  5602 leader_election.cc:400] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12.
I20260709 12:26:33.779585  5602 leader_election.cc:403] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:33.698423 10336 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:33.737665 10470 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:33.786825  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 260157us
I20260709 12:26:33.765686 10218 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.709894 10151 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:33.658504 10047 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:33.804316 10470 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:33.815208 10473 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:33.816090  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:33.816584 10464 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:33.816813 10259 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:33.850596 10464 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:33.782899 10281 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:33.851226 10464 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:33.851873 10281 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.783213 10358 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:33.854239 10358 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:33.806418 10218 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:33.753459 10319 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:33.816164 10458 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:26:33.816217 10431 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:26:33.871505  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 200326us
I20260709 12:26:33.849623 10414 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:33.859854 10023 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:33.883623  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 104123us
I20260709 12:26:33.867537 10385 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:33.868222 10477 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:33.850920 10259 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:33.894196  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.898017  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:33.906160  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:33.907994 10250 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.909366  5561 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: []
I20260709 12:26:33.909829 10151 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:33.911492 10250 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:33.910998 10477 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.869676 10473 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:33.915282 10251 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:33.897394 10023 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:33.951442 10023 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:33.932062 10385 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:33.953387 10385 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.932286 10250 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:33.962201 10250 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:33.963505 10471 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260709 12:26:33.965041 10250 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:33.880084 10431 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:33.961192 10469 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 12:26:33.970839  5703 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:33.959615 10251 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:33.965224 10487 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:33.972915 10250 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:34.028522 10259 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:34.042325  9932 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:34.042615 10259 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:26:34.043526 10259 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:34.043202 10358 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.043861 10259 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:34.045037 10319 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:34.045004 10259 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.045714 10400 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:34.046125 10400 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:34.046293 10259 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:34.046845 10431 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:34.047001 10400 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.047520 10047 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:34.047997 10400 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:34.049948 10452 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:34.053399 10484 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:26:34.055160 10464 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:34.076721  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.916244 10336 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.040889 10250 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:34.042491 10466 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.235s	user 0.001s	sys 0.001s
W20260709 12:26:34.037601 10481 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
W20260709 12:26:34.073170  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 1.49452e+06us
I20260709 12:26:34.066967 10487 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:34.059639 10446 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.623s	user 0.011s	sys 0.000s
W20260709 12:26:34.084807  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 518456us
I20260709 12:26:34.084678 10446 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.085999 10457 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:34.087234 10446 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:34.057552 10473 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:34.039412  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:34.057170  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:34.088874  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 494545us
I20260709 12:26:34.089679  5555 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:34.071439  5553 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:34.060655  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 798) took 1766 ms. Trace:
I20260709 12:26:34.078606  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:34.083899 10444 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:26:34.089622  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:34.031433 10218 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:34.078327 10423 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.625s	user 0.008s	sys 0.004s
I20260709 12:26:34.094884  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:34.091259  5701 rpcz_store.cc:276] 0709 12:26:32.293915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.294054 (+   139us) service_pool.cc:224] Handling call
0709 12:26:32.294367 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:32.317041 (+ 22674us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:32.317089 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:32.348095 (+ 31006us) log.cc:844] Serialized 26 byte log entry
0709 12:26:33.336682 (+988587us) spinlock_profiling.cc:243] Waited 4.08 ms on lock 0x7b48001701d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:26:33.536996 (+200314us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b180011c080. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:26:33.686067 (+149071us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:33.686127 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:33.686225 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:34.060434 (+374209us) raft_consensus.cc:1685] finished
0709 12:26:34.060454 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:34.060588 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10839,"raft.queue_time_us":15277,"raft.run_cpu_time_us":30304,"raft.run_wall_time_us":1022290,"spinlock_wait_cycles":59233408,"thread_start_us":148765,"threads_started":1,"wal-append.queue_time_us":1413652}
I20260709 12:26:34.096516  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:34.097389 10273 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:34.097751 10273 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.098318 10273 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:34.098414 10457 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:34.097718  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:34.100199  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60870.3us
I20260709 12:26:34.102833  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:34.105430  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:34.107195  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:34.107266  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:34.114238  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:34.099428 10446 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:34.126981  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:34.122970  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:34.112648 10457 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:34.112772 10431 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:34.136664  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:34.138470  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:34.136854  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:34.139392  6279 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:34.138830  6275 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:34.123467 10023 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:34.122707 10452 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:34.123659  9932 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:34.141541  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:34.142778  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:34.146235  6272 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:34.121575 10493 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:34.110185 10461 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.148260  6276 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:26:34.148836 10432 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.149644  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:34.138904 10319 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:34.084084 10487 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:34.151464  5675 leader_election.cc:336] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.605s (SENT)
I20260709 12:26:34.159386  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:34.152666  6277 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:34.154490  9932 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:34.155050 10495 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:34.164873  5631 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:34.145267 10452 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:34.172410  5558 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:34.147112 10431 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:34.178989  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:34.179770  6277 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 13: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:34.174981 10473 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:34.180644 10169 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.042s
W20260709 12:26:34.189424 10412 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.608s	user 0.010s	sys 0.003s
I20260709 12:26:34.175527 10452 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:34.177244  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:34.158993 10358 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:34.158313 10319 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:34.215224  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:34.199728 10273 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.216840 10476 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:26:34.217814 10446 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:34.218210 10446 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:34.218642 10446 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:34.220247 10431 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:34.221279 10497 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:26:34.208374  9932 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:34.222887  9932 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.225090 10496 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:34.225522 10496 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:34.226317 10496 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.210800 10319 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:34.227600 10496 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:34.158672 10432 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.205250 10470 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:34.233594 10432 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:34.234642 10358 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:34.235163 10358 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:34.235543 10358 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:34.238461 10250 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:34.238597 10023 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:26:34.239770  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 88347.1us
W20260709 12:26:34.240336  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:34.245055  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 489498us
I20260709 12:26:34.245952  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:34.240638 10218 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:34.247048  5602 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:34.247488 10358 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.247830 10470 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:34.248811  9958 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12.
I20260709 12:26:34.251000  9958 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:26:34.251188 10446 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.241135 10464 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:34.241516 10458 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:34.253621 10446 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:34.254343 10436 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:34.255455  9945 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:34.255784 10413 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:34.256215 10413 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:34.256474  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 874) took 2290 ms. Trace:
I20260709 12:26:34.242839 10461 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:34.253279 10458 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.259832  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:34.260672  5553 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:34.261101 10458 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:34.262197  5559 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:34.268263  5552 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:34.268783  5555 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:34.271306 10358 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:34.241747  5675 leader_election.cc:336] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.777s (SENT)
I20260709 12:26:34.257143 10413 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.283296  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:34.286191  5627 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:34.287108  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:34.282253  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:34.256862  6271 rpcz_store.cc:276] 0709 12:26:31.966466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:31.968487 (+  2021us) service_pool.cc:224] Handling call
0709 12:26:32.154978 (+186491us) spinlock_profiling.cc:243] Waited 8.02 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:32.171512 (+ 16534us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:32.194544 (+ 23032us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:34.256221 (+2061677us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b54005f5d50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:34.256386 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":55811712}
I20260709 12:26:34.294528  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:34.294906  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:34.295495  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53767.5us
I20260709 12:26:34.295985  5627 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
W20260709 12:26:34.296200  5675 leader_election.cc:336] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.910s (SENT)
I20260709 12:26:34.296721  5636 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:34.296979  6275 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
W20260709 12:26:34.296012  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 50148.2us
I20260709 12:26:34.301663  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:34.303853  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:34.306644  5634 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:34.308019  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:34.311388  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:34.248021  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:34.313625  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:34.314495  5630 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:34.318773  5633 raft_consensus.cc:2410] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
W20260709 12:26:34.323761  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 216605us
I20260709 12:26:34.325183  5559 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:34.327132  5556 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:34.289171 10440 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:34.289849 10412 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.296427  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:34.331327  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 83376.3us
I20260709 12:26:34.333200 10270 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:34.336520 10454 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:34.342463 10270 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:34.342466 10454 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.343377 10454 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:34.299140 10047 raft_consensus.cc:2764] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:26:34.345116  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:34.347081  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:34.350723  5703 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:34.354591  5561 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:34.357817  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:34.365342  5634 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:26:34.335973 10413 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:34.348886 10496 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:26:34.374554  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:34.374483  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:34.376735  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:34.467239  5561 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:34.467842  5703 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:34.348747 10400 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:34.382160  5553 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:34.350015 10412 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:34.403443  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.428392  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:34.430070  6272 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:34.357113  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:34.431876  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:34.437851  5456 catalog_manager.cc:5697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 4 to 12, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 12 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:26:34.484927  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 127936us
I20260709 12:26:34.474372  5636 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:34.343868 10270 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.477222  5702 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:34.372687 10398 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:34.389338  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47910us
I20260709 12:26:34.489094  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:34.354660 10341 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 12:26:34.353632 10441 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 12:26:34.492256  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:34.499269  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:34.426964 10432 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.362258 10452 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.507097  6276 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: [7.8-11.9]
I20260709 12:26:34.527143  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:34.527845  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:34.543149  5561 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
W20260709 12:26:34.546430  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 250242us
W20260709 12:26:34.547341  5675 leader_election.cc:336] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.132s (SENT)
I20260709 12:26:34.497538 10458 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:34.398874 10461 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.109s
I20260709 12:26:34.555601  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:34.569630  5556 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: [11.8-11.8]
W20260709 12:26:34.572043  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.552986  5456 catalog_manager.cc:5697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:34.489650 10023 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.065s
I20260709 12:26:34.578343  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:34.590221  5634 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:26:34.534350 10288 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:34.544473 10265 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:34.503736 10336 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
I20260709 12:26:34.600158  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:34.601713  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:34.605311  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:34.612053  5627 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:34.599352  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:34.607363  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60070.2us
I20260709 12:26:34.505067  9844 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 12:26:34.570622 10446 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:26:34.613481  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.527864 10430 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 12:26:34.617158  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:34.609160  5456 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 3 to 11, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 11 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:26:34.617851  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:34.616763  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:34.576551 10493 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:34.631119  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:34.634939  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:34.637770  5710 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 11: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:34.618760  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:34.619445  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:34.622900 10273 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:34.497371 10400 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.640717  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:34.583129 10461 consensus_queue.cc:1243] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 12.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:26:34.534106 10432 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.645195  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:34.645843  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:34.506980 10398 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.644677  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:34.608388 10265 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.566905 10429 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:26:34.663918  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:34.577984 10411 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260709 12:26:34.653787  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:34.627897 10458 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:34.628427 10494 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:26:34.616936 10288 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.661204  6273 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:34.656986 10265 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:34.567471 10451 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
I20260709 12:26:34.665238  5456 catalog_manager.cc:5697] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 8, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 8 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:34.608166 10214 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260709 12:26:34.629249 10423 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:34.685056  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2384) took 1260 ms. Trace:
I20260709 12:26:34.691548  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:34.639976 10446 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:34.699224  5708 rpcz_store.cc:276] 0709 12:26:33.424433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:33.424610 (+   177us) service_pool.cc:224] Handling call
0709 12:26:33.432867 (+  8257us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:33.476971 (+ 44104us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:33.477006 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:33.477532 (+   526us) log.cc:844] Serialized 26 byte log entry
0709 12:26:33.599715 (+122183us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:33.599750 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:34.684647 (+1084897us) spinlock_profiling.cc:243] Waited 43.6 ms on lock 0x7b540006f754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:34.684694 (+    47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:34.684743 (+    49us) raft_consensus.cc:1685] finished
0709 12:26:34.684757 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:34.684978 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":122,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"spinlock_wait_cycles":100374784,"thread_start_us":121747,"threads_started":1,"wal-append.queue_time_us":148583,"wal-append.run_cpu_time_us":2242,"wal-append.run_wall_time_us":1008048}
I20260709 12:26:34.688989 10415 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:34.713222  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:34.647763 10487 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 12:26:34.712658  5456 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 6 to 10, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 10 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:26:34.758098  5559 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:34.760834  5456 catalog_manager.cc:5697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 5 to 8, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 8 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:34.705876 10492 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:34.764735  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:34.769865  5555 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:34.715847 10415 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:34.774859  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:34.779929  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:34.780556  5708 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:34.755748 10341 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.785897  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:34.769337 10492 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:34.710799 10423 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.789893  5456 catalog_manager.cc:5697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 6 to 8, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 8 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:26:34.794480  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:34.799280 10503 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.384s	user 0.000s	sys 0.006s
I20260709 12:26:34.800824  5693 tablet_service.cc:1558] Processing DeleteTablet for tablet 2acd6f5df0204b2295547e021cd8cb9f with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 5)) from {username='slave'} at 127.0.0.1:38456
I20260709 12:26:34.801483  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2119) took 1287 ms. Trace:
W20260709 12:26:34.801712  5675 leader_election.cc:336] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.190s (SENT)
I20260709 12:26:34.801946  5631 rpcz_store.cc:276] 0709 12:26:33.513553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:33.555294 (+ 41741us) service_pool.cc:224] Handling call
0709 12:26:34.801271 (+1245977us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b540032bed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:34.801409 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1262346496}
W20260709 12:26:34.802804  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.803403  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:34.804522 10494 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:34.805050 10494 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.806738 10288 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:34.806979  5457 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 272) took 45809 ms (client timeout 15000 ms). Trace:
I20260709 12:26:34.807871  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2281) took 1356 ms. Trace:
I20260709 12:26:34.808367  5558 rpcz_store.cc:276] 0709 12:26:33.451363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:33.451544 (+   181us) service_pool.cc:224] Handling call
0709 12:26:33.451984 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:34.173107 (+721123us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:34.807639 (+634532us) spinlock_profiling.cc:243] Waited 720 ms on lock 0x7b54001bd354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:34.807811 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1656140672}
W20260709 12:26:34.809250 10319 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.582s	user 0.003s	sys 0.007s
I20260709 12:26:34.809779 10319 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.811249  6278 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:34.795347 10341 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.813279 10431 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:34.813063 10415 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.814492 10415 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:34.815078 10341 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:34.815932  5456 catalog_manager.cc:5697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:34.777092 10473 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:26:34.816931 10509 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 12:26:34.817096 10473 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:34.817719  5631 raft_consensus.cc:2410] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:26:34.817816  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2283) took 1036 ms. Trace:
I20260709 12:26:34.817934 10473 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.818284  5557 rpcz_store.cc:276] 0709 12:26:33.781369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:33.905869 (+124500us) service_pool.cc:224] Handling call
0709 12:26:34.817616 (+911747us) spinlock_profiling.cc:243] Waited 854 ms on lock 0x7b54003c4bd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:34.817757 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1965052800}
I20260709 12:26:34.818737  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:34.809374 10507 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 12:26:34.834522 10369 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:34.813704 10431 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:34.820103  5708 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:34.804003  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:34.787567 10411 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.829056 10400 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:34.831100 10423 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:34.788040 10385 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:34.840098 10448 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.779s	user 0.007s	sys 0.012s
I20260709 12:26:34.843423 10446 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.835032 10369 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:34.790439 10470 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:34.781522 10412 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:34.790905 10458 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.791347 10023 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:34.853473  5456 catalog_manager.cc:5697] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:26:34.856813  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 52730.7us
I20260709 12:26:34.867293 10273 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:34.867393  5456 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 6 to 7, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 7 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:34.867740 10369 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.868889 10369 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:34.869585  5456 catalog_manager.cc:5697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 5 to 8. New cstate: current_term: 8 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:26:34.870899  5456 catalog_manager.cc:5697] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 7 to 9, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 9 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:26:34.871626  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:34.872784 10250 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:34.854540 10411 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.873384 10250 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.874289 10319 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:34.874819 10411 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:34.844748 10506 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.300s	user 0.001s	sys 0.000s
W20260709 12:26:34.852833 10511 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 12:26:34.851131 10452 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:34.847437 10473 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:34.876703  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:34.877948  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:34.878858 10452 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:34.879245 10452 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:34.879601 10452 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:26:34.851013 10510 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
W20260709 12:26:34.845158 10514 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:26:34.872270  5456 catalog_manager.cc:5697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 1 to 9. New cstate: current_term: 9 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:26:34.866140  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 71717.2us
I20260709 12:26:34.849448 10432 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:34.861377  9844 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:34.850772 10398 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:34.862049  5631 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:34.881057 10461 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:34.886310  5456 catalog_manager.cc:5697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:34.868047 10273 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:34.904045  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:34.876066 10250 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:34.922032 10494 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:34.922619 10494 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:34.923027  5443 catalog_manager.cc:5028] TS f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): tablet 2acd6f5df0204b2295547e021cd8cb9f (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:26:34.923161 10494 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.924264  5456 catalog_manager.cc:5697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 6 to 9, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 9 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:34.926532  5456 catalog_manager.cc:5697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 2 to 8, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 8 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
W20260709 12:26:34.807395  5457 rpcz_store.cc:269] 0709 12:25:48.951579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.954156 (+  2577us) service_pool.cc:224] Handling call
0709 12:26:34.755599 (+45801443us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:22.560737 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:22.561597 (+   860us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:22.561665 (+    68us) write_op.cc:435] Acquired schema lock
0709 12:26:22.561693 (+    28us) tablet.cc:662] Decoding operations
0709 12:26:22.579635 (+ 17942us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:22.579728 (+    93us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:22.579756 (+    28us) tablet.cc:685] Acquiring locks for 16 operations
0709 12:26:22.581072 (+  1316us) tablet.cc:701] Row locks acquired
0709 12:26:22.581123 (+    51us) write_op.cc:260] PREPARE: finished
0709 12:26:22.581294 (+   171us) write_op.cc:270] Start()
0709 12:26:22.581463 (+   169us) write_op.cc:276] Timestamp: P: 1783599982581232 usec, L: 0
0709 12:26:22.581510 (+    47us) op_driver.cc:348] REPLICATION: starting
0709 12:26:22.677593 (+ 96083us) log.cc:844] Serialized 6198 byte log entry
0709 12:26:23.059567 (+381974us) op_driver.cc:464] REPLICATION: finished
0709 12:26:23.229383 (+169816us) write_op.cc:301] APPLY: starting
0709 12:26:23.229534 (+   151us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:23.832926 (+603392us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:23.833052 (+   126us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:24.542890 (+709838us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:24.543425 (+   535us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=32,mrs_lookups=0
0709 12:26:24.543474 (+    49us) write_op.cc:312] APPLY: finished
0709 12:26:24.543981 (+   507us) log.cc:844] Serialized 148 byte log entry
0709 12:26:24.684622 (+140641us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:24.696940 (+ 12318us) write_op.cc:454] Released schema lock
0709 12:26:24.744605 (+ 47665us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:24.827240 (+ 82635us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:24.827789 (+   549us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":34811,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":102705,"apply.run_cpu_time_us":55435,"apply.run_wall_time_us":1562693,"cfile_cache_hit":62,"cfile_cache_hit_bytes":64790,"cfile_cache_miss":4,"cfile_cache_miss_bytes":6217,"cfile_init":2,"lbm_read_time_us":18650,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":10,"num_ops":16,"prepare.queue_time_us":374045,"prepare.run_cpu_time_us":96370,"prepare.run_wall_time_us":318953,"raft.queue_time_us":131248,"raft.run_cpu_time_us":5316,"raft.run_wall_time_us":23691,"replication_time_us":477864,"thread_start_us":597835,"threads_started":5,"wal-append.queue_time_us":345823,"wal-append.run_cpu_time_us":20476,"wal-append.run_wall_time_us":2232533}]]}
I20260709 12:26:34.910200  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:34.901265 10423 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 12:26:34.911821  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:34.916659  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:34.880813 10023 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:34.843765 10265 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.882961 10462 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:34.938629  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:34.935734  5631 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 10: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:34.936102  5557 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:34.935719  5456 catalog_manager.cc:5697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:26:34.944609  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:34.915587 10431 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.946669  5558 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:26:34.946274  5556 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:26:34.946274  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:34.947546  5456 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:34.949685  5456 catalog_manager.cc:5697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 5 to 9, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 9 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:26:34.952447  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 74568.4us
I20260709 12:26:34.953456  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:34.955147  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:34.903617 10288 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 12:26:34.954375  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2379) took 2351 ms. Trace:
I20260709 12:26:34.957931  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:34.958493 10451 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:26:34.959717 10214 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:34.960085 10214 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:34.958287  5705 rpcz_store.cc:276] 0709 12:26:32.596080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.596243 (+   163us) service_pool.cc:224] Handling call
0709 12:26:32.610453 (+ 14210us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:32.622217 (+ 11764us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:32.622249 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:32.802910 (+180661us) log.cc:844] Serialized 52 byte log entry
0709 12:26:32.962828 (+159918us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:33.348811 (+385983us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b48001701d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:26:34.399200 (+1050389us) log.cc:844] Serialized 14 byte log entry
0709 12:26:34.938277 (+539077us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:34.938361 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:34.938402 (+    41us) raft_consensus.cc:1685] finished
0709 12:26:34.947328 (+  8926us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:34.947469 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1514196,"prepare.queue_time_us":1463632,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":89226,"raft.run_cpu_time_us":13582,"raft.run_wall_time_us":729858,"spinlock_wait_cycles":9403648,"thread_start_us":580303,"threads_started":3,"wal-append.queue_time_us":572558,"wal-append.run_cpu_time_us":2442,"wal-append.run_wall_time_us":1003921}
I20260709 12:26:34.960916 10214 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:34.958776 10250 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:26:34.956038  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:34.961073  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:34.922648 10273 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:34.963003  5456 catalog_manager.cc:5697] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 3 to 6, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 6 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:34.965021  5553 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:34.940596 10259 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:34.966871  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:34.968174  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:34.969996  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:34.970688  5631 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:34.961117 10429 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 12:26:34.962862 10461 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:26:34.964274 10454 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.620s	user 0.013s	sys 0.001s
I20260709 12:26:34.962173 10470 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:34.988646  5456 catalog_manager.cc:5697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 5 to 8, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 8 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:34.992978  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:34.966833 10412 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:34.994824 10412 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:34.995216 10412 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:34.996698 10523 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:34.997179 10218 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:26:34.978825 10214 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:34.997617 10218 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:34.998160 10023 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.998720 10218 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.999449 10023 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:34.997540 10526 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:34.954339 10411 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:26:34.978586 10250 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:34.970917 10242 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:35.003182  5456 catalog_manager.cc:5697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 2 to 7. New cstate: current_term: 7 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:34.979633 10431 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:34.895023 10492 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.013792  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:34.970726 10259 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.017309 10495 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:26:35.019456 10499 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:35.021279 10265 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:35.021749  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:35.023433  5632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2373) took 2734 ms (client timeout 1950 ms). Trace:
I20260709 12:26:35.002084 10218 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:26:35.023847  5632 rpcz_store.cc:269] 0709 12:26:32.289287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.290721 (+  1434us) service_pool.cc:224] Handling call
0709 12:26:35.023231 (+2732510us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540032d2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:35.023343 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2484161792}
I20260709 12:26:35.024523  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:35.015647  5631 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:35.025012 10433 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:35.025570 10433 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:35.025909  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.027134  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.026695 10433 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:34.946360 10462 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.028290  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.041066 10445 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:35.041620 10445 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:35.025044  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.044255  5632 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 11: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:35.023072 10250 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.012084  5456 catalog_manager.cc:5697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:35.020028 10499 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:35.006217 10430 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:35.048895  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:35.064543  5456 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 4 to 7, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 7 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:35.067212  5553 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 11. Current term is 12. Ops: []
W20260709 12:26:35.073472  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 105339us
I20260709 12:26:34.992110 10454 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.074373  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:35.074537  5555 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
W20260709 12:26:35.075834  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.073643 10288 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:35.076185 10528 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:35.077298 10442 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:35.077636 10462 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:35.052938 10433 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:35.081765  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:35.065883  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:35.084679  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.086839 10526 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:35.119505  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.088954 10442 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:35.120468 10442 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:35.108023 10192 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:35.120496  5557 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:35.081219 10385 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:26:35.119145  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44845.4us
I20260709 12:26:35.088850  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.089561  5555 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:26:35.073827 10461 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:35.042488 10445 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.112993  5632 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:26:35.085206 10494 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:35.112576  5631 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:35.144706 10385 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:35.119622 10524 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:35.145378 10385 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:35.087625 10531 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:26:35.123807  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:26:35.112118 10413 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.633s	user 0.008s	sys 0.003s
W20260709 12:26:35.113245  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47448.3us
I20260709 12:26:35.097213 10423 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.129069 10417 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:35.164913 10417 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.167266 10417 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.129153  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:35.129707  5632 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: [13.8-15.9]
I20260709 12:26:35.088634 10288 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.122854 10230 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:35.124802 10461 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:35.169991 10288 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.170045 10230 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:35.127892 10270 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:35.170594 10230 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:35.131995 10336 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:35.171504 10336 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:35.174463  5633 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:35.171005 10270 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.126307 10499 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.176998 10270 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.177357 10499 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:35.124788  9945 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:35.182631  9844 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:35.182426  9945 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.183490 10423 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.170604 10288 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:35.186009 10423 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:35.188127  5555 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:35.189311  5631 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:35.195345 10230 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.133491 10532 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:35.121356 10192 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.131434 10454 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.092759  5628 raft_consensus.cc:2368] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13.
W20260709 12:26:35.154289  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.091475 10528 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.157236 10526 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:35.161363  5554 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 9. Current term is 10. Ops: []
W20260709 12:26:35.174696  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.195797 10436 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:35.197058  5705 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:35.198434  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.132332 10445 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:35.108577 10462 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:35.176139  5633 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:26:35.210001  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:35.214805  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.248831  5702 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:35.221666  5629 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
W20260709 12:26:35.224161  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 100448us
I20260709 12:26:35.227298  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.250744  5701 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:35.230921  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.229326 10473 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:35.231124  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.243620  5710 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:35.245545  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.222167 10358 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.247042  5704 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:35.251286  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:35.257368 10473 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:35.258706 10230 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.171021 10461 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.232582 10442 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.261970  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:35.262773  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:35.230453 10351 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.262786  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:35.258364 10473 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.252123  5557 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
W20260709 12:26:35.264745  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.268801  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:35.239725 10436 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:35.277529  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:35.277573  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.280377  5706 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 11: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:35.278549  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:35.279063  5710 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:26:35.280983  6276 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:35.283133  6277 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:35.284583  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.285929  5701 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 8. (term mismatch)
I20260709 12:26:35.286268  5701 pending_rounds.cc:85] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 7
I20260709 12:26:35.286475  5701 pending_rounds.cc:107] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:26:35.286701  5701 raft_consensus.cc:2889] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:35.286979  5701 pending_rounds.cc:107] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 11.9
I20260709 12:26:35.287184  5701 raft_consensus.cc:2889] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:35.229072 10336 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:35.289382 10336 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:35.230140 10532 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.291040  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:35.265403 10358 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.295069  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:35.294505 10192 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:35.295796  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:35.295249  9945 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.298454  6274 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:35.302441  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:35.298597  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.305965  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.298970  6273 raft_consensus.cc:2368] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13.
I20260709 12:26:35.300446  5704 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:26:35.272281 10337 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:35.270136 10470 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:35.306617  5707 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:35.306488  5702 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:35.321589  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:35.316915  5708 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:35.255702 10462 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.292696 10385 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.294301 10528 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:35.345089  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.306030 10273 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 12:26:35.358173 10273 consensus_queue.cc:1243] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:26:35.275260 10351 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.360915 10462 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:35.318006 10417 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:35.361745 10417 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:35.362214 10351 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:35.333287 10250 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.364007 10417 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:26:35.363765 10250 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.309983 10337 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:35.365824 10250 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.354841  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:35.355489 10369 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:35.356159 10398 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:26:35.357159  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.353227  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:35.376991  5706 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:35.274649 10412 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.381002  5560 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:35.385170  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:35.386972  5601 leader_election.cc:400] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13.
I20260709 12:26:35.387746  5601 leader_election.cc:403] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:35.399667  5709 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:26:35.345610 10470 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:35.374356 10528 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.419934  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.420696  5557 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:35.423079  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.424654  5701 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:35.425572  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.423554  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.427958  5709 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 13: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:35.422812  5706 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:35.434126 10358 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.435765  5706 pending_rounds.cc:85] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 4
I20260709 12:26:35.433811  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.440732  5706 pending_rounds.cc:107] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260709 12:26:35.441291  5706 raft_consensus.cc:2889] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:35.388343 10462 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 12:26:35.451635  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.457177 10470 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.415380 10270 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:26:35.393231 10532 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.479434  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:35.432075 10452 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.552s	user 0.013s	sys 0.002s
I20260709 12:26:35.480831  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:35.483139  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:35.476001  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:35.484920  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.295110 10526 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.486361  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.487053  5709 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:35.491794  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:35.423864 10528 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.486280 10452 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.498872 10452 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:35.499285 10534 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.486s	user 0.000s	sys 0.002s
I20260709 12:26:35.502631  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 891) took 1223 ms. Trace:
I20260709 12:26:35.490535 10192 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:35.503094  6278 rpcz_store.cc:276] 0709 12:26:34.279299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.279437 (+   138us) service_pool.cc:224] Handling call
0709 12:26:34.279744 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:34.811720 (+531976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:35.502380 (+690660us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b54006379d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:35.502568 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1221851392}
I20260709 12:26:35.503983 10192 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:35.505726 10192 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:35.506225  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2384) took 1224 ms. Trace:
I20260709 12:26:35.506610  5629 rpcz_store.cc:276] 0709 12:26:34.281803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.281940 (+   137us) service_pool.cc:224] Handling call
0709 12:26:35.506059 (+1224119us) spinlock_profiling.cc:243] Waited 873 ms on lock 0x7b540018c754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:35.506166 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2008089472}
I20260709 12:26:35.507650 10270 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13.
I20260709 12:26:35.512728  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.490227 10423 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:35.514104 10423 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:35.514480 10423 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:35.516535  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:35.378940 10436 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.520586 10493 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:35.521029 10493 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:35.522146  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.526301  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.526974  5629 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:35.529665  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.532121  5701 raft_consensus.cc:2410] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:26:35.532819  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:35.536289 10477 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.537675 10493 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.538066  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:35.536530 10542 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 12:26:35.541332  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.547753  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.548473  5629 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:35.549762  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:35.549925  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:35.550412  5629 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:35.551285  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:35.542896  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.551878  5627 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:35.518292 10337 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.558677  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:35.564507  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:35.566468  5633 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:35.569095  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:35.575764 10477 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:26:35.583680 10023 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.584s	user 0.009s	sys 0.000s
I20260709 12:26:35.586229  5710 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:35.592018  5705 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:35.572764  9932 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:35.572186 10336 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:35.616531  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:35.621081  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 129359us
I20260709 12:26:35.589327  9945 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:35.633988  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.610607 10412 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:35.626783  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.591958 10337 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:35.668569  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.653043 10521 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:35.670380 10521 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.682128  5633 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:35.615334  9932 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:35.660846 10445 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.695580  5629 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:35.693457  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:35.701006  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:35.697248  5627 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:35.689927  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:35.702979  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:35.703539  5553 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:35.652647 10446 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:35.645249 10453 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:35.703743  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.708165  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:35.708444  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.711921  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:35.704259  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.708660  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:35.717339  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:35.724078  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:35.725098 10358 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:35.726075 10358 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:35.727309 10358 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:35.728048  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.728758  5631 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:35.729871  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.726195 10492 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:35.733604  5631 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 11: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:35.622908 10336 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.723986  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:35.720688  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:35.725106  5636 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 10: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:35.734977  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:35.704896  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.704574 10412 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.699505  9932 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:35.706650 10521 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.723356 10453 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.716045 10445 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.711104 10496 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:35.666261  9945 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.750087 10477 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:35.706825 10417 raft_consensus.cc:2764] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:26:35.771728  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:35.754944 10351 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:35.736331 10528 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:35.711058 10043 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:35.760226  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:35.738044 10265 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.035s	sys 0.035s
I20260709 12:26:35.706686 10440 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:35.741492 10431 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:26:35.696790 10499 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:35.738628  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:35.773526  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.758389 10336 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:35.714648 10446 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:35.779935 10265 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.035s	sys 0.035s
I20260709 12:26:35.781318 10452 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:35.783650 10440 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:35.775776 10351 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:35.773479 10496 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:35.785149 10259 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:35.789425  5706 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:26:35.793772  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:35.812175  5706 raft_consensus.cc:606] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:35.812588  5706 raft_consensus.cc:614] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:35.800900  5631 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:35.814596  5631 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 12:26:35.801657  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63096us
I20260709 12:26:35.757073 10412 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.848174  5627 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:35.849128 10412 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.805119  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.850131  5558 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:35.851166  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.806392  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.853255  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.854385  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:35.854959  5559 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:35.807834  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.852169  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.856494 10431 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:26:35.857568 10431 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:35.806524  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:35.858469  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52026.8us
I20260709 12:26:35.859638  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:35.860867  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.861510  5704 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:35.861646  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2139) took 1096 ms. Trace:
I20260709 12:26:35.862224  5628 rpcz_store.cc:276] 0709 12:26:34.765659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.871337 (+105678us) service_pool.cc:224] Handling call
0709 12:26:35.861362 (+990025us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b540009ecd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:35.861554 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":504438144}
I20260709 12:26:35.800971  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.864223 10336 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:35.821815  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:35.821673 10440 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.795037 10499 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.823377 10496 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.813279 10369 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:35.778825 10477 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:35.800316 10423 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:35.868394  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:35.828688  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.829730  5555 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:35.876156  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:35.831718 10442 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.793699 10445 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:35.837471 10504 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.143s	user 0.006s	sys 0.008s
W20260709 12:26:35.838227 10556 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 12:26:35.838992 10557 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:26:35.840495  9945 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:35.841496 10453 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:35.814057 10351 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:35.841693  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.918381  5706 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:26:35.842784  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.844218  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:35.845851 10446 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:35.852113  5627 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:35.857906 10431 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:35.819265  5631 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:35.889459 10487 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.865810  5554 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:35.872292  5630 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
W20260709 12:26:35.877198  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.882256  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.878895  5633 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
W20260709 12:26:35.942204  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.883618  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.889472  6274 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:35.894120  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:35.879067  5675 leader_election.cc:400] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13.
W20260709 12:26:35.946120  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 67172.7us
I20260709 12:26:35.892115  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:35.896510  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:35.949440  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53079.3us
I20260709 12:26:35.950136  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:35.950544  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56587.2us
I20260709 12:26:35.951939  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:35.954087  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:35.894915  5559 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:35.902667  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:35.906159  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:35.908322  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:35.908828  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.956759  6277 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:35.909564  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:35.910887  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:35.914062  6278 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:26:35.915350  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:35.915815  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.964944  5632 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:35.917061  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.919868  5709 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:35.864992 10281 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:35.920967  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:35.922701  5629 raft_consensus.cc:2368] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14.
I20260709 12:26:35.873145 10369 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:35.870986 10499 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:35.864833 10336 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:35.864315  5628 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:26:35.916270 10412 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:35.958313  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:35.994716  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 46352.3us
I20260709 12:26:35.974849  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.976574  5557 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:26:35.974910  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 83030.5us
I20260709 12:26:35.997057  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:35.958420  6280 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:35.983306  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:36.009826  5627 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:26:35.981276 10369 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:35.976019 10281 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:35.980713  9945 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:35.995412 10549 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:35.873773 10477 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:35.995544 10288 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.980572 10417 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:35.993935 10414 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:36.007030 10528 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.976893 10487 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.026866  5631 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
W20260709 12:26:36.028079  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 145931us
I20260709 12:26:36.032922 10358 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:36.004468 10473 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:36.061717 10473 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.062279 10473 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:36.065229 10492 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:26:35.980460 10526 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:26:36.065333 10433 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:36.066805 10492 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:36.067005 10526 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:36.067327 10492 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:36.067631 10526 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:35.926764 10452 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:36.065549 10023 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:26:36.054535  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:36.019445 10281 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:36.070324  6280 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:26:36.068709  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:36.069504  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:36.072674  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:36.074152  5631 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:36.004961 10412 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.071739  6271 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:36.023936 10549 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.041178 10265 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:36.089167  5631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:36.094540  5631 raft_consensus.cc:606] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:36.054076 10358 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:36.077047 10499 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:36.129416  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:36.099936  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2290) took 1840 ms. Trace:
I20260709 12:26:36.096184 10549 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:36.098260  5631 raft_consensus.cc:614] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:36.130645  5560 rpcz_store.cc:276] 0709 12:26:34.259565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.261315 (+  1750us) service_pool.cc:224] Handling call
0709 12:26:34.261618 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:35.381474 (+1119856us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:36.099618 (+718144us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54000ec4d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:36.099863 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2566617344}
I20260709 12:26:36.141320  6271 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 13: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:26:36.146030  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 149362us
I20260709 12:26:36.029943 10431 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.149152  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:36.154049  5706 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:36.096580 10433 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:36.048924 10288 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.068677 10526 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.074573 10218 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
I20260709 12:26:36.096885 10265 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:36.069868 10528 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.128623 10358 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:36.165601  5706 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:36.128707 10250 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:36.170187  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 220125us
W20260709 12:26:36.148212  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:36.137163 10415 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:36.180249  5631 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:36.151643 10499 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:36.164319 10433 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:36.091737 10336 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:36.189571  5631 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:26:36.188791  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:36.190971  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:36.192337  5633 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:36.144199 10023 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:26:36.184675 10431 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:36.199242 10250 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.205356  5629 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:26:36.207134  5629 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:26:36.194829  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:36.198539 10415 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:36.195343 10433 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:36.210187 10453 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:36.210593 10453 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:36.210968 10453 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:26:36.214694  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 467053us
I20260709 12:26:36.215425  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:36.174953 10265 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:36.224798  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:36.098984 10492 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.177608 10423 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:36.242132 10414 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.238370  6236 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:36.247388  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:36.218834  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:36.243396  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:36.244797 10532 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:36.216239  5554 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
W20260709 12:26:36.252249  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:36.252395  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 163121us
I20260709 12:26:36.253575  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:26:36.255110  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 263700us
I20260709 12:26:36.234861 10218 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:36.149401 10533 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.061s	user 0.004s	sys 0.015s
I20260709 12:26:36.269152  6279 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:36.282608 10417 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:36.289364 10563 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
W20260709 12:26:36.291689 10562 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:26:36.278076  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:36.311468  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:36.250415 10526 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:36.257431 10270 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:36.251871 10423 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.277427 10496 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.121s
I20260709 12:26:36.313673 10218 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.269321  9932 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:36.315825 10417 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:36.268677 10568 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.180s
I20260709 12:26:36.308918 10528 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:36.267987 10442 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:36.433231  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 244523us
W20260709 12:26:36.466012  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 207330us
W20260709 12:26:36.308028 10533 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.220s	user 0.005s	sys 0.015s
I20260709 12:26:36.515031  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2404) took 1083 ms. Trace:
I20260709 12:26:36.317205 10492 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:36.303462 10452 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:36.258169 10414 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:36.259923 10532 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:36.520994  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 243002us
W20260709 12:26:36.567607  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:36.646215  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:36.647013  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:36.422538 10499 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:36.536690 10564 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.371s	user 0.000s	sys 0.002s
I20260709 12:26:36.578148 10270 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:36.514124 10336 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:26:36.588281  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:36.536926 10552 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:36.602054  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:26:36.666455  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 64529us
I20260709 12:26:36.617259  5701 rpcz_store.cc:276] 0709 12:26:35.423565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:35.425300 (+  1735us) service_pool.cc:224] Handling call
0709 12:26:36.447088 (+1021788us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54002d14d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:36.506720 (+ 59632us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":242072448}
W20260709 12:26:36.576960 10337 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.911s	user 0.011s	sys 0.003s
I20260709 12:26:36.619347  5601 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:36.640233  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:36.673194  6274 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 14
W20260709 12:26:36.653357  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:36.593598 10487 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:36.589968 10577 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.529934 10398 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:26:36.630587 10581 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.551255 10423 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.631659 10452 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.691237  6279 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:26:36.651119 10532 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:36.568709 10473 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.642427 10412 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:36.650499 10259 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.705855  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:36.730139  6274 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:36.754660  6276 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:36.703230 10337 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.642756 10549 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:36.777433 10549 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:36.767073 10580 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:26:36.721381 10577 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:36.779994 10577 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:36.780606 10577 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:36.781147 10577 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.783517 10577 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:36.784034 10577 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:36.784660  9932 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:36.784663 10577 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.785732 10288 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:36.787273 10577 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:36.729506 10496 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.790658 10496 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.792723 10577 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:36.793262 10577 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.793794 10496 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:36.722115 10341 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:36.794903 10341 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.795416 10341 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:36.796543 10192 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:36.796744 10575 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:36.797348 10192 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:36.798882 10577 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:36.801393  9945 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.779s	user 0.015s	sys 0.001s
I20260709 12:26:36.801908  9945 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:36.803642 10218 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:36.805079 10453 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.594s	user 0.012s	sys 0.003s
I20260709 12:26:36.805613 10453 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:36.809485 10423 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:36.815243  5705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2400) took 1480 ms (client timeout 1733 ms). Trace:
W20260709 12:26:36.815986  5705 rpcz_store.cc:269] 0709 12:26:35.334802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:35.339350 (+  4548us) service_pool.cc:224] Handling call
0709 12:26:36.814949 (+1475599us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54002f4c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:36.815134 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":494548736}
W20260709 12:26:36.748703 10565 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.480s	user 0.001s	sys 0.000s
I20260709 12:26:36.774415 10414 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:36.799504 10192 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:36.729655 10452 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:36.787536 10417 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:36.789723 10576 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.201s	user 0.001s	sys 0.000s
I20260709 12:26:36.758062 10413 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:36.850792 10351 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.876s	user 0.014s	sys 0.000s
I20260709 12:26:36.851409 10351 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:36.854663 10433 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.645s	user 0.008s	sys 0.004s
I20260709 12:26:36.855211 10433 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:36.839892 10218 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:36.856814 10218 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.857105 10415 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:36.858155 10351 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:36.859022  6275 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:36.859786  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2407) took 1154 ms. Trace:
I20260709 12:26:36.860183  5703 rpcz_store.cc:276] 0709 12:26:35.704967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:35.713397 (+  8430us) service_pool.cc:224] Handling call
0709 12:26:35.713761 (+   364us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:35.714234 (+   473us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:35.714275 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:35.733972 (+ 19697us) log.cc:844] Serialized 52 byte log entry
0709 12:26:35.931694 (+197722us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:35.981697 (+ 50003us) log.cc:844] Serialized 14 byte log entry
0709 12:26:35.981978 (+   281us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:35.982085 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:36.217904 (+235819us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b18003c5920. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:36.852370 (+634466us) spinlock_profiling.cc:243] Waited 3.08 ms on lock 0x7b18003c5920. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:36.859536 (+  7166us) raft_consensus.cc:1685] finished
0709 12:26:36.859558 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:36.859720 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3017,"prepare.queue_time_us":50393,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":7177,"raft.queue_time_us":882,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":14146,"spinlock_wait_cycles":31142528,"thread_start_us":160862,"threads_started":1,"wal-append.queue_time_us":259421}
W20260709 12:26:36.864845 10571 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.298s	user 0.000s	sys 0.003s
I20260709 12:26:36.858999 10218 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:36.868649  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:36.870651 10433 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:36.875280 10341 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.879402  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2141) took 2102 ms. Trace:
I20260709 12:26:36.879853  5630 rpcz_store.cc:276] 0709 12:26:34.777483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:35.165666 (+388183us) service_pool.cc:224] Handling call
0709 12:26:35.872036 (+706370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:35.940499 (+ 68463us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:36.879204 (+938705us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b540009ecd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:36.879337 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1623807872}
I20260709 12:26:36.882159 10473 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:36.882599 10341 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:36.828584 10250 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:36.827884 10337 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.884145 10493 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:36.763038 10259 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:36.885497 10442 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:26:36.887949 10579 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.075s	user 0.000s	sys 0.002s
W20260709 12:26:36.890142 10265 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.639s	user 0.013s	sys 0.000s
W20260709 12:26:36.900990  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 233543us
W20260709 12:26:36.931725  5710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2402) took 1587 ms (client timeout 1807 ms). Trace:
W20260709 12:26:36.943636  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:36.949070  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:36.949468  5703 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:26:36.950023  5636 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:36.950227  5703 raft_consensus.cc:606] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:36.950760  5703 raft_consensus.cc:614] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:36.948238  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:36.952903  5630 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:36.947665  5710 rpcz_store.cc:269] 0709 12:26:35.344679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:35.344806 (+   127us) service_pool.cc:224] Handling call
0709 12:26:36.931357 (+1586551us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b54002f4254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:36.931615 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":524750208}
I20260709 12:26:36.957203  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:36.958580  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:36.916213 10453 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:36.890877 10413 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:36.840191 10270 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:36.959894 10369 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.923s	user 0.011s	sys 0.003s
I20260709 12:26:36.977923  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2156) took 1100 ms. Trace:
I20260709 12:26:36.978206  6236 leader_election.cc:400] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14.
I20260709 12:26:36.978696  6236 leader_election.cc:403] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:36.978477  5633 rpcz_store.cc:276] 0709 12:26:35.877772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:35.974508 (+ 96736us) service_pool.cc:224] Handling call
0709 12:26:36.977704 (+1003196us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b540018d154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:36.977845 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":452189568}
I20260709 12:26:36.980152  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:36.982158 10043 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14.
W20260709 12:26:36.982731 10281 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.895s	user 0.010s	sys 0.003s
I20260709 12:26:36.981827  5630 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:36.982821 10043 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:36.997350  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:36.975291  5636 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:36.999732  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:36.966650 10591 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.987820  5635 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:36.947389 10265 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.004892 10043 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:36.968266 10341 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:26:36.976188 10431 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:36.967890 10398 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 12:26:37.017674  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.041101  5703 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:37.020995  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.042062  5636 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:37.042364  5710 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:26:37.033972  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.042870  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:37.038264  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:37.038559  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:37.039240  5629 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:26:37.042418  5636 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:37.047206  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.047811  5706 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:37.048593  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:36.730791 10581 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.051544  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:37.051688 10496 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.051875  5633 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:26:37.076774 10496 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.079638 10496 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:37.048853 10585 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.048666 10250 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:37.053108  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:37.054450  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.057560 10259 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:37.058184 10595 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:37.059696 10337 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.062402  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:37.065160  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.065834 10413 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.066239 10454 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:37.103307  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:37.046968  5636 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.071995  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:37.129163  5705 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:37.129040  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:36.983108 10351 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.058s
I20260709 12:26:36.985800 10369 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.092095  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:37.135983  5634 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:26:37.093600  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:37.093613  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 492349us
I20260709 12:26:37.097075  5709 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:36.988147 10281 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.141532  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.098497  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2411) took 1552 ms. Trace:
I20260709 12:26:37.067142  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:37.080437 10445 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:26:37.104789  5525 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.017s	sys 0.005s
W20260709 12:26:37.144248  5525 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.017s	sys 0.005s
I20260709 12:26:37.144575  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:37.106945 10595 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:26:37.106793  5676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.852s	user 0.008s	sys 0.016s
W20260709 12:26:37.150475  5676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.896s	user 0.008s	sys 0.016s
W20260709 12:26:37.151403  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 897872us
I20260709 12:26:37.152931  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.153419  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.110415  5701 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b21feec8630479b94ed9384256483fc"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:26:37.154455  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:37.095136 10415 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:26:37.130522  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:37.097348 10250 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:37.100013  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 480750us
I20260709 12:26:37.132607  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2154) took 1263 ms. Trace:
I20260709 12:26:37.142895  5628 rpcz_store.cc:276] 0709 12:26:35.546541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:36.013622 (+467081us) service_pool.cc:224] Handling call
0709 12:26:36.301160 (+287538us) spinlock_profiling.cc:243] Waited 6.47 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:36.301348 (+   188us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:36.301688 (+   340us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:26:36.301969 (+   281us) log.cc:844] Serialized 14 byte log entry
0709 12:26:37.097992 (+796023us) log.cc:844] Serialized 14 byte log entry
0709 12:26:37.098223 (+   231us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:37.098268 (+    45us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:26:37.098319 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:37.098365 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:37.098454 (+    89us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3870,"prepare.queue_time_us":5011,"prepare.run_cpu_time_us":5848,"prepare.run_wall_time_us":173620,"raft.queue_time_us":101633,"raft.run_cpu_time_us":655,"raft.run_wall_time_us":153304,"spinlock_wait_cycles":31326464,"thread_start_us":795279,"threads_started":2,"wal-append.queue_time_us":636474}
I20260709 12:26:37.170802  5635 rpcz_store.cc:276] 0709 12:26:35.869130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:35.996731 (+127601us) service_pool.cc:224] Handling call
0709 12:26:36.868438 (+871707us) spinlock_profiling.cc:243] Waited 5.69 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:26:37.132338 (+263900us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54000c2954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:37.132502 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":285288704}
I20260709 12:26:37.100013 10337 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.109006 10473 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:26:37.134473  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:37.138648  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.139201  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.184890  6279 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:37.185879  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.186482  5558 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:37.187271  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:37.141310  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47809.5us
I20260709 12:26:37.144663  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:37.188825  6275 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:37.147207  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:37.188870  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:37.189981  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.191179  5552 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:37.191716  5636 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:37.075169 10043 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:37.193023 10043 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.148736  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:37.194485  6273 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:37.154809  5701 raft_consensus.cc:606] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:37.195151  5701 raft_consensus.cc:614] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:37.155393  6272 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:37.154309  5560 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:37.198263  5701 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "addbcf2919194d2e866b5e8826422851"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:26:37.017396 10591 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.095378 10585 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.188222  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.200016  6271 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:26:37.200459  5558 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:37.159471  6276 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:37.200989 10591 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.130739  5710 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:37.161099  6280 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:26:37.201018 10585 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.203985 10585 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.204035 10251 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.204542 10585 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.204639 10251 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.206562 10585 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.206712 10251 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.207046 10585 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.207285 10251 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:37.207698 10251 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.207563 10585 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.165591  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:37.166790  5557 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:37.211818 10440 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.212457  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:37.213153  5635 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:26:37.214149  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:37.214421  5635 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20260709 12:26:37.214553  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 147507us
I20260709 12:26:37.215240 10412 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:37.215410 10414 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:37.215922 10412 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.216198 10414 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:26:37.217007 10369 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:37.217700 10552 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:37.218055 10412 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.218472  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:37.219352  5628 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:37.219318 10281 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:37.214414  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:37.220966  9932 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.221428  9932 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.221812  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:37.222848 10265 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:37.222944 10573 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:37.143994 10445 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:37.223901 10573 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:26:37.224138 10445 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.225170 10445 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:37.213865  5629 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:26:37.226548  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:37.227329  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:37.227361  6272 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:37.227931  6273 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:37.219244 10446 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.010s
I20260709 12:26:37.228850 10595 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.215251  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:37.229058  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:37.230779  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:37.231643 10417 raft_consensus.cc:2764] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:26:37.233336  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:37.162307  5561 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I20260709 12:26:37.230927  5628 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
W20260709 12:26:37.134600  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.215191  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:37.235579  5630 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:37.235698  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:37.176076  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.107340 10454 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.175890 10398 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:37.175889 10337 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.180120 10567 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:37.148656 10496 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:37.208081 10251 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:37.234603  5561 pending_rounds.cc:85] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 10
I20260709 12:26:37.256850  5561 pending_rounds.cc:107] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 12:26:37.257195  5561 raft_consensus.cc:2889] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:37.146442 10358 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.954s	user 0.014s	sys 0.000s
I20260709 12:26:37.170789 10461 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:37.238173  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:37.239241 10192 raft_consensus.cc:2764] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:26:37.237398  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:37.291558  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 54215.9us
W20260709 12:26:37.291275  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53187.3us
I20260709 12:26:37.282964  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:37.154146  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:37.155141 10533 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:37.246291  6279 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:37.248071  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:37.250191  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.162927 10487 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:37.292936  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:37.242024  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:26:37.301957  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 147882us
I20260709 12:26:37.258651  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:37.263880  5634 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:37.218072 10552 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:37.312153  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:37.242276 10281 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:37.313500  5676 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:37.322149  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.322551  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:37.323834  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:37.314560  5630 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:37.320681  6276 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 12:26:37.330004  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 88054.5us
W20260709 12:26:37.330047  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 71482.1us
I20260709 12:26:37.331274  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:37.328820  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:37.242727 10398 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:37.333938 10575 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:37.342455 10575 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.342792 10452 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.343379 10585 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.343185 10552 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.344586 10552 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:37.321763 10577 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:37.348729 10604 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:37.349419 10604 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.349763 10604 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:37.312726 10442 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:37.312417 10487 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.352681 10442 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.353219 10442 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:37.321237 10281 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.353700 10281 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:37.354511 10487 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:37.355049 10606 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:37.355659 10492 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.722s	user 0.006s	sys 0.006s
I20260709 12:26:37.355573 10606 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.342833 10575 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:37.357841 10606 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:37.359552 10250 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:37.290562 10499 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:37.356736  5556 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:37.313808 10493 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:26:37.361016  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:37.361524  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48104.5us
I20260709 12:26:37.370579  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:37.328131  9844 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:26:37.377031  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45901.5us
I20260709 12:26:37.333698  9945 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:37.258437 10440 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.341485 10023 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:26:37.388659  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57110.9us
I20260709 12:26:37.340854 10473 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:37.396605  5561 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:26:37.399013  5561 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:26:37.400409  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:37.400707  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.401678  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.405298  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.405395  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:37.360894 10336 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:37.348760 10577 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:37.371920 10526 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:37.369928 10436 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:37.458814  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.301846 10358 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.378163 10606 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:37.407403  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.368822 10431 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:37.471182  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.433674  5630 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:26:37.377120 10499 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.456223  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2413) took 1903 ms. Trace:
I20260709 12:26:37.492920  5554 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:37.432163  9945 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:37.457406 10259 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:37.309067 10461 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.372860 10493 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:37.341069 10528 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 12:26:37.383951  9844 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.263432 10567 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.411294  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:37.514451  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 103247us
I20260709 12:26:37.515515  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:37.410010  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.431778  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:26:37.388469 10470 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.727s	user 0.004s	sys 0.057s
I20260709 12:26:37.474577 10591 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:26:37.407552  5552 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:37.523316  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.524272  5552 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:37.525585  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:37.526317  5552 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:37.470018 10526 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:37.470139 10606 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.462184 10336 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.537352 10336 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:37.469986  5560 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:37.477311  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:37.470693  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:37.470664  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 280626us
I20260709 12:26:37.540766  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.388700 10023 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:37.543939 10023 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:37.545643  5552 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:37.456489 10492 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:26:37.548520  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 891) took 1452 ms. Trace:
I20260709 12:26:37.548696 10492 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.549381 10492 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:37.549256  6280 rpcz_store.cc:276] 0709 12:26:36.097765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:36.119304 (+ 21539us) service_pool.cc:224] Handling call
0709 12:26:36.153455 (+ 34151us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:37.202649 (+1049194us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:37.548047 (+345398us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54005f5d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:37.548428 (+   381us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2316676864}
I20260709 12:26:37.452243 10288 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.551856 10288 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.553344 10413 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:26:37.554112 10288 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.432209 10597 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:37.554266 10413 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:37.492059  5627 rpcz_store.cc:276] 0709 12:26:35.553028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:36.010971 (+457943us) service_pool.cc:224] Handling call
0709 12:26:37.001514 (+990543us) spinlock_profiling.cc:243] Waited 3.41 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:37.001732 (+   218us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:37.001930 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:26:37.033306 (+ 31376us) log.cc:844] Serialized 14 byte log entry
0709 12:26:37.455761 (+422455us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:37.455841 (+    80us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:37.455919 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:37.456012 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:37.456156 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38007,"prepare.queue_time_us":37651,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":39454,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":69,"spinlock_wait_cycles":7840512,"thread_start_us":338397,"threads_started":1}
I20260709 12:26:37.554894 10413 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:37.405491 10601 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:37.555692 10601 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:37.408872  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:37.406373  5561 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:37.556733 10601 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.558293 10567 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.459818 10473 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:37.411782 10454 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.461308 10577 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:37.559146 10412 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:26:37.559387 10473 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.459396 10600 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.099s
I20260709 12:26:37.483654 10431 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:37.524078  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 92427.6us
I20260709 12:26:37.478440 10598 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
W20260709 12:26:37.521991 10470 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.868s	user 0.004s	sys 0.057s
I20260709 12:26:37.470562  5558 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:37.554338  9932 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.557129  5557 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:37.507117 10452 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.509788 10493 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:37.470274 10398 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.492493 10581 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:37.518608  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:37.573568  5705 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:37.576964  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:37.584482  5706 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:37.588728  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.532088 10526 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:37.530002 10414 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:37.313694 10533 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.533360 10453 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:37.595355  6279 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:37.564906 10606 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:37.563097 10358 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.573197  9844 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:37.565447 10270 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:26:37.507809 10259 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.586908 10250 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 12:26:37.607267  5701 raft_consensus.cc:606] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:37.669970  5701 raft_consensus.cc:614] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:37.617658  5703 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:37.623796  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.671392  5705 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:37.628027  5708 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:26:37.578831 10585 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 12:26:37.578648 10569 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:26:37.616501 10414 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:37.650055  5561 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:37.680701  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:37.597126 10446 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:37.658008  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:37.648948  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:37.555727 10413 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.614759  9932 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 12:26:37.586719 10477 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260709 12:26:37.674028  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.690215  5558 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:37.631923  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.677423  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:37.692507  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.692660  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:37.693005  5703 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:37.591323 10614 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.082s
W20260709 12:26:37.697350  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 120459us
I20260709 12:26:37.693739  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.698720  5708 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:37.601981 10596 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.076s
I20260709 12:26:37.601457 10605 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.097s
I20260709 12:26:37.685875  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:37.720602  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.750265  5708 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:37.751541  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:37.664273 10337 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:37.720947  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.752907  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.704063  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:37.754285  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:37.754459  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 50472.1us
I20260709 12:26:37.755393  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:37.755535  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.716434 10631 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:37.726979  5703 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:37.763851  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.764533  5703 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:37.610069  9945 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.729333 10192 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:37.732153  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:37.753187  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.756667  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:37.687935 10414 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:37.759984  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 74213.7us
W20260709 12:26:37.725359  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 73139.5us
I20260709 12:26:37.769882 10043 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:37.727398 10454 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:37.774897 10415 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:37.763131 10631 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:37.769680  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.769547 10499 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.701215 10598 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.696112 10440 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.691594 10612 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:37.838097 10612 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.838500 10612 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:37.725072 10470 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:37.756171  5705 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:37.797170 10602 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:26:37.789093  5526 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:37.795559  5706 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 14: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:37.787556 10487 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:26:37.709600 10270 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.765048  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:37.753446 10573 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:26:37.773993 10560 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:26:37.771096  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.769598 10265 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:26:37.851688  5708 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:26:37.848045  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:26:37.855737  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:37.812994  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.774461 10413 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:37.815222 10597 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:37.823050 10575 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.827486 10442 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.796948 10528 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:37.834043  5558 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:37.834658  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:37.788283 10599 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260709 12:26:37.836609 10337 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:26:37.837307 10533 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:37.837536  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:37.850597 10573 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.766798 10423 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:26:37.800233 10630 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.801964 10417 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:37.858695  5702 raft_consensus.cc:2410] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:26:37.722420 10446 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.804903 10043 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:37.854597  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:26:37.843830 10626 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260709 12:26:37.812134 10461 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.867646  5708 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
W20260709 12:26:37.868929  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 79890.4us
I20260709 12:26:37.862794 10453 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:37.835383  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:37.875754  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 110770us
I20260709 12:26:37.937894  5554 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:37.938014  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:37.873881  5558 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:37.880957  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46372.4us
W20260709 12:26:37.883605  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 92614.5us
I20260709 12:26:37.939172  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:26:37.795773  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63682.3us
I20260709 12:26:37.877957 10431 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:37.798516 10192 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.947039  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:37.951282  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:37.897639  5634 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:26:37.891999  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.939100  5558 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.852725 10270 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:37.836205 10281 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.865654 10581 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.839848 10023 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:37.879562 10604 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.529s	user 0.009s	sys 0.004s
I20260709 12:26:37.912886 10528 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.950008 10599 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:37.885104 10477 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:37.934847  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:37.957598 10452 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:26:37.908007 10532 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.065s	user 0.013s	sys 0.000s
I20260709 12:26:37.907155  5561 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:37.940326  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:37.847384 10598 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:37.920075 10573 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:37.991153  5554 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:37.955564  5558 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:37.863915 10607 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.301s	user 0.001s	sys 0.000s
I20260709 12:26:37.974227 10265 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:37.875691 10625 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:37.951279 10552 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.606s	user 0.014s	sys 0.000s
I20260709 12:26:37.925207 10417 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:37.998868  5634 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:26:37.937290 10622 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:37.932549  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 91168.5us
I20260709 12:26:37.954036 10461 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:37.980736  5706 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:37.878129 10415 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:26:37.982649  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 47889.8us
I20260709 12:26:37.988302 10351 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.973690  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:38.054605  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 80986.6us
I20260709 12:26:37.906785 10588 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:37.991856  5554 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:37.945726 10533 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:26:37.969220 10453 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.988914 10599 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:38.061067 10192 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.062098 10599 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.062815 10528 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:37.980036 10259 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:37.990485 10605 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:37.876323 10597 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.937777 10621 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:38.039208  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 98946.3us
I20260709 12:26:38.041225  5705 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:26:38.044395 10417 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:38.045557 10423 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:26:37.989986 10624 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.256s	user 0.001s	sys 0.000s
I20260709 12:26:37.989975 10623 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:38.049759  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:37.977041 10608 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.325s	user 0.000s	sys 0.002s
I20260709 12:26:37.931293 10630 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.093425 10630 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.094393  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:37.922828 10631 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:37.977237 10581 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.095726  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:38.055307  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:38.099340  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44103.3us
I20260709 12:26:38.056334  5554 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:38.063447  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 79647.2us
W20260709 12:26:38.043460 10613 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.296s	user 0.000s	sys 0.001s
I20260709 12:26:37.978873 10604 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:37.987985 10452 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:38.082795  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:38.066764 10605 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:26:38.065913 10577 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.506s	user 0.008s	sys 0.004s
I20260709 12:26:37.990741 10532 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.064613 10602 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:38.119628 10577 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:38.097858  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 48173.8us
I20260709 12:26:38.121383 10532 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.121690 10577 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:38.122089  6273 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:38.058549 10588 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:38.122861 10588 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:38.123150  6279 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:26:38.123857  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:38.124542  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:38.125464  5709 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:38.127815  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.128791  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.128902  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.086808 10423 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.083704 10601 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:38.046458 10499 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:26:38.048391 10461 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:38.122781  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:38.109779  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:38.100581  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:38.130997  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.141459  5627 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:38.131481  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.133394  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:37.992417 10043 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.101984 10265 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:38.138780  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:38.143615  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:38.144677  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:38.108774 10477 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:38.090359 10623 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.177250  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:38.149327  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:38.177953  5555 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:38.177802  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:38.150228  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:38.065596 10351 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.117504 10452 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:38.130968 10605 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:38.168991  5634 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:38.089425 10412 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 12:26:38.170370  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:38.173312  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:38.147133 10619 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.272s	user 0.000s	sys 0.001s
I20260709 12:26:38.048431 10622 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.179898  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.200718  6280 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:38.166522  5557 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:38.137127 10423 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:38.120302 10602 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:38.184182  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.146488 10461 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:38.185573  5556 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:26:38.046720 10625 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.192309  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:38.194172  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:38.151437 10415 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:38.167361 10643 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:38.183099 10445 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.954s	user 0.013s	sys 0.000s
I20260709 12:26:38.189421 10446 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.184262 10623 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:38.205247  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.228590  5636 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:38.229923  5636 pending_rounds.cc:85] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 6
I20260709 12:26:38.230307  5636 pending_rounds.cc:107] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 12:26:38.230611  5636 raft_consensus.cc:2889] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:38.210119  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:38.179370 10477 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.185412 10442 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.147704 10601 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.196964 10265 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:38.186967 10532 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 12:26:38.198494 10358 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
I20260709 12:26:38.246609  5634 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:26:38.258774  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.084623 10621 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:38.237995 10617 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.291s	user 0.001s	sys 0.000s
I20260709 12:26:38.256714  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:38.218662 10023 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.223821 10533 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:38.214759 10646 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:38.276969  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 901) took 1138 ms. Trace:
I20260709 12:26:38.264616  5702 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:26:38.219707 10415 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.238451 10477 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:38.277410  5633 rpcz_store.cc:276] 0709 12:26:37.138165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:37.138641 (+   476us) service_pool.cc:224] Handling call
0709 12:26:37.160271 (+ 21630us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:37.263274 (+103003us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:37.263321 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:37.263869 (+   548us) log.cc:844] Serialized 26 byte log entry
0709 12:26:37.690425 (+426556us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:37.690486 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:37.690575 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:38.276759 (+586184us) raft_consensus.cc:1685] finished
0709 12:26:38.276784 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:38.276910 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":133283,"raft.queue_time_us":84840,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":34398,"thread_start_us":349187,"threads_started":1,"wal-append.queue_time_us":490260}
I20260709 12:26:38.280107  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:38.251504 10265 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:38.286284  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 430641us
I20260709 12:26:38.297013  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:38.299295  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:38.304044  5558 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I20260709 12:26:38.305395  5558 pending_rounds.cc:85] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 10
I20260709 12:26:38.305755  5558 pending_rounds.cc:107] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 12:26:38.306041  5558 raft_consensus.cc:2889] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:38.309489  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:38.311017  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:38.311553  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:38.311715  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:38.313655  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:38.279831 10415 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:38.283703 10192 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:38.316819  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:38.287487 10250 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:38.320734  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:38.296949 10265 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:38.301229 10023 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.306802 10496 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:38.324333  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:38.325263 10620 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.520s	user 0.002s	sys 0.004s
I20260709 12:26:38.327910  5702 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:38.315546 10567 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:38.317005 10604 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.340505 10351 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.315311 10412 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:26:38.319845 10192 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:38.321249 10581 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.345429  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:38.346176  5702 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:26:38.328653 10533 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:38.228251 10643 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.331931 10250 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:38.318375 10602 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.352394 10651 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:38.322638 10598 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:38.337849 10597 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.362337  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:38.336202 10588 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.356289 10651 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.364038 10644 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.364614 10615 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:38.365206 10601 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.368502 10594 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
W20260709 12:26:38.369060 10633 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.255s	user 0.000s	sys 0.001s
W20260709 12:26:38.369921  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:38.373356  6274 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:38.386821 10643 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:38.387435 10250 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.333956 10023 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.411386  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:38.301578 10646 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.361109 10288 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:38.390580 10615 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:38.392997 10526 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.761s	user 0.011s	sys 0.001s
W20260709 12:26:38.393503  9844 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.725s	user 0.008s	sys 0.003s
W20260709 12:26:38.399463 10493 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.769s	user 0.011s	sys 0.000s
I20260709 12:26:38.400481 10644 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:38.404040  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 111980us
W20260709 12:26:38.416492  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:38.403959 10638 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.224s	user 0.000s	sys 0.001s
I20260709 12:26:38.403327 10192 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.461701  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:38.536259  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:38.536427  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:38.480253  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:38.420531  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:38.548451  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:38.537410  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:38.550356  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 129911us
I20260709 12:26:38.551244 10656 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:26:38.527097 10251 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.226s	user 0.007s	sys 0.007s
W20260709 12:26:38.477437 10414 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.672s	user 0.013s	sys 0.000s
I20260709 12:26:38.470381  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.577046  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:38.473570 10654 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
W20260709 12:26:38.420722 10336 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.883s	user 0.011s	sys 0.002s
W20260709 12:26:38.479400 10606 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.825s	user 0.011s	sys 0.000s
I20260709 12:26:38.434026 10651 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.423311 10412 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:26:38.483855 10413 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.619s	user 0.009s	sys 0.000s
I20260709 12:26:38.570657  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:38.577145  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:38.592691  6278 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
W20260709 12:26:38.423233 10436 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.944s	user 0.012s	sys 0.002s
I20260709 12:26:38.439064 10023 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.437269 10415 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.477280 10440 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:38.584040  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.466163 10625 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.570062  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:38.538139 10288 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:38.588893  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:38.539058 10615 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:38.527436 10621 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:38.566977 10581 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:38.591338 10606 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.589049 10336 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.567320 10597 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:38.441676 10452 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.598830  5629 raft_consensus.cc:2410] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:26:38.421213 10250 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:38.539685 10492 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.990s	user 0.012s	sys 0.003s
W20260709 12:26:38.587486 10612 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.749s	user 0.008s	sys 0.004s
I20260709 12:26:38.591717 10412 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.502693 10477 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.538730 10493 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.656407  5708 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:26:38.657428 10493 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.657310 10612 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.604185  5634 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:38.607052  5701 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:38.607613  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.547009  9844 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.578584 10251 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.525867 10657 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.091s
I20260709 12:26:38.585325 10414 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.654822  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2294) took 1024 ms. Trace:
I20260709 12:26:38.539229 10526 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.660400  5636 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:38.660431  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2291) took 1032 ms. Trace:
I20260709 12:26:38.659286  6274 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:38.659210  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:38.692741  5708 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:38.691851  5705 rpcz_store.cc:276] 0709 12:26:37.630066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:37.840322 (+210256us) service_pool.cc:224] Handling call
0709 12:26:37.840622 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:38.076716 (+236094us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:38.654480 (+577764us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54000e01d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:38.654740 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":460703616}
I20260709 12:26:38.694182  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.661864  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.694865  5710 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:38.695305  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.695740  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:38.696080  5705 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:38.696832  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.691984  5703 rpcz_store.cc:276] 0709 12:26:37.627902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:37.806364 (+178462us) service_pool.cc:224] Handling call
0709 12:26:37.806855 (+   491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:37.817006 (+ 10151us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:37.817061 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:37.817683 (+   622us) log.cc:844] Serialized 26 byte log entry
0709 12:26:38.307093 (+489410us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:38.307135 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:38.307219 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:38.659832 (+352613us) raft_consensus.cc:1685] finished
0709 12:26:38.659855 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:38.660301 (+   446us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9411,"raft.queue_time_us":126910,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":20180,"thread_start_us":454957,"threads_started":1,"wal-append.queue_time_us":505013}
I20260709 12:26:38.697686  5705 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 11. Current term is 12. Ops: []
W20260709 12:26:38.672195 10658 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:26:38.669724 10412 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:38.679268  9844 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.686690 10251 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.687358  5553 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:26:38.671666 10492 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.606360 10462 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.605623 10440 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:38.663408  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:38.733592  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 70346.2us
I20260709 12:26:38.733817  5701 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:38.610416  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.698458  5635 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:38.734375  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:38.689465 10573 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.653s	user 0.007s	sys 0.006s
I20260709 12:26:38.671773 10043 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.670967 10597 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:38.735656  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:38.692255 10588 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:38.738833  5705 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:26:38.748447 10644 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:38.752579  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:38.759526 10270 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:26:38.761929 10250 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.748523  5704 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: []
I20260709 12:26:38.765383  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:38.767001  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:38.741532 10493 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:38.742902 10621 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:38.778868 10606 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:38.614115 10288 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.745645  9844 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:38.742611 10597 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:38.683874 10623 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.769984 10023 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.740967 10462 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.743427 10573 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.777506 10493 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.780606 10621 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:38.787087 10461 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.564s	user 0.010s	sys 0.005s
I20260709 12:26:38.783995 10606 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:38.801097 10573 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.801378 10606 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:38.801769 10664 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:26:38.802330 10606 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.804641  6276 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 12:26:38.745105 10043 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.809077 10043 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:38.787305 10423 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.568s	user 0.011s	sys 0.003s
I20260709 12:26:38.809765 10423 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.797794  5632 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:38.812909  5631 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:38.791200 10023 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.814914 10398 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:38.815526 10023 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.798591 10492 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:38.817296 10423 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.820302  5633 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:26:38.822209  5702 raft_consensus.cc:2368] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13.
I20260709 12:26:38.798760 10414 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.797125 10493 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:38.829972 10452 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.799213 10288 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:38.824081 10612 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.823766 10251 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.853222 10251 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.830907 10526 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:38.831853 10265 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.855585 10251 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.856464 10265 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.785388 10440 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.857940 10440 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:38.858498 10605 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.662s	user 0.009s	sys 0.003s
I20260709 12:26:38.858975 10605 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:38.833976 10417 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.737s	user 0.008s	sys 0.007s
I20260709 12:26:38.860688 10417 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.862924 10417 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.835246 10336 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.799630 10621 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:38.836303 10477 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.823303 10577 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:38.825868 10634 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:38.843775  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 76864.4us
I20260709 12:26:38.868602  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:38.864224  5627 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:38.821985 10600 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:38.870020  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:38.822070 10461 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:38.881896  5632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2155) took 3012 ms (client timeout 2002 ms). Trace:
W20260709 12:26:38.882768  5632 rpcz_store.cc:269] 0709 12:26:35.869347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:35.982381 (+113034us) service_pool.cc:224] Handling call
0709 12:26:38.881682 (+2899301us) spinlock_profiling.cc:243] Waited 2.8 s on lock 0x7b5400438d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:38.881816 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6434337664}
I20260709 12:26:38.883925  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.830945 10623 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.887578  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.888482  5632 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:38.889873  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:38.876273 10336 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:38.873637 10600 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:38.873761 10526 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:38.891986  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:38.901654  5632 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:38.893131 10336 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.876137 10634 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.882761 10412 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.893187 10452 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:38.920342 10412 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:38.920974 10452 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.921419 10452 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:38.923390  5632 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:38.909291 10336 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:38.925244  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.925899  5628 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:26:38.926668  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:38.927461  9844 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:38.928926  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:38.929944 10612 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:38.930358 10612 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.930696 10612 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:38.932015 10369 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:38.932646 10369 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.933601 10575 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.937475 10336 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.939779  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:38.926735  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:38.941781  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:38.942409  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:38.944839  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:38.945356  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:38.941325  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:38.951843  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:38.946923  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:38.952657  5632 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
W20260709 12:26:38.953205  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:38.953087  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:38.953496  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:38.953965  5632 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:38.956645  6272 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:38.957790 10605 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.958436 10600 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.959497 10600 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:38.952190  6279 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:38.961088 10622 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:38.929963 10611 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.086s	user 0.003s	sys 0.000s
I20260709 12:26:38.964296  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:38.964506 10265 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:38.973488 10265 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:38.974437 10265 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.976995 10281 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.978497  6278 raft_consensus.cc:2410] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 15 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:26:38.979497  6276 raft_consensus.cc:2368] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16.
I20260709 12:26:38.972254  5554 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:38.982137  6236 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:38.982422  6279 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:38.984443  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:38.985116  6276 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:26:38.986147  5632 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:38.989277  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:38.939141 10369 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:38.968480 10581 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:38.991199 10043 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:38.991734 10043 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:38.967176 10615 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.991951 10634 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:38.979732 10597 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:38.997014  5634 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:26:39.000200  5634 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:38.997920  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:38.969134 10461 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:39.008646 10615 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:39.014681 10612 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.016957  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:39.017838  5675 leader_election.cc:400] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16.
I20260709 12:26:39.017850  6279 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:39.018349  5675 leader_election.cc:403] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 12:26:39.018550 10043 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:39.005235 10351 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:39.019125  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:39.019423 10351 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:39.019848 10351 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:39.019948 10581 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:26:39.020710  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:39.021286  6278 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:39.021529 10581 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:39.021615 10281 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:39.023396 10281 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:39.024289 10281 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.024782  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:39.025484  5701 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:39.026292  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:39.027318  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:39.027079 10493 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:39.026396  6279 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:39.039433  6276 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:39.040948  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:39.045917  5554 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 12:26:39.046361  5554 pending_rounds.cc:85] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 11
I20260709 12:26:39.047981  5554 pending_rounds.cc:107] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 12:26:39.048329  5554 raft_consensus.cc:2889] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:39.046522 10612 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:39.049067  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 879) took 1069 ms. Trace:
I20260709 12:26:39.052542  5554 rpcz_store.cc:276] 0709 12:26:37.979474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:37.979633 (+   159us) service_pool.cc:224] Handling call
0709 12:26:37.991041 (+ 11408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:38.971987 (+980946us) spinlock_profiling.cc:243] Waited 629 ms on lock 0x7b580068f3d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5ce0aa 00007f4ffa650447 00007f4ffa668fe7 00007f4ffa66a7de 00007f4ffa674b49 00007f4ffa675416 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4
0709 12:26:39.048833 (+ 76846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:39.049015 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1446345600}
I20260709 12:26:39.071237  5708 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:39.072620  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.069908 10023 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:39.098686 10023 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:39.099557 10640 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:39.126151 10341 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:39.126451 10597 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:39.126783 10341 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:39.127926 10341 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.129436 10341 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:39.133302 10336 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:39.134712 10023 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.116297 10640 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:39.136229  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 902) took 1898 ms. Trace:
I20260709 12:26:39.136722  5629 rpcz_store.cc:276] 0709 12:26:37.237732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:37.237875 (+   143us) service_pool.cc:224] Handling call
0709 12:26:37.518518 (+280643us) spinlock_profiling.cc:243] Waited 4.68 ms on lock 0x7b5c0018800c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffdfdc161 00007f4ffe17da79 00007f4fff6379b6 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:39.136000 (+1617482us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b54001f2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:39.136158 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2299834496}
I20260709 12:26:39.139232  5634 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:39.138556  5629 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:39.142031 10631 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:39.142227  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:39.142861  6275 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:26:39.143092  5629 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
W20260709 12:26:39.147863  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:39.154971  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:39.163159 10655 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.746s	user 0.003s	sys 0.000s
I20260709 12:26:39.164194  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 913) took 1040 ms. Trace:
I20260709 12:26:39.164674  6280 rpcz_store.cc:276] 0709 12:26:38.123807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:38.140635 (+ 16828us) service_pool.cc:224] Handling call
0709 12:26:39.163941 (+1023306us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b54006379d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:39.164123 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49608576}
I20260709 12:26:39.185899  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:39.187137  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:39.213040  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 903) took 1666 ms. Trace:
I20260709 12:26:39.215446 10336 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260709 12:26:39.231276 10647 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:39.231712 10647 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:39.232604 10647 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:39.233917 10618 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.387s	user 0.000s	sys 0.008s
I20260709 12:26:39.234035 10647 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:39.237975  5627 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:26:39.213608  6273 rpcz_store.cc:276] 0709 12:26:37.546671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:37.546818 (+   147us) service_pool.cc:224] Handling call
0709 12:26:37.547122 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:38.122487 (+575365us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:39.212059 (+1089572us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b54004fa6d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:39.212957 (+   898us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1321910016}
I20260709 12:26:39.243010  6278 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:39.255282  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:39.254272  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:39.227515 10594 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:39.215911 10265 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:39.288895 10265 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:39.290019 10265 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:39.269793 10629 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.003s
I20260709 12:26:39.276520 10445 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:26:39.294967 10445 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:39.295363 10445 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:39.295289 10452 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:39.296413 10445 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:39.297313 10595 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.276561 10336 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:39.276577 10625 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:39.297820 10595 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.298336 10336 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:39.298580 10625 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:39.298987 10625 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:39.299314 10336 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.299752 10625 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:39.301021 10625 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:39.297847 10445 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:39.216361 10473 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20260709 12:26:39.266693 10575 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:39.305480 10575 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.307459 10575 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:39.280179  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.280642 10351 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.304909 10629 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:39.278712 10646 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:39.311236  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:39.330869  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:39.337352  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:39.338685  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:26:39.346895  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.326802 10629 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:39.375154  5703 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:39.378288  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:39.381733  5675 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.974s (SENT) [suppressed 10 similar messages]
I20260709 12:26:39.361915 10473 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:39.376220  6275 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:26:39.388274  5636 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:39.395804  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:39.400115  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:39.402314  5636 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:39.374239 10595 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:39.416217  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:39.419327  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:39.423974  5703 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 11 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 4.
I20260709 12:26:39.436867  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2296) took 1791 ms. Trace:
W20260709 12:26:39.437069  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.437350  5706 rpcz_store.cc:276] 0709 12:26:37.645328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:37.841427 (+196099us) service_pool.cc:224] Handling call
0709 12:26:37.841711 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:38.044904 (+203193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:39.436593 (+1391689us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400323054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:39.436803 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":319143296}
I20260709 12:26:39.443919  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:39.423233 10351 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:39.453897 10452 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:39.425346 10640 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:39.457162  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 75462.5us
W20260709 12:26:39.458788  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.446947 10414 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.459700  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:39.460438  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.423765 10629 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:39.463294 10621 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.599s	user 0.020s	sys 0.023s
I20260709 12:26:39.471911  5706 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:39.472652  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:39.473382  6278 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:39.475986  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:39.469185 10595 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:39.480716  6275 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:39.482122  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:39.485006  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.486574  6273 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:26:39.468470 10351 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.455173 10581 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16.
I20260709 12:26:39.490729  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2426) took 1363 ms. Trace:
I20260709 12:26:39.490950  5632 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:26:39.491274  5631 rpcz_store.cc:276] 0709 12:26:38.127748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:38.131118 (+  3370us) service_pool.cc:224] Handling call
0709 12:26:39.490394 (+1359276us) spinlock_profiling.cc:243] Waited 657 ms on lock 0x7b540032bed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:39.490662 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1510903168}
I20260709 12:26:39.487840 10640 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:26:39.493495  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:39.494047 10640 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:39.494983 10640 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.495445  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:39.492748  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:39.497311  6276 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:26:39.497848  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:39.475950 10454 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.498513  5631 raft_consensus.cc:2368] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16.
I20260709 12:26:39.501828  6276 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:26:39.500890  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:39.502323  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:39.502503 10462 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:39.502848  5631 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:39.503072  6275 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:39.504161  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.486886 10600 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:39.466607 10414 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.486886 10629 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:39.512007  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:26:39.513430 10688 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:26:39.514675  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:39.514712  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:39.500437 10477 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:39.517127  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:39.479233 10621 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:39.485949 10595 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.518747  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:39.528294  6276 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:39.520470  5631 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:39.530126  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:39.530869  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:39.531230  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:39.531649  6275 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:39.524566  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:39.526871  5701 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:26:39.532277  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:39.532933  5631 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:39.498471 10351 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.503806 10454 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.539951  5705 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:39.540946  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:39.516649 10600 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:39.543324  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:39.543972 10452 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:39.544279 10575 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:26:39.544826 10615 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:26:39.546139 10250 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:39.541896  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:39.563365  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.564384  6272 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 12:26:39.568157  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.580999  5631 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:26:39.555444 10552 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.007s
I20260709 12:26:39.632180  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2441) took 1397 ms. Trace:
I20260709 12:26:39.560237 10575 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:39.555117 10265 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:26:39.634265  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.645947  5633 rpcz_store.cc:276] 0709 12:26:38.235120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:38.746018 (+510898us) service_pool.cc:224] Handling call
0709 12:26:38.783453 (+ 37435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:38.821782 (+ 38329us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:39.631783 (+810001us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b540018c754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:39.632075 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":83941248}
I20260709 12:26:39.647743  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:26:39.638193  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 69638.7us
W20260709 12:26:39.586120 10685 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:26:39.642449  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:39.653934  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 915) took 1371 ms. Trace:
I20260709 12:26:39.654517  5628 rpcz_store.cc:276] 0709 12:26:38.282283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:38.973627 (+691344us) service_pool.cc:224] Handling call
0709 12:26:38.973961 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:39.184605 (+210644us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:39.184636 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:39.227902 (+ 43266us) log.cc:844] Serialized 26 byte log entry
0709 12:26:39.458280 (+230378us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:39.458319 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:39.458393 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:39.653666 (+195273us) raft_consensus.cc:1685] finished
0709 12:26:39.653695 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:39.653862 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":208379,"raft.queue_time_us":208626,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"thread_start_us":180871,"threads_started":1,"wal-append.queue_time_us":239252}
W20260709 12:26:39.641335 10692 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
W20260709 12:26:39.644972 10694 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
W20260709 12:26:39.675791 10691 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.171s	user 0.000s	sys 0.001s
I20260709 12:26:39.580300 10612 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.077s
I20260709 12:26:39.677157  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:39.681977  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2286) took 2093 ms. Trace:
W20260709 12:26:39.682638  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.676391  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2170) took 1306 ms. Trace:
W20260709 12:26:39.688896  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 145622us
I20260709 12:26:39.688479  5632 rpcz_store.cc:276] 0709 12:26:38.371056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:39.000566 (+629510us) service_pool.cc:224] Handling call
0709 12:26:39.490662 (+490096us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:39.491855 (+  1193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:39.676135 (+184280us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x7b540032bed0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:39.676306 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1126181120}
I20260709 12:26:39.690235  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:39.690963  5632 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:39.687276  5704 rpcz_store.cc:276] 0709 12:26:37.588308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:37.589386 (+  1078us) service_pool.cc:224] Handling call
0709 12:26:37.601263 (+ 11877us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:38.769535 (+1168272us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:39.681741 (+912206us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400323f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:39.681915 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2631153280}
I20260709 12:26:39.694420  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:39.632967 10398 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 12:26:39.697252  5704 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:39.695152  5633 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:39.706736  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.704010 10595 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:39.714970 10415 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:39.715399 10351 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:39.720199 10454 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:39.699632 10265 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:26:39.732084  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 106392us
W20260709 12:26:39.742406  5710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2424) took 2206 ms (client timeout 2405 ms). Trace:
W20260709 12:26:39.742908  5710 rpcz_store.cc:269] 0709 12:26:37.536156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:37.536484 (+   328us) service_pool.cc:224] Handling call
0709 12:26:39.742151 (+2205667us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54002d23d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:39.742314 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2652401920}
I20260709 12:26:39.744910  5707 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:39.747620  5710 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 12:26:39.748111  5710 pending_rounds.cc:85] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 7
I20260709 12:26:39.748415  5710 pending_rounds.cc:107] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 12:26:39.703473 10499 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:26:39.748802  5710 raft_consensus.cc:2889] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:39.734022 10414 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:39.734274 10265 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:39.749941  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:39.721910 10621 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:39.688576 10600 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:39.763438  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.761775  5633 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:39.765543  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:39.760587 10615 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:39.772456 10615 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:39.773643  5706 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:26:39.775005  5706 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:39.775200 10412 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:39.775377 10615 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:39.775619 10412 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:39.776057 10615 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.776285  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2440) took 1022 ms. Trace:
I20260709 12:26:39.776919  5701 rpcz_store.cc:276] 0709 12:26:38.754320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:39.027087 (+272767us) service_pool.cc:224] Handling call
0709 12:26:39.776127 (+749040us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x7b5400075654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:39.776232 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1089320320}
I20260709 12:26:39.776889  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:39.780184 10615 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:39.783068 10615 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:39.784880  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:39.786547  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 917) took 1613 ms. Trace:
I20260709 12:26:39.787199 10615 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:39.787545 10701 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 12:26:39.787786 10615 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:39.790863  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2427) took 1662 ms. Trace:
I20260709 12:26:39.793370  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2287) took 2204 ms. Trace:
I20260709 12:26:39.793902  5707 rpcz_store.cc:276] 0709 12:26:37.588624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:37.596565 (+  7941us) service_pool.cc:224] Handling call
0709 12:26:39.744705 (+2148140us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:39.745332 (+   627us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:39.793057 (+ 47725us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b54002d23d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:39.793278 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4929529600}
I20260709 12:26:39.786579  5710 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:26:39.791703  5635 rpcz_store.cc:276] 0709 12:26:38.128111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:38.128552 (+   441us) service_pool.cc:224] Handling call
0709 12:26:39.790621 (+1662069us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b540032dcd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:39.790781 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1308656000}
I20260709 12:26:39.787408  6271 rpcz_store.cc:276] 0709 12:26:38.173227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:38.173391 (+   164us) service_pool.cc:224] Handling call
0709 12:26:38.173734 (+   343us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:38.384143 (+210409us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:26:38.566968 (+182825us) log.cc:844] Serialized 14 byte log entry
0709 12:26:38.680671 (+113703us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b18003f3400. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:26:38.732540 (+ 51869us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:38.741048 (+  8508us) log.cc:844] Serialized 26 byte log entry
0709 12:26:38.741345 (+   297us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:26:38.741380 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:39.786052 (+1044672us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b5400055854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:39.786133 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:39.786208 (+    75us) raft_consensus.cc:1685] finished
0709 12:26:39.786227 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:39.786466 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":205054,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":4990,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":10408,"spinlock_wait_cycles":459062400,"thread_start_us":201141,"threads_started":2,"wal-append.queue_time_us":197438}
I20260709 12:26:39.808382  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 834) took 1470 ms. Trace:
I20260709 12:26:39.756724 10265 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:39.808881  5702 rpcz_store.cc:276] 0709 12:26:38.337546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:38.346915 (+  9369us) service_pool.cc:224] Handling call
0709 12:26:39.797207 (+1450292us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b5400210554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:39.808274 (+ 11067us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":989362944}
I20260709 12:26:39.794742 10615 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:39.815263  5635 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:26:39.841313  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2448) took 1030 ms. Trace:
I20260709 12:26:39.841818  5635 rpcz_store.cc:276] 0709 12:26:38.810957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:39.812600 (+1001643us) service_pool.cc:224] Handling call
0709 12:26:39.813012 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:39.840924 (+ 27912us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:39.841231 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:39.744752 10454 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
I20260709 12:26:39.770445 10499 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:39.857561 10499 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.816401 10656 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:39.819530 10631 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.675s	user 0.012s	sys 0.009s
I20260709 12:26:39.843588  5635 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 12:26:39.811769 10601 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:39.861925  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2450) took 1050 ms. Trace:
I20260709 12:26:39.861809 10601 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.862485 10615 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:39.863065  5709 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:39.818209 10415 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:39.863137 10615 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.864045 10601 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:39.863937 10415 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:39.865231 10594 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.577s	user 0.007s	sys 0.005s
I20260709 12:26:39.865381  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:39.866304 10415 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:39.829020 10412 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.867110 10415 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:39.867506 10412 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:39.865667 10594 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:39.867651 10415 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:39.860857 10656 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:39.862319  5635 rpcz_store.cc:276] 0709 12:26:38.811517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:39.842828 (+1031311us) service_pool.cc:224] Handling call
0709 12:26:39.843274 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:39.861629 (+ 18355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:39.861864 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:39.882285  6272 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
W20260709 12:26:39.915462  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:39.882536  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.874918  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2438) took 1121 ms. Trace:
I20260709 12:26:39.984899  5708 rpcz_store.cc:276] 0709 12:26:38.753789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:38.926413 (+172624us) service_pool.cc:224] Handling call
0709 12:26:39.874654 (+948241us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54000e01d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:39.874839 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":330131200}
I20260709 12:26:39.901921  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:39.885499  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 921) took 1309 ms. Trace:
I20260709 12:26:39.875439  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 916) took 1399 ms. Trace:
I20260709 12:26:39.986814  6274 rpcz_store.cc:276] 0709 12:26:38.576002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:38.576792 (+   790us) service_pool.cc:224] Handling call
0709 12:26:39.885285 (+1308493us) spinlock_profiling.cc:243] Waited 81 ms on lock 0x7b54002a8354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:39.885426 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":186301312}
I20260709 12:26:39.987823  5629 rpcz_store.cc:276] 0709 12:26:38.475926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:39.167468 (+691542us) service_pool.cc:224] Handling call
0709 12:26:39.680110 (+512642us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:39.680338 (+   228us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:39.680803 (+   465us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:39.680835 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:39.681318 (+   483us) log.cc:844] Serialized 26 byte log entry
0709 12:26:39.760971 (+ 79653us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:39.761030 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:39.761096 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:39.875118 (+114022us) raft_consensus.cc:1685] finished
0709 12:26:39.875140 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:39.875372 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4467,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":6491,"spinlock_wait_cycles":2991872,"thread_start_us":55017,"threads_started":1,"wal-append.queue_time_us":175623}
W20260709 12:26:39.987115  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 85328.9us
W20260709 12:26:39.994992  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:39.987689 10398 raft_consensus.cc:2764] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:26:39.995864  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:39.996083 10414 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:39.996599 10414 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:39.996857  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:39.997053  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:39.998267  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:39.998948 10595 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:39.999184 10414 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:39.870396 10594 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:39.998615  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:40.000180  5629 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:39.999948 10594 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:39.872946 10708 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:40.000967 10708 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:40.001592 10703 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:26:40.002184 10594 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:40.002274 10703 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:40.003262 10708 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:39.985994  5675 leader_election.cc:400] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16.
I20260709 12:26:40.007005  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:40.010917  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:40.010543  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.011299 10398 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:26:40.018639 10632 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:40.023927 10023 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:40.025436 10526 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:40.025925 10526 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:40.026962 10526 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:40.030539 10526 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:40.038100  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:40.039601  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:40.040314  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:40.041062  5706 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:40.041152  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:40.042752  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:40.049681  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2312) took 1512 ms. Trace:
I20260709 12:26:40.050170  5705 rpcz_store.cc:276] 0709 12:26:38.537030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:38.743485 (+206455us) service_pool.cc:224] Handling call
0709 12:26:38.743749 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:39.540870 (+797121us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:40.049321 (+508451us) spinlock_profiling.cc:243] Waited 796 ms on lock 0x7b54002f4254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:40.049529 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1830672640}
I20260709 12:26:40.051434  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:40.060868  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.063707  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:40.064216  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:40.064761  5701 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:40.065040  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:40.065582  5707 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:40.066082  6272 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:26:40.068161  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:40.068657  6234 leader_election.cc:400] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13.
I20260709 12:26:40.069116  6234 leader_election.cc:403] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:40.070817  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:40.071480  5706 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:40.077405  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:39.869563 10493 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:39.884268 10577 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.088769  5629 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:40.080832  5701 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:40.080737  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:40.083019  5631 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: [11.8-11.8]
W20260709 12:26:40.088246  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.089927  5702 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:40.089463  5706 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:40.091763  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:40.091250  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:40.092711  5701 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:26:40.070032 10699 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
I20260709 12:26:40.098521  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:40.100857  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:40.105208  5707 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:26:40.106684 10640 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:26:40.111621 10598 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:40.112433 10598 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:40.112371 10493 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:40.113940  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:40.115921 10632 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:40.117942  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:40.114926  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:40.121268 10632 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:40.122403 10632 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:40.122900 10632 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:40.124118 10632 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:39.899806 10575 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:40.127494  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.135219  5705 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:40.138710  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:40.149749  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.158034  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:40.158130  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:40.159799  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:40.160738  5701 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:40.165279  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:40.167461  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:40.166007  5710 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:26:40.199733  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2330) took 2766 ms. Trace:
I20260709 12:26:40.201588  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:40.200171  5553 rpcz_store.cc:276] 0709 12:26:37.400230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:37.508134 (+107904us) service_pool.cc:224] Handling call
0709 12:26:37.508521 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:38.727117 (+1218596us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:40.165986 (+1438869us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400082354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:40.166233 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2710400896}
I20260709 12:26:40.147503 10595 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13.
I20260709 12:26:40.220569  5561 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:40.154723 10581 raft_consensus.cc:2764] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:26:40.165205 10594 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.222543  5556 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:40.126006 10577 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:40.136248 10621 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260709 12:26:40.165737 10336 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:40.222990  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:40.149200 10575 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:40.263060 10412 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:40.264430 10700 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:40.268711 10700 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:40.229008 10704 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:40.279767  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.269649 10700 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:40.277637  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:40.280495  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:40.290387  5709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2443) took 1535 ms (client timeout 1945 ms). Trace:
W20260709 12:26:40.291440  5709 rpcz_store.cc:269] 0709 12:26:38.755079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:38.997598 (+242519us) service_pool.cc:224] Handling call
0709 12:26:40.290010 (+1292412us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b5400066654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:40.290303 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1981573888}
I20260709 12:26:40.242714 10594 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:40.278146 10704 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:40.314538  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2442) took 1559 ms. Trace:
I20260709 12:26:40.315006  5704 rpcz_store.cc:276] 0709 12:26:38.754829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:39.704835 (+950006us) service_pool.cc:224] Handling call
0709 12:26:39.710800 (+  5965us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:39.711207 (+   407us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:26:39.712513 (+  1306us) log.cc:844] Serialized 14 byte log entry
0709 12:26:39.819428 (+106915us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:39.821917 (+  2489us) log.cc:844] Serialized 52 byte log entry
0709 12:26:39.822138 (+   221us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:39.822190 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:39.822286 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:40.314303 (+492017us) raft_consensus.cc:1685] finished
0709 12:26:40.314323 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:40.314473 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3131,"prepare.queue_time_us":1744,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":8637,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":3511,"thread_start_us":103155,"threads_started":1,"wal-append.queue_time_us":481513}
I20260709 12:26:40.318395 10705 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:26:40.318841 10705 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:40.248576 10336 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:40.319662 10705 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:40.278402 10412 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:40.336936 10720 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.349579 10720 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:40.244007 10709 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260709 12:26:40.351255 10412 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:40.353013 10709 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:40.353528 10709 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:40.356266  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 111274us
I20260709 12:26:40.357183 10615 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:40.358222 10477 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.836s	user 0.018s	sys 0.007s
I20260709 12:26:40.358711 10477 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:40.364697  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:40.365823  5560 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:26:40.367539  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:40.369267  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.008s	sys 0.005s
W20260709 12:26:40.370735  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.008s	sys 0.005s
W20260709 12:26:40.373555  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.382604  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:40.383261  5560 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:40.386809  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:40.377919 10629 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.862s	user 0.017s	sys 0.013s
I20260709 12:26:40.384402  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:40.375027 10341 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.245s	user 0.021s	sys 0.019s
I20260709 12:26:40.393101 10599 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:26:40.421218 10695 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.905s	user 0.005s	sys 0.004s
I20260709 12:26:40.398902  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:40.400067  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:40.422508  5703 raft_consensus.cc:2410] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:26:40.400820 10452 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.401283  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:40.425302  5557 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:40.425426  6272 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:26:40.402330 10250 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:40.404757 10192 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:40.405699 10493 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:40.378924  5701 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:26:40.374449 10605 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260709 12:26:40.378995 10600 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:26:40.408584  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 922) took 1592 ms. Trace:
I20260709 12:26:40.388692 10358 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.415055  5556 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:40.397841  5553 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:26:40.438221  6272 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:26:40.421273 10599 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:40.445212  5632 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.009s	sys 0.020s
W20260709 12:26:40.446261  5632 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.009s	sys 0.021s
I20260709 12:26:40.431412 10493 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:40.413035  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.430967  5636 rpcz_store.cc:276] 0709 12:26:38.816654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:39.988997 (+1172343us) service_pool.cc:224] Handling call
0709 12:26:40.077591 (+ 88594us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:40.292650 (+215059us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:40.292697 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:40.292766 (+    69us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:40.292804 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:40.408286 (+115482us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54001cf7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:40.408366 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:40.408515 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":494004096}
I20260709 12:26:40.427445 10250 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:40.456746  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 515408us
I20260709 12:26:40.457490  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:40.428144  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:26:40.461431  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 393791us
I20260709 12:26:40.468181  6272 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:26:40.473519  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2446) took 1717 ms. Trace:
I20260709 12:26:40.474102  5703 rpcz_store.cc:276] 0709 12:26:38.755882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:39.676907 (+921025us) service_pool.cc:224] Handling call
0709 12:26:40.473258 (+796351us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b5400210f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:40.473441 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1689670528}
W20260709 12:26:40.479242  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51164.7us
I20260709 12:26:40.480571 10615 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:40.481149 10615 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:40.481527 10615 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:40.484845 10477 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:26:40.487004  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.431717 10452 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:40.488462  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:40.447057 10657 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:40.490795  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.448104 10415 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:40.537252  5553 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:40.511621  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2445) took 1701 ms. Trace:
I20260709 12:26:40.540261  5557 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:40.541575  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:40.542549  5557 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:40.543429  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:40.546237  5559 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:40.548739  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:40.441248 10358 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:40.460337 10704 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:40.556704  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:40.557544  5553 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:40.550273  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:40.451465  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2447) took 1639 ms. Trace:
I20260709 12:26:40.465641 10412 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:40.583158  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:40.465464 10351 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.559012  5633 rpcz_store.cc:276] 0709 12:26:38.810675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:39.778996 (+968321us) service_pool.cc:224] Handling call
0709 12:26:39.779343 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:39.779550 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:26:40.147449 (+367899us) log.cc:844] Serialized 14 byte log entry
0709 12:26:40.449675 (+302226us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:40.449726 (+    51us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:40.449769 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:40.449835 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:40.449966 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":424364,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":41,"raft.queue_time_us":81,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":2709,"thread_start_us":668963,"threads_started":2}
I20260709 12:26:40.593719  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:40.594554  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:40.595103  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:40.594552  5558 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 13: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:40.596515 10454 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:26:40.597931 10599 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:40.599282 10599 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:40.510684 10477 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:40.530908  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 462560us
I20260709 12:26:40.428623 10709 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:40.610383  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:40.611406 10452 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:40.607671 10631 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.005s
I20260709 12:26:40.613097 10493 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:40.592145 10250 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:40.616403 10265 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.797s	user 0.011s	sys 0.001s
I20260709 12:26:40.596253 10577 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:40.607626 10351 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:40.632206  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:40.550056 10629 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:40.633981  5559 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:40.608271 10412 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:40.655702  5553 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:40.552325 10657 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:40.662101  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:40.662762  5553 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:40.645480 10715 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
W20260709 12:26:40.754339  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.667464 10270 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:40.683691 10462 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:40.683450 10526 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.643618  5556 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:26:40.720332 10575 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:40.726240 10705 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:40.731652  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:40.737005  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.745812 10594 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:40.538524  5632 rpcz_store.cc:276] 0709 12:26:38.810036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:39.700113 (+890077us) service_pool.cc:224] Handling call
0709 12:26:39.700443 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:39.700636 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:26:39.845892 (+145256us) log.cc:844] Serialized 14 byte log entry
0709 12:26:40.511133 (+665241us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:40.511217 (+    84us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:40.511291 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:40.511382 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:40.511548 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80119,"prepare.queue_time_us":426511,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":85415,"thread_start_us":736259,"threads_started":2}
I20260709 12:26:40.750157 10657 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:40.751292  5633 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:40.779507  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:40.603236 10598 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 12:26:40.602447 10621 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 12:26:40.644196 10415 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.126s
I20260709 12:26:40.646247 10724 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.134s
I20260709 12:26:40.753666 10629 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:40.648726 10600 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:40.623926 10477 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:40.641609 10720 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:40.780226  5632 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:40.653069 10265 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:40.772710 10270 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:40.844982  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:40.801477  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 336287us
W20260709 12:26:40.836380  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 481243us
W20260709 12:26:40.799611 10732 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 12:26:40.783314 10336 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:40.846279  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:40.780967 10526 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:40.835819 10577 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.780179 10462 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:40.866106  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:40.867905  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:40.871069  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:40.849273 10412 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:40.877074 10734 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260709 12:26:40.878386 10733 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.237s	user 0.000s	sys 0.001s
I20260709 12:26:40.882380 10462 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:40.882544 10629 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:40.892954  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46821.6us
W20260709 12:26:40.876111 10731 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
I20260709 12:26:40.874111 10336 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.842928 10600 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:40.889762 10351 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:40.894351 10358 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:40.872614 10577 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:40.760295  5553 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:41.000315  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:41.001894  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:41.002934  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:41.002760  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 488209us
I20260709 12:26:41.004006  5553 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:41.005621  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:41.007404  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:41.008865  5552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.002s	sys 0.003s
I20260709 12:26:41.009382  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:40.762042  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:41.009727  5552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.002s	sys 0.003s
I20260709 12:26:41.009811  5554 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:41.011267  5558 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:40.778713  5559 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:41.012851  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:41.020836  5556 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:26:41.021338  5629 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.018s	sys 0.015s
W20260709 12:26:41.021847  5629 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.018s	sys 0.015s
I20260709 12:26:41.022648  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2457) took 2052 ms. Trace:
I20260709 12:26:41.024502  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2194) took 1712 ms. Trace:
I20260709 12:26:41.025033  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.025038  5629 rpcz_store.cc:276] 0709 12:26:39.311988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:40.242795 (+930807us) service_pool.cc:224] Handling call
0709 12:26:40.367381 (+124586us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:40.371365 (+  3984us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:40.372294 (+   929us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:40.391086 (+ 18792us) log.cc:844] Serialized 52 byte log entry
0709 12:26:41.022702 (+631616us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:41.023515 (+   813us) log.cc:844] Serialized 14 byte log entry
0709 12:26:41.023936 (+   421us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:41.024060 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:41.024140 (+    80us) raft_consensus.cc:1685] finished
0709 12:26:41.024164 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:41.024439 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3869,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":12343,"thread_start_us":630720,"threads_started":1,"wal-append.queue_time_us":571476}
I20260709 12:26:41.027779  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.029656  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.023229  5628 rpcz_store.cc:276] 0709 12:26:38.969684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:40.043980 (+1074296us) service_pool.cc:224] Handling call
0709 12:26:40.044311 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:40.044598 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:26:40.452113 (+407515us) log.cc:844] Serialized 14 byte log entry
0709 12:26:41.021715 (+569602us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:41.021841 (+   126us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:41.021965 (+   124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:41.022097 (+   132us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:41.022387 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":220189,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"thread_start_us":861287,"threads_started":2,"wal-append.queue_time_us":501730}
I20260709 12:26:41.032001  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.033896  5628 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:41.035269  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.030243  5552 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:26:41.049789 10192 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.617s	user 0.012s	sys 0.000s
W20260709 12:26:41.038810  5634 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.007s	sys 0.014s
W20260709 12:26:41.051079  5634 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.008s	sys 0.014s
I20260709 12:26:41.052696  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 919) took 2422 ms. Trace:
I20260709 12:26:41.054064 10265 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:41.053126  5634 rpcz_store.cc:276] 0709 12:26:38.631944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:39.236533 (+604589us) service_pool.cc:224] Handling call
0709 12:26:40.424018 (+1187485us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:40.424223 (+   205us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:40.424454 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:26:40.443249 (+ 18795us) log.cc:844] Serialized 14 byte log entry
0709 12:26:41.051853 (+608604us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:41.051967 (+   114us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:41.052217 (+   250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:41.052320 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:41.052604 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":19140,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"spinlock_wait_cycles":19081344,"thread_start_us":607958,"threads_started":1,"wal-append.queue_time_us":515030}
I20260709 12:26:41.055083 10265 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:26:41.055780  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:41.038516  5631 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.004s	sys 0.014s
I20260709 12:26:40.920715 10336 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:41.081406  5631 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.005s	sys 0.019s
I20260709 12:26:41.083153  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2459) took 1772 ms. Trace:
I20260709 12:26:41.061241 10477 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:41.038978  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.084596 10477 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:41.059396 10358 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 12:26:41.066208 10526 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:41.039397  5636 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:41.046708  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.090237  5552 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:41.091253 10526 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:41.091574  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.038040  5629 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:41.058606 10270 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:41.057207  5630 raft_consensus.cc:2410] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 12:26:41.096022 10265 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:41.098709 10600 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:41.098809 10265 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:41.099053  5630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2168) took 2747 ms (client timeout 1974 ms). Trace:
I20260709 12:26:41.099359  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 930) took 1449 ms. Trace:
W20260709 12:26:41.099670  5630 rpcz_store.cc:269] 0709 12:26:38.351844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:38.536011 (+184167us) service_pool.cc:224] Handling call
0709 12:26:41.098727 (+2562716us) spinlock_profiling.cc:243] Waited 2.52 s on lock 0x7b54000c0b54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:41.098956 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5792480000}
I20260709 12:26:41.100930 10458 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.102370  5630 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:26:41.101835 10458 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.080690  5634 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:41.087409 10358 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:41.107041 10250 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:41.107264 10358 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:41.109030  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2200) took 1063 ms. Trace:
I20260709 12:26:41.035694  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:41.109426 10358 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:41.111706 10594 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:41.099813  5633 rpcz_store.cc:276] 0709 12:26:39.649728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:41.098059 (+1448331us) service_pool.cc:224] Handling call
0709 12:26:41.098604 (+   545us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:41.098905 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:41.098936 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:41.098974 (+    38us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:41.099013 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:41.099074 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:41.099175 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:41.117463 10632 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.118571 10632 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:41.115518 10577 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:41.100602 10265 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:41.123665 10265 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:41.092145 10526 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.128510  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.129187  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.128538  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.131374  6279 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:41.133266 10526 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:41.037621  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.121838 10632 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:41.133906 10612 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:41.147624 10612 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:41.148322  5628 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:41.149480 10612 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.151293 10612 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:41.153640 10632 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.154903 10632 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:41.155817  5629 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 12:26:41.141105 10656 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:41.142360  5633 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:41.142761 10358 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:41.049834  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.109580  5634 rpcz_store.cc:276] 0709 12:26:40.045561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:41.107543 (+1061982us) service_pool.cc:224] Handling call
0709 12:26:41.108222 (+   679us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:41.108548 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:41.108576 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:41.108646 (+    70us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:41.108688 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:41.108767 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:41.108949 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:41.162113  5629 pending_rounds.cc:85] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 10
I20260709 12:26:41.087597  5632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52cee1d09af749489de4689a67047497"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:41.175040  5632 raft_consensus.cc:606] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
W20260709 12:26:41.139029  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 103592us
I20260709 12:26:41.176110  5632 raft_consensus.cc:614] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:41.173243  5629 pending_rounds.cc:107] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 12:26:41.181213  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2199) took 1130 ms. Trace:
I20260709 12:26:41.197705  5630 rpcz_store.cc:276] 0709 12:26:40.045269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:41.101394 (+1056125us) service_pool.cc:224] Handling call
0709 12:26:41.101931 (+   537us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:41.165937 (+ 64006us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:41.175199 (+  9262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:41.189509  5629 raft_consensus.cc:2889] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:41.201589  5553 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
W20260709 12:26:41.210333  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:41.083732  5631 rpcz_store.cc:276] 0709 12:26:39.310323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:40.090548 (+780225us) service_pool.cc:224] Handling call
0709 12:26:40.091098 (+   550us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:40.091340 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:26:40.439407 (+348067us) log.cc:844] Serialized 14 byte log entry
0709 12:26:41.082422 (+643015us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:41.082539 (+   117us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:41.082622 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:41.082750 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:41.083064 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33791,"prepare.queue_time_us":177669,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":176,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":150,"spinlock_wait_cycles":12705024,"thread_start_us":956064,"threads_started":2,"wal-append.queue_time_us":519410}
I20260709 12:26:41.232658  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 929) took 1680 ms. Trace:
I20260709 12:26:41.233227  5629 rpcz_store.cc:276] 0709 12:26:39.549228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:41.154833 (+1605605us) service_pool.cc:224] Handling call
0709 12:26:41.155252 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:41.227469 (+ 72217us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:41.229762 (+  2293us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:41.235682  5556 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:26:41.241897  5556 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:26:41.250043  5559 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:41.253356  5631 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: [13.8-13.8]
W20260709 12:26:41.254231  5635 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.010s	sys 0.005s
W20260709 12:26:41.254650  5635 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.010s	sys 0.006s
I20260709 12:26:41.255046 10714 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:41.257541  5556 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:41.263064 10632 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:41.264882  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.697s	user 0.015s	sys 0.026s
I20260709 12:26:41.263789  6280 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: [13.10-13.10]
W20260709 12:26:41.266518  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:41.260082  5553 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:41.267920  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.275698 10398 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.276551 10398 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.278970 10398 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:41.255767 10714 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:41.281391  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.015s	sys 0.027s
I20260709 12:26:41.282824  5554 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 12:26:41.281507  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 305803us
W20260709 12:26:41.293534  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68050.2us
W20260709 12:26:41.290742  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 831277us
I20260709 12:26:41.299971  5634 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:26:41.294552  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.304224  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.304958  6278 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:41.305362  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.306944  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:41.323021 10751 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.362s	user 0.000s	sys 0.000s
W20260709 12:26:41.332403  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.006s	sys 0.009s
W20260709 12:26:41.332875  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.006s	sys 0.010s
I20260709 12:26:41.339264 10632 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:41.340521  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.341364  6280 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
W20260709 12:26:41.343760  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:41.344338 10458 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:41.347743 10336 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:41.348997  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:41.352219 10702 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.352752 10720 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.352967 10702 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.353387 10720 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:41.355614 10702 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:41.355788 10720 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:41.348894  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2453) took 2502 ms. Trace:
I20260709 12:26:41.356133 10702 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.377712  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:41.378741  5559 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:41.384052  6275 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:26:41.365876 10358 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.387038  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:41.393247  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:41.386212  5627 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.003s	sys 0.027s
I20260709 12:26:41.394225  5556 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:41.395438  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.361775  5635 rpcz_store.cc:276] 0709 12:26:38.846390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:39.990030 (+1143640us) service_pool.cc:224] Handling call
0709 12:26:39.990400 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:39.990647 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:26:40.539373 (+548726us) log.cc:844] Serialized 14 byte log entry
0709 12:26:41.311684 (+772311us) log.cc:844] Serialized 14 byte log entry
0709 12:26:41.348458 (+ 36774us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:41.348529 (+    71us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:41.348615 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:41.348690 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:41.348839 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":102803,"prepare.queue_time_us":292886,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":76,"raft.queue_time_us":30128,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":4438,"thread_start_us":1129316,"threads_started":2,"wal-append.queue_time_us":759228}
W20260709 12:26:41.377267 10752 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.417s	user 0.000s	sys 0.004s
W20260709 12:26:41.399237  5627 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.004s	sys 0.028s
I20260709 12:26:41.408510  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.409785  5556 raft_consensus.cc:2410] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
W20260709 12:26:41.411196  5560 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.984s	user 0.012s	sys 0.013s
I20260709 12:26:41.411504 10598 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.411733  5560 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:26:41.412324  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 887) took 1030 ms. Trace:
I20260709 12:26:41.412128 10598 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:41.412799  5560 rpcz_store.cc:276] 0709 12:26:40.382216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:40.382345 (+   129us) service_pool.cc:224] Handling call
0709 12:26:41.412246 (+1029901us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:41.414027  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:41.415091 10598 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:41.414940  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:41.416239  5703 raft_consensus.cc:2376] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15. Re-sending same reply.
I20260709 12:26:41.416595  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:41.417645  5708 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:26:41.418409 10750 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 12:26:41.418411 10742 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.412s	user 0.006s	sys 0.000s
I20260709 12:26:41.419768 10358 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:41.421600  5560 raft_consensus.cc:2376] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12. Re-sending same reply.
I20260709 12:26:41.429347  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:41.397158 10336 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.431875 10336 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:41.432761  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:41.434386 10336 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:41.435034  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:41.435221 10458 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:26:41.439445 10744 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.479s	user 0.000s	sys 0.001s
I20260709 12:26:41.440590 10454 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:41.443171 10454 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:26:41.443913 10454 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:41.448706  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.449218 10458 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:41.451083  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.452461  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 920) took 2775 ms. Trace:
I20260709 12:26:41.453279 10493 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:41.454815 10477 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:26:41.455694 10412 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:41.443621 10351 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.013s
I20260709 12:26:41.441041  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 937) took 1051 ms. Trace:
W20260709 12:26:41.459913 10615 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.945s	user 0.011s	sys 0.011s
I20260709 12:26:41.461266  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:41.462270 10702 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.481264  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:41.463492  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:41.464231  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 152596us
I20260709 12:26:41.464821  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:41.466411  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.467124 10358 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:41.505352  5560 raft_consensus.cc:2376] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13. Re-sending same reply.
I20260709 12:26:41.468464 10709 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:41.469945  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.471251  5557 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:26:41.472057  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.473384  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.474432  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:41.475533 10753 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.514s	user 0.004s	sys 0.000s
I20260709 12:26:41.476621  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.481990  5628 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:26:41.481805  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:41.481463  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:41.483355  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.471940 10250 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:41.467926  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:41.508132  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:41.480299 10714 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:41.510468  5554 raft_consensus.cc:2376] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16. Re-sending same reply.
W20260709 12:26:41.481698 10741 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.522s	user 0.000s	sys 0.001s
I20260709 12:26:41.513366  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:41.518477  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.534623 10702 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:41.535243  5554 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:41.535921 10657 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 12:26:41.518692  5632 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:41.518816 10717 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.538779  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:41.538496 10717 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.511945 10615 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.520179  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.541525  5634 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:41.535413  5559 raft_consensus.cc:2376] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12. Re-sending same reply.
I20260709 12:26:41.542500 10615 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:41.543310 10615 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:26:41.520385  5630 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:41.520474  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:41.545199  5555 raft_consensus.cc:2376] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16. Re-sending same reply.
I20260709 12:26:41.545842 10656 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:41.546249 10656 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:41.546331  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:41.547214  5555 raft_consensus.cc:2410] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:26:41.547137 10656 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:41.547930  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2341) took 1011 ms. Trace:
I20260709 12:26:41.548308  5558 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 10. (term mismatch)
I20260709 12:26:41.548383 10250 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:41.548828  5558 pending_rounds.cc:85] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 9
I20260709 12:26:41.549165  5558 pending_rounds.cc:107] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260709 12:26:41.549418  5558 raft_consensus.cc:2889] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:41.549597 10250 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:41.521157  5556 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:41.522048  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.551621  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.526721 10612 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:26:41.552834  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.553725  5556 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:41.553611  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.554311  5631 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:41.554603  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.520196  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 921) took 1497 ms. Trace:
I20260709 12:26:41.520931  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.551863  5553 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 13: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:41.556528  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:41.557292  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:41.557060 10454 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.557686  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.558104  5553 raft_consensus.cc:2410] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:26:41.552878 10612 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:41.558789 10612 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:41.558970  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:41.527177  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 126592us
I20260709 12:26:41.562582  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.562760  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:41.563730 10724 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:41.563782  5629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "733a5b54e4dd4f86999821705033529f"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:41.559147 10612 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:41.543710  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.491945  5627 rpcz_store.cc:276] 0709 12:26:38.676921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:39.238776 (+561855us) service_pool.cc:224] Handling call
0709 12:26:39.681728 (+442952us) spinlock_profiling.cc:243] Waited 4.01 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:26:39.681911 (+   183us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:39.682107 (+   196us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:26:39.768270 (+ 86163us) log.cc:844] Serialized 14 byte log entry
0709 12:26:40.455879 (+687609us) spinlock_profiling.cc:243] Waited 22.6 ms on lock 0x7b1800489d60. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a
0709 12:26:40.549554 (+ 93675us) log.cc:844] Serialized 14 byte log entry
0709 12:26:40.716500 (+166946us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x7b1800489d60. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:26:40.936567 (+220067us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b1800489d60. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:41.451784 (+515217us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:41.451897 (+   113us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:41.451996 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:41.452111 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:41.452335 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32985,"prepare.queue_time_us":265161,"prepare.run_cpu_time_us":158,"prepare.run_wall_time_us":7256,"raft.queue_time_us":66712,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":54249,"spinlock_wait_cycles":166678144,"thread_start_us":1059591,"threads_started":3,"wal-append.queue_time_us":460597}
I20260709 12:26:41.564155  5559 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:41.596859  5559 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:41.566951  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.597916  5559 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:41.575661  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:41.555651  6276 rpcz_store.cc:276] 0709 12:26:40.022331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:40.036335 (+ 14004us) service_pool.cc:224] Handling call
0709 12:26:40.036816 (+   481us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:40.037344 (+   528us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:40.037389 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:40.430225 (+392836us) spinlock_profiling.cc:243] Waited 24.4 ms on lock 0x7b1800039340. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a
0709 12:26:40.699078 (+268853us) spinlock_profiling.cc:243] Waited 7.69 ms on lock 0x7b2c00040818. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:26:40.699735 (+   657us) log.cc:844] Serialized 26 byte log entry
0709 12:26:41.049771 (+350036us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:41.049896 (+   125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:41.050037 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:41.474706 (+424669us) spinlock_profiling.cc:243] Waited 4.26 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:41.519843 (+ 45137us) raft_consensus.cc:1685] finished
0709 12:26:41.519864 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:41.520027 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6262,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":5517,"spinlock_wait_cycles":83575680,"thread_start_us":349332,"threads_started":1,"wal-append.queue_time_us":260452}
I20260709 12:26:41.603679 10351 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:41.501308  6274 rpcz_store.cc:276] 0709 12:26:40.389051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:40.389177 (+   126us) service_pool.cc:224] Handling call
0709 12:26:40.395011 (+  5834us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:40.797003 (+401992us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:26:40.797050 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:40.797866 (+   816us) log.cc:844] Serialized 26 byte log entry
0709 12:26:41.270844 (+472978us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:41.270924 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:41.271061 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:41.440518 (+169457us) raft_consensus.cc:1685] finished
0709 12:26:41.440548 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:41.440784 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16602,"raft.queue_time_us":93136,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":52414,"thread_start_us":837388,"threads_started":2,"wal-append.queue_time_us":555617}
I20260709 12:26:41.604285 10351 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:41.521627 10458 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:41.605315 10351 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:41.607012 10705 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.823s	user 0.014s	sys 0.004s
I20260709 12:26:41.607884  5561 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:26:41.608666  5554 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:41.498543 10477 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:41.565266  5632 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
W20260709 12:26:41.608877  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44643.6us
I20260709 12:26:41.609802 10477 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:41.528715  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 136350us
I20260709 12:26:41.568641  5556 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:41.611886  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:41.578460 10412 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.514195  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 934) took 1044 ms. Trace:
I20260709 12:26:41.617702  5553 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:41.576052 10717 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:41.599273  5559 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:41.600957  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:41.589859  5628 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:26:41.620836  5629 raft_consensus.cc:606] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
W20260709 12:26:41.584513  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 86484.9us
W20260709 12:26:41.594440  5456 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35600 (request call id 276) took 30569 ms (client timeout 15000 ms). Trace:
I20260709 12:26:41.621191  5629 raft_consensus.cc:614] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:26:41.616264  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 135102us
I20260709 12:26:41.619503  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.548336  5557 rpcz_store.cc:276] 0709 12:26:40.536141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:40.544230 (+  8089us) service_pool.cc:224] Handling call
0709 12:26:41.547729 (+1003499us) spinlock_profiling.cc:243] Waited 912 ms on lock 0x7b54000ea6d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:41.547882 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2097263488}
I20260709 12:26:41.600281  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:41.629966  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 103464us
I20260709 12:26:41.630867  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:41.631778 10632 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:41.632637  5558 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:41.632234  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:41.644459  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:41.616277  5633 rpcz_store.cc:276] 0709 12:26:40.464457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:41.177029 (+712572us) service_pool.cc:224] Handling call
0709 12:26:41.234632 (+ 57603us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:41.266085 (+ 31453us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:26:41.290523 (+ 24438us) log.cc:844] Serialized 14 byte log entry
0709 12:26:41.508581 (+218058us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:41.508636 (+    55us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:26:41.508676 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:41.508736 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:41.508843 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19469,"prepare.queue_time_us":26870,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":17456,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":15228,"thread_start_us":178787,"threads_started":1,"wal-append.queue_time_us":312503}
I20260709 12:26:41.631619 10598 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:41.627635 10477 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:41.632279 10632 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:41.643286 10615 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:26:41.686504 10656 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:41.689782  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 89674.5us
I20260709 12:26:41.690038 10700 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:26:41.690547  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 150275us
I20260709 12:26:41.691257  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:41.711177  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 890) took 1324 ms. Trace:
I20260709 12:26:41.673923 10598 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:41.713696  5561 rpcz_store.cc:276] 0709 12:26:40.386492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:40.386605 (+   113us) service_pool.cc:224] Handling call
0709 12:26:41.710922 (+1324317us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54003a89d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:41.711091 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2806284032}
I20260709 12:26:41.744235 10454 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:41.744467  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:26:41.745536  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 80101.5us
I20260709 12:26:41.746001  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:41.746513 10725 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.076s
I20260709 12:26:41.748205 10462 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.690990 10632 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:41.711171 10700 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:41.633045 10621 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.116s
I20260709 12:26:41.714166 10615 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:41.754946 10598 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:41.782264  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 106361us
I20260709 12:26:41.762657 10462 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.790558 10462 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:41.792557  5704 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:26:41.793270  5704 raft_consensus.cc:606] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:41.793646  5704 raft_consensus.cc:614] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:41.794039  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:41.795161  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:41.795904  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:41.799649  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 156510us
I20260709 12:26:41.800195  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:41.802265 10724 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:41.796128  5454 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:26:41.802608 10615 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:41.804211  5454 catalog_manager.cc:5697] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 4 to 9, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 9 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:41.807866  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:41.800942 10462 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:41.811801 10718 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:41.812682 10718 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:41.780139 10717 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:41.814139  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:41.778767 10700 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:41.819993  5559 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
W20260709 12:26:41.821224  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:41.827512  5559 pending_rounds.cc:85] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 7
I20260709 12:26:41.831669 10412 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:41.832434 10705 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:41.832865 10705 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:41.833216 10705 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:41.833571  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:41.819269 10718 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:41.833446 10717 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:41.836933  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 92629.6us
I20260709 12:26:41.837198  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:41.836553  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 127676us
I20260709 12:26:41.838944  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:41.840055  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:41.840883  5558 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:41.841053  5560 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:26:41.841884  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.842321  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.842936 10351 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:41.843597  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.845247 10720 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:26:41.846818 10612 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.847911 10499 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:26:41.848667  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 216547us
I20260709 12:26:41.848768  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.849107 10612 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:41.849680  5553 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:41.828440  5559 pending_rounds.cc:107] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 12:26:41.852939  5559 raft_consensus.cc:2889] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:41.836524 10717 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.884474  5553 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
W20260709 12:26:41.870934  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58710.1us
I20260709 12:26:41.861415 10702 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:26:41.893970  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:41.890609  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:41.852581 10745 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:26:41.898379  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.883980 10250 raft_consensus.cc:2764] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
W20260709 12:26:41.901087  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 73870.6us
I20260709 12:26:41.893708 10702 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:41.903448  6237 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:41.916743  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 118867us
I20260709 12:26:41.919644  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:41.897293 10720 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:41.921306  5553 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:41.906618 10462 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:41.921681 10526 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:41.922225 10412 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:41.923161  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:41.925262  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:41.925868 10725 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:26:41.931602  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 101122us
I20260709 12:26:41.933494  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:41.941300 10754 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:41.941819 10754 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:41.942201 10754 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:41.944543 10601 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:41.892508  5454 catalog_manager.cc:5697] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 4 to 7, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 7 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:26:41.945175 10601 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:41.947576 10601 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:41.909497 10724 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:41.621378  5456 rpcz_store.cc:269] 0709 12:26:11.024836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.025260 (+   424us) service_pool.cc:224] Handling call
0709 12:26:41.594129 (+30568869us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:36.061332 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:36.062023 (+   691us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:36.062071 (+    48us) write_op.cc:435] Acquired schema lock
0709 12:26:36.062094 (+    23us) tablet.cc:662] Decoding operations
0709 12:26:36.064036 (+  1942us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:36.064160 (+   124us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:36.064196 (+    36us) tablet.cc:685] Acquiring locks for 26 operations
0709 12:26:36.126472 (+ 62276us) tablet.cc:701] Row locks acquired
0709 12:26:36.126538 (+    66us) write_op.cc:260] PREPARE: finished
0709 12:26:36.126767 (+   229us) write_op.cc:270] Start()
0709 12:26:36.126952 (+   185us) write_op.cc:276] Timestamp: P: 1783599996126677 usec, L: 0
0709 12:26:36.126999 (+    47us) op_driver.cc:348] REPLICATION: starting
0709 12:26:36.128015 (+  1016us) log.cc:844] Serialized 10008 byte log entry
0709 12:26:37.679347 (+1551332us) op_driver.cc:464] REPLICATION: finished
0709 12:26:38.038383 (+359036us) write_op.cc:301] APPLY: starting
0709 12:26:38.038504 (+   121us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:38.937332 (+898828us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:38.937447 (+   115us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:39.116407 (+178960us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:39.117066 (+   659us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=52,key_file_lookups=52,delta_file_lookups=52,mrs_lookups=0
0709 12:26:39.117145 (+    79us) write_op.cc:312] APPLY: finished
0709 12:26:39.152791 (+ 35646us) log.cc:844] Serialized 228 byte log entry
0709 12:26:39.363062 (+210271us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:39.389408 (+ 26346us) write_op.cc:454] Released schema lock
0709 12:26:39.414869 (+ 25461us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:39.429475 (+ 14606us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:39.429516 (+    41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2195716,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":351061,"apply.run_cpu_time_us":62321,"apply.run_wall_time_us":1391063,"cfile_cache_hit":106,"cfile_cache_hit_bytes":112807,"num_ops":26,"prepare.queue_time_us":462670,"prepare.run_cpu_time_us":24684,"prepare.run_wall_time_us":505435,"raft.queue_time_us":644985,"raft.run_cpu_time_us":10120,"raft.run_wall_time_us":397246,"replication_time_us":1492000,"thread_start_us":1400393,"threads_started":5,"wal-append.queue_time_us":904481,"wal-append.run_cpu_time_us":56032,"wal-append.run_wall_time_us":2651703}]]}
I20260709 12:26:41.911665 10702 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:41.936457  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.935567 10526 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:41.957949  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.930204 10612 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.935922 10412 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:41.931787 10577 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:41.972661  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:41.974637  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.974552  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.972563  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.979691  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:26:41.980687  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:41.992643 10780 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:41.991443  5454 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 8 to 9, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 9 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:41.993403 10780 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:41.995252  5552 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:41.994506  5454 catalog_manager.cc:5697] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 3 to 6, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 6 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:41.996068 10772 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 12:26:41.979516 10724 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.988621  6274 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:41.991504  5704 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:26:42.000795 10778 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:42.001209  5704 raft_consensus.cc:606] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:41.948194 10776 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:26:42.001639  5704 raft_consensus.cc:614] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:42.001489 10778 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:41.966954 10526 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:42.000087  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:42.003216  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:42.006765  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.973381 10412 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:42.010572  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:42.015843  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:42.016639 10632 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:42.017517  5631 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:42.018790 10632 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:42.019295 10781 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.022457  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:41.957577 10462 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.027912  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:42.031342  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:42.036731  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:42.037477  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:42.040478  5454 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 6 to 7, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 7 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:42.044263  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:41.986846 10714 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:26:42.048120  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:42.050763 10724 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:42.052451  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:42.052899  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:42.053045  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:42.086038  5557 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:42.054052  5561 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
I20260709 12:26:42.054312  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:42.054641  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.087370  5560 raft_consensus.cc:2410] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:26:42.056986  5554 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:26:42.057569  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.061484 10351 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:42.062585 10778 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:42.013092 10718 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.001344 10598 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.986205 10720 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:41.994678 10780 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:42.069543  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2364) took 1039 ms. Trace:
W20260709 12:26:42.075181 10493 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.560s	user 0.008s	sys 0.000s
I20260709 12:26:42.079223 10700 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:42.081223 10462 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:42.085337  5553 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:42.114176  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.114828  5560 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:42.088943  5559 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:42.115537  5553 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:41.984795 10577 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:42.095063  5558 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:42.099241  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.010102 10612 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:42.043107  5454 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 4 to 8, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 8 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:26:42.111472  6271 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:42.101555  5552 rpcz_store.cc:276] 0709 12:26:41.029929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:41.091192 (+ 61263us) service_pool.cc:224] Handling call
0709 12:26:42.069367 (+978175us) spinlock_profiling.cc:243] Waited 903 ms on lock 0x7b54000271d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:42.069483 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2076374400}
I20260709 12:26:42.096434  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:42.120164  5559 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:26:42.085475 10776 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.019834 10781 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:42.142674  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.143850  5560 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:42.147509  5561 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:42.157205  5555 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:26:42.157794  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 232702us
I20260709 12:26:42.168004  5454 catalog_manager.cc:5697] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:42.170202 10754 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.170733 10781 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:42.171496  5629 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:42.060211 10705 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.171389 10657 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:42.173048 10718 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:42.173265 10657 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:42.173770 10657 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:42.176337 10778 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.170491 10250 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:26:42.190907 10705 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:42.178269  5556 raft_consensus.cc:2368] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13.
I20260709 12:26:42.192440  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.192896 10700 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:42.193027  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.193601 10784 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.193776  5556 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:26:42.194645  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:42.195297  5556 raft_consensus.cc:2376] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12. Re-sending same reply.
W20260709 12:26:42.160516  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 64179.3us
I20260709 12:26:42.176924 10462 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:42.180940  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.212086  5561 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:42.176990 10778 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.185091 10773 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.185622 10598 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:42.185985  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:42.215663  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.186249 10454 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.186760  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.185642  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.193145  5558 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:42.196319  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.179944  5557 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:42.184633 10782 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:42.217905  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 286434us
I20260709 12:26:42.223357 10462 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.171308  5454 catalog_manager.cc:5697] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:42.231747  5556 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:42.249643  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.250681  5556 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:42.233402  5554 raft_consensus.cc:2368] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13.
I20260709 12:26:42.235176  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:42.251732  5557 raft_consensus.cc:2376] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12. Re-sending same reply.
I20260709 12:26:42.178761 10656 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:42.267084  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.194183 10784 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.276611  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.276929  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.268359  5555 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:42.268458  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.280946  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:42.282714  5557 raft_consensus.cc:2410] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
W20260709 12:26:42.284912  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:42.225033  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:42.267132  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 95869.7us
I20260709 12:26:42.302655  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:42.307569  5561 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:42.308622  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:42.308772  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.309782  5555 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:26:42.307410  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 319442us
I20260709 12:26:42.267498 10656 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:42.300974  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:42.316702  5560 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:42.335559  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:42.336495  5557 raft_consensus.cc:2376] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11. Re-sending same reply.
I20260709 12:26:42.214794 10776 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.343915  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:42.357750  5629 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:42.325142 10656 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:42.224499 10773 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:42.363452  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:42.318266 10754 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:42.369534  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 68712.3us
I20260709 12:26:42.370688  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:42.323318 10458 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:42.322775 10717 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:42.372062  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 449062us
I20260709 12:26:42.375766  5557 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:42.306229 10454 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:42.389551  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:42.393985  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:42.395107  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:42.280296 10782 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:42.401784  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:42.434330  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:42.450574  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 48949.8us
W20260709 12:26:42.455024  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:42.459033  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.459829  5557 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:42.456085  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:42.473021  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 96069.6us
I20260709 12:26:42.474080  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:42.474378  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:42.417253 10705 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:26:42.482435  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 63432.1us
I20260709 12:26:42.482626 10777 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:26:42.577085  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:42.580143  5629 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:42.583105  5629 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:42.583848  5676 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:42.383705 10717 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.385715 10458 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.581548  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:42.434511 10632 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:42.476886 10250 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:42.476953 10718 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:42.515607 10612 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:42.604740  5629 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.612859 10759 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:42.577215 10709 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20260709 12:26:42.615949  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:42.651180  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:42.616192 10782 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:42.616565 10493 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:26:42.657851  6276 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:26:42.658236  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:42.658624 10493 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:42.633880 10612 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:42.616849 10784 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:42.617460  5705 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:26:42.660776  5705 raft_consensus.cc:606] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:42.660391 10612 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:42.660979 10792 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.187s	user 0.000s	sys 0.000s
I20260709 12:26:42.661180  5705 raft_consensus.cc:614] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:42.618392 10454 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:42.618701 10632 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:42.662660 10776 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:42.662760 10612 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:42.663806  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.664024 10632 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:42.619688  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:42.620430 10462 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:42.590080 10720 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:42.621088 10783 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:26:42.665427  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45836.4us
I20260709 12:26:42.666973 10783 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:42.621618 10250 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:42.622351 10791 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.188s	user 0.000s	sys 0.000s
I20260709 12:26:42.669049  5709 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:26:42.622999 10788 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:26:42.626631 10412 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.587s	user 0.004s	sys 0.004s
I20260709 12:26:42.696949  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.697413  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.584333 10577 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:42.637295  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:42.590770 10786 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:42.609670  5454 catalog_manager.cc:5697] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 6 to 7, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 7 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:42.643330  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:42.509074  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:42.714138  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 205140us
I20260709 12:26:42.648277  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:42.648440  5629 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:42.646019  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:42.668315  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:42.671213  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.671795  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:42.670812  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.659729  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:42.668932  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:42.686429  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:42.698547  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.717694  5555 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:42.671873 10778 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:42.703007  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51937.7us
I20260709 12:26:42.741372  5559 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:42.637374  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:42.644814 10714 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:42.742440  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:42.672923 10621 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:42.713336  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:42.716866  5701 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:42.715906  5704 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:42.716775  5710 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:42.719823  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:42.748971  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:42.749583  5706 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:42.749867 10717 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:42.750983  6273 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:42.692986 10250 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.752655  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:42.764987  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 96799.7us
I20260709 12:26:42.765375  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:42.769212  6275 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:42.768788  5632 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:26:42.777412  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.789417  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:42.754684 10657 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.580s	user 0.010s	sys 0.003s
I20260709 12:26:42.800859  6278 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:42.808019  5555 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:42.813315  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:42.716714  5561 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:26:42.813004  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:42.821265  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 78908.9us
I20260709 12:26:42.822119  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:42.771663 10526 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.743s	user 0.011s	sys 0.002s
I20260709 12:26:42.824491  5705 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:42.764456 10705 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:42.821182  5709 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:42.754926 10788 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:42.825381  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 105969us
I20260709 12:26:42.693637 10462 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.079s
I20260709 12:26:42.711862 10412 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.838579  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:42.839018  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:42.654130 10759 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.839202  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:42.848562  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:42.840096  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:42.842433  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:42.832065  6236 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:42.847078  5701 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:42.848183  5704 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:42.857280  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:42.858569  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:42.859553  5632 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:42.871093  5629 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 12:26:42.881722 10632 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:42.882241 10632 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:42.882607 10632 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:42.887383 10621 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:42.887966 10621 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:42.888409 10621 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:42.893317  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 288935us
W20260709 12:26:42.893518  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61548.4us
I20260709 12:26:42.894985 10724 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:42.895396 10724 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:42.895758 10724 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:42.896389 10612 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:42.896972 10612 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:42.899494 10612 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:42.896401  5634 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:26:42.900610  5634 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:42.901451  5634 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.902391  5634 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:26:42.902745  5602 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.568s (SENT)
I20260709 12:26:42.903394  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:42.904320 10724 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:42.909552  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:42.969255  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 59816.4us
I20260709 12:26:42.970471  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:42.911677  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:26:42.972326  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60721.9us
I20260709 12:26:42.852219 10778 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:42.935324 10759 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:42.868599 10754 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:42.939127 10725 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.973s	user 0.012s	sys 0.005s
I20260709 12:26:42.765182 10786 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:42.825683 10657 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:42.940423  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61230.6us
I20260709 12:26:42.941884 10250 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:42.944514  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:42.945835  5634 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 8. (term mismatch)
I20260709 12:26:42.829608 10526 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.949043 10621 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.950269 10632 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:42.954075  5557 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:43.005941  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:42.955317  5560 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:26:42.851325 10577 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:42.858151 10782 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:42.959364 10412 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:42.974560  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:42.924660 10788 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:43.021452  5635 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:43.022590  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:43.024475  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:42.991295 10601 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:42.978047 10477 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:42.978595 10787 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:42.979749  5559 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:26:42.983166 10705 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:26:42.973265 10801 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 12:26:42.984336 10778 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:42.871779  5629 pending_rounds.cc:85] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 6
I20260709 12:26:43.036698  5629 pending_rounds.cc:107] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 12:26:43.037000  5629 raft_consensus.cc:2889] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:43.029989 10632 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:42.993134  5634 pending_rounds.cc:85] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 7
I20260709 12:26:43.038434  5634 pending_rounds.cc:107] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 15.8
I20260709 12:26:43.038801  5634 raft_consensus.cc:2889] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:43.002854  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 100165us
I20260709 12:26:42.972580 10717 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:26:43.012883  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 243866us
I20260709 12:26:43.013553  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:43.016281  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:43.046746  5560 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:43.019375  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.020326  5630 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:42.995939 10725 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:26:43.026937  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:42.976817  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:43.030853  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.032162  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:43.050455  5628 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:43.011468 10577 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:43.011651 10782 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:43.002768 10754 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:43.031452 10787 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:43.045183  6275 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:26:43.053040  5555 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:43.062427  6271 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
W20260709 12:26:43.053612  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 239422us
I20260709 12:26:43.041553 10705 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:43.059249  5634 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:26:43.043053  5707 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:26:43.066461  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:43.069991  5701 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:43.072044  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:43.083222  5705 raft_consensus.cc:2410] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260709 12:26:43.060808 10702 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.074s	user 0.011s	sys 0.003s
I20260709 12:26:43.039543  5454 catalog_manager.cc:5697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 5 to 7, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 7 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:43.089993  5706 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:26:43.096925  5706 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 8. (term mismatch)
I20260709 12:26:43.097396  5706 pending_rounds.cc:85] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 7
I20260709 12:26:43.097651  5706 pending_rounds.cc:107] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260709 12:26:43.056860 10577 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:43.097877  5706 raft_consensus.cc:2889] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:43.099958  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.091578  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:43.058562 10782 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:43.102326  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:43.086688  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:43.059710 10458 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:43.068776 10787 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:43.061372 10724 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:43.111392  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:43.112008  5709 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 13: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:43.110638 10702 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:43.116025 10632 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:43.117019 10816 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:26:43.119513  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 74390.7us
I20260709 12:26:43.120538  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:43.232474  5701 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:43.070245 10705 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:43.084484 10526 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:43.060226 10398 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:26:43.055539 10250 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:26:43.116524 10812 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:43.065760 10783 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:43.106578 10714 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:43.037112 10778 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:43.108227 10601 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 12:26:43.056376 10700 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:26:43.218607 10621 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:43.155865  5705 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:26:43.243268  5554 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:43.218006 10773 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:43.250638  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:43.162002  5628 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:43.155839  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:43.222965  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.230975  5560 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:43.237569  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:43.284807  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 166343us
I20260709 12:26:43.240692  6278 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:26:43.115360 10811 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:26:43.231814 10632 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:43.234328 10816 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:43.239765  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.295519  5709 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:43.298970  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.294891 10632 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:43.303651  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:43.318804  5706 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:43.264560 10807 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.256s
I20260709 12:26:43.321415  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.321703 10702 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:43.322181  5709 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:43.323331  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:43.305629 10657 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:43.299714 10816 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:43.300314 10621 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:26:43.331637  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:43.334053  5701 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:43.338464  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:43.300803 10786 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:43.339169  6278 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:43.343688  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.313094 10754 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:43.347074  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.349318  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:43.336488 10621 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:43.342581 10829 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:26:43.367693  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:43.373422  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 135916us
I20260709 12:26:43.344836 10526 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:26:43.368471  6276 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.010s	sys 0.006s
I20260709 12:26:43.347246 10835 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:43.377734  6276 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.010s	sys 0.006s
I20260709 12:26:43.380270  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:43.353205 10412 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:43.383080  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:26:43.383416  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.384406  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:43.388962 10577 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:43.395443 10830 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
I20260709 12:26:43.371811 10621 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:43.403313 10705 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:43.403932  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 932) took 1205 ms. Trace:
I20260709 12:26:43.399171  5706 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:43.399719 10601 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:43.385264 10835 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:43.429697  5705 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:43.401674 10577 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:43.401706  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:43.405444 10831 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:26:43.406700  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:43.409018 10705 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:43.413674  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2236) took 1224 ms. Trace:
I20260709 12:26:43.439627  5708 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
W20260709 12:26:43.376305 10656 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.978s	user 0.016s	sys 0.001s
I20260709 12:26:43.444036 10656 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:43.446650 10656 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:43.416996 10612 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:26:43.448815  5702 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:43.387303 10754 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:26:43.385804 10818 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:26:43.449908 10835 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:43.450498  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.451843  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.452339  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.453027  5701 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:43.453271 10835 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:43.453351  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.429028  6271 rpcz_store.cc:276] 0709 12:26:42.198538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:42.214122 (+ 15584us) service_pool.cc:224] Handling call
0709 12:26:42.267537 (+ 53415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:43.066489 (+798952us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:43.403625 (+337136us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54004c7a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:43.403859 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1823443840}
I20260709 12:26:43.443089  5554 rpcz_store.cc:276] 0709 12:26:42.189101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:42.276648 (+ 87547us) service_pool.cc:224] Handling call
0709 12:26:43.413475 (+1136827us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b54000eb5d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:43.413614 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2220506368}
W20260709 12:26:43.442103  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 401801us
I20260709 12:26:43.454002  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:43.455034  5602 leader_election.cc:336] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.585s (SENT)
I20260709 12:26:43.461248  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:43.462356  6278 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:43.465065  6276 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:43.424674 10773 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:43.466192  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:43.398878 10412 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:43.477784  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:43.453930 10835 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:43.494051  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:43.495800  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 111515us
I20260709 12:26:43.493486  5705 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:43.475126 10412 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:43.504380  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:43.507473 10621 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:43.478727 10657 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:26:43.500828  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 133768us
I20260709 12:26:43.513116  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:43.502193  5706 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:43.514835 10787 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:43.515961  5706 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:26:43.516867  5708 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: [13.8-13.8]
W20260709 12:26:43.519311  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 64410.4us
W20260709 12:26:43.520012 10824 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.343s	user 0.002s	sys 0.000s
W20260709 12:26:43.520148  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:43.520679  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:43.522486  5602 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.548s (SENT)
I20260709 12:26:43.522848  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:43.523944 10632 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:43.525290 10700 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:43.529876 10700 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:43.532477 10778 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:26:43.533015 10700 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:43.535456 10621 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:43.526433  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:43.538928 10826 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.334s	user 0.000s	sys 0.002s
I20260709 12:26:43.539364  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:43.541014 10702 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:43.541062 10837 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:43.541505 10250 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:26:43.527269 10458 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:26:43.542358  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:43.542559  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:43.527712  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:43.543151  5554 raft_consensus.cc:2368] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12.
I20260709 12:26:43.543541  6237 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:43.543726  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:43.544201  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:43.544759  5701 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:43.544914  5552 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:43.545766  5560 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:26:43.536471  5709 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:43.546605 10725 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:43.546763  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:43.547106 10725 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:43.547432  5629 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:43.547506 10725 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:43.536091 10782 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:43.549609 10832 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.255s	user 0.001s	sys 0.000s
I20260709 12:26:43.537935 10632 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:43.526286 10787 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:43.528332 10657 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:43.544212 10759 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:43.551134 10657 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:43.551169 10759 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:43.552009 10657 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:43.479367 10462 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:26:43.535962  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:43.555014  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:43.558986  5629 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:26:43.560003 10759 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:43.564155  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:43.565117  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:43.564000 10842 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:26:43.565766  5559 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:26:43.565852  5560 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:43.567484  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:43.571012  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:43.571247  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:43.573733  5555 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:43.574275  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:43.576027  5552 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 11: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:43.576808  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:43.578470  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:43.575534  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:43.579543  5553 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:43.552702 10803 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:43.549664 10782 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:43.588519 10812 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:43.589113 10812 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:43.591833 10632 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:43.593602 10754 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:43.594298 10754 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:43.599404  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:43.596685 10724 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:26:43.601789 10807 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:26:43.619693 10724 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:43.620790 10724 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:43.622133 10838 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:26:43.628324  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:43.629053  5702 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:43.636729  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 137981us
I20260709 12:26:43.602901 10785 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:26:43.643193  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:43.644047  5629 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
W20260709 12:26:43.640954  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 163886us
I20260709 12:26:43.648681 10833 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:43.649502 10833 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:43.650765 10833 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:43.656332 10412 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:43.658646 10398 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:43.659690 10398 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:43.658962 10412 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:43.688390 10412 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:43.602411 10803 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:43.689931 10803 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:43.694730 10412 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:43.695443 10412 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:43.633641 10612 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:43.711949  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2220) took 1303 ms. Trace:
I20260709 12:26:43.681604  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:43.700690  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:43.728288 10788 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:26:43.730113  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 92340.9us
I20260709 12:26:43.731405  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:43.735527 10788 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:26:43.766050 10783 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:43.766799 10783 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:43.791446  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:43.712551  5630 rpcz_store.cc:276] 0709 12:26:42.378259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:42.378395 (+   136us) service_pool.cc:224] Handling call
0709 12:26:43.660887 (+1282492us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x7b54000c0b54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:43.682096 (+ 21209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1413123456}
I20260709 12:26:43.640661 10835 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:43.850035  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:43.669869 10785 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:43.824575 10612 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:43.823572 10462 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:43.844101 10499 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:43.876269 10499 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:43.876885 10499 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:43.877854 10848 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:43.878811 10848 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:26:43.847720 10833 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:26:43.872143  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:43.883569 10816 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.554s	user 0.006s	sys 0.005s
I20260709 12:26:43.884248 10816 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:43.863940 10398 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:43.863754 10785 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:43.871963 10612 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:43.891383  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:43.892020  5703 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:43.893567 10847 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:43.894071 10847 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:43.894428 10847 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:43.894589 10816 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:43.874927 10462 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:43.872221  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:43.902456  5702 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:43.947295 10657 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:43.949436  5706 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:26:43.950459 10632 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:26:43.926994 10823 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.679s	user 0.000s	sys 0.011s
I20260709 12:26:43.952337 10725 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.003432  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:44.004091  5704 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:43.978554 10754 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:43.981177 10601 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.538s	user 0.009s	sys 0.000s
W20260709 12:26:43.983639  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 243315us
I20260709 12:26:43.987530  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:43.993432  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 346790us
I20260709 12:26:44.034526  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2492) took 1519 ms. Trace:
I20260709 12:26:44.035100  5633 rpcz_store.cc:276] 0709 12:26:42.577547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:42.625793 (+ 48246us) service_pool.cc:224] Handling call
0709 12:26:42.626136 (+   343us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:42.626632 (+   496us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:42.626662 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:42.627132 (+   470us) log.cc:844] Serialized 26 byte log entry
0709 12:26:42.984819 (+357687us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:42.984953 (+   134us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:42.985028 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:43.875624 (+890596us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:44.034132 (+158508us) raft_consensus.cc:1685] finished
0709 12:26:44.034160 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:44.034434 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":200,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":180,"spinlock_wait_cycles":5672192,"thread_start_us":357318,"threads_started":1,"wal-append.queue_time_us":488673}
I20260709 12:26:44.038350  5635 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:44.039765  5630 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:44.023504  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:44.041105  5706 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 13: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:44.038959 10725 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:43.997628 10621 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:43.996172  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:44.000223  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:44.007390  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:44.058827  5628 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:26:44.058979  5701 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:44.049492  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:44.062588  5704 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:44.063851  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:43.995069  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:44.004555 10816 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:44.074402  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:44.075845  5557 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:44.072289  5454 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.002s	sys 0.016s
I20260709 12:26:43.956848 10783 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:44.122510 10783 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:44.132834  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:44.133836  5708 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 16 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:26:44.134713  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:44.135246  5708 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:44.136404 10759 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:44.137941  5557 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
W20260709 12:26:44.072656  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.010s	sys 0.009s
I20260709 12:26:44.105289  5701 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:44.106118  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:44.146639  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.010s	sys 0.010s
I20260709 12:26:44.111271  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:44.091526  5706 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:26:44.041395 10601 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:44.117338  5702 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:26:44.139166  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:44.085680  5628 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
W20260709 12:26:44.151418  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 607632us
I20260709 12:26:44.157281  5703 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
W20260709 12:26:44.106243 10812 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.517s	user 0.008s	sys 0.004s
W20260709 12:26:44.112488  5454 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.003s	sys 0.016s
I20260709 12:26:44.086238 10700 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:44.187870  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:44.080226 10632 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:44.188894  5633 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:44.079686 10857 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:26:44.192428  5704 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
W20260709 12:26:44.194317  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 93579.4us
I20260709 12:26:44.107975 10776 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.100s
W20260709 12:26:44.203768  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:44.205506  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 683001us
W20260709 12:26:44.206224  5602 leader_election.cc:336] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.571s (SENT)
I20260709 12:26:44.206759  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:44.207732  5629 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:44.191378 10783 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:44.088748 10816 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:44.191872 10632 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:44.218166  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:44.192123 10857 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:44.237208  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:44.237656  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:44.199373 10598 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:44.239204  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:44.201162 10835 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:44.240756  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:44.187295 10812 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.250590  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:44.253283  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 222391us
I20260709 12:26:44.265012  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:44.267741  5557 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:44.268445  5559 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:26:44.268733  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:44.270246  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:44.191210 10836 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:26:44.267511  5561 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:44.220292 10783 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:44.193529 10700 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.273348  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:44.296883  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:44.221469 10725 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:44.227108 10632 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:44.299225  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:44.299371  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:44.299719  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:44.300204  5555 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:44.300364  5559 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:44.299849  5561 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:44.307673  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:44.308322  5554 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:44.308496  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 102301us
I20260709 12:26:44.297571 10783 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:44.312299 10462 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:44.278698 10754 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:44.229619 10847 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.272120 10598 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.271457 10835 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.343847  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2474) took 1342 ms. Trace:
I20260709 12:26:44.342908  5559 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:44.321693 10657 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:44.355696  5636 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:44.307943 10725 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.301177 10499 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.432372 10598 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:44.433307 10499 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:44.434986  5556 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:26:44.437345  6276 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:44.440845  5708 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
W20260709 12:26:44.370239  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 288303us
I20260709 12:26:44.371521  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:44.442312  5559 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:44.372380  5555 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:44.344524  5710 rpcz_store.cc:276] 0709 12:26:42.992469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:42.999866 (+  7397us) service_pool.cc:224] Handling call
0709 12:26:43.015886 (+ 16020us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:43.059192 (+ 43306us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:26:43.103269 (+ 44077us) log.cc:844] Serialized 14 byte log entry
0709 12:26:43.351917 (+248648us) log.cc:844] Serialized 14 byte log entry
0709 12:26:43.522026 (+170109us) spinlock_profiling.cc:243] Waited 4.56 ms on lock 0x7b1800089920. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:43.530057 (+  8031us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b1800089920. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1
0709 12:26:43.530773 (+   716us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b1800089920. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:43.979134 (+448361us) spinlock_profiling.cc:243] Waited 8.28 ms on lock 0x7b1800089920. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9
0709 12:26:43.980525 (+  1391us) spinlock_profiling.cc:243] Waited 922 us on lock 0x7b1800089920. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:26:44.333876 (+353351us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:44.333929 (+    53us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:44.333977 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:44.334047 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:44.334170 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":86,"prepare.queue_time_us":46792,"prepare.run_cpu_time_us":5156,"prepare.run_wall_time_us":36713,"raft.queue_time_us":31636,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":11997,"spinlock_wait_cycles":65793152,"thread_start_us":521614,"threads_started":2,"wal-append.queue_time_us":340014}
I20260709 12:26:44.447510  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:44.403076 10747 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:44.404804  6277 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:44.409837  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:44.412020  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:44.415432  6279 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: []
I20260709 12:26:44.416472  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2468) took 1747 ms. Trace:
I20260709 12:26:44.417693  6275 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:44.419047  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:44.421813 10816 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:44.454332  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:44.461225  6274 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:26:44.461393  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:44.432786 10835 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:44.458197  5707 rpcz_store.cc:276] 0709 12:26:42.669493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:42.669950 (+   457us) service_pool.cc:224] Handling call
0709 12:26:42.670299 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:43.068289 (+397990us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:44.376133 (+1307844us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54002d0ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:44.416392 (+ 40259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":809106176}
I20260709 12:26:44.476263 10804 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:44.477275 10351 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:44.450012 10783 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:44.507462  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:44.509016  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:44.510820  5557 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:44.347512 10857 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.525409  5557 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 12:26:44.529470  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 220257us
W20260709 12:26:44.530164  5602 leader_election.cc:336] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.945s (SENT)
I20260709 12:26:44.507215 10804 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:26:44.533703  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 377328us
I20260709 12:26:44.496308 10598 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 12:26:44.496104 10336 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:26:44.538599  5557 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:26:44.540657  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 953) took 1845 ms. Trace:
I20260709 12:26:44.548796  5629 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:26:44.544322  5633 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:26:44.549119  5705 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:44.550406  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:44.551271  5708 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 16: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:44.552276  5559 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
W20260709 12:26:44.545274  5675 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.543s (SENT) [suppressed 48 similar messages]
I20260709 12:26:44.513095 10351 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.554582  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:44.576546  5710 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:44.570739 10862 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:44.556962  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 81927.7us
I20260709 12:26:44.553141  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:44.573572  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:44.574323  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 435222us
I20260709 12:26:44.550290  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:44.582733  5602 leader_election.cc:336] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.977s (SENT)
I20260709 12:26:44.582860  5599 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:44.583256  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:44.583988  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:44.575230  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:44.586583  5703 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:44.570919 10833 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.690s	user 0.010s	sys 0.006s
I20260709 12:26:44.547384  6280 rpcz_store.cc:276] 0709 12:26:42.695513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:42.695802 (+   289us) service_pool.cc:224] Handling call
0709 12:26:42.741627 (+ 45825us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:43.012762 (+271135us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:26:43.030462 (+ 17700us) log.cc:844] Serialized 14 byte log entry
0709 12:26:43.099729 (+ 69267us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:43.099779 (+    50us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:43.099820 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:44.540469 (+1440649us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b54002e3954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:44.540510 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:44.540606 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22544,"prepare.queue_time_us":28215,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":40550,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"spinlock_wait_cycles":616684928,"thread_start_us":61860,"threads_started":1,"wal-append.queue_time_us":202849,"wal-append.run_cpu_time_us":13302,"wal-append.run_wall_time_us":1235473}
I20260709 12:26:44.600263  5704 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:26:44.593142  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:44.615404  5708 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:44.550856 10700 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:44.573334 10598 consensus_queue.cc:1243] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 13.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:26:44.574157 10812 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:44.643862  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 50808.4us
I20260709 12:26:44.652446  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:44.652755  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:44.653080  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:44.655369  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:44.656399  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:44.657117  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:44.655751  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:44.652192  5635 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: []
W20260709 12:26:44.696039  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:44.656785  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:44.649736  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:44.480008 10747 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.768291  5702 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:44.591590 10862 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:44.700906  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 108716us
W20260709 12:26:44.769910  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 196141us
I20260709 12:26:44.677273  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:44.570451 10657 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.072s
I20260709 12:26:44.570384 10458 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.073s
I20260709 12:26:44.616178 10601 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:26:44.651968 10412 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.956s	user 0.010s	sys 0.007s
W20260709 12:26:44.647660 10612 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.756s	user 0.012s	sys 0.003s
I20260709 12:26:44.614562 10871 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:44.633193 10865 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:26:44.648373 10867 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 12:26:44.653702 10863 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
W20260709 12:26:44.673734 10866 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
W20260709 12:26:44.661254 10859 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
I20260709 12:26:44.779842  5554 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:44.791687  5705 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
W20260709 12:26:44.649273 10860 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
I20260709 12:26:44.793358  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:44.785200  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:44.798528  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:44.774798 10785 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.882s	user 0.014s	sys 0.002s
I20260709 12:26:44.677510 10754 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:44.641614 10880 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:44.649567 10783 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:44.770294 10847 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:44.809043  6280 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
W20260709 12:26:44.815680  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 270413us
W20260709 12:26:44.873682  5561 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.559s	user 0.010s	sys 0.001s
I20260709 12:26:44.874223  5561 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:44.776530 10869 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:44.788838 10601 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:44.801241 10881 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:44.873960  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:44.803464 10871 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:44.894752  5707 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:44.804399 10612 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.803520 10412 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:44.860812 10868 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:44.870544 10725 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:44.875767 10477 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:44.910439  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:44.914479 10612 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:44.918121 10873 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
W20260709 12:26:44.920233 10874 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:26:44.896539 10869 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:44.922217 10881 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:44.923384 10881 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:44.923897 10869 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:44.927027 10856 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.890s	user 0.000s	sys 0.002s
I20260709 12:26:44.918208 10351 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:44.929294 10788 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:44.929772 10788 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:44.930063  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 916) took 1368 ms. Trace:
W20260709 12:26:44.931491 10759 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.795s	user 0.008s	sys 0.001s
I20260709 12:26:44.936414 10862 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:44.855072 10876 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.096s
I20260709 12:26:44.938795 10876 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:44.939183 10876 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:44.940563  5627 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:44.941804 10398 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:44.943563  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:44.946242 10747 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:44.874951 10880 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.949319 10880 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:44.898852 10871 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:44.950248 10849 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:44.950826 10849 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:44.951557 10857 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:44.953933 10412 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:44.956578  5632 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:44.961791 10849 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:44.962522 10849 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:44.962997 10849 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:44.965095 10849 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:26:44.895027 10812 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:44.966145 10812 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:44.968678  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 183590us
I20260709 12:26:44.867544 10785 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.930858 10788 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.904708 10883 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:44.787390 10804 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:44.894546 10601 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:44.972438  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 319643us
I20260709 12:26:44.906206 10847 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:44.933795 10477 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:44.930598  5558 rpcz_store.cc:276] 0709 12:26:43.561765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:43.561881 (+   116us) service_pool.cc:224] Handling call
0709 12:26:43.562248 (+   367us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:43.578218 (+ 15970us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:43.578257 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:43.582533 (+  4276us) log.cc:844] Serialized 52 byte log entry
0709 12:26:43.856153 (+273620us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:43.856216 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:43.856313 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:44.926246 (+1069933us) spinlock_profiling.cc:243] Waited 2.75 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:44.929763 (+  3517us) raft_consensus.cc:1685] finished
0709 12:26:44.929784 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:44.929984 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8684,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":6900,"spinlock_wait_cycles":6352384,"thread_start_us":231633,"threads_started":1,"wal-append.queue_time_us":435794}
W20260709 12:26:44.988183  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:44.997557  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 320340us
I20260709 12:26:44.998728  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:45.002189  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:45.005317  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.002311  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:45.009639  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:45.034312  6276 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
I20260709 12:26:45.005026 10880 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 12:26:44.997218 10816 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:45.061317 10816 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.063498 10816 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:45.064267  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:45.064886  5707 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:44.990150 10847 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:45.013626 10632 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.692s	user 0.010s	sys 0.002s
I20260709 12:26:45.044179  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:45.075687  5705 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:45.045137  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:45.047808  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48465.2us
I20260709 12:26:45.079525  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:45.048547  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:45.048776  5703 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:26:45.050910  6277 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:26:44.940035 10876 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.009333 10869 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:45.088171  5709 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 16: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:45.051882 10857 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:45.052960  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:45.095325  5707 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
W20260709 12:26:45.047808 10885 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:26:45.002776 10812 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:45.057413 10893 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:45.059994  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:45.037611 10849 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260709 12:26:45.044118  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:44.984117 10621 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 8, Time since last communication: 0.058s
I20260709 12:26:45.006629 10783 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:45.135902 10785 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:45.136027 10783 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.139045  5706 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:45.140331 10621 consensus_queue.cc:1243] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 16.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:26:45.142719  5701 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:26:45.143133  5701 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:45.095587 10854 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:26:45.038100 10458 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
W20260709 12:26:45.112928  6234 leader_election.cc:336] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.815s (SENT)
I20260709 12:26:45.075656 10847 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:45.158274  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:45.145404 10871 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:45.160801  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 81371.8us
I20260709 12:26:45.161950  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.144042  5701 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.164037  5701 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:26:45.166389  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53582.2us
I20260709 12:26:45.134941 10849 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:26:45.092676  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:45.126189  5705 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:26:45.152904  5706 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:45.155431 10816 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:45.078568 10632 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.173629  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:45.180299  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 178196us
I20260709 12:26:45.148944 10869 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:45.187641  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:45.188330  6277 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:26:45.148808 10657 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20260709 12:26:45.179720 10880 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:45.191028  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:45.150228 10857 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.191473  6275 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:45.199681  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:45.199930  5710 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 12:26:45.183183 10892 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:45.197113  5630 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:45.201635 10878 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:26:45.201838  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:45.202553  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:45.203141  5635 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:45.204016 10632 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:45.205474 10871 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:45.207962  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:45.208619  6272 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:26:45.202548  5634 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:26:45.183586 10816 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:45.180058 10754 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:45.190488  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.183379 10876 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:45.241976 10754 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.195953  5636 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:45.233521 10892 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:45.194029  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 145697us
I20260709 12:26:45.192801  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:45.226137 10816 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:45.196188 10700 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:45.190994 10849 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:45.242154  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:45.213047 10783 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:45.271781  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:45.236135 10881 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:45.246549  5701 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:26:45.252723  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:45.252944 10880 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:45.252749  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:45.253547  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2394) took 1019 ms. Trace:
I20260709 12:26:45.253468  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:45.275547  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:45.255087  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:45.254319  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:45.259654 10869 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:45.265573  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 72952.8us
I20260709 12:26:45.226444  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:45.275655  5553 rpcz_store.cc:276] 0709 12:26:44.234217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:44.236449 (+  2232us) service_pool.cc:224] Handling call
0709 12:26:44.270237 (+ 33788us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:44.604841 (+334604us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:44.604884 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:44.641193 (+ 36309us) log.cc:844] Serialized 26 byte log entry
0709 12:26:45.078279 (+437086us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:45.078319 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:45.078388 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:45.253338 (+174950us) raft_consensus.cc:1685] finished
0709 12:26:45.253359 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:45.253494 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":266480,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":27255,"thread_start_us":612860,"threads_started":2,"wal-append.queue_time_us":423630}
I20260709 12:26:45.234761 10458 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:45.271059  5709 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:45.273591  5555 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:26:45.267690 10882 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:26:45.312992 10882 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:45.313941 10812 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:45.314237 10882 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.314757 10812 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:45.302675 10621 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
I20260709 12:26:45.316788  5704 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:26:45.317590 10857 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:45.318436  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:45.319617 10754 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:45.319713  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:45.320747  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:45.320674  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:45.321379  5703 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:45.271054 10849 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:45.322700 10849 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.323976 10849 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:45.305310 10835 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260709 12:26:45.325613  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:45.304163 10881 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.327620  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:45.328775  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:45.330557 10881 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:45.333165  5552 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:45.336326  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:45.336716 10892 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.275729  5558 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:45.199393 10884 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260709 12:26:45.341483  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:45.248108  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57705.1us
I20260709 12:26:45.342911  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.226605 10632 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:26:45.343698  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:45.283385  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.343011  5553 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:45.344873  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:45.349715  5559 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:45.344254  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:45.350154  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:45.197836  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:45.303994  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:45.352139  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.353950  5561 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:45.358858  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:45.360057 10754 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:45.360888 10754 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:45.362118 10754 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.358705  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:45.375476  5559 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:26:45.378183  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:45.378376  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:45.340552 10458 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:45.381783  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 108270us
I20260709 12:26:45.306565 10786 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:45.271826 10871 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.079s
I20260709 12:26:45.318449 10857 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:26:45.385401 10725 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:26:45.350790 10880 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 12:26:45.392951 10632 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:45.395123 10601 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:45.399096 10900 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.163s	user 0.000s	sys 0.000s
I20260709 12:26:45.402894 10700 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:45.403091 10812 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:45.412410  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:45.444128  5559 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:26:45.416396  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.418022  5709 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:45.420152  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.438802  6278 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:45.409737 10884 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:45.442330  5705 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:26:45.410115 10786 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:45.472186 10786 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:45.434228 10794 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:26:45.428435 10621 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:45.440670  5558 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:45.452126 10902 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:45.442853 10880 consensus_queue.cc:1243] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:26:45.479648  5629 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:45.480724 10601 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:45.481227  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:45.481477  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44758.8us
I20260709 12:26:45.482070  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2500) took 1498 ms. Trace:
I20260709 12:26:45.482689  6237 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:45.483003  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.483012 10499 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:26:45.483829  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:45.484048 10499 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:45.484441 10499 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:45.484751 10462 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:45.484974 10780 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:26:45.485467 10780 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:45.482427  5632 rpcz_store.cc:276] 0709 12:26:43.983807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:43.991129 (+  7322us) service_pool.cc:224] Handling call
0709 12:26:44.003478 (+ 12349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:44.957040 (+953562us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:45.481854 (+524814us) spinlock_profiling.cc:243] Waited 953 ms on lock 0x7b540032d2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:45.482015 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2191417088}
I20260709 12:26:45.486219 10780 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:45.484519 10804 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:45.417093 10857 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:45.489508 10857 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.490523 10857 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:45.464666 10835 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:45.488637 10725 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:45.492733 10725 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:45.493399 10477 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:45.493604 10725 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.494323 10870 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:26:45.494901 10870 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:45.452888 10458 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.495903 10458 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:45.494367 10499 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.500638  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:45.501194  5705 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:45.483086  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63053.8us
I20260709 12:26:45.503237  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.509284  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.495747 10870 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.444290 10700 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:45.526551  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:45.527645  5556 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:26:45.513075  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 170224us
I20260709 12:26:45.538007  5704 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:26:45.540666  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 162557us
I20260709 12:26:45.541380  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:45.541980  5705 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:45.542747  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:45.543828  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:45.513387 10794 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:26:45.521569 10745 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:45.525969 10870 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:26:45.557184  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 72863us
I20260709 12:26:45.522106 10398 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:45.559134  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:45.568225  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:45.572026  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:45.546581 10862 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:26:45.548513 10526 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:45.576435  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:45.586963  5556 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:26:45.588459  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:45.564823 10632 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:45.590444 10632 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:45.576018 10621 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:45.595821 10883 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:45.596292 10883 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:45.596599 10847 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.597077 10883 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.598192 10883 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:45.578372 10700 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.600841 10816 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.603121 10499 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:45.576088 10782 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 12:26:45.604115 10804 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:45.575945 10398 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.606769 10398 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:45.607345 10794 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:26:45.607906 10794 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:45.608556 10700 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:45.609447 10869 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:45.610813  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.611346  5552 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:45.539425 10812 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:26:45.612866  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:45.615724  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:45.616479  5554 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:45.590898 10632 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:45.609894 10869 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:45.634120 10869 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.635542 10869 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:45.632591 10462 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:45.577936 10783 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:45.645051  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.615876  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:45.609167 10794 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.652976  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:45.579254 10776 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.068s
W20260709 12:26:45.649438 10865 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.770s	user 0.016s	sys 0.046s
I20260709 12:26:45.644227 10849 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:45.677462 10849 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:45.678736 10849 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:45.684278 10889 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.595s	user 0.000s	sys 0.003s
I20260709 12:26:45.640332 10786 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.668164 10862 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:45.669335  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:45.672837 10876 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:45.674028 10776 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:45.687665  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:45.640856 10847 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:45.651824 10526 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:26:45.720413  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 109714us
I20260709 12:26:45.721728  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:45.725270  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 56023.4us
I20260709 12:26:45.686794 10786 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:45.647074 10868 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.072s
I20260709 12:26:45.693735 10835 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:26:45.743392  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:45.743885  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 977) took 1092 ms. Trace:
I20260709 12:26:45.727018  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:26:45.696594 10865 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.821s	user 0.016s	sys 0.047s
I20260709 12:26:45.736550  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.739010 10876 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:45.732895 10917 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:45.741032 10526 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:45.725777 10780 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:45.744292  5627 rpcz_store.cc:276] 0709 12:26:44.651556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:44.652482 (+   926us) service_pool.cc:224] Handling call
0709 12:26:45.733653 (+1081171us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b54001de2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:45.743824 (+ 10171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":586532608}
I20260709 12:26:45.746400  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:45.746568  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:45.774740  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:45.775154  5629 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:45.775413  6276 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:45.749150  6274 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15. Ops: []
W20260709 12:26:45.753400  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:45.725415 10412 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:45.725237 10462 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:45.762115  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.795921  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:45.773725  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:45.776257  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:45.776371  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 118247us
I20260709 12:26:45.740310 10598 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:45.779436  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.725617 10862 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:45.725886 10776 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:45.781589  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:45.769054 10458 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:26:45.768831 10865 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:45.769423 10621 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:45.770756 10816 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:45.806493  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:45.776753 10912 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:26:45.809676  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:45.810396  5632 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
W20260709 12:26:45.810945  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48927.8us
I20260709 12:26:45.811731  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:45.813305  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:45.813958  5635 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:26:45.817802  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.818763  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:45.819368  5631 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:45.821534  5627 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:26:45.823395  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:45.824210  5554 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:45.782029 10917 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:45.830077  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:45.836318  5630 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:26:45.831000  5599 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:45.833141  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:45.837085  5628 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:45.800396 10918 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:45.822733  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:45.893191 10776 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.831036 10601 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:45.894512 10621 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.807577 10783 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:45.809795 10718 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:45.779124 10882 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:26:45.802361 10598 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:45.807788 10881 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:45.892422 10526 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.894336 10865 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:45.911214  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70555.8us
I20260709 12:26:45.958827 10912 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:26:45.889293 10919 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:26:45.801329 10462 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:45.997571 10718 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:45.807224 10878 raft_consensus.cc:2764] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:26:45.821808 10794 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:46.002810 10794 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:46.003290 10794 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:46.003218 10783 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.004705 10836 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:46.005141 10836 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:46.005985 10836 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.006983 10772 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:46.007270 10836 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:46.007403 10772 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:45.838568 10848 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:26:45.862043 10412 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:46.008322 10772 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:45.916828  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:45.830641 10902 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:45.946112  5628 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:45.793736 10914 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
W20260709 12:26:45.957378  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 150907us
I20260709 12:26:45.961933  5630 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
W20260709 12:26:45.972764  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:45.977726  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 65571.4us
I20260709 12:26:46.024142  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:45.916208 10899 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
I20260709 12:26:45.996155  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:46.034229  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:45.874838  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63169.6us
I20260709 12:26:46.002619 10718 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:45.817689 10458 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:46.027026 10398 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:26:46.028673 10876 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:26:46.029422 10914 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:45.989727  5554 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:45.999218 10912 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:46.043677  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:46.052875  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:46.034853  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:46.009559 10772 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:46.040123  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:46.043578  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 225690us
I20260709 12:26:46.025660 10477 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:45.916970 10862 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.060642  5553 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:45.984187 10632 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.041592 10850 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:26:46.073909  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:46.049096 10912 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:46.078725 10921 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:26:46.078820 10927 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260709 12:26:46.031020 10878 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:26:45.915864 10917 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.045687 10876 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:46.075758  5630 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:46.080596 10876 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:46.068495  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:46.081563 10922 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:26:46.067356 10785 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:46.084815 10785 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:46.045534 10892 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 12:26:46.085817 10892 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:46.086591 10892 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:46.086084 10785 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.009739 10462 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.061657 10657 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:46.041035 10794 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.097106  5630 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
W20260709 12:26:46.097268  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 73524.7us
W20260709 12:26:46.102804  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 296661us
I20260709 12:26:46.105234  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:46.105608  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52824.2us
I20260709 12:26:46.106925  5705 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:46.080253 10924 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:46.076500 10780 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:46.114128  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:46.070873 10477 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.087704 10892 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.099048 10850 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:46.048727 10598 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.109613 10657 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:46.160379  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48862.8us
I20260709 12:26:46.174722  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:46.176187  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:46.181818 10632 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:46.184522 10931 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 12:26:46.185693 10931 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:46.186223 10931 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:46.187176 10931 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.194466 10912 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.196026  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:46.196698  5701 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:46.204734  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:46.205605  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:46.205878  5707 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:46.206835  5703 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:26:46.207142  5553 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
W20260709 12:26:46.208470  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 46547.7us
W20260709 12:26:46.162402  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48363.6us
W20260709 12:26:46.165131  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 96787.1us
I20260709 12:26:46.210546  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:46.110405 10918 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:46.211064  5703 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 12:26:46.210916  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:46.169503 10882 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:46.219175  5703 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:26:46.164783 10934 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:26:46.205248 10883 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.604s	user 0.011s	sys 0.002s
I20260709 12:26:46.196990 10892 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:46.169241 10780 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
W20260709 12:26:46.238252  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:46.238116  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.202203 10941 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:46.235316  5703 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:26:46.123409 10725 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.102s
I20260709 12:26:46.239091  5629 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:46.121832 10458 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:46.252425 10932 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 12:26:46.253545 10780 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:46.254033 10780 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:46.255110 10780 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.235116 10462 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:46.257269  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.257680 10850 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.258502 10936 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:46.259213 10936 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.122400 10924 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:46.249941 10869 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.614s	user 0.015s	sys 0.000s
I20260709 12:26:46.230203 10882 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.251137 10930 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:46.262318  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.229565 10398 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 12:26:46.230763 10934 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:46.256383 10477 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:46.265357  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:46.196630 10657 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.260535 10939 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:26:46.274112  5701 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:46.277506  5602 leader_election.cc:400] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12.
I20260709 12:26:46.240365 10813 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:26:46.308126  5629 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:26:46.320387  5602 leader_election.cc:403] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:46.322011  5709 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:26:46.275777 10780 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:46.319490  5703 raft_consensus.cc:2410] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
W20260709 12:26:46.323966  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 85560.6us
I20260709 12:26:46.325062  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:46.332559 10865 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:46.333055 10865 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:46.333473 10865 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:46.333928 10924 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:46.334585 10918 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.336724 10477 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:46.337268 10477 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.338524 10912 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:46.339738 10477 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:46.339927 10794 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:46.340366  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:46.341061 10462 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12.
I20260709 12:26:46.272804 10941 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.342381 10876 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.343228 10941 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:46.344743  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:46.346411  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.341095  5701 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 12: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:46.347110  5630 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:46.350061  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:46.351891  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 119444us
I20260709 12:26:46.353309  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:46.301944 10930 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:46.302337 10812 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:26:46.356207  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:46.358258  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.362740  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:46.371619  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:46.373457  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.374270  5630 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:46.361229  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:46.385651  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.319483 10934 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:46.390326  5556 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:26:46.379065  6237 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:46.399196  5703 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:46.399699  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:46.402808  5554 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 18: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:46.401388  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:46.405943  5703 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:46.408123 10462 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:46.399416 10398 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.010s
I20260709 12:26:46.409083 10462 consensus_queue.cc:1243] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 14.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:26:46.409765 10462 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:46.410104 10462 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:46.363364 10847 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 12:26:46.410987 10462 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.412189 10462 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:46.416725  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:46.417634 10772 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:46.417725 10876 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:46.418243 10772 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.348438 10939 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:46.422600  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.357810 10883 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:46.359676 10477 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:26:46.409116 10943 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:26:46.459697  5702 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:46.424569  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:46.423983  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45025us
I20260709 12:26:46.360770 10812 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:46.439438  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:46.370513 10917 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:26:46.440346  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:46.452242  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.473338  5629 raft_consensus.cc:2410] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:26:46.474427  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.475288  5629 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:26:46.359930 10930 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:46.398550 10902 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:46.428745  5630 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:26:46.426275  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:46.462366  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:46.440301 10747 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:26:46.495918  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:46.487504  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:46.422508 10772 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:46.422144 10657 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:46.513218  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 50060.4us
I20260709 12:26:46.487308  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 969) took 2068 ms. Trace:
I20260709 12:26:46.515710  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.487501  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:46.520102  5629 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:26:46.360337 10941 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.101s
I20260709 12:26:46.569633  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:46.525327  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:46.471407 10621 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:46.555047  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 129022us
I20260709 12:26:46.489984  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.587774  5702 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:46.589079  5675 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.543s (SENT)
I20260709 12:26:46.589775  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:46.593847  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:46.595399  6278 rpcz_store.cc:276] 0709 12:26:44.418457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:44.514853 (+ 96396us) service_pool.cc:224] Handling call
0709 12:26:44.528859 (+ 14006us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:45.445926 (+917067us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:46.487018 (+1041092us) spinlock_profiling.cc:243] Waited 910 ms on lock 0x7b54002ce7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:46.487236 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2092092800}
I20260709 12:26:46.597163  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2489) took 1554 ms. Trace:
I20260709 12:26:46.597582  5710 rpcz_store.cc:276] 0709 12:26:45.042326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:45.042451 (+   125us) service_pool.cc:224] Handling call
0709 12:26:45.047777 (+  5326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:45.200296 (+152519us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:46.596898 (+1396602us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54001991d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:46.597108 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":349346176}
I20260709 12:26:46.604871  5628 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:46.516108 10847 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:46.620952  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.621971  5629 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:46.622900  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:46.612990  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:46.560868 10948 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:46.625068  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.629498  5632 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:46.632548  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:46.519071 10917 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:46.569975 10882 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:46.631973  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.631121  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.597155 10621 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.536278 10747 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:46.676635 10747 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.541479 10936 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:46.664148  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:46.685072  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.686067  5630 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:46.576049 10961 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:26:46.488370 10902 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:46.625944 10836 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.608s	user 0.015s	sys 0.000s
I20260709 12:26:46.613816 10725 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:46.541213 10398 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.092s
I20260709 12:26:46.614318 10794 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:26:46.673393 10960 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:26:46.619236 10847 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:46.784116 10954 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.185s	user 0.000s	sys 0.000s
I20260709 12:26:46.785238  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.786471  5636 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:26:46.787783  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.795755 10847 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:46.744580 10788 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:46.758704 10902 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:46.688941  5629 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:46.689757  5702 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:46.563812 10812 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.698138 10499 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:46.699754  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2502) took 1088 ms. Trace:
I20260709 12:26:46.702011  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.702296  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.827167  5635 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:46.827497  5631 raft_consensus.cc:2410] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:26:46.704571  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:46.828747  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 124257us
I20260709 12:26:46.829704  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:46.705148 10883 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:46.705770 10816 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:46.706336 10917 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.706970  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:46.845490  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:46.846192  5703 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:46.847530  5703 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:26:46.707943 10882 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:46.711861  5707 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:46.848758 10882 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.851049 10882 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:46.852425 10632 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:46.852751 10883 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:46.852914 10632 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:46.853163 10883 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:46.853916 10632 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.712926  5704 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:46.854626 10883 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.661517 10948 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.857497  5704 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:26:46.858436 10948 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:46.729785  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.859405  5704 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:46.859896  5632 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:46.636451 10939 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.740293  5710 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:26:46.860505 10870 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:46.860993 10870 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:46.861810 10870 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.862787 10870 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:46.763038  5628 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:46.718586  5630 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:46.865140  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.865264  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.865908  5630 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:26:46.866995 10949 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.540s	user 0.000s	sys 0.001s
I20260709 12:26:46.825845  5705 rpcz_store.cc:276] 0709 12:26:45.611039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:45.612547 (+  1508us) service_pool.cc:224] Handling call
0709 12:26:46.699513 (+1086966us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54002d19d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:46.699671 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1133897472}
I20260709 12:26:46.777509 10794 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:46.868661 10794 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:46.871251 10812 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:46.825680 10499 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:46.873588 10836 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:46.874070 10499 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.802664 10881 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:46.853256 10941 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:26:46.837868  5675 leader_election.cc:336] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.627s (SENT)
I20260709 12:26:46.878070  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:46.879567 10783 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:46.866999  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.881372  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.881469 10598 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:46.882488  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.867842  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.837177  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.842056  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.890146  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.854456  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.853158  5707 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
W20260709 12:26:46.891777  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53971.4us
I20260709 12:26:46.838747  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.893100  5631 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:46.782590 10934 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:46.892640  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 63040.6us
I20260709 12:26:46.896490  5703 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:26:46.901957  5630 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:26:46.903885  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:46.907361  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.908509  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:46.911113  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:46.909785  5630 raft_consensus.cc:2376] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13. Re-sending same reply.
I20260709 12:26:46.915082  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:46.894869 10499 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:46.909514 10780 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.572s	user 0.010s	sys 0.000s
I20260709 12:26:46.919958  5636 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:46.926525  5633 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:46.915699  5630 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:26:46.926626  6278 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
W20260709 12:26:46.928203  5675 leader_election.cc:336] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.645s (SENT)
I20260709 12:26:46.876017 10881 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:46.904768 10883 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:46.939129  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 983) took 1135 ms. Trace:
I20260709 12:26:46.941313  5634 rpcz_store.cc:276] 0709 12:26:45.804110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:45.804468 (+   358us) service_pool.cc:224] Handling call
0709 12:26:45.825301 (+ 20833us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:46.032029 (+206728us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:46.032061 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:46.032102 (+    41us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:46.032137 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:46.938808 (+906671us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54001f2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:46.938881 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:46.939054 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":474964352}
I20260709 12:26:46.956918 10880 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:46.957376 10880 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:46.958603 10880 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:46.959941 10880 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:46.951244  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:46.965037  6278 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:26:46.967736  5630 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:26:46.916316 10882 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:26:46.986169  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:46.990437  5633 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:26:46.933522 10892 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:46.999469  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:47.011569  5636 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:47.005859  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:47.012496  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:47.014196  5634 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
W20260709 12:26:47.014334  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:47.009531 10477 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 12:26:46.978909 10816 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:46.968037 10718 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 12:26:47.021061 10718 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:47.021457 10718 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:47.022399 10718 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:47.024207 10718 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:46.933836 10836 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20260709 12:26:47.031191 10836 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:46.977994 10914 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.032796 10914 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:47.033926 10953 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.705s	user 0.000s	sys 0.002s
I20260709 12:26:47.035974 10914 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:47.036800  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:47.008692 10921 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
W20260709 12:26:47.043079  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:47.077247  5558 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:26:47.053143 10621 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:46.982149 10882 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:47.031817 10836 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:47.079679  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:47.080809  5634 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:26:47.060155  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:47.083504  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.041s	sys 0.069s
W20260709 12:26:47.084101  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.041s	sys 0.070s
W20260709 12:26:47.084470  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 622925us
I20260709 12:26:47.061736  5552 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:26:47.062518  5631 raft_consensus.cc:1217] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Deduplicated request from leader. Original: 11.9->[13.10-15.11]   Dedup: 15.11->[]
I20260709 12:26:47.071903  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:47.074863  5628 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:26:47.041805 10947 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.715s	user 0.005s	sys 0.000s
W20260709 12:26:47.086985  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:47.079384 10754 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.570s	user 0.020s	sys 0.067s
W20260709 12:26:47.088845 10754 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.582s	user 0.020s	sys 0.067s
I20260709 12:26:47.095384 10836 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:47.076180 10941 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.097255 10941 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.099713 10941 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:47.100736  5708 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260709 12:26:47.101137  5708 pending_rounds.cc:85] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 8
I20260709 12:26:47.101367  5708 pending_rounds.cc:107] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 12:26:47.101606  5708 raft_consensus.cc:2889] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:47.103291  5557 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:26:47.082872 10899 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:47.113181 10899 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.113924  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:47.114813  5636 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 12: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:47.115418 10899 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:47.083184 10848 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.115713  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:47.074590 10865 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.732s	user 0.012s	sys 0.000s
I20260709 12:26:47.116137 10848 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.116647 10865 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:47.118290 10848 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:47.118659 10865 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:47.079505 10882 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:46.944080 10930 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:47.108157 10776 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.109314 10934 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:47.135325  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:47.110281  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:47.111311 10914 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.136050  5633 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:47.073257 10892 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:47.143632  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:47.119791  5631 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: [13.10-15.11]
I20260709 12:26:47.146958  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:47.105837  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:47.100926  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:47.148192  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:47.148795  5627 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:47.149075  5633 raft_consensus.cc:2376] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14. Re-sending same reply.
I20260709 12:26:47.103211 10780 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:26:47.100148  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:47.121764 10847 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:47.152601  6279 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:47.134176  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:47.153625  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:26:47.140846  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:47.104867  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:47.161232  5709 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:26:47.154038  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:47.103366 10881 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:47.171156  5706 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:47.176293  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:47.172259 10870 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:26:47.180301  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:47.181016  5636 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:47.119817 10794 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:47.183346  6274 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:47.142997 10848 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.175978 10598 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.184681 10848 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.185317 10598 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.187173 10848 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:47.151991 10914 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.189936 10836 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:47.190415 10836 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:47.190836 10836 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:47.193493 10794 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:47.197350  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:47.196288  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:47.204536 10871 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260709 12:26:47.203539 10934 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260709 12:26:47.208316  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:47.200871  6279 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17. Ops: []
I20260709 12:26:47.209800  5703 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:26:47.210052  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:47.211194  5553 raft_consensus.cc:2368] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17.
I20260709 12:26:47.211717 10632 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:47.169040 10462 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260709 12:26:47.212610 10462 consensus_queue.cc:1243] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 10.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:26:47.213171 10462 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:47.213483 10462 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:47.214340 10462 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.5, Last appended by leader: 5, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.214747 10847 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:47.215638 10462 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:47.215893  5628 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 11: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:47.170264 10893 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 12:26:47.219681 10893 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:26:47.220059 10893 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:47.201669 10780 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:47.221136 10893 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.222084 10780 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:47.222591 10893 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:47.223667  5630 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:47.172154 10786 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 12:26:47.202137  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:47.226183  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:47.226449  5628 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:47.227449  5635 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:47.170663 10759 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.231531  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:47.203811  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:47.248600  5630 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:47.219990  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:47.250135  5703 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:47.229730  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:47.251370  5627 raft_consensus.cc:2410] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:26:47.211825 10785 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:47.219941 10914 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:47.257050  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:47.230897  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:47.214655  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:47.258610  5634 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:47.232825  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:47.196818  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:47.252512  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:26:47.234853  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:47.259599  5709 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:47.235329  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:47.260659  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:47.238536  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:47.240553  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:47.261498  5707 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:47.241041  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:47.261860  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:47.262657  5553 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:26:47.233678  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:47.263891  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:47.259615  5636 raft_consensus.cc:2376] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13. Re-sending same reply.
I20260709 12:26:47.264326  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:47.265880  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:47.265923  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:47.267175  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:47.268006  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:47.268105  5706 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:26:47.269323  5553 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:26:47.248214 10870 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:47.258682 10759 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.271186  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:47.271378 10870 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.272938  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:47.277032  5630 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:47.281000  5558 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:47.142094 10965 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.111s
I20260709 12:26:47.286502  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:47.287343  5558 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:47.279172  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:47.206233 10892 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:47.288859  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:47.254176 10785 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:47.264695 10883 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:26:47.318243  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:47.331372  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:47.341554  5561 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:47.343288  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:47.290684  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:47.303858  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:47.311482  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:47.313927  5556 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:47.146895 10776 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:47.352968  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:47.355743  6272 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:47.357358  5553 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:26:47.351723  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:47.286368 10941 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:47.358656 10941 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:47.359066  5553 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:26:47.359206 10941 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:47.361418  5633 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:26:47.367908  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:47.369411  5553 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:26:47.291028 10912 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:47.381390  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:47.382269  5561 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:26:47.365111  5556 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:47.292387 10930 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:47.292742 10754 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:26:47.399969  5633 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:26:47.278136 10882 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:47.361362 10975 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:26:47.283054 10881 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.082s
I20260709 12:26:47.327612 10883 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:47.317129 10785 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:47.417783  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:47.421129  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:47.421645  5703 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:26:47.421659  5633 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 15. Current term is 16. Ops: []
I20260709 12:26:47.429080  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:47.430907  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:47.364464 10892 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:47.378918 10880 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.441702  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:47.450726  5555 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:47.446305  5633 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:26:47.411137 10912 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:47.452754  5633 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:47.454751  5633 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:47.456245  5633 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.445969 10883 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:47.469732  5553 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 12:26:47.453703 10930 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:26:47.455098 10912 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:47.479061  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:47.478647  5554 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:47.504488  5561 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:26:47.493024  6272 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:26:47.449734 10499 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
W20260709 12:26:47.517797 10976 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:26:47.518332 10598 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:47.453853 10880 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.479643 10952 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.499359 10931 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:47.517866 10499 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:26:47.520102 10930 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:47.536315 10941 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:47.539047 10499 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:47.618781  5552 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:47.540716 10931 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.539778 10952 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:47.620499 10962 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:26:47.611286 10868 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:26:47.650228  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 201994us
I20260709 12:26:47.634987 10848 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:47.663710  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:47.664149  5628 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:26:47.665690  6272 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 13: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:47.666101  5628 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
W20260709 12:26:47.666535  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:47.667364  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:47.668179  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:47.668852  5628 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:47.671418  5553 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:26:47.678007  5558 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260709 12:26:47.678110  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 257065us
I20260709 12:26:47.642302 10882 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:47.683519  5554 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:26:47.653508 10917 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:47.671316 10979 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.218s	user 0.001s	sys 0.000s
I20260709 12:26:47.663873 10868 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:47.728948  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48770.9us
I20260709 12:26:47.701030 10816 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:47.703406 10759 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:47.678913 10930 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:47.706411 10931 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:47.709555  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:47.747392  5554 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:47.723228 10598 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:47.723580  5630 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:47.724341 10924 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:26:47.725018 10929 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:47.730049  5556 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:26:47.671156 10980 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.213s	user 0.001s	sys 0.000s
W20260709 12:26:47.724850 10985 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:26:47.716988 10880 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:47.749135  5553 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:47.694522 10477 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:47.759253  5554 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 12:26:47.773818 10462 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.558s	user 0.005s	sys 0.009s
I20260709 12:26:47.774813  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:47.809404  5630 raft_consensus.cc:2410] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:26:47.810441  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:47.811132  5630 raft_consensus.cc:606] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:47.811518  5630 raft_consensus.cc:614] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:47.730063 10868 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:47.748957 10893 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.526s	user 0.013s	sys 0.000s
I20260709 12:26:47.739403 10816 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:47.797119  5628 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:47.881152  5630 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:26:47.688325 10499 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:47.732743 10917 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.907642  5628 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:26:47.925580  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:47.929409  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2225) took 1529 ms. Trace:
I20260709 12:26:47.930091  5629 rpcz_store.cc:276] 0709 12:26:46.400296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:46.895590 (+495294us) service_pool.cc:224] Handling call
0709 12:26:46.896170 (+   580us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:46.896578 (+   408us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:46.896615 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:46.897088 (+   473us) log.cc:844] Serialized 26 byte log entry
0709 12:26:47.135538 (+238450us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:47.135580 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:47.135654 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:47.929057 (+793403us) raft_consensus.cc:1685] finished
0709 12:26:47.929093 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:47.929322 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":89,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":9378,"thread_start_us":202258,"threads_started":1,"wal-append.queue_time_us":461767}
I20260709 12:26:47.773973 10883 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:47.788520 10812 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.122s
I20260709 12:26:47.753042 10780 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.158s
I20260709 12:26:47.936297  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2228) took 1487 ms. Trace:
W20260709 12:26:47.938505 10982 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.307s	user 0.001s	sys 0.001s
W20260709 12:26:47.938737 10718 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.914s	user 0.012s	sys 0.000s
I20260709 12:26:47.795168 10804 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.124s
I20260709 12:26:47.898404 10965 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.803453 10882 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.128s
I20260709 12:26:47.942158  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 890) took 1467 ms. Trace:
I20260709 12:26:47.934808 10998 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.901335 11002 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:47.943989  5631 rpcz_store.cc:276] 0709 12:26:46.448912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:46.901959 (+453047us) service_pool.cc:224] Handling call
0709 12:26:47.062333 (+160374us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:47.157098 (+ 94765us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:47.936017 (+778919us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54001cf2d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:47.936221 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":336090880}
W20260709 12:26:47.943125 10986 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:26:47.864315 10477 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:47.943063 10988 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
I20260709 12:26:47.981845  5702 rpcz_store.cc:276] 0709 12:26:46.474771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:46.834735 (+359964us) service_pool.cc:224] Handling call
0709 12:26:46.835088 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:46.835522 (+   434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:46.835565 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:46.836126 (+   561us) log.cc:844] Serialized 26 byte log entry
0709 12:26:47.201103 (+364977us) spinlock_profiling.cc:243] Waited 42.4 ms on lock 0x7b1800462fa0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:26:47.646281 (+445178us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:47.646796 (+   515us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:47.646871 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:47.941815 (+294944us) raft_consensus.cc:1685] finished
0709 12:26:47.941841 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:47.942091 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":117,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":131,"spinlock_wait_cycles":97519872,"thread_start_us":442414,"threads_started":1,"wal-append.queue_time_us":951350}
W20260709 12:26:48.018704  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 295916us
W20260709 12:26:48.007715  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 275084us
I20260709 12:26:48.008666  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 876) took 2837 ms. Trace:
I20260709 12:26:48.013195 10718 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.025125  5708 rpcz_store.cc:276] 0709 12:26:45.147174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:45.147446 (+   272us) service_pool.cc:224] Handling call
0709 12:26:45.160889 (+ 13443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:47.102027 (+1941138us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:47.984382 (+882355us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b54002f5b54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:47.984635 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4460347136}
I20260709 12:26:48.026343 10718 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.999331 10747 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.031453  5705 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:26:48.032307  5705 raft_consensus.cc:606] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
W20260709 12:26:48.031486  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 284241us
I20260709 12:26:48.032848  5705 raft_consensus.cc:614] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:48.036887  5602 leader_election.cc:400] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17.
I20260709 12:26:48.037256  5602 leader_election.cc:403] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:26:48.037425  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 111929us
W20260709 12:26:48.022374 10870 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.730s	user 0.018s	sys 0.010s
I20260709 12:26:48.048666  5707 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:26:48.049520  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:48.050324  5708 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
W20260709 12:26:48.050611  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 150649us
I20260709 12:26:48.053655  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:48.054739  5702 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 10.
I20260709 12:26:48.056083  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:48.056913  5702 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:48.057821  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:48.059363  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:48.016290 10998 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:47.989245 10965 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:48.047751 10870 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.752s	user 0.021s	sys 0.011s
I20260709 12:26:48.057757  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:26:48.040060 10984 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.352s	user 0.001s	sys 0.000s
I20260709 12:26:48.027817 10952 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:48.072336  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:48.080201  5708 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:48.060567 10812 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.082680  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:48.086977 10780 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:26:48.066941 11009 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:48.089257 11009 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:48.090098 10996 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:26:48.090262 11009 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.091629 10914 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.093231 10884 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.093722 10884 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.094422  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 958) took 1398 ms. Trace:
I20260709 12:26:48.094792  6274 rpcz_store.cc:276] 0709 12:26:46.696006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:46.696165 (+   159us) service_pool.cc:224] Handling call
0709 12:26:46.696525 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:47.183723 (+487198us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:48.094184 (+910461us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b54006121d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:48.094365 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1119137536}
I20260709 12:26:48.078238 10893 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:26:48.099212 10995 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.076s	user 0.000s	sys 0.002s
I20260709 12:26:48.068980 10865 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17.
I20260709 12:26:48.088539 10965 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.043139 10747 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.113857  5631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:48.021283 11002 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.075475 10804 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.048767 10940 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 12:26:48.131747  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:48.083884 10870 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.084631 10598 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:26:48.138224 10998 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.138252  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2531) took 1067 ms. Trace:
I20260709 12:26:48.140215 10477 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:48.081756 10948 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
I20260709 12:26:48.106818 10812 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.146952 11002 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:48.147562 10632 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.935s	user 0.015s	sys 0.000s
W20260709 12:26:48.135949 10902 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.310s	user 0.010s	sys 0.005s
I20260709 12:26:48.148157 10718 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.160534 10804 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:48.167168  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:48.170078  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:48.187438  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:48.092147 10914 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.171756  5627 rpcz_store.cc:276] 0709 12:26:47.070542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:47.257418 (+186876us) service_pool.cc:224] Handling call
0709 12:26:47.257800 (+   382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:47.265429 (+  7629us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:47.265457 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:47.294494 (+ 29037us) log.cc:844] Serialized 26 byte log entry
0709 12:26:47.629023 (+334529us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:47.629070 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:47.629149 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:48.138016 (+508867us) raft_consensus.cc:1685] finished
0709 12:26:48.138043 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:48.138189 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7178,"raft.queue_time_us":9788,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":9533,"thread_start_us":303166,"threads_started":1,"wal-append.queue_time_us":753418}
I20260709 12:26:48.125543 10893 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:26:48.176491  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 135101us
W20260709 12:26:48.172621 10836 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.981s	user 0.014s	sys 0.001s
I20260709 12:26:48.174711 10871 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:26:48.177650 10785 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.723s	user 0.015s	sys 0.000s
I20260709 12:26:48.191099  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:48.195123  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 143354us
I20260709 12:26:48.183533 10783 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.171658 10870 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.201216  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:48.189425 10993 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
W20260709 12:26:48.204916  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:48.206452  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:48.209373  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 164821us
I20260709 12:26:48.211902  5553 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
W20260709 12:26:48.213981 10912 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.692s	user 0.011s	sys 0.003s
I20260709 12:26:48.214402 10747 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:48.223153  5561 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 10. (term mismatch)
I20260709 12:26:48.223803  5561 pending_rounds.cc:85] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 9
I20260709 12:26:48.224123  5561 pending_rounds.cc:107] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260709 12:26:48.224471  5561 raft_consensus.cc:2889] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:48.195025 10462 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:48.193053 10865 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:48.195573 10871 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:48.208843 10759 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:48.215061 10747 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.015s
I20260709 12:26:48.210700 10783 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.208029 10785 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.185298 10902 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.198745 10836 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.244989  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2530) took 1318 ms. Trace:
I20260709 12:26:48.240167 10816 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.245574  5635 rpcz_store.cc:276] 0709 12:26:46.926349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:47.228242 (+301893us) service_pool.cc:224] Handling call
0709 12:26:47.233732 (+  5490us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:47.254176 (+ 20444us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:47.254207 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:47.254651 (+   444us) log.cc:844] Serialized 26 byte log entry
0709 12:26:47.626214 (+371563us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:47.626265 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:47.626327 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:48.244679 (+618352us) raft_consensus.cc:1685] finished
0709 12:26:48.244710 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:48.244907 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20029,"raft.queue_time_us":20757,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":16100,"spinlock_wait_cycles":9856,"thread_start_us":353308,"threads_started":1,"wal-append.queue_time_us":944387}
I20260709 12:26:48.240751  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:48.239435  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 194262us
I20260709 12:26:48.232939 10893 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:48.250090  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:48.248390 10816 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.214444 10912 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.255985  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2532) took 1185 ms. Trace:
I20260709 12:26:48.256466  5634 rpcz_store.cc:276] 0709 12:26:47.070886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:47.272035 (+201149us) service_pool.cc:224] Handling call
0709 12:26:47.272342 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:47.276864 (+  4522us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:47.276894 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:47.284972 (+  8078us) log.cc:844] Serialized 26 byte log entry
0709 12:26:47.641534 (+356562us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:47.641575 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:47.641645 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:48.255756 (+614111us) raft_consensus.cc:1685] finished
0709 12:26:48.255785 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:48.255926 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4116,"raft.queue_time_us":9985,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":5559,"spinlock_wait_cycles":526720,"thread_start_us":356016,"threads_started":1,"wal-append.queue_time_us":761976}
I20260709 12:26:48.259577 10870 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.260492 10865 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:48.261478 10836 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.263478 10912 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.264720 10462 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:26:48.267486  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:48.269627 10865 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:26:48.271858  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:48.252666 11005 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260709 12:26:48.272712  5632 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
W20260709 12:26:48.279357  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 70773.9us
I20260709 12:26:48.235121 10871 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:48.282423  5631 raft_consensus.cc:606] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:48.282934  5631 raft_consensus.cc:614] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:48.251202 10759 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:48.281355  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:48.281917  5559 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:26:48.284088  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:48.289016  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:48.291291  5561 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:26:48.290675  5556 raft_consensus.cc:2410] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:26:48.254205 10725 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 12:26:48.316934  5559 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:48.289705 10865 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260709 12:26:48.317484  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:48.317468 10868 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.329530  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:48.330531 10868 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.315627 11023 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:26:48.331347  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:48.327271 11016 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:26:48.332676 11006 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:26:48.327215 10785 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.326160 11025 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.008s
I20260709 12:26:48.335973 11006 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:48.332145 11023 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:48.336786 10931 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:26:48.336622 11006 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.337172 10931 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:48.337540 10914 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:48.338058 10931 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.338790 11006 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:48.339159 10902 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.339536  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:48.327492 10865 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.340556 10865 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.341372  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:48.342533 10865 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:48.332597 10632 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:48.343029  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:48.343216 10948 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:48.343562 10632 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:48.345464 10725 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:48.345865 10725 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:48.346638 10725 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.283803 10893 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:26:48.340742 11022 log.cc:927] Time spent T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260709 12:26:48.349014 10892 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:48.356304 10892 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:48.356665  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:48.349156 10893 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:48.357334  5707 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:26:48.357784 10893 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.359649  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2427) took 1300 ms. Trace:
I20260709 12:26:48.349335 10812 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.361280  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2515) took 1126 ms. Trace:
I20260709 12:26:48.361670  5706 rpcz_store.cc:276] 0709 12:26:47.234860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:47.235109 (+   249us) service_pool.cc:224] Handling call
0709 12:26:48.361111 (+1126002us) spinlock_profiling.cc:243] Waited 7.25 ms on lock 0x7b54002d23d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:48.361217 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16674048}
I20260709 12:26:48.348266 10725 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:26:48.370378 10983 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.720s	user 0.000s	sys 0.005s
W20260709 12:26:48.371186  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:48.371816 10812 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.372303 10812 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.350764  5636 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:26:48.352603  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:48.349958 10759 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.374541 10812 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.352787 10940 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:48.376037  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 887) took 1902 ms. Trace:
I20260709 12:26:48.376449  5701 rpcz_store.cc:276] 0709 12:26:46.473767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:46.733309 (+259542us) service_pool.cc:224] Handling call
0709 12:26:46.733662 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:46.857490 (+123828us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:46.857520 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:46.858029 (+   509us) log.cc:844] Serialized 26 byte log entry
0709 12:26:47.150632 (+292603us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:26:47.254585 (+103953us) log.cc:844] Serialized 14 byte log entry
0709 12:26:47.255133 (+   548us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:48.271103 (+1015970us) spinlock_profiling.cc:243] Waited 5.12 ms on lock 0x7b180035d340. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:48.375214 (+104111us) spinlock_profiling.cc:243] Waited 4.54 ms on lock 0x7b180035d340. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419
0709 12:26:48.375732 (+   518us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54001a8954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:48.375781 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:48.375832 (+    51us) raft_consensus.cc:1685] finished
0709 12:26:48.375845 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:48.375979 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3347,"prepare.queue_time_us":103654,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":1792,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":12136,"spinlock_wait_cycles":306165760,"thread_start_us":292130,"threads_started":1,"wal-append.queue_time_us":358709}
I20260709 12:26:48.380853  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2290) took 1125 ms. Trace:
I20260709 12:26:48.381266  5560 rpcz_store.cc:276] 0709 12:26:47.255528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:47.263669 (+  8141us) service_pool.cc:224] Handling call
0709 12:26:48.380667 (+1116998us) spinlock_profiling.cc:243] Waited 814 ms on lock 0x7b54003a98d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:48.380793 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1871664512}
I20260709 12:26:48.321959 11020 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:48.383136 10462 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:48.353480  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:48.386660  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:48.387434  5635 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:26:48.387248 10632 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:48.388190  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2250) took 1689 ms. Trace:
I20260709 12:26:48.388588  5632 rpcz_store.cc:276] 0709 12:26:46.699202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:46.917540 (+218338us) service_pool.cc:224] Handling call
0709 12:26:46.917801 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:48.282262 (+1364461us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:48.387957 (+105695us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b540032cdd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:48.388132 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3115585792}
I20260709 12:26:48.353570  5559 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:26:48.353674 10499 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:48.360041  5555 rpcz_store.cc:276] 0709 12:26:47.059263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:47.062219 (+  2956us) service_pool.cc:224] Handling call
0709 12:26:47.080244 (+ 18025us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:47.451160 (+370916us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:48.359423 (+908263us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b5400227ed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:48.359591 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":851639168}
I20260709 12:26:48.387352  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:48.392032 10848 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:48.397586 10848 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:48.399768  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:48.347630 10883 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:48.339327 10931 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:48.352932  5708 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:26:48.403072 10776 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:48.394767  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:48.394934  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:48.395218 11023 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:26:48.355655  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:48.348776 10804 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.410338  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:48.352186  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:48.411623  5709 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:48.352324  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:48.397076  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:48.412659  5710 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:48.393009  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:48.404997  5636 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:48.416309  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:48.420922 10940 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:48.421222 11017 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:48.421708 11017 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:48.422083 11017 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:48.423641 10939 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:48.434795  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:48.420843 11024 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:26:48.438153  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:48.438812  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:48.439553 11024 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:48.441942 11024 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:48.443560 11020 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:26:48.445891 11018 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:48.446473 10499 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:26:48.448575 11019 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:48.438062  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:48.451196 11020 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.452606 11025 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:48.452983 11025 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:48.453320 11025 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:48.417634 11011 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:26:48.439718 10776 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:26:48.451740 11020 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.418838 10812 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:48.425643 10892 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.463706  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:48.493831 11024 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.507290 10883 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.508116 10883 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.509109  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:48.510007  5636 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:48.463287 10776 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:26:48.512643  5555 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:26:48.512774  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:48.515506  5559 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 13: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:48.475368 10812 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:48.526754  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:48.528839  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:48.529657  6272 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:48.528218  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:48.488566 10747 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.537226  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:48.537776  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:48.537956  5630 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:48.534284  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:48.534127  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:48.538771  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:48.513109 10902 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.561918  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:48.538312  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:48.564193 10893 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:48.577847  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:48.568846  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:48.583249  5630 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:48.518699 10632 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:48.571282  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:48.518488 10934 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.513553 10948 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 12:26:48.592984  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:48.602407  6272 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:48.592109 10836 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:26:48.604596  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 93164.7us
I20260709 12:26:48.605011 10836 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:48.515281 10940 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
I20260709 12:26:48.514976 10924 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.067s
I20260709 12:26:48.516656 11006 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 12:26:48.576979 10912 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:48.576006 10813 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.625276  5704 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:48.626710  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:48.572626 11019 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:48.575254 11011 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:48.629338  6272 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:26:48.546962 10865 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.074s
I20260709 12:26:48.631492  5709 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:48.574826 10747 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.586267 10902 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.594894 11023 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:26:48.656095  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:48.690923  5636 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:26:48.755724  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:48.609290 10987 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:26:48.730239  6279 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:26:48.737579  5633 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:48.702497  5630 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:48.630510 10912 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:48.741106  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:48.767424  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:48.610889 10836 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:48.606105 11010 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.818322  5629 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:26:48.818133 11010 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.818830 10902 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.819504 10902 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.720685 11020 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:48.609606 10934 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.824673 10934 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.824846 11010 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:48.810496  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 54887.2us
I20260709 12:26:48.825764 11043 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260709 12:26:48.826376  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:48.828877  5703 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:48.830000  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:48.830626 10870 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:26:48.831167 10870 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:48.832011 10870 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.832504  5704 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:26:48.834331 10931 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:48.656025 11019 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:48.835397  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 94398.5us
I20260709 12:26:48.831188  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:48.836550  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:48.836875  5632 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:48.809244  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:48.627326 11024 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.611023 11017 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.809764 10598 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:48.855911  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:48.856837  6274 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 12:26:48.815431  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 281439us
W20260709 12:26:48.683871 11029 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:26:48.808621  5706 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:26:48.817166  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:48.839812  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:48.837227  5704 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:26:48.861603  5636 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
I20260709 12:26:48.843521  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:48.845381  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:48.846539  5710 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:48.655350 11011 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:48.724946 10804 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:26:48.858662  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 91384.1us
I20260709 12:26:48.819988 10902 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.688532 11007 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:48.872977  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:48.812268 10912 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:48.876863  5630 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:48.859690  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:48.734540 10892 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.835001 10931 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.882781  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:48.889547  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:48.958956  5636 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:48.890092  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:48.960160  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 70196.5us
I20260709 12:26:48.841666 11019 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:48.962903  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:48.891754  5701 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:48.897930  5709 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:26:48.853895 10865 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:26:48.857438 10868 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:48.956180  5633 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:26:48.735832 11025 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.837898 10871 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.968832  5636 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:48.969290  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:48.875406 11018 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.665273 10813 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.682037 10812 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.977006  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2273) took 1745 ms. Trace:
I20260709 12:26:48.874452 10941 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.104s
I20260709 12:26:48.987589 10952 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:49.028731 10952 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:49.029587 10952 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:48.873769 10598 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.030958 10902 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.019078 11045 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:26:49.033881 11045 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:49.017350 11018 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.034674 11045 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:49.036670  5634 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:26:49.037772 10871 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:48.999043 10924 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260709 12:26:49.039431 10924 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:49.039814 10924 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:49.040681 11052 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:49.042091 10813 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.044899  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:48.880002 10499 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:26:49.045892 10499 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:49.046855 10499 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:48.818830 10848 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.051162 10848 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.051857  5629 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:26:48.982966  5556 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:26:48.987133 10718 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:48.983016  5558 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:49.054888 10718 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.057304 10718 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.057521  5632 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:48.992755  5552 raft_consensus.cc:2368] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12.
I20260709 12:26:48.987353 10816 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:49.061702  5552 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:26:49.061575 10816 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.063599 10816 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:48.995591 11006 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.975131 11009 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:48.970039 10865 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:49.091398  5676 leader_election.cc:400] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12.
I20260709 12:26:49.091867  5676 leader_election.cc:403] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:49.003075 10868 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:48.901995 10893 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:26:49.015714  5633 rpcz_store.cc:276] 0709 12:26:47.231546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:47.452158 (+220612us) service_pool.cc:224] Handling call
0709 12:26:47.452613 (+   455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:48.382888 (+930275us) spinlock_profiling.cc:243] Waited 922 ms on lock 0x7b18001bb880. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6ad646 00007f4ffa5e758e 00007f4ffa668f9e 00007f4ffa66a7de 00007f4ffa674b49 00007f4ffa675416 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4
0709 12:26:48.737257 (+354369us) spinlock_profiling.cc:243] Waited 7.2 ms on lock 0x7b58005ca2d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5ce0aa 00007f4ffa650447 00007f4ffa668fe7 00007f4ffa66a7de 00007f4ffa674b49 00007f4ffa675416 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4
0709 12:26:48.976727 (+239470us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:48.976951 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2137140736}
W20260709 12:26:49.004256  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:48.881659 11011 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:49.006381  5710 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:26:49.019898 11021 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:48.880505 10934 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.109s
I20260709 12:26:48.955864 11007 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.025291 10747 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:49.026149 10939 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.602s	user 0.008s	sys 0.004s
I20260709 12:26:48.889860 10804 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.027629 11017 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:48.994037  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:49.065632  5629 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:26:49.121680  5561 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:49.067517  5557 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:26:48.974431 11031 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 12:26:49.001202  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.033905 11025 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:49.110571  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 248554us
I20260709 12:26:49.082144 11018 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:49.093729 10865 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:49.066466  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:49.099249 10929 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:26:49.084635 10462 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 12:26:49.112483 11021 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:49.090734 10398 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 12:26:49.090886 11006 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.096890 10940 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:26:49.093089 11009 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.159415  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:49.155588  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:49.125891 10598 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:49.157058  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.005s	sys 0.017s
I20260709 12:26:49.124570 10848 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:49.106142 10948 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 12:26:49.131242 10939 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:26:49.162421  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.005s	sys 0.017s
W20260709 12:26:49.168143  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 545915us
I20260709 12:26:49.171663  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.154541 11021 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:49.114718 10924 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.174849 10924 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:49.164060 10948 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:49.176333 10948 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:49.177260 10948 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.178829 10948 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:49.161634 11051 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 12:26:49.185645  6272 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:49.190749  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.163669 11017 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:49.196784  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:49.175432 11046 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.093s
I20260709 12:26:49.201011  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.205281  6272 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 9.
I20260709 12:26:49.202065  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:49.205933  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 140255us
I20260709 12:26:49.206610  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:49.170979 10848 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.213445 10836 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.214335 10848 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.215626 10836 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:49.202898  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:49.203235  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 344909us
I20260709 12:26:49.159313 10881 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 12:26:49.221484 11007 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.221980 10848 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:49.223230 10848 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:49.223803 10848 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:49.225185 10848 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.226327 10848 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:49.173342 10868 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.203680 11017 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:49.231874 11017 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:49.234689  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:49.237584  5636 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 11. (term mismatch)
I20260709 12:26:49.238021  5636 pending_rounds.cc:85] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 10
I20260709 12:26:49.238358  5636 pending_rounds.cc:107] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 15.11
I20260709 12:26:49.238618  5636 raft_consensus.cc:2889] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:49.241863  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.242638  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:49.243703  5628 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260709 12:26:49.244083  5628 pending_rounds.cc:85] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 8
I20260709 12:26:49.244303  5628 pending_rounds.cc:107] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 12:26:49.244609  5628 raft_consensus.cc:2889] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:49.245808  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.238063 10398 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:49.240379 11010 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:49.260255  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.249873 10930 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:49.262236 10930 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.264526 10930 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.266400  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:49.266283 10939 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:49.267095 10939 consensus_queue.cc:1243] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 12.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:26:49.267732  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.270057 10912 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.273782  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:49.279479 11053 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260709 12:26:49.280951 10783 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:49.257864 11019 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.283015 10804 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.283493 11019 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.284709 11009 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.256517 10398 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:49.285581 10398 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:49.243194 10914 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:49.290017 10914 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.291069  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.292263  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:49.296213  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:49.305361  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 349421us
I20260709 12:26:49.287892 10865 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.302141  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:49.293618 10912 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.333287  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:49.344405  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:49.347045  5703 raft_consensus.cc:2410] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:26:49.318552 10930 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:49.295586 11010 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.301820 10598 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:26:49.349216  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:49.329932 10871 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:49.328637 11007 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:49.330595 10939 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:26:49.341331 10914 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.330226 10836 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:49.375650  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.383817  5635 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:49.352885 10930 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:49.393102  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:49.351703 11003 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:49.397467  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.396834  5635 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:26:49.403488  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:49.404496  5635 pending_rounds.cc:85] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 8
I20260709 12:26:49.404915  5635 pending_rounds.cc:107] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 12:26:49.405216  5635 raft_consensus.cc:2889] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:49.386811 11061 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:26:49.372649 10939 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:49.406692  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:49.410622  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:49.411562  5708 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:49.348898 11017 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.386785 11011 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.383881 10816 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:49.425333  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.428709  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 1005) took 1073 ms. Trace:
I20260709 12:26:49.429059  5632 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:26:49.429098  5634 rpcz_store.cc:276] 0709 12:26:48.355306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:48.355439 (+   133us) service_pool.cc:224] Handling call
0709 12:26:49.428528 (+1073089us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b5400439c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:49.428649 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1443537792}
I20260709 12:26:49.426851  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.411049 10598 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:26:49.430632 11058 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:49.450150 11067 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:26:49.450418 11058 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:49.451768 11056 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 12:26:49.451854  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:49.453151  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:49.453429  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:39496 (request call id 30) took 1348 ms. Trace:
I20260709 12:26:49.433547 11042 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:26:49.454332  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:49.469012 10924 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:49.469594 10924 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:49.411036 10939 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:49.473975  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.475384  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.475914  5707 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:49.477572  5559 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:49.480906  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 966) took 1276 ms. Trace:
W20260709 12:26:49.482301 11032 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.478s	user 0.002s	sys 0.000s
I20260709 12:26:49.481328  5554 rpcz_store.cc:276] 0709 12:26:48.204768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:48.205511 (+   743us) service_pool.cc:224] Handling call
0709 12:26:48.205853 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:48.231286 (+ 25433us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:48.231318 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:48.235838 (+  4520us) log.cc:844] Serialized 26 byte log entry
0709 12:26:48.500701 (+264863us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:48.500766 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:49.480483 (+979717us) spinlock_profiling.cc:243] Waited 4.97 ms on lock 0x7b54001c5cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:49.480533 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:49.480584 (+    51us) raft_consensus.cc:1685] finished
0709 12:26:49.480597 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:49.480839 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15502,"raft.queue_time_us":21818,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":6229,"spinlock_wait_cycles":11434240,"thread_start_us":248790,"threads_started":1,"wal-append.queue_time_us":448764}
I20260709 12:26:49.485603 11010 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.388556 11063 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:49.414875 10783 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.080s
I20260709 12:26:49.395217 10836 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.487218  6279 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:49.470629 10924 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.488962 10836 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.398440 11003 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:49.489645 11003 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:49.489673  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 970) took 1215 ms. Trace:
I20260709 12:26:49.490245  5553 rpcz_store.cc:276] 0709 12:26:48.274747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:48.275499 (+   752us) service_pool.cc:224] Handling call
0709 12:26:48.275816 (+   317us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:48.276168 (+   352us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:48.276196 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:48.282205 (+  6009us) log.cc:844] Serialized 52 byte log entry
0709 12:26:48.490708 (+208503us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:48.801134 (+310426us) log.cc:844] Serialized 14 byte log entry
0709 12:26:48.830700 (+ 29566us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:48.830769 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:49.481647 (+650878us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:49.489365 (+  7718us) raft_consensus.cc:1685] finished
0709 12:26:49.489474 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:49.489615 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27420,"prepare.queue_time_us":326068,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":29750,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":35528,"spinlock_wait_cycles":260607360,"thread_start_us":433837,"threads_started":2,"wal-append.queue_time_us":567266}
I20260709 12:26:49.433185  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:49.493965  5710 raft_consensus.cc:2368] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14.
I20260709 12:26:49.437075 10930 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.439145  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.495615  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:49.385784 10871 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.498286  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.385551 11007 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.499415  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.438548 11011 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.493878  5555 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:49.501771 11007 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.442130  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.503288  5706 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:49.447587  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.447006 10759 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:49.447826 10902 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:49.486405 11063 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:49.504978 10902 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:49.505005 10759 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.505962 10902 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.507288 10902 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:49.499439  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:49.508119  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.391989 10914 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.117s
I20260709 12:26:49.508996  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:49.509796 10934 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:49.510012 10892 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:26:49.510485 10934 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:49.486469 10917 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.510980 10934 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:49.511648  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:49.512063  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:49.493278 11017 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.506879 11063 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.512269 10934 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.516065  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.435854  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:49.516927  6272 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 13: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:26:49.517439  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 81649.3us
I20260709 12:26:49.540858  5675 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:49.449448 11030 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.825s	user 0.001s	sys 0.000s
I20260709 12:26:49.541627  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.512645 11059 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:49.542640 11059 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:49.542796  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.543043 11059 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:49.543551 10884 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.545562 11006 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.547549 11065 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:49.518730 11017 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:49.548106 11065 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.519168  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.550206  5709 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:26:49.550931 11065 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.551090  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.552119  5709 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:49.553166  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:49.553750  5709 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:49.444307  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:49.589437  5556 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:49.525200 10883 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.527657 10598 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:49.590762 10598 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.528245 11042 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:49.455509 10931 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:49.513159  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:49.488636 10924 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:49.528191  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:49.534847  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:49.535596  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:49.538846 11051 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:49.603178 10598 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.539151  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:49.604673  5710 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:49.540004 10759 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.454214  5631 rpcz_store.cc:276] 0709 12:26:48.104652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:48.104813 (+   161us) service_pool.cc:224] Handling call
0709 12:26:49.453239 (+1348426us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b540032cdd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa66c853 00007f4fff63a60a 00007f4ff8bc07d4 00007f4ff8bc0762 00007f4ff8bc0691 00007f4ff8bc060c 00007f4ff8bbf8c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:49.453348 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":361533312}
I20260709 12:26:49.554776 10929 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 11. Current term is 12.
I20260709 12:26:49.555912 10398 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.524796 11058 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.459839 11045 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.565397  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:49.565636 11064 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:26:49.566706  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:49.569149 10725 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:49.627312 11058 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:49.569541 11017 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:49.572427  5601 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:49.629088 11058 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.629413 11017 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:49.629902  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 57594.7us
I20260709 12:26:49.575446  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.631551 11017 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:49.631620  6272 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:49.580300  5552 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:26:49.544029  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:49.633229  5706 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:26:49.592176 11042 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.633602 11058 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.634047  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.629118 10725 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:49.634720 10725 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:49.634943 11042 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:49.635329  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.600304 11046 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:26:49.636548  6274 raft_consensus.cc:2368] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16.
W20260709 12:26:49.628103  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61468.4us
I20260709 12:26:49.584102 11059 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.585304 11003 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.660414  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:49.587007 10939 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.669842  6276 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: []
I20260709 12:26:49.591619  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:49.595616  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:49.670881  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 79364.5us
I20260709 12:26:49.610569  5708 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:49.415854 10912 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.185s
I20260709 12:26:49.623754  5709 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:26:49.632200 10759 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:26:49.646648  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 247127us
I20260709 12:26:49.635932  5706 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:49.638244  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:49.647413 10929 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:49.685118  5629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:49.588452 11021 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.598188  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.672611  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:49.685811  5629 raft_consensus.cc:606] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:49.686158  5708 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:49.687595  6273 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:49.680734  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.688779  5709 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:49.678187  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:49.684072  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.686249  5629 raft_consensus.cc:614] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:49.683012  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.665057  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:49.690764  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.691411  6272 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:49.691457  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.693068  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.694569  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.669389 11042 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:49.697285  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:49.697307  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.697832  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.698046  5709 raft_consensus.cc:2376] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13. Re-sending same reply.
I20260709 12:26:49.699011  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.699055  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.700313  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:49.700276  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.710388  5709 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:49.689597  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:49.727686  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:49.728844  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63913.8us
I20260709 12:26:49.729646  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:49.676196 11058 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:49.731771  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.733206  5634 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:49.734975  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:49.687650 11051 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:49.713287 10929 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:49.741665  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.743825  5558 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:26:49.744601  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.744834  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:49.747644  5634 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:49.635820 10725 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:49.751451  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:26:49.748821  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:49.753590  6275 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:49.753618  6272 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:49.735529 11070 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:49.732316 10914 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:49.727805 10884 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 12:26:49.734804 10718 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:49.735805 11058 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:49.745633 11051 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:49.741638 11007 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:49.770131  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:49.741083  5524 leader_election.cc:400] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14.
I20260709 12:26:49.769378  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.770980  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.771428  6277 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:26:49.752918 10929 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:49.760341  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.772444  5524 leader_election.cc:403] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:49.775260  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:49.760972  5708 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:49.781221  5635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:49.781945  5635 raft_consensus.cc:606] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:49.782387  5635 raft_consensus.cc:614] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:49.783807  5635 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: []
W20260709 12:26:49.790172  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49269.1us
I20260709 12:26:49.791879  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:49.793268  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:49.757190 11070 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:49.831172  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:26:49.820128  5634 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:49.793664  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:49.806113  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.769464 11058 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:49.836577  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:49.837174  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.839896  5708 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:49.841042  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:49.841703  5708 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:49.790832  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:49.844537  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53820.3us
I20260709 12:26:49.848755  6272 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:26:49.818077 10718 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:26:49.776935 11051 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:49.789090 11052 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:49.838054  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44890.6us
I20260709 12:26:49.859098  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:49.767522 10914 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:49.861369 11072 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 12:26:49.853856 11064 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:49.847244 10725 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:49.897861  5630 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:49.856313  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:49.837661 11070 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:49.847851 11007 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:49.776043 10868 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.078s
I20260709 12:26:49.863680  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.944015  5710 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:26:49.945446  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.946329  5710 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:49.802798 10912 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:49.845953 10917 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:26:49.861549 11063 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:49.919315 10883 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:26:49.866585 10812 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.017s	sys 0.004s
W20260709 12:26:49.919494  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60470.8us
I20260709 12:26:49.956677  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:49.866827 11052 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:49.863432 10398 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:49.932341 11045 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.894292 10884 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:26:49.933674  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 77453.7us
I20260709 12:26:49.949074  5708 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:49.981706  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:49.921672 10902 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:49.925623 11059 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.927371 11064 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:49.886771 10786 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.867s	user 0.009s	sys 0.019s
W20260709 12:26:49.951100 10870 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.020s	sys 0.020s
I20260709 12:26:49.957921  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:49.931007  5627 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.015s	sys 0.034s
W20260709 12:26:49.999732  5627 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.692s	user 0.015s	sys 0.034s
I20260709 12:26:49.970463  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:50.000813  5709 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:26:50.002039  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:50.002743  5709 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:49.971846 11055 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.943s	user 0.002s	sys 0.002s
I20260709 12:26:49.971310 11042 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:49.952250 10917 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:49.963837 10812 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.904s	user 0.019s	sys 0.005s
I20260709 12:26:49.943012 11065 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:49.950575 10930 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:49.950987 11003 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.964076 10883 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:49.955222 11063 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:49.988291  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:49.972596 11007 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:26:50.010219  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52360.9us
I20260709 12:26:49.997822  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:50.012434  5629 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:26:49.978628 10940 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:50.019660 10917 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:49.986654 11021 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:49.983927 10836 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:49.998826 10786 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.977s	user 0.010s	sys 0.024s
W20260709 12:26:50.016644 11077 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.178s	user 0.002s	sys 0.000s
I20260709 12:26:49.992727 11064 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:50.071532  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:50.075026  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:50.071626  5635 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:50.011129 11006 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:50.094089  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:50.096154  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 108084us
I20260709 12:26:49.986845 10884 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:50.034291 10883 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:50.075089 11072 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:26:50.127297  5635 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:49.988866 10902 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.028908 11015 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:50.034605 11063 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:26:50.081480 11074 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.279s	user 0.000s	sys 0.001s
I20260709 12:26:50.081648 10930 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:50.112303 11006 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.112859  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:50.113340 11094 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:26:50.116780 10914 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:50.154002  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:50.112062  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:50.120260 10870 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:50.155450 10882 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:50.155805 10870 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.156623 10884 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:50.158023 10870 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:50.083415 10868 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:26:50.134142  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62744.6us
I20260709 12:26:50.159072  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:50.087606 11003 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:26:50.023989 11042 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.161844 11042 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:50.066684 11065 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.146917  5701 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
W20260709 12:26:50.156270 11001 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 2.118s	user 0.007s	sys 0.007s
I20260709 12:26:50.097048  5709 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:50.098986 10836 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:50.166188 11085 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 12:26:50.135421 11059 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:50.137049 11015 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:50.174075  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:50.171907 11058 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:50.175689  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:50.162581  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2442) took 1182 ms. Trace:
I20260709 12:26:50.184110  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:50.203054  5560 rpcz_store.cc:276] 0709 12:26:48.980384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:48.980894 (+   510us) service_pool.cc:224] Handling call
0709 12:26:48.981317 (+   423us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:49.295578 (+314261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:49.295625 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:49.296377 (+   752us) log.cc:844] Serialized 52 byte log entry
0709 12:26:49.639407 (+343030us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:49.639441 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:49.639504 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:50.162350 (+522846us) raft_consensus.cc:1685] finished
0709 12:26:50.162375 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:50.162523 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10350,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":16840,"thread_start_us":609729,"threads_started":2,"wal-append.queue_time_us":437007}
I20260709 12:26:50.204550  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:50.184482  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:50.187031  5704 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:26:50.174397  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:50.208644  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:50.187119  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:50.208953  5701 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:50.187835  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:50.188489 11092 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 12:26:50.188679 11078 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.320s	user 0.000s	sys 0.002s
I20260709 12:26:50.220932  5709 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:26:50.222513 11107 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:50.223008 11107 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:50.223430 11107 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:50.188834  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:50.189560 11087 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:26:50.226850  5555 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:50.191383 10884 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:50.192423 10893 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:50.227901 10893 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:50.229823 11038 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.573s	user 0.002s	sys 0.000s
I20260709 12:26:50.179435 10929 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:50.196656 10902 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:50.197873 11098 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:26:50.199708 11065 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:50.172940 10930 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:50.213085 11100 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:50.213617 11003 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:50.197916 11006 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:50.231076  5556 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:50.231980  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:50.256261  5704 raft_consensus.cc:2410] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 14 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:26:50.181830 10868 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:50.206233 10836 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:50.251049  5631 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: [15.9-15.9]
W20260709 12:26:50.258495  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.847s (SENT) [suppressed 25 similar messages]
W20260709 12:26:50.245232 10924 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.628s	user 0.011s	sys 0.001s
I20260709 12:26:50.257452  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:50.260058  5706 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 13: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:50.266881  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 107883us
I20260709 12:26:50.268023  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:50.268795  5708 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:50.269654  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:50.307327  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:50.309971  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:50.312387  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:50.313019  5555 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:50.313591  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:50.317335  5633 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:26:50.253244 11100 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:50.256858 10893 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:50.245296 10884 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.070s
I20260709 12:26:50.245960 10917 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.315630 10902 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:50.318318 11059 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:50.330412 11100 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:50.315233 11109 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.087s
I20260709 12:26:50.337105 11007 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:26:50.366261 11110 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:50.366559 11066 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:26:50.366914 11079 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.115s
I20260709 12:26:50.352255 11059 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:50.355656 10902 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:50.434512 11064 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.435833  5701 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:50.384830  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:50.420461 11106 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:50.421409 11083 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
W20260709 12:26:50.422013 11104 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
W20260709 12:26:50.422577 11095 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.216s	user 0.000s	sys 0.001s
I20260709 12:26:50.457376 11110 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:50.460880 11110 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:50.422447 11070 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.425870 10917 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:50.352139 10868 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.483390 10929 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:50.427431 11058 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:50.464751  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 913) took 2005 ms. Trace:
I20260709 12:26:50.368431 10883 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:50.478410 11101 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20260709 12:26:50.483206 11106 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.266820 11015 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:50.501618  5702 rpcz_store.cc:276] 0709 12:26:48.459031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:48.459200 (+   169us) service_pool.cc:224] Handling call
0709 12:26:48.459573 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:48.460150 (+   577us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:48.460189 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:48.460726 (+   537us) log.cc:844] Serialized 26 byte log entry
0709 12:26:49.458025 (+997299us) spinlock_profiling.cc:243] Waited 999 us on lock 0x7b1800450880. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:26:49.934807 (+476782us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:49.974788 (+ 39981us) log.cc:844] Serialized 14 byte log entry
0709 12:26:49.975136 (+   348us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:49.975250 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:50.464429 (+489179us) raft_consensus.cc:1685] finished
0709 12:26:50.464531 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:50.464687 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":170,"prepare.queue_time_us":38957,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":412,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":8628,"spinlock_wait_cycles":4138624,"thread_start_us":476587,"threads_started":1,"wal-append.queue_time_us":1788595}
I20260709 12:26:50.377027 11066 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:50.509793 11064 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:50.501441 10718 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.641s	user 0.006s	sys 0.005s
I20260709 12:26:50.533895  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:50.480973 11007 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:50.539680  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:50.543398  5702 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:50.543130 11015 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:50.519057 11058 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:50.536773 11066 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:50.549558  5702 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:26:50.542220 10718 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14.
I20260709 12:26:50.573230  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2404) took 1309 ms. Trace:
I20260709 12:26:50.573704  5709 rpcz_store.cc:276] 0709 12:26:49.263650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:50.222465 (+958815us) service_pool.cc:224] Handling call
0709 12:26:50.223062 (+   597us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:50.234465 (+ 11403us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:26:50.337384 (+102919us) log.cc:844] Serialized 14 byte log entry
0709 12:26:50.572865 (+235481us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:50.572922 (+    57us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:50.572974 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:50.573042 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:50.573170 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75742,"prepare.queue_time_us":114577,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"thread_start_us":234438,"threads_started":1}
I20260709 12:26:50.513844 10902 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:50.567168 10893 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:50.597514  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:50.569851 11111 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:50.569262 11105 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:50.599282  5554 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
W20260709 12:26:50.604525  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 346005us
I20260709 12:26:50.597841  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:50.605582  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:50.600217  5709 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:26:50.546195 11007 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.610605  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:50.610638  5702 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:50.611232  5560 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:50.541608 11059 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.5, Last appended by leader: 5, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:50.576272 11112 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:50.623947  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:50.608258  5709 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:26:50.568813 11058 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.634414  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:50.635742  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:50.641371  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:50.576659 11063 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.646636  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 928) took 1072 ms. Trace:
I20260709 12:26:50.609956 10930 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:50.656993  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:50.651883  5704 rpcz_store.cc:276] 0709 12:26:49.573995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:50.305790 (+731795us) service_pool.cc:224] Handling call
0709 12:26:50.306146 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:50.306401 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:26:50.587585 (+281184us) log.cc:844] Serialized 14 byte log entry
0709 12:26:50.644892 (+ 57307us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:50.645095 (+   203us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:50.645698 (+   603us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:50.646226 (+   528us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:50.646501 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37765,"prepare.queue_time_us":45434,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"thread_start_us":294586,"threads_started":2}
I20260709 12:26:50.664296  5709 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:50.664703  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:50.665400  5704 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:50.665968  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:50.666803  5709 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 11.
I20260709 12:26:50.669118  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:50.604234 10893 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:50.640949 10598 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:50.641436 11110 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:50.680573  5552 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
W20260709 12:26:50.681070  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68222.5us
I20260709 12:26:50.682736  5557 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 15. Current term is 16. Ops: []
I20260709 12:26:50.685189  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:50.646557 11064 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:50.687062  5560 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:26:50.688768  5552 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 11.
I20260709 12:26:50.621748 11107 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.691959  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:50.693631  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:50.609186 11111 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:50.694491  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 66947.8us
I20260709 12:26:50.616159 11105 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.705035  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:50.686411 11127 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:50.706732  5554 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:26:50.633183 11112 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.671712 10930 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:50.711953  5704 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:50.691140 11070 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:50.701645  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:50.697355 10884 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:50.724793 11124 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 12:26:50.726341  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2394) took 1506 ms. Trace:
I20260709 12:26:50.726778  5703 rpcz_store.cc:276] 0709 12:26:49.219626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:49.234921 (+ 15295us) service_pool.cc:224] Handling call
0709 12:26:50.726069 (+1491148us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b5400451fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:50.726245 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":236798208}
I20260709 12:26:50.689872 10598 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:50.731071  5708 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:50.738516  5703 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:50.710820 10883 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:50.694427 11064 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:50.733902  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:50.688953 11110 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.717142 11127 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.748286  5709 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:26:50.777967  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:50.756606  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:50.786993  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:50.787799  5633 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:50.763620  5703 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20260709 12:26:50.773379  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:50.757139  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:50.791797  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:50.790370  5703 pending_rounds.cc:85] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 6
I20260709 12:26:50.792995  5703 pending_rounds.cc:107] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20260709 12:26:50.793387  5703 raft_consensus.cc:2889] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:50.765951  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 969) took 1254 ms. Trace:
I20260709 12:26:50.795445  6274 rpcz_store.cc:276] 0709 12:26:49.511877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:49.512900 (+  1023us) service_pool.cc:224] Handling call
0709 12:26:50.765706 (+1252806us) spinlock_profiling.cc:243] Waited 43.8 ms on lock 0x7b5400055854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:50.765864 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":100648320}
I20260709 12:26:50.774085  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:50.797960  5560 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:50.800637  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:50.801352  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:50.801998  5552 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:50.802292  5560 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:50.815568  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:50.760403 11123 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:26:50.816272  5556 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:26:50.822650 11130 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:50.822891  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 66394.4us
W20260709 12:26:50.763242 11051 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.892s	user 0.013s	sys 0.000s
I20260709 12:26:50.740487 10884 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:50.757050 11106 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:50.762902 10871 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:50.765674 11058 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:50.764250 10598 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:50.842895  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:50.860994  5634 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:50.761624 11064 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:50.870965  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 188231us
I20260709 12:26:50.851814  5560 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:26:50.744921 11129 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.092s
I20260709 12:26:50.772257 11082 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:50.769953 11112 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:50.863621 11126 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 12:26:50.870486 10598 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:50.873378  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:50.880198  5553 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:26:50.870757 11058 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:50.885118 10893 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:50.880914 10930 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:50.927532  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54247.6us
I20260709 12:26:50.852980 11051 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:50.921862 10598 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:50.942034 11134 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:50.942623 11134 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:50.943980 10598 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:50.923442 11112 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:50.859676 11130 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:50.935645 10725 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.001s	user 0.008s	sys 0.006s
W20260709 12:26:50.956552  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 121826us
I20260709 12:26:50.927891 11063 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:50.988906  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 967) took 1479 ms. Trace:
I20260709 12:26:50.990087  6276 rpcz_store.cc:276] 0709 12:26:49.509582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:49.511983 (+  2401us) service_pool.cc:224] Handling call
0709 12:26:49.512642 (+   659us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:49.670270 (+157628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:50.988668 (+1318398us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b5400591ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:50.988849 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":360960384}
I20260709 12:26:50.939977 11105 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:50.962116 11120 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:26:50.937518 11129 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:50.889499 11082 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:50.988238 11051 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:51.015027  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:50.951479 11112 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:50.998701 10725 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:50.999869 10893 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260709 12:26:51.006731 11129 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:51.039541 11128 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.232s	user 0.001s	sys 0.001s
I20260709 12:26:50.997923 11131 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.255s
I20260709 12:26:51.019872 11134 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:26:51.025413 11110 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:51.060547 11141 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260709 12:26:51.046159 11129 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:51.056470 11115 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.239s	user 0.000s	sys 0.001s
W20260709 12:26:51.052011 11075 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.307s	user 0.003s	sys 0.008s
I20260709 12:26:51.084496 11127 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:51.087460 10952 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:51.057757 10725 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:51.058480 11135 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:51.085511 10902 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
W20260709 12:26:51.159682 11100 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.770s	user 0.009s	sys 0.004s
W20260709 12:26:51.174961  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 215988us
W20260709 12:26:51.179306  5676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.015s	sys 0.005s
W20260709 12:26:51.179726  5676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.015s	sys 0.005s
I20260709 12:26:51.180680  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:51.095715 11112 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:51.192994 11145 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:51.193862 11145 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:51.198379 11151 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
W20260709 12:26:51.199225 11133 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.274s	user 0.001s	sys 0.001s
I20260709 12:26:51.204838  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2571) took 1210 ms. Trace:
I20260709 12:26:51.205272  5627 rpcz_store.cc:276] 0709 12:26:49.994556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:50.133229 (+138673us) service_pool.cc:224] Handling call
0709 12:26:50.135053 (+  1824us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:50.178150 (+ 43097us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:50.178227 (+    77us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:50.182618 (+  4391us) log.cc:844] Serialized 26 byte log entry
0709 12:26:50.514893 (+332275us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:50.514956 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:50.515057 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:51.204587 (+689530us) raft_consensus.cc:1685] finished
0709 12:26:51.204608 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:51.204772 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29444,"raft.queue_time_us":48112,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":10952,"thread_start_us":268585,"threads_started":1,"wal-append.queue_time_us":406154}
I20260709 12:26:51.203414 11127 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:51.212215 11127 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:51.216012  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2539) took 1373 ms. Trace:
I20260709 12:26:51.216643  5702 rpcz_store.cc:276] 0709 12:26:49.842742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:50.613625 (+770883us) service_pool.cc:224] Handling call
0709 12:26:50.614065 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:50.616882 (+  2817us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:50.616929 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:50.628664 (+ 11735us) log.cc:844] Serialized 26 byte log entry
0709 12:26:50.723679 (+ 95015us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:50.733390 (+  9711us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:50.733486 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:51.215692 (+482206us) raft_consensus.cc:1685] finished
0709 12:26:51.215723 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:51.215913 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2319,"raft.queue_time_us":2431,"spinlock_wait_cycles":12358016,"thread_start_us":29358,"threads_started":1,"wal-append.queue_time_us":173818}
I20260709 12:26:51.222797 11127 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:51.173903 11100 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:51.239288  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:51.239952  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 462045us
I20260709 12:26:51.240147  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:51.240835  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:51.247448 11145 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:51.242051  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:51.167928 11135 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:51.243542 11139 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:26:51.256964  5629 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:51.253448  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:51.189553 10725 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:51.256341 11018 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:26:51.242900 11130 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:51.265427 10865 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:51.272470 11154 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
W20260709 12:26:51.284489 11144 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.231s	user 0.001s	sys 0.000s
I20260709 12:26:51.258339 11082 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:51.269274 11015 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.724s	user 0.010s	sys 0.004s
I20260709 12:26:51.268694 11018 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:51.304078 11150 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:26:51.304955 11150 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:51.290055 11135 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:51.300981 11018 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:51.313679 11153 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:51.340777 11153 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:51.342185  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:51.343055  5705 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:51.346575 11100 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:51.378552  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2544) took 1521 ms. Trace:
I20260709 12:26:51.379212  5708 rpcz_store.cc:276] 0709 12:26:49.857473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:50.738660 (+881187us) service_pool.cc:224] Handling call
0709 12:26:50.739121 (+   461us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:50.739625 (+   504us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:50.739665 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:50.752462 (+ 12797us) log.cc:844] Serialized 52 byte log entry
0709 12:26:50.873204 (+120742us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:50.922420 (+ 49216us) log.cc:844] Serialized 14 byte log entry
0709 12:26:50.941612 (+ 19192us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:50.941688 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:51.378208 (+436520us) raft_consensus.cc:1685] finished
0709 12:26:51.378237 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:51.378462 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19903,"prepare.queue_time_us":48097,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":4571,"raft.queue_time_us":10985,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":12808,"thread_start_us":80154,"threads_started":1,"wal-append.queue_time_us":153172}
I20260709 12:26:51.382330  5705 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
W20260709 12:26:51.388628  5705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2409) took 2123 ms (client timeout 1847 ms). Trace:
W20260709 12:26:51.397461  5705 rpcz_store.cc:269] 0709 12:26:49.264940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:49.504262 (+239322us) service_pool.cc:224] Handling call
0709 12:26:51.341949 (+1837687us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b5800053198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:26:51.388453 (+ 46504us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":32473856}
W20260709 12:26:51.409020 11066 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.839s	user 0.014s	sys 0.000s
I20260709 12:26:51.409766 11066 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:51.306167 11150 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:51.432137 11153 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:51.440531  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 443924us
I20260709 12:26:51.442602  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:51.444591  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:51.447660  5702 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:51.445084  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:51.449331  5703 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:51.445436  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:51.443580  5709 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 9. Current term is 10. Ops: []
I20260709 12:26:51.470305  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:51.471297  5708 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:51.441373 11129 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:51.501153  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:51.474871 11023 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.135s	user 0.012s	sys 0.032s
W20260709 12:26:51.503727 10917 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.001s	sys 0.035s
W20260709 12:26:51.510154  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 255435us
W20260709 12:26:51.508035 11023 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.171s	user 0.012s	sys 0.032s
I20260709 12:26:51.486269 11127 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
W20260709 12:26:51.510418 10917 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.001s	sys 0.035s
I20260709 12:26:51.518003 10924 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:51.559520 11159 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:26:51.581182 11105 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:26:51.581712 11105 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:51.574311 10924 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:51.582690 11105 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:51.585090 11105 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:51.586681 11111 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:51.581209 11132 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.660s	user 0.004s	sys 0.000s
W20260709 12:26:51.588275 11114 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.104s	user 0.001s	sys 0.000s
I20260709 12:26:51.595867 11129 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:51.584090 11066 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:51.638155  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 977) took 1054 ms. Trace:
W20260709 12:26:51.649521  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 408047us
I20260709 12:26:51.654680  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:51.657176  5702 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:51.659178  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:51.638998  5559 rpcz_store.cc:276] 0709 12:26:50.583491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:50.584543 (+  1052us) service_pool.cc:224] Handling call
0709 12:26:50.584830 (+   287us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:50.585204 (+   374us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:50.585229 (+    25us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:50.610068 (+ 24839us) log.cc:844] Serialized 52 byte log entry
0709 12:26:50.774274 (+164206us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:51.065196 (+290922us) log.cc:844] Serialized 14 byte log entry
0709 12:26:51.068171 (+  2975us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:51.068267 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:51.560345 (+492078us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:51.637012 (+ 76667us) raft_consensus.cc:1685] finished
0709 12:26:51.637265 (+   253us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:51.637786 (+   521us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32412,"prepare.queue_time_us":100626,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":2813,"raft.queue_time_us":5871,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":183,"spinlock_wait_cycles":6142336,"thread_start_us":372917,"threads_started":2,"wal-append.queue_time_us":204984}
I20260709 12:26:51.662451  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:51.665858  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:51.668874  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:51.671010  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:51.682562 10759 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:51.684762 10804 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:51.684774 10783 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:51.685448 10725 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:51.685812 10804 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:51.686035 10783 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:51.686455 10725 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:51.689683 10783 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:51.690490 10804 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:51.683712 10759 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:51.725260  5708 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 13: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:51.733693 11156 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:26:51.736199 10759 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:51.734398 11156 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:51.579281 11018 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:51.759876  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:51.761067  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:51.761554 11018 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:51.761946  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:51.758044  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2302) took 2438 ms. Trace:
W20260709 12:26:51.768729  5630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2299) took 2087 ms (client timeout 1999 ms). Trace:
W20260709 12:26:51.769316  5630 rpcz_store.cc:269] 0709 12:26:49.680975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:49.683775 (+  2800us) service_pool.cc:224] Handling call
0709 12:26:51.768148 (+2084373us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b540043a154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:51.768409 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":476829952}
I20260709 12:26:51.770853  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2297) took 2097 ms. Trace:
I20260709 12:26:51.761972  5708 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:51.747556 11156 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:51.771376  5631 rpcz_store.cc:276] 0709 12:26:49.673012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:49.673236 (+   224us) service_pool.cc:224] Handling call
0709 12:26:49.675214 (+  1978us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:50.258555 (+583341us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:51.769971 (+1511416us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b54001dd8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:51.770553 (+   582us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1323482368}
I20260709 12:26:51.763478  5561 rpcz_store.cc:276] 0709 12:26:49.319509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:49.319664 (+   155us) service_pool.cc:224] Handling call
0709 12:26:49.320077 (+   413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:49.388096 (+ 68019us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:49.388131 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:49.388669 (+   538us) log.cc:844] Serialized 26 byte log entry
0709 12:26:49.802640 (+413971us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:49.802693 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:51.757098 (+1954405us) spinlock_profiling.cc:243] Waited 67.2 ms on lock 0x7b540016ba54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:51.757234 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:51.757334 (+   100us) raft_consensus.cc:1685] finished
0709 12:26:51.757350 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:51.757783 (+   433us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7414,"raft.queue_time_us":1196,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":9852,"spinlock_wait_cycles":154589952,"thread_start_us":366259,"threads_started":1,"wal-append.queue_time_us":373184,"wal-append.run_cpu_time_us":1563,"wal-append.run_wall_time_us":1004014}
I20260709 12:26:51.779372  5707 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:51.784055  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:51.789726  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:51.790313  5706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2415) took 2434 ms (client timeout 2354 ms). Trace:
W20260709 12:26:51.790895  5706 rpcz_store.cc:269] 0709 12:26:49.356104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:49.681619 (+325515us) service_pool.cc:224] Handling call
0709 12:26:51.789804 (+2108185us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b540026fa54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:51.790103 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1291948672}
I20260709 12:26:51.807888  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:51.811856 10893 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:51.812655 11176 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:51.813335 10939 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:51.826009 11007 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:51.829582  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:51.829907  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:51.830246 10725 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:51.830574  6276 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:26:51.833750  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:51.817471 10893 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:51.837567  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45254.3us
I20260709 12:26:51.842583 11051 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:51.845021 11082 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:51.845532 10893 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:51.846074 11052 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:51.834998  5702 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
W20260709 12:26:51.856752  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 202467us
W20260709 12:26:51.859515  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 341014us
I20260709 12:26:51.868723 11082 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:51.871482  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:51.872040 10881 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:51.873167 10725 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:51.873042 10881 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:51.874231 10725 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:51.876691  5705 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:26:51.862875  6276 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 10.
I20260709 12:26:51.876695 11105 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:51.884886 11105 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:51.881603 11051 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:51.872018 11082 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:51.916702  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52456.4us
W20260709 12:26:51.909536  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:51.871583  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:51.913077  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 975) took 1330 ms. Trace:
I20260709 12:26:51.905889  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2445) took 2922 ms. Trace:
I20260709 12:26:51.930704  5708 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:26:51.926227  5555 rpcz_store.cc:276] 0709 12:26:50.582947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:50.583825 (+   878us) service_pool.cc:224] Handling call
0709 12:26:50.773575 (+189750us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:50.773808 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:50.773846 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:50.773904 (+    58us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:50.773950 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:50.774019 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:51.912704 (+1138685us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54000eb0d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:51.912897 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":435669248}
I20260709 12:26:51.927306  5558 rpcz_store.cc:276] 0709 12:26:48.982823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:49.055486 (+ 72663us) service_pool.cc:224] Handling call
0709 12:26:49.055786 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:49.749353 (+693567us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:51.905003 (+2155650us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b54000271d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:51.905588 (+   585us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1581858048}
I20260709 12:26:51.888268 11105 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:51.897344 10924 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:51.906620 11082 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:51.893944 10881 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:51.888394 11107 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:51.890977 11043 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:51.951570 10718 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:51.957497 10884 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:51.970687  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:51.890408 11051 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:51.953313 11018 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:26:51.977742 10794 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:51.979292  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2398) took 2725 ms. Trace:
I20260709 12:26:51.980813  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:51.942656 11007 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:51.984045  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:51.958904 11059 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:51.986730 10794 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:51.987826 10848 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:51.984442  5701 rpcz_store.cc:276] 0709 12:26:49.254051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:50.213989 (+959938us) service_pool.cc:224] Handling call
0709 12:26:50.214323 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:50.436324 (+222001us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:51.978802 (+1542478us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b54002f6054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:51.979099 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":508802432}
I20260709 12:26:51.986143  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:51.992084  5703 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:26:51.958773 11066 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:26:51.995174  5710 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:26:51.992726 10848 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:51.999027 11059 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.001899 11059 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:52.002187 10783 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:26:52.002841 10783 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:52.004572 10783 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.006637 10783 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:52.007737 10914 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:26:52.008255 10914 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:52.016572 11134 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:26:52.017323 11134 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:52.018978 11134 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:51.981642 11043 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.016611 10914 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.022683 10914 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:51.952343 10718 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:52.066401  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 195167us
I20260709 12:26:52.026137  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:51.991159 10725 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:52.028867  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:52.025503  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2405) took 2761 ms (client timeout 1905 ms). Trace:
W20260709 12:26:52.076059  5707 rpcz_store.cc:269] 0709 12:26:49.263943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:49.527912 (+263969us) service_pool.cc:224] Handling call
0709 12:26:52.025257 (+2497345us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b54003e6f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:52.025424 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4764038528}
I20260709 12:26:52.077545  5559 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:26:52.078639  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 92351.2us
I20260709 12:26:52.080123  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:52.080178 11102 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:52.083099  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:52.073109 11134 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:52.022802 11043 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:51.997700 11007 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.103735  5708 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 13 index: 6. (index mismatch)
I20260709 12:26:52.106253  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:52.107749  6278 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:52.110149 10848 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:52.124303 11007 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:52.129325  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 973) took 1547 ms. Trace:
I20260709 12:26:52.118129  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:52.131992 11011 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:52.132853 11011 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:52.023916 11082 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.116457 10931 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:52.116787 11165 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:26:52.117624  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 170854us
I20260709 12:26:52.118031 11059 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:52.118221  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:52.110904 10924 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:52.129814  5557 rpcz_store.cc:276] 0709 12:26:50.582302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:50.583665 (+  1363us) service_pool.cc:224] Handling call
0709 12:26:50.601475 (+ 17810us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:50.683171 (+ 81696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:52.128687 (+1445516us) spinlock_profiling.cc:243] Waited 81.1 ms on lock 0x7b54001c43d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:52.129083 (+   396us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":186439552}
I20260709 12:26:52.145875 11059 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.158813  5631 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 13: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:52.158933  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.159839  5633 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:52.190382 10794 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:52.191020 11082 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:52.190714 10893 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.191519  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.235777  5628 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:52.203766 10893 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:52.318192  5554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 979) took 1718 ms (client timeout 2039 ms). Trace:
W20260709 12:26:52.318861  5554 rpcz_store.cc:269] 0709 12:26:50.600112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:50.605325 (+  5213us) service_pool.cc:224] Handling call
0709 12:26:52.234417 (+1629092us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b54000267d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:52.317979 (+ 83562us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":230924160}
I20260709 12:26:52.204581 11165 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:52.309484 11011 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:52.190405 10931 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.364962  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.389837 11165 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:52.404933  5633 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
W20260709 12:26:52.510094 11156 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.722s	user 0.013s	sys 0.000s
I20260709 12:26:52.510779 11156 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.512249  5709 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:26:52.591853 11082 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:52.592958 11082 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:52.657636 11134 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.552s	user 0.012s	sys 0.000s
W20260709 12:26:52.676116  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:52.678045 11082 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:52.680310  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:52.681154  5557 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:52.682476 10972 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:52.683224 10972 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:52.685696 10972 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:52.686442  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 75279.4us
W20260709 12:26:52.689801  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 557950us
I20260709 12:26:52.690522 10718 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:52.695545  5557 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
W20260709 12:26:52.696455  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 590875us
I20260709 12:26:52.698854  5636 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:26:52.699036  5627 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:26:52.702672  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.718712  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:52.717741  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:52.722959  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2584) took 2026 ms. Trace:
I20260709 12:26:52.722831  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:52.723453  5629 rpcz_store.cc:276] 0709 12:26:50.696412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:50.696536 (+   124us) service_pool.cc:224] Handling call
0709 12:26:50.696836 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:51.259844 (+563008us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:52.722423 (+1462579us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x7b5400438354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:52.722757 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1287570816}
I20260709 12:26:52.725422  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:52.726478 10462 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:26:52.727012 10462 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:52.703336 11059 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:52.727699  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:52.730355 10462 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.721093  5553 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
W20260709 12:26:52.734540  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 540859us
I20260709 12:26:52.736624 10912 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:52.737396 10912 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:52.738535 11031 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:26:52.739015 11031 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:52.739495 11031 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:26:52.741044  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:52.742291 11007 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:52.743095 11007 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.744733 11065 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:52.745352 11065 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.746157 11007 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:52.747505 10783 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.740s	user 0.009s	sys 0.001s
I20260709 12:26:52.747797 11065 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:52.749661  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 652959us
W20260709 12:26:52.750818  5560 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.017s	sys 0.028s
W20260709 12:26:52.752398  5560 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.017s	sys 0.028s
I20260709 12:26:52.750952 10893 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:26:52.757604 11059 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:52.757783 10893 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:52.758395 11059 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.759186 10893 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.761188 11059 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:52.727456 11051 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:26:52.766628 11051 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:52.775885 11051 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:26:52.776254  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:52.777531  5707 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:26:52.779448  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:52.769043 11156 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:52.772254  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:52.780887  5704 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
W20260709 12:26:52.773046 11183 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.053s	user 0.003s	sys 0.000s
W20260709 12:26:52.782375 10914 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.706s	user 0.007s	sys 0.003s
I20260709 12:26:52.780180  5706 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:52.795663  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:52.796450  5710 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:52.796589  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:52.797642  5707 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:52.798367  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:52.799186  5704 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:52.800621 11059 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:26:52.801352 11059 consensus_queue.cc:1243] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 10.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:26:52.804889  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:52.805986  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:52.807694 11181 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.131s	user 0.002s	sys 0.000s
I20260709 12:26:52.805347 11167 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:52.807996 11099 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:52.809166 11099 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:52.809078 11167 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:52.809675 11099 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:52.731963 10931 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:52.813971  5709 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:26:52.815699 11167 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:52.816447 11059 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:52.817101 11059 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:52.817672 11059 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:52.740229 10912 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:52.826897 11112 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:52.827176  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:52.828487 10848 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:52.828893 10848 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:52.829319 10848 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:52.831672  5704 pending_rounds.cc:85] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 4
I20260709 12:26:52.835368  5707 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:52.837091 11065 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:26:52.838187 11065 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:26:52.838573 11065 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:52.838689  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.844550  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:52.834743  5704 pending_rounds.cc:107] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 10.5
I20260709 12:26:52.848114  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:52.848970  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.847365  5704 raft_consensus.cc:2889] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:52.854167 11134 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:52.856819 11142 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:52.857307 11142 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:52.857650 11142 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:52.851784 10902 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.012s
I20260709 12:26:52.862491  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:52.864748  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:52.873626  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:52.879683  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.874889 11156 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:52.887092  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:52.890655  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.877322 11065 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:52.911653  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.912247  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:52.916556  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.917374  5628 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:52.921698  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.895471 11156 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:52.928915  5629 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
W20260709 12:26:52.976724  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:52.968756  5630 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 17: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:52.975666  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.924789 11065 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:26:52.960865  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 65022.3us
I20260709 12:26:52.976066  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.980566  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:52.977854  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:53.022063  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.950673  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:53.022603  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:53.022784  5631 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 10.
I20260709 12:26:52.942119 10783 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:53.023183  5703 raft_consensus.cc:2376] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13. Re-sending same reply.
I20260709 12:26:53.023965  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:53.024576  5703 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 9. (term mismatch)
I20260709 12:26:53.024721  5706 raft_consensus.cc:2368] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15.
I20260709 12:26:53.024930  5703 pending_rounds.cc:85] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 8
I20260709 12:26:53.025187  5703 pending_rounds.cc:107] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 12:26:53.025418  5703 raft_consensus.cc:2889] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:53.025867  5706 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:26:53.027033  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:53.027659  5705 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:53.028087  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:52.951835 10848 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:53.031589  5526 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.024s	sys 0.046s
W20260709 12:26:53.032019  5526 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.024s	sys 0.046s
I20260709 12:26:53.032788  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:53.036549  5705 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:26:52.975148 10871 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:52.951325  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.953771 11156 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:53.042796  5635 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:26:53.032341  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 825827us
I20260709 12:26:53.045245  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:52.938879 10931 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:53.047808  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:53.051887  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:53.053227  5632 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:53.055689  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:53.063592  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:53.086882  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:53.088009  5634 raft_consensus.cc:2410] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 18 because replica has last-logged OpId of term: 17 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:26:53.070003 10783 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:26:53.090478  5629 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:26:53.046463 11031 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:53.041767 10871 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:53.097374  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:53.097759  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:53.098532  5710 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:26:53.100487 11182 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.748s	user 0.004s	sys 0.024s
I20260709 12:26:53.082602 10848 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:53.102231  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:53.105990  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:53.107064 10718 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:53.107573 10718 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:53.108026 10718 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:52.957090  5452 catalog_manager.cc:5697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 12 to 14, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 14 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:53.116770  5632 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:26:53.085482 10931 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:53.125456  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:53.120538 11082 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:53.146638  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:53.136557 11082 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:53.178714 11082 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:53.192725 10783 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:26:53.194082 10783 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:53.194495 10783 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:53.197988  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:53.198652  5557 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:26:53.199983  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:53.200253  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:53.200387  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:53.200719  5555 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:53.201071  5561 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:53.140980 11142 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:53.249053 11185 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
W20260709 12:26:53.163069  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:53.160974  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:53.139511 10912 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:26:53.161616  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:53.163057 11031 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:53.165802 11099 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:53.168931 11051 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:53.170135 11059 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:53.172585  5634 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:26:53.178088 10931 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:53.244155  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:53.296419  5710 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:53.297279  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:53.297363  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:53.298246  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:53.312546  5557 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 18: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:53.299854  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:53.311229  5703 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:26:53.311213  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:53.195617 10783 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:53.316128  5555 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:53.320078  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:53.324076 11011 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.910s	user 0.035s	sys 0.076s
I20260709 12:26:53.277604 10912 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:53.333420 11011 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.918s	user 0.035s	sys 0.076s
W20260709 12:26:53.334512  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 194317us
I20260709 12:26:53.347595  5559 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:53.354897  5632 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
W20260709 12:26:53.362497  5701 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.036s	sys 0.061s
I20260709 12:26:53.363085  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:53.338013 10912 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:53.364784  5554 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:26:53.364856  5701 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.036s	sys 0.061s
I20260709 12:26:53.378084  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:53.384742  5560 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.026s	sys 0.043s
W20260709 12:26:53.385382  5560 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.026s	sys 0.043s
I20260709 12:26:53.390300  5635 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
W20260709 12:26:53.387300  5708 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.005s	sys 0.054s
W20260709 12:26:53.395427  5708 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.005s	sys 0.055s
I20260709 12:26:53.397284  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:53.399880  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.007s	sys 0.022s
W20260709 12:26:53.400272  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.007s	sys 0.023s
W20260709 12:26:53.421816  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:53.398716  5632 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 12.
W20260709 12:26:53.427292  6234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.008s	sys 0.026s
W20260709 12:26:53.429772  6234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.008s	sys 0.026s
W20260709 12:26:53.397964 10462 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.015s	sys 0.014s
W20260709 12:26:53.432132  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 655313us
I20260709 12:26:53.396863 10783 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:53.450062 10462 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.015s	sys 0.015s
I20260709 12:26:53.459003  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:53.451352 11134 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.014s	sys 0.016s
I20260709 12:26:53.465492 11112 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:53.467171  5634 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:26:53.468403  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:53.469552 11051 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:53.470628  5452 catalog_manager.cc:5697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 11. New cstate: current_term: 11 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:53.474161  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:53.462491  5632 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: []
W20260709 12:26:53.477763  5704 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.004s	sys 0.005s
W20260709 12:26:53.495429  5704 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.004s	sys 0.005s
I20260709 12:26:53.476017  5634 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:53.477098  5636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.001s	sys 0.009s
W20260709 12:26:53.499039  5636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.001s	sys 0.009s
W20260709 12:26:53.386312  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.004s	sys 0.024s
W20260709 12:26:53.500732  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.004s	sys 0.025s
W20260709 12:26:53.501726  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 743645us
W20260709 12:26:53.457222 11165 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.003s	sys 0.023s
W20260709 12:26:53.476997 11134 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.014s	sys 0.017s
I20260709 12:26:53.514813 11142 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:53.514874  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.003s	sys 0.007s
W20260709 12:26:53.517824  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.003s	sys 0.007s
I20260709 12:26:53.520241 10871 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:53.520808 10870 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.010s	sys 0.049s
I20260709 12:26:53.519600  5703 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
W20260709 12:26:53.521301 10870 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.010s	sys 0.050s
I20260709 12:26:53.522344 11099 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:53.523419  5634 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:53.496383 11112 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:53.511406 11165 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.003s	sys 0.023s
I20260709 12:26:53.529610  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:53.544027  5707 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.009s	sys 0.016s
I20260709 12:26:53.511375 11134 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:53.548539  5634 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:53.566152  5631 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
W20260709 12:26:53.546044  5707 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.009s	sys 0.016s
I20260709 12:26:53.537536 11059 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:53.548638 10893 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.012s	sys 0.028s
I20260709 12:26:53.507047 10718 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:53.607156  6276 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:26:53.605005  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:53.617830  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:53.624899  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:53.601070 10893 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.013s	sys 0.029s
I20260709 12:26:53.620047  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 995) took 1591 ms. Trace:
W20260709 12:26:53.632694  5630 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.005s	sys 0.018s
W20260709 12:26:53.633318  5630 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.005s	sys 0.018s
W20260709 12:26:53.644917  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:53.644212  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 185288us
I20260709 12:26:53.648967  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:53.649573  5710 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:53.651638  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:53.652415  5701 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:53.658057  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 610280us
W20260709 12:26:53.656508  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 287615us
I20260709 12:26:53.671701  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:53.626107  5560 rpcz_store.cc:276] 0709 12:26:52.028832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:52.028962 (+   130us) service_pool.cc:224] Handling call
0709 12:26:52.029306 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:52.029693 (+   387us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:26:52.754956 (+725263us) log.cc:844] Serialized 14 byte log entry
0709 12:26:53.619379 (+864423us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:53.619750 (+   371us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:53.619804 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:53.619864 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:53.619987 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10813,"prepare.queue_time_us":100710,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":152956,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":27297,"thread_start_us":1583183,"threads_started":3,"wal-append.queue_time_us":722751}
I20260709 12:26:53.685946 11225 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:53.687019 11225 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:53.688984 10930 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:53.689716 10930 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:53.693704 10930 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:53.696884 10930 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:53.697854 11225 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:53.653391 10931 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:53.714285 10931 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:53.637990 10870 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:53.718950 10870 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:53.722185 10870 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:53.722436  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:53.723062 10870 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:53.723414  5706 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:26:53.717686 10931 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:53.723852 10870 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:53.715873  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:53.729101  5702 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:53.727854 10870 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:53.730134 10870 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:53.730966 10870 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:53.729266 11225 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:53.733232 11225 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:53.738106 11225 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:53.739607 11225 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:53.740489 11225 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:53.743465 11225 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:53.738798 10870 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:53.745105 10870 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:53.749802  5710 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
W20260709 12:26:53.755040  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 46666.7us
I20260709 12:26:53.712617  5709 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:53.770501  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:53.772815  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:53.773931  5705 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 15: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:53.774941  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:53.775125  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:53.776005  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:53.776306  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:53.751663 11201 raft_consensus.cc:2764] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
W20260709 12:26:53.800916  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44896.6us
I20260709 12:26:53.678544 11156 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:53.789498  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:53.814160  5705 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:26:53.812657  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:53.810451  5706 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:26:53.815361  5555 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:53.791460  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:53.791908  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:53.778414  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:53.699801 10930 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:53.953593  6280 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:26:53.821374  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 156416us
W20260709 12:26:53.956277  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 408221us
W20260709 12:26:53.947822  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 171847us
I20260709 12:26:53.915670  5634 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 16 index: 14. (term mismatch)
W20260709 12:26:53.955055  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 196710us
W20260709 12:26:53.960640  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 109206us
I20260709 12:26:53.965207  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:53.862309  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 211837us
I20260709 12:26:53.916851  5710 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:26:53.967260  5634 pending_rounds.cc:85] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 13
W20260709 12:26:53.958006  5601 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.722s (SENT)
I20260709 12:26:53.958033  5702 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
W20260709 12:26:53.969485  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 152298us
I20260709 12:26:53.971457  5634 pending_rounds.cc:107] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 15.14
I20260709 12:26:53.971850  5634 raft_consensus.cc:2889] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:53.963881  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:53.916424 11204 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.377s	user 0.002s	sys 0.000s
W20260709 12:26:53.916041 11208 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
W20260709 12:26:53.862038 11195 log.cc:927] Time spent T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.298s	user 0.000s	sys 0.001s
I20260709 12:26:53.781541 10718 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:53.979240  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:53.707926 11134 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:54.022190  5701 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:53.860572 10902 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:54.023044  5705 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:26:53.970880  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:53.862973 11199 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.210s	user 0.001s	sys 0.000s
W20260709 12:26:53.982476  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:53.982363  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:53.982923 11214 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.165s	user 0.000s	sys 0.000s
I20260709 12:26:53.983264 11223 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:54.036407 11223 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:54.038214 10462 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:26:54.038437 10930 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:53.983678  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:54.038866 10462 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:53.984074  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.040027 10462 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:54.041499  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.042075  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.042732  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.043268  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.043901  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:53.762223 11112 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:54.045047 11209 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:54.045516 11209 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:54.045853 11209 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:53.985306  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:53.616299  5452 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 5 to 11. New cstate: current_term: 11 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:53.986671  6272 raft_consensus.cc:2368] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18.
I20260709 12:26:53.778514 11200 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:53.992194  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:53.948808  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.058692  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:54.005784 10870 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:26:54.000962 11156 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:54.006495  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:54.066648  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:53.855525  5454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 275) took 64308 ms (client timeout 15000 ms). Trace:
I20260709 12:26:53.788514 11149 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:26:54.015461  5602 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.722s (SENT)
W20260709 12:26:53.793309 11196 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260709 12:26:54.018905  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:53.976680 10931 raft_consensus.cc:2764] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:26:54.052704  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:54.027925  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57051us
W20260709 12:26:54.022998 10783 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.555s	user 0.007s	sys 0.003s
I20260709 12:26:54.059216  6271 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
W20260709 12:26:54.067238  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60822.3us
W20260709 12:26:54.084100  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62369.3us
I20260709 12:26:53.835760 11225 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.214s
I20260709 12:26:53.778652 11206 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.378s
I20260709 12:26:53.855312 10912 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:54.038884 11223 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:54.065655 11200 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:54.103773 11227 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:54.131052  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:54.129465  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:54.109957 11167 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:26:54.052027 10902 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:54.137912  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:54.140601  5631 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:26:54.128274 11219 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:26:54.123430 10783 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:26:54.142200  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 156088us
I20260709 12:26:54.155679  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:54.146068  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:26:54.148689 11213 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.232s	user 0.002s	sys 0.000s
I20260709 12:26:54.148978  5632 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:54.141535 11223 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:54.153232  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.153960  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:54.128893 11220 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:54.221091  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2578) took 1447 ms. Trace:
I20260709 12:26:54.214438  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.165936  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:54.244671  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 224211us
I20260709 12:26:54.242713  6273 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:26:54.243314  5708 rpcz_store.cc:276] 0709 12:26:52.773376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:52.773535 (+   159us) service_pool.cc:224] Handling call
0709 12:26:52.773930 (+   395us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:52.774436 (+   506us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:52.774466 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:52.775138 (+   672us) log.cc:844] Serialized 52 byte log entry
0709 12:26:53.395966 (+620828us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:26:53.396038 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:53.396138 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:54.220688 (+824550us) raft_consensus.cc:1685] finished
0709 12:26:54.220721 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:54.221004 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6081,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"thread_start_us":620278,"threads_started":1,"wal-append.queue_time_us":749255}
W20260709 12:26:54.200608  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58662.9us
I20260709 12:26:54.245268  6275 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:54.164980  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:54.141404 11082 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.962s	user 0.014s	sys 0.000s
I20260709 12:26:54.164661  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:54.249320  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:54.157831 11065 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:54.137856 11227 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:54.216600 11232 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20260709 12:26:54.138316 10931 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:54.261428  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 101259us
I20260709 12:26:54.263512 11065 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:54.280593 11065 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:54.289999 11215 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.433s	user 0.004s	sys 0.001s
I20260709 12:26:54.297569  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:54.299038 11234 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.302s	user 0.001s	sys 0.008s
W20260709 12:26:54.266878  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 101040us
I20260709 12:26:54.301578  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:54.217201 11207 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:26:54.268882  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:54.137247 11200 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:54.208719 10871 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:54.267143  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:54.270057 11227 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:54.302991  6279 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 13: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:26:54.271338  5601 leader_election.cc:336] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 2.277s (SENT)
I20260709 12:26:54.318068  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:54.309953  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 43999.5us
I20260709 12:26:54.243191 11149 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:54.240921 10912 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:54.313473  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 98632.8us
I20260709 12:26:54.314409 11219 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:26:54.323006 11216 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.417s	user 0.004s	sys 0.000s
W20260709 12:26:54.323536  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 58436.1us
I20260709 12:26:54.324185 11202 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:26:54.302943 11059 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:26:54.314622 10972 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:26:54.327050  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 334931us
W20260709 12:26:54.333566  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:54.338238  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:54.339552 11242 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:54.340030 11242 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:54.340415 11242 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:54.343072  5701 pending_rounds.cc:85] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 8
I20260709 12:26:54.343575  5701 pending_rounds.cc:107] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:26:54.343822  5701 raft_consensus.cc:2889] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:54.345408  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:54.241195  5452 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 11 to 14. New cstate: current_term: 14 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:26:54.347214 11212 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.342s	user 0.002s	sys 0.000s
I20260709 12:26:54.265187 11082 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:54.362064  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:54.362697  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:54.358651  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.359356  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:54.363924  5710 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:26:54.359869 11230 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 12:26:54.361173  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:54.362561  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44916.9us
I20260709 12:26:54.333244  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:54.364462  5675 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:54.141008 11198 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.199s
I20260709 12:26:54.321897 11149 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:54.321636 10871 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:54.361449  6237 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:54.375545 11243 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:26:54.384948  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:54.300083 11231 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.049s
I20260709 12:26:54.377135  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:54.315234 11225 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
W20260709 12:26:54.431777  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54726.5us
I20260709 12:26:54.335387 11165 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:54.334456 11202 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:54.433761  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60890.2us
I20260709 12:26:54.350764 11051 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:26:54.435562  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44624.2us
I20260709 12:26:54.436051  6278 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:26:54.335286 10972 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:54.348245 10902 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:54.443198  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:54.443382  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:54.386806 11059 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:54.439041 11223 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:26:54.468158  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.466066  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:54.444574 10871 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:54.448880  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 485059us
I20260709 12:26:54.348390  5452 catalog_manager.cc:5697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 9 to 12. New cstate: current_term: 12 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:54.490517  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:26:54.463200  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 490274us
W20260709 12:26:54.470255 11210 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.617s	user 0.004s	sys 0.000s
I20260709 12:26:54.470745  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.471405  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:54.488703  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 50651.5us
I20260709 12:26:54.491904  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:54.504531 11220 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:54.505926  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:54.496683 11051 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:54.507818 11235 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:54.462495 10972 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:54.497741 11202 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:54.500260 11209 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:54.432758 11156 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:54.501029  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 57767.4us
I20260709 12:26:54.501761 11059 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:54.502972 11241 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:54.506726 11223 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:54.461927 11212 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:26:54.393831 11112 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:54.469318 11011 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:54.442165 10893 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:54.513733  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:54.491253 11239 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.242s	user 0.000s	sys 0.001s
I20260709 12:26:54.443238 11218 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:54.522302  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.513263 11165 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:54.548802  5601 leader_election.cc:336] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.806s (SENT)
W20260709 12:26:54.525821  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 59873.4us
I20260709 12:26:54.549927  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:54.390151 10912 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.125s
I20260709 12:26:54.532853  5452 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 9 to 14. New cstate: current_term: 14 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:26:54.530530  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:54.539296  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.544526  5634 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:54.516752 11206 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:26:54.558246  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:54.562911  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:54.563666  5553 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:54.577376 11235 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:54.577594  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:54.578069 11235 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:54.581521 11241 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:26:54.580920  5452 catalog_manager.cc:5697] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 9, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 9 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:54.582052 11241 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:54.582682  5452 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 7 to 11. New cstate: current_term: 11 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:26:54.583045 11241 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:54.584673 11245 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.160s	user 0.000s	sys 0.009s
W20260709 12:26:54.080138  5454 rpcz_store.cc:269] 0709 12:25:49.358232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:49.358998 (+   766us) service_pool.cc:224] Handling call
0709 12:26:53.655883 (+64296885us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:44.297638 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:44.498456 (+200818us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:44.509620 (+ 11164us) write_op.cc:435] Acquired schema lock
0709 12:26:44.512887 (+  3267us) tablet.cc:662] Decoding operations
0709 12:26:44.924545 (+411658us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:44.924803 (+   258us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:44.924839 (+    36us) tablet.cc:685] Acquiring locks for 11 operations
0709 12:26:44.925901 (+  1062us) tablet.cc:701] Row locks acquired
0709 12:26:44.925968 (+    67us) write_op.cc:260] PREPARE: finished
0709 12:26:44.926199 (+   231us) write_op.cc:270] Start()
0709 12:26:44.926372 (+   173us) write_op.cc:276] Timestamp: P: 1783600004926115 usec, L: 0
0709 12:26:44.926506 (+   134us) op_driver.cc:348] REPLICATION: starting
0709 12:26:44.927428 (+   922us) log.cc:844] Serialized 4284 byte log entry
0709 12:26:45.593959 (+666531us) op_driver.cc:464] REPLICATION: finished
0709 12:26:45.978320 (+384361us) write_op.cc:301] APPLY: starting
0709 12:26:45.994201 (+ 15881us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:47.108885 (+1114684us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:47.109021 (+   136us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:47.359252 (+250231us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:47.359589 (+   337us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=22,key_file_lookups=22,delta_file_lookups=22,mrs_lookups=0
0709 12:26:47.359635 (+    46us) write_op.cc:312] APPLY: finished
0709 12:26:47.360044 (+   409us) log.cc:844] Serialized 105 byte log entry
0709 12:26:47.909188 (+549144us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:47.932876 (+ 23688us) write_op.cc:454] Released schema lock
0709 12:26:47.949955 (+ 17079us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:48.075227 (+125272us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:48.078203 (+  2976us) write_op.cc:454] Released schema lock
0709 12:26:49.975332 (+1897129us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
Metrics: {"mutex_wait_us":142585,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":241402,"apply.run_cpu_time_us":99896,"apply.run_wall_time_us":2175148,"cfile_cache_hit":46,"cfile_cache_hit_bytes":50107,"num_ops":11,"prepare.queue_time_us":682480,"prepare.run_cpu_time_us":65667,"prepare.run_wall_time_us":968575,"raft.queue_time_us":343146,"raft.run_cpu_time_us":4791,"raft.run_wall_time_us":60850,"replication_time_us":667249,"spinlock_wait_cycles":5403904,"thread_start_us":1688702,"threads_started":5,"wal-append.queue_time_us":901050,"wal-append.run_cpu_time_us":49223,"wal-append.run_wall_time_us":4546019}]]}
I20260709 12:26:54.562515 11218 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:54.592833 11254 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:26:54.595309 11082 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:54.596626 11165 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:54.538272 11011 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:54.599833 11011 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:54.543133 11201 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.063s
I20260709 12:26:54.601233 11201 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:26:54.601514 11209 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:54.601577 11201 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:26:54.602566  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44428.6us
I20260709 12:26:54.602988 11206 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:26:54.603396 11206 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:54.603777 11206 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:54.548141 10930 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:26:54.613188  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 141915us
I20260709 12:26:54.549741 11134 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:26:54.612063  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.635555  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.638885 11134 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:54.626021  5601 leader_election.cc:336] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.744s (SENT)
W20260709 12:26:54.591418 11203 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.357s	user 0.001s	sys 0.002s
W20260709 12:26:54.644848  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:54.591509 11218 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:54.545300 11112 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:54.612162 10912 consensus_queue.cc:1243] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 15.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:26:54.564596 10931 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:54.605607 11167 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:26:54.655117  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50536.5us
W20260709 12:26:54.633437 11240 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.216s	user 0.000s	sys 0.001s
I20260709 12:26:54.670768  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:54.615625 11051 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:54.674180  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:54.675387  6278 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:54.617807 11235 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:54.682420  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:54.683070  5553 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:54.639421 11134 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:26:54.681425  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:54.685123  5560 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:54.695902  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.678037 11226 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:54.699549  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 338164us
I20260709 12:26:54.678943 11167 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:54.602599 11201 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:54.705698 10930 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:54.725332 10930 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:54.725785 10930 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:54.726727  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:54.683409 10783 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:54.729071 11201 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:26:54.732867  6275 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
I20260709 12:26:54.605520  5452 catalog_manager.cc:5697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 11. New cstate: current_term: 11 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:54.736629 11221 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:54.737149 11221 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:54.738144  5633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2311) took 1862 ms (client timeout 1771 ms). Trace:
I20260709 12:26:54.737602  5452 catalog_manager.cc:5697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 6 to 11. New cstate: current_term: 11 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:54.738025 11221 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:54.744758 11221 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:26:54.748683  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:54.748514 11260 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:26:54.739707  5452 catalog_manager.cc:5697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 2 to 11, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 11 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
W20260709 12:26:54.756842 11205 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.904s	user 0.006s	sys 0.002s
I20260709 12:26:54.757779 10930 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:54.767683  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2586) took 1955 ms. Trace:
W20260709 12:26:54.738559  5633 rpcz_store.cc:269] 0709 12:26:52.876225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:52.876419 (+   194us) service_pool.cc:224] Handling call
0709 12:26:54.737930 (+1861511us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b540008b6d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:54.738072 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3807624576}
I20260709 12:26:54.684938 11243 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:54.768324 10930 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:54.787241 11243 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:54.787806 11243 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:54.740585 11167 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:54.768138  5704 rpcz_store.cc:276] 0709 12:26:52.812666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:52.812845 (+   179us) service_pool.cc:224] Handling call
0709 12:26:52.830619 (+ 17774us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:52.894224 (+ 63605us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:52.894258 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:52.913283 (+ 19025us) log.cc:844] Serialized 52 byte log entry
0709 12:26:53.525069 (+611786us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:26:53.525125 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:53.525222 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:54.675523 (+1150301us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:54.767185 (+ 91662us) raft_consensus.cc:1685] finished
0709 12:26:54.767212 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:54.767596 (+   384us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3326,"raft.queue_time_us":8162,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":5545,"spinlock_wait_cycles":26393984,"thread_start_us":564411,"threads_started":1,"wal-append.queue_time_us":744707}
I20260709 12:26:54.751516 11253 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:26:54.792550 11253 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:54.793581 11253 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:54.801481 11220 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:26:54.706641  5601 leader_election.cc:336] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.945s (SENT)
I20260709 12:26:54.706290  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:54.753618 11242 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:54.780089  5553 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:54.804415  5709 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:26:54.803610  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:54.808301  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:54.757815 11248 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:26:54.816110  6275 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:26:54.766552  5452 catalog_manager.cc:5697] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 6 to 12, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 12 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:26:54.756476 11251 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 12:26:54.809933  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:54.814690  5560 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
W20260709 12:26:54.817651  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 238306us
I20260709 12:26:54.824539  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:54.820087  5633 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: []
W20260709 12:26:54.826231  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 172754us
I20260709 12:26:54.824311  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:54.829638  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2588) took 1952 ms. Trace:
I20260709 12:26:54.831982  5707 rpcz_store.cc:276] 0709 12:26:52.876752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:52.876887 (+   135us) service_pool.cc:224] Handling call
0709 12:26:52.882970 (+  6083us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:52.888907 (+  5937us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:52.888929 (+    22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:52.915977 (+ 27048us) log.cc:844] Serialized 26 byte log entry
0709 12:26:53.571558 (+655581us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:53.571629 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:53.571752 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:54.590493 (+1018741us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:54.818622 (+228129us) raft_consensus.cc:1685] finished
0709 12:26:54.818645 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:54.829522 (+ 10877us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8600,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":1785,"spinlock_wait_cycles":27857152,"thread_start_us":647801,"threads_started":1,"wal-append.queue_time_us":1207816}
I20260709 12:26:54.821719  5627 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: []
I20260709 12:26:54.768072 10972 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:54.848826  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:54.852965 11011 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:54.759919 11149 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:54.687988 11223 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:54.860828  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.818629 11220 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:54.863143  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:54.864390  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.816272 10848 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:26:54.853161  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:54.868815  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:54.869720  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.870707  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:54.871397  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:54.878443  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:54.881834  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.882894  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:54.883580  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 176962us
W20260709 12:26:54.884178  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:54.884233  6280 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:26:54.871639  6274 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:26:54.885948  6271 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:54.887475  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:54.888549  6273 raft_consensus.cc:2376] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12. Re-sending same reply.
I20260709 12:26:54.861745 10931 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:54.892421  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.862551 11198 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:26:54.900734  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:54.757601 11206 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:54.902050  5705 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:26:54.822376 10930 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 12:26:54.904928 11149 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:54.905745  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.906735  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:54.906764  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.862141 11253 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:26:54.897536  5601 leader_election.cc:336] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.945s (SENT)
I20260709 12:26:54.772744 11226 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:54.882680  5675 connection.cc:256] RPC call timeout handler was delayed by -1.4263496350004061s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1771s for a response
I20260709 12:26:54.901981  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:54.905939 11227 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.583s	user 0.012s	sys 0.000s
I20260709 12:26:54.911484  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:54.907042  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.898706  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:54.898257  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:54.848307 10870 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 12:26:54.853152  5452 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 7 to 9, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 9 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:54.912690  5707 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:26:54.912786  5709 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:26:54.907585 11206 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:54.891238 11254 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:54.893389 11221 raft_consensus.cc:2764] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:26:54.899156 11220 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:54.919979 11209 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:26:54.920277 11256 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:54.925601  5452 catalog_manager.cc:5697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 10, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 10 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:26:54.918596  5601 leader_election.cc:336] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.897s (SENT)
I20260709 12:26:54.915941 11227 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:54.917526 11241 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:54.932931  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 69055.5us
I20260709 12:26:54.927289  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:54.900876 11249 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
I20260709 12:26:54.933192  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:54.927491 11263 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:26:54.959393  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:54.917418 10931 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:54.927394 10930 consensus_queue.cc:1243] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:26:54.963188 11242 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:54.962935  5452 catalog_manager.cc:5697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 3 to 9, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 9 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:54.963745 11243 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:54.965245 11226 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:54.981616  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 48525.2us
I20260709 12:26:54.983297  5558 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:54.992996  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:54.990798  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:55.000145  5452 catalog_manager.cc:5697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 12, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 12 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } }
I20260709 12:26:55.002518  5452 catalog_manager.cc:5697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:26:55.004671 11011 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:55.005010  5452 catalog_manager.cc:5697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 9 to 10, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 10 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:26:54.944680 11149 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:26:55.009073  5710 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:54.983860 11219 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:54.983027 11031 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:54.955890 11167 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:55.033104  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 73844.4us
W20260709 12:26:55.038475  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53807.6us
I20260709 12:26:55.011806 11226 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:26:55.047415  6234 connection.cc:256] RPC call timeout handler was delayed by -1.5165605059996778s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1718s for a response
W20260709 12:26:55.048127  6234 connection.cc:256] RPC call timeout handler was delayed by -1.4850605060000817s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1753s for a response
I20260709 12:26:55.052640  6275 raft_consensus.cc:2368] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14.
I20260709 12:26:54.983668 11241 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:54.950304 11254 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.058795  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:55.062868  5602 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.531s (SENT)
I20260709 12:26:55.031797 11198 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:55.047515 11226 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:55.067361  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:55.041296 11099 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 12:26:55.081040  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:55.085585 11241 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:55.085721 11254 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:55.091300 11261 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260709 12:26:55.092828 11065 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.812s	user 0.011s	sys 0.002s
I20260709 12:26:55.093278 11065 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:55.093358  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44394.5us
W20260709 12:26:55.094880 11258 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.370s	user 0.000s	sys 0.002s
I20260709 12:26:55.094810  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:55.095953 11065 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:55.096458 10783 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:55.096908 10783 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:55.096949 11227 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:55.097682 11243 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:55.098028 11273 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:55.099798 11219 raft_consensus.cc:2764] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election decision vote started in defunct term 17: won
I20260709 12:26:55.101238  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:55.101630 10972 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:55.102928  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:55.103322 11167 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:55.103477  5552 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:55.096982 11266 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:26:55.007925  5452 catalog_manager.cc:5697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 4 to 12, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 12 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:55.106745 11257 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:55.107111  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:55.107286 11257 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.107736  5556 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:55.109416 11257 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:55.110183 11209 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:55.110633 11209 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:55.113528 11250 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.381s	user 0.001s	sys 0.000s
I20260709 12:26:55.116555 11222 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:55.117031 11222 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:55.117895 11222 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:55.118995 11222 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:55.121049 11218 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:55.121256 11112 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:55.121582 11218 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:55.122855  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:55.125574  5702 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:26:55.126781 11229 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:26:55.127242 11229 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:55.128137  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:55.128901  5636 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:26:55.129004  5705 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:55.129184  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:55.129834  5704 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:26:55.130509  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:55.131047  5635 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:55.131400  5703 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:26:55.075599 11226 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:55.137264  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:55.137861  5628 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:55.138989 10718 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:26:55.139907 10718 consensus_queue.cc:1243] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 19.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:26:55.143663 11149 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:26:55.144064 11149 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:55.050093 11220 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.144671  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:55.145268  5630 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:55.144874 11149 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:55.145967  5554 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:55.146127 11149 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:26:55.106465  5452 catalog_manager.cc:5697] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 7 to 12, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 12 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:26:55.149106  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 66214.8us
I20260709 12:26:55.149216  5631 raft_consensus.cc:1217] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Deduplicated request from leader. Original: 11.6->[14.7-14.7]   Dedup: 14.7->[]
I20260709 12:26:55.152626  5631 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:26:55.128333  5708 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
W20260709 12:26:55.149812  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:55.126896  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:55.155434  5702 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:55.151010  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:55.156428  5704 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:55.153504  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:55.159938  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:55.166033 10848 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:26:55.166479 10848 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:55.164239 11254 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:55.167135  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:55.165138  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:55.167676  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:55.169896  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:55.175289  5552 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:55.148447 10870 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:55.157981 10871 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:55.189465  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:55.067124 11228 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.176s
I20260709 12:26:55.166936 10848 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:55.060551 10931 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:55.208006  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:55.208666  5627 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:55.198940  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:55.197005 10871 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.212181 11255 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:55.187098 11218 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:55.194826 11031 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.019s
I20260709 12:26:55.183496 11243 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 12:26:55.215751 11272 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:26:55.216125 11243 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:55.216518 11243 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:55.217314 11243 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:55.218402 11243 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:55.195392 10902 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 12:26:55.208464 10893 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.012s
I20260709 12:26:55.221422 11059 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:55.166777 11262 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:26:55.221796 11059 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:55.190948 11241 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.035s
I20260709 12:26:55.222769 11241 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:55.222617 11059 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:55.223202 11241 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:55.223841 11059 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:55.220019  6276 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
W20260709 12:26:55.214808  5601 leader_election.cc:336] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.593s (SENT)
I20260709 12:26:55.225559 11256 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:26:55.195555 11167 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:55.216136 11272 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:55.220212  5636 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
W20260709 12:26:55.232405  5602 leader_election.cc:336] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.503s (SENT)
I20260709 12:26:55.214038 11257 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:55.183591 10870 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:55.237424  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:55.195715 10972 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:26:55.247562  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:55.249352  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:55.251034  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:55.232470  5635 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:55.172310 11254 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.268275  5601 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:26:55.242602  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 224021us
W20260709 12:26:55.253563  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:55.253711  5633 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
W20260709 12:26:55.266045  5602 leader_election.cc:336] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.628s (SENT)
I20260709 12:26:55.270362  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:55.268841  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:55.271689  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:55.272603  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:55.238932 11272 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:55.282325  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:55.274580  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:55.271559  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:55.275957  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:55.207979 11226 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.278190  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:55.279002  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:55.288237  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:55.277673  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:55.296509  5710 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:55.196558 11134 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.303539 11230 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:26:55.290962  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:55.305192  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:55.270681  5452 catalog_manager.cc:5697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:55.305851  6278 raft_consensus.cc:2368] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17.
I20260709 12:26:55.306226 10871 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:55.306694  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:55.306393  5452 catalog_manager.cc:5697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:55.307883  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:55.246635 11257 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.291185  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:55.292903  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:55.293889  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 98684.4us
I20260709 12:26:55.298472  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:26:55.301426  5602 leader_election.cc:336] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.744s (SENT)
W20260709 12:26:55.301877 11200 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.974s	user 0.013s	sys 0.001s
W20260709 12:26:55.322261  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 54062.3us
I20260709 12:26:55.363663  5602 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:55.321866  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:55.337124  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:55.374634  6279 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 16 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 12:26:55.345383  5630 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:26:55.352582  5710 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:26:55.345830  5628 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:55.320585  5561 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:55.315140  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:55.313723  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:55.374543  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 83472.5us
I20260709 12:26:55.382017  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:55.324613  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53171.5us
W20260709 12:26:55.313243 11246 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.770s	user 0.000s	sys 0.003s
I20260709 12:26:55.268054 11255 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:55.384790  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:55.380266  6274 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:55.386155  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:55.128188 11229 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.363653 11266 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:55.411310 11275 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:26:55.411895 11275 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:55.412868 11275 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.397158 11256 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:55.416404 11269 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.145s
I20260709 12:26:55.416900 11266 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:55.417614 11269 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:55.418035 11269 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:55.418171 10870 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:55.419399 10870 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:55.418965 11269 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.417920 11266 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.421440 11254 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:55.207593 11198 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.424198  5710 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 12:26:55.426160 11233 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:26:55.426947 11233 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:55.428543 11233 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:55.387403 11165 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:55.410200 11219 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:26:55.383823 11156 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:26:55.442598  5708 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:26:55.442864 11233 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:55.353039 11235 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:55.338223 11209 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:55.291828 11218 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:26:55.290431 10462 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:26:55.381798 10848 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:55.315913 10783 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:55.462201  5707 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:55.469939  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:55.313066 11226 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:55.308266  5452 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 10 to 12, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 12 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:26:55.457617  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 61136.5us
W20260709 12:26:55.479955  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:55.481925 10783 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:55.483566  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:55.484207 11165 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:55.484714 11165 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:55.485256 11165 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:55.487870 11257 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:55.472179  5627 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:55.490630  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:55.492093  6272 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20260709 12:26:55.492575  6272 pending_rounds.cc:85] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 13
I20260709 12:26:55.492856  6272 pending_rounds.cc:107] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:26:55.493119  6272 raft_consensus.cc:2889] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:55.475804  5704 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:55.491230  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:55.502559  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 201150us
W20260709 12:26:55.503475  5602 leader_election.cc:336] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.806s (SENT)
I20260709 12:26:55.503963  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:55.510044  5601 leader_election.cc:400] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14.
I20260709 12:26:55.510572  5601 leader_election.cc:403] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:55.386801 11200 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:55.463173 10848 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:55.522276  5629 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: [13.9-15.10]
W20260709 12:26:55.527536  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 47886us
W20260709 12:26:55.529475  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:55.531174  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2608) took 1204 ms. Trace:
W20260709 12:26:55.531533  5602 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.875s (SENT)
I20260709 12:26:55.531987  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:55.531574  5701 rpcz_store.cc:276] 0709 12:26:54.327140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:54.327299 (+   159us) service_pool.cc:224] Handling call
0709 12:26:54.342833 (+ 15534us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:54.344490 (+  1657us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:54.344523 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:54.345017 (+   494us) log.cc:844] Serialized 26 byte log entry
0709 12:26:54.438836 (+ 93819us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:54.438864 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:54.439252 (+   388us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:55.244119 (+804867us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:55.530836 (+286717us) raft_consensus.cc:1685] finished
0709 12:26:55.530864 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:55.531081 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6664448,"thread_start_us":93412,"threads_started":1,"wal-append.queue_time_us":146500}
I20260709 12:26:55.517062 11226 raft_consensus.cc:2764] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election decision vote started in defunct term 13: won
I20260709 12:26:55.478513 11218 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.045s
I20260709 12:26:55.538936  5555 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:55.539772  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:55.540458  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 151619us
W20260709 12:26:55.544318  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:55.545548  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:55.542416  6272 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
I20260709 12:26:55.533461 11112 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14.
I20260709 12:26:55.530519 11255 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:26:55.556069  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 86238.3us
I20260709 12:26:55.475150  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:55.559562  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:55.561895  5630 raft_consensus.cc:606] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:55.562309  5630 raft_consensus.cc:614] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:26:55.561053  5602 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.994s (SENT) [suppressed 49 similar messages]
I20260709 12:26:55.549170 10848 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:26:55.575232  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 100537us
I20260709 12:26:55.555626 11257 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:26:55.582492  5602 leader_election.cc:336] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.897s (SENT)
I20260709 12:26:55.582993  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:55.566154 10902 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:26:55.592659  5524 leader_election.cc:400] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15.
I20260709 12:26:55.593465  5524 leader_election.cc:403] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:26:55.597432  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:55.561373 11202 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
W20260709 12:26:55.602643  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:55.609876 10848 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:55.610507 10848 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:55.620666  5452 catalog_manager.cc:5697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 7 to 9, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 9 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:55.622581  5452 catalog_manager.cc:5697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 10, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 10 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:26:55.623449 11200 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:55.574445 11272 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.612042 10848 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:55.626164 11272 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:55.627610  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:55.587713 11112 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:26:55.628381  5706 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:55.628984 11112 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:55.629439 11112 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:55.596299 11059 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:55.630429 11165 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:55.593948 10902 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:55.631479 10902 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:55.632678 11165 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:55.633419  5452 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:55.645833  6278 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: []
I20260709 12:26:55.647702  5706 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
W20260709 12:26:55.653537  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:55.656605  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72118.1us
I20260709 12:26:55.658609  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:55.659294  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:55.659689 10912 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:55.660086  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:55.661985 11149 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:26:55.662621 11149 consensus_queue.cc:1243] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:26:55.663846 11149 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:55.665606 11149 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:26:55.666334  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2484) took 1335 ms. Trace:
I20260709 12:26:55.666453 11149 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:55.666766  5555 rpcz_store.cc:276] 0709 12:26:54.330610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:54.330751 (+   141us) service_pool.cc:224] Handling call
0709 12:26:54.331032 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:55.541976 (+1210944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:55.665970 (+123994us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54003c41d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:55.666268 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2777504640}
I20260709 12:26:55.672906  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:55.683599  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:55.684324  5556 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:26:55.684679  5557 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:55.685802  5557 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:55.686204  5558 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:55.683674  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:55.687273  5559 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:55.630509 11112 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:55.694319 11112 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:55.668059 11149 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:55.697299  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "733a5b54e4dd4f86999821705033529f"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:55.697829  5559 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:26:55.699994  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2611) took 1314 ms. Trace:
I20260709 12:26:55.700704  5629 rpcz_store.cc:276] 0709 12:26:54.385707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:54.433382 (+ 47675us) service_pool.cc:224] Handling call
0709 12:26:54.464799 (+ 31417us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:55.522764 (+1057965us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:55.699718 (+176954us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54001f2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:55.699924 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2431121664}
I20260709 12:26:55.703130  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:55.697997  5630 raft_consensus.cc:606] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:55.705816  5630 raft_consensus.cc:614] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:55.708491  6279 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:55.701658  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:55.710175  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:26:55.720988  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 46101.3us
I20260709 12:26:55.723080  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:55.723867  5559 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:55.724865 11225 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:26:55.724972  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:55.725646  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:55.726375  5557 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:55.726575  5556 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:55.725835 11225 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:55.735383  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:55.738170  6278 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: []
I20260709 12:26:55.740427  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:55.743475  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:55.745783  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:55.775307  5631 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: []
I20260709 12:26:55.777532  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:55.780342  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 180144us
I20260709 12:26:55.781286  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:55.782053  5552 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:55.782929  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:55.778409  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:55.783671  5706 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:55.735577 11134 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:55.795681  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:55.795518  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:55.798339  5703 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:26:55.818919  5710 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 19
W20260709 12:26:55.798585  5602 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.896s (SENT)
I20260709 12:26:55.739702 11233 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:55.932257  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:55.932293  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 73654.7us
I20260709 12:26:55.896926  5552 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:26:55.933568  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:55.748546 11226 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:55.820878  5630 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
W20260709 12:26:55.938594  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:55.894899  5706 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:26:55.940516  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:55.945417  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 165100us
I20260709 12:26:55.946872  5526 leader_election.cc:400] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17.
I20260709 12:26:55.947559  5526 leader_election.cc:403] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 12:26:55.951380  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:55.941949  5710 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:26:55.942924  5554 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 15. Current term is 16. Ops: []
I20260709 12:26:55.944164  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:55.953914  5704 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:55.970995  5704 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:55.974998  5710 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:26:55.797010 11219 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:55.979931  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:55.981139  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:55.978884 11223 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:55.982457  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 183911us
I20260709 12:26:55.986656 11223 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:55.988806 11223 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:55.988893  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:55.989347  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:55.990232  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:55.990618  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:55.991242  5629 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:55.992434  5636 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: []
W20260709 12:26:55.950356 11280 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:26:55.995513  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:55.996025  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:55.996771  5553 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:56.003953  5556 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:56.004956  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.006098  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.011689  5556 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:26:55.950269 11233 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:55.968598 10912 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15.
W20260709 12:26:56.018435  5602 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.939s (SENT)
I20260709 12:26:56.015553  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:56.018275  5558 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:56.022775  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.018954  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:56.048763 10848 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:26:56.050601  5602 leader_election.cc:336] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.945s (SENT)
I20260709 12:26:56.051077  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:56.051716 11209 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:56.051823 10848 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:56.050702  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:56.052472 11233 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:56.054414  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:55.982812 11242 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.072s
I20260709 12:26:55.986200 11218 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:56.042616 11256 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 12:26:55.982650 10783 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:56.056115  5629 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:26:56.077512  5555 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:55.978693 11221 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:56.057149  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:56.043092 11257 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.013s
I20260709 12:26:56.073837  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.018878 11149 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17.
I20260709 12:26:56.091382  5629 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:26:55.964228 11225 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:56.088414 11288 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:56.103394 11288 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:56.103775 11288 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:56.109057  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.109943  5560 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:26:56.110818  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.111577  5560 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 13: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:56.113161  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1005) took 1257 ms. Trace:
I20260709 12:26:56.116029  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:56.085008 11220 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.938s	user 0.018s	sys 0.004s
I20260709 12:26:56.116506  6276 rpcz_store.cc:276] 0709 12:26:54.856020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:54.868463 (+ 12443us) service_pool.cc:224] Handling call
0709 12:26:56.112945 (+1244482us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b54005910d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:56.113070 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":625996672}
I20260709 12:26:56.074415 11209 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:26:56.127461  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 180763us
W20260709 12:26:56.112336 11300 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.071s	user 0.002s	sys 0.001s
I20260709 12:26:56.084673 11221 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:26:56.144429  5602 leader_election.cc:336] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.500s (SENT)
W20260709 12:26:56.145169  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 175936us
I20260709 12:26:56.077987 11272 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 12:26:56.159584  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:26:56.129088 11220 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.982s	user 0.018s	sys 0.004s
I20260709 12:26:56.161000  5553 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:56.163959  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.167959  5705 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:56.174472 11206 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:26:56.180927  5559 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: []
W20260709 12:26:56.197810  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 202825us
I20260709 12:26:56.078755 11241 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:56.204625  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2494) took 1061 ms. Trace:
I20260709 12:26:56.205245  5561 rpcz_store.cc:276] 0709 12:26:55.143345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.157634 (+ 14289us) service_pool.cc:224] Handling call
0709 12:26:55.165892 (+  8258us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:55.379276 (+213384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:56.204266 (+824990us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b54000ea6d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:56.204544 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":355193600}
I20260709 12:26:56.201913  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:56.216061  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.211205  6237 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:56.225404  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.222766  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.161005 11253 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:26:56.242848  5553 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:56.244697 11267 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:26:56.246644 11233 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:26:56.249747 10902 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.617s	user 0.006s	sys 0.007s
I20260709 12:26:56.250382 10902 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.251936  5636 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:56.255013 10902 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:56.269393 11221 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.271891 11221 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:26:56.273330 11305 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:26:56.319075  5601 leader_election.cc:336] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.745s (SENT)
I20260709 12:26:56.302795  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.223634 11311 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.090s
I20260709 12:26:56.224181 11272 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.086s
I20260709 12:26:56.306644  5633 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:56.306879 11267 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:26:56.307659  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:56.315299 11303 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.300s	user 0.001s	sys 0.001s
I20260709 12:26:56.319753 11229 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:56.320459 11288 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:56.241115  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:56.322840  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:56.323302  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 255169us
I20260709 12:26:56.325690  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:56.337510  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 100929us
I20260709 12:26:56.338677  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.339840  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.339948  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:56.340426 11253 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:26:56.340571  6273 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:26:56.340952  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:56.341461  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:56.341797  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:56.342607 10902 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:26:56.332836 10783 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:56.208824 11257 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.114s
I20260709 12:26:56.336829  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.322445  5561 raft_consensus.cc:2376] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15. Re-sending same reply.
W20260709 12:26:56.343727  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 141893us
I20260709 12:26:56.342856  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:56.346141  6279 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:56.346181  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.347002  5553 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:56.347751  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.349216  5553 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:26:56.349979  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.350433 11272 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:56.350801  5553 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:26:56.351897  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.352058  5553 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:26:56.353679  5557 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 13 index: 6. (index mismatch)
I20260709 12:26:56.352973 10783 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.354770  5557 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:56.352507  5561 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:56.357721  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:56.348529 11203 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.860s	user 0.000s	sys 0.005s
I20260709 12:26:56.358494  5553 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 13: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:26:56.356750  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:56.361624  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:56.364740  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.365895  5560 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:56.367105  6273 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:26:56.367841  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.368865  5561 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:26:56.368739  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:56.368954  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:56.369971 11099 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:26:56.370362 11320 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:26:56.370676 11288 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:56.372450 11209 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.372951 11209 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:56.340968 11253 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:56.375159 11209 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:56.370522 11099 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:56.375545 11253 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.376231 11099 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.378453  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.378706 11320 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:26:56.379765  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.350843 11272 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:56.381328  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.381834  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:56.382059  5555 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:26:56.382695 11233 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:56.383054  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.384969  5555 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:56.383709 11149 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:26:56.389135 11243 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.390033 11243 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:56.396548  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:56.398079 11311 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.398705 11311 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.399581  6271 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:56.400029  6271 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:56.401047  6271 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.385288 11235 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:26:56.404889 11235 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:56.405897 11235 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:56.385910 10912 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:26:56.409886 10912 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:26:56.410250 10912 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:56.410390  6272 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:26:56.416705  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.416944  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:56.417918 11149 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:26:56.418179  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:56.418382 11149 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:56.418777  5630 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:26:56.420250  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.421476  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:56.424162  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1055) took 1038 ms. Trace:
I20260709 12:26:56.425269  5628 rpcz_store.cc:276] 0709 12:26:55.386143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.386341 (+   198us) service_pool.cc:224] Handling call
0709 12:26:55.386913 (+   572us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:55.456657 (+ 69744us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:55.456709 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:55.456775 (+    66us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:55.456828 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:56.390689 (+933861us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b540018d154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:56.390733 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:56.424101 (+ 33368us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":159554944}
W20260709 12:26:56.432325 11298 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.391s	user 0.001s	sys 0.001s
I20260709 12:26:56.434046 11219 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:56.434193  5557 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:56.434978 11219 consensus_queue.cc:1243] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 16.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:26:56.435796 11200 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:26:56.436256 11200 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:56.437248 11200 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:56.439291  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 228169us
I20260709 12:26:56.383477 11233 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:26:56.441322  6271 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:56.386374  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.429106 11320 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:56.447800 11243 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:56.444736 11209 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:56.445088 11227 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:56.444772 11223 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:26:56.459777  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:56.428746  5560 raft_consensus.cc:2376] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13. Re-sending same reply.
I20260709 12:26:56.460963  5703 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 12:26:56.450222  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:26:56.456312  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:56.458640  5630 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:56.458561  6279 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:26:56.466950  6278 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 10. Current term is 11. Ops: []
I20260709 12:26:56.467059  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90c64867bae6493c846d941815aa1074"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:26:56.470464  5630 raft_consensus.cc:606] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:56.474094  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:56.471726  5630 raft_consensus.cc:614] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:56.482544  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.483251  5555 raft_consensus.cc:2376] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15. Re-sending same reply.
I20260709 12:26:56.486697  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.487903  5560 raft_consensus.cc:2376] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15. Re-sending same reply.
I20260709 12:26:56.489125  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.490363  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.491154 11219 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.490370  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.508741  5555 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:56.491739 11223 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:26:56.492763  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:56.505807  5560 raft_consensus.cc:2376] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15. Re-sending same reply.
I20260709 12:26:56.517043  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:56.493794 11149 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.495218 11292 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:26:56.495522  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.499239  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.501010 11320 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:26:56.487175  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:56.449399 11235 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:56.448766 11209 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:56.509972  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:56.470680 11262 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:26:56.517915  5632 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:56.520231  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:56.524782  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:56.530048  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:56.533923  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:56.511580 11223 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:56.545300  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.525290 11257 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:26:56.510195 11267 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 12:26:56.527750 11320 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:56.530824 11292 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:56.576952  5555 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
W20260709 12:26:56.565246  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:56.512820 11219 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:56.583495  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.584378  5555 raft_consensus.cc:2376] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13. Re-sending same reply.
I20260709 12:26:56.543098 11149 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:56.545312 10783 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:56.521955 11288 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 12:26:56.544917 10931 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:56.583127 11320 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:56.596207  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.613191 11292 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.633302  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:56.643504  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.644462  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:56.647264  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:56.612107 11311 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:56.631054 10783 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:56.654270  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.658867  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.662336  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.663106  5555 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:26:56.672209  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.651475 10931 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:56.674304  5555 raft_consensus.cc:2376] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15. Re-sending same reply.
W20260709 12:26:56.663106  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:56.657934 10783 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:56.686108  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.546376 10912 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.703732  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.701905  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.707821  5555 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:56.673282 11220 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:56.682416 11288 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:56.716993  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:56.719622  5555 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:26:56.720189  5552 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
W20260709 12:26:56.721745  5602 leader_election.cc:336] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.195s (SENT)
I20260709 12:26:56.729732  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:56.738561  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:26:56.740713 11330 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:26:56.740859  5554 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:56.742053  5554 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 13. Ops: []
I20260709 12:26:56.739133 11257 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.744263 11257 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:56.745018 11167 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260709 12:26:56.746464 11257 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:56.747404 11276 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:56.747828 11276 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:56.737463 11243 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:56.740005  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 155034us
I20260709 12:26:56.740422  5555 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
I20260709 12:26:56.748533 11243 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.749481 11202 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:26:56.748903 11276 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.681255 11223 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:56.778199  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:56.886927  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:56.888649  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:56.890539  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:56.884300  5703 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:26:56.891368  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:56.801290  5552 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
W20260709 12:26:56.784323  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:56.902035  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:56.904089  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:56.886891  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 171868us
W20260709 12:26:56.913414  5602 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.591s (SENT)
W20260709 12:26:56.909921  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 242753us
I20260709 12:26:56.916857  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:56.924652  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:56.925400  5633 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:56.926466  5557 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:26:56.730223 11288 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:56.903317 11198 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.939038  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:56.904834 11257 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:56.723062 11220 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.947163  5558 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:26:56.954133  5557 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:26:56.956043  5558 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:26:56.965098  5558 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:26:56.965948  5555 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:26:56.941839 11219 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:56.948130 11149 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.971027  5558 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 12:26:56.970177 10870 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:56.972770 11288 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:56.941300 11198 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.976531  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:56.977671 11220 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:56.796936 10931 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:56.980290 11229 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.635s	user 0.010s	sys 0.000s
I20260709 12:26:56.980101 10783 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:56.983381  5558 pending_rounds.cc:85] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 7
W20260709 12:26:56.983737  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 93012us
I20260709 12:26:57.017482  5558 pending_rounds.cc:107] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 12:26:56.970240 11336 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.017915  5558 raft_consensus.cc:2889] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:57.020318  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 81675.9us
W20260709 12:26:57.023241  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 84309.3us
I20260709 12:26:57.018793  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:57.025537  5554 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:26:56.958864 11257 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:56.981864 11344 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:26:57.060871  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.034658  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:57.004129 10870 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.025275 11202 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:26:56.984473 11099 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
W20260709 12:26:57.088609  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:56.999255 11149 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:57.088616 11332 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:26:57.027243 11218 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:57.074911 11221 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:26:57.074594 11243 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:57.028050 11336 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:57.134943 11200 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:57.136040 11350 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:57.137251 10783 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:57.137521 10870 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:57.028205 11343 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
I20260709 12:26:57.099696 11351 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:57.150995 11336 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:57.109459 11202 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:26:57.152936 11342 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.171s	user 0.000s	sys 0.001s
I20260709 12:26:57.109249 11292 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
W20260709 12:26:57.156610  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 124976us
I20260709 12:26:57.228744  5706 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:26:57.266148  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.247805  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:57.240272 11235 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.709s	user 0.008s	sys 0.004s
W20260709 12:26:57.245189 11339 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.281s	user 0.000s	sys 0.001s
I20260709 12:26:57.278271 11351 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:57.186614 11243 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:57.290647  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.295892  5554 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
W20260709 12:26:57.291611  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:57.292114  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.220459 11225 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:57.231624 11257 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:57.163296 11288 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:57.240916 11362 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:57.269191 11229 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.237599 11149 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:57.315533 11257 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:57.317878 11202 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:57.302663  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 274188us
I20260709 12:26:57.320269 11318 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:26:57.294287 11320 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.643s	user 0.006s	sys 0.006s
W20260709 12:26:57.323881  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.197279 11350 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:57.143720 11218 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.333326  6277 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:26:57.382143  5701 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:26:57.384872  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:57.385648  5709 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:57.320549 11360 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.388576 10972 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.320634 11225 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.385092 11320 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.380033 11318 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:26:57.375705 11198 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:57.418766  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2330) took 1402 ms. Trace:
I20260709 12:26:57.333737 11356 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:57.402348  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:57.426807  5701 raft_consensus.cc:2368] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13.
I20260709 12:26:57.311196 11243 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:57.306313 11361 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.101s
I20260709 12:26:57.424710  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:57.432770  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 106622us
I20260709 12:26:57.320811 11351 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:57.435977  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:57.330966 11362 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:57.442654  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:57.332296 11229 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.329456 11288 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.444116  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:57.447021  5704 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:26:57.415952 11344 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:26:57.419426  5556 rpcz_store.cc:276] 0709 12:26:56.016817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:56.017051 (+   234us) service_pool.cc:224] Handling call
0709 12:26:56.017419 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:56.036904 (+ 19485us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:26:56.170998 (+134094us) log.cc:844] Serialized 14 byte log entry
0709 12:26:57.137937 (+966939us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b18000a50a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a
0709 12:26:57.138355 (+   418us) log.cc:844] Serialized 14 byte log entry
0709 12:26:57.318685 (+180330us) spinlock_profiling.cc:243] Waited 8.87 ms on lock 0x7b18000a50a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:26:57.418249 (+ 99564us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:57.418330 (+    81us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:57.418411 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:57.418494 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:57.418673 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":773,"prepare.queue_time_us":144756,"raft.queue_time_us":26459,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":17626,"spinlock_wait_cycles":29345536,"thread_start_us":422857,"threads_started":3,"wal-append.queue_time_us":9452}
I20260709 12:26:57.380050 11257 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:57.451664  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:57.462710  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.431021 11149 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:57.462919  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:57.461712  5556 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:57.400769 10972 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.457345  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:57.474998  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:57.416328 11360 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:57.386538 11272 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.481513  5628 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:26:57.480329  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:57.481832  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:57.480363  5554 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:57.521173  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:57.459713 11366 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:57.523954  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:57.525708  5554 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:26:57.535323  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.535661 11225 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:57.527947 11351 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:57.536679  5634 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:26:57.536720 11353 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:26:57.537263 11324 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:26:57.537683 11324 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:57.538566 11324 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:57.539495 11364 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:26:57.539927 11324 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:57.544163 11198 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:26:57.547196  5552 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:26:57.547618  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:57.548943 11198 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:26:57.549502 11198 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:57.549899 11198 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:57.550205 11243 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:57.521807 11318 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.553606 10870 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:57.553789 11366 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:57.554183 10870 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:57.555981 11366 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:57.527769 11369 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:57.433365 11356 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.553082 11318 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:26:57.568567  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.556708  5633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9ff38a877a89473392a3e27657516383"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
W20260709 12:26:57.589599  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.589630  5633 raft_consensus.cc:606] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:26:57.590272  5633 raft_consensus.cc:614] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:57.592650  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:26:57.594821  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:57.595702  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:26:57.597220  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:57.596146  5452 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.176s	user 0.003s	sys 0.026s
W20260709 12:26:57.598532  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:57.535699 11326 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.200s	user 0.005s	sys 0.000s
I20260709 12:26:57.525663 10902 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:57.477728 11149 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.626947  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:26:57.533282 11359 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:26:57.634094  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 193280us
I20260709 12:26:57.562012 11288 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:57.563177 11320 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:57.556262 10870 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:57.587347 10462 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:57.556717 11350 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:57.639989  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:57.622088  5452 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.203s	user 0.004s	sys 0.026s
W20260709 12:26:57.651372  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:57.656850  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 212822us
I20260709 12:26:57.645807 11366 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:26:57.604811 11266 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:26:57.671166  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:57.649490 11243 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.587831 11369 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:26:57.667021  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 114040us
I20260709 12:26:57.559971 11344 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:26:57.700706  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.632786 10902 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.709594  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2340) took 1370 ms. Trace:
I20260709 12:26:57.714962  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:57.714138  5635 rpcz_store.cc:276] 0709 12:26:56.339044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:56.343733 (+  4689us) service_pool.cc:224] Handling call
0709 12:26:56.349552 (+  5819us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:56.387544 (+ 37992us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:56.387599 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:56.387638 (+    39us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:56.387676 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:57.709331 (+1321655us) spinlock_profiling.cc:243] Waited 31 ms on lock 0x7b5400241654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:57.709389 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:57.709526 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":71272320}
I20260709 12:26:57.721098  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:57.703686 11229 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:57.725898  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55741.4us
I20260709 12:26:57.726817  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:57.727643  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:57.713408 11218 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:57.661276 11359 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.060s
I20260709 12:26:57.721155 11362 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:57.743708  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:57.709000 11243 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.744838  5701 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:57.763123  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:57.765417 11360 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:26:57.766105 11200 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.611s	user 0.007s	sys 0.006s
I20260709 12:26:57.767524  5560 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:26:57.757843 11381 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:57.771636  5704 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:57.772591 11218 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:26:57.746539 11269 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:57.742951 11379 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 12:26:57.783260 11385 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:26:57.785120 11385 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:26:57.753144 11257 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:57.787024 11343 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:26:57.788731 11356 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:57.791350  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:57.758097 11375 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:26:57.794241  5556 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:57.742677 11374 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:57.799213 11374 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:57.799876 11374 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:26:57.802335 11381 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:26:57.805079  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:57.810655  5704 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:57.800926 11343 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20260709 12:26:57.817749 11241 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:57.818069  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2376) took 1429 ms. Trace:
I20260709 12:26:57.818459  5557 rpcz_store.cc:276] 0709 12:26:56.388828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:56.927566 (+538738us) service_pool.cc:224] Handling call
0709 12:26:56.953762 (+ 26196us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:56.954799 (+  1037us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:57.817825 (+863026us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b54003c4bd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:57.817987 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25789184}
I20260709 12:26:57.805958 11356 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:26:57.821126 11356 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:57.821578 11356 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:26:57.823208  6280 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 997) took 2930 ms (client timeout 1691 ms). Trace:
W20260709 12:26:57.823724  6280 rpcz_store.cc:269] 0709 12:26:54.892670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:54.893686 (+  1016us) service_pool.cc:224] Handling call
0709 12:26:57.822914 (+2929228us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b54002e3954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:57.823087 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4555218176}
W20260709 12:26:57.828748  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 124736us
I20260709 12:26:57.829315  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:57.830801  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:26:57.834779  5557 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:26:57.836378  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:57.836653  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:57.841150  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.844861  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:57.845570  5704 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:26:57.811075 11149 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:57.879783  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:57.880786  5526 leader_election.cc:336] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.716s (SENT)
I20260709 12:26:57.855782  5555 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 12:26:57.881547  5555 pending_rounds.cc:85] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 8
I20260709 12:26:57.881803  5555 pending_rounds.cc:107] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 12:26:57.882019  5555 raft_consensus.cc:2889] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:57.882314  5555 pending_rounds.cc:107] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 15.10
I20260709 12:26:57.882532  5555 raft_consensus.cc:2889] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:57.843513 11362 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:26:57.860445  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.843211 10931 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:57.868018 11275 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.887224 11225 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.891892 11275 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.891463  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:57.889271  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:57.894805  5526 connection.cc:256] RPC call timeout handler was delayed by -1.1592948359998445s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15280000000000002s for a response
I20260709 12:26:57.871776  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:57.876873 11356 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:57.877774 11369 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:57.879310  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:57.825798 11198 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.853447  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2388) took 1446 ms. Trace:
I20260709 12:26:57.880978  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:57.907698  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.907755  5560 rpcz_store.cc:276] 0709 12:26:56.406981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:56.513434 (+106453us) service_pool.cc:224] Handling call
0709 12:26:57.853177 (+1339743us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54000eb5d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:57.853370 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2839627008}
I20260709 12:26:57.848260  5524 leader_election.cc:400] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13.
I20260709 12:26:57.910028  5524 leader_election.cc:403] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
W20260709 12:26:57.885481  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.887204 11368 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.012s
I20260709 12:26:57.869740 11243 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:57.892598  5635 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: []
I20260709 12:26:57.898226  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:57.899791  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:57.900707  5629 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:26:57.901249 11350 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.902365 11356 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:57.903353  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.925697  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.904708  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.905619  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.899621 11378 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:57.879781  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.915560  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.915483  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.939687  5636 raft_consensus.cc:2410] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 17 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
I20260709 12:26:57.887773 10931 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:57.906243 11099 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.945050  5704 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:57.928577  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:57.945523  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.946592  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.960942  5630 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:57.948200  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.948295  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.916471 10870 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:57.892283 11225 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.937958 11227 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13.
I20260709 12:26:57.964998  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.918740 11362 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:26:57.973945  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.974243  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.980904  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:57.980207  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55723.8us
I20260709 12:26:57.983074  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.983944 11227 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:26:57.984859 11198 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:57.986722 10902 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:57.987622  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:57.988263 11149 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:26:57.988714 11149 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:57.989890  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:57.990625 10972 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:57.947902 10931 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:57.938409 11343 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:57.940008 11350 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:57.947775 11378 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:57.998044  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:58.639540  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:58.038929  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:58.658174  5703 raft_consensus.cc:2410] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:26:58.603143  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1077) took 1088 ms. Trace:
I20260709 12:26:58.276947  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2650) took 1015 ms. Trace:
I20260709 12:26:58.658742  5628 rpcz_store.cc:276] 0709 12:26:57.514965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:57.526467 (+ 11502us) service_pool.cc:224] Handling call
0709 12:26:57.527045 (+   578us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:57.632481 (+105436us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:57.632544 (+    63us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:57.632590 (+    46us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:57.632631 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.602742 (+970111us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b540043a154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:58.602815 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:58.603049 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":241904896}
I20260709 12:26:58.613634  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2652) took 1334 ms. Trace:
I20260709 12:26:58.659456  5706 rpcz_store.cc:276] 0709 12:26:57.261711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:57.261995 (+   284us) service_pool.cc:224] Handling call
0709 12:26:57.262590 (+   595us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:57.425091 (+162501us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:26:57.445156 (+ 20065us) log.cc:844] Serialized 14 byte log entry
0709 12:26:57.641987 (+196831us) log.cc:844] Serialized 14 byte log entry
0709 12:26:57.662013 (+ 20026us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:57.662159 (+   146us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:57.662211 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.276539 (+614328us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b5400451fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:58.276609 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:58.276845 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19696,"prepare.queue_time_us":19567,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":2204,"raft.queue_time_us":25759,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":6759,"spinlock_wait_cycles":363711232,"thread_start_us":189702,"threads_started":1,"wal-append.queue_time_us":270312}
I20260709 12:26:58.604729  5702 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:58.041769  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2502) took 2043 ms. Trace:
I20260709 12:26:58.662846  5559 rpcz_store.cc:276] 0709 12:26:55.998221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.999214 (+   993us) service_pool.cc:224] Handling call
0709 12:26:56.015596 (+ 16382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:56.181405 (+165809us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.041540 (+1860135us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b5400025dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:58.041711 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":379744640}
W20260709 12:26:58.380452  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:57.998374  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:26:58.664911  5553 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.001s	sys 0.031s
I20260709 12:26:57.998188  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:26:58.665540  5553 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.001s	sys 0.031s
I20260709 12:26:58.208487  6276 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:26:58.037118 11209 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:58.023128 11149 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:58.148463  6278 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:26:58.089668 11374 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:58.669193 11149 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:58.669894  5633 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 12. Current term is 13. Ops: []
I20260709 12:26:58.274631  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:58.671042 11374 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:58.206146  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:58.351480  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:58.041196  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1020) took 1582 ms. Trace:
W20260709 12:26:58.197793 11307 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 2.095s	user 0.005s	sys 0.000s
I20260709 12:26:57.975028 11099 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:58.675866  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2628) took 3194 ms. Trace:
I20260709 12:26:57.977447 10870 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:58.676239  5708 rpcz_store.cc:276] 0709 12:26:55.481210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.488427 (+  7217us) service_pool.cc:224] Handling call
0709 12:26:55.488771 (+   344us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:55.489155 (+   384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:55.489193 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:55.489844 (+   651us) log.cc:844] Serialized 52 byte log entry
0709 12:26:55.730543 (+240699us) spinlock_profiling.cc:243] Waited 6.4 ms on lock 0x7b18000a2ca0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:26:56.057174 (+326631us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:56.089886 (+ 32712us) log.cc:844] Serialized 14 byte log entry
0709 12:26:56.228635 (+138749us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:56.228747 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:58.197010 (+1968263us) spinlock_profiling.cc:243] Waited 6 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:26:58.675668 (+478658us) raft_consensus.cc:1685] finished
0709 12:26:58.675689 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:58.675813 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3100,"prepare.queue_time_us":33740,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":17613,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":151,"spinlock_wait_cycles":28517888,"thread_start_us":464724,"threads_started":2,"wal-append.queue_time_us":606381}
I20260709 12:26:58.678793  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:58.679383  5708 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:58.679546 10870 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:58.347604  5630 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:58.680303  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:58.112484 11400 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.089s	user 0.004s	sys 0.000s
W20260709 12:26:58.029964  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:58.681273  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2457) took 1155 ms. Trace:
W20260709 12:26:58.681777  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:58.249831  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:26:58.301777  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:58.335879  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:58.085731 11398 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.112s	user 0.000s	sys 0.005s
W20260709 12:26:58.121035 11401 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.100s	user 0.000s	sys 0.007s
I20260709 12:26:58.672338  6273 rpcz_store.cc:276] 0709 12:26:56.459181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:56.459305 (+   124us) service_pool.cc:224] Handling call
0709 12:26:56.478059 (+ 18754us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:56.507121 (+ 29062us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:56.507170 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:56.507785 (+   615us) log.cc:844] Serialized 26 byte log entry
0709 12:26:57.563706 (+1055921us) spinlock_profiling.cc:243] Waited 1.35 ms on lock 0x7b1800380e60. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:26:57.701547 (+137841us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:57.701618 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.040741 (+339123us) spinlock_profiling.cc:243] Waited 7.46 ms on lock 0x7b54004f9cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:58.040812 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:58.040878 (+    66us) raft_consensus.cc:1685] finished
0709 12:26:58.040899 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:58.041091 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26724,"raft.queue_time_us":24413,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":9990,"spinlock_wait_cycles":20271744,"thread_start_us":137552,"threads_started":1,"wal-append.queue_time_us":1263309}
I20260709 12:26:57.968545 11365 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.123777  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 945) took 2846 ms. Trace:
I20260709 12:26:58.137966  5701 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:58.318792  6275 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
W20260709 12:26:58.126448  6277 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1004) took 2876 ms (client timeout 1716 ms). Trace:
I20260709 12:26:58.710145  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:58.708874  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 862770us
I20260709 12:26:58.713871  5703 raft_consensus.cc:2410] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 12:26:58.714614  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2464) took 1055 ms. Trace:
I20260709 12:26:58.038932  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:58.716248  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 677357us
I20260709 12:26:58.687014  5627 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:58.282279  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:58.721097  6279 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:58.661064  5710 rpcz_store.cc:276] 0709 12:26:57.280095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:57.289053 (+  8958us) service_pool.cc:224] Handling call
0709 12:26:57.289434 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:57.468537 (+179103us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:57.468571 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:57.469139 (+   568us) log.cc:844] Serialized 26 byte log entry
0709 12:26:57.662681 (+193542us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:57.662737 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.603629 (+940892us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b540026fa54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:58.603692 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:58.603773 (+    81us) raft_consensus.cc:1685] finished
0709 12:26:58.603793 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:58.613544 (+  9751us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20925,"raft.queue_time_us":19920,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":11370,"spinlock_wait_cycles":366967552,"thread_start_us":135178,"threads_started":1,"wal-append.queue_time_us":305652}
W20260709 12:26:58.713198  6277 rpcz_store.cc:269] 0709 12:26:55.250501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.272323 (+ 21822us) service_pool.cc:224] Handling call
0709 12:26:58.126159 (+2853836us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b54002a8354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:26:58.126341 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4032862080}
W20260709 12:26:58.163394 11253 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.757s	user 0.007s	sys 0.005s
W20260709 12:26:58.689673  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:58.682760  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:58.691665  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:58.691708  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:26:58.492125  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:58.694768  5706 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: [16.10-18.11]
I20260709 12:26:58.734184  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2459) took 1176 ms. Trace:
I20260709 12:26:58.681681  5708 rpcz_store.cc:276] 0709 12:26:57.525352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.680057 (+1154705us) service_pool.cc:224] Handling call
0709 12:26:58.681166 (+  1109us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:58.722659  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:58.737789  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:58.738847  5710 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:26:58.117936 11371 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.715096  5703 rpcz_store.cc:276] 0709 12:26:57.658902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.709894 (+1050992us) service_pool.cc:224] Handling call
0709 12:26:58.714536 (+  4642us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:58.685415 11404 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 12:26:58.747855  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:58.679584 11275 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:58.727021  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:26:58.756897  5631 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:26:58.118908 11327 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.784s	user 0.005s	sys 0.003s
I20260709 12:26:58.674136  5704 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:26:58.730851 11253 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.338s	user 0.010s	sys 0.024s
I20260709 12:26:58.757687  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:57.998667 10902 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:58.016072 11198 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.004885 11356 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:58.208530 11325 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:58.676527 11099 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:58.021983 11225 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:58.712085  5705 rpcz_store.cc:276] 0709 12:26:55.277613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:55.277818 (+   205us) service_pool.cc:224] Handling call
0709 12:26:55.326875 (+ 49057us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:56.168538 (+841663us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.123330 (+1954792us) spinlock_profiling.cc:243] Waited 841 ms on lock 0x7b5400067054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:58.123656 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1933608832}
I20260709 12:26:58.705528 10783 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:58.117836 11318 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:58.207402 11324 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:58.726738 11272 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:58.022892 11362 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:58.734613  5706 rpcz_store.cc:276] 0709 12:26:57.558269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.694087 (+1135818us) service_pool.cc:224] Handling call
0709 12:26:58.694495 (+   408us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:58.733803 (+ 39308us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.734110 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:58.762878  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:58.735111 11312 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:26:58.745355 10870 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:58.777518  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63536.7us
I20260709 12:26:58.798497  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:58.800591 11099 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:58.801189 11099 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:58.801690 11099 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:58.711961 11343 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:58.729252 11365 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:58.793265  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1023) took 1499 ms. Trace:
I20260709 12:26:58.804275  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:58.759307  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:58.755779 11371 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:58.765190 11275 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 12:26:58.784813 11226 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:26:58.817550  6275 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:58.811292  5631 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:26:58.815353  5553 rpcz_store.cc:276] 0709 12:26:57.294278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:57.294419 (+   141us) service_pool.cc:224] Handling call
0709 12:26:57.312765 (+ 18346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:57.313118 (+   353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:57.313147 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:57.315103 (+  1956us) log.cc:844] Serialized 26 byte log entry
0709 12:26:58.029996 (+714893us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b1800355060. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:26:58.666636 (+636640us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:58.708136 (+ 41500us) log.cc:844] Serialized 14 byte log entry
0709 12:26:58.708430 (+   294us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.708500 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:58.786474 (+ 77974us) raft_consensus.cc:1685] finished
0709 12:26:58.793032 (+  6558us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:58.793201 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":60217,"raft.queue_time_us":4543,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":17425,"spinlock_wait_cycles":45460992,"thread_start_us":636303,"threads_started":1,"wal-append.queue_time_us":1388174}
I20260709 12:26:58.812667  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2656) took 1175 ms. Trace:
I20260709 12:26:58.828896  5705 rpcz_store.cc:276] 0709 12:26:57.637723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.788699 (+1150976us) service_pool.cc:224] Handling call
0709 12:26:58.811913 (+ 23214us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:58.812171 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:58.812211 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:58.812271 (+    60us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:58.812319 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.812411 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:58.812587 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:58.770561 11198 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:58.774217 11356 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:58.803221 11382 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:58.809824 10870 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:58.840941 11321 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:58.776245 11325 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:58.786713 10783 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:58.862543 11343 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:26:58.863690  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:58.864673  5710 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: [15.12-17.13]
I20260709 12:26:58.851747  5708 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: []
I20260709 12:26:58.865859 11350 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:58.866338  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:58.866824 11350 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:58.867532 11350 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:58.860003 10870 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:58.826107 11226 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:58.860522 11321 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:58.862709 11325 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:58.864588  6275 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:58.790068 11318 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:58.792760 11324 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:58.790413 11225 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 12:26:58.771169 10902 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:58.870729 11378 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:58.871872  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2400) took 1044 ms. Trace:
I20260709 12:26:58.867661  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:58.857937 11371 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:58.884148  5705 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:26:58.940794  6274 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:26:58.942734  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2659) took 1235 ms. Trace:
I20260709 12:26:58.943280  5705 rpcz_store.cc:276] 0709 12:26:57.707562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.852701 (+1145139us) service_pool.cc:224] Handling call
0709 12:26:58.853086 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:58.942293 (+ 89207us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.942652 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:58.945145  5705 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:26:58.946197  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2660) took 1237 ms. Trace:
I20260709 12:26:58.946787  5705 rpcz_store.cc:276] 0709 12:26:57.708626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.944310 (+1235684us) service_pool.cc:224] Handling call
0709 12:26:58.944708 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:58.945571 (+   863us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.946079 (+   508us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:58.942746  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:58.949780  6278 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:26:58.893247  6277 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:26:58.864192 10783 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:58.888760 11408 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:58.952483  5705 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: [11.7-13.8]
I20260709 12:26:58.956230  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2468) took 1231 ms. Trace:
I20260709 12:26:58.956874  5705 rpcz_store.cc:276] 0709 12:26:57.724699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.947872 (+1223173us) service_pool.cc:224] Handling call
0709 12:26:58.952021 (+  4149us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:58.955669 (+  3648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.956140 (+   471us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:58.938622  5561 rpcz_store.cc:276] 0709 12:26:57.827947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:57.830687 (+  2740us) service_pool.cc:224] Handling call
0709 12:26:57.840881 (+ 10194us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:57.841689 (+   808us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:57.841748 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:57.843290 (+  1542us) log.cc:844] Serialized 26 byte log entry
0709 12:26:57.917638 (+ 74348us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:57.941867 (+ 24229us) log.cc:844] Serialized 14 byte log entry
0709 12:26:57.975843 (+ 33976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:57.975976 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:58.871637 (+895661us) raft_consensus.cc:1685] finished
0709 12:26:58.871659 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:58.871809 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":27779,"prepare.run_wall_time_us":5705,"raft.queue_time_us":43737,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":5152,"spinlock_wait_cycles":24911744,"thread_start_us":100606,"threads_started":2,"wal-append.queue_time_us":131513}
I20260709 12:26:58.961818  6275 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:26:58.970021  6276 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:58.972287  6280 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.021s	sys 0.025s
I20260709 12:26:58.974764  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1016) took 1096 ms. Trace:
I20260709 12:26:58.960809  5705 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:26:58.979681  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2469) took 1254 ms. Trace:
I20260709 12:26:58.885937 11227 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.079s
I20260709 12:26:58.938637 10870 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:58.965061 11225 consensus_queue.cc:1243] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 17.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:26:58.885334 11382 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:58.980048  5705 rpcz_store.cc:276] 0709 12:26:57.725519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.959830 (+1234311us) service_pool.cc:224] Handling call
0709 12:26:58.960401 (+   571us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:58.979297 (+ 18896us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.979600 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:58.982815 11377 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.944s	user 0.042s	sys 0.064s
W20260709 12:26:59.008800  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 262500us
I20260709 12:26:59.016531  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:59.017390  5631 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:26:59.017676  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:58.969097 11371 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.004244  6276 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:59.013247  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:58.983222 11378 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:26:59.013280  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:59.020027  5705 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
W20260709 12:26:59.022634  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 218475us
W20260709 12:26:59.023859  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:59.016167 11377 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.988s	user 0.043s	sys 0.065s
I20260709 12:26:59.021855  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:59.027608  5633 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:26:59.015229 11225 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:26:59.031248  5675 leader_election.cc:336] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.691s (SENT)
I20260709 12:26:59.042405  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:59.044306  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:59.044857  5705 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
W20260709 12:26:59.020509  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 272750us
I20260709 12:26:59.028378  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:26:59.016795 11381 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:26:59.024868  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:59.023144 10870 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:26:59.048995  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:59.026769 11382 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:26:59.055819  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 313322us
I20260709 12:26:59.040757 10783 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.064613  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:59.058154  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:59.031780 11408 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:26:59.045295 11313 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.655s	user 0.000s	sys 0.025s
I20260709 12:26:59.069659  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:59.073050 11313 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.682s	user 0.000s	sys 0.027s
I20260709 12:26:59.065328 11198 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:59.084658  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:59.081413 10783 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:59.068965  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:59.053140 11417 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:59.053949 11381 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:59.107216 11408 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:59.118673 11417 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:59.098308 10870 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:59.100889 10902 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:59.101696 11423 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:26:59.101934 11429 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:59.102751 11421 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:26:59.023422 11371 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:59.047886 11225 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:59.108318  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:59.048677 11267 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.151365 11267 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:59.154330 11371 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:59.154633 11267 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:59.155023 11371 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:26:59.156476 10871 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:26:59.096670  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:26:59.120536  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 65660.7us
I20260709 12:26:59.161653  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:59.162976  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:26:59.092088 11321 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:59.107890 11356 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:59.094483 11419 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:59.111352 11324 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:59.107491 11420 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:26:59.277314 11420 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:59.146037  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 77197.7us
W20260709 12:26:59.278097 11418 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.188s	user 0.000s	sys 0.001s
I20260709 12:26:59.279278  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2655) took 1837 ms. Trace:
I20260709 12:26:59.029269 11378 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:59.279699  5709 rpcz_store.cc:276] 0709 12:26:57.442216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.710549 (+1268333us) service_pool.cc:224] Handling call
0709 12:26:58.751115 (+ 40566us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:58.751553 (+   438us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:58.751593 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:58.758396 (+  6803us) log.cc:844] Serialized 26 byte log entry
0709 12:26:59.031547 (+273151us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:59.031604 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:59.031679 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:59.279073 (+247394us) raft_consensus.cc:1685] finished
0709 12:26:59.279095 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:59.279218 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":115060,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"thread_start_us":236368,"threads_started":1,"wal-append.queue_time_us":496300}
I20260709 12:26:59.281461  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:26:59.281589 11408 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:26:59.282012  5709 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:59.135993 11425 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:26:59.237082 11423 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:26:59.150576 11225 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:59.283905 11423 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:59.278849 11419 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:59.124405 11318 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:59.286779 11419 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:59.140479 11350 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:59.137856 11429 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:59.274712  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 135705us
I20260709 12:26:59.281937 11378 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:59.265753 10870 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:59.335063  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:59.272640 11421 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:26:59.275312 11321 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:59.372159  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:59.373303  5631 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
W20260709 12:26:59.375012  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 283808us
I20260709 12:26:59.348955 11431 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.298916 11436 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:26:59.381366  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:59.350718 11415 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.307s	user 0.002s	sys 0.000s
I20260709 12:26:59.267073 11099 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:59.295449 11429 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:26:59.416163  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 358083us
I20260709 12:26:59.343237 11324 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:26:59.368847 10931 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.359s	user 0.003s	sys 0.012s
I20260709 12:26:59.351670 11321 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:59.424121  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:59.422807  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:26:59.426613  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 263678us
W20260709 12:26:59.427184  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:59.427636  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:59.429612  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:26:59.433493  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2657) took 1774 ms. Trace:
W20260709 12:26:59.425390  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:59.278115 11420 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:59.419112 11436 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:59.433995  5706 rpcz_store.cc:276] 0709 12:26:57.659327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.794227 (+1134900us) service_pool.cc:224] Handling call
0709 12:26:58.794671 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:58.794975 (+   304us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:26:59.050009 (+255034us) log.cc:844] Serialized 14 byte log entry
0709 12:26:59.433060 (+383051us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:59.433129 (+    69us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:59.433195 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:59.433288 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:59.433428 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":315418,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":2773,"raft.queue_time_us":129636,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":25442,"thread_start_us":611202,"threads_started":3,"wal-append.queue_time_us":371611}
I20260709 12:26:59.443190  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:59.352223 10870 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:59.439239  5633 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 14: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:26:59.446214  5709 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
W20260709 12:26:59.446920  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 350383us
I20260709 12:26:59.451534  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:59.438292  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:26:59.433755 11321 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:59.456521  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:26:59.468166  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:59.434471 11253 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:26:59.437829 11356 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:59.473578  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:59.475004  5709 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 13: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:26:59.441114 11381 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:59.419713 11431 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:59.446511 11436 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:26:59.493360 11272 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.699s	user 0.014s	sys 0.000s
I20260709 12:26:59.433050 11408 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 12:26:59.434255 10931 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:59.433996 11324 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:59.473498 11253 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:59.497527  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.503417  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:59.472855  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:59.501380  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:26:59.501981  6273 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:26:59.504117  5558 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:26:59.504724  5709 raft_consensus.cc:2410] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 20 because replica has last-logged OpId of term: 17 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:26:59.474056 11267 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:59.503147  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.470948 11225 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:26:59.440747 11149 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260709 12:26:59.505414  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.519244  5556 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:26:59.520139  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.520846  5556 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:26:59.513059  5552 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:26:59.513319  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.523124  5553 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:26:59.437777 11424 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 12:26:59.520874  5706 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: []
W20260709 12:26:59.514756  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 86637us
I20260709 12:26:59.475874 11430 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:59.476902 11356 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:59.526419  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2470) took 1794 ms. Trace:
I20260709 12:26:59.547367  5706 rpcz_store.cc:276] 0709 12:26:57.731674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:59.520226 (+1788552us) service_pool.cc:224] Handling call
0709 12:26:59.520635 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:59.526155 (+  5520us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:59.526344 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:59.560182  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:59.544092  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.562929  5552 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:59.544833  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.568572  5556 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:59.570207  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.548153  5709 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: []
I20260709 12:26:59.571966  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2472) took 1830 ms. Trace:
I20260709 12:26:59.493904 10783 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:59.572407  5709 rpcz_store.cc:276] 0709 12:26:57.741770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:59.526280 (+1784510us) service_pool.cc:224] Handling call
0709 12:26:59.547801 (+ 21521us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:59.571627 (+ 23826us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:59.571893 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:59.512633 11267 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:59.573277  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 121868us
I20260709 12:26:59.585119  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:26:59.576551  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.577685  5558 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:59.579417  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.512485 11198 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:26:59.512364 11253 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:59.594030  5552 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:26:59.595295  5709 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:26:59.600405  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2473) took 1828 ms. Trace:
I20260709 12:26:59.600764  5709 rpcz_store.cc:276] 0709 12:26:57.771733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:59.575179 (+1803446us) service_pool.cc:224] Handling call
0709 12:26:59.581271 (+  6092us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:59.600091 (+ 18820us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:59.600322 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:59.518631 11225 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:59.596726  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.558668 11356 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:26:59.575953  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 151933us
I20260709 12:26:59.573724 11447 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:59.592744 11267 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:59.602707 11225 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:26:59.590839 11417 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:26:59.610339 11322 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:26:59.590219 11198 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:59.603528  5556 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:26:59.609777  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2474) took 1818 ms. Trace:
I20260709 12:26:59.630355  5709 rpcz_store.cc:276] 0709 12:26:57.791580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:59.601580 (+1810000us) service_pool.cc:224] Handling call
0709 12:26:59.602447 (+   867us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:59.609397 (+  6950us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:59.609438 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:59.609491 (+    53us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:59.609529 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:59.609594 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:59.609719 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:59.605870  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.490774 11421 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:59.626650 11453 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:59.632925  5558 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:26:59.634052  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.649092  5556 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:59.624571 11408 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:26:59.602413 11446 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:59.623971 11427 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.306s	user 0.000s	sys 0.001s
I20260709 12:26:59.622823 11447 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:59.650048 11322 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:59.576211 11430 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:59.675216  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2461) took 2116 ms. Trace:
I20260709 12:26:59.787364  5708 rpcz_store.cc:276] 0709 12:26:57.567851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.757263 (+1189412us) service_pool.cc:224] Handling call
0709 12:26:58.784371 (+ 27108us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:58.865611 (+ 81240us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:59.674697 (+809086us) spinlock_profiling.cc:243] Waited 62.9 ms on lock 0x7b54001982d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:26:59.675120 (+   423us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":144636288}
I20260709 12:26:59.788125  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:59.674103  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:59.789965  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.790817  5556 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:26:59.643332 11198 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:26:59.792562  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1023) took 1301 ms. Trace:
W20260709 12:26:59.792841  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 207836us
I20260709 12:26:59.649451 11223 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:26:59.794904  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:59.793076  6271 rpcz_store.cc:276] 0709 12:26:58.490892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.683327 (+192435us) service_pool.cc:224] Handling call
0709 12:26:59.673871 (+990544us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:26:59.792396 (+118525us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26623872}
I20260709 12:26:59.650522 11408 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:26:59.800830  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:26:59.802250  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:26:59.784319 11406 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.662s	user 0.005s	sys 0.031s
I20260709 12:26:59.805491  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:59.810034  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:26:59.811940  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:59.809597 11220 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:59.707867 11417 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.817572  5708 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 12:26:59.738276 11435 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.274s	user 0.000s	sys 0.001s
W20260709 12:26:59.754397 11442 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 12:26:59.821278  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:26:59.811075 11454 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.785799 11276 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:26:59.669088 10931 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:26:59.855912  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.856734  5558 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:26:59.828573  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:59.732255 10783 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:26:59.834087 10912 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:26:59.829380  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.829082  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:26:59.799222 11423 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.840773  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2477) took 2028 ms. Trace:
I20260709 12:26:59.875171  5708 rpcz_store.cc:276] 0709 12:26:57.812005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:59.796315 (+1984310us) service_pool.cc:224] Handling call
0709 12:26:59.816974 (+ 20659us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:59.840295 (+ 23321us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:59.840689 (+   394us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:59.843221  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.679171 11446 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:59.812630 11408 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:26:59.822640  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 202516us
I20260709 12:26:59.851250  5635 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:26:59.818509 11149 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:26:59.830545 11218 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:59.865465  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:26:59.866330  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:00.120544  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2460) took 2561 ms. Trace:
W20260709 12:27:00.121058  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 254795us
I20260709 12:27:00.121304  5710 rpcz_store.cc:276] 0709 12:26:57.558678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.739951 (+1181273us) service_pool.cc:224] Handling call
0709 12:26:58.741793 (+  1842us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:58.865221 (+123428us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.120116 (+1254895us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54003ad4d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:00.120469 (+   353us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":281895808}
I20260709 12:26:59.795332 11099 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:59.876955  5556 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:27:00.126204  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.806955 11453 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:26:59.880991  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:26:59.854745 11431 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:26:59.851517 11324 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:26:59.881070 11445 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.265s	user 0.000s	sys 0.001s
I20260709 12:26:59.872931 11437 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:26:59.854096 11417 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:26:59.908385 11459 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:26:59.867678 10783 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:26:59.863314 11454 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.093657 11356 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:26:59.881422 11423 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.075973 11371 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.089196 11457 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:27:00.089781 11321 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.613s	user 0.011s	sys 0.006s
I20260709 12:26:59.913841  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:00.117856  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.163393  5708 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:27:00.120601  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:00.122195  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:00.164942  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:00.129021  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.165725 11444 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:00.166227 11444 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:00.166590 11444 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:00.166599  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:00.167853 11365 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:00.134253  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:00.135354  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:00.146756  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:00.119110 11149 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:00.132090 10931 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:00.154986 11467 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.127815  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:00.149670 11430 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:00.204725  5524 connection.cc:256] RPC call timeout handler was delayed by -1.9380395849998422s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.165s for a response
W20260709 12:27:00.202540  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55867.6us
I20260709 12:27:00.208098  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2665) took 2221 ms. Trace:
I20260709 12:27:00.131345 11218 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.208699  5710 rpcz_store.cc:276] 0709 12:26:57.987135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.207061 (+2219926us) service_pool.cc:224] Handling call
0709 12:27:00.207492 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:00.207705 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:00.207741 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:00.207797 (+    56us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:27:00.207842 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.207910 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:00.208032 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:00.200282  5556 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:00.212086  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:00.211910  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2666) took 2206 ms. Trace:
I20260709 12:27:00.157626 11417 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:00.214911  5710 rpcz_store.cc:276] 0709 12:26:58.005247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.210802 (+2205555us) service_pool.cc:224] Handling call
0709 12:27:00.211294 (+   492us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:00.211508 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:00.211544 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:00.211609 (+    65us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:00.211652 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.211721 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:00.211846 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:00.162359 11437 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:00.221683  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2667) took 2198 ms. Trace:
I20260709 12:27:00.226536  5710 rpcz_store.cc:276] 0709 12:26:58.023585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.220202 (+2196617us) service_pool.cc:224] Handling call
0709 12:27:00.220849 (+   647us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:00.221094 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:00.221131 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:00.221188 (+    57us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:00.221251 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.221424 (+   173us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:00.221591 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:00.229144  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2668) took 2204 ms. Trace:
I20260709 12:27:00.229606  5710 rpcz_store.cc:276] 0709 12:26:58.024487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.228019 (+2203532us) service_pool.cc:224] Handling call
0709 12:27:00.228478 (+   459us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:00.228716 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:00.228750 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:00.228807 (+    57us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:00.228854 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.228924 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:00.229074 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:27:00.224715  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59384.8us
I20260709 12:27:00.240307  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 958) took 1254 ms. Trace:
I20260709 12:27:00.240906  5710 rpcz_store.cc:276] 0709 12:26:58.986248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.230794 (+1244546us) service_pool.cc:224] Handling call
0709 12:27:00.231158 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:00.239746 (+  8588us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:00.239796 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:00.239861 (+    65us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:27:00.239947 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.240052 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:00.240233 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1524}
I20260709 12:27:00.243157  5710 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:27:00.203835 11149 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:00.208247 11457 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:00.203907 10931 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:00.214630 11365 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:27:00.232338  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:00.160027 11371 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.251189  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.252924  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:00.257217 11471 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:00.257799 11356 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:00.258530  5636 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 12:27:00.259927 11463 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:27:00.260912 11099 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 12:27:00.262522 11460 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
W20260709 12:27:00.263311 11267 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.645s	user 0.010s	sys 0.000s
I20260709 12:27:00.263718 11267 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.264560  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2663) took 2456 ms. Trace:
W20260709 12:27:00.264875 11462 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260709 12:27:00.323302  5709 rpcz_store.cc:276] 0709 12:26:57.807733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:59.631574 (+1823841us) service_pool.cc:224] Handling call
0709 12:26:59.635574 (+  4000us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:59.652815 (+ 17241us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:59.652870 (+    55us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:26:59.661692 (+  8822us) log.cc:844] Serialized 78 byte log entry
0709 12:26:59.720947 (+ 59255us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:26:59.866802 (+145855us) log.cc:844] Serialized 14 byte log entry
0709 12:26:59.867255 (+   453us) log.cc:844] Serialized 14 byte log entry
0709 12:26:59.872235 (+  4980us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:59.872316 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:00.264284 (+391968us) raft_consensus.cc:1685] finished
0709 12:27:00.264305 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:00.264483 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":863,"prepare.queue_time_us":664468,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":106,"raft.queue_time_us":5001,"raft.run_cpu_time_us":506,"raft.run_wall_time_us":45856,"thread_start_us":166060,"threads_started":2,"wal-append.queue_time_us":422283}
I20260709 12:27:00.264201 11149 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.265643 11417 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:00.267452  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.270495 11227 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:00.273679 11218 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:00.277835 11410 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.778s	user 0.010s	sys 0.010s
W20260709 12:27:00.279158  5632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2361) took 2273 ms (client timeout 1668 ms). Trace:
I20260709 12:27:00.282202 11454 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:00.199925 11220 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:27:00.283082 11198 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.287195  6272 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
W20260709 12:27:00.289337  5701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2466) took 2572 ms (client timeout 1650 ms). Trace:
I20260709 12:27:00.216265 11467 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.291412 11288 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.295120  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2398) took 3317 ms. Trace:
I20260709 12:27:00.294997  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:00.296087  5704 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2662) took 2506 ms (client timeout 2155 ms). Trace:
W20260709 12:27:00.346400  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 51473.4us
I20260709 12:27:00.347321  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:00.301725 11409 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.302662  5558 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:00.346220  5555 rpcz_store.cc:276] 0709 12:26:56.977507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:57.016936 (+ 39429us) service_pool.cc:224] Handling call
0709 12:26:57.855380 (+838444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:57.887424 (+ 32044us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.294929 (+2407505us) spinlock_profiling.cc:243] Waited 838 ms on lock 0x7b54000eb5d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:00.295060 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1926670720}
I20260709 12:27:00.300800 11423 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:00.307590 11455 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.259s	user 0.002s	sys 0.000s
I20260709 12:27:00.257812 11471 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.308117 11453 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:00.173859 11321 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.253995  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 959) took 1267 ms. Trace:
I20260709 12:27:00.258420 11356 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:27:00.316495  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56185.8us
W20260709 12:27:00.317683  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 165732us
I20260709 12:27:00.316215  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:00.365260  5710 rpcz_store.cc:276] 0709 12:26:58.986598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.242309 (+1255711us) service_pool.cc:224] Handling call
0709 12:27:00.242713 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:00.243577 (+   864us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.253918 (+ 10341us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:00.247815 11457 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:00.252177 10931 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:00.337363  5632 rpcz_store.cc:269] 0709 12:26:58.005388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.278941 (+2273553us) service_pool.cc:211] Skipping call since client already timed out
0709 12:27:00.279067 (+   126us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:27:00.238322 11437 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:00.390817 11416 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 1.290s	user 0.001s	sys 0.001s
I20260709 12:27:00.391391  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:00.392834 11437 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:00.252039 11365 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:00.343176  5701 rpcz_store.cc:269] 0709 12:26:57.716759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:57.762789 (+ 46030us) service_pool.cc:224] Handling call
0709 12:27:00.289028 (+2526239us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b5400323054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:00.289259 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":563780864}
I20260709 12:27:00.393757 11365 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:27:00.393940 11434 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.290s	user 0.001s	sys 0.001s
I20260709 12:27:00.342218 11227 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.351742 11409 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.398341 11457 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:00.399031 11374 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.399562 11374 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.348889 11468 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.344341 11266 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.329367 11417 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:00.337968 11446 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:00.366740  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:00.336803  5629 raft_consensus.cc:2410] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:27:00.331133 11473 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:00.342590 11371 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:00.379081 11198 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:00.429143  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:00.429769  6271 raft_consensus.cc:2376] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13. Re-sending same reply.
I20260709 12:27:00.430631  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:00.431651  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:00.380894  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.381311  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2404) took 2547 ms. Trace:
I20260709 12:27:00.382640 11220 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:27:00.383849 11440 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.801s	user 0.003s	sys 0.000s
I20260709 12:27:00.433430 11149 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:00.385154  5631 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:27:00.434778  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.435994  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.436161 11266 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:00.290266  6276 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1021) took 1800 ms (client timeout 1654 ms). Trace:
I20260709 12:27:00.437443 11267 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:00.437345  6276 rpcz_store.cc:269] 0709 12:26:58.489738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.667216 (+177478us) service_pool.cc:224] Handling call
0709 12:27:00.290084 (+1622868us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b5400343ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:00.290183 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":545596800}
W20260709 12:27:00.307288 11458 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.360s	user 0.001s	sys 0.001s
I20260709 12:27:00.438644  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:00.409157  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.357947 11223 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:00.444051 11446 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:00.444252 11223 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:00.444712 11223 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:00.444898 11371 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:27:00.445457 11371 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:00.445847 11371 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:00.422662 11468 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.423885 11227 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:00.425065 11374 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:00.425431 11321 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:00.433290  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.389595  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:00.367146 11257 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:00.437227  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:00.346294  5704 rpcz_store.cc:269] 0709 12:26:57.789867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.639108 (+849241us) service_pool.cc:224] Handling call
0709 12:27:00.295800 (+1656692us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b5400278654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:00.295968 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":36786432}
I20260709 12:27:00.432684  5557 rpcz_store.cc:276] 0709 12:26:57.834336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:57.846714 (+ 12378us) service_pool.cc:224] Handling call
0709 12:26:57.895702 (+ 48988us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:57.898177 (+  2475us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:57.898208 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:57.906873 (+  8665us) log.cc:844] Serialized 26 byte log entry
0709 12:26:57.999226 (+ 92353us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:26:57.999352 (+   126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:57.999510 (+   158us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:00.277230 (+2277720us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:00.381009 (+103779us) raft_consensus.cc:1685] finished
0709 12:27:00.381030 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:00.381243 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":850,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"spinlock_wait_cycles":42254720,"thread_start_us":56161,"threads_started":1,"wal-append.queue_time_us":281574}
I20260709 12:27:00.387050 11322 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:00.439447  6276 raft_consensus.cc:2410] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:27:00.444667 11446 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.422816 11409 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:00.447417  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:00.370162 11453 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.444729  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.448966  5708 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 16: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:00.427752 11417 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:00.485723  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:00.350822 11288 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.491644  5636 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:27:00.487428  5628 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 18
W20260709 12:27:00.486999  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 218325us
I20260709 12:27:00.439443 11473 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:00.490167  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:00.489642  5631 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 18: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:00.495689  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.496704  5631 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:00.497807  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.502159  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.504290  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.506253  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:00.489527  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:00.492923  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:00.493892  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:00.493996  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 139879us
W20260709 12:27:00.489549 11225 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.843s	user 0.014s	sys 0.001s
I20260709 12:27:00.513374  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:00.513626  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.515125  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:00.517189  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 94750.6us
I20260709 12:27:00.511729  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:00.514226  5636 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:00.518745  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:00.518833  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.479722 11267 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:27:00.524173  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.520781  5707 raft_consensus.cc:2410] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:27:00.523065  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:00.519585  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:00.525506  5553 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:27:00.529019  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:00.530659  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:00.531507  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:00.534881  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:00.538728  5635 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:00.499352 11257 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:00.541006  5628 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:00.499636 11473 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:00.543744  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:27:00.546033  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 81401.5us
I20260709 12:27:00.548801  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:00.550221  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:00.509606 11321 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.041s
W20260709 12:27:00.547871 11253 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.946s	user 0.011s	sys 0.003s
I20260709 12:27:00.521883 11225 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.596055 11485 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:27:00.578539  6276 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
W20260709 12:27:00.657554  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.650s (SENT) [suppressed 25 similar messages]
I20260709 12:27:00.579352 11257 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:00.622701  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 111025us
I20260709 12:27:00.655987  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:00.565605 11322 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:00.665261  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:00.661535  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 142845us
W20260709 12:27:00.680913 11408 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.736s	user 0.012s	sys 0.006s
W20260709 12:27:00.676301 11436 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.173s	user 0.013s	sys 0.006s
I20260709 12:27:00.692061  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:00.692822  5560 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:27:00.676733 11429 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.244s	user 0.009s	sys 0.008s
I20260709 12:27:00.684151 11424 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:27:00.702185  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44678.1us
I20260709 12:27:00.576989 11444 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.687361 11321 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:27:00.706629  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:00.689607 11322 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:00.689926 11481 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:00.728574  5706 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260709 12:27:00.729077  5706 pending_rounds.cc:85] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 8
I20260709 12:27:00.729372  5706 pending_rounds.cc:107] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 12:27:00.729660  5706 raft_consensus.cc:2889] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:00.729775  6276 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: []
W20260709 12:27:00.712895 11413 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 2.045s	user 0.006s	sys 0.000s
I20260709 12:27:00.716475 11424 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:00.701817 11408 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.718361 11489 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:27:00.718849 11226 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.720244 11453 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:00.709587 11480 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:00.676821 11253 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:00.727989  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2648) took 3502 ms (client timeout 2216 ms). Trace:
I20260709 12:27:00.733876  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1026) took 2009 ms. Trace:
W20260709 12:27:00.747992  5707 rpcz_store.cc:269] 0709 12:26:57.225223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:57.225663 (+   440us) service_pool.cc:224] Handling call
0709 12:27:00.721845 (+3496182us) spinlock_profiling.cc:243] Waited 3.25 s on lock 0x7b540006f754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:00.727867 (+  6022us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7469856512}
I20260709 12:27:00.742543  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1019) took 2273 ms. Trace:
I20260709 12:27:00.748625  6277 rpcz_store.cc:276] 0709 12:26:58.724179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.724417 (+   238us) service_pool.cc:224] Handling call
0709 12:26:58.724799 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:58.951135 (+226336us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.733612 (+1782477us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b5400591ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:00.733806 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":379209600}
I20260709 12:27:00.707031 11467 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:00.735082 11471 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:00.750224  6279 rpcz_store.cc:276] 0709 12:26:58.469484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.741080 (+271596us) service_pool.cc:224] Handling call
0709 12:26:58.741540 (+   460us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:58.873381 (+131841us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:58.873411 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:58.873889 (+   478us) log.cc:844] Serialized 26 byte log entry
0709 12:26:59.072444 (+198555us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:26:59.072505 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.393375 (+1320870us) spinlock_profiling.cc:243] Waited 8.36 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:00.742220 (+348845us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b5400771d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:00.742275 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:00.742320 (+    45us) raft_consensus.cc:1685] finished
0709 12:27:00.742336 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:00.742481 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":321761920,"thread_start_us":198120,"threads_started":1,"wal-append.queue_time_us":229261}
I20260709 12:27:00.712026 11223 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.760761  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:00.741957 11489 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:00.704519 11429 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.728466 11481 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:00.762084 11446 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:00.763827 11253 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:00.764294 11226 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:00.768105 11409 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:27:00.712941 11225 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:00.769922  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 83979.5us
I20260709 12:27:00.771795  6277 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:27:00.735875 11321 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:00.739825 11476 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:00.701378 11436 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:00.788312  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:00.833473  5557 raft_consensus.cc:2368] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16.
I20260709 12:27:00.796765  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:00.784898 11412 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 2.115s	user 0.009s	sys 0.021s
I20260709 12:27:00.831055  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:00.796743  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1025) took 1007 ms. Trace:
I20260709 12:27:00.837476  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:00.839701  6277 rpcz_store.cc:276] 0709 12:26:59.789240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.771024 (+981784us) service_pool.cc:224] Handling call
0709 12:27:00.771474 (+   450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:00.796452 (+ 24978us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.796675 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:00.838778  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:00.756196 11479 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260709 12:27:00.844623  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2446) took 3506 ms. Trace:
I20260709 12:27:00.846403  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:00.744827 11417 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.769142 11431 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.066s
I20260709 12:27:00.845834  5702 rpcz_store.cc:276] 0709 12:26:57.338692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:57.338874 (+   182us) service_pool.cc:224] Handling call
0709 12:26:58.604455 (+1265581us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:58.662012 (+ 57557us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.844419 (+2182407us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540026fa50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:00.844565 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2805604096}
I20260709 12:27:00.787477 11481 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:00.852783  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:00.857975  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61334.9us
I20260709 12:27:00.861426  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2639) took 1438 ms. Trace:
I20260709 12:27:00.879896  5630 rpcz_store.cc:276] 0709 12:26:59.422558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.852413 (+1429855us) service_pool.cc:224] Handling call
0709 12:27:00.861363 (+  8950us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:00.780664 10931 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.845041 11467 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:27:00.785499 11471 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.071s
I20260709 12:27:00.868505  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:00.890014  5556 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:00.878122  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:00.890726  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.891337  5630 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:27:00.891978  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:00.838269 11476 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:00.878392  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:00.881326  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:00.894215  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:00.851027 11312 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:00.848688 11409 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:00.882277  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:00.882870  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:00.902208  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1079) took 3365 ms. Trace:
I20260709 12:27:00.883186  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:00.883456  5701 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:27:00.850008 11490 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:00.836658 11149 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
W20260709 12:27:00.885188 11478 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260709 12:27:00.886422  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:00.914634  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:00.837980 11365 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:00.896436  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 57802.5us
I20260709 12:27:00.898692  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:00.915858  5704 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:00.915966  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:00.898859  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:00.899586  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:00.916697  5709 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:27:00.902760  5634 rpcz_store.cc:276] 0709 12:26:57.537762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:57.538652 (+   890us) service_pool.cc:224] Handling call
0709 12:26:57.539102 (+   450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:57.539360 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:57.539387 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:58.035266 (+495879us) spinlock_profiling.cc:243] Waited 49.7 ms on lock 0x7b2c00000de8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:26:58.035757 (+   491us) log.cc:844] Serialized 26 byte log entry
0709 12:26:58.400238 (+364481us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:26:58.400392 (+   154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.400885 (+   493us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:00.730614 (+2329729us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:00.901913 (+171299us) raft_consensus.cc:1685] finished
0709 12:27:00.901945 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:00.902131 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":146274560,"thread_start_us":352831,"threads_started":1,"wal-append.queue_time_us":633888}
I20260709 12:27:00.914793  5705 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:00.917105  5556 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:27:00.916942  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:00.915256  5710 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:00.918278  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:00.918447  5524 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:00.918879  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:00.919209 11225 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:00.919632 11198 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:27:00.919835 11225 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:00.892796  5707 raft_consensus.cc:2376] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15. Re-sending same reply.
I20260709 12:27:00.900151  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:00.920470 11225 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:00.894145  5703 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:00.920974 11198 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:00.921456  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:00.922060 11198 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:00.922430 11480 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:00.922649 11198 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:00.923100 11480 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:00.923264  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:00.924220 11480 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.901335 11409 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:00.929186 11409 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:00.930228 11490 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.853248 11322 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.932996 11490 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:00.933983  5702 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:27:00.935204 11475 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:00.857300 11489 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.935774 11475 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:00.936270 11475 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:00.936641 11489 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:00.895416 11457 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.940404  5553 raft_consensus.cc:2368] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15.
I20260709 12:27:00.941008  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:00.941408  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:26:58.976106  6280 rpcz_store.cc:276] 0709 12:26:57.877943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:57.878155 (+   212us) service_pool.cc:224] Handling call
0709 12:26:57.878485 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:57.878705 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:26:58.117453 (+238748us) log.cc:844] Serialized 14 byte log entry
0709 12:26:58.974314 (+856861us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:58.974378 (+    64us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:58.974437 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:58.974524 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:58.974694 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":789137,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":469,"thread_start_us":894268,"threads_started":2}
I20260709 12:27:00.942127  6276 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:00.930647 10931 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:00.894605 11476 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:00.945660  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.945833 11473 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.947633 11473 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:00.947988  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2661) took 3193 ms. Trace:
I20260709 12:27:00.948488  5706 rpcz_store.cc:276] 0709 12:26:57.754016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:59.556419 (+1802403us) service_pool.cc:224] Handling call
0709 12:27:00.722106 (+1165687us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:00.730291 (+  8185us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.947589 (+217298us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540006f750. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:00.947821 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2680094464}
I20260709 12:27:00.949030 11408 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:00.888551 11371 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:00.954471 11371 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:00.954530 11408 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:27:00.955528 11223 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:00.955899 11149 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:00.956290 11149 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:00.957134 11149 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:00.957670  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:00.958408 11149 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:00.959123  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:00.956310  6272 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1029) took 2231 ms (client timeout 1790 ms). Trace:
W20260709 12:27:00.960810  6272 rpcz_store.cc:269] 0709 12:26:58.725205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.736570 (+ 11365us) service_pool.cc:224] Handling call
0709 12:27:00.956103 (+2219533us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54005f5d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:00.956228 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3433309056}
I20260709 12:27:00.962400  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:00.964002  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:00.964174  5702 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:27:00.964641  5706 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:00.965636  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:00.970451  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1035) took 1101 ms. Trace:
I20260709 12:27:00.970963  5555 rpcz_store.cc:276] 0709 12:26:59.869892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.351295 (+481403us) service_pool.cc:224] Handling call
0709 12:27:00.351653 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:00.408844 (+ 57191us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:00.408875 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:00.408916 (+    41us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:00.408946 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.970164 (+561218us) spinlock_profiling.cc:243] Waited 57 ms on lock 0x7b540023ffd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:00.970240 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:00.970383 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":131047680}
W20260709 12:27:00.972992  6273 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1022) took 2247 ms (client timeout 1634 ms). Trace:
W20260709 12:27:00.973521  6273 rpcz_store.cc:269] 0709 12:26:58.725808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:58.733245 (+  7437us) service_pool.cc:224] Handling call
0709 12:27:00.972771 (+2239526us) spinlock_profiling.cc:243] Waited 743 ms on lock 0x7b5400534dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:00.972903 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1709603712}
I20260709 12:27:00.975461  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:00.976181  5703 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
I20260709 12:27:00.974736  5710 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:00.979223  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:00.982955 10870 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:00.985244  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1036) took 1196 ms. Trace:
I20260709 12:27:00.985760  6274 rpcz_store.cc:276] 0709 12:26:59.788688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:59.839618 (+ 50930us) service_pool.cc:224] Handling call
0709 12:26:59.861685 (+ 22067us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:00.312147 (+450462us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:00.312179 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:00.349052 (+ 36873us) log.cc:844] Serialized 26 byte log entry
0709 12:27:00.680427 (+331375us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:27:00.682451 (+  2024us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:00.984701 (+302250us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b540020c454. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:00.984760 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:00.984832 (+    72us) raft_consensus.cc:1685] finished
0709 12:27:00.984854 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:00.985149 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":119,"raft.queue_time_us":44715,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":11750,"spinlock_wait_cycles":687661824,"thread_start_us":431077,"threads_started":2,"wal-append.queue_time_us":372715}
I20260709 12:27:00.989434 11447 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:00.980238 11149 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:01.002436 10783 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:01.004926  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:00.997421 11149 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:01.007824  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2480) took 1137 ms. Trace:
I20260709 12:27:01.008244  5708 rpcz_store.cc:276] 0709 12:26:59.870735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.164002 (+293267us) service_pool.cc:224] Handling call
0709 12:27:01.007628 (+843626us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b54002f6054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:01.007765 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":484351744}
I20260709 12:27:01.011360 11288 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:01.015802 11266 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:01.012480 11149 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.028705  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:01.030213  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:27:01.031397  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:01.036448  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:01.036412  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:01.038208  5557 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:01.039490  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:00.895212 11467 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:01.021054 11225 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.025172 11321 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.055790 11225 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:00.851548 11424 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.048777 11365 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:00.911882 11479 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:01.050215 11429 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:00.877933 11453 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 12:27:00.910197 11488 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:00.948498 11444 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:01.058394  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:01.059641  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:01.059641  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:01.062566  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:01.070156  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:01.065413  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:01.067728  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:01.071779  5629 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:01.068053  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:01.072717  5557 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:01.077463  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:01.082582  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:01.088137  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:01.066864 11417 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:01.100791  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 70693.8us
I20260709 12:27:01.102973  5561 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:01.103283 11223 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:01.103931 11446 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.104423  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:01.104411 11446 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:01.105427 11403 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 2.786s	user 0.000s	sys 0.015s
I20260709 12:27:00.925848 11421 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.140s
I20260709 12:27:01.107301  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:01.109475  5557 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:01.110029  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:01.113391  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:01.114074  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:01.114744  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:01.067127 11436 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:01.132588  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1034) took 1264 ms. Trace:
I20260709 12:27:01.140336  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:01.145299  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:01.142539  5557 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:01.148801  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:01.150699  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:01.151750  5557 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:01.143079  5552 rpcz_store.cc:276] 0709 12:26:59.868983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.235528 (+366545us) service_pool.cc:224] Handling call
0709 12:27:00.252907 (+ 17379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:00.369571 (+116664us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:00.369606 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:00.369651 (+    45us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:00.369697 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:01.132214 (+762517us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54001be254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:01.132283 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:01.132487 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":23913,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"spinlock_wait_cycles":267251072}
I20260709 12:27:01.068607 11488 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:01.157991  5705 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: []
I20260709 12:27:01.159173  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:01.160989  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:01.162618  5706 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:01.169080  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2507) took 3869 ms. Trace:
I20260709 12:27:01.155377 11487 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:01.114512 11444 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:01.106639 11272 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.064s
I20260709 12:27:01.219473  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:01.174629  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:01.178905  5708 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:27:01.155603 11450 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:01.129199  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:01.186272  5703 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:01.115828 11448 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:01.157277 11267 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:01.153448 11288 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:01.182008  5554 rpcz_store.cc:276] 0709 12:26:57.300135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:57.795299 (+495164us) service_pool.cc:224] Handling call
0709 12:26:57.795799 (+   500us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:57.809336 (+ 13537us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:57.809773 (+   437us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:57.828241 (+ 18468us) log.cc:844] Serialized 26 byte log entry
0709 12:26:57.948281 (+120040us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:26:57.948323 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:57.948417 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:01.104773 (+3156356us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:01.157035 (+ 52262us) raft_consensus.cc:1685] finished
0709 12:27:01.157067 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:01.169016 (+ 11949us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":76,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":45434240,"thread_start_us":69600,"threads_started":1,"wal-append.queue_time_us":234388}
I20260709 12:27:01.229811  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:01.176486 11474 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:27:01.231817  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57275.6us
I20260709 12:27:01.241747  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:01.184453 11488 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:01.237325  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:01.245431  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:01.213193 11487 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:01.258584  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 129516us
I20260709 12:27:01.259325  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:01.250981  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:01.257797  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:01.260600  5554 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:01.230310 11450 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.265301  5675 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:01.276906  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:01.231092 11267 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:01.185086 11459 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
I20260709 12:27:01.273725  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:01.275367  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2516) took 1778 ms. Trace:
I20260709 12:27:01.266075  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:01.292586  5560 rpcz_store.cc:276] 0709 12:26:59.497058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:59.497207 (+   149us) service_pool.cc:224] Handling call
0709 12:27:01.275016 (+1777809us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54001af254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:01.275285 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2727272704}
W20260709 12:27:01.283197  5633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2638) took 2196 ms (client timeout 1772 ms). Trace:
I20260709 12:27:01.239523 11438 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:27:01.285760  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:01.294804  5633 rpcz_store.cc:269] 0709 12:26:59.074945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:59.097347 (+ 22402us) service_pool.cc:224] Handling call
0709 12:27:01.271247 (+2173900us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b5400438354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:01.271418 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":674892032}
I20260709 12:27:01.296902  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:01.297755  5552 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:01.256960 11275 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:27:01.231102 11356 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 12:27:01.265059 11487 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:01.279608 11209 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:01.262580 11379 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:01.276095 11448 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:01.267726 11444 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:01.270324 11354 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:27:01.280018 11450 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:01.299621  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:01.277992 11490 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:01.280202 11436 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:01.289567 11267 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:01.285059 11365 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:01.244827 11288 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.316819 11226 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:01.305577 10783 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:01.334148 11490 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:01.334619  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:01.334820 11417 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:01.335175 11436 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:01.358971  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:01.338142  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:01.309036 11500 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:27:01.363379  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 85568.2us
I20260709 12:27:01.328435 11448 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:01.362517  5552 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:27:01.375294  5557 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:01.377540  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:01.393832 11409 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.356905 10783 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:01.405370 10783 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:01.414255  5556 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:01.413137 10783 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.363502 11226 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.419005 11226 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:01.363752 11436 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:01.420373 10783 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:01.416497 11409 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:01.421131 11504 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:27:01.344717 11503 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:27:01.431397  5676 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.4.65.62:38921, user_credentials={real_user=slave}} blocked reactor thread for 54309.2us
W20260709 12:27:01.425573  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48112.3us
I20260709 12:27:01.426589  5629 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
W20260709 12:27:01.428670  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:01.438848  5554 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:01.368042 11502 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:01.374099 11444 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:27:01.449914  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:01.457928  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 212562us
I20260709 12:27:01.448464  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:01.459017  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:01.462898  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:01.464344  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:01.464737  5602 leader_election.cc:400] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16.
I20260709 12:27:01.465054  5602 leader_election.cc:403] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:27:01.465163  5560 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:01.466372  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:01.467952  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:01.469020  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:01.469296  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:01.448967 11198 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.526s	user 0.011s	sys 0.000s
I20260709 12:27:01.440688 11417 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:01.440603 11503 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:01.447495 11502 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.473784  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:01.474395  5705 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:01.474685  5709 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:01.480932  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:01.471184  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:27:01.491415  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:01.513559  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:01.492720  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:01.474917 11431 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:27:01.471463 11321 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:01.473920 11497 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:01.525481  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:01.525872 11507 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:27:01.527302  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:01.528048 11509 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:27:01.473832 11356 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:27:01.512709 11485 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:27:01.486217 11444 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:01.521040  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:01.532472  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:01.480862 11198 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.538188 11288 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:01.535801  5705 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:27:01.483551 11503 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:01.545082 11508 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.173s	user 0.000s	sys 0.000s
I20260709 12:27:01.485597 11374 raft_consensus.cc:2764] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
W20260709 12:27:01.552953 11511 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:27:01.483896 11490 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.555814 11409 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:01.467607 11365 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
W20260709 12:27:01.565649 11489 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.628s	user 0.011s	sys 0.000s
W20260709 12:27:01.566483  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:01.566689  5702 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: []
I20260709 12:27:01.471449 10871 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:01.483678 11502 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:01.483238 11438 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:01.486838 11362 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:01.576375  5560 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:27:01.574962 11475 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.638s	user 0.009s	sys 0.004s
I20260709 12:27:01.598212 11226 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:01.644577 11362 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:01.671223 11362 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.673233 11226 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.676764 11226 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:01.625564  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 173962us
W20260709 12:27:01.678866  5599 connection.cc:256] RPC call timeout handler was delayed by -1.7205669620002482s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2155s for a response
I20260709 12:27:01.588063 11444 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:01.581435 11485 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:01.649832  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:01.629282 10871 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:01.623036  5709 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
W20260709 12:27:01.637881 11476 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.691s	user 0.008s	sys 0.004s
I20260709 12:27:01.700757  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:01.628540 11374 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:01.627764 11420 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:01.681238 11514 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:01.701761  5560 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:27:01.705799  5561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2532) took 2127 ms (client timeout 1721 ms). Trace:
I20260709 12:27:01.633869 11438 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:27:01.706364  5561 rpcz_store.cc:269] 0709 12:26:59.576630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:59.579025 (+  2395us) service_pool.cc:224] Handling call
0709 12:27:01.704011 (+2124986us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b540044e154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:01.704139 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3451622144}
I20260709 12:27:01.707942  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:01.688735 11354 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.647979 11198 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:01.716169  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1089) took 2662 ms. Trace:
I20260709 12:27:01.701107 10871 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:01.713351  5561 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:01.626157 11409 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.716624  5627 rpcz_store.cc:276] 0709 12:26:59.053983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:59.056192 (+  2209us) service_pool.cc:224] Handling call
0709 12:26:59.056489 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:59.737892 (+681403us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:26:59.775637 (+ 37745us) log.cc:844] Serialized 14 byte log entry
0709 12:27:00.749606 (+973969us) spinlock_profiling.cc:243] Waited 1.7 ms on lock 0x7b1800249d60. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:27:01.194307 (+444701us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:01.194391 (+    84us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:01.194452 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:01.715873 (+521421us) spinlock_profiling.cc:243] Waited 681 ms on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:01.715945 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:01.716108 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13251,"prepare.queue_time_us":21873,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":14633,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":7695,"spinlock_wait_cycles":1570120192,"thread_start_us":431002,"threads_started":2,"wal-append.queue_time_us":1290203}
I20260709 12:27:01.684916 11475 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.703066 11476 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.705753 11267 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.709863 11448 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.707126 11374 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.728056 11480 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:01.718628 11354 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.768240 11409 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:01.713277 11514 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:01.771414 11518 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
W20260709 12:27:01.801792  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 122039us
I20260709 12:27:01.736297 11444 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.772809 11420 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:01.787305 11223 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:01.787456 11409 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:01.762472 11485 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.821444  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:01.778151 11480 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.832423  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:01.833815  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:01.810771 11420 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.828497 11354 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:01.839013 11488 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.561s	user 0.010s	sys 0.001s
I20260709 12:27:01.839201 11420 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:01.839529 11488 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.840551 11356 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 12:27:01.843477 11488 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:01.845786 11365 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:01.838793 11354 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:27:01.835453 11362 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:01.818773 11409 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.829283 11198 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:27:01.852144  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49090.5us
W20260709 12:27:01.830745 11487 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.503s	user 0.011s	sys 0.000s
I20260709 12:27:01.833627 11490 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:01.816167 11223 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.858866 11487 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.865612 11487 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:01.855715 11362 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.868597  5705 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:01.869002 11476 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:01.870654 11431 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16.
I20260709 12:27:01.873466 11480 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:01.873651 11468 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:01.875392 11223 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:01.867518 10871 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.878396 10871 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:01.878911 11267 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:01.879477 11267 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.879933 11267 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.882176  6273 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:27:01.884918 11514 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:01.885605  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:01.885836 11446 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:01.886091 11448 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:01.886358  6272 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:27:01.888684 11450 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.538s	user 0.005s	sys 0.005s
W20260709 12:27:01.889864  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 412969us
I20260709 12:27:01.890789 11475 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:01.891891  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:01.875334 11379 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:01.889986  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:01.894894  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:01.896107  6271 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:01.898044  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:01.889106 11450 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.901279  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:01.903596  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:01.902562  6277 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:01.906191  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:01.891909 11481 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.018s	user 0.004s	sys 0.009s
I20260709 12:27:01.910013  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:01.909934  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:01.913133  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:01.914094  6271 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:01.905368  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:01.896652 11379 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:01.921161  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:01.922986 11267 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:01.908798 11374 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:01.905750 11490 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:27:01.925501 11409 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:01.906078 11476 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:27:01.932737 11450 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:01.933571 11467 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:01.934621 11467 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:01.934895 11438 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.917868 11362 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:01.917770 11481 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:01.942564 11436 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.522s	user 0.004s	sys 0.010s
I20260709 12:27:01.943181 11436 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:01.952340 11505 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.522s	user 0.004s	sys 0.000s
I20260709 12:27:01.952862 11481 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:01.953316 11503 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.953570 11502 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:01.955422 11477 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:27:01.955930 11477 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:01.957149 11477 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:01.958406 11477 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:01.961735 11502 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:01.962575 11503 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:01.996971  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:01.997869  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:02.001529  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:02.008875  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 152925us
I20260709 12:27:02.010510  5552 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:01.904917  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:02.016785  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 111826us
I20260709 12:27:02.018473  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:02.022024  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:02.025158  5553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2408) took 1581 ms (client timeout 1790 ms). Trace:
I20260709 12:27:02.025655  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:02.026104  5553 rpcz_store.cc:269] 0709 12:27:00.443710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.526716 (+ 83006us) service_pool.cc:224] Handling call
0709 12:27:02.024419 (+1497703us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b54001c48d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:02.024856 (+   437us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":898678144}
I20260709 12:27:02.019753  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:02.029421  5560 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 15. Ops: []
I20260709 12:27:02.012105  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:02.033985  5555 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:27:02.041782 11409 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:02.042459  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:02.043751  5553 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:02.043129 11409 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.045603  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2424) took 1481 ms. Trace:
I20260709 12:27:02.045641  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:02.053220  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1029) took 1732 ms. Trace:
I20260709 12:27:02.060695 11453 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:02.061748  5708 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: []
I20260709 12:27:02.062081  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:02.061862 11453 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.063169  5703 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:02.063596  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:02.065235  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:02.067034  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:02.068768 11481 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:27:02.068971  5557 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:27:02.070574  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:02.055593  6279 rpcz_store.cc:276] 0709 12:27:00.320768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.780109 (+459341us) service_pool.cc:224] Handling call
0709 12:27:00.780601 (+   492us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:00.788302 (+  7701us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:00.788427 (+   125us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:00.998467 (+210040us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b1800361de0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a
0709 12:27:00.999526 (+  1059us) spinlock_profiling.cc:243] Waited 5.68 ms on lock 0x7b2c00040818. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:27:01.000118 (+   592us) log.cc:844] Serialized 26 byte log entry
0709 12:27:01.244196 (+244078us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:01.244229 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:01.244388 (+   159us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:01.951790 (+707402us) spinlock_profiling.cc:243] Waited 4.77 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:02.052546 (+100756us) raft_consensus.cc:1685] finished
0709 12:27:02.052583 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:02.052964 (+   381us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7179,"raft.queue_time_us":12487,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":38578,"spinlock_wait_cycles":29006208,"thread_start_us":210135,"threads_started":1,"wal-append.queue_time_us":328689}
I20260709 12:27:02.075613  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:02.077790  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:02.078249  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:02.079584  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:02.081270  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:02.084573  5556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2421) took 1567 ms (client timeout 2010 ms). Trace:
W20260709 12:27:02.085242  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:02.085121  5556 rpcz_store.cc:269] 0709 12:27:00.516679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.918049 (+401370us) service_pool.cc:224] Handling call
0709 12:27:02.083746 (+1165697us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b540023f354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:02.084042 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1138933632}
I20260709 12:27:02.086570  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:02.088886 10871 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:02.090158 10871 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.096980  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:02.054709  5555 rpcz_store.cc:276] 0709 12:27:00.564698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.972159 (+407461us) service_pool.cc:224] Handling call
0709 12:27:02.021343 (+1049184us) spinlock_profiling.cc:243] Waited 65.6 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:27:02.045483 (+ 24140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":150923520}
I20260709 12:27:02.099795  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:02.100728 11209 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:02.066888  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:02.067860  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:02.101210 11209 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:02.042466  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:02.069947 11436 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:02.114565  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 89646us
I20260709 12:27:02.113961  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:02.117683  5701 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:02.114109  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:02.127820 11481 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:02.128873 11374 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:02.129410 11536 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:02.129703 11480 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:27:02.130395  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 49375.2us
I20260709 12:27:02.133324 11485 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:27:02.135474  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:02.135272 11529 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:27:02.135989 11534 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:27:02.136739 11529 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:02.137688 11534 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:27:02.129451 11481 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:02.138193 11534 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:02.139017  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:02.137209 11529 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:02.140724 11528 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:27:02.147140 11209 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:02.150344  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:02.152485  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:02.152948  5552 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:02.153139 11488 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:27:02.154140  6275 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 12:27:02.156632  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 90124us
I20260709 12:27:02.169373 11453 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:02.129410 11374 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:02.171855 11374 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:02.172508  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:02.139854 11534 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:02.176074 11534 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:02.177412 11209 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:02.180714  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:02.181108  5710 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14. Ops: [13.9-13.9]
W20260709 12:27:02.184578  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 93050.9us
I20260709 12:27:02.186053 11446 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:02.186715 11446 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:02.187507 11446 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:02.189630 11409 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:02.195281 11457 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:02.199047 11198 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:27:02.199788 11528 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:02.201267  5553 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:02.230561  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:02.231211 11431 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:02.231894 11431 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.232491 11495 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.010s
I20260709 12:27:02.234452 11431 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:02.231264  5556 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:02.241284  6271 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:02.242303  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:02.243245  5558 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:02.245958  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:02.246754  5559 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:02.247325  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 63311.4us
I20260709 12:27:02.247794  5556 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:02.248588 11267 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:02.248808  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:02.249028 11267 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:02.250126 11267 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.251093  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1038) took 2380 ms. Trace:
I20260709 12:27:02.251487  5560 rpcz_store.cc:276] 0709 12:26:59.871108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:01.715843 (+1844735us) service_pool.cc:224] Handling call
0709 12:27:02.028795 (+312952us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.030240 (+  1445us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.250845 (+220605us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54001c48d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:02.250984 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":709547776}
I20260709 12:27:02.256876 11528 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.260124  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:02.261126  5555 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:02.264706 11453 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:27:02.266450 11453 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:27:02.266878 11453 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:02.242388  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:02.274281  5634 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260709 12:27:02.268666  5553 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:02.271180  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:02.269280  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:02.275094  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.284840 11267 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:02.294487  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1045) took 2235 ms. Trace:
I20260709 12:27:02.295120  5553 rpcz_store.cc:276] 0709 12:27:00.059378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.289626 (+2230248us) service_pool.cc:224] Handling call
0709 12:27:02.293621 (+  3995us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.293885 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:02.293913 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:02.293981 (+    68us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:02.294021 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.294212 (+   191us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:02.294376 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.270090  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:02.270901  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.298561  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.303218 10871 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:02.309446 11528 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:02.186836 11542 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.123s
I20260709 12:27:02.309859  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:02.310303  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:02.310787  5560 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:02.311738  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:02.312424  5556 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:02.314059 11471 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:02.314149  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:02.317371  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:02.298842  5553 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:27:02.351168  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2406) took 1888 ms. Trace:
I20260709 12:27:02.351755  5553 rpcz_store.cc:276] 0709 12:27:00.443191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.297985 (+1854794us) service_pool.cc:224] Handling call
0709 12:27:02.298512 (+   527us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:02.330911 (+ 32399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.331188 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.370952  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:02.374637  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:02.267951 11453 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:02.376914  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.386466  5627 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:02.387990  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:02.392827  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 80451.7us
I20260709 12:27:02.449401  5553 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:27:02.396638 11471 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:02.454597  5702 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:02.456671  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2407) took 2009 ms. Trace:
I20260709 12:27:02.457223  5553 rpcz_store.cc:276] 0709 12:27:00.443459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.369187 (+1925728us) service_pool.cc:224] Handling call
0709 12:27:02.396388 (+ 27201us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:02.452673 (+ 56285us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.453145 (+   472us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.453809 11480 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.460807 11480 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:02.463726 11480 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:02.464444  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:02.465690  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47226.4us
I20260709 12:27:02.466768  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:02.467579 11480 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:02.467998 11480 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:02.468408 11480 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:02.468497 11379 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.516s	user 0.004s	sys 0.016s
I20260709 12:27:02.469120 11379 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.471830 11379 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:02.472604  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1041) took 2477 ms. Trace:
I20260709 12:27:02.473053  5554 rpcz_store.cc:276] 0709 12:26:59.878164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.050709 (+2172545us) service_pool.cc:224] Handling call
0709 12:27:02.051254 (+   545us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.071768 (+ 20514us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:02.071810 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:02.071898 (+    88us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:02.071954 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.353787 (+281833us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b54003a93d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:02.353849 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:02.355730 (+  1881us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46279424}
W20260709 12:27:02.474002 11477 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.515s	user 0.013s	sys 0.003s
I20260709 12:27:02.434767 11198 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:02.477792 11438 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:02.477568 11198 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.478672  6271 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:02.479969 11198 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:02.480861 11453 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:02.481734 11480 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:02.486495  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:02.491739  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.492712  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:02.493186 11365 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.647s	user 0.022s	sys 0.001s
I20260709 12:27:02.493707 11471 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:02.494086  5630 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 12. Current term is 13. Ops: []
W20260709 12:27:02.494230 11467 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.504s	user 0.020s	sys 0.000s
I20260709 12:27:02.494511 11534 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:02.494578  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53925.9us
I20260709 12:27:02.536300  5559 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:02.504225  5560 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:02.512706  5553 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:27:02.545997  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2411) took 2099 ms. Trace:
I20260709 12:27:02.546418  5553 rpcz_store.cc:276] 0709 12:27:00.446500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.506905 (+2060405us) service_pool.cc:224] Handling call
0709 12:27:02.512056 (+  5151us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:02.545670 (+ 33614us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.545930 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.517313  5558 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:02.510457  5555 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
W20260709 12:27:02.505899  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 111629us
W20260709 12:27:02.497538  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 478128us
I20260709 12:27:02.537748  5627 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:02.545735  5557 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:27:02.549299  5554 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:02.542591  5556 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
W20260709 12:27:02.553290  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:02.537412  5559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 1054) took 1771 ms (client timeout 2096 ms). Trace:
I20260709 12:27:02.543469  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.554785  5634 raft_consensus.cc:2410] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:27:02.512283 10871 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:02.555123  5559 rpcz_store.cc:269] 0709 12:27:00.765959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:00.971136 (+205177us) service_pool.cc:224] Handling call
0709 12:27:02.245602 (+1274466us) spinlock_profiling.cc:243] Waited 64.9 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:27:02.537354 (+291752us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149286784}
I20260709 12:27:02.565469  5631 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:02.567317 11534 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.569552 10871 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.546566 11374 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:02.537667 11477 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:02.572116  5602 leader_election.cc:400] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15.
W20260709 12:27:02.574234  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 254138us
I20260709 12:27:02.585261  5558 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15. Ops: []
W20260709 12:27:02.601356  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:02.610467  5602 leader_election.cc:403] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:27:02.586330  5553 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:02.572602  5561 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:27:02.583403  6278 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:02.578675  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:02.611907  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2419) took 2163 ms. Trace:
I20260709 12:27:02.584843  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:02.580933  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:02.584446  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1042) took 2674 ms. Trace:
I20260709 12:27:02.617641  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2413) took 2170 ms. Trace:
I20260709 12:27:02.618120  5553 rpcz_store.cc:276] 0709 12:27:00.446970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.568688 (+2121718us) service_pool.cc:224] Handling call
0709 12:27:02.586073 (+ 17385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.615253 (+ 29180us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.617579 (+  2326us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.614729  5558 rpcz_store.cc:276] 0709 12:27:00.448873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.584585 (+2135712us) service_pool.cc:224] Handling call
0709 12:27:02.585028 (+   443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.611569 (+ 26541us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.611837 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.619762  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:02.583014  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:02.582417  5636 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:02.588454  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:02.622210  5636 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:02.616166  5552 rpcz_store.cc:276] 0709 12:26:59.911091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.168277 (+2257186us) service_pool.cc:224] Handling call
0709 12:27:02.168810 (+   533us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:02.169456 (+   646us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:02.169487 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:02.170333 (+   846us) log.cc:844] Serialized 26 byte log entry
0709 12:27:02.427962 (+257629us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:02.428024 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.428275 (+   251us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:02.584132 (+155857us) raft_consensus.cc:1685] finished
0709 12:27:02.584162 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:02.584374 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":933,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":172,"thread_start_us":257029,"threads_started":1,"wal-append.queue_time_us":340467}
I20260709 12:27:02.581928  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:02.563479 11365 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:02.564257 11467 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:02.635322  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:02.635632 11467 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:02.572204 10871 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:02.632916  5633 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:02.628145  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2412) took 2176 ms. Trace:
I20260709 12:27:02.640954  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:02.641249  5554 rpcz_store.cc:276] 0709 12:27:00.446745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.474210 (+2027465us) service_pool.cc:224] Handling call
0709 12:27:02.474629 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.549750 (+ 75121us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.623040 (+ 73290us) spinlock_profiling.cc:243] Waited 74.4 ms on lock 0x7b540044e154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:02.623254 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":171215616}
I20260709 12:27:02.642246  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:02.642634  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:02.643318  5554 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:02.643489  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.647488 11467 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:02.577564 11479 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:02.634670 11489 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:02.636135 11362 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.637002  5636 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:02.638427  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.648109  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:02.651542 11547 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:27:02.630703  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:02.634469  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46115us
I20260709 12:27:02.654028  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:02.654091  5707 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:02.655671  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:02.656467  5627 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 14: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:02.657387  5629 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:27:02.658529  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:02.664819  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:02.628537 11446 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.669188  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:02.672711  5627 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 14: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:02.672854  5552 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 18: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:02.669610  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:02.585446 11223 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.115s
I20260709 12:27:02.581377 11438 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.082s
I20260709 12:27:02.682823  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:02.674317  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55347.2us
I20260709 12:27:02.504873 11529 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:02.673533  5553 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:02.661018 11534 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:02.692698  5636 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:02.694083  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:02.696244  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.585733 11549 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.087s
I20260709 12:27:02.700042  5555 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: [13.8-15.9]
I20260709 12:27:02.703761  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.704927 11420 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.743929 11420 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.721066  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1048) took 1956 ms. Trace:
I20260709 12:27:02.722586  5635 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:02.747803  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:02.728395  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 145552us
I20260709 12:27:02.750577  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:02.751516 11534 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:02.728741  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:02.753023  5702 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:02.728807  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.754499  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.755698  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.729754  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1049) took 1965 ms. Trace:
I20260709 12:27:02.749086  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2425) took 2126 ms. Trace:
I20260709 12:27:02.729887  5705 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:02.730731  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:02.756731  5557 rpcz_store.cc:276] 0709 12:27:00.764625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.728808 (+1964183us) service_pool.cc:224] Handling call
0709 12:27:02.729160 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.729368 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:02.729392 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:02.729461 (+    69us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:02.729500 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.729574 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:02.729688 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.732265  5704 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:27:02.747052  5560 rpcz_store.cc:276] 0709 12:27:00.764281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.720021 (+1955740us) service_pool.cc:224] Handling call
0709 12:27:02.720428 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.720684 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:02.720706 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:02.720754 (+    48us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:02.720782 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.720834 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:02.720968 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.693459 11436 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:02.759161 11198 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:27:02.763620 11198 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:02.734445 11480 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:02.764537 11446 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:02.734826  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1039) took 2863 ms. Trace:
I20260709 12:27:02.665915 11477 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.764948 11198 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.765339  5561 rpcz_store.cc:276] 0709 12:26:59.871679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:01.723085 (+1851406us) service_pool.cc:224] Handling call
0709 12:27:01.723591 (+   506us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:02.624721 (+901130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.734565 (+109844us) spinlock_profiling.cc:243] Waited 814 ms on lock 0x7b540041a0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:02.734774 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1871411584}
I20260709 12:27:02.766598 11198 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:02.736109  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:02.767066  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1050) took 2002 ms. Trace:
I20260709 12:27:02.767462  5559 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:02.768400 11477 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:02.736486  5629 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:02.770561  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.771683  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.737159  5636 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:27:02.773406  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.774235  5636 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:02.740880 11551 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:27:02.689837  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:02.742326 11362 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:02.759231 11436 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:02.733374  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:02.759400  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:02.749120  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:02.799115  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:02.760460  5627 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:02.799757  5703 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:02.757187  5555 rpcz_store.cc:276] 0709 12:27:00.623188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.669459 (+2046271us) service_pool.cc:224] Handling call
0709 12:27:02.699684 (+ 30225us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:02.748731 (+ 49047us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.749022 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.762187  5557 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:02.767468  5558 rpcz_store.cc:276] 0709 12:27:00.764894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.619046 (+1854152us) service_pool.cc:224] Handling call
0709 12:27:02.688049 (+ 69003us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.766164 (+ 78115us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:02.766200 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:02.766262 (+    62us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:02.766292 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.766850 (+   558us) spinlock_profiling.cc:243] Waited 77.8 ms on lock 0x7b54000eb0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:02.766895 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:02.767002 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":179026816}
I20260709 12:27:02.786330  5554 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
W20260709 12:27:02.759438  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 187594us
I20260709 12:27:02.794622  5630 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:02.801791  5556 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: [11.6-13.7]
I20260709 12:27:02.581465 11471 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.804845  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1052) took 2039 ms. Trace:
I20260709 12:27:02.805271  5557 rpcz_store.cc:276] 0709 12:27:00.765423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.761541 (+1996118us) service_pool.cc:224] Handling call
0709 12:27:02.762007 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.804599 (+ 42592us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.804791 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:27:02.805805  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 139153us
I20260709 12:27:02.809515  5553 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:02.809983  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.748473 11420 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:02.812103  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.813351  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:02.789677 11467 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:02.816089  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:02.817727  5631 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:27:02.775272 11503 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:02.805048  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:02.818650  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.818974  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:02.823262  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:02.816030  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:02.826663  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:02.827983  5561 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260709 12:27:02.820082  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:02.832428  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:02.834640  5558 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:02.835444  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2549) took 1777 ms. Trace:
I20260709 12:27:02.835835  5558 rpcz_store.cc:276] 0709 12:27:01.057730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.806785 (+1749055us) service_pool.cc:224] Handling call
0709 12:27:02.834409 (+ 27624us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.835168 (+   759us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.835383 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.842286  5710 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:02.842476  5630 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 18: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:02.842576  5636 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:02.844916  5702 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:02.841374  5704 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:02.825363 11467 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:02.848145  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2429) took 1750 ms. Trace:
I20260709 12:27:02.849669  5629 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:02.849994  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:02.850730  5557 rpcz_store.cc:276] 0709 12:27:01.097591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.806260 (+1708669us) service_pool.cc:224] Handling call
0709 12:27:02.839933 (+ 33673us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.847678 (+  7745us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:02.847728 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:02.847793 (+    65us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:27:02.847853 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.847936 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:02.848078 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.796897 11479 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:02.855140  5555 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 12. Current term is 13. Ops: []
I20260709 12:27:02.855926  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2426) took 1771 ms. Trace:
I20260709 12:27:02.856326  5555 rpcz_store.cc:276] 0709 12:27:01.084633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.801091 (+1716458us) service_pool.cc:224] Handling call
0709 12:27:02.831199 (+ 30108us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.855638 (+ 24439us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.855860 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.843379 11529 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:27:02.861752  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56806.8us
I20260709 12:27:02.864898  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:02.863574  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:02.866860  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:02.847179 11489 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:27:02.871062  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77548.3us
I20260709 12:27:02.853562 11467 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:02.876680 11374 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:02.880474 11356 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:02.881561  6271 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: []
I20260709 12:27:02.852677  5627 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "733a5b54e4dd4f86999821705033529f"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:02.901762  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2423) took 2367 ms. Trace:
I20260709 12:27:02.885124 11436 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.854455 11534 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:02.897197  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:02.898200  5557 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:27:02.899142  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2556) took 1658 ms. Trace:
I20260709 12:27:02.905311  5553 rpcz_store.cc:276] 0709 12:27:01.241224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.826507 (+1585283us) service_pool.cc:224] Handling call
0709 12:27:02.847543 (+ 21036us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.898684 (+ 51141us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:02.898725 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:02.898786 (+    61us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:02.898837 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.898922 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:02.899072 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.909237  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1059) took 1540 ms. Trace:
I20260709 12:27:02.909744  5552 rpcz_store.cc:276] 0709 12:27:01.369110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.847848 (+1478738us) service_pool.cc:224] Handling call
0709 12:27:02.869885 (+ 22037us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.908738 (+ 38853us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:02.908784 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:02.908876 (+    92us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:02.908930 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.909021 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:02.909162 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.911541  5552 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:02.900512  5560 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:02.902261  5556 rpcz_store.cc:276] 0709 12:27:00.534554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.650157 (+2115603us) service_pool.cc:224] Handling call
0709 12:27:02.801461 (+151304us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:02.802213 (+   752us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.901574 (+ 99361us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b5400026cd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:02.901706 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":346902400}
I20260709 12:27:02.863003 11487 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:02.901868  5627 raft_consensus.cc:606] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:27:02.913563  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2431) took 1428 ms. Trace:
I20260709 12:27:02.917160  5557 rpcz_store.cc:276] 0709 12:27:01.485419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.854605 (+1369186us) service_pool.cc:224] Handling call
0709 12:27:02.880156 (+ 25551us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:02.913219 (+ 33063us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.913493 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.913674  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2430) took 1801 ms. Trace:
I20260709 12:27:02.918428  5560 rpcz_store.cc:276] 0709 12:27:01.112794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.822884 (+1710090us) service_pool.cc:224] Handling call
0709 12:27:02.861687 (+ 38803us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.913361 (+ 51674us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.913610 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.916134 11487 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:02.920518 11487 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:02.921150  5556 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:27:02.922936  5556 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:27:02.927493  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2433) took 1092 ms. Trace:
I20260709 12:27:02.928639  5552 rpcz_store.cc:276] 0709 12:27:01.824857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.910929 (+1086072us) service_pool.cc:224] Handling call
0709 12:27:02.911324 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.916921 (+  5597us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.917188 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.924808  5701 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:02.930836 11362 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:02.939364  5557 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:02.941534  5709 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:27:02.913136 11534 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:02.963701  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:02.964139  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:02.965068  6272 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:02.924474 11489 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:02.940490 11436 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:02.976511  6271 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:02.912304 11479 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:02.984782  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:02.936558 11549 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:27:02.986783  5552 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:02.986407  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:02.990600  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:02.991554  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:02.995393  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1058) took 1630 ms. Trace:
I20260709 12:27:02.996060  5558 rpcz_store.cc:276] 0709 12:27:01.364551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.838265 (+1473714us) service_pool.cc:224] Handling call
0709 12:27:02.902605 (+ 64340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.987083 (+ 84478us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:02.987108 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:02.994693 (+  7585us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:02.994784 (+    91us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.994880 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:02.995225 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:02.998821  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:03.003325 11489 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:03.008533  6272 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:03.009462  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:03.010823  6271 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 17: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:03.015942  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:03.019122  5628 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:02.947150 11356 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 12:27:03.027019 11356 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:27:03.027402 11356 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:27:03.028512  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:03.029066  5552 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:27:03.030790  5709 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:27:03.031483  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:03.032765  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:03.036293  5634 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:27:03.039611  5710 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: [11.6-13.7]
I20260709 12:27:03.041508  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:03.042088  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:03.042873  5704 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:03.043457  5710 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:03.044325  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:03.044807  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:03.046213  5708 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:27:03.046888  5701 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:02.942472 11374 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.106s
I20260709 12:27:03.048899 11374 consensus_queue.cc:1243] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:27:03.050101  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:03.049104 11362 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:27:03.051157  5557 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:27:03.051227 11362 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:03.052343  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:03.052992  5557 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:03.052892 11209 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:03.052651 11362 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:03.053629 11209 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:03.053891  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:03.054495  5557 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:27:03.054618 11209 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:03.056658 11365 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:03.057260 11365 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:03.057477  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:03.061408  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:03.062918 11365 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:02.980130 11534 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:27:03.065061  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49348.6us
I20260709 12:27:03.065253  5557 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:27:03.065874 11475 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:03.066293  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2544) took 2063 ms. Trace:
I20260709 12:27:03.073323  5558 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:27:03.073796  5556 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:27:03.120671  5561 rpcz_store.cc:276] 0709 12:27:01.003111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.766365 (+1763254us) service_pool.cc:224] Handling call
0709 12:27:02.810244 (+ 43879us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:02.830892 (+ 20648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:03.066056 (+235164us) spinlock_profiling.cc:243] Waited 43.4 ms on lock 0x7b54001c5cd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:03.066207 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":99911936}
I20260709 12:27:03.078315  5703 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:03.028261 11356 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:03.080610  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:03.148939  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 396769us
I20260709 12:27:03.091979  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:03.092425 11487 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:03.096380 11266 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:03.159837  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:03.096915 11551 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15.
I20260709 12:27:03.048180  5708 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
W20260709 12:27:03.097532  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 322611us
I20260709 12:27:03.097868  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:03.098742  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:03.105324  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:03.106185  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:03.114348  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:03.171337  5709 raft_consensus.cc:2368] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12.
I20260709 12:27:03.148402 11223 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:03.172581  5708 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15. Ops: [11.7-14.9]
I20260709 12:27:03.177100 11266 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:03.177553 11266 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:03.177997 11487 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:03.178961  6276 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:03.178469 11266 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:03.179798 11266 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:03.179961  5704 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:03.119220 11374 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.187630 11374 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:03.120582 11479 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:03.163275  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 178611us
I20260709 12:27:03.176196 11272 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:03.185086 11379 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:03.194093 11379 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:03.195112 11379 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:03.185472  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 87709us
I20260709 12:27:03.200469 11475 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:27:03.201403 11477 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:27:03.201579  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 50577.6us
I20260709 12:27:03.201874 11477 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:03.186795  5701 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:27:03.202896 11551 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:27:03.202953 11477 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:03.203775 11420 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.013s
I20260709 12:27:03.204478 11477 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:03.204689 11420 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:03.205191 11420 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:03.205615  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:03.205667 11549 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:27:03.206085 11420 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:03.206705 11549 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:03.207108 11549 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:03.207145 11374 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:03.175534  5710 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:03.172266 11489 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:03.192605 11479 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:03.216071  5599 leader_election.cc:400] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12.
I20260709 12:27:03.207342 11420 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:03.149264 11365 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:03.191663  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:03.217633  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 43620.6us
I20260709 12:27:03.208113  6279 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:03.229863 11489 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:03.230675 11453 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.686s	user 0.012s	sys 0.000s
I20260709 12:27:03.231000 11489 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:03.208406 11446 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:03.232365 11446 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:03.231506  5552 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:27:03.227964  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:03.204604 11551 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:27:03.165748 11529 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 12:27:03.243624  5635 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:03.246362  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2428) took 2160 ms. Trace:
I20260709 12:27:03.249110  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:03.253017  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:03.174556 11467 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:03.232785 11475 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:03.200891 11554 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:27:03.241071 11446 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:03.216485 11487 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:03.243109 11453 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:03.246969 11551 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:03.278743 11559 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:03.282719  5560 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:03.235587 11374 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 12:27:03.258240 11475 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:03.291972  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 82894.9us
I20260709 12:27:03.275645 11480 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:03.271580 11453 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:03.311082  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 444299us
I20260709 12:27:03.314023  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:03.286414 11559 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:03.296655 11475 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:03.291714 11529 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:27:03.339785  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 480470us
I20260709 12:27:03.316522 11453 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:03.246745 11549 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:03.328979 11471 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:03.429986  6276 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:03.406278  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2432) took 1915 ms. Trace:
W20260709 12:27:03.411136  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:03.320041 11480 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:03.422081 11570 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.121s
W20260709 12:27:03.423408  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 362059us
I20260709 12:27:03.425014 11549 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:03.404120 11198 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.317023 11374 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.105s
I20260709 12:27:03.430804  5555 rpcz_store.cc:276] 0709 12:27:01.491058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.862124 (+1371066us) service_pool.cc:224] Handling call
0709 12:27:02.927115 (+ 64991us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.976816 (+ 49701us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:27:02.977358 (+   542us) log.cc:844] Serialized 14 byte log entry
0709 12:27:03.405890 (+428532us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:03.405965 (+    75us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:03.406022 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:03.406095 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:03.406220 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":359,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":12099,"thread_start_us":276600,"threads_started":1}
I20260709 12:27:03.436409  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:03.333981 11529 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:03.331396 11559 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:03.305094 11487 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:03.440644 11550 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.575s	user 0.005s	sys 0.000s
W20260709 12:27:03.441867 11567 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 12:27:03.442569 11198 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:03.457300  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:03.459969  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 211076us
W20260709 12:27:03.475656  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 161793us
I20260709 12:27:03.476876  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:03.497879  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:03.411018 11551 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:03.464413 11362 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:03.655388  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:03.626709 11209 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.036s	sys 0.039s
W20260709 12:27:03.631669 11567 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.168s	user 0.000s	sys 0.000s
W20260709 12:27:03.676067 11209 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.036s	sys 0.039s
I20260709 12:27:03.461416 11575 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.219s
I20260709 12:27:03.665863 11471 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:03.698570  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 264059us
W20260709 12:27:03.711635  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:03.679571 11467 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:03.750660  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:03.748617 11551 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:27:03.679999 11362 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:03.795400 11571 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:03.795953 11571 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:03.796442 11571 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:03.766548 11529 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:03.801821 11529 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:03.774608 11209 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:27:03.811411  5706 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:03.817229 11529 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:27:03.823691  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.005s	sys 0.012s
W20260709 12:27:03.841768  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.006s	sys 0.012s
W20260709 12:27:03.842669  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 601154us
W20260709 12:27:03.845348 11580 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:27:03.845526  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:03.699587 11480 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:03.849295 11572 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.153s	user 0.000s	sys 0.000s
I20260709 12:27:03.850960  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2514) took 1062 ms. Trace:
I20260709 12:27:03.851565  5705 rpcz_store.cc:276] 0709 12:27:02.788426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.788631 (+   205us) service_pool.cc:224] Handling call
0709 12:27:02.808664 (+ 20033us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:02.808942 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:02.808979 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:02.834113 (+ 25134us) log.cc:844] Serialized 26 byte log entry
0709 12:27:03.300450 (+466337us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:03.300486 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:03.300570 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:03.850735 (+550165us) raft_consensus.cc:1685] finished
0709 12:27:03.850754 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:03.850877 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":438511,"threads_started":1,"wal-append.queue_time_us":640876}
I20260709 12:27:03.717862 11471 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:03.823094 11495 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.008s	sys 0.010s
I20260709 12:27:03.791980 11588 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:03.877080  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 400308us
I20260709 12:27:03.839278 11529 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:27:03.892474 11495 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.009s	sys 0.010s
W20260709 12:27:03.909395  5674 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.003s	sys 0.012s
W20260709 12:27:03.911737  5674 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.004s	sys 0.012s
W20260709 12:27:03.907522 11567 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 12:27:03.928406  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:03.935835  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:03.936743  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:03.248507  5554 rpcz_store.cc:276] 0709 12:27:01.085946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.803030 (+1717084us) service_pool.cc:224] Handling call
0709 12:27:02.818996 (+ 15966us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:02.826162 (+  7166us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:27:02.868781 (+ 42619us) log.cc:844] Serialized 14 byte log entry
0709 12:27:03.245571 (+376790us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:03.245668 (+    97us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:03.245767 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:03.246093 (+   326us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:03.246286 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":496,"prepare.queue_time_us":23678,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":5399,"raft.queue_time_us":5370,"thread_start_us":305579,"threads_started":1}
I20260709 12:27:03.937960  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:03.938642 11587 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:27:03.939534  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 94121us
W20260709 12:27:03.939893 11477 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.735s	user 0.008s	sys 0.002s
I20260709 12:27:03.942008 11356 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:03.942059  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:03.909960 11487 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:03.917004 11581 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:03.918045 11198 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:03.915997 11588 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:03.961160  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2418) took 1390 ms. Trace:
I20260709 12:27:03.961690  5634 rpcz_store.cc:276] 0709 12:27:02.570821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.576758 (+  5937us) service_pool.cc:224] Handling call
0709 12:27:02.577214 (+   456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:03.036773 (+459559us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:03.960800 (+924027us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b54000c1f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:03.961056 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1054736256}
I20260709 12:27:03.963088 11588 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:03.964314 11362 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:03.966048 11471 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:03.967134 11581 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:03.967717 11593 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:03.968868 11529 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:03.969991 11581 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:03.970582 11591 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.113s
I20260709 12:27:03.971577 11591 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:03.971905 11591 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:03.972805 11591 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:03.973634 11266 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.763s	user 0.012s	sys 0.000s
I20260709 12:27:03.974220 11591 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:03.974560 11495 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:27:03.975921 11477 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:27:03.976586  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2546) took 2949 ms. Trace:
I20260709 12:27:03.970177 11529 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:03.944732 11534 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.793s	user 0.013s	sys 0.000s
I20260709 12:27:03.977063  5559 rpcz_store.cc:276] 0709 12:27:01.027477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.768143 (+1740666us) service_pool.cc:224] Handling call
0709 12:27:02.768544 (+   401us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:02.768970 (+   426us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:27:02.769331 (+   361us) log.cc:844] Serialized 14 byte log entry
0709 12:27:03.106236 (+336905us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:03.120307 (+ 14071us) log.cc:844] Serialized 52 byte log entry
0709 12:27:03.122087 (+  1780us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:27:03.122367 (+   280us) log.cc:844] Serialized 14 byte log entry
0709 12:27:03.122583 (+   216us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:03.122671 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:03.809460 (+686789us) spinlock_profiling.cc:243] Waited 4.76 ms on lock 0x7b18000d4140. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:27:03.976170 (+166710us) raft_consensus.cc:1685] finished
0709 12:27:03.976199 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:03.976447 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":25660,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":108,"raft.queue_time_us":577,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":13370,"spinlock_wait_cycles":10941312,"thread_start_us":336518,"threads_started":1,"wal-append.queue_time_us":482645}
W20260709 12:27:03.977175 11579 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 12:27:03.987872  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45909us
I20260709 12:27:03.988909  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:03.989315 11589 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 12:27:03.990376 11471 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:27:03.990594 11475 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.546s	user 0.004s	sys 0.004s
I20260709 12:27:03.991091 11475 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:03.991516  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:03.992976 11471 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:03.997342 11475 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:04.000130 11479 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:04.005698 11453 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.568s	user 0.010s	sys 0.000s
I20260709 12:27:04.006188 11453 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.005076  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:04.008968  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:04.009244 11602 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:04.009872 11602 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:04.010147 11475 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:27:04.011627 11420 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.789s	user 0.007s	sys 0.003s
I20260709 12:27:04.011281 11602 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.011828  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:04.009876 11471 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:04.012943 11471 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:04.013326 11471 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:04.015264 11576 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.601s	user 0.007s	sys 0.020s
I20260709 12:27:04.021718 11575 consensus_queue.cc:1243] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:27:04.025784 11575 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:04.024580 11420 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:27:04.023634  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:04.037091  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:04.038334  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:04.050614  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 110906us
W20260709 12:27:04.054613  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 198527us
W20260709 12:27:04.046190  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:04.054936  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:04.020215 11446 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.733s	user 0.009s	sys 0.000s
I20260709 12:27:04.058044  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1060) took 2431 ms. Trace:
I20260709 12:27:04.041282 11575 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:04.058517  5553 rpcz_store.cc:276] 0709 12:27:01.627190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.906443 (+1279253us) service_pool.cc:224] Handling call
0709 12:27:02.906903 (+   460us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:02.907297 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:02.907334 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:02.907959 (+   625us) log.cc:844] Serialized 26 byte log entry
0709 12:27:03.245038 (+337079us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:03.464174 (+219136us) log.cc:844] Serialized 14 byte log entry
0709 12:27:03.464415 (+   241us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:03.464556 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:04.014572 (+550016us) spinlock_profiling.cc:243] Waited 32.3 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:04.050869 (+ 36297us) raft_consensus.cc:1685] finished
0709 12:27:04.057776 (+  6907us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:04.057967 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":564429,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":36,"raft.queue_time_us":291827,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":127,"spinlock_wait_cycles":74338048,"thread_start_us":527326,"threads_started":2,"wal-append.queue_time_us":505929}
I20260709 12:27:04.045053 11479 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:27:04.078269  5555 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.035s	sys 0.071s
W20260709 12:27:04.078668  5555 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.035s	sys 0.071s
W20260709 12:27:04.078229  5527 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.010s	sys 0.021s
W20260709 12:27:04.085924  5527 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.010s	sys 0.021s
W20260709 12:27:04.086613  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:04.056785 11446 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.089670  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:04.090618  5702 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:27:04.045162 11600 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 12:27:04.098220  5703 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:27:04.028246 11534 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.098569  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:04.099086  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 240757us
I20260709 12:27:04.099290  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:04.103351  5707 raft_consensus.cc:2376] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15. Re-sending same reply.
I20260709 12:27:04.105543  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:04.076022 11575 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:27:04.075734 11570 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.002s	sys 0.018s
W20260709 12:27:04.111259 11570 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.002s	sys 0.018s
I20260709 12:27:04.111712 11570 consensus_queue.cc:1243] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:27:04.083060 11365 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.085022 11420 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:27:04.098716 11559 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.619s	user 0.012s	sys 0.001s
I20260709 12:27:04.117982  5558 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:04.118083  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:27:04.118744  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:04.121961  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:04.087754 11198 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:04.060554 11453 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.089778 11209 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:04.106678 11362 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:04.117357 11549 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.667s	user 0.012s	sys 0.000s
I20260709 12:27:04.057405 11594 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.060s
I20260709 12:27:04.201727 11198 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.206838 11613 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:04.209981  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:04.212604  5553 raft_consensus.cc:2376] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18. Re-sending same reply.
W20260709 12:27:04.210216  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 92346.9us
I20260709 12:27:04.214238  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:04.231925  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 126486us
I20260709 12:27:04.173132 11479 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:04.171527 11587 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.207082 11570 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:27:04.239027  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:04.252183  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 194574us
W20260709 12:27:04.282011  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 225498us
I20260709 12:27:04.201269 11365 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.171945 11559 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.239360 11479 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:04.125303 11593 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:04.216598 11223 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.083s
I20260709 12:27:04.233438 11581 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.068s
I20260709 12:27:04.327562 11581 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:27:04.292619 11570 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:04.328533 11570 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:04.294137 11595 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:04.329401  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 112781us
I20260709 12:27:04.329233 11595 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.329710 11570 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.331133 11570 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:04.331622 11595 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.211274 11362 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.335209 11362 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:04.298722 11420 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.325311  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:04.311180 11356 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.314908 11479 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:04.335903 11534 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:04.253541 11587 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:04.339506  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 388491us
I20260709 12:27:04.343650 11587 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.339489 11420 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.281960 11606 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.107s
I20260709 12:27:04.347186 11420 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.311425 11594 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:04.348284  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 993) took 1489 ms. Trace:
I20260709 12:27:04.341400 11595 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:04.349020 11446 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:04.343545  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:04.348121 11606 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:04.316797 11607 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.338973  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:04.348899  5706 rpcz_store.cc:276] 0709 12:27:02.858663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.863332 (+  4669us) service_pool.cc:224] Handling call
0709 12:27:04.348069 (+1484737us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b5400324454. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:04.348191 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1235745024}
I20260709 12:27:04.340351 11617 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:04.357208 11597 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:04.339387 11603 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.247s
I20260709 12:27:04.360097 11612 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:27:04.360397 11209 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.362094 11615 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:27:04.320593 11609 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:04.350108 11356 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:04.375556  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 250324us
W20260709 12:27:04.389446  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45981.4us
W20260709 12:27:04.395320  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 60901.8us
I20260709 12:27:04.393923 10871 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:27:04.400777  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:04.364342 11571 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.567s	user 0.009s	sys 0.000s
I20260709 12:27:04.391386 11617 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:04.375298 11606 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:04.410339  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 71504.9us
I20260709 12:27:04.365373 11365 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:04.371369 11559 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.395179 11487 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:04.397296 11609 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:27:04.424822  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 138967us
I20260709 12:27:04.425958 11365 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:04.427840 11365 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:27:04.434481  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:04.348798 11594 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:04.419377 11551 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.624s	user 0.010s	sys 0.003s
I20260709 12:27:04.400897 11480 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:04.407482 11362 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.439263 11621 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:27:04.440495  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:04.431416 11467 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.444716  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:04.451900  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:27:04.436861  5675 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.871s (SENT)
I20260709 12:27:04.460830  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:04.454299  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:04.432267 11609 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:04.455739  5556 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:04.459731  5554 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 10.
W20260709 12:27:04.467056  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 105747us
I20260709 12:27:04.418159 11617 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:04.468794  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:04.448609 11223 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:04.408079 11571 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.477403  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:04.478041  5554 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:04.478892  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:04.479825  5553 raft_consensus.cc:2410] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 17 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:27:04.480036  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:04.395815 11597 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.397591 11607 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.436021 11487 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:04.485339  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47280.7us
W20260709 12:27:04.485870  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:04.467674 11467 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.489251  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:04.494061  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 81613.2us
I20260709 12:27:04.476673 11480 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:04.498556  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:04.501402  5675 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.954s (SENT)
I20260709 12:27:04.504585  5556 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:04.509267  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:04.511150 11503 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:04.511528 11503 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:04.500339 11595 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:04.524204 11487 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:04.500841 11480 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:04.525666  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:04.526427  5557 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:04.528821  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:39496 (request call id 37) took 1723 ms. Trace:
I20260709 12:27:04.529206  5627 rpcz_store.cc:276] 0709 12:27:02.826244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.831967 (+  5723us) service_pool.cc:224] Handling call
0709 12:27:04.528543 (+1696576us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b5400294620. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5ed5e7 00007f4ffa66cc68 00007f4ffa66c84a 00007f4fff63a60a 00007f4ff8bc07d4 00007f4ff8bc0762 00007f4ff8bc0691 00007f4ff8bc060c 00007f4ff8bbf8c2 00007f4ff89ad48c
0709 12:27:04.528740 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":255290240}
I20260709 12:27:04.530866 11420 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.531389 11420 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.525874 11480 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:04.533567  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:04.534723  5557 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:27:04.540879  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2559) took 2375 ms. Trace:
I20260709 12:27:04.481871 11362 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.541401  5560 rpcz_store.cc:276] 0709 12:27:02.165094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.925550 (+760456us) service_pool.cc:224] Handling call
0709 12:27:02.939018 (+ 13468us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:03.283273 (+344255us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:04.532338 (+1249065us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b540044e654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:04.540812 (+  8474us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":789829888}
I20260709 12:27:04.575295  5554 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
W20260709 12:27:04.575938  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:04.489272 11621 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
W20260709 12:27:04.533056 11471 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.519s	user 0.010s	sys 0.004s
W20260709 12:27:04.603498  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55423.3us
I20260709 12:27:04.487200 11223 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.516319 11559 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:04.608686  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:04.609678  5561 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:27:04.541846 11611 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.474880 11593 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.586169 11595 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.495930 11606 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.620882 11606 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:04.624601 11620 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:04.626116 11619 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.626860 11619 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.628544 11618 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:04.629168 11618 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.604714 11266 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.619720  5554 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:27:04.652024 11266 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.654485 11266 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:04.655676 11591 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.681s	user 0.007s	sys 0.008s
W20260709 12:27:04.636986 11622 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
W20260709 12:27:04.640852  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 146156us
I20260709 12:27:04.609930 11356 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.645251 11467 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.646100 11223 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:04.512341 11503 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.647362  5705 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:04.613467 11559 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.649736 11607 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:04.651329 11362 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:04.641654 11594 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:04.663614  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:04.664548  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:04.610953 11487 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:04.667611  5556 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 12:27:04.670851  5524 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:04.672806  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:04.679657 11559 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.610471 11471 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.693516  5704 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:04.684279 11618 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:04.700440  5675 leader_election.cc:336] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.178s (SENT)
I20260709 12:27:04.701215  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:04.706208 11598 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:04.706696 11598 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:04.708839 11503 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:27:04.709499 11564 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.376s	user 0.025s	sys 0.000s
I20260709 12:27:04.709775 11471 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:04.711495 11602 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:04.711585  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1057) took 2030 ms. Trace:
I20260709 12:27:04.712018 11602 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:04.707576 11598 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.717959  5556 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18. Ops: []
W20260709 12:27:04.703390 11529 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.686s	user 0.009s	sys 0.005s
I20260709 12:27:04.656260 11595 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:04.719584  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48836.7us
I20260709 12:27:04.618875 11611 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.722044  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:04.712109  6278 rpcz_store.cc:276] 0709 12:27:02.681238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.687691 (+  6453us) service_pool.cc:224] Handling call
0709 12:27:02.718120 (+ 30429us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:02.718640 (+   520us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:02.718678 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:02.719169 (+   491us) log.cc:844] Serialized 26 byte log entry
0709 12:27:02.979660 (+260491us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:02.979700 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:02.979771 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:04.708697 (+1728926us) spinlock_profiling.cc:243] Waited 41.5 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:04.711255 (+  2558us) raft_consensus.cc:1685] finished
0709 12:27:04.711277 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:04.711515 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3081,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"spinlock_wait_cycles":95364864,"thread_start_us":260021,"threads_started":1,"wal-append.queue_time_us":565691}
I20260709 12:27:04.694489 11467 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.019s
I20260709 12:27:04.717912  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2578) took 1766 ms. Trace:
I20260709 12:27:04.712772 11446 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:04.731647 11446 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:04.732219 11559 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:04.732911 11420 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.732803 11559 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.726526  5552 rpcz_store.cc:276] 0709 12:27:02.951361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:03.029852 (+ 78491us) service_pool.cc:224] Handling call
0709 12:27:03.030140 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:03.233275 (+203135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:04.713333 (+1480058us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b54001bec54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:04.717845 (+  4512us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":462368512}
I20260709 12:27:04.734906 11559 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.735651  5552 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:27:04.737110 11597 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.719955 11581 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:27:04.740706  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1067) took 1652 ms. Trace:
I20260709 12:27:04.741238  5558 rpcz_store.cc:276] 0709 12:27:03.088692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:03.166595 (+ 77903us) service_pool.cc:224] Handling call
0709 12:27:03.189984 (+ 23389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:04.124174 (+934190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:04.740330 (+616156us) spinlock_profiling.cc:243] Waited 928 ms on lock 0x7b54002279d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:04.740622 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2133572480}
I20260709 12:27:04.743149 11549 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:27:04.743732 11549 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:04.744618 11571 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.744975 11549 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:04.750146  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:04.750774 11626 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:27:04.751868 11446 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.755576  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:04.756279 11374 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:04.756830 11374 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:04.757944 11374 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.758764  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:04.759544 11374 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:04.759581  6279 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:04.761153 10871 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:04.761586 10871 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:04.762169  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:04.762645 10871 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.764269  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:04.730967 11595 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.766170 11602 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.767170  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:04.767930 11602 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:04.769299  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:04.771988  5633 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:04.774281  5634 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:04.775784  5628 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:04.750247 11611 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:04.777223 11611 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.747077 11549 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:04.779381  6271 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 12.
I20260709 12:27:04.764305 10871 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:04.791039  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:04.795099  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:04.796254  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:04.797473  5634 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:27:04.797842  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:04.796312  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:04.795069  5631 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:04.800873  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:04.802948  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 105373us
I20260709 12:27:04.803730  5628 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:27:04.727864 11471 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:27:04.804526 11471 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:04.805027 11471 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:04.800312  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:04.809562  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:04.810365  5636 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:04.811239  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:04.812107  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:04.812274  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:04.812669 11480 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.817133  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:04.668241 11609 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:04.826526  5675 leader_election.cc:336] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.233s (SENT)
I20260709 12:27:04.828074  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:04.828413  6275 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:27:04.828437  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:04.823292  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:04.829880  6276 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:04.830170  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:04.832209 11615 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:27:04.832778 11575 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.709s	user 0.009s	sys 0.002s
I20260709 12:27:04.833280 11575 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.834043  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:04.834640  5630 raft_consensus.cc:2376] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12. Re-sending same reply.
W20260709 12:27:04.838239  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:04.841738  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:04.844249  5553 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:04.846541  6279 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20260709 12:27:04.846935  6279 pending_rounds.cc:85] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 13
W20260709 12:27:04.840672 11578 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.075s	user 0.014s	sys 0.008s
I20260709 12:27:04.777943 11611 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.850571  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:04.886186  5628 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:04.832816 11615 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:04.886439  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:04.853161 11577 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.427s	user 0.013s	sys 0.022s
I20260709 12:27:04.853673  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:04.894500  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:04.894738  5630 raft_consensus.cc:2410] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:27:04.854251  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:04.897054  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1068) took 1808 ms. Trace:
I20260709 12:27:04.854287  5704 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:27:04.856037  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:04.856405  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:04.900336  5705 raft_consensus.cc:2376] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12. Re-sending same reply.
I20260709 12:27:04.900624  5703 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 13: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:04.856719  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:04.857997  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:04.901510  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:04.901906  5708 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:04.902223  6278 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:27:04.902601 11615 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.904027 11615 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:04.848452 11495 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.904943 11495 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.907091 11495 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:04.908047 11198 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.848176 11436 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.862371 11471 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.863384  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:04.848776 11598 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:04.868516 11619 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:04.869781 11479 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.928550  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:04.929715  5633 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:04.930188 11479 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:04.871706  5635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2670) took 2132 ms (client timeout 2255 ms). Trace:
I20260709 12:27:04.930884  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:04.931042  5635 rpcz_store.cc:269] 0709 12:27:02.739313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:02.808274 (+ 68961us) service_pool.cc:224] Handling call
0709 12:27:04.871477 (+2063203us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b54001f2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:04.871603 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":822188800}
I20260709 12:27:04.932006  5633 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:27:04.932737  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:04.933440  5635 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:04.872484 11575 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:04.872726  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:04.933461  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:04.876739 11489 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:04.878507 11446 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.942848  5628 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:27:04.943259 11446 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.880048  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:04.945989  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 66062.5us
I20260709 12:27:04.878543 11480 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:04.948822 11480 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:04.949510 11479 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:04.950021 11609 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:04.951025 11479 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:04.935527 11436 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.951473 11479 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:04.880681 11600 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:27:04.952464 11600 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:04.883070  5627 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:04.942114  6276 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:04.858773  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:04.897513  5555 rpcz_store.cc:276] 0709 12:27:03.089006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:03.442915 (+353909us) service_pool.cc:224] Handling call
0709 12:27:03.443246 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:03.456547 (+ 13301us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:27:04.079027 (+622480us) log.cc:844] Serialized 14 byte log entry
0709 12:27:04.314811 (+235784us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:04.314880 (+    69us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:04.314938 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:04.896841 (+581903us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b54000267d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:04.896887 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:04.897000 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106875,"prepare.queue_time_us":353183,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":108253,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":4492,"spinlock_wait_cycles":29698944,"thread_start_us":725281,"threads_started":2,"wal-append.queue_time_us":258876}
I20260709 12:27:04.909142  5554 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14. Ops: []
W20260709 12:27:04.916754  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53465.5us
I20260709 12:27:04.953699  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:04.948134  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:04.954876  5636 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:27:04.924695 11617 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:04.926803 11559 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:27:04.927659  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:04.920529  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:04.936051  5559 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 18: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:27:04.936265  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 114038us
I20260709 12:27:04.942423 11489 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:04.847275  6279 pending_rounds.cc:107] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:27:04.963508  6279 raft_consensus.cc:2889] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:04.953967  5702 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:04.954649  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:04.921942 11495 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:04.925544  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:04.956427  5631 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:27:04.965340  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:04.971300  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:04.974473  5708 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:27:04.986891  5706 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:04.991197  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:04.994277  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:04.994665  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:04.997870  5710 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:27:05.002419  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:05.003413  5702 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:05.007285  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:04.957311 11595 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.994102 11551 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:04.941468 11198 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.054s
I20260709 12:27:04.934906 11272 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
I20260709 12:27:05.016475  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:04.987672 11477 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:27:05.027689  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:05.029902  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 104427us
I20260709 12:27:04.979579 11575 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:27:04.972445 11475 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:05.027212  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:04.997367 11619 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:04.996984 11495 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:05.064170  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:05.065887  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:05.068166  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:05.072960  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "832269e5980743c48521740b22eefa9f"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:05.074048  5630 raft_consensus.cc:606] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:27:05.074496  5630 raft_consensus.cc:614] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:27:05.078027  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8b21feec8630479b94ed9384256483fc"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:05.078686  5630 raft_consensus.cc:606] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:27:05.079474  5630 raft_consensus.cc:614] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:27:05.081226  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:05.082562  5635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:05.083122  5635 raft_consensus.cc:606] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:27:05.084759  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:27:05.061095 11594 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.088922 11594 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:05.091867 11594 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:05.054188 11570 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.092847 11570 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:05.094643 11570 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:27:05.026885 11571 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.095395 11571 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:05.064402  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:05.070423  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:05.069780  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:05.068758  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:05.029876 11356 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:04.979115 11420 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.069s
I20260709 12:27:05.097107  5631 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:27:05.002496 11365 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 12:27:05.023988 11436 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:05.070474 11495 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:05.048843 11601 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:05.050802 11487 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260709 12:27:05.115979  5556 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:05.025767 11209 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.080s
I20260709 12:27:05.055244 11575 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:05.025398 11581 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.075s
I20260709 12:27:05.056146 11619 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:05.137198  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:05.052909 11610 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.074s
I20260709 12:27:05.062706 11477 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:05.144093  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:05.144315  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:05.168025  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 168078us
I20260709 12:27:05.144764  5455 catalog_manager.cc:5697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 7 to 11, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 11 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:27:05.145941 11581 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:05.146677  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:05.149682 11554 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:05.140446 11575 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:05.150739 11619 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:05.140527  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:05.154588 11617 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:05.156635 11489 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:05.157835  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.158998  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.160810 11611 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:05.161888  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:05.162420 11503 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.003s
I20260709 12:27:05.163480 11266 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:05.106148 11356 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:05.164831  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.202945  5707 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:27:05.031337 11600 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:05.142184  5552 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:05.036252 11609 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.082s
I20260709 12:27:05.033470 11595 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:05.166615  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.015s	sys 0.061s
W20260709 12:27:05.166543  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 108989us
I20260709 12:27:05.145107  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:05.183259 11554 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:05.197760  5704 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
W20260709 12:27:05.200825  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 220710us
I20260709 12:27:05.035264 11551 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:05.252601  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:05.252743  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.253947  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.255786  5552 raft_consensus.cc:2410] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:27:05.203307  5710 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:05.164182 11585 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 12:27:05.260859  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:05.205118  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:05.223181  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.015s	sys 0.061s
I20260709 12:27:05.223726  5558 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:05.182848 11581 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:05.222441  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:05.265919  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.268458  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.255620  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:05.258800  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:05.259859  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:05.196707  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:05.262369  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:05.259295  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.270179  5559 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:05.271653  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:05.272562  5709 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:05.273491  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:05.276729  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54454.1us
I20260709 12:27:05.276729  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:05.279069  6276 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:27:05.265256  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.269912  5560 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:05.206787 11356 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:05.306190  5631 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:27:05.307179  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:05.307842 11570 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.308386 11570 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:05.310151  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.310491  5627 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:27:05.310972  5703 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:05.312017  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:05.312731  5630 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:27:05.313104 11585 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:27:05.313649 11585 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:05.313879  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:05.314775  5632 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:05.314848 11585 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:05.315711  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:05.224223 11571 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:05.282970  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.312136  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.318997  5703 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:05.253623 11467 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.288750 11600 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:05.254223 11587 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.294878 11554 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:05.295768 11551 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:05.256260 11489 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:27:05.253852 11475 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:05.221164 11365 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.075s
I20260709 12:27:05.297802 11595 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:05.210440 11618 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.088s
I20260709 12:27:05.299335  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:05.193156 11601 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.066s
I20260709 12:27:05.300591 11374 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:05.301451  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:05.301878  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.303092 11636 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:05.305523 11643 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:05.308708  6275 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:05.281761  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:05.319087  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:05.319866  5710 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
W20260709 12:27:05.421393  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 102400us
I20260709 12:27:05.284257 11477 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:05.283500  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:05.326337  5708 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:05.329875  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:05.330673  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:05.326776  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 273276us
W20260709 12:27:05.424265  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 94448.5us
I20260709 12:27:05.322016 11617 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:05.387962  5701 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:27:05.388867  6279 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:27:05.389552  5709 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:05.425800  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:05.459013  5708 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 10. (term mismatch)
I20260709 12:27:05.459626  5708 pending_rounds.cc:85] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 9
I20260709 12:27:05.459941  5708 pending_rounds.cc:107] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 14.10
I20260709 12:27:05.460305  5708 raft_consensus.cc:2889] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:05.335602 11585 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:05.478179  5560 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
W20260709 12:27:05.499971  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 74254.9us
I20260709 12:27:05.500998  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:05.482944  5635 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:05.394042 11374 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:05.326952 11467 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:05.531363  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:05.545378  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:05.529614  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 247968us
W20260709 12:27:05.545768  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 86550us
I20260709 12:27:05.546077  6272 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
W20260709 12:27:05.546427  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:05.546594  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:05.546820  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:05.387722 11636 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:05.545794  5635 raft_consensus.cc:2376] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20. Re-sending same reply.
I20260709 12:27:05.393741 11643 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:05.549427  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:05.550045  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:05.336977 11587 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:05.555246  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47143.4us
I20260709 12:27:05.458169 11618 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:05.338660 11475 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:05.464530 11362 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:27:05.526716 11598 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.592s	user 0.011s	sys 0.000s
I20260709 12:27:05.608625  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:05.621172  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:05.565167  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:05.634313  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 87781.5us
I20260709 12:27:05.635275  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:05.637042  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 353614us
I20260709 12:27:05.531721 11633 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:05.638792  5634 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:27:05.639631  6279 raft_consensus.cc:2368] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18.
W20260709 12:27:05.584533 11645 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:27:05.684890  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:05.698139  5635 raft_consensus.cc:2376] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14. Re-sending same reply.
W20260709 12:27:05.676131  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 55073.7us
I20260709 12:27:05.550648 11570 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:05.700541  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:05.557229 11489 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:05.604411 11601 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:27:05.692039  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.707113  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:05.706948  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:05.542209 11602 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.753s	user 0.007s	sys 0.004s
I20260709 12:27:05.548471 11374 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:05.492175 11617 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:05.639077 11643 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:05.682116 11549 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.889s	user 0.010s	sys 0.002s
I20260709 12:27:05.690119 11475 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:05.570544 11636 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:05.637568 11362 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:05.737318  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:05.632344 11618 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:05.764716  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:05.707032 11489 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:05.707219 11209 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:05.702203 11436 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.699260 11480 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:05.776131  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58439.6us
I20260709 12:27:05.706167 11503 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:05.782333  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:05.697531 11646 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:05.766364 11362 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:05.765769 11595 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:27:05.719959 11633 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:27:05.784631 11628 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:27:05.774387 11489 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:05.774658 11618 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:05.778147 11209 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:05.825291  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:05.766201 11609 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 12:27:05.722174 11495 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.091s
I20260709 12:27:05.807965 11272 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:05.823983 11642 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:27:05.789639 11436 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:05.833768 11209 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:05.866763  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 320003us
I20260709 12:27:05.840201 11475 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:27:05.861120  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:05.804997 11503 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:05.875326  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:05.876674  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 139624us
I20260709 12:27:05.800871 11480 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:05.853693  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 288661us
I20260709 12:27:05.878738  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:05.846266 11272 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:05.880055 11671 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:05.890827  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:05.894699  5675 leader_election.cc:400] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18.
I20260709 12:27:05.895035  5675 leader_election.cc:403] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:27:05.861954 11669 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:27:05.893118 11639 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.390s	user 0.000s	sys 0.001s
I20260709 12:27:05.887848 11272 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:05.904882  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:05.931579  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53525.3us
I20260709 12:27:05.894699 11653 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:27:05.901681 11659 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:05.941660  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:05.933154  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:05.901875 11671 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:05.927789  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53415.7us
I20260709 12:27:05.904718 11633 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18.
I20260709 12:27:05.962363 11666 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:05.962507  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:05.952818  5635 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:05.915421 11467 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:05.940654 11587 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:05.949150 11648 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
W20260709 12:27:05.954979 11619 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.744s	user 0.010s	sys 0.000s
I20260709 12:27:05.982405  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.986321  5709 raft_consensus.cc:2410] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 17 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:27:05.949108 11673 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:05.988543  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:05.968840 11671 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:05.991020 11671 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:27:05.992942 11656 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 12:27:05.992617 11665 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
W20260709 12:27:05.994371 11651 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 12:27:06.002398 11619 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.004004  5706 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:06.008533 11587 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 12:27:06.013835  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51383.1us
I20260709 12:27:06.015583  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:05.962924 11666 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:06.016707  5635 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:05.928478 11663 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:06.019013  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:05.958002 11446 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:06.025393  5701 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
W20260709 12:27:06.023126  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 66281.3us
I20260709 12:27:05.911155 11669 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.033057  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:06.033905  5707 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:06.035120  5704 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:27:06.042249  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:06.044239  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:06.005505 11619 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:06.046284  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:06.011606 11503 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:06.011693 11653 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:06.053046 11653 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:06.052380 11671 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:27:06.053609 11671 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:06.054217 11671 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:06.054147 11653 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.057296 11436 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:06.058367 11644 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.219s	user 0.000s	sys 0.001s
I20260709 12:27:06.024227 11633 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.059440 11633 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.060981  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:06.061738 11633 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:06.025454 11663 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:06.062546 11663 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:06.062948  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:06.063695  5552 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:06.064952  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:06.018537 11467 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:27:06.054780 11312 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:06.065577  5627 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:06.015134 11617 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:06.064564 11650 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.314s	user 0.000s	sys 0.001s
W20260709 12:27:06.072755  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:06.038834 11362 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.023187 11666 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:06.077642  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:06.084961 11647 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 12:27:06.087450  5559 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:06.088815  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:06.089458  5627 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:06.045855 11480 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:06.092731 11549 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.093261 11549 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.094309 11581 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:06.095158 11356 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.789s	user 0.011s	sys 0.000s
I20260709 12:27:06.095346 11549 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.095531 11581 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:06.095897 11615 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.096416 11615 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.098099 11471 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.098495 11615 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.098645 11669 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.099022 11615 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:06.099493 11615 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.100713 11571 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.101441  5556 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 14 index: 6. (index mismatch)
I20260709 12:27:06.101250 11571 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.103214 11571 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.103659 11571 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.048691 11642 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.105461 11642 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.106410  5632 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:27:06.107591 11642 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.065223 11587 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:06.115442 11587 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:06.115940 11587 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:06.116623 11362 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:06.072911 11312 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:06.107564  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:06.123080  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2679) took 1272 ms. Trace:
I20260709 12:27:06.092108  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:06.123466  5634 rpcz_store.cc:276] 0709 12:27:04.883626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:04.883860 (+   234us) service_pool.cc:224] Handling call
0709 12:27:06.122872 (+1239012us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b54001dd8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:06.123022 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1712097536}
I20260709 12:27:06.124769  5633 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:27:06.124552  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:05.969179 11209 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.071096 11446 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:06.071228 11503 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.139065  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:06.142489  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:06.144455  6275 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:06.131105  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:06.139603  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:06.078845 11575 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.880s	user 0.011s	sys 0.001s
I20260709 12:27:06.095597 11356 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.146545  6279 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:06.140389  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:06.146749  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:06.147660  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 103521us
W20260709 12:27:06.140836  6234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.015s	sys 0.018s
W20260709 12:27:06.148762  6234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.015s	sys 0.018s
W20260709 12:27:06.152165  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 516935us
I20260709 12:27:06.149061 11575 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.146366 11487 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.120229 11479 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.150593 11611 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.016s
I20260709 12:27:06.121694 11312 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.141989  5627 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:06.145298  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:06.138782 11676 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:06.147301  5710 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:06.141141  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:06.150451  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:06.129654 11597 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:06.143939 11549 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:27:06.151336  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:06.152531  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:06.152959  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:06.155836  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:06.162156  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:06.173210  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2724) took 1015 ms. Trace:
I20260709 12:27:06.179993  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:06.525796  5709 raft_consensus.cc:2368] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14.
W20260709 12:27:06.523660  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 372377us
W20260709 12:27:06.524995  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 372098us
I20260709 12:27:06.197805  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:06.523660  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 371283us
I20260709 12:27:06.153219 11420 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:27:06.523533 11595 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:06.158545 11575 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:06.525939  5706 rpcz_store.cc:276] 0709 12:27:05.157697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:05.158288 (+   591us) service_pool.cc:224] Handling call
0709 12:27:05.158562 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:06.004438 (+845876us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:06.172931 (+168493us) spinlock_profiling.cc:243] Waited 845 ms on lock 0x7b54001a9354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:06.173130 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1943914368}
I20260709 12:27:06.171068 11618 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.527558  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:06.519268 11479 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:27:06.520334 11652 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.754s	user 0.002s	sys 0.002s
I20260709 12:27:06.531364 11559 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:27:06.195148 11666 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.169296 11209 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:06.200771 11549 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:06.533413 11549 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:27:06.534624 11666 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:06.536226 11618 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:06.204734  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:06.175566  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:06.182675  5675 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.507s (SENT)
I20260709 12:27:06.104075 11571 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.232875  6275 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:06.537555  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:06.196377  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:27:06.205350 11585 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.683s	user 0.014s	sys 0.000s
I20260709 12:27:06.238137  5632 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:06.541584  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2444) took 1037 ms. Trace:
I20260709 12:27:06.542016  5632 rpcz_store.cc:276] 0709 12:27:05.503976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.107349 (+603373us) service_pool.cc:224] Handling call
0709 12:27:06.541526 (+434177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:06.542491 11690 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.543000  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:06.543238 11690 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.544250  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:06.544557 11559 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:06.545120  5632 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:06.545463 11690 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.545205 11559 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.546420  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:06.546906  5632 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:06.547750 11559 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.176491 11312 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:06.153942 10871 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:06.548163  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:06.548743 10871 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:06.549604  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:06.549733 10871 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.551087  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:06.215961 11587 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.219172 11667 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.553272 11667 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.554579 11587 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.149487 11503 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.556869  5633 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:06.557801 11503 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.155879 11356 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:06.270313 11489 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.284026 11551 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:06.559263 11551 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.560344 11489 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:06.305743 11615 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.165573  5553 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:27:06.565230  5559 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:27:06.566447  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:06.555559 11667 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.567198  5553 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:06.569051  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:06.308252 11374 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.572s	user 0.007s	sys 0.006s
I20260709 12:27:06.570168 11374 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.573449 11374 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:06.308533 11643 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.565s	user 0.004s	sys 0.007s
I20260709 12:27:06.574508 11643 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.577215 11643 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:06.308950 11671 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:06.303474 11636 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.529s	user 0.010s	sys 0.002s
I20260709 12:27:06.579003 11636 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:06.579766 11671 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.580684  5631 raft_consensus.cc:2368] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17.
I20260709 12:27:06.316850  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:06.317531 11581 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.318251  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:06.320555  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:06.326272 11597 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.331499 11611 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.613037 11611 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.304394 11272 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.617019 11272 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:06.337265 11600 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.008s	user 0.013s	sys 0.001s
I20260709 12:27:06.343042 11686 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.525301 11379 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:06.618356 11686 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.619131 11667 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:06.619730 11667 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:06.620101 11667 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:06.373080 11198 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:06.620653 11198 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:06.622064 11198 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.623903 11198 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:06.387277 11687 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.624485 11581 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.624768 11687 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.626885 11581 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:06.391284 11675 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.307s	user 0.004s	sys 0.000s
I20260709 12:27:06.396544 11654 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:06.402995  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2592) took 1289 ms. Trace:
I20260709 12:27:06.416517 11653 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.630259  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2606) took 1458 ms. Trace:
I20260709 12:27:06.630599 11653 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.630654  5560 rpcz_store.cc:276] 0709 12:27:05.172199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:05.489980 (+317781us) service_pool.cc:224] Handling call
0709 12:27:05.537206 (+ 47226us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:05.705294 (+168088us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:05.705341 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:05.728900 (+ 23559us) log.cc:844] Serialized 52 byte log entry
0709 12:27:05.885361 (+156461us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:27:05.885423 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:05.885538 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:06.390950 (+505412us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b240013f658. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff740b9c4 00007f4ff7409acb 00007f4ff75c4dc9 00007f4ff75c4a55 00007f4ff75c409d 00007f4ffa1f610e 00007f4ffa20d88c 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221
0709 12:27:06.629822 (+238872us) raft_consensus.cc:1685] finished
0709 12:27:06.629857 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:06.630190 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":217954,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":7676,"spinlock_wait_cycles":99075456,"thread_start_us":249159,"threads_started":2,"wal-append.queue_time_us":246088}
I20260709 12:27:06.632937 11653 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.476490 11660 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:06.491055  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2610) took 1317 ms. Trace:
I20260709 12:27:06.162994  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:05.293145  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:06.633714  5561 rpcz_store.cc:276] 0709 12:27:05.173301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:05.897936 (+724635us) service_pool.cc:224] Handling call
0709 12:27:05.914978 (+ 17042us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:06.075641 (+160663us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:06.075671 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:06.075709 (+    38us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:27:06.075751 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:06.490731 (+414980us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b540023fd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:06.490793 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:06.490978 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":369035520}
I20260709 12:27:06.635207  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:06.159080 11487 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.527269  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:06.210353 11676 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:06.629408  5555 rpcz_store.cc:276] 0709 12:27:05.113998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:05.127151 (+ 13153us) service_pool.cc:224] Handling call
0709 12:27:05.144193 (+ 17042us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:05.148311 (+  4118us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:05.148346 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:05.149033 (+   687us) log.cc:844] Serialized 52 byte log entry
0709 12:27:05.320159 (+171126us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:05.388582 (+ 68423us) log.cc:844] Serialized 14 byte log entry
0709 12:27:06.175395 (+786813us) spinlock_profiling.cc:243] Waited 51.1 ms on lock 0x7b1800376120. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:27:06.396239 (+220844us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:06.400440 (+  4201us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:06.400498 (+    58us) raft_consensus.cc:1685] finished
0709 12:27:06.402754 (+  2256us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:06.402929 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":99708,"prepare.queue_time_us":18070,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":6052,"raft.queue_time_us":321320,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":15829,"spinlock_wait_cycles":117589888,"thread_start_us":391236,"threads_started":2,"wal-append.queue_time_us":235649}
I20260709 12:27:06.538399  6279 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:06.633383 11686 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:06.510648 11601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.006s	sys 0.006s
I20260709 12:27:06.525166 11602 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:27:06.645351 11571 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:06.567210 11551 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:06.526451  5627 raft_consensus.cc:2410] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:27:06.961665  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2443) took 1507 ms. Trace:
I20260709 12:27:06.962019  5627 rpcz_store.cc:276] 0709 12:27:05.454020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.179729 (+725709us) service_pool.cc:224] Handling call
0709 12:27:06.961603 (+781874us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:06.962906  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:06.357152  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:06.967196  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2430) took 1659 ms. Trace:
I20260709 12:27:06.583519  5634 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:06.530187  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:07.111459 11436 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:06.181607  5635 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
W20260709 12:27:07.126850  5635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2446) took 1554 ms (client timeout 1721 ms). Trace:
W20260709 12:27:07.127305  5635 rpcz_store.cc:269] 0709 12:27:05.572858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:05.988282 (+415424us) service_pool.cc:224] Handling call
0709 12:27:07.126777 (+1138495us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:07.128471  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:06.455925  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:06.584205  5629 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:27:07.145709  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2441) took 1788 ms (client timeout 1555 ms). Trace:
W20260709 12:27:07.146214  5629 rpcz_store.cc:269] 0709 12:27:05.357637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:05.717439 (+359802us) service_pool.cc:224] Handling call
0709 12:27:07.145491 (+1428052us) spinlock_profiling.cc:243] Waited 816 ms on lock 0x7b5400439254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:07.145635 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1876489472}
I20260709 12:27:06.583635  5704 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:06.530560  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:07.150854  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 620304us
W20260709 12:27:06.637787  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 110606us
I20260709 12:27:07.154162  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:06.583981  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 263482us
I20260709 12:27:07.156780  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:07.160544 11479 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:07.161252 11479 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:07.161608 11479 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:07.161967 11479 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:06.647727  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:07.164680  5555 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:07.168341 11420 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:27:07.169173 11420 consensus_queue.cc:1243] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:27:07.170473 11420 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:07.171013 11420 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.656592  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:06.632696 11597 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:06.604000 11618 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:07.176982  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:06.536296 11575 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:27:06.646381 11601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.006s	sys 0.006s
I20260709 12:27:06.644025 11487 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:07.182090  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:06.633785 11660 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:07.184697 11660 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:06.348377 11663 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:06.649911 11676 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.189664  5555 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:06.665817 11687 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:07.190780  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:06.722980  5601 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.552s (SENT) [suppressed 24 similar messages]
I20260709 12:27:06.777706  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2607) took 1605 ms. Trace:
I20260709 12:27:07.191551  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2458) took 1032 ms. Trace:
W20260709 12:27:06.785473  5554 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.697s	user 0.011s	sys 0.019s
I20260709 12:27:06.861328 11475 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:27:06.869134  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:06.872905 11554 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:07.193640 11475 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:07.191627  5557 rpcz_store.cc:276] 0709 12:27:05.172505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:05.502482 (+329977us) service_pool.cc:224] Handling call
0709 12:27:05.555870 (+ 53388us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:27:05.777888 (+222018us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:05.777920 (+    32us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:27:05.806402 (+ 28482us) log.cc:844] Serialized 78 byte log entry
0709 12:27:06.747792 (+941390us) spinlock_profiling.cc:243] Waited 9.22 ms on lock 0x7b18000d7200. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:27:06.768610 (+ 20818us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:06.776647 (+  8037us) log.cc:844] Serialized 14 byte log entry
0709 12:27:06.777067 (+   420us) log.cc:844] Serialized 14 byte log entry
0709 12:27:06.777267 (+   200us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:06.777352 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:06.777478 (+   126us) raft_consensus.cc:1685] finished
0709 12:27:06.777493 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:06.777642 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12029,"prepare.queue_time_us":8165,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":66,"raft.queue_time_us":264948,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":5238,"spinlock_wait_cycles":21217024,"thread_start_us":249726,"threads_started":3,"wal-append.queue_time_us":930585}
I20260709 12:27:07.194099 11475 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:07.194576 11475 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:07.194620  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:07.195554  5557 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 17: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:06.875164  5628 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
W20260709 12:27:07.192126  5554 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.104s	user 0.011s	sys 0.019s
I20260709 12:27:07.196426  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:07.197476  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1086) took 1879 ms. Trace:
W20260709 12:27:06.878242  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:06.882802 11686 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:07.197713  5557 raft_consensus.cc:2410] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:27:07.198558  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2457) took 1040 ms. Trace:
I20260709 12:27:06.910224  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2445) took 1354 ms. Trace:
I20260709 12:27:07.199261 11696 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:07.199425  5633 rpcz_store.cc:276] 0709 12:27:05.556267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.155539 (+599272us) service_pool.cc:224] Handling call
0709 12:27:06.909989 (+754450us) spinlock_profiling.cc:243] Waited 31.3 ms on lock 0x7b5400438d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:06.910127 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":71949056}
I20260709 12:27:07.199992 11696 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.200814  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:07.201277 11663 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:07.201442  5633 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:07.201931 11663 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:07.202421 11696 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:06.910456 11667 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:06.914697  6279 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:06.927840  5632 raft_consensus.cc:2376] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15. Re-sending same reply.
I20260709 12:27:07.203402  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1060) took 1021 ms. Trace:
W20260709 12:27:06.936671  5631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2432) took 1619 ms (client timeout 1507 ms). Trace:
I20260709 12:27:07.204435 11667 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:06.911758  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 594990us
I20260709 12:27:06.967571  5630 rpcz_store.cc:276] 0709 12:27:05.307375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.356844 (+1049469us) service_pool.cc:224] Handling call
0709 12:27:06.967121 (+610277us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:27:07.204545  5631 rpcz_store.cc:269] 0709 12:27:05.317294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.060706 (+743412us) service_pool.cc:224] Handling call
0709 12:27:06.936390 (+875684us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x7b540009e2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:06.936579 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1193395456}
I20260709 12:27:07.205472 11379 raft_consensus.cc:2764] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:27:06.655620 11549 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.206032  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:07.197633  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2687) took 2149 ms. Trace:
I20260709 12:27:07.207229  5628 rpcz_store.cc:276] 0709 12:27:05.048617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:05.048749 (+   132us) service_pool.cc:224] Handling call
0709 12:27:07.197434 (+2148685us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54003afa54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:07.197548 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4087904640}
I20260709 12:27:06.640815 11653 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:27:06.992393  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 809697us
W20260709 12:27:06.982223 11672 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.945s	user 0.006s	sys 0.000s
I20260709 12:27:06.999635  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:07.018263 11654 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:06.986692  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:07.018460  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:06.996374  5455 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:27:07.019050 11666 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:07.056736 11690 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:07.069625  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2612) took 1749 ms. Trace:
W20260709 12:27:07.096546  5452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35600 (request call id 277) took 32278 ms (client timeout 15000 ms). Trace:
I20260709 12:27:06.204442  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:07.113062  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 582902us
I20260709 12:27:07.205677  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:07.112615 11436 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.216063 11687 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:07.216236  5630 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:07.216540 11687 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:07.216985  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:07.213505 11690 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:07.217571 11436 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:07.142593  5634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2440) took 1797 ms (client timeout 2392 ms). Trace:
I20260709 12:27:07.142006  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:07.176592 11597 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:07.173487  5560 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:27:07.214334  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:07.212222 11654 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:07.218848  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2454) took 1061 ms. Trace:
I20260709 12:27:07.177987 11684 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:07.219318 11436 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:06.646755 11611 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:07.218195  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:07.221817 11690 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:07.213632  5559 rpcz_store.cc:276] 0709 12:27:05.320524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.088203 (+767679us) service_pool.cc:224] Handling call
0709 12:27:06.114324 (+ 26121us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:06.565904 (+451580us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:07.069139 (+503235us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b54001bec54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:07.069535 (+   396us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1009391488}
W20260709 12:27:07.214838  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 1.01047e+06us
I20260709 12:27:07.218472  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:07.215250  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:07.224071  6276 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:07.214269  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2727) took 1895 ms. Trace:
I20260709 12:27:06.163499  5556 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:07.225488  5703 rpcz_store.cc:276] 0709 12:27:05.318363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:05.454049 (+135686us) service_pool.cc:224] Handling call
0709 12:27:05.522657 (+ 68608us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:05.696670 (+174013us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:05.696706 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:05.697324 (+   618us) log.cc:844] Serialized 52 byte log entry
0709 12:27:05.857049 (+159725us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:05.875537 (+ 18488us) log.cc:844] Serialized 14 byte log entry
0709 12:27:05.876138 (+   601us) log.cc:844] Serialized 14 byte log entry
0709 12:27:05.876377 (+   239us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:05.876477 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:06.057538 (+181061us) spinlock_profiling.cc:243] Waited 5.22 ms on lock 0x7b18001ded20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:27:06.654552 (+597014us) spinlock_profiling.cc:243] Waited 3.12 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:07.214048 (+559496us) raft_consensus.cc:1685] finished
0709 12:27:07.214073 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:07.214213 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1714,"prepare.queue_time_us":53645,"prepare.run_cpu_time_us":194,"prepare.run_wall_time_us":5486,"raft.queue_time_us":71394,"raft.run_cpu_time_us":391,"raft.run_wall_time_us":46466,"spinlock_wait_cycles":19171328,"thread_start_us":218520,"threads_started":2,"wal-append.queue_time_us":238237}
I20260709 12:27:07.227015  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:07.186803  5553 raft_consensus.cc:2368] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15.
I20260709 12:27:07.227627  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2446) took 1180 ms. Trace:
I20260709 12:27:07.228636  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2452) took 1071 ms. Trace:
I20260709 12:27:07.229086 11597 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:27:07.217682  6275 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:07.229000  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:07.230515  5702 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
W20260709 12:27:07.218130  5634 rpcz_store.cc:269] 0709 12:27:05.344970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.126028 (+781058us) service_pool.cc:224] Handling call
0709 12:27:07.142408 (+1016380us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b5400438354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:07.142531 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1007659904}
I20260709 12:27:07.231526 11597 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:07.198287 11686 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:07.236178  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:07.204869  5636 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:27:06.960879  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:07.199227  5557 rpcz_store.cc:276] 0709 12:27:06.158492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.196112 (+1037620us) service_pool.cc:224] Handling call
0709 12:27:07.198486 (+  2374us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:07.207005  5632 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:07.228034  5556 rpcz_store.cc:276] 0709 12:27:06.046889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.102182 (+ 55293us) service_pool.cc:224] Handling call
0709 12:27:06.163307 (+ 61125us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:07.225681 (+1062374us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:07.227327 (+  1646us) spinlock_profiling.cc:243] Waited 60.8 ms on lock 0x7b54001bdd50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:07.227550 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":139946880}
I20260709 12:27:07.205806  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:07.197885  5554 rpcz_store.cc:276] 0709 12:27:05.318217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.086575 (+768358us) service_pool.cc:224] Handling call
0709 12:27:06.086901 (+   326us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:06.087250 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:06.087280 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:06.087905 (+   625us) log.cc:844] Serialized 52 byte log entry
0709 12:27:07.197105 (+1109200us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:07.197151 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:07.197224 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:07.197281 (+    57us) raft_consensus.cc:1685] finished
0709 12:27:07.197294 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:07.197423 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4395,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":111,"thread_start_us":1108838,"threads_started":1,"wal-append.queue_time_us":217696}
I20260709 12:27:07.053310 11581 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:07.216960 11687 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:07.220510 11654 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:07.215296  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:07.113099 11551 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:07.213457 11666 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:07.241562 11686 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:07.214078  5452 rpcz_store.cc:269] 0709 12:26:34.818199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:34.818508 (+   309us) service_pool.cc:224] Handling call
0709 12:26:35.179248 (+360740us) spinlock_profiling.cc:243] Waited 5.85 ms on lock 0x7b5400002490. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754e0b2 00007f50004c7759 00007f500045fa76 00007f5000430d58 00007f500052f04e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc
0709 12:27:07.096281 (+31917033us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:57.317313 (+     0us) spinlock_profiling.cc:243] Waited 52.8 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ff75680ac 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffe0a716c
0709 12:26:57.752577 (+435264us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:57.753386 (+   809us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:57.753442 (+    56us) write_op.cc:435] Acquired schema lock
0709 12:26:57.753460 (+    18us) tablet.cc:662] Decoding operations
0709 12:26:57.834788 (+ 81328us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:57.834890 (+   102us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:57.834916 (+    26us) tablet.cc:685] Acquiring locks for 26 operations
0709 12:26:58.026335 (+191419us) tablet.cc:701] Row locks acquired
0709 12:26:58.026389 (+    54us) write_op.cc:260] PREPARE: finished
0709 12:26:58.026702 (+   313us) write_op.cc:270] Start()
0709 12:26:58.027033 (+   331us) write_op.cc:276] Timestamp: P: 1783600018026611 usec, L: 0
0709 12:26:58.027092 (+    59us) op_driver.cc:348] REPLICATION: starting
0709 12:26:58.027984 (+   892us) log.cc:844] Serialized 10004 byte log entry
0709 12:27:00.388752 (+2360768us) spinlock_profiling.cc:243] Waited 6.87 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:00.755784 (+367032us) op_driver.cc:464] REPLICATION: finished
0709 12:27:01.115996 (+360212us) write_op.cc:301] APPLY: starting
0709 12:27:01.116114 (+   118us) tablet.cc:1367] starting BulkCheckPresence
0709 12:27:01.843987 (+727873us) tablet.cc:1370] finished BulkCheckPresence
0709 12:27:01.844055 (+    68us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:27:02.083281 (+239226us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:27:02.084058 (+   777us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=52,key_file_lookups=52,delta_file_lookups=52,mrs_lookups=0
0709 12:27:02.084217 (+   159us) write_op.cc:312] APPLY: finished
0709 12:27:02.308747 (+224530us) log.cc:844] Serialized 228 byte log entry
0709 12:27:02.553185 (+244438us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:27:02.570083 (+ 16898us) write_op.cc:454] Released schema lock
0709 12:27:02.570174 (+    91us) write_op.cc:341] FINISH: Updating metrics
0709 12:27:02.570794 (+   620us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:27:02.570825 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":825084,"reported_tablets":48,"spinlock_wait_cycles":13459712,"child_traces":[["op",{"apply.queue_time_us":351353,"apply.run_cpu_time_us":58581,"apply.run_wall_time_us":1454754,"cfile_cache_hit":106,"cfile_cache_hit_bytes":112807,"num_ops":26,"prepare.queue_time_us":1301284,"prepare.run_cpu_time_us":118288,"prepare.run_wall_time_us":1310532,"raft.queue_time_us":300017,"raft.run_cpu_time_us":26802,"raft.run_wall_time_us":202471,"replication_time_us":2681511,"spinlock_wait_cycles":137368704,"thread_start_us":2742678,"threads_started":5,"wal-append.queue_time_us":1391154,"wal-append.run_cpu_time_us":34158,"wal-append.run_wall_time_us":3528242}]]}
I20260709 12:27:07.223042  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:07.237564  5552 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:27:07.238646  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:07.245592  5556 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:27:07.202433 11663 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.219766 11436 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:07.220369  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:07.271593  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 51340.9us
I20260709 12:27:07.272784  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:07.213153  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:07.274379  5561 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:07.274834  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2455) took 1116 ms. Trace:
I20260709 12:27:07.274922  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:07.275146  5561 rpcz_store.cc:276] 0709 12:27:06.157952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.212698 (+1054746us) service_pool.cc:224] Handling call
0709 12:27:07.274781 (+ 62083us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:07.275645  6276 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:07.276531 11581 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:27:07.277390  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:07.279376  6276 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:27:07.279244 11660 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:27:07.281001  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:07.281832  6276 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:06.656448 11636 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:27:07.282667 11581 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:07.219195  5560 rpcz_store.cc:276] 0709 12:27:06.157674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.651474 (+493800us) service_pool.cc:224] Handling call
0709 12:27:07.218784 (+567310us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:07.283595 11581 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:07.284130  5560 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:27:07.284730  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2453) took 1127 ms. Trace:
I20260709 12:27:07.285030  5560 rpcz_store.cc:276] 0709 12:27:06.157417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.283564 (+1126147us) service_pool.cc:224] Handling call
0709 12:27:07.283860 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:07.284486 (+   626us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:07.284681 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:07.180698 11601 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:07.287045 11601 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.276258  5561 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:07.288976  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2451) took 1132 ms. Trace:
I20260709 12:27:07.203869  6279 rpcz_store.cc:276] 0709 12:27:06.182494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.182633 (+   139us) service_pool.cc:224] Handling call
0709 12:27:07.203333 (+1020700us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:07.289299  5561 rpcz_store.cc:276] 0709 12:27:06.156876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.275796 (+1118920us) service_pool.cc:224] Handling call
0709 12:27:07.276100 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:07.288710 (+ 12610us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:07.288924 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:07.239178  5631 raft_consensus.cc:2410] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 12:27:07.292157  5631 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:07.246296  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:07.295256  5705 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:07.186892 11575 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:27:07.253314  6275 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:07.147917  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:07.297703  5629 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:27:07.299257  5629 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260709 12:27:07.300467  5629 pending_rounds.cc:85] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 8
I20260709 12:27:07.300812  5629 pending_rounds.cc:107] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 12:27:07.301170  5629 raft_consensus.cc:2889] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:07.302739  5629 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:27:07.304456  5629 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 9. (term mismatch)
I20260709 12:27:07.309340  5629 pending_rounds.cc:85] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 8
I20260709 12:27:07.309738  5629 pending_rounds.cc:107] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260709 12:27:07.310127  5629 raft_consensus.cc:2889] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:07.213984  5455 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:07.269332  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:07.228981  5553 rpcz_store.cc:276] 0709 12:27:06.157145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.568532 (+411387us) service_pool.cc:224] Handling call
0709 12:27:07.228452 (+659920us) spinlock_profiling.cc:243] Waited 617 ms on lock 0x7b540044f054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:07.228576 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1418643840}
I20260709 12:27:07.311200  5703 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:07.042900  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:07.312014  5558 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:07.260898  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:27:07.322925  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 599957us
I20260709 12:27:07.192013  5555 rpcz_store.cc:276] 0709 12:27:06.158786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.190479 (+1031693us) service_pool.cc:224] Handling call
0709 12:27:07.191494 (+  1015us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:07.326848 11636 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:07.327454  5555 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 12:27:07.327381 11636 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:07.328698  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 67920.5us
I20260709 12:27:07.328707 11654 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:07.329542  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:07.329562 11654 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:07.330351 11653 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:07.330765 11653 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:07.331197 11653 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:07.330864 11654 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.329811 11636 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:07.335348 11695 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:07.336053 11695 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.341563 11695 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:07.342077 11653 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:07.212037  5675 connection.cc:256] RPC call timeout handler was delayed by -1.026828180999928s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1688s for a response
I20260709 12:27:07.232010 11597 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:07.355975  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:07.345432  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:07.356984  5703 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:07.358289  5628 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:27:07.358712 11636 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:07.359097  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2434) took 2036 ms. Trace:
I20260709 12:27:07.359280 11636 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:07.359860 11636 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:07.271225 11686 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:27:07.363296  5633 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:07.245857 11379 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:07.364178  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:07.364702  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:07.364796 11379 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:07.364951  5710 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:27:07.365569  5708 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:07.365886 11379 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.366498  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:07.208272 11549 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:07.268138 11198 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:07.369063  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:07.368963 11198 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.369751  5709 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:07.370770  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:07.370590 11636 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.371148 11198 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:07.371431  5557 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:27:07.372058  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:07.372684  5554 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:27:07.373138  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2615) took 1016 ms. Trace:
I20260709 12:27:07.373551  5557 rpcz_store.cc:276] 0709 12:27:06.356304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.372179 (+1015875us) service_pool.cc:224] Handling call
0709 12:27:07.372568 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:07.372777 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:07.372802 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:07.372850 (+    48us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:07.372889 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:07.372943 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:07.373075 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:07.376999  5630 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:07.378082  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:07.359522  5628 rpcz_store.cc:276] 0709 12:27:05.322291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.269904 (+1947613us) service_pool.cc:224] Handling call
0709 12:27:07.357947 (+ 88043us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:07.358753 (+   806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:07.359012 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:07.393553  5455 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:27:07.236675 11684 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:07.396874  5703 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:07.399113  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1138) took 1014 ms. Trace:
I20260709 12:27:07.399652  5630 rpcz_store.cc:276] 0709 12:27:06.385235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.376508 (+991273us) service_pool.cc:224] Handling call
0709 12:27:07.376821 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:07.398759 (+ 21938us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:07.399025 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:07.401398  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2431) took 2093 ms. Trace:
W20260709 12:27:07.405809  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 48459.5us
I20260709 12:27:07.401818  5636 rpcz_store.cc:276] 0709 12:27:05.307600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.196470 (+888870us) service_pool.cc:224] Handling call
0709 12:27:07.204493 (+1008023us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:07.237085 (+ 32592us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:07.401195 (+164110us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54003afa50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:07.401335 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2317282176}
W20260709 12:27:07.408927  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 79458us
I20260709 12:27:07.410020  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:07.413048  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:07.418113  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:07.420204  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:07.420943  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:07.268761 11690 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:07.345574 11436 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:07.441107  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:07.442385  5630 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:07.445279  5627 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 6. (term mismatch)
I20260709 12:27:07.445824  5627 pending_rounds.cc:85] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 5
I20260709 12:27:07.446125  5627 pending_rounds.cc:107] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260709 12:27:07.446439  5627 raft_consensus.cc:2889] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:07.446834  5627 pending_rounds.cc:107] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20260709 12:27:07.447129  5627 raft_consensus.cc:2889] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:07.438903  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:07.460458  5706 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 10. (term mismatch)
I20260709 12:27:07.460927  5706 pending_rounds.cc:85] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 9
I20260709 12:27:07.461598  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:07.467900  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:07.462888  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:07.470027  5707 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:07.471158  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:07.425082 11667 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:07.466814  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:07.473888  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:07.466879  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:07.461151  5706 pending_rounds.cc:107] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 15.10
I20260709 12:27:07.481927  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:07.482852  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:07.466154  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2447) took 1880 ms. Trace:
I20260709 12:27:07.483623  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:07.485139  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 211820us
I20260709 12:27:07.485687  5633 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:07.481391  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:07.484975  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 140180us
I20260709 12:27:07.476964 11667 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:07.477288  5636 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:27:07.484553  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:07.492190  5630 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:07.478829  5706 raft_consensus.cc:2889] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:27:07.488085  5675 leader_election.cc:336] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.582s (SENT)
I20260709 12:27:07.494442  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:07.488878  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:07.488998  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:07.495839  6278 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:07.496465  5627 rpcz_store.cc:276] 0709 12:27:05.586084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.398034 (+1811950us) service_pool.cc:224] Handling call
0709 12:27:07.444837 (+ 46803us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:07.465813 (+ 20976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:07.466079 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:07.499101  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:07.464133 11601 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:07.485405 11660 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:07.461118 11697 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:07.500393  6280 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:07.512979  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:07.475854 11551 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:27:07.526273  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 101013us
I20260709 12:27:07.502818  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:07.520004  5706 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:27:07.536026  6278 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:27:07.507714  5704 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13.
I20260709 12:27:07.537197  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2736) took 1332 ms. Trace:
I20260709 12:27:07.537643  5706 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:27:07.537588  5704 rpcz_store.cc:276] 0709 12:27:06.205012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.317924 (+112912us) service_pool.cc:224] Handling call
0709 12:27:07.537131 (+1219207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:07.538905  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:07.539095 11611 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:07.539732 11611 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:07.540443 11611 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:07.534591 11697 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:07.541399 11697 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:07.543697  5630 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:07.533861 11660 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.546413 11660 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:07.548219  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:07.458171 11690 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.422577 11684 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:07.576519  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:07.549939 11697 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:07.581446  5632 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18. Ops: []
W20260709 12:27:07.561137  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 79797.8us
I20260709 12:27:07.490666 11581 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:07.567488  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:07.583827  5636 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:07.499475 11667 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:07.570899  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:07.575448  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:07.512279 11601 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:07.553500  5701 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:07.501408 11587 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:07.580253  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:07.550930 11489 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:07.580736  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:07.584543  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:07.575742  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:07.603761  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 119666us
I20260709 12:27:07.534335 11209 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:27:07.603528  6278 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:07.595503  5707 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:07.513513 11695 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.078s
I20260709 12:27:07.611291  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:07.613579  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:07.587539 11676 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:27:07.617707  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 65691.6us
I20260709 12:27:07.501434 11704 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.108s
I20260709 12:27:07.596166 11601 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:07.585847 11420 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:07.595717 11653 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:07.515957 11711 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.113s
I20260709 12:27:07.623348  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:07.618930  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:07.485860 11597 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:07.618050 11587 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.009s
W20260709 12:27:07.828927  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 210117us
W20260709 12:27:07.764314 11479 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.595s	user 0.008s	sys 0.002s
W20260709 12:27:07.733559  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:07.666620 11636 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:07.752166  6276 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: []
W20260709 12:27:07.661772  5582 debug-util.cc:398] Leaking SignalData structure 0x7b080024c220 after lost signal to thread 4359
I20260709 12:27:07.647738 11687 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:07.751472 11710 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:07.665889 11697 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:07.753691  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 140334us
I20260709 12:27:07.748001 11686 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:07.628856  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1141) took 1242 ms. Trace:
I20260709 12:27:07.876236  5632 rpcz_store.cc:276] 0709 12:27:06.386186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.238268 (+852082us) service_pool.cc:224] Handling call
0709 12:27:07.238554 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:07.581949 (+343395us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:07.628496 (+ 46547us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b5400240754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:07.628769 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":788025600}
I20260709 12:27:07.658125 11690 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:07.659442 11704 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:07.661288 11718 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:07.615798 11489 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.596196 11581 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:07.746206 11475 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.551s	user 0.009s	sys 0.003s
I20260709 12:27:07.661159 11551 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:07.644979 11715 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:27:07.639611 11684 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:07.660271 11663 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:07.745608  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.006s	sys 0.001s
W20260709 12:27:07.902252  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.710s	user 0.008s	sys 0.002s
I20260709 12:27:07.637799 11701 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:27:07.810901  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:07.649857 11209 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:07.912382 11209 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:07.887898 11686 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:07.913251 11209 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:07.913421 11686 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:07.904146 11618 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:07.914666 11725 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:07.745136  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1140) took 1359 ms. Trace:
I20260709 12:27:07.915825 11436 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:07.916687  5635 rpcz_store.cc:276] 0709 12:27:06.385951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.376445 (+990494us) service_pool.cc:224] Handling call
0709 12:27:07.426216 (+ 49771us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:07.426483 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:27:07.498317 (+ 71834us) log.cc:844] Serialized 14 byte log entry
0709 12:27:07.744643 (+246326us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:07.744718 (+    75us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:07.744795 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:07.744892 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:07.745050 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21330,"prepare.queue_time_us":164362,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":1635,"raft.queue_time_us":68452,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":122,"thread_start_us":279227,"threads_started":2,"wal-append.queue_time_us":321912}
I20260709 12:27:07.608302 11666 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:07.918610 11436 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:07.743820  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 173274us
W20260709 12:27:07.836943 11694 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.222s	user 0.000s	sys 0.003s
I20260709 12:27:07.830009  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:07.860833  5582 debug-util.cc:398] Leaking SignalData structure 0x7b08003c7ac0 after lost signal to thread 5493
W20260709 12:27:07.922968  5582 debug-util.cc:398] Leaking SignalData structure 0x7b08000a0220 after lost signal to thread 5585
W20260709 12:27:07.821317 11554 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.628s	user 0.005s	sys 0.006s
W20260709 12:27:07.927646  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 322327us
W20260709 12:27:07.865052  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 292835us
W20260709 12:27:07.866247  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 255029us
I20260709 12:27:07.879179  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:07.886579 11714 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.230s	user 0.001s	sys 0.001s
I20260709 12:27:07.891336  5628 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
W20260709 12:27:07.896706 11713 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.257s	user 0.007s	sys 0.000s
I20260709 12:27:07.898154 11722 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:07.898028 11475 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.618393 11676 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:07.919440  5702 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:07.960703  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:07.961658  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:07.962256  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:07.935411  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:07.963217  5556 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:07.922719 11686 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:07.888664 11718 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:07.859004 11479 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:07.938562  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:07.942220  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63166.8us
W20260709 12:27:07.942919  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.004s	sys 0.008s
I20260709 12:27:07.943724  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:07.981626  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.004s	sys 0.008s
I20260709 12:27:07.937237  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:07.904759 11551 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:07.960698  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:07.923255  5582 debug-util.cc:398] Leaking SignalData structure 0x7b08003ad280 after lost signal to thread 5586
I20260709 12:27:07.964027  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:07.967305  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:07.982191  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 766929us
W20260709 12:27:07.982676  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45551us
I20260709 12:27:07.985612  5705 raft_consensus.cc:2368] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14.
I20260709 12:27:07.986129  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:07.987703  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44156.9us
I20260709 12:27:08.019752  5560 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:07.943547 11554 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:07.941576 11689 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:08.023108  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:08.022284  5555 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:07.952507 11663 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:08.016676 11587 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:27:08.025362 11739 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:07.969386 11676 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:07.961184 11704 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:08.035856  5556 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
W20260709 12:27:08.027381  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 104140us
I20260709 12:27:08.110422  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:08.111865  5556 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:27:08.033124  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:08.113150  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 80139.5us
I20260709 12:27:08.040387  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:08.025414  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:08.025568  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:08.137442  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:08.149057  5556 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:27:08.035416  5582 debug-util.cc:398] Leaking SignalData structure 0x7b080027b220 after lost signal to thread 5660
I20260709 12:27:08.016958 11549 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:27:08.150813  5582 debug-util.cc:398] Leaking SignalData structure 0x7b08000d47c0 after lost signal to thread 5661
I20260709 12:27:07.982542 11684 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:08.135489  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 305525us
I20260709 12:27:08.137815  5705 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:27:08.042943 11611 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:08.036809  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:08.020501 11731 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:07.964684 11722 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.112903 11636 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:27:08.025647 11725 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:08.162042 11587 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:08.249344 11705 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.609s	user 0.000s	sys 0.008s
I20260709 12:27:08.249679 11587 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:08.191813  5710 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: []
I20260709 12:27:08.030256 11663 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:08.251407  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2699) took 1847 ms. Trace:
I20260709 12:27:08.251652  5559 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:08.173295 11718 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:08.253144 11749 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:27:08.253266 11718 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:08.254163 11663 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:08.212687  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 187318us
I20260709 12:27:08.159843  5707 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:08.174183 11701 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:08.256210 11701 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.040812 11741 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:08.257973 11741 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.258550 11701 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:08.152279  5582 debug-util.cc:398] Leaking SignalData structure 0x7b08003223c0 after lost signal to thread 5734
I20260709 12:27:08.038538 11554 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:08.260425  5582 debug-util.cc:398] Leaking SignalData structure 0x7b080028b380 after lost signal to thread 6304
W20260709 12:27:08.260752  5582 debug-util.cc:398] Leaking SignalData structure 0x7b080028b0a0 after lost signal to thread 6305
W20260709 12:27:08.261215  5582 debug-util.cc:398] Leaking SignalData structure 0x7b080031cb80 after lost signal to thread 11467
W20260709 12:27:08.261440  5582 debug-util.cc:398] Leaking SignalData structure 0x7b08002c1b80 after lost signal to thread 11471
I20260709 12:27:08.261929 11684 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:08.262406  5582 debug-util.cc:398] Leaking SignalData structure 0x7b08003d71c0 after lost signal to thread 11642
W20260709 12:27:08.262946  5582 debug-util.cc:398] Leaking SignalData structure 0x7b08001d62e0 after lost signal to thread 11657
W20260709 12:27:08.263185  5582 debug-util.cc:398] Leaking SignalData structure 0x7b08001d01e0 after lost signal to thread 11658
W20260709 12:27:08.263419  5582 debug-util.cc:398] Leaking SignalData structure 0x7b0800270d80 after lost signal to thread 11662
W20260709 12:27:08.263643  5582 debug-util.cc:398] Leaking SignalData structure 0x7b0800356000 after lost signal to thread 11664
W20260709 12:27:08.263911  5582 debug-util.cc:398] Leaking SignalData structure 0x7b0800393b20 after lost signal to thread 11669
I20260709 12:27:08.263974 11684 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:08.106781 11697 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:08.266812 11697 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:08.030048 11689 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.228157 11653 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:08.167954  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:08.264722  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:08.163259 11739 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:27:08.160445  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 125772us
I20260709 12:27:08.250202 11731 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:08.283264  5524 leader_election.cc:400] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14.
I20260709 12:27:08.283681  5524 leader_election.cc:403] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:27:08.249012 11725 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.261351 11554 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:08.113494 11704 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:08.292668 11704 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:08.293459 11479 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:08.029954 11717 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.377s	user 0.000s	sys 0.004s
W20260709 12:27:08.158758  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:08.264278  5582 debug-util.cc:398] Leaking SignalData structure 0x7b08002e8880 after lost signal to thread 11673
I20260709 12:27:08.260108 11741 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:08.287065 11739 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:27:08.298116  5582 debug-util.cc:398] Leaking SignalData structure 0x7b08003f02e0 after lost signal to thread 11679
I20260709 12:27:08.251979  5631 rpcz_store.cc:276] 0709 12:27:06.407584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.292935 (+885351us) service_pool.cc:224] Handling call
0709 12:27:07.293324 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:07.293774 (+   450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:07.293814 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:07.294388 (+   574us) log.cc:844] Serialized 26 byte log entry
0709 12:27:07.489821 (+195433us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:07.489872 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:07.489955 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:08.114603 (+624648us) spinlock_profiling.cc:243] Waited 84.7 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:08.251035 (+136432us) raft_consensus.cc:1685] finished
0709 12:27:08.251061 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:08.251328 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2916,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"spinlock_wait_cycles":194726656,"thread_start_us":194960,"threads_started":1,"wal-append.queue_time_us":345162}
I20260709 12:27:08.042397 11575 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:27:08.306907  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53019.4us
I20260709 12:27:08.309388  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:08.310256  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:08.311038  5627 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:08.316551  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:08.325011  5632 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:27:08.326089  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 158253us
I20260709 12:27:08.175261 11436 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:08.327371  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:08.250351 11611 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:08.339309  5556 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
W20260709 12:27:08.340963  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 76342.1us
I20260709 12:27:08.025065 11551 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:08.329219  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:08.344326  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:08.345136  5635 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:08.341760  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:08.347234  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 236292us
I20260709 12:27:08.348102  5707 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:27:08.352838  6272 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:27:08.357672  5627 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
W20260709 12:27:08.032377 11713 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:27:08.297101 11687 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:08.366931  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:08.367900  6276 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:08.029500 11724 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:27:08.367982  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:08.369071  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:08.360584  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 246688us
I20260709 12:27:08.307056 11654 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14.
I20260709 12:27:08.307001 11575 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:08.360052 11663 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:08.324666 11653 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:08.038609 11420 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:08.439469  5558 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
W20260709 12:27:08.439352  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 65393.8us
I20260709 12:27:08.375198  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:08.376123  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:08.382292  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:08.336632 11436 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:08.446623  5705 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 19 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
W20260709 12:27:08.439142  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70183.4us
I20260709 12:27:08.370277  5629 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:08.450028  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:08.400501  5635 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:08.373705 11701 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:08.374145 11479 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:08.443667  5675 connection.cc:256] RPC call timeout handler was delayed by -1.1501612530000784s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1777s for a response
I20260709 12:27:08.388636 11575 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:08.366768 11697 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:08.394447 11667 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.796s	user 0.009s	sys 0.003s
I20260709 12:27:08.397898  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1089) took 2042 ms. Trace:
W20260709 12:27:08.443073  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 67058.3us
I20260709 12:27:08.374245 11581 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:08.499475  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51794.9us
I20260709 12:27:08.453414  5555 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
W20260709 12:27:08.499399  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56940.4us
I20260709 12:27:08.407017 11722 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:08.374728 11724 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:08.499965  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:08.382637  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:08.468206  6276 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 20: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:08.418061  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.473169  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:08.436436  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:08.381203 11729 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:27:08.446450 11717 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:27:08.405517 11686 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:08.462986  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.559245  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2741) took 1323 ms. Trace:
I20260709 12:27:08.513837 11728 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:08.522161  5560 rpcz_store.cc:276] 0709 12:27:06.355073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.285776 (+930703us) service_pool.cc:224] Handling call
0709 12:27:07.286051 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:08.020222 (+734171us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:08.397570 (+377348us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b54000267d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:08.397815 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1686925568}
I20260709 12:27:08.559701  5702 rpcz_store.cc:276] 0709 12:27:07.235512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.235809 (+   297us) service_pool.cc:224] Handling call
0709 12:27:08.559020 (+1323211us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b540006efd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:08.559182 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1568854912}
W20260709 12:27:08.509003  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0771655559997271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2211s for a response
I20260709 12:27:08.528910  6234 leader_election.cc:400] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14.
W20260709 12:27:08.565812  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0285655559996485s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22650000000000003s for a response
I20260709 12:27:08.566038  6234 leader_election.cc:403] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:27:08.378829 11487 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:27:08.566615  5675 leader_election.cc:336] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.688s (SENT)
I20260709 12:27:08.446100 11741 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:08.387898 11489 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:08.561594 11601 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.894s	user 0.014s	sys 0.000s
I20260709 12:27:08.380046 11636 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.138s
I20260709 12:27:08.536789  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2706) took 1054 ms. Trace:
I20260709 12:27:08.593969  5628 rpcz_store.cc:276] 0709 12:27:07.482471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.482696 (+   225us) service_pool.cc:224] Handling call
0709 12:27:07.502546 (+ 19850us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:07.930513 (+427967us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:08.536373 (+605860us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b540009ecd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:08.536704 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":893367552}
I20260709 12:27:08.447185 11687 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:27:08.541611  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:08.596144  5705 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:08.543401  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:08.564030 11690 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:08.394690 11689 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:08.602548 11724 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:08.603988 11724 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:08.403896 11653 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:08.550799  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77816.9us
W20260709 12:27:08.407019 11716 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.769s	user 0.003s	sys 0.003s
I20260709 12:27:08.454519 11436 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:08.543787 11697 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 12:27:08.596439 11744 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 13. Current term is 14.
I20260709 12:27:08.401942 11731 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:08.520105  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:08.638141  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 118164us
I20260709 12:27:08.639158  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:08.515648 11721 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:27:08.563409  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 127135us
I20260709 12:27:08.641718  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:08.642942 11721 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:27:08.643815 11721 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:08.518340 11198 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:08.644572 11737 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:08.644644 11198 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.645490 11737 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:08.445865 11420 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:08.647148 11198 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:08.380990 11738 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.164s
I20260709 12:27:08.531503  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.527946 11475 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:08.580919  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 221781us
I20260709 12:27:08.652630 11489 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:27:08.653663 11720 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:08.653715  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:08.654242 11720 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:08.655437 11720 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:08.532656  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 158219us
I20260709 12:27:08.452963 11748 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.111s
W20260709 12:27:08.591408  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62634.9us
I20260709 12:27:08.544080 11701 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:08.454068 11704 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:08.593092  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1139) took 2207 ms. Trace:
I20260709 12:27:08.676661 11686 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:27:08.676554  5634 rpcz_store.cc:276] 0709 12:27:06.385705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.357798 (+972093us) service_pool.cc:224] Handling call
0709 12:27:07.358097 (+   299us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:07.407579 (+ 49482us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:07.407611 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:07.425331 (+ 17720us) log.cc:844] Serialized 26 byte log entry
0709 12:27:07.552977 (+127646us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:07.555936 (+  2959us) log.cc:844] Serialized 14 byte log entry
0709 12:27:07.556261 (+   325us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:07.556402 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:07.899724 (+343322us) spinlock_profiling.cc:243] Waited 67.4 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:08.592782 (+693058us) raft_consensus.cc:1685] finished
0709 12:27:08.592881 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:08.593031 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57522,"prepare.queue_time_us":72744,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":106115,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":29938,"spinlock_wait_cycles":154940160,"thread_start_us":74721,"threads_started":2,"wal-append.queue_time_us":149164}
I20260709 12:27:08.677794 11701 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.594552  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.542907  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:08.678965  5633 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:27:08.679345  5704 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:27:08.547801  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:08.680732  5701 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:08.599197  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:08.514372 11479 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.647537 11420 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:08.592742 11487 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:08.683549  5555 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:08.620445  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:08.683367 11487 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.684247  5707 raft_consensus.cc:2376] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 13. Re-sending same reply.
I20260709 12:27:08.620436  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.403028 11663 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.686941 11663 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:08.577747 11729 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:08.535756 11667 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:08.671231  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:08.690148  5560 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:08.691114  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:08.406265 11379 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:08.656036 11697 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:08.580711  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:08.648768 11636 consensus_queue.cc:1243] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 18.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:27:08.582358 11660 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:08.664315  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:08.671240 11209 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.757s	user 0.013s	sys 0.000s
I20260709 12:27:08.672495  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:08.644187 11721 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:08.694267  5705 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:08.600747 11690 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:08.705700  5704 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:08.727371  5554 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:08.711228  5702 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:08.725199  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 132066us
W20260709 12:27:08.707356  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:08.717753  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53495.3us
W20260709 12:27:08.671777  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52582.7us
I20260709 12:27:08.720642  5602 leader_election.cc:400] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15.
I20260709 12:27:08.579912 11728 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.722671  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:08.725237  6280 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:08.725510  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:08.656523 11475 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:27:08.728636  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 56255.6us
I20260709 12:27:08.683398 11479 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:08.730676  5602 leader_election.cc:403] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:27:08.713670 11720 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:27:08.733355  5675 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.810s (SENT)
I20260709 12:27:08.597707 11601 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.733209  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:08.711265 11379 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:08.766703  6278 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:27:08.771770  5702 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:08.716717 11198 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 12:27:08.732650 11697 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:08.749425  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:08.750046 11765 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.129s	user 0.000s	sys 0.003s
I20260709 12:27:08.731117 11690 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:08.750862 11686 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 14. Current term is 15.
I20260709 12:27:08.725037 11554 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:08.752498 11762 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:08.726540 11209 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.711596 11744 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:08.755218  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:08.744588 11774 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:08.813694 11774 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:08.756083 11704 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:08.756448 11748 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:27:08.815066 11774 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:08.816001 11775 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:27:08.807680  6274 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:08.757650 11618 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:08.816747 11618 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:08.817413  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2616) took 2445 ms. Trace:
I20260709 12:27:08.817835  5554 rpcz_store.cc:276] 0709 12:27:06.371807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.373447 (+1001640us) service_pool.cc:224] Handling call
0709 12:27:07.425655 (+ 52208us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:08.727749 (+1302094us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:08.817179 (+ 89430us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54003a98d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:08.817356 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2993222400}
I20260709 12:27:08.819113  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:08.820791  6272 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
W20260709 12:27:08.823364  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:08.726783 11721 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:08.825961  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:08.827015  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:08.827735  5628 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:08.738639  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:08.765098  5701 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:08.828495  5627 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:08.762733  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:08.773036  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.829254  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:08.829841  5633 raft_consensus.cc:2376] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14. Re-sending same reply.
I20260709 12:27:08.830068  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.830811  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:08.831197 11755 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:27:08.724434 11611 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 12:27:08.732488  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:08.832679  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 100262us
I20260709 12:27:08.789249  6275 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:08.721593 11436 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.833874 11689 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:08.835206 11436 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:08.734773 11660 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:08.835213 11618 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:08.836979 11686 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:08.837342 11687 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:08.837911 11687 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:08.837738 11686 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.779969 11475 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:27:08.724121 11761 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.068s
I20260709 12:27:08.838930 11687 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.803951 11762 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:08.841499 11744 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:08.841617 11663 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:08.842114 11744 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:08.842309 11663 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:08.723961 11731 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 12:27:08.843284 11744 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:08.844663 11549 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:08.845109 11549 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:08.803830 11554 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:08.840054  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.850847  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:08.785512  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52180.8us
W20260709 12:27:08.851759  5675 leader_election.cc:336] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.105s (SENT)
I20260709 12:27:08.852174  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:08.806699 11741 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:27:08.734944 11759 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
I20260709 12:27:08.831435  5631 raft_consensus.cc:2376] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20. Re-sending same reply.
I20260709 12:27:08.868562  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:08.831240  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:08.870121  5552 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:08.833822  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:08.787249  6271 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:08.832990 11721 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:08.837618 11618 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:08.837339  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:08.871245  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:08.809865 11732 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:27:08.796613  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.795379  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.849030  5627 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:27:08.849689  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:08.850157  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:08.853168  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.811020  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.862190  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:08.766031 11734 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 12:27:08.920004  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:08.920919  5555 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:27:08.922264  5553 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:08.832759 11748 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:08.873276  5631 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:27:08.922394  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 164361us
I20260709 12:27:08.923738  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:08.924172  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 71167.9us
I20260709 12:27:08.927368  5709 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
W20260709 12:27:08.924872  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53748us
I20260709 12:27:08.840149 11687 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:08.873574  5630 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:27:08.865742  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.931646  5561 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
W20260709 12:27:08.923897  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 73837.4us
I20260709 12:27:08.933504  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:08.845877 11549 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:08.939718  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.939769  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.935113  5632 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:08.941396  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:08.941963  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:08.942448  5554 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:08.943109  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.930469  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:08.937167  5701 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:08.939014  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:08.841713 11767 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.237s	user 0.000s	sys 0.002s
W20260709 12:27:08.865060 11676 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.706s	user 0.011s	sys 0.001s
I20260709 12:27:08.864780 11554 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:08.953749  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:08.963438  5561 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:27:08.952538  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.964615  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.946146  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:08.951550  5632 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:08.947402  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:08.873946 11715 raft_consensus.cc:2764] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
W20260709 12:27:08.953720  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 212751us
I20260709 12:27:08.955403  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:08.968657  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2627) took 1028 ms. Trace:
I20260709 12:27:08.969466  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:08.970314  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 118587us
I20260709 12:27:08.885831 11686 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:08.761775 11729 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:08.988379  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:08.938107 11755 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:08.972952  5559 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:08.965842  5627 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:08.979472  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:08.979830  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:08.981523  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:08.932986 11666 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:08.866791 11741 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:08.926211 11748 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:08.952435 11718 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.699s	user 0.015s	sys 0.001s
I20260709 12:27:08.919883 11732 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:08.985239  5558 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:27:08.969142  5557 rpcz_store.cc:276] 0709 12:27:07.939845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.940241 (+   396us) service_pool.cc:224] Handling call
0709 12:27:07.940663 (+   422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:07.972853 (+ 32190us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:07.972909 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:07.985824 (+ 12915us) log.cc:844] Serialized 26 byte log entry
0709 12:27:08.377285 (+391461us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:27:08.377346 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:08.377446 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:08.684309 (+306863us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:08.958096 (+273787us) raft_consensus.cc:1685] finished
0709 12:27:08.958123 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:08.968592 (+ 10469us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33620,"raft.queue_time_us":21620,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":11729,"spinlock_wait_cycles":55931264,"thread_start_us":263181,"threads_started":1,"wal-append.queue_time_us":493157}
I20260709 12:27:09.022411  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:09.023020  5705 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:09.023429  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:08.973996  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:08.977699  5702 pending_rounds.cc:85] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 6
I20260709 12:27:08.946961  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:08.996516  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:09.024230  5701 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:09.024940  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:09.026882  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:09.024205  5702 pending_rounds.cc:107] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 12:27:09.027470  5702 raft_consensus.cc:2889] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:09.011282  5704 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:09.004424  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:09.011840  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:09.011547  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:09.004711  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:09.028720  5632 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 19: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:08.996398  5553 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:08.957121 11479 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:09.019029  5560 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:09.028973  5627 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
W20260709 12:27:09.029556  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:09.030135  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:09.030694  5627 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:08.989794 11618 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:27:09.032979  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:09.035090 11704 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:08.956773 11738 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.061s
I20260709 12:27:08.957687 11654 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.060s
I20260709 12:27:08.993505 11755 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:08.965725 11676 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.041612  5710 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:09.054569  5555 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 11.
I20260709 12:27:08.841754 11762 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:09.021124 11748 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:09.089582  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:09.037902 11666 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:09.118820  5632 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:09.119329 11666 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:09.106112 11755 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:08.964128 11711 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 12:27:09.130664  5559 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
W20260709 12:27:09.044518  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:09.100879  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:09.050587  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:09.102274  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1021) took 1163 ms. Trace:
I20260709 12:27:09.103192  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:08.968055 11420 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
I20260709 12:27:08.994376 11601 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:08.983911 11379 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.097102 11479 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.119903  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:09.030326  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:27:09.145426  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 115177us
I20260709 12:27:08.995430 11686 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:09.109509 11718 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:09.005086 11660 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:09.137789 11778 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
I20260709 12:27:09.097343 11667 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:09.133565  5708 rpcz_store.cc:276] 0709 12:27:07.939280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.939978 (+   698us) service_pool.cc:224] Handling call
0709 12:27:07.940382 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:08.162074 (+221692us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:08.162114 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:08.162597 (+   483us) log.cc:844] Serialized 26 byte log entry
0709 12:27:08.513095 (+350498us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:08.513155 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:08.513233 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:08.841021 (+327788us) spinlock_profiling.cc:243] Waited 55 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:09.087806 (+246785us) raft_consensus.cc:1685] finished
0709 12:27:09.087924 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:09.102186 (+ 14262us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":235427,"raft.queue_time_us":239260,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":19619,"spinlock_wait_cycles":126386944,"thread_start_us":350167,"threads_started":1,"wal-append.queue_time_us":417761}
W20260709 12:27:09.131908 11551 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.747s	user 0.011s	sys 0.005s
I20260709 12:27:09.140035  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:27:09.183122  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 229463us
I20260709 12:27:09.131454  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:09.155794 11575 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.610s	user 0.011s	sys 0.000s
W20260709 12:27:09.261701  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:09.274544  5701 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:09.133304 11475 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:09.268998  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:09.285670  5632 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:09.155253  5627 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:09.276659  5707 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:09.277379  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:09.290313 11654 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:09.290715 11654 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:09.290757  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:09.291167 11654 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:09.294275 11715 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:09.296489 11782 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:09.297161 11730 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:09.297564 11730 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:09.297888 11730 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:09.299446 11654 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.301678 11654 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:09.306157 11479 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:09.307235 11721 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:09.195963 11687 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:09.157312 11781 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:27:09.311722 11676 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:09.313066 11779 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.316s	user 0.001s	sys 0.000s
W20260709 12:27:09.301775 11785 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 12:27:09.332165  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:09.204921  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:09.347924  5710 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 16: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:27:09.181927 11760 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.368s	user 0.000s	sys 0.001s
I20260709 12:27:09.351850  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:09.356289  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 290993us
I20260709 12:27:09.362155  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:09.360767  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:09.365655  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:09.366228  5701 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:09.145045 11732 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.158144 11747 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:09.157295 11772 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:09.380491  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:09.383616  5552 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.513s	user 0.009s	sys 0.001s
I20260709 12:27:09.384521  5552 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:09.369585  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:09.374235  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:09.374202  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:09.391389  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:09.396991  5710 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:27:09.394825  5627 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:09.398447  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:09.399118  5627 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:09.408133  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:09.408620  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 268672us
I20260709 12:27:09.409833  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:09.286988 11475 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:09.284531 11575 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.415675 11575 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:09.416458 11701 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.008s	sys 0.012s
I20260709 12:27:09.416867 11575 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:27:09.417080 11701 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.008s	sys 0.012s
W20260709 12:27:09.417328  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:09.417645 11575 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:09.416567  5636 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:09.419127 11581 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:09.419476 11581 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:09.420065  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2715) took 1104 ms. Trace:
I20260709 12:27:09.420449  5636 rpcz_store.cc:276] 0709 12:27:08.315390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:08.315574 (+   184us) service_pool.cc:224] Handling call
0709 12:27:08.347128 (+ 31554us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:09.419284 (+1072156us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:09.419834 (+   550us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54001f2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:09.420003 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2458998528}
I20260709 12:27:09.421018  5627 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:09.421624  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:09.422374  5636 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 14: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:09.422284  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:09.423297  5627 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:09.423375  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:09.428941 11475 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:09.429997 11475 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:09.431764 11744 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:09.433990  5627 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:09.435362  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:09.436048  5627 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:09.424095  5636 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:09.438390 11756 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:09.438776  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:09.438920 11756 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.439392  5636 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 19
W20260709 12:27:09.442354  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77383.9us
I20260709 12:27:09.443698 11479 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:09.444839 11479 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:09.445413 11479 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.447858 11420 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:09.448401 11420 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:09.283061 11551 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.157331 11773 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.426s
I20260709 12:27:09.409003  5632 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:09.368767 11718 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:09.407101  5701 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:09.455163  5707 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 12:27:09.456406  5627 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:09.392534 11747 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:09.145349 11741 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.458035  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:09.459525  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:09.158474 11689 consensus_queue.cc:1175] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f613ba929b1046d3b5aed909414a95bf" responder_term: 15 status { last_received { term: 15 index: 12 } last_committed_idx: 12 last_received_current_leader { term: 15 index: 12 } }
I20260709 12:27:09.461969  5561 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: [17.15-17.15]
I20260709 12:27:09.392910 11788 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:09.464578 11773 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:09.397660 11772 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.403227 11726 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:09.397889 11782 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:09.502071  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:09.408077 11781 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:09.468317 11718 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:09.469223 11789 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 12:27:09.471287 11554 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.453675 11732 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:27:09.458748 11747 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:09.462509  5627 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:09.422428 11549 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:27:09.482161  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:09.487943  5636 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:09.496945  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:09.528921  5560 raft_consensus.cc:2368] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19.
I20260709 12:27:09.475154 11676 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:09.514930  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 105216us
I20260709 12:27:09.475443 11601 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:27:09.518833  5634 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.010s	sys 0.021s
I20260709 12:27:09.488978 11741 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:09.478259 11773 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:09.519368 11739 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.005s	sys 0.026s
I20260709 12:27:09.529937  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:09.502313 11726 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:09.493243 11788 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:09.562696  5627 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:09.529965 11667 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:09.563233  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:09.565418  5634 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.010s	sys 0.022s
I20260709 12:27:09.565966  6276 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:09.577397  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:09.578926  5710 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 12:27:09.588961  5710 pending_rounds.cc:85] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 7
I20260709 12:27:09.589290  5710 pending_rounds.cc:107] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 12:27:09.589553  5710 raft_consensus.cc:2889] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:27:09.590960  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:09.563678  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:09.582566  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:09.489790 11420 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.580892 11726 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:09.581017 11697 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:27:09.616789  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:09.572404 11684 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:09.620801  5710 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:27:09.622402  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:09.625486  6278 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
W20260709 12:27:09.626859  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63723.2us
I20260709 12:27:09.627959  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:09.568131 11676 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.629992  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:27:09.631166  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:09.609753 11756 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:09.610109 11715 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:09.616384 11739 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:09.642970 11420 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:09.639066 11684 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:09.597654 11667 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:09.690565  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:09.640969 11690 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.834s	user 0.011s	sys 0.005s
I20260709 12:27:09.698525  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:09.640214 11475 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:09.711046 11715 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:09.712375 11663 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.713485 11676 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:09.713994 11720 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.948s	user 0.008s	sys 0.008s
I20260709 12:27:09.714735 11554 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:09.792865 11715 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:09.797094  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:09.645675 11773 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:09.718066 11772 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:09.623868 11782 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:09.830145 11796 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.234s	user 0.002s	sys 0.000s
I20260709 12:27:09.616272 11781 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.648545 11788 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:09.715215 11802 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 12:27:09.725593  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:09.728135  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:09.574991 11755 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.715309 11761 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:09.731978  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:09.732712  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:09.842108  6278 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 17
W20260709 12:27:09.734207  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 106369us
I20260709 12:27:09.843166  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:09.712141 11784 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.428s	user 0.003s	sys 0.000s
I20260709 12:27:09.711071 11762 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:09.816130  5706 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:09.733582 11686 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:09.847535  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:09.846462  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:09.816497 11787 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.281s	user 0.000s	sys 0.002s
I20260709 12:27:09.847429  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:09.855350  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:09.857506  6279 raft_consensus.cc:2368] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19.
I20260709 12:27:09.860117  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:09.861567  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:09.803354 11475 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:09.841720 11761 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:09.846098 11739 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:27:09.877255  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 109540us
I20260709 12:27:09.845616 11551 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:09.851800 11773 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:09.878162  5636 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 15. (term mismatch)
I20260709 12:27:09.814088 11690 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.872226  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:09.870556 11774 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.009s	sys 0.009s
I20260709 12:27:09.866667 11687 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:27:09.888655  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:09.882723  5675 leader_election.cc:336] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.777s (SENT)
I20260709 12:27:09.881757  5636 pending_rounds.cc:85] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 14
I20260709 12:27:09.867476 11737 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:09.850512 11601 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:09.890724  5636 pending_rounds.cc:107] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 17.15
W20260709 12:27:09.886778  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:09.882645 11794 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.275s	user 0.000s	sys 0.002s
I20260709 12:27:09.886948 11809 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:27:09.878106 11704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.822s	user 0.008s	sys 0.011s
I20260709 12:27:09.881966 11697 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:09.853544 11686 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:09.885726 11730 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.587s	user 0.012s	sys 0.000s
I20260709 12:27:09.892930  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2455) took 1060 ms. Trace:
W20260709 12:27:09.902586 11704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.008s	sys 0.011s
W20260709 12:27:09.887115 11774 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.009s	sys 0.009s
I20260709 12:27:09.905982  5633 rpcz_store.cc:276] 0709 12:27:08.832227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:08.848004 (+ 15777us) service_pool.cc:224] Handling call
0709 12:27:08.848411 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:08.848755 (+   344us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:08.848791 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:08.849221 (+   430us) log.cc:844] Serialized 26 byte log entry
0709 12:27:09.144558 (+295337us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:09.390756 (+246198us) log.cc:844] Serialized 14 byte log entry
0709 12:27:09.424622 (+ 33866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:09.424694 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:09.892690 (+467996us) raft_consensus.cc:1685] finished
0709 12:27:09.892711 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:09.892872 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33611,"prepare.queue_time_us":228277,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":85225,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":13219,"thread_start_us":365364,"threads_started":2,"wal-append.queue_time_us":427443}
I20260709 12:27:09.892431  5636 raft_consensus.cc:2889] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:09.886240  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:27:09.901847  5675 leader_election.cc:336] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.991s (SENT)
I20260709 12:27:09.912454  5636 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 15. Current term is 16. Ops: []
W20260709 12:27:09.912732  5675 leader_election.cc:336] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.008s (SENT)
I20260709 12:27:09.913331  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:09.914223  5675 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.211s (SENT)
W20260709 12:27:09.902058  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 69066.6us
I20260709 12:27:09.898479  5629 raft_consensus.cc:2368] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17.
W20260709 12:27:09.895148 11797 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.199s	user 0.001s	sys 0.001s
I20260709 12:27:09.885105 11720 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:09.916718  5602 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:09.924000 11690 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:09.924710  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:09.925397  6274 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:09.926690 11809 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:27:09.927250 11809 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:09.927553  5631 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:09.928383 11809 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.929551 11726 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.929605  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2718) took 1063 ms. Trace:
I20260709 12:27:09.933307  5631 rpcz_store.cc:276] 0709 12:27:08.866214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:08.938780 (+ 72566us) service_pool.cc:224] Handling call
0709 12:27:09.929442 (+990662us) spinlock_profiling.cc:243] Waited 982 ms on lock 0x7b540008c0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:09.929552 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2258891008}
I20260709 12:27:09.934862  5631 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:27:09.936370 11660 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:09.936932 11660 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.938134  6276 raft_consensus.cc:2410] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:27:09.939031 11660 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:09.940327  5702 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:09.940582  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1067) took 1501 ms. Trace:
I20260709 12:27:09.941226 11804 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:09.940994  6275 rpcz_store.cc:276] 0709 12:27:08.438862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:08.439891 (+  1029us) service_pool.cc:224] Handling call
0709 12:27:08.440273 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:08.833280 (+393007us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:09.940337 (+1107057us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b5400343ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:09.940524 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":801974144}
I20260709 12:27:09.941737 11667 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:09.942559 11808 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:09.943243  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1109) took 1120 ms. Trace:
I20260709 12:27:09.942833 11726 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:09.945969 11209 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:09.947409  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2460) took 1096 ms. Trace:
W20260709 12:27:09.947991  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:09.949513 11755 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:09.938763 11601 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.917672 11487 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:09.947760  5630 rpcz_store.cc:276] 0709 12:27:08.851497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:08.851767 (+   270us) service_pool.cc:224] Handling call
0709 12:27:08.869378 (+ 17611us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:08.931415 (+ 62037us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:09.947214 (+1015799us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b540032d7d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:09.947351 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":39816064}
I20260709 12:27:09.943671  5559 rpcz_store.cc:276] 0709 12:27:08.822963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:09.017248 (+194285us) service_pool.cc:224] Handling call
0709 12:27:09.127025 (+109777us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:09.148476 (+ 21451us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:09.943043 (+794567us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b54000eb0d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:09.943183 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":251728256}
I20260709 12:27:09.956595 11799 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:09.959123  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:09.962239  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:09.962809  5630 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:09.963666  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:09.964083  5630 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:09.883129 11793 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.035s
W20260709 12:27:09.966259  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49590.2us
I20260709 12:27:09.967195  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:09.967564  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:09.968072  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:09.969168  6278 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:09.910221 11704 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:09.975538 11799 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:09.980891 11724 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:27:09.983845  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:09.984701 11660 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:27:09.975695 11549 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:09.986956 11728 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:09.988695 11747 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.989302 11597 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:27:09.989840  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:09.982187 11806 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260709 12:27:09.944170 11475 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:09.976603 11667 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:10.083057  5631 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 10. Current term is 11. Ops: []
I20260709 12:27:10.025285  5630 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
W20260709 12:27:09.974413 11794 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:27:09.909600 11730 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.965874 11720 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:09.978488 11774 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:27:09.992130 11787 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:27:10.088824 11804 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:10.089373 11793 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:10.089589 11804 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.090044 11793 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:09.983435 11436 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.102s
I20260709 12:27:10.080364 11799 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:10.087069 11697 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.093581 11581 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.095077  5636 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: []
W20260709 12:27:10.023892 11748 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.866s	user 0.008s	sys 0.005s
W20260709 12:27:10.002928 11744 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.571s	user 0.010s	sys 0.001s
I20260709 12:27:10.084422  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:10.099539  5630 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:10.099926  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2479) took 1286 ms. Trace:
I20260709 12:27:10.085551  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:09.911015 11761 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.084587 11667 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:09.973773 11487 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.104s
I20260709 12:27:09.993847 11732 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.091s
I20260709 12:27:10.080588 11728 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:10.100507  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:09.974823 11209 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.079s
I20260709 12:27:10.094604 11581 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:27:10.103989  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 120233us
I20260709 12:27:10.103897  5627 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:10.101267  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:10.103104  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2453) took 1276 ms. Trace:
I20260709 12:27:10.092604 11704 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.128041  5675 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:10.129554 11728 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:10.130034 11728 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:10.130137  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2644) took 1306 ms. Trace:
I20260709 12:27:10.130542 11728 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:10.132741  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:10.135114 11791 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.838s	user 0.000s	sys 0.003s
I20260709 12:27:10.089597 11684 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:10.093863 11799 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.113629  5630 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:27:10.110052  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 120105us
I20260709 12:27:10.101956  5554 rpcz_store.cc:276] 0709 12:27:08.813193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:08.963350 (+150157us) service_pool.cc:224] Handling call
0709 12:27:08.963746 (+   396us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:08.967010 (+  3264us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:08.967043 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:08.982972 (+ 15929us) log.cc:844] Serialized 52 byte log entry
0709 12:27:09.515996 (+533024us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:09.778064 (+262068us) log.cc:844] Serialized 14 byte log entry
0709 12:27:09.778522 (+   458us) log.cc:844] Serialized 14 byte log entry
0709 12:27:10.099575 (+321053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:10.099665 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:10.099707 (+    42us) raft_consensus.cc:1685] finished
0709 12:27:10.099723 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:10.099867 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2835,"prepare.queue_time_us":503472,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":85,"raft.queue_time_us":97530,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":18152,"thread_start_us":977703,"threads_started":3,"wal-append.queue_time_us":564044}
I20260709 12:27:10.130667  5558 rpcz_store.cc:276] 0709 12:27:08.823583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:09.021494 (+197911us) service_pool.cc:224] Handling call
0709 12:27:09.021831 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:09.097977 (+ 76146us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:27:09.113995 (+ 16018us) log.cc:844] Serialized 14 byte log entry
0709 12:27:09.285865 (+171870us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:09.286644 (+   779us) log.cc:844] Serialized 26 byte log entry
0709 12:27:09.286823 (+   179us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:27:09.286858 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:09.286935 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:10.112907 (+825972us) spinlock_profiling.cc:243] Waited 5.32 ms on lock 0x7b180008b2a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:27:10.129810 (+ 16903us) raft_consensus.cc:1685] finished
0709 12:27:10.129923 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:10.130068 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21656,"prepare.queue_time_us":15968,"raft.queue_time_us":80087,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":24567,"spinlock_wait_cycles":12226432,"thread_start_us":165566,"threads_started":1,"wal-append.queue_time_us":334567}
W20260709 12:27:10.147198  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:10.148558  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:10.148855  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:10.097729 11774 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:10.136444  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:10.141533  6274 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:10.107169 11667 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:10.128244  5629 rpcz_store.cc:276] 0709 12:27:08.826639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:08.826764 (+   125us) service_pool.cc:224] Handling call
0709 12:27:10.102902 (+1276138us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54000c0b54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:10.103043 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2462247296}
I20260709 12:27:10.145246  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:10.106921 11755 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:27:10.130185 11820 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:27:10.153564  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:10.155112  6274 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:10.156072  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:10.157212  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:10.157661  5630 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:10.158551  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:10.100494 11748 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.137115 11747 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:10.151295 11817 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 12:27:10.159932  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:10.159941  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:10.183947  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:10.176007  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1162) took 1158 ms. Trace:
I20260709 12:27:10.162354  5629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:10.195632  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2560) took 1210 ms. Trace:
I20260709 12:27:10.196064  5706 rpcz_store.cc:276] 0709 12:27:08.985038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:08.985154 (+   116us) service_pool.cc:224] Handling call
0709 12:27:10.195421 (+1210267us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b54002f5154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:10.195566 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1758855552}
I20260709 12:27:10.194880  5701 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:27:10.194845  5631 rpcz_store.cc:276] 0709 12:27:09.017730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:09.947990 (+930260us) service_pool.cc:224] Handling call
0709 12:27:09.949540 (+  1550us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:10.083508 (+133968us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:10.175738 (+ 92230us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54001cf7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:10.175940 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":306402176}
I20260709 12:27:10.198722  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:10.199478  5630 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:10.199717  5631 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: [15.10-15.10]
W20260709 12:27:10.197783  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:10.201104  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:10.201141  5558 raft_consensus.cc:2410] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
W20260709 12:27:10.103725 11819 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.146s	user 0.008s	sys 0.007s
I20260709 12:27:10.152954 11774 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:10.204607  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 68805.6us
I20260709 12:27:10.196856  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:10.211614  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:10.212880  5552 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:10.218050  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:10.177265 11786 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.876s	user 0.002s	sys 0.000s
I20260709 12:27:10.219270 11436 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:10.179450 11720 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:10.197106 11744 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:10.222532 11748 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:10.233264  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:10.160457 11755 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:10.217880 11732 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:10.195152  5629 raft_consensus.cc:606] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:27:10.283481  5629 raft_consensus.cc:614] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:27:10.249327  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:10.250653  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:10.254390  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:10.264731 11436 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:10.242606 11209 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:10.178879 11379 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:10.288256 11436 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:10.265213  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1034) took 1328 ms. Trace:
I20260709 12:27:10.265861  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:10.292982 11487 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:10.293787 11715 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:10.294059  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:10.295925 11209 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.296051  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:10.283715 11744 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.299625 11720 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:10.283373 11732 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:10.300796 11732 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:10.301393 11744 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:10.298832 11487 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:10.303874  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:10.294312 11715 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.304800 11487 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:10.305975  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2448) took 1497 ms. Trace:
I20260709 12:27:10.291527 11667 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.273918  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:10.308523  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:10.308895  5560 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:10.304280 11209 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:10.203173 11809 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.079s
I20260709 12:27:10.311172 11809 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:10.311507 11809 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:10.174129 11793 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:10.174362 11829 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:10.312744 11829 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:10.312469 11809 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.313163 11829 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:10.306664 11715 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:10.313889 11809 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:10.315511 11798 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.550s	user 0.001s	sys 0.000s
W20260709 12:27:10.277355  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51979.4us
I20260709 12:27:10.292555  5703 rpcz_store.cc:276] 0709 12:27:08.936961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:08.938306 (+  1345us) service_pool.cc:224] Handling call
0709 12:27:08.938691 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:08.960923 (+ 22232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:08.960975 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:08.970040 (+  9065us) log.cc:844] Serialized 26 byte log entry
0709 12:27:09.149451 (+179411us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:09.149503 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:09.149598 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:10.134415 (+984817us) spinlock_profiling.cc:243] Waited 52.9 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:10.232839 (+ 98424us) raft_consensus.cc:1685] finished
0709 12:27:10.232875 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:10.265133 (+ 32258us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21743,"raft.queue_time_us":22017,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":16784,"spinlock_wait_cycles":121579776,"thread_start_us":122545,"threads_started":1,"wal-append.queue_time_us":289492}
I20260709 12:27:10.319289  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:10.172547 11823 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:10.282294  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:10.269881  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:10.219043 11731 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:10.284678  5557 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:10.258296  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:10.273808 11747 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:10.284684  5553 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 17: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:10.173720 11730 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:10.242826 11687 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:10.345008 11687 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.347481 11687 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:10.347983  6275 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:27:10.348289  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:10.349133  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:10.351728  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:10.281769 11660 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 12:27:10.354424  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:10.355464 11755 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.355854  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:10.276340 11636 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:10.357084 11636 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.359223 11636 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:10.205716 11781 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.081s
I20260709 12:27:10.360088 11737 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:10.360639 11737 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.362731 11737 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:10.364065  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:10.364843  6278 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:10.364925 11209 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:10.365427 11209 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.367364 11436 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.162894 11832 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:10.279470  5630 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:10.320231  6274 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:10.206259 11420 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:10.317544  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2450) took 1492 ms. Trace:
I20260709 12:27:10.220765  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:10.321637  5560 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:10.395993  5559 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:27:10.271240  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 111363us
I20260709 12:27:10.340427  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1019) took 2401 ms. Trace:
W20260709 12:27:10.397154 11805 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.680s	user 0.000s	sys 0.002s
I20260709 12:27:10.397324  5709 rpcz_store.cc:276] 0709 12:27:07.938721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:07.939242 (+   521us) service_pool.cc:224] Handling call
0709 12:27:10.340112 (+2400870us) spinlock_profiling.cc:243] Waited 908 ms on lock 0x7b5400323054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:10.340283 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2087715712}
I20260709 12:27:10.399299  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1075) took 1065 ms. Trace:
I20260709 12:27:10.399722  6273 rpcz_store.cc:276] 0709 12:27:09.333572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:09.333737 (+   165us) service_pool.cc:224] Handling call
0709 12:27:09.357461 (+ 23724us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:09.357897 (+   436us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:09.357941 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:09.371109 (+ 13168us) log.cc:844] Serialized 26 byte log entry
0709 12:27:09.511708 (+140599us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:09.511774 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:09.511879 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:10.328581 (+816702us) spinlock_profiling.cc:243] Waited 7.96 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:10.398812 (+ 70231us) raft_consensus.cc:1685] finished
0709 12:27:10.398847 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:10.399102 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":241,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":14951,"spinlock_wait_cycles":18316544,"thread_start_us":132671,"threads_started":1,"wal-append.queue_time_us":245419}
I20260709 12:27:10.402701 11823 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:10.367942 11209 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:10.306646  5635 rpcz_store.cc:276] 0709 12:27:08.808861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:08.823769 (+ 14908us) service_pool.cc:224] Handling call
0709 12:27:08.824099 (+   330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:08.845639 (+ 21540us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:08.845669 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:08.846138 (+   469us) log.cc:844] Serialized 26 byte log entry
0709 12:27:09.020118 (+173980us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:09.282038 (+261920us) log.cc:844] Serialized 14 byte log entry
0709 12:27:09.478917 (+196879us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:09.479011 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:10.176475 (+697464us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:10.305689 (+129214us) raft_consensus.cc:1685] finished
0709 12:27:10.305787 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:10.305919 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":98627,"prepare.queue_time_us":352402,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":156634,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":13318,"spinlock_wait_cycles":45340928,"thread_start_us":464860,"threads_started":3,"wal-append.queue_time_us":312060}
I20260709 12:27:10.405293  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:10.406126  5635 raft_consensus.cc:2376] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14. Re-sending same reply.
I20260709 12:27:10.407315  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:10.285077 11554 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:10.372888  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:10.377107  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:10.378448 11420 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.374349 11487 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:10.380693 11781 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:10.370457 11720 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:10.372535  5701 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:10.428401  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:10.410948  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:10.411573  5634 rpcz_store.cc:276] 0709 12:27:08.825124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:08.825256 (+   132us) service_pool.cc:224] Handling call
0709 12:27:08.825573 (+   317us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:08.846644 (+ 21071us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:08.846684 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:08.874040 (+ 27356us) log.cc:844] Serialized 26 byte log entry
0709 12:27:09.450497 (+576457us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b48002727d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:27:09.587507 (+137010us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:09.587553 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:09.587634 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:10.314904 (+727270us) spinlock_profiling.cc:243] Waited 7.47 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:10.317337 (+  2433us) raft_consensus.cc:1685] finished
0709 12:27:10.317357 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:10.317489 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20721,"raft.queue_time_us":36141,"raft.run_cpu_time_us":5145,"raft.run_wall_time_us":585114,"spinlock_wait_cycles":41013248,"thread_start_us":617838,"threads_started":1,"wal-append.queue_time_us":649585}
I20260709 12:27:10.423885  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:10.426251  5628 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:10.436214  5629 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:10.419148  5552 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:10.443511  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:10.485615  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:10.344914 11728 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.419767 11689 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:10.469782  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:10.501410  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:10.332553 11833 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.125s
I20260709 12:27:10.483443  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:10.503952  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:10.437721 11597 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:10.326980 11731 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.510455  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:10.435379 11575 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:10.427719 11554 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.642395  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:10.689265  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 205890us
W20260709 12:27:10.680147  6279 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1091) took 1746 ms (client timeout 1763 ms). Trace:
I20260709 12:27:10.490262 11724 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:10.636448  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:10.608793  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:10.721915  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:10.722558  6279 rpcz_store.cc:269] 0709 12:27:08.923654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:08.923782 (+   128us) service_pool.cc:224] Handling call
0709 12:27:10.669694 (+1745912us) spinlock_profiling.cc:243] Waited 888 ms on lock 0x7b54002922d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:10.669870 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2042590976}
I20260709 12:27:10.530023 11420 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:10.737186 11839 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.219s	user 0.000s	sys 0.000s
I20260709 12:27:10.512195  5634 raft_consensus.cc:2376] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15. Re-sending same reply.
I20260709 12:27:10.486732 11747 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:27:10.677510  6273 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:27:10.752338  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:10.762609  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:10.484676 11773 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:10.449337 11487 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:10.735765 11728 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:10.528879 11721 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:27:10.807751  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:10.759778  5629 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:27:10.771598  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 408571us
W20260709 12:27:10.543665 11762 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.693s	user 0.004s	sys 0.009s
I20260709 12:27:10.603997 11696 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:10.789754  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:10.427960 11781 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:27:10.794549  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:10.702960 11587 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:10.629657 11581 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:10.813016  6275 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:27:10.848829  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:10.849715  5703 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:27:10.850720  5631 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:27:10.852002  5702 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: [11.9-12.10]
I20260709 12:27:10.852568  5631 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:27:10.854367  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:10.771978 11420 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:27:10.628311 11209 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:10.877107  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:10.530040 11597 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:10.764909 11715 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:10.742846 11724 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:10.508005 11689 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.857515  5634 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:27:10.815522 11731 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:10.859645  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:10.629977 11575 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.883494  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:10.884326  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:10.911548  5706 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:10.807119 11755 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:27:10.845916 11829 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.532s	user 0.008s	sys 0.004s
I20260709 12:27:10.890413  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:10.916234  5708 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:10.891100  5631 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 12:27:10.898722 11732 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.597s	user 0.005s	sys 0.006s
W20260709 12:27:10.895637  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 131366us
I20260709 12:27:10.898077 11730 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:10.539471 11774 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.904211 11436 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:10.880815 11667 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:10.903893 11734 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260709 12:27:10.876848 11581 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.906894 11724 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.857803 11696 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.903875 11759 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:10.877565 11611 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:10.908623 11487 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.909771 11689 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:10.912802  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1157) took 1939 ms. Trace:
I20260709 12:27:10.966696  5633 rpcz_store.cc:276] 0709 12:27:08.973617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:09.911289 (+937672us) service_pool.cc:224] Handling call
0709 12:27:09.911642 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:09.919384 (+  7742us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:09.919416 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:09.919460 (+    44us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:09.919496 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:10.912490 (+992994us) spinlock_profiling.cc:243] Waited 7.53 ms on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:10.912580 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:10.912732 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17318272}
I20260709 12:27:10.807461 11721 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:10.840942 11833 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.085s
W20260709 12:27:10.918358  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 286014us
W20260709 12:27:10.975862  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:10.903082  5705 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:10.938982  5706 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:10.913938 11731 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:10.944715  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:10.955454  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 296100us
I20260709 12:27:10.898332 11209 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.950194 11654 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:10.962441 11611 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:10.917215 11755 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:11.026094  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:10.968149  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:10.931505 11420 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 12:27:10.951383 11730 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:10.918411 11829 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.962743 11597 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:10.986254 11731 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.991089 11724 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:11.010911 11554 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.008037 11842 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.005s
I20260709 12:27:11.008037 11788 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:10.981405 11689 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:10.991449 11479 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:11.014999 11774 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.015240 11209 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:11.017263 11667 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:27:11.017514 11812 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.306s	user 0.006s	sys 0.000s
I20260709 12:27:11.022641 11696 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:10.931201 11732 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:11.024994  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:11.027791  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:11.060236 11762 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:11.030061  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:11.097121  5702 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 16
W20260709 12:27:11.069883 11782 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.167s	user 0.011s	sys 0.043s
I20260709 12:27:11.050346  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:10.918311 11781 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:10.980376 11759 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.053371 11581 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:11.030172 11654 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:11.090114  5602 leader_election.cc:400] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19.
I20260709 12:27:11.109822  5602 leader_election.cc:403] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:27:11.064802 11729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.745s	user 0.006s	sys 0.047s
I20260709 12:27:11.025601 11832 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:11.114200 11654 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:11.114696 11731 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.105053 11762 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.010251  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:27:11.102784 11782 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.202s	user 0.014s	sys 0.043s
I20260709 12:27:11.132978  6273 raft_consensus.cc:2368] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
W20260709 12:27:11.085417 11811 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.368s	user 0.014s	sys 0.014s
I20260709 12:27:11.104943  6274 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:11.052565 11730 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:11.112901 11636 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19.
W20260709 12:27:11.113373 11729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.794s	user 0.006s	sys 0.049s
W20260709 12:27:11.113433 11813 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.395s	user 0.000s	sys 0.008s
I20260709 12:27:11.112696  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:11.114319 11832 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.099092  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:11.135744  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:11.026263 11611 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:11.166306  5710 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:27:11.176564 11781 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:11.180656  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:11.181748  5709 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 16 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 6.
I20260709 12:27:11.096004 11667 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:11.196163 11837 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.204s	user 0.001s	sys 0.000s
I20260709 12:27:11.181126 11829 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:11.198514 11840 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 12:27:11.068744 11689 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:11.209062  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2772) took 1025 ms. Trace:
I20260709 12:27:11.064585 11788 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.209653  5707 rpcz_store.cc:276] 0709 12:27:10.183168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:10.185385 (+  2217us) service_pool.cc:224] Handling call
0709 12:27:10.185749 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:10.186023 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:10.186063 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:10.186618 (+   555us) log.cc:844] Serialized 26 byte log entry
0709 12:27:10.679303 (+492685us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:27:10.679346 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:10.679419 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:11.208743 (+529324us) raft_consensus.cc:1685] finished
0709 12:27:11.208773 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:11.208984 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":473676,"threads_started":1,"wal-append.queue_time_us":733468}
I20260709 12:27:11.211553 11636 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:27:11.174471 11809 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.859s	user 0.010s	sys 0.003s
I20260709 12:27:11.158754 11731 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:11.243335  5635 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:27:11.249622  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2721) took 2382 ms. Trace:
I20260709 12:27:11.250161  5632 rpcz_store.cc:276] 0709 12:27:08.866905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:09.465975 (+599070us) service_pool.cc:224] Handling call
0709 12:27:09.466297 (+   322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:09.466660 (+   363us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:09.466690 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:09.467212 (+   522us) log.cc:844] Serialized 26 byte log entry
0709 12:27:09.582487 (+115275us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:09.609144 (+ 26657us) log.cc:844] Serialized 14 byte log entry
0709 12:27:09.720024 (+110880us) spinlock_profiling.cc:243] Waited 3.58 ms on lock 0x7b180024e740. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:27:09.720271 (+   247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:09.726425 (+  6154us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:10.917464 (+1191039us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:11.249347 (+331883us) raft_consensus.cc:1685] finished
0709 12:27:11.249378 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:11.249536 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19640,"prepare.queue_time_us":28269,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":133304,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":46472,"spinlock_wait_cycles":53816064,"thread_start_us":114853,"threads_started":1,"wal-append.queue_time_us":249249}
W20260709 12:27:11.236341  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 123781us
I20260709 12:27:11.217921 11729 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:27:11.265475  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:11.266695  5552 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:11.259902 11829 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:11.271752 11829 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.272739 11832 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:11.273526 11715 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:11.274647  5553 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:27:11.278277  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:11.280409  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:11.281450  5557 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 18: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:11.283486  5559 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 10. (term mismatch)
I20260709 12:27:11.284118  5559 pending_rounds.cc:85] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 9
I20260709 12:27:11.284646  5559 pending_rounds.cc:107] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 15.10
I20260709 12:27:11.284962  5559 raft_consensus.cc:2889] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:11.287425  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:11.288589  5559 raft_consensus.cc:2410] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:27:11.228943 11732 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.296427  5675 leader_election.cc:400] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:11.297013  5675 leader_election.cc:403] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
W20260709 12:27:11.295476  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 286472us
I20260709 12:27:11.319060  5702 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:11.321132  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:11.326846  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:11.338841  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:11.340508 11729 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.003s
W20260709 12:27:11.343792 11795 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.623s	user 0.009s	sys 0.005s
I20260709 12:27:11.345126  5709 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 13. (term mismatch)
I20260709 12:27:11.345497  5709 pending_rounds.cc:85] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 12
I20260709 12:27:11.345716  5709 pending_rounds.cc:107] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260709 12:27:11.345777 11755 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:11.346016  5709 raft_consensus.cc:2889] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:11.347599 11790 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:11.347795 11762 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:11.348173 11790 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:11.349318  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:11.348441 11601 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:11.247563 11731 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:11.359897  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:11.303907 11841 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:11.358453 11654 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:11.369611 11587 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.521s	user 0.010s	sys 0.002s
I20260709 12:27:11.375114  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:11.361285 11420 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:11.375594 11654 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.379016 11689 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.364578 11601 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:11.389905 11804 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:11.366020 11788 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.375876 11832 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:11.397454  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:11.419741  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1154) took 2495 ms. Trace:
I20260709 12:27:11.471630  5627 rpcz_store.cc:276] 0709 12:27:08.924235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:09.581234 (+656999us) service_pool.cc:224] Handling call
0709 12:27:10.103723 (+522489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:10.127741 (+ 24018us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:11.419557 (+1291816us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54000c0b50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:11.419683 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1134299392}
I20260709 12:27:11.434530  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1076) took 1083 ms. Trace:
I20260709 12:27:11.473707  6275 rpcz_store.cc:276] 0709 12:27:10.350956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:10.363375 (+ 12419us) service_pool.cc:224] Handling call
0709 12:27:10.363662 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:10.847057 (+483395us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:11.434297 (+587240us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b54002eed54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:11.434473 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":988485504}
I20260709 12:27:11.435487  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2478) took 1218 ms. Trace:
I20260709 12:27:11.475775  5635 rpcz_store.cc:276] 0709 12:27:10.251295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:10.406977 (+155682us) service_pool.cc:224] Handling call
0709 12:27:11.435315 (+1028338us) spinlock_profiling.cc:243] Waited 835 ms on lock 0x7b5400438854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:11.435434 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1920210176}
I20260709 12:27:11.477418  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:11.478550  5635 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
W20260709 12:27:11.420163  5628 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2457) took 2571 ms (client timeout 1907 ms). Trace:
W20260709 12:27:11.480443  5628 rpcz_store.cc:269] 0709 12:27:08.848370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:08.849466 (+  1096us) service_pool.cc:224] Handling call
0709 12:27:11.420005 (+2570539us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b540018c254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:11.420099 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3370333056}
I20260709 12:27:11.481859  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:11.482658  5628 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:11.440519  5552 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:11.445464  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:11.447757  5633 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:27:11.447892  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:11.488281  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:11.488919  5701 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:11.449568  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1160) took 2454 ms. Trace:
I20260709 12:27:11.491130  5636 rpcz_store.cc:276] 0709 12:27:08.995580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:09.913469 (+917889us) service_pool.cc:224] Handling call
0709 12:27:09.913792 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:10.095716 (+181924us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:11.449340 (+1353624us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b54001cfcd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:11.449499 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":416484096}
I20260709 12:27:11.449880  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:11.450490  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:11.494277  5629 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:11.451061  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:11.457216  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:11.498281  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:11.499821  5701 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:27:11.499987  6237 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:11.424345  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:11.446838  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:11.501293  5701 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:27:11.392282  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:11.417605  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2483) took 1269 ms. Trace:
I20260709 12:27:11.449084  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:11.512248  5558 rpcz_store.cc:276] 0709 12:27:10.148107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:10.148274 (+   167us) service_pool.cc:224] Handling call
0709 12:27:11.417381 (+1269107us) spinlock_profiling.cc:243] Waited 51.6 ms on lock 0x7b54001c48d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:11.417528 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":118568704}
I20260709 12:27:11.513173  5709 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:11.493227  5631 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:11.437582  5703 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260709 12:27:11.507268  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48990.2us
W20260709 12:27:11.437151  5705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2564) took 2399 ms (client timeout 1605 ms). Trace:
W20260709 12:27:11.517122  5705 rpcz_store.cc:269] 0709 12:27:09.037703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:09.044166 (+  6463us) service_pool.cc:224] Handling call
0709 12:27:11.436966 (+2392800us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b54003e6f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:11.437086 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3936227200}
I20260709 12:27:11.445892  5630 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:27:11.424533  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:11.516523  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:11.544472  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:11.546386  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:11.555408  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55506.9us
I20260709 12:27:11.557684  5601 leader_election.cc:400] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19.
I20260709 12:27:11.559839 11549 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:11.560619 11549 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.563683 11549 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:11.436165 11831 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.274s	user 0.000s	sys 0.010s
W20260709 12:27:11.565564  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 120178us
W20260709 12:27:11.401081 11814 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.684s	user 0.002s	sys 0.005s
W20260709 12:27:11.446858 11844 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.078s	user 0.000s	sys 0.002s
I20260709 12:27:11.429404 11731 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.406970 11761 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:11.423651 11663 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:11.428565 11597 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:11.402312 11686 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:11.637861  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:11.433357 11832 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.399786 11601 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.405386 11667 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:11.456061 11587 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:11.600085  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:11.673811 11587 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:11.412313 11721 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:11.589497  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:11.674341 11587 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:11.674691  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 85315.9us
I20260709 12:27:11.675256 11587 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.567384 11636 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 12:27:11.678856 11856 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:11.679502 11856 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:11.679936 11856 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:11.680473 11689 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:11.553964 11696 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:11.393138 11715 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:11.449230 11823 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:11.432996 11730 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:11.499253 11479 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:11.442140 11575 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.411079 11759 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.422509 11790 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:11.430701 11829 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.564814 11781 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:11.564982 11744 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:11.569787 11817 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:27:11.638803  5627 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:27:11.577484  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:11.570448 11549 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:27:11.665771  5629 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:11.432878 11420 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.605734 11799 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:11.604118 11833 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:11.673079 11667 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:11.553416 11850 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.085s
I20260709 12:27:11.567040 11809 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.072s
I20260709 12:27:11.687381  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:11.766706  5630 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:11.687983  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:11.688788  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:11.770956  5633 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:11.689711  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:11.772001  5635 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:11.681627 11804 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:11.653334 11654 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.612421 11732 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.082s
I20260709 12:27:11.724713  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:11.726223  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:11.597630 11720 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.090s
I20260709 12:27:11.668583 11597 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.743275  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:11.703222 11855 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:11.693190 11601 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.050s
I20260709 12:27:11.727345 11209 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:11.760201  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:11.861480 11209 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:11.724306 11781 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:11.862462 11781 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:11.863659 11209 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:11.865326  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:11.866114  6274 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 14: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:11.860235 11420 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.867084  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:11.867100 11832 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:11.774725  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:11.868340  6273 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:27:11.783814  5627 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:11.695809 11823 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.869438  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:11.871097  6273 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260709 12:27:11.871430  6273 pending_rounds.cc:85] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 9
I20260709 12:27:11.871624 11823 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:11.758673 11611 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.545s	user 0.013s	sys 0.001s
I20260709 12:27:11.704046 11715 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.057s
I20260709 12:27:11.872320  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:11.873054  6280 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:11.872651 11611 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.875276 11611 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.773495  5636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:11.877107  5636 raft_consensus.cc:606] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:27:11.736558 11730 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:11.877708 11730 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:11.878424  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:11.879207  6277 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:11.879782 11730 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.702090 11479 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.880394  6280 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:27:11.880403 11781 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:11.881894  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:11.882220  5631 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:11.882751 11781 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.844684  5630 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:11.844707  5708 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:27:11.845322  5676 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.821s	user 0.012s	sys 0.023s
W20260709 12:27:11.846007  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 102861us
I20260709 12:27:11.885110  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:11.846426  5675 leader_election.cc:336] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.519s (SENT)
I20260709 12:27:11.850608  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:11.887285  6277 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:27:11.851395  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:11.887984  5676 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.012s	sys 0.023s
I20260709 12:27:11.855741  5633 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:11.748014 11799 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:11.857668  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:11.786064 11667 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:11.836501 11723 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:11.861723  5629 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:11.839860 11732 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:11.879809  5561 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:27:11.890017  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:11.871641  6273 pending_rounds.cc:107] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 12:27:11.918386  6273 raft_consensus.cc:2889] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:11.891201  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:11.892262  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:11.859908 11597 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:11.893980  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:11.896034  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:11.896874  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 309511us
I20260709 12:27:11.919605  5708 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:11.925452  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:11.896270  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 864785us
I20260709 12:27:11.899042 11799 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:11.884196 11198 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:11.892798 11744 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:27:11.928550  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:11.921404  5710 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:11.838363 11859 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:11.913836 11696 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:11.946235  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:11.809314 11804 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:11.941534  6279 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.014s	sys 0.025s
W20260709 12:27:11.957472  6279 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.014s	sys 0.025s
I20260709 12:27:11.771323 11833 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:11.913354 11781 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:27:11.951978 11838 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.015s	sys 0.012s
W20260709 12:27:11.952837 11860 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.215s	user 0.000s	sys 0.001s
I20260709 12:27:11.953023 11420 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:27:11.953522 11788 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:27:11.953743 11823 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 12:27:11.943586  5676 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.078s (SENT) [suppressed 19 similar messages]
W20260709 12:27:12.002642  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59088.4us
I20260709 12:27:12.004309  5703 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:27:11.954174 11597 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:27:11.954722 11857 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260709 12:27:11.917486 11732 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:11.955111  5629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8c6474db7a70405694509f6976d850ce"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:12.012398  5629 raft_consensus.cc:606] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:11.955329 11667 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:11.961433  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:27:12.013450  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 52132.6us
I20260709 12:27:12.014300  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:12.014989  5636 raft_consensus.cc:2368] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17.
I20260709 12:27:12.014758  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:11.947409  5708 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:12.015591  5627 raft_consensus.cc:2376] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15. Re-sending same reply.
I20260709 12:27:12.016700  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:12.017791  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:12.018929  5630 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:27:11.963315  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:11.953106  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:12.020785  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 67769.2us
I20260709 12:27:12.021073  5630 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:27:12.021696  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:12.022614  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:11.921633 11815 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:11.988365 11704 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.024s	sys 0.016s
I20260709 12:27:12.005564  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:11.928511 11686 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.064s
I20260709 12:27:12.009936  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:11.968178 11817 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:12.010152  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 617906us
I20260709 12:27:11.949154 11198 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.000320 11850 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:11.886314 11855 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:12.024814  5525 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.015s	sys 0.019s
I20260709 12:27:12.025626  5701 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:12.023810  5630 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: [17.10-17.10]
W20260709 12:27:12.041903  5525 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.015s	sys 0.019s
I20260709 12:27:11.965837 11696 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.040232  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:12.043954  5627 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:12.044234  5552 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260709 12:27:12.044641  5552 pending_rounds.cc:85] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 10
I20260709 12:27:12.044875  5552 pending_rounds.cc:107] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 12:27:12.045109  5552 raft_consensus.cc:2889] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:27:12.026496  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63308.6us
I20260709 12:27:12.045560  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.010759  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:12.034291  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:12.027539  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:12.017726  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:12.000771 11832 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:12.009132 11597 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:11.965145 11859 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:11.921150 11723 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:12.048269  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:12.049293  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.071612  5555 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:12.072338  5560 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
W20260709 12:27:12.075570  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53009.9us
I20260709 12:27:12.050312  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:12.051599  5556 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:12.008121 11744 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:12.025182 11704 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.025s	sys 0.016s
I20260709 12:27:12.035161 11479 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:12.064052  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:12.024314 11575 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:12.031268 11663 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:12.032506 11730 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:12.086186  5558 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:27:12.077805  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:12.053174  5559 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:27:12.064419 11597 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:27:12.090745  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:12.084555  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:12.037705 11861 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:12.085682  6278 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:12.088910  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:12.058138  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:12.053042 11731 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:12.098258  5636 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:12.101066  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.078642 11734 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:12.102622  5557 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:12.104714  5627 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 12:27:12.090984 11841 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.722s	user 0.014s	sys 0.000s
I20260709 12:27:12.085031 11790 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:27:12.108258  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 50228.7us
I20260709 12:27:12.091084 11773 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:12.111281 11696 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:12.081944 11654 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.030s
I20260709 12:27:12.101694 11788 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:12.114526  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:12.068581 11832 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.116189 11855 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:12.172313  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:12.151216 11479 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:12.110368 11601 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:12.032184 11815 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.086364 11663 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:12.091493 11575 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:12.116663 11744 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.093988 11732 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.117486 11734 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:12.118721  5561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 1115) took 1909 ms (client timeout 1742 ms). Trace:
I20260709 12:27:12.119344  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.121405  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:12.192739  5561 rpcz_store.cc:269] 0709 12:27:10.208875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:10.208991 (+   116us) service_pool.cc:224] Handling call
0709 12:27:12.118512 (+1909521us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b540041a0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:12.118659 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3819178240}
I20260709 12:27:12.193323  5555 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:12.118112  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:12.194595  5636 raft_consensus.cc:2368] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16.
I20260709 12:27:12.195830  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:12.040839 11817 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.054965 11850 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:12.124861  6272 raft_consensus.cc:1217] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Deduplicated request from leader. Original: 16.13->[19.14-19.14]   Dedup: 19.14->[]
I20260709 12:27:12.128839 11549 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:12.133179  5554 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
W20260709 12:27:12.136919 11870 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:27:12.140452  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.143458  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:12.145351 11597 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:12.146737 11721 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:12.095109 11880 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.060s
I20260709 12:27:12.085194 11747 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
W20260709 12:27:12.150614  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61804.7us
I20260709 12:27:12.241302  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:12.076046 11759 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20260709 12:27:12.110332 11731 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:12.095736 11774 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:12.158288  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:12.245195  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:12.163106  6278 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:27:12.096928 11861 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:12.113265 11723 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:12.153405 11879 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:12.114782 11859 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:12.152720 11790 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:12.158313 11829 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:12.161903 11841 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:12.191442  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.191635  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:12.123464 11872 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
I20260709 12:27:12.197685  5701 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:12.198379  5552 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:12.199477  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:12.264432  5561 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 17: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:12.216684  6272 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:27:12.215972  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:12.265687  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.217418  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.267798  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:12.268654  5555 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:12.269217 11734 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:12.260908 11790 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:12.175194 11883 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:12.198297  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.213464  5636 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:12.274179 11597 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:12.274889 11597 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:12.275005  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:12.275378 11597 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.275900  5636 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:12.275826 11731 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:12.239113 11855 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:12.222347  5675 leader_election.cc:400] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17.
I20260709 12:27:12.277673  5675 leader_election.cc:403] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
W20260709 12:27:12.278499  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 56338.7us
I20260709 12:27:12.232455  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.280542  5558 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:12.281422  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.282029  5558 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:12.187896 11663 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:12.234072  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.234629  5631 raft_consensus.cc:2368] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:12.236164  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:12.238051  6279 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:12.283216  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:12.283218  5559 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:12.186976 11479 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.239110 11732 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:12.182740 11788 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.267992 11575 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:12.273968 11759 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:12.277091  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:12.274245  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.252971  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:12.671535  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:12.291833  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.290354  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.823230  5557 raft_consensus.cc:2410] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 15 because replica has last-logged OpId of term: 14 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 12:27:12.823802  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.825577  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:12.693743  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:12.698124  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.698632  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.827268  5552 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:27:12.344954  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 317513us
W20260709 12:27:12.344463  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 328793us
W20260709 12:27:12.701021  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.010s	sys 0.001s
I20260709 12:27:12.296868  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.705928  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.339075  5636 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:12.710714  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:12.288223  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:12.713143  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:12.714622  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:12.719349  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.721191  5628 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:27:12.721057  5559 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
W20260709 12:27:12.400173  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:12.845662  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:12.293301  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:12.735893  5701 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90c64867bae6493c846d941815aa1074"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
W20260709 12:27:12.847357  5555 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.578s	user 0.017s	sys 0.009s
I20260709 12:27:12.847985  5701 raft_consensus.cc:606] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:27:12.848733  5701 raft_consensus.cc:614] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:27:12.287884  5709 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:12.850775 11848 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:27:12.289583  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 150146us
I20260709 12:27:12.446106 11198 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:12.301050 11209 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:12.504720 11601 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:12.890271 11601 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.257696 11715 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:12.292301 11720 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17.
I20260709 12:27:12.777159 11420 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:12.893651 11420 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.894743 11601 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:12.425848 11479 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:12.477659 11597 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:12.896890 11720 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:12.779630 11731 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:12.897864 11731 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.897742 11720 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:12.779045 11663 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:12.236047 11686 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.035s
I20260709 12:27:12.216459 11549 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:12.250731 11587 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:12.780889 11721 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:12.296360 11723 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.079s
I20260709 12:27:12.297511 11724 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.079s
I20260709 12:27:12.341962 11759 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.960891 11721 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:12.961689 11914 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:12.962248 11914 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:12.963739 11759 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:12.439963 11773 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:12.964938 11773 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:12.967739 11587 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:12.258281 11774 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:12.255558 11782 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:12.451912 11788 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:12.470841 11799 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.513s	user 0.008s	sys 0.000s
I20260709 12:27:12.981711 11799 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:12.248974 11804 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:12.458014 11815 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:12.465034 11817 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:12.300551 11829 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:12.988862 11829 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:12.269238 11832 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:12.463681 11833 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:12.418319 11841 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:12.431114 11850 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:12.296103 11855 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:13.019973 11841 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:13.021399  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:13.022171 11855 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:13.022406  5701 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:27:12.463447 11856 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.783s	user 0.005s	sys 0.005s
I20260709 12:27:13.023514 11856 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:13.026219 11856 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:12.259657 11879 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:12.457427 11882 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 12:27:12.272161 11883 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:13.033797 11883 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:13.035002  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2591) took 1704 ms. Trace:
I20260709 12:27:13.035475  5702 rpcz_store.cc:276] 0709 12:27:11.329992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:11.364144 (+ 34152us) service_pool.cc:224] Handling call
0709 12:27:11.364587 (+   443us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:11.392346 (+ 27759us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:11.392414 (+    68us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:11.548249 (+155835us) log.cc:844] Serialized 52 byte log entry
0709 12:27:12.025123 (+476874us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:12.066825 (+ 41702us) log.cc:844] Serialized 14 byte log entry
0709 12:27:12.067516 (+   691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:12.067806 (+   290us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:13.034446 (+966640us) raft_consensus.cc:1685] finished
0709 12:27:13.034496 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:13.034760 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37820,"prepare.queue_time_us":81934,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":2523,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":111021,"thread_start_us":408261,"threads_started":1,"wal-append.queue_time_us":663907}
W20260709 12:27:12.449787 11887 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.211s	user 0.000s	sys 0.002s
I20260709 12:27:13.038352 11774 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:12.452894 11891 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
W20260709 12:27:12.468691 11902 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.196s	user 0.002s	sys 0.000s
W20260709 12:27:12.804814  5635 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.012s	sys 0.007s
W20260709 12:27:13.044373  5635 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.861s	user 0.013s	sys 0.007s
I20260709 12:27:13.045749  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1183) took 1497 ms. Trace:
I20260709 12:27:13.046149  5635 rpcz_store.cc:276] 0709 12:27:11.548651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:11.797325 (+248674us) service_pool.cc:224] Handling call
0709 12:27:11.797715 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:11.798013 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:27:12.183143 (+385130us) log.cc:844] Serialized 14 byte log entry
0709 12:27:13.045106 (+861963us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:13.045211 (+   105us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:13.045323 (+   112us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.045456 (+   133us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:13.045658 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14733,"prepare.queue_time_us":85998,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":1452,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":149,"thread_start_us":1179707,"threads_started":2,"wal-append.queue_time_us":621710}
I20260709 12:27:13.047381 11759 raft_consensus.cc:993] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:27:12.819958  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:13.048590 11759 raft_consensus.cc:1081] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Signalling peer f613ba929b1046d3b5aed909414a95bf to start an election
W20260709 12:27:13.048949  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 229385us
I20260709 12:27:12.829734  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.828541  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:12.830875  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.011s	sys 0.002s
I20260709 12:27:12.831722  5560 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:12.826526  5561 raft_consensus.cc:2376] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15. Re-sending same reply.
I20260709 12:27:12.839162  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:12.832260  5553 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 11.
W20260709 12:27:12.852406  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:12.847388  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:12.848026  5555 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:12.865820  5552 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:27:12.868479  5554 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:12.874292  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:12.875068 11909 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:12.877763  5631 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 17: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:12.905890 11720 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:12.903352  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:12.902086  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:12.900684  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:12.904614  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:12.906642  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:12.909044 11880 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:27:12.910142  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48254.9us
I20260709 12:27:12.910471 11731 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:12.910925 11881 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:13.060949 11881 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:13.061504 11881 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:13.062793  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:13.063706  5555 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:12.913494  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:12.902032  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 649024us
I20260709 12:27:13.060988 11909 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:13.065276  5559 raft_consensus.cc:2376] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17. Re-sending same reply.
I20260709 12:27:13.066455  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:13.066627  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:13.067435  5559 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:13.067677 11909 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:12.923007  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:12.928576  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:13.068820  5630 raft_consensus.cc:606] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:27:13.069227  5630 raft_consensus.cc:614] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:27:12.954989 11420 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:12.956597  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 663251us
I20260709 12:27:12.959033 11663 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:12.959784  5632 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:12.932077 11549 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:12.964318 11914 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:12.968297 11587 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:12.510878 11790 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:13.080060 11601 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:13.081238 11790 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:13.081559  5557 raft_consensus.cc:2368] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17.
I20260709 12:27:12.986083  5634 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:12.985641 11721 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:12.256152 11861 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:13.084667 11817 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:13.085150 11817 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:13.085554 11817 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:13.088770  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:27:13.090094 11910 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:27:13.027226  6274 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:13.043876  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2801) took 1093 ms. Trace:
I20260709 12:27:13.093174  5710 rpcz_store.cc:276] 0709 12:27:11.950755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:11.950909 (+   154us) service_pool.cc:224] Handling call
0709 12:27:11.951231 (+   322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:11.959281 (+  8050us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:11.959326 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:11.960040 (+   714us) log.cc:844] Serialized 26 byte log entry
0709 12:27:12.165891 (+205851us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:12.165939 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:12.166148 (+   209us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:13.043359 (+877211us) raft_consensus.cc:1685] finished
0709 12:27:13.043404 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:13.043668 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8537,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":129,"thread_start_us":205474,"threads_started":1,"wal-append.queue_time_us":281281}
I20260709 12:27:13.009456 11799 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:13.095399 11773 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:13.023963 11841 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:13.057942  5556 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:13.096536  5710 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8c6474db7a70405694509f6976d850ce"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:38468
I20260709 12:27:13.097200  5710 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:13.097389  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:13.069885  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:13.078262 11914 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:13.082793  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 169904us
I20260709 12:27:13.088526 11782 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:13.094724 11549 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:13.097576  5710 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:13.097651  5705 raft_consensus.cc:2410] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:27:13.058197  5703 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:13.031006 11879 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:13.100658  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:13.102309 11804 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:13.118780 11804 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:13.102531  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:13.122676 11850 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:13.122816 11686 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:27:13.123276 11850 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:13.123891 11850 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:13.125797 11804 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:13.126554 11804 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:13.127367 11897 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:13.127195 11804 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:13.127941 11897 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:13.128504 11731 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:13.129279 11731 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:13.129801 11848 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:27:13.108841 11721 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:13.131969 11721 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:13.134433 11724 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:13.135156 11724 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:13.109925 11720 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:13.111163 11832 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:13.115304  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:13.123733  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:13.102872  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:13.141067  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 72678us
I20260709 12:27:13.176069  5703 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:13.182301  5561 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:13.182158 11209 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.005s
W20260709 12:27:13.181847  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 53867.3us
I20260709 12:27:13.189221  5558 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:13.190889  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:13.191706  5559 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:13.193850  6277 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:27:13.195230  5701 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:27:13.138828 11883 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:27:13.175830 11910 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.081s	user 0.003s	sys 0.000s
I20260709 12:27:13.196780  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2669) took 1076 ms. Trace:
I20260709 12:27:13.201588  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.136582 11782 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:13.202088  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.182787 11731 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:13.203342  5559 rpcz_store.cc:276] 0709 12:27:12.120741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.068228 (+947487us) service_pool.cc:224] Handling call
0709 12:27:13.196720 (+128492us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:13.207072  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2668) took 1081 ms. Trace:
I20260709 12:27:13.211084  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:13.211279  5635 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:13.207561  5558 rpcz_store.cc:276] 0709 12:27:12.120453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.188558 (+1068105us) service_pool.cc:224] Handling call
0709 12:27:13.188993 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:13.201834 (+ 12841us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.202013 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:13.195151 11759 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:13.214923  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:13.225124  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2487) took 1120 ms. Trace:
I20260709 12:27:13.223814  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:13.227424  5554 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:13.227998  6280 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:13.228266  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2674) took 1106 ms. Trace:
I20260709 12:27:13.229230  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.230034  5706 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:13.231995  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:13.205780 11663 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:13.232903  6276 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:13.233494 11663 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:13.208871 11804 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:13.236469  6278 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:13.232923 11721 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:13.239045 11663 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:13.240495 11773 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:13.240953  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61666.2us
I20260709 12:27:13.242107  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:13.243449  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 119942us
W20260709 12:27:13.244867  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:13.245293  5710 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:13.246013  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 67348.3us
I20260709 12:27:13.246910  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:13.247505  5710 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:13.248682  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:13.252753  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 73698.9us
I20260709 12:27:13.244864  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:13.256014 11721 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:13.256520 11721 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:13.256770 11897 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:13.256944 11721 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:13.257443 11897 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:13.206164 11904 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.126s
I20260709 12:27:13.228837  5554 rpcz_store.cc:276] 0709 12:27:12.122357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.200061 (+1077704us) service_pool.cc:224] Handling call
0709 12:27:13.223904 (+ 23843us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:13.227900 (+  3996us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.228171 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:13.225605  5629 rpcz_store.cc:276] 0709 12:27:12.055096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:12.055257 (+   161us) service_pool.cc:224] Handling call
0709 12:27:12.080807 (+ 25550us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:12.393697 (+312890us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:12.393745 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:12.414442 (+ 20697us) log.cc:844] Serialized 26 byte log entry
0709 12:27:12.877822 (+463380us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:13.017001 (+139179us) log.cc:844] Serialized 14 byte log entry
0709 12:27:13.116142 (+ 99141us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.154634 (+ 38492us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:13.154709 (+    75us) raft_consensus.cc:1685] finished
0709 12:27:13.154752 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:13.175861 (+ 21109us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11348,"prepare.queue_time_us":103257,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":240734,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":272,"thread_start_us":903578,"threads_started":3,"wal-append.queue_time_us":529241}
I20260709 12:27:13.274184  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:13.279790  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.283509  5557 pending_rounds.cc:85] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 8
I20260709 12:27:13.284021  5557 pending_rounds.cc:107] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:27:13.284325  5557 raft_consensus.cc:2889] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:13.287107 11724 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:13.288508 11696 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:13.288846  5559 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:27:13.289014  5709 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:13.289230  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:13.265057 11773 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:13.297201  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.298820  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2499) took 1131 ms. Trace:
I20260709 12:27:13.299238  5559 rpcz_store.cc:276] 0709 12:27:12.161619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.287968 (+1126349us) service_pool.cc:224] Handling call
0709 12:27:13.288459 (+   491us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:13.292775 (+  4316us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.293030 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:13.303444  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2671) took 1176 ms. Trace:
I20260709 12:27:13.304039  5552 rpcz_store.cc:276] 0709 12:27:12.121284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.195638 (+1074354us) service_pool.cc:224] Handling call
0709 12:27:13.224421 (+ 28783us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:13.297106 (+ 72685us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:13.297155 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:13.297222 (+    67us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:13.297274 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.297363 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:13.297522 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:27:13.309432  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 99280.3us
I20260709 12:27:13.320780  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:13.324625  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:13.324764  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 76424.8us
I20260709 12:27:13.325313  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.325466 11597 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:13.326287  5710 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 16: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:13.327306  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:13.331549 11855 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:13.292466 11881 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:13.333401 11724 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:13.334722 11881 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:13.340473 11861 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:13.341904  5628 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:27:13.343230 11861 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:13.343420  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:13.344197  5701 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:13.345240  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:13.346043  5630 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:13.347002  5555 raft_consensus.cc:2368] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17.
I20260709 12:27:13.320984 11654 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:27:13.347368  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.292057 11601 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:13.327104 11663 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.022s
I20260709 12:27:13.348711  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.349177  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.350761  5704 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:27:13.350850  5702 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:27:13.361893 11856 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:13.275780 11909 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 12:27:13.362630  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1124) took 1124 ms. Trace:
I20260709 12:27:13.362479 11856 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:13.352317  5708 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:27:13.363091  5554 rpcz_store.cc:276] 0709 12:27:12.238493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.361684 (+1123191us) service_pool.cc:224] Handling call
0709 12:27:13.362017 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:13.362218 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:13.362241 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:13.362284 (+    43us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:13.362358 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.362420 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:13.362549 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:13.353010  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2502) took 1224 ms. Trace:
W20260709 12:27:13.352931  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 97602.8us
I20260709 12:27:13.346053  5706 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:13.345265  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.356653 11198 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:13.357539  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:13.357129  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:13.358515  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:13.359225  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:13.359912 11916 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.138s	user 0.003s	sys 0.000s
I20260709 12:27:13.345281  5705 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:27:13.327270  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.360448  6277 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 12:27:13.327739  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 85827.7us
I20260709 12:27:13.324260  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:13.364374  5634 rpcz_store.cc:276] 0709 12:27:12.128581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:12.191390 (+ 62809us) service_pool.cc:224] Handling call
0709 12:27:13.352732 (+1161342us) spinlock_profiling.cc:243] Waited 723 ms on lock 0x7b540032d7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:13.352934 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1662023424}
I20260709 12:27:13.394953  5635 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:13.396690  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:13.394078  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 73412.8us
I20260709 12:27:13.391530  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1125) took 1152 ms. Trace:
I20260709 12:27:13.399993  5554 rpcz_store.cc:276] 0709 12:27:12.238768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.364187 (+1125419us) service_pool.cc:224] Handling call
0709 12:27:13.365494 (+  1307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:13.390957 (+ 25463us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:13.391001 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:13.391062 (+    61us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:13.391104 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.391207 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:13.391417 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:13.404597  6280 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
W20260709 12:27:13.406437  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 82286.1us
I20260709 12:27:13.424685  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:13.425578  5635 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:13.421723  6278 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:13.404135 11597 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:13.424167  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:13.399663 11904 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:13.428323  5552 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:27:13.429171  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:13.431702  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:13.432004  5630 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:13.435773  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2503) took 1252 ms. Trace:
I20260709 12:27:13.438405  5710 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:13.448688  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:13.450356  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:13.421540 11909 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:13.443296  5552 rpcz_store.cc:276] 0709 12:27:12.183082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.370683 (+1187601us) service_pool.cc:224] Handling call
0709 12:27:13.421042 (+ 50359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:13.435480 (+ 14438us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.435713 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:13.448961  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1187) took 1187 ms. Trace:
I20260709 12:27:13.454890  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.394230 11587 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:13.453928  6280 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:27:13.466007  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2666) took 1340 ms. Trace:
I20260709 12:27:13.468866  5556 rpcz_store.cc:276] 0709 12:27:12.119880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.102942 (+983062us) service_pool.cc:224] Handling call
0709 12:27:13.103321 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:13.103586 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:27:13.206484 (+102898us) log.cc:844] Serialized 14 byte log entry
0709 12:27:13.459898 (+253414us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:13.459955 (+    57us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:27:13.460007 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.460080 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:13.460326 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8614,"prepare.queue_time_us":178545,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":17123,"thread_start_us":284566,"threads_started":2}
I20260709 12:27:13.471573  5556 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:27:13.468983  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:13.468990  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.474336  5701 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:13.476131 11855 raft_consensus.cc:2764] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:27:13.462062  5627 rpcz_store.cc:276] 0709 12:27:12.261720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:12.851142 (+589422us) service_pool.cc:224] Handling call
0709 12:27:12.859869 (+  8727us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:12.873210 (+ 13341us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:12.873266 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:12.874233 (+   967us) log.cc:844] Serialized 26 byte log entry
0709 12:27:12.991396 (+117163us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:12.991444 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:12.991509 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:13.359111 (+367602us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:13.441585 (+ 82474us) raft_consensus.cc:1685] finished
0709 12:27:13.441630 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:13.448891 (+  7261us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4030,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":163,"spinlock_wait_cycles":63062272,"thread_start_us":116635,"threads_started":1,"wal-append.queue_time_us":270725}
I20260709 12:27:13.481429  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:13.484297  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47777us
I20260709 12:27:13.411073 11198 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:13.489620  5705 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:27:13.448647 11654 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:13.467751  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2667) took 1332 ms. Trace:
I20260709 12:27:13.488804  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:13.456382 11909 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:13.461463 11904 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:13.490034  5635 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:13.497622  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1126) took 1237 ms. Trace:
I20260709 12:27:13.453586 11897 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:27:13.442040 11774 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20260709 12:27:13.503461 11856 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:13.504750  5554 rpcz_store.cc:276] 0709 12:27:12.260508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.401252 (+1140744us) service_pool.cc:224] Handling call
0709 12:27:13.459952 (+ 58700us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:13.492118 (+ 32166us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:13.492165 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:13.497251 (+  5086us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:13.497315 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.497403 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:13.497547 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:13.507084  5634 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:27:13.509852  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:13.511117  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:13.511698 11850 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:13.512236 11587 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:13.518441  5627 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 12:27:13.519227  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 72410.4us
I20260709 12:27:13.519661 11848 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:13.528849 11654 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:13.500950  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:13.500021 11899 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:13.549871  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:13.550051  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2494) took 1398 ms. Trace:
I20260709 12:27:13.525504 11715 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:13.550915  5559 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:27:13.552022  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:13.554803  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.555011  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.555912  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.555922  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.551673  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1127) took 1200 ms. Trace:
I20260709 12:27:13.557322  5557 rpcz_store.cc:276] 0709 12:27:12.151486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.275314 (+1123828us) service_pool.cc:224] Handling call
0709 12:27:13.283192 (+  7878us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:13.285154 (+  1962us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:13.285205 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:13.285836 (+   631us) log.cc:844] Serialized 26 byte log entry
0709 12:27:13.378295 (+ 92459us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:27:13.378355 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.378469 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:13.523050 (+144581us) raft_consensus.cc:1685] finished
0709 12:27:13.523073 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:13.549951 (+ 26878us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":91979,"threads_started":1,"wal-append.queue_time_us":151382}
I20260709 12:27:13.502836 11880 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:13.503541 11909 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:13.563948  6277 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:13.562575  5552 rpcz_store.cc:276] 0709 12:27:12.301419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.462611 (+1161192us) service_pool.cc:224] Handling call
0709 12:27:13.493900 (+ 31289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:13.501350 (+  7450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:13.501387 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:13.501429 (+    42us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:13.501460 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.501519 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:13.501636 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:13.501482  5561 rpcz_store.cc:276] 0709 12:27:12.120151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.195853 (+1075702us) service_pool.cc:224] Handling call
0709 12:27:13.251440 (+ 55587us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:13.251772 (+   332us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:27:13.308922 (+ 57150us) log.cc:844] Serialized 14 byte log entry
0709 12:27:13.452629 (+143707us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:13.452752 (+   123us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:13.452805 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.452881 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:13.453004 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6416,"prepare.queue_time_us":61341,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":23487,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":11392,"thread_start_us":127762,"threads_started":2}
I20260709 12:27:13.578357  6279 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:13.565830  6274 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1099) took 1692 ms (client timeout 1548 ms). Trace:
I20260709 12:27:13.577106  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:13.586510  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1123) took 1342 ms. Trace:
I20260709 12:27:13.586951  5559 rpcz_store.cc:276] 0709 12:27:12.238191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.359775 (+1121584us) service_pool.cc:224] Handling call
0709 12:27:13.360221 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:13.560552 (+200331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.580231 (+ 19679us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b540023ffd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:13.580434 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":396436864}
I20260709 12:27:13.583376  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:13.581913  6274 rpcz_store.cc:269] 0709 12:27:11.865342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:11.866826 (+  1484us) service_pool.cc:224] Handling call
0709 12:27:13.528601 (+1661775us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54002909d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:13.558093 (+ 29492us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2665579264}
I20260709 12:27:13.596690  6272 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:13.586391  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:13.571332 11880 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:13.597035  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:13.605794  6280 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:13.575380 11848 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:13.614820  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:13.616902  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 192736us
I20260709 12:27:13.624797  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:13.559933 11817 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:13.624707 11861 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:13.634670 11861 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:13.637110  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:13.610805 11880 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:13.574532 11899 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:13.636518  5634 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:27:13.655665 11601 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:27:13.589926 11804 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:27:13.636299  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:13.587841 11773 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:27:13.635106 11861 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:13.663410 11799 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:13.656169 11601 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:13.583556 11715 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:13.664006  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:13.662111 11848 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:13.666982  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45301.7us
I20260709 12:27:13.635618 11549 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:27:13.666087  5455 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 176) took 62326 ms (client timeout 15000 ms). Trace:
W20260709 12:27:13.684638  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 113614us
I20260709 12:27:13.580521 11597 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:13.694875  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 61438.8us
I20260709 12:27:13.670733 11799 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:13.670945 11774 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:13.710472 11774 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:13.710810 11938 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:27:13.710942 11774 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:13.712182  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:13.715420  6275 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:13.693776  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:13.653913 11721 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:13.716764  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59924.3us
I20260709 12:27:13.717048  5703 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:13.718221  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:13.718287 11721 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:13.719127 11721 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:13.719688  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:13.721060 11899 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:13.721773  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:13.722352 11198 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:13.722772  5703 raft_consensus.cc:2410] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
W20260709 12:27:13.725435  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 169711us
I20260709 12:27:13.726498  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:13.727753 11936 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:27:13.728454 11799 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:13.737039 11881 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:13.742039 11721 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:13.685603 11918 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.057s	user 0.000s	sys 0.002s
I20260709 12:27:13.678014 11601 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:27:13.563691  5451 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 12 to 17, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 17 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:13.796749  5451 catalog_manager.cc:5697] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 6 to 14. New cstate: current_term: 14 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:27:13.798772  5451 catalog_manager.cc:5697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 9 to 19. New cstate: current_term: 19 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:13.801797 11817 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:13.686244  5455 rpcz_store.cc:269] 0709 12:26:11.337664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.337995 (+   331us) service_pool.cc:224] Handling call
0709 12:27:13.663469 (+62325474us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:27:08.289643 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:27:08.578120 (+288477us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:27:08.588899 (+ 10779us) write_op.cc:435] Acquired schema lock
0709 12:27:08.592466 (+  3567us) tablet.cc:662] Decoding operations
0709 12:27:08.602165 (+  9699us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:27:08.602267 (+   102us) write_op.cc:641] Partition lock acquired for write op
0709 12:27:08.602299 (+    32us) tablet.cc:685] Acquiring locks for 4 operations
0709 12:27:08.602816 (+   517us) tablet.cc:701] Row locks acquired
0709 12:27:08.602866 (+    50us) write_op.cc:260] PREPARE: finished
0709 12:27:08.603051 (+   185us) write_op.cc:270] Start()
0709 12:27:08.603225 (+   174us) write_op.cc:276] Timestamp: P: 1783600028602989 usec, L: 0
0709 12:27:08.603251 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 12:27:08.782407 (+179156us) log.cc:844] Serialized 1646 byte log entry
0709 12:27:10.221222 (+1438815us) op_driver.cc:464] REPLICATION: finished
0709 12:27:10.763325 (+542103us) write_op.cc:301] APPLY: starting
0709 12:27:10.806894 (+ 43569us) tablet.cc:1367] starting BulkCheckPresence
0709 12:27:11.400898 (+594004us) tablet.cc:1370] finished BulkCheckPresence
0709 12:27:11.401013 (+   115us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:27:11.403653 (+  2640us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:27:11.403866 (+   213us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=8,key_file_lookups=8,delta_file_lookups=8,mrs_lookups=0
0709 12:27:11.403913 (+    47us) write_op.cc:312] APPLY: finished
0709 12:27:11.404147 (+   234us) log.cc:844] Serialized 49 byte log entry
0709 12:27:12.008908 (+604761us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:27:12.012802 (+  3894us) write_op.cc:454] Released schema lock
0709 12:27:12.031133 (+ 18331us) write_op.cc:341] FINISH: Updating metrics
0709 12:27:12.079501 (+ 48368us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:27:12.085060 (+  5559us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3597537,"reported_tablets":31,"spinlock_wait_cycles":15104,"child_traces":[["op",{"apply.queue_time_us":488346,"apply.run_cpu_time_us":34618,"apply.run_wall_time_us":1332667,"cfile_cache_hit":18,"cfile_cache_hit_bytes":20847,"num_ops":4,"prepare.queue_time_us":531663,"prepare.run_cpu_time_us":60659,"prepare.run_wall_time_us":806970,"raft.queue_time_us":246787,"raft.run_cpu_time_us":80508,"raft.run_wall_time_us":313111,"replication_time_us":1617797,"thread_start_us":1366964,"threads_started":5,"wal-append.queue_time_us":1049527,"wal-append.run_cpu_time_us":31864,"wal-append.run_wall_time_us":3043364}]]}
I20260709 12:27:13.704746  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:13.705380 11715 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:13.709025 11850 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:13.702025  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:13.666868 11904 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:13.783664  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:13.795245  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:13.825723  5635 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:13.804807  6279 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
W20260709 12:27:13.811035  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 91482.9us
I20260709 12:27:13.832252  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:13.823848 11931 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.190s	user 0.000s	sys 0.004s
I20260709 12:27:13.828303  5627 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:13.840983  5629 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:13.827271 11914 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:13.849437 11817 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:27:13.850251 11817 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:13.851393 11817 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:13.837405  5675 connection.cc:256] RPC call timeout handler was delayed by -1.214596408000034s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1836s for a response
W20260709 12:27:13.853883  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0158507319997625s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
I20260709 12:27:13.837435 11549 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:13.865387  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:13.845372 11654 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:13.849517 11833 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:13.827482 11881 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:13.863873 11549 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:13.847270 11914 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:13.909106  5601 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:13.909772 11899 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:13.910034  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:13.910539 11723 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:13.911918  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:13.860499  5451 catalog_manager.cc:5697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 14 to 19. New cstate: current_term: 19 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:13.883033 11833 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:13.913594 11601 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:13.914227  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1122) took 1676 ms. Trace:
I20260709 12:27:13.923436  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:13.925773  5632 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:13.917724  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:13.928058  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 119941us
I20260709 12:27:13.904454 11549 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:13.931303  6274 raft_consensus.cc:2410] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:27:13.915174 11724 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:27:13.909758 11881 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:13.940145  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 213851us
I20260709 12:27:13.920627 11914 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:13.929566  5558 rpcz_store.cc:276] 0709 12:27:12.237835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.354771 (+1116936us) service_pool.cc:224] Handling call
0709 12:27:13.355182 (+   411us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:13.355615 (+   433us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:13.355645 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:13.356173 (+   528us) log.cc:844] Serialized 26 byte log entry
0709 12:27:13.625617 (+269444us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:13.625876 (+   259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:13.626054 (+   178us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:13.913917 (+287863us) raft_consensus.cc:1685] finished
0709 12:27:13.913943 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:13.914149 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3288,"raft.queue_time_us":15118,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":2098,"thread_start_us":258314,"threads_started":1,"wal-append.queue_time_us":471316}
I20260709 12:27:13.943867  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:13.944753  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:13.931600 11833 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:13.952886  6279 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:27:13.936136 11597 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:13.949050  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:13.955053  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:13.955981  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:13.905781 11209 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:27:13.958146  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:13.963696  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:13.964731  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:13.965449  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:13.966396  5630 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:13.968101  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:13.974370  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:13.976511 11723 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:14.003813  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 86945us
I20260709 12:27:13.984917  5451 catalog_manager.cc:5697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 3 to 16, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 16 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:13.985859 11601 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:13.977954  5707 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:27:13.942358 11720 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:14.005826  5629 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:14.010777  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:14.005311  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 60655.2us
I20260709 12:27:14.007285  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:13.996615  5636 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:13.892997 11721 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:14.012290  5632 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:27:14.014750  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:14.020885  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:13.993234 11597 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:13.937498 11899 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.027786  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:14.047780 11850 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.008s
I20260709 12:27:14.047822  5676 leader_election.cc:400] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17.
I20260709 12:27:14.051029  5676 leader_election.cc:403] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 12:27:14.022068 11952 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:27:14.053467  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49680.6us
I20260709 12:27:14.036723 11597 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:14.060469  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:14.074440  5628 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: [21.13-21.13]
W20260709 12:27:14.073642  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 125322us
I20260709 12:27:14.077771  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:14.078678  6276 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:14.052800 11731 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:14.057189 11663 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:14.056849 11198 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:14.055759 11855 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:27:14.090519  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 161757us
I20260709 12:27:14.089982 11832 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:14.084039 11952 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:14.071341 11720 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.084379 11773 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.029s
I20260709 12:27:14.083060 11861 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.124476 11855 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:14.128086  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:14.081885 11909 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:14.150115  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:14.154753 11909 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:14.145006 11880 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:14.156751  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:14.157434  5705 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
W20260709 12:27:14.139348  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 99022.7us
I20260709 12:27:14.097956 11663 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:14.114679  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 100574us
W20260709 12:27:14.158236  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 60938.9us
I20260709 12:27:14.158717  5703 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:14.080489 11774 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.162719  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:14.142707  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 65041.5us
I20260709 12:27:14.381893 11899 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:14.107455  5451 catalog_manager.cc:5697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 4 to 15, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 15 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:14.260179 11198 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:14.239023  5675 leader_election.cc:336] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.812s (SENT)
W20260709 12:27:14.215204  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47782.1us
I20260709 12:27:14.188637 11549 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:14.243247  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 147516us
I20260709 12:27:14.220345 11804 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:27:14.227787  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 187191us
W20260709 12:27:14.220872  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:14.188122  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 233289us
I20260709 12:27:14.296682 11960 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:14.290282  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:14.255921 11949 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.199s	user 0.001s	sys 0.001s
I20260709 12:27:14.340863 11587 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:14.458074 11940 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.802s	user 0.002s	sys 0.002s
I20260709 12:27:14.334358 11855 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:14.460531 11953 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.348s	user 0.003s	sys 0.000s
W20260709 12:27:14.462044 11959 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
W20260709 12:27:14.447067  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 266913us
I20260709 12:27:14.340590 11964 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.311s
I20260709 12:27:14.168800  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:14.158860 11636 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:14.501936  6276 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:14.189342  5635 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 12:27:14.190002  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:14.194954  5705 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:14.262908  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:14.224576  5708 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
W20260709 12:27:14.512373  6271 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.508s	user 0.011s	sys 0.001s
I20260709 12:27:14.241820 11880 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:14.551219  6271 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
W20260709 12:27:14.253616 11950 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.177s	user 0.002s	sys 0.000s
I20260709 12:27:14.509706  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:14.534763  5703 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:14.570753  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:14.570628  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:14.168605 11759 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:14.323503 11961 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.331s
I20260709 12:27:14.580665 11759 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:14.579375 11549 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:14.581444 11817 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:14.581960 11817 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:14.582381 11817 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:14.582123 11759 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.580044 11601 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:14.584561  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:14.318753 11723 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:14.551373  5627 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:27:14.257495 11951 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
W20260709 12:27:14.581120  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 412410us
I20260709 12:27:14.589481  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:14.590451  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:14.591250  6237 leader_election.cc:400] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17.
I20260709 12:27:14.591642  6237 leader_election.cc:403] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:27:14.525776 11587 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.395044 11947 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.212s
I20260709 12:27:14.593267  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:14.593954  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:14.553755  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 314763us
I20260709 12:27:14.595067 11773 raft_consensus.cc:2764] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:27:14.595495  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:14.595885 11773 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:14.595834  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:14.597677  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:14.598351  5558 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:14.598845  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:14.599686 11759 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:14.522022 11848 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:14.600523 11209 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:14.600575 11848 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:14.601440  5561 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 12:27:14.579011 11861 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:14.601971 11848 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:14.603264 11848 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:14.558009  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:14.558398  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:14.559391  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:14.562793  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:14.563398 11960 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:14.565112  5451 catalog_manager.cc:5697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 11 to 14, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 14 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:27:14.575881 11855 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:14.578416 11964 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:27:14.579231  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 69597.6us
I20260709 12:27:14.579828 11720 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:14.522475 11829 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:27:14.573400  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44452.2us
I20260709 12:27:14.584730 11601 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:14.592281 11971 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:14.552675 11198 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.594251 11587 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:14.641932 11601 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:14.635280 11855 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:14.645753  5451 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 11 to 13, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 13 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:14.642190 11971 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:14.648833 11971 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:14.630779 11964 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:14.650323 11964 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.605422  5705 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:14.245831 11856 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:14.606909  5710 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:14.548151  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:14.600055  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:14.297079 11897 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:14.611002  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:14.614248  5557 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: []
W20260709 12:27:14.609438  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 316393us
I20260709 12:27:14.297394 11782 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:14.628961  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:14.619697  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:14.322820 11663 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.216092 11799 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.642748  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:14.319082 11833 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.557478  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:14.677361 11960 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:14.615864 11881 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:14.604966 11969 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17.
I20260709 12:27:14.679152  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:14.669659  5558 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:14.401077 11952 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:14.730917  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 210447us
I20260709 12:27:14.698972  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:14.779706  5701 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:14.731383  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:14.743957  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:14.681465  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:14.723932  5705 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:27:14.783264  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:14.735066  5561 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:14.757264  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:14.765415  5553 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:14.698086 11909 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:14.730028 11209 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:14.630600 11829 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:14.784888  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 181506us
I20260709 12:27:14.647773  5451 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 14 to 16, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 16 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:14.678946 11774 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:14.785596  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:14.792486  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 91618.3us
I20260709 12:27:14.814742  5561 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:14.690025 11549 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:27:14.795548  5702 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: []
W20260709 12:27:14.796846  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 190738us
I20260709 12:27:14.757041 11855 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:14.778030 11964 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:14.679781 11947 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:14.815333  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:27:14.817576  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 74721.2us
W20260709 12:27:14.826848  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 215934us
I20260709 12:27:14.696913 11897 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:14.791119 11723 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:27:14.853897  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 234332us
I20260709 12:27:14.811162 11881 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:14.757751 11782 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:14.860009  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 80968.7us
W20260709 12:27:14.866520  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 226484us
I20260709 12:27:14.822763 11549 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:14.857808  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:14.878140  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:14.879256  5710 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:14.832157 11947 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:27:14.883221  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:14.883498  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:14.865118 11597 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.786s	user 0.013s	sys 0.003s
I20260709 12:27:14.855827 11961 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:14.811069 11909 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.864766 11723 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:14.900269  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:14.908892  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:14.882304 11731 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:14.881176 11720 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:14.910815  5627 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:27:14.887333 11960 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:14.899693 11962 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.372s	user 0.002s	sys 0.000s
W20260709 12:27:14.896765 11970 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.420s	user 0.002s	sys 0.000s
I20260709 12:27:14.913986 11198 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:14.914556 11961 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:14.915330  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:14.918028  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:14.912199 11721 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:14.918265  5451 catalog_manager.cc:5697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:14.923894  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:14.924705  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 241959us
I20260709 12:27:14.925552  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:14.910640 11731 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:14.892921 11597 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:14.932243  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 66035.1us
I20260709 12:27:14.923666  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:14.938552  5629 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 13.
I20260709 12:27:14.939451  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:14.955145  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 101718us
I20260709 12:27:14.943660  5705 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:14.955955  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:14.928056  5628 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:27:14.948294 11961 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.898105 11829 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.934396 11982 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
I20260709 12:27:14.927029 11720 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:14.928548 11960 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:15.049324  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:14.988484  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:15.089704  5705 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
W20260709 12:27:14.948510 11979 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:27:14.987737  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:14.937769 11198 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:15.093813  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:14.978256  5708 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
W20260709 12:27:15.103889  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 212001us
I20260709 12:27:15.109536  6275 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 20: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:14.951203 11731 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:15.000015 11209 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:14.983660 11975 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:27:15.017045 11993 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
W20260709 12:27:14.959327 11965 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.365s	user 0.001s	sys 0.001s
I20260709 12:27:15.120677  5701 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:27:15.010079 11799 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:15.109982 11198 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:15.114565 11880 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:15.125113  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:14.985108 11833 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:15.004725 11973 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.263s	user 0.000s	sys 0.001s
I20260709 12:27:14.983403 11977 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:14.949882 11721 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:15.110878 11715 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:14.985491 11909 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:15.177359  5705 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
W20260709 12:27:15.168641  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 170187us
I20260709 12:27:15.126718 11209 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:15.222007  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 96972.4us
I20260709 12:27:15.008898 11997 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.160s
I20260709 12:27:15.130723 11782 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:15.259963 11817 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.677s	user 0.011s	sys 0.003s
W20260709 12:27:15.261036  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 302209us
I20260709 12:27:15.276903 11817 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.948863 11947 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:15.260267 11832 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.120s	user 0.004s	sys 0.014s
I20260709 12:27:15.267251  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:27:15.233249 11914 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.224s	user 0.018s	sys 0.000s
I20260709 12:27:15.013692 11723 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:14.957023 11549 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:15.269408 11909 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:15.269582 11597 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:15.292769  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:15.294096  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:15.251801 11799 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:15.252068 11880 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:15.268420 11209 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:15.287807 11984 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.306s	user 0.002s	sys 0.000s
I20260709 12:27:15.332049  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:15.390331 11960 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:15.344025  5558 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:27:15.346189 11988 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.347s	user 0.000s	sys 0.002s
W20260709 12:27:15.347174 11996 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:27:15.347055  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:15.329493 11601 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.683s	user 0.009s	sys 0.005s
W20260709 12:27:15.348706 12000 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 12:27:15.349674  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1094) took 1331 ms. Trace:
I20260709 12:27:15.350005 11721 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:15.327764 11832 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17.
I20260709 12:27:15.403409  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1098) took 1466 ms. Trace:
I20260709 12:27:15.350266  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:15.350494 11977 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:15.404091  5708 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:27:15.354744  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 86957.5us
I20260709 12:27:15.355314 11909 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:15.348737 11999 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:15.356727 11856 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:15.359370  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2624) took 1204 ms. Trace:
W20260709 12:27:15.363323 11985 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.381s	user 0.001s	sys 0.000s
I20260709 12:27:15.368490  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:15.371160 11654 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:15.020478  5451 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 14 to 17. New cstate: current_term: 17 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:27:15.380400  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 492172us
I20260709 12:27:15.267336 11833 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
W20260709 12:27:15.383508  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 421939us
I20260709 12:27:15.383600 11994 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:27:15.384503 11773 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:27:15.385355  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:15.385716 11947 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:15.387399 11817 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:15.445619  5701 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:27:15.446704  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:15.389374 11720 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:15.396734  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:15.352890 11549 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:27:15.378783 11799 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:15.405462  5675 leader_election.cc:336] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.729s (SENT)
I20260709 12:27:15.398828 11982 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:27:15.400367  6273 rpcz_store.cc:276] 0709 12:27:14.011929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:14.012062 (+   133us) service_pool.cc:224] Handling call
0709 12:27:14.012523 (+   461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:14.035720 (+ 23197us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:14.035765 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:14.054872 (+ 19107us) log.cc:844] Serialized 26 byte log entry
0709 12:27:14.237545 (+182673us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:14.237592 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:14.237670 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:15.342881 (+1105211us) raft_consensus.cc:1685] finished
0709 12:27:15.342903 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:15.343049 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3868,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":12722,"thread_start_us":153395,"threads_started":1,"wal-append.queue_time_us":317699}
I20260709 12:27:15.403846  6280 rpcz_store.cc:276] 0709 12:27:13.936712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:13.936874 (+   162us) service_pool.cc:224] Handling call
0709 12:27:13.937373 (+   499us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:14.090607 (+153234us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:14.090638 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:14.107451 (+ 16813us) log.cc:844] Serialized 26 byte log entry
0709 12:27:14.317203 (+209752us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:14.317735 (+   532us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:14.317818 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:15.403149 (+1085331us) raft_consensus.cc:1685] finished
0709 12:27:15.403171 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:15.403342 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19452,"raft.queue_time_us":20354,"raft.run_cpu_time_us":53,"raft.run_wall_time_us":53,"thread_start_us":204123,"threads_started":2,"wal-append.queue_time_us":456032}
I20260709 12:27:15.425141  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:15.432019  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:15.440075  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:15.442651  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:15.452653  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:15.329675 11914 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:15.426808 11856 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:15.466624  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:15.434095 11774 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:15.403203 11601 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:15.460587  5555 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.006s	sys 0.002s
W20260709 12:27:15.479065  5555 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.007s	sys 0.002s
I20260709 12:27:15.423835 11977 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:15.424707  5706 rpcz_store.cc:276] 0709 12:27:14.154602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:14.158340 (+  3738us) service_pool.cc:224] Handling call
0709 12:27:14.241825 (+ 83485us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:14.359644 (+117819us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:27:14.804501 (+444857us) log.cc:844] Serialized 14 byte log entry
0709 12:27:15.358787 (+554286us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b18003c1960. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a
0709 12:27:15.359023 (+   236us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:15.359073 (+    50us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:27:15.359130 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:15.359195 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:15.359313 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22513,"prepare.queue_time_us":477041,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":3976,"raft.queue_time_us":119,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"spinlock_wait_cycles":3013504,"thread_start_us":340483,"threads_started":2,"wal-append.queue_time_us":302668}
I20260709 12:27:15.467907  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:15.454891 11799 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:15.475919  5554 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.681s	user 0.010s	sys 0.005s
W20260709 12:27:15.451745 11964 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.601s	user 0.009s	sys 0.004s
I20260709 12:27:15.483604  5708 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:15.484966  5554 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:15.486294  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:15.489056  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 50754.7us
I20260709 12:27:15.491184 11914 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:15.491680  5703 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:15.492071 11880 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:15.452236 11855 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 12:27:15.498189  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:15.496877 11799 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:15.501140  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:15.501526 11833 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:27:15.505647 11909 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:15.457743 11654 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:15.509378 11969 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:27:15.426645 11999 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:15.479527 11774 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:15.453999 11881 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
W20260709 12:27:15.513479  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 108049us
I20260709 12:27:15.480046 12009 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:15.516700 11601 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:15.518989 11997 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:15.551870  5558 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
I20260709 12:27:15.477326 11960 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:15.497046  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:15.561488  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:15.636830  5559 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:15.637212  5629 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:15.569828  5557 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:15.550958 11587 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:27:15.579093  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 81023.7us
W20260709 12:27:15.599152 11663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.005s	sys 0.009s
W20260709 12:27:15.595947  5553 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.783s	user 0.008s	sys 0.005s
I20260709 12:27:15.642334  5553 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
W20260709 12:27:15.598006  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 172729us
W20260709 12:27:15.560789 11971 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.912s	user 0.014s	sys 0.003s
I20260709 12:27:15.642227  5558 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260709 12:27:15.586320  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 139550us
W20260709 12:27:15.632947  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 190407us
W20260709 12:27:15.602279 11759 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.002s	user 0.012s	sys 0.003s
I20260709 12:27:15.638814  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:15.589797  5554 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:27:15.641501  6277 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:15.600034 12010 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:15.613445  5707 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:15.642846  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2700) took 1033 ms. Trace:
I20260709 12:27:15.653313  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:15.642225 11663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.006s	sys 0.009s
I20260709 12:27:15.663007  5553 rpcz_store.cc:276] 0709 12:27:14.609411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:14.609529 (+   118us) service_pool.cc:224] Handling call
0709 12:27:15.642791 (+1033262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:15.638815 11997 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:15.666023  5556 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:27:15.665279  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:15.668216  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 95669.4us
I20260709 12:27:15.648710 11952 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:15.690281 11654 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:15.678544  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:15.682792  5557 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:15.665688 12010 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:15.703127 11587 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:15.703712 11587 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:15.705343 12014 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260709 12:27:15.706552 11848 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.103s	user 0.011s	sys 0.011s
I20260709 12:27:15.697151 11909 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:15.706959 11597 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:15.707670 11597 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:15.708542 11804 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:27:15.707993 11961 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.002s
I20260709 12:27:15.662766 11833 raft_consensus.cc:2764] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:27:15.646068 11971 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:15.676783 12020 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:27:15.716917  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63003.6us
I20260709 12:27:15.645889 12018 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.059s
I20260709 12:27:15.568610  5451 catalog_manager.cc:5697] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 9 to 13, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 13 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:15.700518 12010 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:27:15.718398  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53182.2us
I20260709 12:27:15.661958 11969 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
W20260709 12:27:15.721195  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 69713us
I20260709 12:27:15.734750  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:15.738796  6271 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:15.723361  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:15.720412 11909 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:15.747571  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:15.750582  5558 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:15.751577  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:15.745628  5632 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:15.743021  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:15.732903 12018 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:15.744940  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:15.722308  5629 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:15.760833  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:15.739966 11198 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.507s	user 0.004s	sys 0.004s
I20260709 12:27:15.732226 11654 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:15.769547 12022 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:27:15.770440 12022 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:15.770792 12022 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:15.723876  5451 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 11 to 15, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 15 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:15.756593 11549 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:15.771665 12022 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:15.737500 11848 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.047s
I20260709 12:27:15.773061 12022 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:15.773418 11897 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:15.765275 11855 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:15.775377 11855 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:15.775794 11855 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:15.777874  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:15.778717 11601 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:15.778820  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:15.779129 11601 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:15.771718 11774 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:15.733709 11999 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:15.780746 11969 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:15.781306 11969 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:15.781981  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1064) took 1225 ms. Trace:
I20260709 12:27:15.782155 11969 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:15.782389  5703 rpcz_store.cc:276] 0709 12:27:14.556567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:14.573547 (+ 16980us) service_pool.cc:224] Handling call
0709 12:27:14.574100 (+   553us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:15.558748 (+984648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:15.781742 (+222994us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b54003ad4d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:15.781918 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2109819264}
I20260709 12:27:15.767520 12019 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 12:27:15.783530 11198 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:15.784694  6279 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:15.786832  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:15.770968 11799 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:15.732302 11960 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 12:27:15.779572 11601 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:27:15.790511 11990 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.806s	user 0.003s	sys 0.000s
I20260709 12:27:15.790987 11914 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:15.757558 12018 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:15.768046 11654 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:15.792733 11654 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:15.780329 11997 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:15.766355  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:15.708226 11209 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:15.818652  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 73795.5us
I20260709 12:27:15.820734  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:15.824586  5554 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:15.824828  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:15.742261 11773 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 12:27:15.821548  6273 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:15.833354  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:15.861461  5632 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:15.756791 11759 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:15.786957 11977 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:15.841012 11909 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:15.842374  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:15.842906 11721 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:27:15.843400 12024 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
I20260709 12:27:15.823542 11856 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:15.844487 11731 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.649s	user 0.009s	sys 0.002s
I20260709 12:27:15.846356 11209 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:15.846674  5628 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:27:15.787055 11198 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:15.857371  5451 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 10 to 12, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 12 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:27:15.864629 11759 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:15.867041  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2619) took 1725 ms (client timeout 1728 ms). Trace:
W20260709 12:27:15.874336  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 108033us
I20260709 12:27:15.875159 12022 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:27:15.875912  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44061.3us
I20260709 12:27:15.878762 11773 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:27:15.879513 12007 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.532s	user 0.000s	sys 0.002s
I20260709 12:27:15.879787 12017 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:27:15.880910 11720 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:27:15.895140  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:15.894781  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:15.945240  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 101601us
W20260709 12:27:15.922287  5707 rpcz_store.cc:269] 0709 12:27:14.141906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:14.149772 (+  7866us) service_pool.cc:224] Handling call
0709 12:27:15.866620 (+1716848us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400323054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:15.866901 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2963044736}
I20260709 12:27:15.891638 11977 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:15.832330 12009 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:15.890954 11715 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:15.945880  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:15.949638  5559 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:15.960044  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2694) took 1343 ms. Trace:
I20260709 12:27:15.921689 11723 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:15.962097 11964 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:27:15.962699 11897 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:15.962895 11850 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:15.963579 11978 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:15.964304  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:15.964651  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:15.913578 12019 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:27:15.964742  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:15.964306  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:15.965160  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:15.928992 11774 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:15.933734 11773 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:15.920588 11731 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:15.965597  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.167763 11978 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.169317  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:16.016844 11975 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:16.170547 11983 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:16.171353 11983 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.173776 11983 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:16.144788  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47101.5us
I20260709 12:27:16.176693  5631 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:16.181058  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.017329  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:15.966775  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:16.148847  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:16.066023  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:16.199503  5628 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 16 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 12:27:16.192186  5554 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:16.202803  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 249896us
I20260709 12:27:15.960985  5555 rpcz_store.cc:276] 0709 12:27:14.607410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:14.607536 (+   126us) service_pool.cc:224] Handling call
0709 12:27:14.636173 (+ 28637us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:14.636486 (+   313us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:14.636524 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:14.648299 (+ 11775us) log.cc:844] Serialized 26 byte log entry
0709 12:27:14.898649 (+250350us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:15.345416 (+446767us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:15.480448 (+135032us) log.cc:844] Serialized 14 byte log entry
0709 12:27:15.929200 (+448752us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:15.929277 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:15.929320 (+    43us) raft_consensus.cc:1685] finished
0709 12:27:15.950668 (+ 21348us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:15.950841 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13909,"prepare.queue_time_us":462693,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":199924,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":21776,"spinlock_wait_cycles":40314240,"thread_start_us":1250942,"threads_started":3,"wal-append.queue_time_us":350497}
W20260709 12:27:16.203469  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54734.3us
I20260709 12:27:16.204941  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:16.205880  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:16.023711  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2699) took 1414 ms. Trace:
I20260709 12:27:16.203661  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:16.205629  5633 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:16.168143 11994 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:16.223218 11994 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:16.224449  5635 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:16.224243 11994 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.225546 11994 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:16.166405 11721 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.063s
I20260709 12:27:16.235514 11881 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:16.166039 11856 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.238135 11731 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:15.954731 11947 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:16.199059 11978 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.202193 11774 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.222154  5631 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:16.240243  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.209036 11850 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.241039  5631 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 17: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:16.241066 11774 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.242277  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.238135  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:15.953567 12017 consensus_queue.cc:1243] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 16.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:27:16.218410 11964 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:27:16.241928 11947 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.202103  5627 raft_consensus.cc:2368] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18.
I20260709 12:27:16.242347  5632 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:16.257380  5558 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:16.247747  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:16.101333 11880 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:15.944247 11720 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:16.243721  5631 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:16.174840 11975 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:15.954996 11980 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:16.258074  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:16.257438  5636 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:27:15.974126 11715 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:27:16.205484 11587 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.501s	user 0.016s	sys 0.000s
I20260709 12:27:16.271986  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.102205 11977 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:16.003605 11982 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:16.265355  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:16.033957 11759 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:16.267992  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 64428.2us
I20260709 12:27:16.217933  5560 rpcz_store.cc:276] 0709 12:27:14.608938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:14.609078 (+   140us) service_pool.cc:224] Handling call
0709 12:27:14.645553 (+ 36475us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:14.721981 (+ 76428us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:27:14.785550 (+ 63569us) log.cc:844] Serialized 14 byte log entry
0709 12:27:15.384125 (+598575us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:15.711004 (+326879us) spinlock_profiling.cc:243] Waited 3.38 ms on lock 0x7b2c000fb608. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:27:15.711492 (+   488us) log.cc:844] Serialized 26 byte log entry
0709 12:27:15.711624 (+   132us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:15.711656 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:15.711743 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:15.878802 (+167059us) spinlock_profiling.cc:243] Waited 9.52 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:16.023369 (+144567us) raft_consensus.cc:1685] finished
0709 12:27:16.023392 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:16.023529 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":329365,"prepare.queue_time_us":60944,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":332680,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":34131,"spinlock_wait_cycles":29667584,"thread_start_us":274295,"threads_started":1,"wal-append.queue_time_us":465136}
I20260709 12:27:16.296043  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.293771  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.301862  5630 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:16.302956  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.270567  5555 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 12:27:16.032616 12009 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:16.025748 12010 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.004575 12022 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.263s
I20260709 12:27:16.296972  5628 raft_consensus.cc:2368] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 17.
I20260709 12:27:16.032281 11601 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.305889  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.306493  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.316445  5631 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:16.242425 11850 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.271642 11880 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.268874 11774 raft_consensus.cc:2764] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:27:16.265556 11721 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:16.271983 12038 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:27:16.328255 11721 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.328910 12038 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:16.329299 12038 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:16.304520 12019 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:16.305900  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:16.292801 12008 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:16.294703 11964 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:16.313302  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:16.243098 11897 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.318122  5554 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:16.316222  5627 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:16.293551 11723 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.308442  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:16.330307 12038 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.313642 12005 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.035s
I20260709 12:27:16.334906 12036 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 12:27:16.335800  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.271804 12020 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 12:27:16.026677  5451 catalog_manager.cc:5697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 11 to 15, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 15 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:16.306458 11982 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.296413 11881 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 12:27:16.340011 11947 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.432948 11975 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.341877 12009 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.433845 11829 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:16.434008 11982 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:16.434653 12010 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:16.436949  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 131164us
I20260709 12:27:16.437644  5632 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 16 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 7.
I20260709 12:27:16.331664  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:16.300189 11587 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.304960 11980 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:16.448007  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 116429us
I20260709 12:27:16.448760  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:16.439097  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 125230us
W20260709 12:27:16.397528  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:16.314201 11952 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.085s
I20260709 12:27:16.369799  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.348477  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.459287  5635 raft_consensus.cc:2368] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18.
W20260709 12:27:16.454797  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 159032us
I20260709 12:27:16.368758  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:16.331606 11855 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.555s	user 0.011s	sys 0.000s
I20260709 12:27:16.463199  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:16.459177  5628 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
W20260709 12:27:16.341264 11654 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.548s	user 0.011s	sys 0.000s
I20260709 12:27:16.430059 11880 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.431471 11601 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.425274 11721 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:16.475217  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.481002  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.461959 11994 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:16.441571 12010 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:16.444248 12022 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:16.471113  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:16.452678 11982 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:16.455077 12008 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:16.483429  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.500819  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.466728  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:16.484683  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.502328  5553 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:16.501643  5631 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:16.468976 11855 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.465903 11774 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:16.497444  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.480603 12038 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:16.460105 12027 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 12:27:16.528839  5451 catalog_manager.cc:5697] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 3 to 14, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 14 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:16.531117  5451 catalog_manager.cc:5697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:16.489840 12036 consensus_queue.cc:1243] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 22.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:27:16.469774 12019 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:27:16.510136 11994 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:16.533864 12019 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:16.534037 11994 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.517781  5628 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:16.519351 11964 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.521939 11897 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.523731 11587 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.525300 11980 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:16.513108 12020 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:16.471915 11654 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.495939 12022 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:16.540128  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.540030  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:16.342388 11977 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.499577 12005 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:16.493992  5633 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:16.534623 12019 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.542825  5558 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:16.503801 12045 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:16.540854  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.572181  5632 raft_consensus.cc:2410] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:27:16.547528  5554 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:16.518491 11952 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:16.549954  5631 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 19: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:16.498354 12010 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.552822 11994 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:16.555286  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58435.5us
I20260709 12:27:16.582357  5634 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:16.559178  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.547991 12022 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:16.537390 11774 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.564894 11971 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.565222  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:16.541671 11975 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:16.544123 11832 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:16.575096  5553 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:16.601627  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:16.602269  5553 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:16.562202 12008 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.580906  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:16.557379  5451 catalog_manager.cc:5697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 13, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 13 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:27:16.505784 11982 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.583177  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:27:16.596294  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 133153us
I20260709 12:27:16.548280 12020 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.606173  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:16.606539  5708 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:27:16.611807  5704 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:27:16.611881  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:16.605458  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:16.610742  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:16.609475  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:16.616590  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:16.617750  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:16.619493 11994 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:16.619776 11909 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:16.614786 11855 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.620319 11909 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:16.620772 11909 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:16.621286 11855 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:16.621758 11855 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:16.622117 11855 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:16.624780 11723 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:16.629362  5709 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:27:16.630676  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:16.631771  5675 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:16.633404 11995 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.630s	user 0.004s	sys 0.002s
I20260709 12:27:16.633736 12019 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:16.603268 11832 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:16.634760  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:16.636439 11982 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.637516  5451 catalog_manager.cc:5697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 11 to 13, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 13 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:27:16.641052 12010 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:16.641782  5555 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:27:16.641673  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:16.709507  5632 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:16.644594  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:16.710345  5636 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:16.645774 11774 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:16.645630 11715 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.646582 11723 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:16.608050 12045 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:16.614756 12005 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:16.657672  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:16.658052  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1104) took 1276 ms. Trace:
I20260709 12:27:16.658418 12020 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.607723 11975 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.663515  5553 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:16.669224  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:16.670168  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:16.671236  5704 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:16.679141  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:16.732535  6273 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
W20260709 12:27:16.732779  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 101099us
I20260709 12:27:16.681107  6274 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:16.682255  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:16.733572  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:16.734244  6277 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 19: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:16.682713  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:16.685024  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:16.690099  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 59514.2us
I20260709 12:27:16.701745  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:16.713654  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:16.736542  5701 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:27:16.673440 11991 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:16.723973  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:16.726711  5706 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:16.738513  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:16.699059  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:16.726667  6279 rpcz_store.cc:276] 0709 12:27:15.381875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:15.382027 (+   152us) service_pool.cc:224] Handling call
0709 12:27:15.382338 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:15.785059 (+402721us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:16.657815 (+872756us) spinlock_profiling.cc:243] Waited 402 ms on lock 0x7b54005f8f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:16.657997 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":924936960}
I20260709 12:27:16.738765  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:16.739410  5704 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:16.678459 11987 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20260709 12:27:16.740332  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:16.740456  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:16.708678 12050 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:16.742331  5559 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:16.686667 12022 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.773480  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:16.773833 12022 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.775732  5707 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:16.735018  5702 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:16.741374 11914 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:16.750289 11991 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.783058  5636 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:16.785540  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:16.786216  5636 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:16.753013 11964 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:16.788432 11964 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:16.789491 11964 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.790024  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:16.752591 11832 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.791196  5702 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:27:16.791271 11832 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:16.791591 11983 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:16.792088 11983 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:16.793185 11983 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:16.753737  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54807.7us
I20260709 12:27:16.795845  5636 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:16.754429 11982 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:16.798039  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.798807 12019 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:27:16.799657 11597 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:16.799798 12019 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:16.755429 11987 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:16.800509 11987 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:16.801548 11987 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.802932 11987 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:16.803314  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.804152  5636 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 20: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:16.757102 11715 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.723227 12004 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:16.749929 11774 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.011s
I20260709 12:27:16.762084  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:16.806386 11549 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:16.806644  5629 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:16.807019 11549 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:16.808198  5629 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:27:16.809015  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1100) took 1855 ms. Trace:
I20260709 12:27:16.809432  6271 rpcz_store.cc:276] 0709 12:27:14.953711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:14.953923 (+   212us) service_pool.cc:224] Handling call
0709 12:27:14.959529 (+  5606us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:15.745642 (+786113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:16.808790 (+1063148us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b54002eed54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:16.808959 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1791702400}
I20260709 12:27:16.810122  5630 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:27:16.811436  5630 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:27:16.762825  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:16.765204  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:16.806351  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:16.752105  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:16.766672  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:16.798842  5633 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 17: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:16.767561 11855 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.768787  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2693) took 2161 ms. Trace:
I20260709 12:27:16.770370 11663 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:16.770730 12017 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:16.832455  5561 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:16.723541 12045 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:16.833410 12045 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:16.798075 11982 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:16.834120 11982 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:16.734513 11723 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.835621 11654 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.800278 12019 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:16.836766  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:16.837280 12019 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.752209  5451 catalog_manager.cc:5697] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:16.749464 12008 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:27:16.750546  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 80456.7us
I20260709 12:27:16.843557  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:16.762274 11975 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:16.810029  5629 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:16.821822  5706 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:16.823879  5632 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
W20260709 12:27:16.828421  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 76408.2us
I20260709 12:27:16.849210  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:16.829052  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 88774us
I20260709 12:27:16.827930  5552 rpcz_store.cc:276] 0709 12:27:14.607188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:14.607308 (+   120us) service_pool.cc:224] Handling call
0709 12:27:14.629634 (+ 22326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:14.682402 (+ 52768us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:14.682455 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:14.722525 (+ 40070us) log.cc:844] Serialized 26 byte log entry
0709 12:27:14.854514 (+131989us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:14.875864 (+ 21350us) log.cc:844] Serialized 14 byte log entry
0709 12:27:14.876148 (+   284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:14.876258 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:15.361176 (+484918us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b1800147a00. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:27:15.370973 (+  9797us) spinlock_profiling.cc:243] Waited 4.38 ms on lock 0x7b1800147a00. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ffe54658f 00007f4ff7585571 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd
0709 12:27:16.632789 (+1261816us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:16.768521 (+135732us) raft_consensus.cc:1685] finished
0709 12:27:16.768541 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:16.768684 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7,"prepare.queue_time_us":21761,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":35059,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":9349,"spinlock_wait_cycles":49417216,"thread_start_us":55008,"threads_started":1,"wal-append.queue_time_us":220699}
I20260709 12:27:16.850381  5675 leader_election.cc:400] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 17.
I20260709 12:27:16.829566  5633 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:27:16.794314 11983 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:16.830317  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 86986.8us
I20260709 12:27:16.818864  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:16.831545  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:16.831429  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:16.730393 12005 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:16.850741  5675 leader_election.cc:403] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:27:16.852218  5633 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:16.853979  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:16.861596  5556 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:16.838723 11774 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:27:16.839226 12038 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:16.864585  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45810.9us
I20260709 12:27:16.832413 11663 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:16.872098  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:16.864732  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:16.836730 11909 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:16.913784  5524 connection.cc:256] RPC call timeout handler was delayed by -1.7653151580002486s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1728s for a response
I20260709 12:27:16.837327 11723 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:16.897835  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:16.846592 11721 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:16.851084 12019 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:16.857488 11989 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:16.917801  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:16.924639  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:16.927726  5629 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:16.928332  5633 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:16.931224 12017 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:16.932210 11654 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:27:16.933189 12044 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:16.934429 12044 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:16.864616 12005 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:16.934829 12044 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:16.937103 12017 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:16.938138 12017 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:16.855407 11963 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:27:16.916735  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 66491.1us
I20260709 12:27:16.859755 11975 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:16.915578 12008 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 17.
I20260709 12:27:16.928269  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:16.959837  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:16.941982  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:16.862949 11976 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:16.949489  5629 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:16.969501  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:16.916680 11663 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:16.909976 11880 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:16.915936 11952 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:16.915473 11999 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:16.981909  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:16.985672  5556 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:16.982112  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:16.935746 12044 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:16.864467 12022 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.098s
I20260709 12:27:16.978754  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:16.986300  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:16.988384  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:16.993269  5552 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 14: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:16.946774 11723 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:16.988885  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60752.3us
I20260709 12:27:16.989894  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:16.990494  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:16.917161 12038 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:16.983980 12054 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.307s	user 0.000s	sys 0.002s
I20260709 12:27:16.998466  5560 raft_consensus.cc:2410] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 20 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 12:27:16.971017 11975 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.006707 12044 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:16.977344 12036 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:17.009577  5633 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:17.025955 11909 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:17.033011  6276 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:17.033370  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:17.033298 11983 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:27:17.035164 11983 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:27:17.036813  5561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 1141) took 1763 ms (client timeout 1985 ms). Trace:
W20260709 12:27:17.037271  5561 rpcz_store.cc:269] 0709 12:27:15.273791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:15.293612 (+ 19821us) service_pool.cc:224] Handling call
0709 12:27:17.036611 (+1742999us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001bd854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:17.036727 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3165913856}
I20260709 12:27:17.015759 11829 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.024s
I20260709 12:27:16.991441 11999 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.042371 11999 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:16.986171 11880 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:17.044091 11880 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:17.044773  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:17.044940 11991 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:17.045065 11855 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:17.071501  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:16.997298 11952 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:17.007324 11654 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:16.982928  5451 catalog_manager.cc:5697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 14, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 14 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:17.084048  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:17.093086  6277 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:17.093583  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:17.017576 12008 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:27:17.049433  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:17.018074 12036 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:17.050729 11773 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:17.052824  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:17.007833 11817 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:17.062316  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:17.057824  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:17.058970  6237 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.985s (SENT) [suppressed 30 similar messages]
I20260709 12:27:17.016130 11715 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
W20260709 12:27:17.080315  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 90520.7us
I20260709 12:27:17.108877  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:27:17.110322  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52582.1us
I20260709 12:27:16.972211 11721 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.085657  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:17.114936  5557 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:16.988020 11597 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.085s
I20260709 12:27:17.135138 11597 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:17.135622 11597 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:17.136647 11597 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:17.138209 11597 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:17.100489  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:17.141100  5560 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:17.142067  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:17.106799  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:17.112852  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:17.111209  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:17.014600 11723 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.117455 11975 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:17.099030 11654 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:17.117728 11721 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:17.118419 11952 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:17.119486 12036 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:17.120447  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:17.121191  5451 catalog_manager.cc:5697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 7 to 19, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 19 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:17.007421 11976 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.101459 11817 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:17.133001 11715 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:17.163781  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:17.133688  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:17.164597  5557 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:17.120451 11773 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:17.159535  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:17.111733 11909 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:17.142086 12022 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:27:17.144785  5675 leader_election.cc:400] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18.
I20260709 12:27:17.229158  5675 leader_election.cc:403] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:27:17.164769  5552 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:17.148326 11855 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:27:17.171387  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58629.3us
I20260709 12:27:17.230743  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:17.237620  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:17.158870 12008 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:17.163386 11817 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:17.164299 11715 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:17.148643 11829 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.107s
I20260709 12:27:17.230122 11721 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:17.267622  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:17.227315 11804 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:17.263098 12036 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:17.272845  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 283740us
I20260709 12:27:17.225781 11914 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:17.280434  5557 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:27:17.260668 12022 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:17.294796  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:17.281147  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:17.300026  5560 raft_consensus.cc:2368] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19.
W20260709 12:27:17.301066  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:17.304826  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:17.279287 11856 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:17.305706  5627 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:17.262563 12045 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:17.313613  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:17.271534 12008 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:17.283134 12049 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:17.323544 12049 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.266124 11982 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.292764 12010 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18.
I20260709 12:27:17.272620 11549 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.329447 11982 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:17.280444 11855 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:17.330989 11549 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:17.331948  5636 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:17.331555 11855 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:17.336483 12049 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:17.337307  5557 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:27:17.337698  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:17.338419  6280 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:17.338564  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:17.339919  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:17.339854  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:17.340610  5702 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:27:17.341434 12049 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:17.341805  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:17.342133 12049 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:17.337322 12008 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:17.342569 12049 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:17.344219 11817 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.346645 11817 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:17.355561 11833 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:17.356945 12010 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:17.357383 12010 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:17.358304 12010 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:17.313444 11897 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:17.299259 11914 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:17.409595  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:17.414425  5627 raft_consensus.cc:1217] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Deduplicated request from leader. Original: 14.8->[15.9-15.9]   Dedup: 15.9->[]
I20260709 12:27:17.416533  5627 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: [15.9-15.9]
W20260709 12:27:17.407027 12009 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.963s	user 0.009s	sys 0.013s
W20260709 12:27:17.419737 12009 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.984s	user 0.010s	sys 0.016s
I20260709 12:27:17.409129 11715 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.421116  5627 raft_consensus.cc:1217] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Deduplicated request from leader. Original: 19.12->[21.13-21.13]   Dedup: 21.13->[]
I20260709 12:27:17.421941  5627 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:27:17.431319 12018 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:17.433413 12018 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:17.404212 11897 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:17.441613 11897 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:17.386519 12005 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.446415 12005 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:17.322472 11856 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.449270 11856 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:17.452602  5703 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
W20260709 12:27:17.454293 12060 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260709 12:27:17.456573 12045 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:17.457890 11976 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:17.466260 12022 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:17.470433  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:17.301594 11654 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:17.494922 12038 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:17.502511  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:17.503372  6272 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:17.510213  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:17.477931 11986 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:17.542529  5553 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:27:17.535354 12038 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:17.543442 12038 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:17.543844 12038 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:17.509227 11914 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:17.606101  6237 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.040s	sys 0.044s
W20260709 12:27:17.611565  6237 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.040s	sys 0.044s
W20260709 12:27:17.613402  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 554417us
W20260709 12:27:17.617924  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:17.628248  5601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.024s	sys 0.071s
I20260709 12:27:17.311695  5451 catalog_manager.cc:5697] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 6 to 14, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 14 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:17.564697 11986 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:17.563668 11723 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:17.495256 12049 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:17.638165  5601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.024s	sys 0.071s
I20260709 12:27:17.638485  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:17.638895 11909 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:17.666780 11989 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.721s	user 0.013s	sys 0.000s
I20260709 12:27:17.643256 11986 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:17.642882  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:17.663126 11663 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.672s	user 0.010s	sys 0.000s
W20260709 12:27:17.708868  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 430136us
W20260709 12:27:17.667505  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:17.640573  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2714) took 1754 ms. Trace:
I20260709 12:27:17.719518  5559 rpcz_store.cc:276] 0709 12:27:15.886249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:16.003896 (+117647us) service_pool.cc:224] Handling call
0709 12:27:16.004426 (+   530us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:16.751690 (+747264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:17.640135 (+888445us) spinlock_profiling.cc:243] Waited 738 ms on lock 0x7b540044eb54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:17.640500 (+   365us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1696606464}
I20260709 12:27:17.686100 11721 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:17.728996  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 584282us
I20260709 12:27:17.732690  6280 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:17.732761  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:17.759491  5559 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:17.758816  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:17.760473  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 122091us
I20260709 12:27:17.761204  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:17.761936  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:17.762754  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:17.763386  5710 raft_consensus.cc:2376] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17. Re-sending same reply.
I20260709 12:27:17.764127  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:17.765446  5634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2515) took 1799 ms (client timeout 1572 ms). Trace:
W20260709 12:27:17.765858  5634 rpcz_store.cc:269] 0709 12:27:15.965631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:16.168978 (+203347us) service_pool.cc:224] Handling call
0709 12:27:17.765294 (+1596316us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b540043a154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:17.765386 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":688946944}
I20260709 12:27:17.766278  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:17.712182 12069 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:17.772465  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:17.721179 11989 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:17.779062  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:17.780141  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:17.781049  5559 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 18: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:17.783457  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:17.784204  5710 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:17.785204  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:17.785948  5634 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 20: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:17.786728  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:17.787382  6272 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:27:17.794147  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 320829us
I20260709 12:27:17.796463  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:17.797837  5559 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:17.799197  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:17.800184  5559 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:17.725605 11663 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.767123 11991 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:17.805248  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:17.773015 11914 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:17.819617  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:17.786712 11549 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:27:17.815220  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54110.6us
I20260709 12:27:17.821506  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:17.772878 11947 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:17.816051  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:17.772842 12005 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:17.764222 11723 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:27:17.824719  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:17.825605  5634 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:17.803526  5710 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:17.794126 12004 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:17.829493  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:17.834095  5560 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:27:17.818665 12064 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
W20260709 12:27:17.858283  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:17.820017 11914 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:17.828672 12004 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:17.862950  5559 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:17.861644  5634 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
W20260709 12:27:17.866712  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45387.5us
I20260709 12:27:17.917706  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:17.917712  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:17.868072  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:17.868846  5710 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:17.920876  5703 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:17.869133  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:17.838277 12084 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:27:17.869964  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:17.874349  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:17.922464  5627 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:17.922874  5553 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:17.875514  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:17.796674 12008 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:17.880735  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:17.863476 11721 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.866266  5558 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: [16.10-16.10]
W20260709 12:27:17.924170  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48761.9us
W20260709 12:27:17.973783  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:17.973951  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:17.974709  5628 raft_consensus.cc:2368] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
I20260709 12:27:17.926132  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:17.975601  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:17.976178  6279 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:17.977427  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:17.978294  6280 raft_consensus.cc:2368] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
I20260709 12:27:17.927075  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2786) took 1471 ms. Trace:
W20260709 12:27:17.927387 12074 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.154s	user 0.000s	sys 0.000s
I20260709 12:27:17.919669 11587 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:17.920778  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2789) took 1464 ms. Trace:
I20260709 12:27:17.824736 11991 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.987032  5628 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260709 12:27:17.934576  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2870) took 1122 ms. Trace:
I20260709 12:27:17.987475  5628 pending_rounds.cc:85] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 9
I20260709 12:27:17.987799  5628 pending_rounds.cc:107] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:27:17.921864 12004 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:17.999464 11997 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:17.936794 12077 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:17.937650  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:18.001626  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 64087.8us
I20260709 12:27:18.003242  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:17.938931  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2856) took 1274 ms. Trace:
I20260709 12:27:18.004400  5710 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: []
W20260709 12:27:17.939370 11880 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.885s	user 0.008s	sys 0.003s
W20260709 12:27:17.857370 11987 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.054s	user 0.014s	sys 0.001s
I20260709 12:27:17.941262 11804 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:27:17.944952  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:17.945536 12073 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.259s	user 0.000s	sys 0.001s
I20260709 12:27:17.946929 11663 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:17.915135 12072 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
I20260709 12:27:17.858135 11549 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:17.948956 12038 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:17.950520  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:17.951472 11715 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:17.895306 11723 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:17.897096 11914 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:17.831003 11947 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:17.953758  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:17.954528 12019 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.014s	user 0.011s	sys 0.000s
I20260709 12:27:17.829547  5451 catalog_manager.cc:5697] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 11 to 16. New cstate: current_term: 16 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:27:17.958627  5704 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:27:17.831923 11909 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:17.837846 11897 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:27:17.960754 12075 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.235s	user 0.000s	sys 0.001s
I20260709 12:27:17.962694 11720 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:17.964628 12049 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:17.967902 12080 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.192s	user 0.002s	sys 0.000s
I20260709 12:27:17.820329 12083 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.098s
I20260709 12:27:17.969224  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:18.130498  5560 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:17.970199  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 1217) took 1329 ms. Trace:
I20260709 12:27:18.103804 11663 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.128033  5627 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:17.981108  6272 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:17.982372  5526 leader_election.cc:400] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
I20260709 12:27:17.978771  5630 rpcz_store.cc:276] 0709 12:27:16.455778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:16.812046 (+356268us) service_pool.cc:224] Handling call
0709 12:27:16.812296 (+   250us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:16.824834 (+ 12538us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:16.824875 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:16.838726 (+ 13851us) log.cc:844] Serialized 26 byte log entry
0709 12:27:17.257656 (+418930us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:17.257705 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:17.257774 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:17.926826 (+669052us) raft_consensus.cc:1685] finished
0709 12:27:17.926848 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:17.927007 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":418550,"threads_started":1,"wal-append.queue_time_us":769951}
I20260709 12:27:18.131733  5636 rpcz_store.cc:276] 0709 12:27:16.640693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:16.806088 (+165395us) service_pool.cc:224] Handling call
0709 12:27:17.969973 (+1163885us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b540032c8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:17.970131 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1186203008}
I20260709 12:27:17.972672 11999 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:17.839468 11832 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.095s
I20260709 12:27:17.988138  5628 raft_consensus.cc:2889] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:17.987550  5708 rpcz_store.cc:276] 0709 12:27:16.811747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:16.814882 (+  3135us) service_pool.cc:224] Handling call
0709 12:27:16.815202 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:16.861205 (+ 46003us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:27:17.227020 (+365815us) log.cc:844] Serialized 14 byte log entry
0709 12:27:17.533514 (+306494us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:17.533592 (+    78us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:17.533658 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:17.934282 (+400624us) spinlock_profiling.cc:243] Waited 45.8 ms on lock 0x7b5400323f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:17.934331 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:17.934507 (+   176us) inbound_call.cc:177] Queueing success response
0709 12:27:17.967342 (+ 32835us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
Metrics: {"mutex_wait_us":60090,"prepare.queue_time_us":542761,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":36812,"raft.queue_time_us":1079,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":70339,"spinlock_wait_cycles":111134464,"thread_start_us":634306,"threads_started":2,"wal-append.queue_time_us":489292}
W20260709 12:27:17.989938 12067 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.291s	user 0.000s	sys 0.003s
I20260709 12:27:17.990463  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:17.983661  5635 rpcz_store.cc:276] 0709 12:27:16.456733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:16.818560 (+361827us) service_pool.cc:224] Handling call
0709 12:27:16.848425 (+ 29865us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:16.848696 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:27:16.863242 (+ 14546us) log.cc:844] Serialized 14 byte log entry
0709 12:27:17.919992 (+1056750us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b1800342220. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:27:17.920224 (+   232us) log.cc:844] Serialized 14 byte log entry
0709 12:27:17.920418 (+   194us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:17.920472 (+    54us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:27:17.920527 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:17.920597 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:17.920720 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1566,"prepare.queue_time_us":721291,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":90,"raft.queue_time_us":230,"raft.run_cpu_time_us":425,"raft.run_wall_time_us":26023,"spinlock_wait_cycles":6208896,"thread_start_us":367241,"threads_started":1,"wal-append.queue_time_us":545969}
I20260709 12:27:17.986425 11952 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:27:18.145052  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 154644us
I20260709 12:27:18.145169 11952 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.145826  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:18.146319  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2793) took 1567 ms. Trace:
I20260709 12:27:18.147058  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:18.147274 12019 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:18.147519 11952 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:18.147863 12019 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:17.993085 11855 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:18.148645 11855 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.148806 12019 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.150782 11855 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:18.150848 11987 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:18.151657 11987 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:17.990427  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:27:18.152881  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 162495us
I20260709 12:27:17.994200  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.155526 12021 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:18.156060  5556 raft_consensus.cc:2410] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:27:18.156390 12021 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:18.157657 12021 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:17.982751 12009 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:17.998796 12036 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.721s	user 0.010s	sys 0.000s
I20260709 12:27:18.007539  5707 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:18.004804  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:18.011088  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1213) took 1371 ms. Trace:
I20260709 12:27:18.010926  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.007673  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:18.161713  5559 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:27:18.011883  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2505) took 1068 ms. Trace:
I20260709 12:27:18.162190  5554 rpcz_store.cc:276] 0709 12:27:16.943331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:16.943481 (+   150us) service_pool.cc:224] Handling call
0709 12:27:18.011661 (+1068180us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b5400082354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:18.011814 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1976982784}
I20260709 12:27:18.021977  6271 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:27:18.004522  5709 rpcz_store.cc:276] 0709 12:27:16.664686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:16.665339 (+   653us) service_pool.cc:224] Handling call
0709 12:27:16.665555 (+   216us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:16.838132 (+172577us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:27:17.307850 (+469718us) log.cc:844] Serialized 14 byte log entry
0709 12:27:17.688873 (+381023us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:17.688930 (+    57us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:17.688987 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:17.938681 (+249694us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54001a9354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:17.938729 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:17.938859 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63449,"prepare.queue_time_us":482660,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":800,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":11170,"spinlock_wait_cycles":396271360,"thread_start_us":799164,"threads_started":2,"wal-append.queue_time_us":555467}
I20260709 12:27:18.042773  5602 leader_election.cc:400] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19.
W20260709 12:27:18.164173  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 149951us
I20260709 12:27:18.100332  5634 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:27:18.108285  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.109154  6279 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:18.167546  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.167820  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.168188  5553 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:18.169024  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.169720  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.170143  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.170387  5553 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 12.
I20260709 12:27:18.171236  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.023067 12077 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:18.020507 11804 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:18.001677 12008 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:18.011305 11597 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:18.091797 11723 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:18.005788 11587 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.103014 11983 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:18.123972  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:18.011219 11880 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:17.979056  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 196200us
I20260709 12:27:18.132738  5526 leader_election.cc:403] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
W20260709 12:27:18.140239  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:18.140811  5630 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:27:18.156584  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2511) took 1154 ms. Trace:
I20260709 12:27:18.233222  5556 rpcz_store.cc:276] 0709 12:27:17.001810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:17.002014 (+   204us) service_pool.cc:224] Handling call
0709 12:27:17.994041 (+992027us) spinlock_profiling.cc:243] Waited 28.5 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:27:18.156521 (+162480us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":65644288}
I20260709 12:27:17.996659 12010 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:18.161453  5629 rpcz_store.cc:276] 0709 12:27:16.639616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:16.993249 (+353633us) service_pool.cc:224] Handling call
0709 12:27:17.018499 (+ 25250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:17.232190 (+213691us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:17.232224 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:17.232270 (+    46us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:17.232304 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:18.010793 (+778489us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b5400240754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:18.010854 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:18.011019 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":490731392}
I20260709 12:27:18.171375  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:18.171887  5561 raft_consensus.cc:2368] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17.
I20260709 12:27:18.185969 12077 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:18.173928  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:18.244416  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70661.2us
I20260709 12:27:18.244524 11198 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:18.088603  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:18.184543  5558 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:27:18.187245  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:18.245630  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 157128us
W20260709 12:27:18.188390  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:18.246906  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:18.160705 12009 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.189623 12076 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:27:18.190672 12083 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:18.118594 11978 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 12:27:18.196543 12101 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:27:18.197317 11987 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:27:18.207908  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:18.211853  5552 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:18.239703 12039 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.146682  5631 rpcz_store.cc:276] 0709 12:27:16.579062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:16.933037 (+353975us) service_pool.cc:224] Handling call
0709 12:27:16.933392 (+   355us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:16.933825 (+   433us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:16.933853 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:16.934419 (+   566us) log.cc:844] Serialized 52 byte log entry
0709 12:27:17.308158 (+373739us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:17.928758 (+620600us) log.cc:844] Serialized 14 byte log entry
0709 12:27:17.929015 (+   257us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:17.929128 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:17.989634 (+ 60506us) spinlock_profiling.cc:243] Waited 32.2 ms on lock 0x7b240029c138. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff740b9c4 00007f4ff7409acb 00007f4ff75c4dc9 00007f4ff75c4a55 00007f4ff75c409d 00007f4ffa1f610e 00007f4ffa20d88c 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221
0709 12:27:18.145999 (+156365us) raft_consensus.cc:1685] finished
0709 12:27:18.146109 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:18.146261 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":612026,"prepare.queue_time_us":621112,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":10591,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":207,"spinlock_wait_cycles":73978624,"thread_start_us":342620,"threads_started":1,"wal-append.queue_time_us":700862}
I20260709 12:27:18.115628 11549 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.164564 11720 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:18.172448 12049 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.201022 11832 consensus_queue.cc:1243] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 16.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260709 12:27:18.236802  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 258044us
I20260709 12:27:18.237672  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:18.171947  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:18.309357  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 137489us
I20260709 12:27:18.310322  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:18.239759  5553 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:18.241715  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:18.313541  6277 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:18.270270 12083 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:18.251075  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:18.237871  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:18.252887  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:18.165804 11663 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.262635  5706 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:18.263600  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:18.233955 11986 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.268639  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:18.345252  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:18.345547  5704 raft_consensus.cc:2410] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:27:18.269608  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:18.277866  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:18.245182 11198 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:18.250444 12022 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.256644 11833 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:18.285838  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 118251us
W20260709 12:27:18.354980  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47181.1us
I20260709 12:27:18.255398 12077 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:18.287250  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:18.297655  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:18.362525  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:18.299427 11947 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:18.264250 11817 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:18.263808 12008 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:18.322510  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.377768  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:18.337947 11654 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:18.325752  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:18.395241  5561 raft_consensus.cc:2368] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17.
I20260709 12:27:18.276952 11992 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:18.232990 11983 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.336239  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:18.340778  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 333162us
I20260709 12:27:18.341944  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:18.260170  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:18.340791  6276 raft_consensus.cc:2410] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:27:18.353704 11198 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:18.359122  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 121303us
I20260709 12:27:18.328805 12069 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:18.317862 12049 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.370805  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:18.320308 12070 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:18.378286  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.365469 11989 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.393671  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:18.398422  6280 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:18.357549 11597 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:27:18.405157  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48919.3us
I20260709 12:27:18.405110 11947 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:18.408859  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:18.408851  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:18.401919  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:18.308535 12039 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.364529 11833 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.339741 12083 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:18.367074 11817 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:18.249967  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:18.260891  5451 catalog_manager.cc:5697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 10 to 12, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 12 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:27:18.446753  5556 raft_consensus.cc:2410] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:27:18.421545  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:18.391584 12038 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:18.426367  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:18.364907 12102 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:18.406256  5558 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:18.427595  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:18.363521 12022 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.410818  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:18.431092  5553 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260709 12:27:18.429356  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:18.456321  5553 pending_rounds.cc:85] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 12
I20260709 12:27:18.457146  5553 pending_rounds.cc:107] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 12:27:18.457448  5553 raft_consensus.cc:2889] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:18.458789  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:18.436203  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:18.413793 12112 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:18.439788  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:18.440541 12070 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:27:18.441179 11549 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:18.428166 11978 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:18.415666 11992 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:18.448125  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:18.344444 11723 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.448912 11832 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.074s
I20260709 12:27:18.524560 12110 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.008s
I20260709 12:27:18.525148 11723 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:18.407399 12069 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:18.456250 11997 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.399811 12008 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.453927  5557 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:18.455696  5704 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:18.484553 12038 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.453660 11198 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.456319  5702 raft_consensus.cc:2376] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17. Re-sending same reply.
I20260709 12:27:18.541088  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:18.541687  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:18.461329 12113 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:27:18.468097  5524 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.608s (SENT)
I20260709 12:27:18.426064 11597 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:18.474033  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:18.475381 12108 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:18.475660  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:18.489002  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:18.502071 12092 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.278s	user 0.001s	sys 0.001s
W20260709 12:27:18.502138 12093 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.281s	user 0.001s	sys 0.001s
I20260709 12:27:18.448770 11721 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:18.445621 11663 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:18.407157 11654 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.438441 12076 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:18.517292  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:18.426687 11989 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.449667 11817 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:18.518159  5560 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
W20260709 12:27:18.522480  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 82663.4us
I20260709 12:27:18.454423 12102 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:18.502892 12104 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:27:18.524219 12109 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:18.525734  5561 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 12:27:18.614053  5561 pending_rounds.cc:85] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 8
I20260709 12:27:18.614425  5561 pending_rounds.cc:107] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:27:18.614713  5561 raft_consensus.cc:2889] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:18.448683  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2528) took 1117 ms. Trace:
I20260709 12:27:18.462154 11947 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:18.532907 12070 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:27:18.615624  5556 rpcz_store.cc:276] 0709 12:27:17.331051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.249709 (+918658us) service_pool.cc:224] Handling call
0709 12:27:18.448591 (+198882us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:18.615958 12102 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:18.515813 11587 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:18.616842 11587 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.617290 11587 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.619391 11587 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:18.620781 11978 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:18.550514  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:18.622177  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:18.623188  5704 raft_consensus.cc:2410] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:27:18.527976 11976 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:18.624531 11976 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:18.625579 11976 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.627342 11976 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:18.565584 11198 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:18.567346  6237 leader_election.cc:400] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17.
I20260709 12:27:18.629185  6237 leader_election.cc:403] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
W20260709 12:27:18.630379  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63528.8us
W20260709 12:27:18.632220 12115 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:27:18.602716 12045 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
I20260709 12:27:18.587728 11991 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:18.641598  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 173580us
I20260709 12:27:18.537050 11997 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:18.580963  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 107012us
I20260709 12:27:18.481775 11909 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260709 12:27:18.530818 12069 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:18.534606 12036 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:18.592253  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 75030.7us
I20260709 12:27:18.593238  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:18.543298 12009 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:18.482735 12112 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.567265 11597 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:18.602607  5676 leader_election.cc:400] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17.
I20260709 12:27:18.578564 11549 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:27:18.605096 12107 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260709 12:27:18.605887 12111 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:27:18.607762 12085 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:18.705590 12085 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:18.705998 12085 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:18.578753 12098 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:18.708173 12098 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.708680 12112 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:18.709434 12112 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.709973 12112 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.710983 12098 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:18.567545 12021 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:18.711903 12021 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.609714  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1215) took 1969 ms. Trace:
I20260709 12:27:18.581560 11721 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:18.715015 12120 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:18.591768 11663 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.611340  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2795) took 2031 ms. Trace:
I20260709 12:27:18.714687  5633 rpcz_store.cc:276] 0709 12:27:16.640117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:17.018659 (+378542us) service_pool.cc:224] Handling call
0709 12:27:17.018959 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:17.395919 (+376960us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:17.395960 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:17.418963 (+ 23003us) log.cc:844] Serialized 26 byte log entry
0709 12:27:17.879653 (+460690us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:17.879695 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:17.879777 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:18.609285 (+729508us) raft_consensus.cc:1685] finished
0709 12:27:18.609322 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:18.609612 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":86873,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":6896,"thread_start_us":706704,"threads_started":2,"wal-append.queue_time_us":592683}
I20260709 12:27:18.716527  5632 rpcz_store.cc:276] 0709 12:27:16.579744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:16.960379 (+380635us) service_pool.cc:224] Handling call
0709 12:27:16.981154 (+ 20775us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:17.393857 (+412703us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:17.393900 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:17.408634 (+ 14734us) log.cc:844] Serialized 52 byte log entry
0709 12:27:17.879010 (+470376us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:17.982834 (+103824us) log.cc:844] Serialized 14 byte log entry
0709 12:27:17.983062 (+   228us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:17.983135 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:18.611074 (+627939us) raft_consensus.cc:1685] finished
0709 12:27:18.611096 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:18.611267 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":92226,"prepare.queue_time_us":113226,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":101201,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":19179,"thread_start_us":752401,"threads_started":2,"wal-append.queue_time_us":603378}
I20260709 12:27:18.612946  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:18.718842  5705 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:18.616122 12070 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:18.459348  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2523) took 1130 ms. Trace:
I20260709 12:27:18.720497  5558 rpcz_store.cc:276] 0709 12:27:17.329367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.259835 (+930468us) service_pool.cc:224] Handling call
0709 12:27:18.459266 (+199431us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:18.721451  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.722651  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:18.723712  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.577057  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:18.723904 12085 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.725886 12085 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:18.726146  5702 raft_consensus.cc:1217] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Deduplicated request from leader. Original: 14.8->[15.9-15.9]   Dedup: 15.9->[]
I20260709 12:27:18.726701  5702 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:27:18.638053 11198 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:18.720276 12070 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:18.737177  5705 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:27:18.590396  5451 catalog_manager.cc:5697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 9 to 12, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 12 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:18.638932 12018 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.640851  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2728) took 1577 ms. Trace:
I20260709 12:27:18.740613  5552 rpcz_store.cc:276] 0709 12:27:17.063062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:17.230245 (+167183us) service_pool.cc:224] Handling call
0709 12:27:17.262532 (+ 32287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:18.283538 (+1021006us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:18.640608 (+357070us) spinlock_profiling.cc:243] Waited 949 ms on lock 0x7b540016ba54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:18.640792 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2182813056}
I20260709 12:27:18.742354  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:18.551563 12038 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.613945 12109 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:18.744618 12109 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:18.745142 12022 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:18.678675  5524 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.614s (SENT)
I20260709 12:27:18.748494 12036 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:18.749205  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 106914us
I20260709 12:27:18.751616 12036 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:18.678874  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 220162us
I20260709 12:27:18.685041  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:18.685302  6274 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:18.635502 11992 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.700014  6275 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 12:27:18.754581  5602 connection.cc:256] RPC call timeout handler was delayed by -1.1226384500000677s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21120000000000003s for a response
I20260709 12:27:18.629601 11986 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:18.700050  5676 leader_election.cc:403] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:27:18.702116  6276 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:27:18.758467 12120 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 17
W20260709 12:27:18.758635  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 157574us
I20260709 12:27:18.702195  5708 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:27:18.715255 11721 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:18.715853 11663 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.717947 12021 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:18.613456 11654 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:18.753243 12008 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:18.724267  5558 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
W20260709 12:27:18.767869  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:18.802508  6272 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 12. (term mismatch)
I20260709 12:27:18.803157  6272 pending_rounds.cc:85] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 11
I20260709 12:27:18.803468  6272 pending_rounds.cc:107] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 12:27:18.803786  6272 raft_consensus.cc:2889] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:18.805490  5552 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 12:27:18.808287  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:18.818492  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48604.6us
I20260709 12:27:18.806465  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2539) took 1360 ms. Trace:
I20260709 12:27:18.788003 11663 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:18.624182  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:18.824985  5552 rpcz_store.cc:276] 0709 12:27:17.446498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.743524 (+1297026us) service_pool.cc:224] Handling call
0709 12:27:18.762127 (+ 18603us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:18.806090 (+ 43963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:18.806391 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:18.671247 12084 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.739173  5451 catalog_manager.cc:5697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 17. New cstate: current_term: 17 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:18.832664  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:18.835391  5634 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:18.837208  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:18.739928 12018 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.849100  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:18.849345  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:18.789700 12110 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.850761 11198 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17.
I20260709 12:27:18.851773  5628 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:27:18.852134  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:18.850134  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:18.854527  5629 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:18.854555  5558 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:18.854990  5631 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 12:27:18.855198  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.715269 11833 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:18.857244  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:18.858300  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:18.865335  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:18.859469  5703 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:18.853422  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:18.858698  5635 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: [15.10-15.10]
I20260709 12:27:18.864058  5552 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:18.826342 11654 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:18.900558  5703 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:18.902338  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:18.903914  5558 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:18.763211 12036 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.094s
I20260709 12:27:18.843048 11947 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:18.906683  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2534) took 1515 ms. Trace:
I20260709 12:27:18.910135  5556 rpcz_store.cc:276] 0709 12:27:17.391189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.616417 (+1225228us) service_pool.cc:224] Handling call
0709 12:27:18.616748 (+   331us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:18.617074 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:18.617104 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:18.617590 (+   486us) log.cc:844] Serialized 26 byte log entry
0709 12:27:18.693450 (+ 75860us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:18.905065 (+211615us) log.cc:844] Serialized 14 byte log entry
0709 12:27:18.906222 (+  1157us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:18.906321 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:18.906393 (+    72us) raft_consensus.cc:1685] finished
0709 12:27:18.906411 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:18.906603 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":859,"raft.queue_time_us":21720,"raft.run_cpu_time_us":56,"raft.run_wall_time_us":56,"thread_start_us":286218,"threads_started":2,"wal-append.queue_time_us":75692}
I20260709 12:27:18.844324 12113 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:18.880422 11198 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:18.912920  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.930871  6275 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:18.866276  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:18.931118 11198 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:18.935923 11198 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:18.843657 12084 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:18.949069  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 82903us
I20260709 12:27:18.931591  5703 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.950801  5703 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:18.863547 11663 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.906123 12083 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.944868  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:18.900485 12110 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.946163 12008 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:18.947029 11909 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:18.947932 12102 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.951719  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:18.922927  5633 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:18.955709  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:18.939988 12022 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:18.860922 12009 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.079s
I20260709 12:27:18.972636  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:18.972844  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:18.974001 11198 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:18.958951  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:18.958882  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2532) took 1626 ms. Trace:
I20260709 12:27:18.978386  5701 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:18.962945  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:18.966032  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:18.979527  5710 raft_consensus.cc:2368] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19.
W20260709 12:27:18.962723 12004 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.963s	user 0.014s	sys 0.001s
I20260709 12:27:18.948400 12005 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.005s
I20260709 12:27:18.995290 12110 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:18.981211  5556 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 18: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:18.996169  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:18.996630  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:18.996827  5636 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:18.997161  5556 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:18.997202 11804 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.795s	user 0.012s	sys 0.000s
I20260709 12:27:18.997699  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:18.998484  5636 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:18.997941 11804 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.999305  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:18.999985  5636 raft_consensus.cc:2368] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:18.920670 11654 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.002835 11654 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:18.982237  5558 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:18.977334  5560 rpcz_store.cc:276] 0709 12:27:17.332190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.599480 (+1267290us) service_pool.cc:224] Handling call
0709 12:27:18.600096 (+   616us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:18.600385 (+   289us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:27:18.850565 (+250180us) log.cc:844] Serialized 14 byte log entry
0709 12:27:18.958504 (+107939us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:18.958562 (+    58us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:18.958618 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:18.958705 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:18.958825 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9359,"thread_start_us":264793,"threads_started":2}
I20260709 12:27:19.006009  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:19.006292  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:19.006675  5560 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:19.006995 11198 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:19.007746  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:18.993384  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:19.009068  5633 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:19.009909  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:18.994086 12004 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.010995  5560 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:27:19.012893  5556 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:19.013762 11198 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:19.014448 11198 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.016033  5703 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:27:18.947856 11833 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:27:18.968813 12085 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:19.020882  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1163) took 1403 ms. Trace:
I20260709 12:27:19.021255  5560 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 20. Ops: []
I20260709 12:27:19.024185  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1164) took 1406 ms. Trace:
I20260709 12:27:18.934937 11947 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.030129  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2542) took 1582 ms. Trace:
I20260709 12:27:19.029794  5556 rpcz_store.cc:276] 0709 12:27:17.617538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.021857 (+1404319us) service_pool.cc:224] Handling call
0709 12:27:19.023083 (+  1226us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.023459 (+   376us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:19.023499 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:19.023622 (+   123us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:19.023720 (+    98us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.023843 (+   123us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:19.024065 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:19.031196  5633 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:18.972293 12112 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:18.982980 12022 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:18.963075 11663 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:18.980907 12124 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:19.030821  5558 rpcz_store.cc:276] 0709 12:27:17.617110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.019787 (+1402677us) service_pool.cc:224] Handling call
0709 12:27:19.020183 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.020397 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:19.020428 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:19.020599 (+   171us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:19.020644 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.020694 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:19.020807 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49408}
W20260709 12:27:19.099376  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:19.079726  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:19.058403  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2526) took 1728 ms. Trace:
W20260709 12:27:19.054653  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 809399us
W20260709 12:27:19.064328  5557 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2530) took 1732 ms (client timeout 1867 ms). Trace:
W20260709 12:27:19.129312  5557 rpcz_store.cc:269] 0709 12:27:17.331627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:17.339689 (+  8062us) service_pool.cc:224] Handling call
0709 12:27:19.064046 (+1724357us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54000ebfd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:19.064237 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2533819776}
I20260709 12:27:19.127656  5553 rpcz_store.cc:276] 0709 12:27:17.330266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.418066 (+1087800us) service_pool.cc:224] Handling call
0709 12:27:18.418503 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:18.473414 (+ 54911us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.058156 (+584742us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b54003a98d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:19.058337 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27886464}
I20260709 12:27:19.130846  5557 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:27:19.131593  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2732) took 1260 ms. Trace:
W20260709 12:27:19.127965  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:19.129647  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:19.135326  5633 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:19.123204  5560 rpcz_store.cc:276] 0709 12:27:17.447321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.019785 (+1572464us) service_pool.cc:224] Handling call
0709 12:27:19.020183 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.029834 (+  9651us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.030072 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30976}
I20260709 12:27:19.098425  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:19.138047  5636 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:19.135780  5553 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:19.069466  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2527) took 1739 ms. Trace:
I20260709 12:27:19.017009 12123 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:19.097795  5705 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:27:19.138809  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2734) took 1267 ms. Trace:
W20260709 12:27:19.120185 12010 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.856s	user 0.007s	sys 0.007s
I20260709 12:27:19.145649 11986 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:27:19.056805 12005 consensus_queue.cc:1243] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:27:19.138839 11198 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:19.099061 12077 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.722s	user 0.010s	sys 0.002s
W20260709 12:27:19.048024 11914 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.941s	user 0.010s	sys 0.000s
I20260709 12:27:19.146682  5558 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:19.136711 12112 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:19.017854 11976 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:19.145699 12098 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.012s
I20260709 12:27:19.141594  5552 raft_consensus.cc:2410] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
W20260709 12:27:19.057749 12103 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
W20260709 12:27:19.152499  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 207010us
I20260709 12:27:19.147104  5553 rpcz_store.cc:276] 0709 12:27:17.871631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.130608 (+1258977us) service_pool.cc:224] Handling call
0709 12:27:19.130844 (+   236us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.138573 (+  7729us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.138755 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:19.029363 12085 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:19.155396  5633 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:19.048625 12121 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:19.132174  5557 rpcz_store.cc:276] 0709 12:27:17.870936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.130204 (+1259268us) service_pool.cc:224] Handling call
0709 12:27:19.130533 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.131275 (+   742us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.131523 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:19.156574  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:19.049211 11997 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.209530  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2740) took 1202 ms. Trace:
I20260709 12:27:19.209800  5557 rpcz_store.cc:276] 0709 12:27:18.007361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.208831 (+1201470us) service_pool.cc:224] Handling call
0709 12:27:19.209117 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.209305 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:19.209331 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:19.209368 (+    37us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:27:19.209388 (+    20us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.209420 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:19.209488 (+    68us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:19.210773  5557 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:19.211061  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:19.211668  5557 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:27:19.057255 12018 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.044037 11804 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.068404 12102 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:19.157935  5556 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:19.158713  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:19.042680 12083 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:19.162446 11817 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.526s	user 0.016s	sys 0.001s
I20260709 12:27:19.138769  5561 rpcz_store.cc:276] 0709 12:27:17.330709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.420620 (+1089911us) service_pool.cc:224] Handling call
0709 12:27:18.435742 (+ 15122us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:18.615312 (+179570us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.069045 (+453733us) spinlock_profiling.cc:243] Waited 84.1 ms on lock 0x7b540023fd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:19.069400 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":193397120}
I20260709 12:27:19.021169 12008 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:19.058941 11597 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.186900  5706 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
W20260709 12:27:19.175570  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 219995us
I20260709 12:27:19.176213  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:19.195622  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:19.022406 12113 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.154927 12021 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:19.139294 12124 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.164394 12038 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:19.221213  6273 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 18: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:19.223984  5557 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:27:19.246304  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:19.248344  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:19.195679 12009 consensus_queue.cc:1243] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 14.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:27:19.251523 12009 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:19.229259  6277 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:19.230254  5633 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:27:19.256035  5633 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:27:19.225656 12008 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:19.257508  5633 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
W20260709 12:27:19.257627 12106 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.896s	user 0.000s	sys 0.004s
I20260709 12:27:19.211998 12085 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:19.225551  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:19.269425  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:19.272708 11999 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:19.273240 11999 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:19.274307 11947 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.173648 11914 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.276430 11914 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:19.277165  5702 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:27:19.245863 12021 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.279158 12085 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.228469 12121 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:19.279853 12021 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:19.280247 12121 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:19.280390  5703 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:27:19.227552 12070 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.282416 12070 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:19.282541 12085 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:19.250353 12131 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:19.283277 12131 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.284504 12124 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.284826 11832 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:19.285085 12124 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:19.285229 11832 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:19.285573 12124 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:19.286306 11721 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.526s	user 0.012s	sys 0.000s
I20260709 12:27:19.286119 11832 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.286903 11721 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.288020 12124 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.258451  5636 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:19.259466 11597 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.252027 12009 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.309927 12131 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:19.262105  6279 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:19.262195  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:19.263091  6271 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1133) took 2227 ms (client timeout 2149 ms). Trace:
W20260709 12:27:19.311232  6271 rpcz_store.cc:269] 0709 12:27:17.035373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:17.044500 (+  9127us) service_pool.cc:224] Handling call
0709 12:27:19.262835 (+2218335us) spinlock_profiling.cc:243] Waited 966 ms on lock 0x7b54004f9cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:19.262973 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2222431104}
I20260709 12:27:19.263224  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1118) took 1280 ms. Trace:
I20260709 12:27:19.312405  6280 rpcz_store.cc:276] 0709 12:27:17.983222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:17.983826 (+   604us) service_pool.cc:224] Handling call
0709 12:27:17.984114 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:18.423851 (+439737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.262992 (+839141us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x7b54005f5d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:19.263165 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":905048832}
I20260709 12:27:19.177944 12112 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.070037  5451 catalog_manager.cc:5697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } }
I20260709 12:27:19.264935  5561 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
W20260709 12:27:19.265017 11723 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.739s	user 0.009s	sys 0.002s
I20260709 12:27:19.266028 12022 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.266352 11909 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:27:19.266780 11989 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:19.233870  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 282214us
I20260709 12:27:19.270630 12004 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:19.227780 11817 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.234170 12110 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:19.274183 11999 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.279588 11663 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:19.287384 11832 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:19.300850 12069 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.539s	user 0.000s	sys 0.012s
I20260709 12:27:19.288252 12081 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:19.288780  5704 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 18: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:19.250487 12083 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:19.289230 11983 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:19.289464 12109 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.544s	user 0.000s	sys 0.012s
I20260709 12:27:19.257292 12008 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:19.290931  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:19.291721  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:19.170207 12120 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17.
I20260709 12:27:19.249272  5557 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:19.169833 12077 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.297376  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:19.413271  5553 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:19.220549 11976 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:19.385143  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:19.416014  6274 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:19.415387  6273 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:19.356472 12131 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:19.392168  6276 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:19.406467 12069 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:19.338074  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 75949.1us
I20260709 12:27:19.396384 11992 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:19.400606  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:19.398818 11909 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 13 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:19.355731 11597 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:19.300577 12101 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.109s
I20260709 12:27:19.311075 12021 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.105s
I20260709 12:27:19.404889 12081 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:19.422010  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 130466us
I20260709 12:27:19.420027  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:19.428010 12009 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:19.431561 11817 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.428114 12022 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.009s
I20260709 12:27:19.437502 12008 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:19.438369  5557 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:19.439738 11721 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:19.440881 12112 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.427074 11992 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.489287  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:19.489892  5557 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:19.490554 11992 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:19.445660 11976 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:19.449312  5701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2864) took 2723 ms (client timeout 1869 ms). Trace:
W20260709 12:27:19.492187  5701 rpcz_store.cc:269] 0709 12:27:16.727398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:16.766354 (+ 38956us) service_pool.cc:224] Handling call
0709 12:27:19.446688 (+2680334us) spinlock_profiling.cc:243] Waited 2.16 s on lock 0x7b540026fa54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:19.446797 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4965582592}
I20260709 12:27:19.493470  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:19.450074  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:19.494261  5561 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:19.495158  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:19.497033  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:19.497579  5561 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:19.402442 12110 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.419703 12004 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:19.419262 12131 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:19.470777 12039 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.481269 12085 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:19.481804 11914 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:19.482971 12009 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:19.485738 12077 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.486774 12081 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.498994  5709 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:19.500762  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:19.418052 12109 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.497936 12120 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 12:27:19.543113  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:19.509238  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:19.510342  6272 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:19.510318  5557 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:19.513911  6271 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:19.516366 12112 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:27:19.517210  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:19.518844 11880 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.568274 12109 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:19.521456  6275 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:19.537814 12085 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.569588  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1107) took 1333 ms. Trace:
I20260709 12:27:19.569777  5701 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 16: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:19.570545  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:19.571305 12085 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:19.571509  5627 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:19.571653  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:19.571888  5710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2631) took 1322 ms (client timeout 1614 ms). Trace:
W20260709 12:27:19.572285  5710 rpcz_store.cc:269] 0709 12:27:18.249890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.251145 (+  1255us) service_pool.cc:224] Handling call
0709 12:27:19.571692 (+1320547us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b54002d19d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:19.571838 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1456135424}
I20260709 12:27:19.569983  6279 rpcz_store.cc:276] 0709 12:27:18.236250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.242598 (+  6348us) service_pool.cc:224] Handling call
0709 12:27:18.243027 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.310822 (+1067795us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.569226 (+258404us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400771d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:19.569521 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2343166720}
I20260709 12:27:19.521868  5526 leader_election.cc:400] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
W20260709 12:27:19.575605  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 53860.4us
I20260709 12:27:19.523401  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:19.523952  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:19.462744 11723 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:19.527956  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:19.532065  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:19.532588  5632 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:27:19.583840  5631 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:19.533133  5561 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:19.534621  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:19.535303  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:19.533875  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:19.582939  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:19.586257  5627 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:19.450935 11597 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.545212  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:19.548117  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:19.561299 11914 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.539752 12103 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:19.595889  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:19.542905 12009 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.596827  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:19.597463  6277 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 13.
I20260709 12:27:19.598225 12009 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:19.565276 12081 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:19.599490  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:19.600067  6277 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:19.600622  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:19.601136  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:19.566098 12004 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:19.602069  5701 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:19.601923 12004 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.589839 12120 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
I20260709 12:27:19.565944  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:19.596000 12103 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.566407 11992 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:19.604053 12004 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:19.604254 11992 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:19.604705 11992 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:19.604980 12069 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:19.605332  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:19.605803 11597 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.611111  5631 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:27:19.567260  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:19.568001  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:19.568313 12136 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:27:19.577056  5628 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:19.579792  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:19.632740  6279 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:19.545186  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:19.581373  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:19.583106  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:19.588261  5557 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:19.533447 12131 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:19.594220 11914 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:19.536041 11721 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:19.601207  5561 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:19.536190 12132 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:19.612672 11723 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:19.612550 12010 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:27:19.613885  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:19.686724  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 72915.8us
I20260709 12:27:19.631747 12039 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:19.617558 11715 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:19.623265  6271 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:19.566932 12121 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.624583  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:19.695881  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 71373.3us
I20260709 12:27:19.696182  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2736) took 1823 ms. Trace:
W20260709 12:27:19.696774  5559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2525) took 2366 ms (client timeout 2112 ms). Trace:
I20260709 12:27:19.638775  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:19.697304  5559 rpcz_store.cc:269] 0709 12:27:17.329952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:17.917415 (+587463us) service_pool.cc:224] Handling call
0709 12:27:19.696512 (+1779097us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b540044f554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:19.696682 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":554400640}
I20260709 12:27:19.697925  5703 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:19.649037  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:19.643263  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 98144.5us
I20260709 12:27:19.699275  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2737) took 1826 ms. Trace:
I20260709 12:27:19.463661 12113 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:19.637704  5710 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:19.649667  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:19.623848 12049 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:19.656862  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:19.661398  5451 catalog_manager.cc:5697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 10 to 13. New cstate: current_term: 13 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:19.816735  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:19.672772  6279 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:19.614035 12009 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:19.690407  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:19.842073  6275 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:27:19.534925 11909 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.700141  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:19.795217 12004 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:19.830273  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:19.855221  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:19.833880  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:27:19.668044  5706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2640) took 1368 ms (client timeout 1553 ms). Trace:
W20260709 12:27:19.856822  5706 rpcz_store.cc:269] 0709 12:27:18.300001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.341597 (+ 41596us) service_pool.cc:224] Handling call
0709 12:27:19.667842 (+1326245us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b54001991d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:19.667972 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1754542848}
I20260709 12:27:19.857630  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1160) took 2846 ms. Trace:
I20260709 12:27:19.857865  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:19.858511  5706 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:19.817369  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:19.840340  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:19.858166  5555 rpcz_store.cc:276] 0709 12:27:17.010961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:17.012615 (+  1654us) service_pool.cc:224] Handling call
0709 12:27:17.961628 (+949013us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:27:17.961811 (+   183us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:18.254786 (+292975us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:18.254841 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:18.254907 (+    66us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:18.254957 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.857087 (+1602130us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b540023ffd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:19.857160 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:19.857317 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":696310912}
I20260709 12:27:19.859877  6279 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 18: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:19.860498  5559 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:19.630349 11978 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.157s
W20260709 12:27:19.853521  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:19.844730  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 49023.3us
I20260709 12:27:19.874199  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:19.680060 11720 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:27:19.849896  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 200348us
I20260709 12:27:19.883384  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:19.884007  5554 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:27:19.704411  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:19.885166  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2743) took 1122 ms. Trace:
W20260709 12:27:19.851718  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:19.885505  5554 rpcz_store.cc:276] 0709 12:27:18.762352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.880310 (+1117958us) service_pool.cc:224] Handling call
0709 12:27:19.883718 (+  3408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.884911 (+  1193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.885109 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:19.699266  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:19.853479  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 153398us
I20260709 12:27:19.862479  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2742) took 1100 ms. Trace:
I20260709 12:27:19.864140  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:19.686741 11977 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:19.680729 11597 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:19.875499  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:19.848984  5703 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:27:19.820991 12136 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 12:27:19.817601 12133 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:19.699849  5552 rpcz_store.cc:276] 0709 12:27:17.872699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.207691 (+1334992us) service_pool.cc:224] Handling call
0709 12:27:19.231556 (+ 23865us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.292412 (+ 60856us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:19.292444 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:19.292485 (+    41us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:19.292539 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.698961 (+406422us) spinlock_profiling.cc:243] Waited 60.6 ms on lock 0x7b54000ec4d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:19.699036 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:19.699201 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":139357440}
I20260709 12:27:19.879686  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:19.881008  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:19.907084  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:19.908643  5555 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:27:19.888021  5561 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:19.886917  5559 rpcz_store.cc:276] 0709 12:27:18.762072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.861360 (+1099288us) service_pool.cc:224] Handling call
0709 12:27:19.861797 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.861983 (+   186us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:19.862007 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:19.862050 (+    43us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:19.862081 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.862130 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:19.862422 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:19.886986  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2745) took 1097 ms. Trace:
I20260709 12:27:19.885169  6271 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
W20260709 12:27:19.897424  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:19.901094  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 201899us
I20260709 12:27:19.696565  5553 rpcz_store.cc:276] 0709 12:27:17.872344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.205415 (+1333071us) service_pool.cc:224] Handling call
0709 12:27:19.205743 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.416978 (+211235us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.695948 (+278970us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54001be754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:19.696114 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":476657920}
I20260709 12:27:19.889871  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:19.908524  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:19.913276  5706 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:19.881150  5557 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:19.910683  5554 rpcz_store.cc:276] 0709 12:27:18.789264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.886301 (+1097037us) service_pool.cc:224] Handling call
0709 12:27:19.886554 (+   253us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.886708 (+   154us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:19.886737 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:19.886776 (+    39us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:19.886807 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.886853 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:19.886940 (+    87us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:19.917979  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:19.909891  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:19.920497  6271 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:19.843469 12039 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:19.841251 12009 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:19.873292 11991 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:19.686319 12132 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.684316 11721 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.927155  5703 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:19.929605  5557 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:19.854172 12004 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:19.880551 12120 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:19.862654 11804 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:19.911705  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:19.931895  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:19.952394  5706 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:19.953365  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:19.854303 11587 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 12:27:19.875854 12077 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:19.970099  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2648) took 1667 ms. Trace:
I20260709 12:27:19.893285 11977 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:19.846061 11715 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:19.901525 12101 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:19.905067 11720 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:19.947696 12004 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:19.915369 11976 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:27:19.942054 12083 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.519s	user 0.009s	sys 0.001s
W20260709 12:27:19.940217 11832 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.538s	user 0.010s	sys 0.000s
I20260709 12:27:19.956509  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:19.965247 11880 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:19.986310  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1093) took 2121 ms. Trace:
I20260709 12:27:19.965406  5451 catalog_manager.cc:5697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 12 to 15, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 15 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:19.967343  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:19.930467 12109 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:19.934759 12009 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:19.931882 11978 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:19.935060 11991 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:19.930948 12133 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:19.970521  5705 rpcz_store.cc:276] 0709 12:27:18.302751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.738242 (+435491us) service_pool.cc:224] Handling call
0709 12:27:18.738614 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:19.146265 (+407651us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.969808 (+823543us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b54002d0ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:19.970030 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":825310592}
I20260709 12:27:19.864259 12010 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:19.972373  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:19.976171  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 287680us
I20260709 12:27:20.001631  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:19.958096 11804 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.921384 11597 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.986721  5707 rpcz_store.cc:276] 0709 12:27:17.865017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:17.937672 (+ 72655us) service_pool.cc:224] Handling call
0709 12:27:17.938072 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:18.160896 (+222824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.986062 (+1825166us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b540033fc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:19.986250 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":91560832}
I20260709 12:27:19.968842 12120 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:19.945276 11999 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:27:19.988757 11880 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:19.975343 12112 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:19.974995 12077 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:19.945453 12008 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:27:19.975514 12101 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:19.923933 11723 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.013177 12069 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:20.005254  5705 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:19.980002 11976 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:20.042263 11976 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:20.042660 12120 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:19.987109 12083 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:20.010637 11597 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:20.049010 11597 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.049546 11597 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.050125 11914 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:20.050525 11914 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:20.051438 11914 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.052858 11597 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:20.015553  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:20.061713  5709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2647) took 1759 ms (client timeout 1901 ms). Trace:
W20260709 12:27:20.062074  5709 rpcz_store.cc:269] 0709 12:27:18.302377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.474859 (+172482us) service_pool.cc:224] Handling call
0709 12:27:20.061521 (+1586662us) spinlock_profiling.cc:243] Waited 803 ms on lock 0x7b5400066654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:20.061657 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1845839232}
I20260709 12:27:20.062727  5552 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:20.063113  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.063365  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2750) took 1113 ms. Trace:
I20260709 12:27:20.063680  5552 rpcz_store.cc:276] 0709 12:27:18.950295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.062251 (+1111956us) service_pool.cc:224] Handling call
0709 12:27:20.062559 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:20.063123 (+   564us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.063318 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:19.992805  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:20.064949  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72217.4us
I20260709 12:27:20.066303  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2651) took 1762 ms. Trace:
I20260709 12:27:20.066664  5708 rpcz_store.cc:276] 0709 12:27:18.303684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.759799 (+456115us) service_pool.cc:224] Handling call
0709 12:27:20.064432 (+1304633us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:20.064746 (+   314us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:20.064773 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:20.064811 (+    38us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:20.064845 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.064897 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:20.066146 (+  1249us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b5400066650. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:20.066246 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2999605120}
W20260709 12:27:20.068866 12143 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
I20260709 12:27:20.002594  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:20.072458  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 69933us
I20260709 12:27:20.064206  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:19.995507 12109 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:20.000458 11991 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:20.013025 12103 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:20.076865  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.079273  5710 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:20.003837 11978 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.083189  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:19.981395  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2735) took 2109 ms. Trace:
I20260709 12:27:20.017109 11977 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:20.028635 12121 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:20.012223 12010 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:20.074265 12109 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:20.033412 12133 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.037084 12132 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:20.007066 12039 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.025906 12101 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:20.039491 11721 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:20.027180 11720 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:20.040817 12069 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:20.024016 11715 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:20.120390 11715 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:20.122704  5561 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:27:20.017609 12112 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:20.125056 12112 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.126397 12112 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:20.015218  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.053020  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2731) took 2182 ms. Trace:
I20260709 12:27:20.053668  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1121) took 1132 ms. Trace:
I20260709 12:27:20.128984 11855 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:20.129254  6274 rpcz_store.cc:276] 0709 12:27:18.921769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.928644 (+  6875us) service_pool.cc:224] Handling call
0709 12:27:18.929086 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.420504 (+491418us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.053438 (+632934us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b54002e3954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:20.053611 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1119321472}
I20260709 12:27:20.056612  5559 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:20.131326  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2747) took 1284 ms. Trace:
I20260709 12:27:20.131916  5559 rpcz_store.cc:276] 0709 12:27:18.846854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.056094 (+1209240us) service_pool.cc:224] Handling call
0709 12:27:20.056439 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:20.131072 (+ 74633us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.131272 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:20.134079  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1167) took 2515 ms. Trace:
I20260709 12:27:20.134691  5556 rpcz_store.cc:276] 0709 12:27:17.618546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.123210 (+1504664us) service_pool.cc:224] Handling call
0709 12:27:19.123522 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.220435 (+ 96913us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.133832 (+913397us) spinlock_profiling.cc:243] Waited 34.2 ms on lock 0x7b54003a93d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:20.134014 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":78732544}
I20260709 12:27:20.057359 12004 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.136847  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:20.137797 12004 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:20.036221 11992 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.139029  5701 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:27:20.139899 11992 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:20.058451  5557 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 9. (term mismatch)
I20260709 12:27:20.140522  5557 pending_rounds.cc:85] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 8
I20260709 12:27:20.140808  5557 pending_rounds.cc:107] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260709 12:27:20.141037  5557 raft_consensus.cc:2889] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:20.141155 12108 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:20.141621 12108 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:20.059991  5555 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:27:20.142444  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2549) took 1246 ms. Trace:
I20260709 12:27:20.142819  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:20.142788  5555 rpcz_store.cc:276] 0709 12:27:18.896532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.059425 (+1162893us) service_pool.cc:224] Handling call
0709 12:27:20.059777 (+   352us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:20.142207 (+ 82430us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.142394 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:20.143694  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:20.073387  5709 raft_consensus.cc:2410] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:27:20.019586 11880 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:20.088260  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:20.145160  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:20.077033 12124 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.079227 11914 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:20.078091 12083 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:20.104741  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:20.086143  5560 rpcz_store.cc:276] 0709 12:27:17.871941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.137780 (+1265839us) service_pool.cc:224] Handling call
0709 12:27:19.138089 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.138277 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:27:19.142124 (+  3847us) log.cc:844] Serialized 14 byte log entry
0709 12:27:19.615875 (+473751us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b1800343ba0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:27:19.981040 (+365165us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:19.981094 (+    54us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:19.981141 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:19.981207 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:19.981331 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7338,"prepare.queue_time_us":1660,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":6784,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":7378,"spinlock_wait_cycles":9644800,"thread_start_us":360075,"threads_started":1}
I20260709 12:27:20.089185 11817 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:20.165361  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:20.113936  5628 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:20.129727 11855 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.141680  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2547) took 1246 ms. Trace:
I20260709 12:27:20.167425  5557 rpcz_store.cc:276] 0709 12:27:18.895867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.057657 (+1161790us) service_pool.cc:224] Handling call
0709 12:27:20.058121 (+   464us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:20.141470 (+ 83349us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.141629 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:20.168908  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:20.169709  5557 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:20.142573 12108 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:20.144299  5553 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:27:20.203508  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:20.086057  5552 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:20.204945  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2751) took 1249 ms. Trace:
I20260709 12:27:20.205369  5552 rpcz_store.cc:276] 0709 12:27:18.955645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.072050 (+1116405us) service_pool.cc:224] Handling call
0709 12:27:20.085881 (+ 13831us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:20.204625 (+118744us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.204875 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:20.083258  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.129200  5558 rpcz_store.cc:276] 0709 12:27:17.870596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.125489 (+1254893us) service_pool.cc:224] Handling call
0709 12:27:19.125789 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:19.163692 (+ 37903us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.052786 (+889094us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b54003a8ed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:20.052959 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47602944}
I20260709 12:27:20.143983  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:20.207417  5555 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:20.129029  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:20.145869  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2886) took 1001 ms. Trace:
I20260709 12:27:20.209334  5701 rpcz_store.cc:276] 0709 12:27:19.144043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.139833 (+995790us) service_pool.cc:224] Handling call
0709 12:27:20.140178 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:20.145536 (+  5358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:20.145562 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:20.145606 (+    44us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:20.145641 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.145709 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:20.145805 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:20.142020  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2635) took 1863 ms. Trace:
I20260709 12:27:20.150875  5556 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:27:20.089812  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2646) took 1787 ms. Trace:
I20260709 12:27:20.212493  5702 rpcz_store.cc:276] 0709 12:27:18.302048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.727553 (+425505us) service_pool.cc:224] Handling call
0709 12:27:18.727901 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:19.277597 (+549696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.089542 (+811945us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b54000e0bd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:20.089753 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1262624640}
I20260709 12:27:20.214689  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.147746  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:20.215896  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 50622us
W20260709 12:27:20.216815  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 69168.9us
I20260709 12:27:20.206516  5705 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:20.210879  5704 rpcz_store.cc:276] 0709 12:27:18.278985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:18.623908 (+344923us) service_pool.cc:224] Handling call
0709 12:27:20.141839 (+1517931us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b54001a9354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:20.141965 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1058071424}
I20260709 12:27:20.220783  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.157364  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:20.153503  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 65326.9us
I20260709 12:27:20.102463 11977 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:20.163182  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:20.232491  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.234369  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.235648  5703 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:20.085049 11774 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:27:20.084152 12100 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 12:27:20.238044 12100 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:20.238590 12100 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:20.239064 12100 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:20.171810  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:20.240883  5557 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:20.173484  6274 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:20.175685 11855 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:20.188127 11804 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:20.189602 11976 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:20.245317 11976 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:20.190022 11817 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:27:20.191295 11978 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:20.191638 11909 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:20.192471 12077 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:20.080302  5451 catalog_manager.cc:5697] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:20.249199  5451 catalog_manager.cc:5697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 11 to 14, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 14 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:27:20.251051  5560 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:20.250689  5451 catalog_manager.cc:5697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 10 to 12, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 12 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:27:20.106766 12084 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:20.193730 12069 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:20.252503  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.254204  5703 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:20.255442  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.226712  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.256237  5703 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:27:20.252228  5451 catalog_manager.cc:5697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 10 to 13. New cstate: current_term: 13 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:20.262405  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:20.157918 12121 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:20.267793  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:20.268684  5710 raft_consensus.cc:2376] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17. Re-sending same reply.
I20260709 12:27:20.269879  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.270650  5710 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
W20260709 12:27:20.272651  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:20.274039 11999 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:20.274569 11999 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:20.220044 12004 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:20.277441 12010 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:20.282490 11983 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:20.283104 11983 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:20.144436 12085 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.058s
I20260709 12:27:20.162213 11597 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:27:20.286888  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55080.4us
I20260709 12:27:20.290634  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.295797  5704 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:20.297117  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.298175  5704 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 23 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 12:27:20.291600  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.166296 11992 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:20.301558  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.301558  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:20.317991  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:20.331887  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.326583  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:20.335246  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.335973  5710 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:20.229918 11721 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:20.229254 12083 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:20.293108 11597 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:20.349037 11597 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:20.200140 12101 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.304296  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:20.342587  5552 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:27:20.343119  5707 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:20.356781  5702 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:20.290354 11833 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:20.324990 12108 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:20.360327  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.364162  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:20.366204  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62005.7us
I20260709 12:27:20.384473  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:20.381533  5602 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.015s	sys 0.022s
I20260709 12:27:20.326562 11992 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:20.202996 12124 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.384459  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:20.385408  5602 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.015s	sys 0.022s
W20260709 12:27:20.364812  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:20.285211 12008 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.071s
W20260709 12:27:20.390733  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 611722us
I20260709 12:27:20.328320 12004 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:20.391785 12004 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:20.393564  5555 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:20.396421 11976 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:20.396991 11976 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:20.293361 11832 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.070s
I20260709 12:27:20.390247  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:20.397471  5703 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
W20260709 12:27:20.398355  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:20.406491  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:20.430971  5702 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:27:20.300200 11804 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20260709 12:27:20.420264  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:20.381989 12131 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.697s	user 0.007s	sys 0.004s
W20260709 12:27:20.441872  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.009s	sys 0.025s
I20260709 12:27:20.453034  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1106) took 1191 ms. Trace:
W20260709 12:27:20.457345  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.010s	sys 0.026s
I20260709 12:27:20.361249 12085 consensus_queue.cc:1243] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 17.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20260709 12:27:20.458705  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 525836us
I20260709 12:27:20.452486  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:20.440126  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49973.5us
I20260709 12:27:20.457549  5702 rpcz_store.cc:276] 0709 12:27:19.261935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.398108 (+1136173us) service_pool.cc:224] Handling call
0709 12:27:20.398430 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:20.450409 (+ 51979us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.452963 (+  2554us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:20.438973  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:20.385720  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 338595us
I20260709 12:27:20.463442  5556 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:27:20.276094 11999 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.399811 11992 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:20.469556  5702 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: []
W20260709 12:27:20.472836  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:20.284215 11983 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.433574 12132 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:20.471042  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:20.474777  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:20.478044  5601 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:20.478266  6271 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.020s	sys 0.024s
W20260709 12:27:20.442307  5600 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.009s	sys 0.025s
W20260709 12:27:20.479835  5600 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.010s	sys 0.026s
W20260709 12:27:20.480208  6271 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.020s	sys 0.024s
I20260709 12:27:20.452009 11978 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.452090 11721 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:20.386006 11833 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:20.484134  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 399397us
I20260709 12:27:20.455170 12081 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.456144 12134 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:20.454746 12009 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.432868 12121 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.050s
I20260709 12:27:20.484376 11997 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.481302 12132 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:20.513543  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:20.465960 12131 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:20.430889 11976 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:20.511646  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:20.519937  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 354031us
I20260709 12:27:20.535899  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:20.506812 12098 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:20.538401  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:20.540234  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:20.541167  5556 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:20.545161  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:20.546118  5560 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:20.263530  5451 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 12 to 15, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 15 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:27:20.550041  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:20.553292  5708 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:20.563171  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:20.568048  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:20.511458 12134 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:20.605101  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 49895.2us
I20260709 12:27:20.605139  5701 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:20.606513  5553 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:20.609197  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:20.609750  5707 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:20.610229  5552 raft_consensus.cc:2368] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17.
I20260709 12:27:20.551226 12010 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:20.623171  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1099) took 1363 ms. Trace:
I20260709 12:27:20.620599  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:20.533990 12132 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:20.511902 11978 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:20.647603  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1184) took 1157 ms. Trace:
I20260709 12:27:20.648038  5554 rpcz_store.cc:276] 0709 12:27:19.489914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.129874 (+639960us) service_pool.cc:224] Handling call
0709 12:27:20.130359 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:20.130834 (+   475us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:20.130867 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:20.131743 (+   876us) log.cc:844] Serialized 26 byte log entry
0709 12:27:20.483592 (+351849us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:20.483677 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.483777 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:20.647369 (+163592us) raft_consensus.cc:1685] finished
0709 12:27:20.647392 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:20.647536 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6774,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"thread_start_us":351292,"threads_started":1,"wal-append.queue_time_us":510049}
I20260709 12:27:20.628885  5707 rpcz_store.cc:276] 0709 12:27:19.259516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.409373 (+1149857us) service_pool.cc:224] Handling call
0709 12:27:20.409795 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:20.610309 (+200514us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.622918 (+ 12609us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b5400210554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:20.623114 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":459323264}
I20260709 12:27:20.514261 12081 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:20.553006 12121 consensus_queue.cc:1243] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260709 12:27:20.637249 12156 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260709 12:27:20.507140 12083 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:20.637737 11991 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.556s	user 0.009s	sys 0.000s
I20260709 12:27:20.637506 12049 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:20.611814 12077 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:27:20.622128 11855 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:20.650713  5707 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:20.653811  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:20.655373  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:20.554493 12155 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:27:20.663655  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:20.607851 11587 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:27:20.665858  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 187838us
W20260709 12:27:20.638212  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 90803.8us
I20260709 12:27:20.683115  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:20.672432  5628 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:20.684474  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:20.684774  5628 raft_consensus.cc:606] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
W20260709 12:27:20.641705 12109 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.533s	user 0.007s	sys 0.001s
I20260709 12:27:20.612432 12134 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:20.676904  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:20.536908 11997 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.685843  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:20.693856  5630 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:20.695204  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:20.686197  5631 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
W20260709 12:27:20.667220  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 228313us
I20260709 12:27:20.686652  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:20.696519  5556 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:20.630079 12110 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:20.624099 12069 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:20.654429 11774 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:20.628482 12010 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:20.688320  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:20.605921 12100 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.709092  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:20.672740 12039 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:20.680953 11720 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.675915 12084 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:20.714707  5629 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
W20260709 12:27:20.745654  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 57400.3us
I20260709 12:27:20.746986  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:20.747772  5632 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:27:20.680718 12038 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:20.732746  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:20.732836  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:20.744731  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:20.623054 11715 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.060s
I20260709 12:27:20.714424  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:20.715221  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:20.756450  5628 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 17: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:20.756955  5633 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:20.763096  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:20.763154  5635 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:27:20.757051  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:20.766449  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 289505us
W20260709 12:27:20.757385  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 104338us
I20260709 12:27:20.712491 11587 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:20.710294 11774 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:20.672601 11991 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:20.706424 12101 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:20.717841 12155 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.788197  6271 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 20: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:27:20.788524  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 152995us
I20260709 12:27:20.766561  5632 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:20.716606 12049 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:20.793455  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:20.796116  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:20.835083  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44317.4us
I20260709 12:27:20.753480 12038 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:20.733247 12069 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:20.743489 12039 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:20.930506  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 139065us
I20260709 12:27:20.838573  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:20.752948 12153 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.704226 12109 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:20.711531 11723 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.076s
W20260709 12:27:20.789006 11909 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.542s	user 0.005s	sys 0.005s
I20260709 12:27:20.753212 12084 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:20.760116 12120 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:20.759716 12004 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:27:20.903513  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 220469us
I20260709 12:27:21.000330  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:20.913964  5636 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 17: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:20.838220 12161 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.007608  5627 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
W20260709 12:27:20.941874  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 236835us
W20260709 12:27:20.943849  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:20.900909 12174 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:20.965602  5633 raft_consensus.cc:2410] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:27:20.942448 12039 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:21.022240  5709 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:21.005686  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:20.846467 11774 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:21.009727  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:21.009907  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:21.013864  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.013440  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:20.763414 11720 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.016664  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.016759  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:20.796846 12173 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.222s
W20260709 12:27:21.024793  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:20.897502 11804 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.105s
I20260709 12:27:21.026490  5704 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:21.076012  5633 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
W20260709 12:27:21.074311  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 64476.5us
I20260709 12:27:21.078888  5631 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 18: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:21.090174  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:21.092885  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 90153.3us
W20260709 12:27:21.103341  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 93665.5us
W20260709 12:27:21.122236  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:21.125054  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 219499us
I20260709 12:27:21.126075  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:21.011819 12152 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.999416 12120 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:21.126708  5636 raft_consensus.cc:2376] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17. Re-sending same reply.
I20260709 12:27:21.137342  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:20.917105 12010 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:21.018049 11833 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:21.143569  5706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2893) took 1618 ms (client timeout 1981 ms). Trace:
W20260709 12:27:21.150944  5706 rpcz_store.cc:269] 0709 12:27:19.524869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:19.953084 (+428215us) service_pool.cc:224] Handling call
0709 12:27:21.143324 (+1190240us) spinlock_profiling.cc:243] Waited 703 ms on lock 0x7b54002d19d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:21.143501 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1615801088}
I20260709 12:27:21.026010 12004 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:20.763017  5451 catalog_manager.cc:5697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 9 to 16. New cstate: current_term: 16 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:21.025429 12009 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:21.175105  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:21.176647 12168 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.331s	user 0.001s	sys 0.000s
W20260709 12:27:21.177127 12159 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.508s	user 0.001s	sys 0.001s
I20260709 12:27:21.177793 12084 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.180984  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.182749  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:21.183786  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.184741  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:21.189261  5451 catalog_manager.cc:5697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 15. New cstate: current_term: 15 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:21.190366  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.199950  5627 raft_consensus.cc:2376] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19. Re-sending same reply.
I20260709 12:27:21.188805 11587 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.139453 12131 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:21.211284  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:21.212767  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 86786.7us
W20260709 12:27:21.216939  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 216722us
I20260709 12:27:21.219336  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:21.228216  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:21.229483  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:21.127491 12077 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.237129  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:21.008910 12172 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.207s
I20260709 12:27:20.971717 12153 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.144590 12120 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:20.928202 12155 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.256557  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:21.213454 12081 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:21.259382  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:21.303257  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:21.304535  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:21.305177  5553 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:21.306079  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:21.307161  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:21.260265  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:21.308023  5560 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:21.308931  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:21.260202  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:21.263536  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:21.243794 11978 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:21.311383 12173 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:21.312179 11978 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.312875 11978 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:21.278983  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:21.030714 12188 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.217s
I20260709 12:27:21.193979 11663 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:27:21.317644  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:21.318252 11663 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:21.318919 11663 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.319392 11663 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.214468 12049 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:21.320518 12049 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:21.321383 12049 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:21.283524 11804 consensus_queue.cc:1243] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:27:21.025970 12174 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:21.358163  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.003s	sys 0.009s
W20260709 12:27:21.361066  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.004s	sys 0.009s
W20260709 12:27:21.361528  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 650551us
I20260709 12:27:21.258077  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:21.299016  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:21.019088 12161 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.246061 12113 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:27:21.361616  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 98170.4us
I20260709 12:27:21.027848 12038 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:21.360769  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 66868.6us
I20260709 12:27:21.364847  5557 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:21.379691  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:21.148782 12152 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:21.278087 12010 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:21.264564 12077 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:21.367980 12181 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:27:21.402802  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 274215us
I20260709 12:27:21.378886 12113 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:21.403340 12181 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:21.403630 12113 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:21.403882 12181 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:21.409584  5630 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:21.298853 12021 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:27:21.410004  5524 leader_election.cc:336] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.547s (SENT)
I20260709 12:27:21.149348 12069 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:21.385567 12049 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:21.322645 12191 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:21.370177 11997 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:21.467031  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 371410us
I20260709 12:27:21.435205  5631 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:21.438367 12174 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:21.415637 12004 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:21.439245 12185 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.193s	user 0.000s	sys 0.000s
I20260709 12:27:21.440524 12110 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 12:27:21.441462 12154 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.888s	user 0.004s	sys 0.000s
I20260709 12:27:21.441866 11855 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:21.442018 12109 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:21.442723 12164 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.536s	user 0.000s	sys 0.001s
I20260709 12:27:21.443375 12155 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:21.445606 12188 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 12:27:21.446949 12167 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.499s	user 0.000s	sys 0.001s
W20260709 12:27:21.447221 12163 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.458s	user 0.000s	sys 0.001s
I20260709 12:27:21.432022 12175 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:27:21.451330 11991 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:21.451228  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:21.452328 12166 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:27:21.452600  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.454061  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2760) took 1556 ms. Trace:
I20260709 12:27:21.426738 11723 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:27:21.410496 12173 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:21.517347 11721 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:21.451668 12181 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.462824  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:21.461961  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52118.4us
I20260709 12:27:21.465236  5630 raft_consensus.cc:2410] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
W20260709 12:27:21.584029  5524 leader_election.cc:336] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.679s (SENT)
I20260709 12:27:21.525734 11983 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:21.468304  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:21.475031  5675 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:21.597805  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 122846us
I20260709 12:27:21.441946 12182 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:21.366513 11978 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:21.441419 11663 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:21.458639 12199 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:21.608318 12199 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:21.608412 12098 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:21.609341 12098 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:21.609834 12199 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.602368 12204 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.611568 12199 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:21.611382 12204 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:21.613319 12153 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:21.614797 12204 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:21.458626 12085 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:21.616541 12085 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:21.617722 12152 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:21.617798 12085 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:21.618129 12098 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.619155 12085 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:21.473591 12151 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.104s	user 0.002s	sys 0.000s
I20260709 12:27:21.358510  5451 catalog_manager.cc:5697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:21.498041  5634 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.730s	user 0.000s	sys 0.009s
W20260709 12:27:21.636590  5634 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.874s	user 0.000s	sys 0.009s
W20260709 12:27:21.638254  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 170025us
I20260709 12:27:21.639228  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:21.506047  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1110) took 1434 ms. Trace:
I20260709 12:27:21.640729 12172 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:27:21.493872  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:21.641412 12172 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:21.641892 12172 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:27:21.642500  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:21.643843  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:21.467060 12100 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:21.514909  5631 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:21.644608  6272 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 19: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:21.640534 12155 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:27:21.625438 12167 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:27:21.474546 12191 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:21.640532  5710 rpcz_store.cc:276] 0709 12:27:20.071210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.742958 (+671748us) service_pool.cc:224] Handling call
0709 12:27:20.768788 (+ 25830us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:20.838657 (+ 69869us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:27:21.073707 (+235050us) log.cc:844] Serialized 14 byte log entry
0709 12:27:21.505117 (+431410us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:21.505373 (+   256us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:21.505511 (+   138us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:21.505847 (+   336us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:21.505983 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":184647,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":65,"raft.queue_time_us":106,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":8682,"thread_start_us":589758,"threads_started":2}
W20260709 12:27:21.486747 12179 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.497s	user 0.000s	sys 0.001s
I20260709 12:27:21.647092  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1183) took 2157 ms. Trace:
I20260709 12:27:21.460623 11909 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:27:21.647758  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:21.560325  5627 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:21.502928 12133 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.648744 12133 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.649181  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.650594  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.650838 12133 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:21.566359  5559 rpcz_store.cc:276] 0709 12:27:19.897949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.135380 (+237431us) service_pool.cc:224] Handling call
0709 12:27:20.145267 (+  9887us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:20.147660 (+  2393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:20.147692 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:20.164892 (+ 17200us) log.cc:844] Serialized 52 byte log entry
0709 12:27:20.516563 (+351671us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:20.548803 (+ 32240us) log.cc:844] Serialized 14 byte log entry
0709 12:27:20.671195 (+122392us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.671324 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:21.453853 (+782529us) raft_consensus.cc:1685] finished
0709 12:27:21.453871 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:21.454005 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2069,"prepare.queue_time_us":31647,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":14158,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":28179,"thread_start_us":231276,"threads_started":1,"wal-append.queue_time_us":441922}
W20260709 12:27:21.528406 11880 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.364s	user 0.011s	sys 0.003s
I20260709 12:27:21.652287 11880 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:21.408231 12161 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:21.532255 12108 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.132s	user 0.010s	sys 0.004s
I20260709 12:27:21.515365 12021 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:21.584535  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:21.654412  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:21.465858  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 50069.9us
I20260709 12:27:21.588706 12195 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:21.590322 12181 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:21.539655 12183 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.572680 11720 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:21.601270 11663 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:21.607846  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2884) took 2464 ms. Trace:
W20260709 12:27:21.620757  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:21.629730 12098 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:21.630816  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1109) took 1559 ms. Trace:
I20260709 12:27:21.602660 12182 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:21.645639  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:21.645442 12077 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:21.571686  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.646909 12191 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:21.654598  5710 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:21.656685  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:21.647492  5561 rpcz_store.cc:276] 0709 12:27:19.489657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.123741 (+634084us) service_pool.cc:224] Handling call
0709 12:27:20.124141 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:20.154470 (+ 30329us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:27:20.212537 (+ 58067us) log.cc:844] Serialized 14 byte log entry
0709 12:27:20.516775 (+304238us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:20.669676 (+152901us) log.cc:844] Serialized 26 byte log entry
0709 12:27:20.670461 (+   785us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:20.670564 (+   103us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:20.670716 (+   152us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:21.646876 (+976160us) raft_consensus.cc:1685] finished
0709 12:27:21.646901 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:21.647034 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42271,"prepare.queue_time_us":57729,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":69,"raft.queue_time_us":8630,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":9453,"thread_start_us":303155,"threads_started":1,"wal-append.queue_time_us":530518}
I20260709 12:27:21.660530  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:21.690209  5554 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:21.660838  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:21.662032  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:21.651939  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.677592 12183 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:21.666100  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 82111.9us
W20260709 12:27:21.693554  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:21.693890  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:21.694242  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:21.603726  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:21.667989  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:21.625212 12131 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:21.594798 11721 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:21.671793  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:21.709127  5557 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:21.665128 12195 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:21.709946  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:21.655644 11992 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.170s	user 0.013s	sys 0.000s
I20260709 12:27:21.714361  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:21.684466  6274 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:21.499666 11855 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:21.679478  5707 rpcz_store.cc:276] 0709 12:27:20.070926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.686900 (+615974us) service_pool.cc:224] Handling call
0709 12:27:20.687213 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:20.704036 (+ 16823us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:20.704072 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:20.716148 (+ 12076us) log.cc:844] Serialized 26 byte log entry
0709 12:27:21.267504 (+551356us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:21.267635 (+   131us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:21.267841 (+   206us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:21.630585 (+362744us) raft_consensus.cc:1685] finished
0709 12:27:21.630611 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:21.630754 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3257,"raft.queue_time_us":3540,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":12751,"thread_start_us":498896,"threads_started":1,"wal-append.queue_time_us":885562}
I20260709 12:27:21.618149 12174 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:21.670732  5708 rpcz_store.cc:276] 0709 12:27:19.143448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.067706 (+924258us) service_pool.cc:224] Handling call
0709 12:27:20.067987 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:20.554708 (+486721us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:21.607563 (+1052855us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b540006f754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:21.607773 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1115681024}
I20260709 12:27:21.714596  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:21.692260  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.696914  5703 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:21.697314  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:21.702804  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 102441us
I20260709 12:27:21.710907  5636 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:21.670524 11663 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:21.685928 12181 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:21.742393  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:21.748641  5708 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:21.749646  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2676) took 1599 ms. Trace:
I20260709 12:27:21.748116  5554 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:21.750185  5708 rpcz_store.cc:276] 0709 12:27:20.149764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:21.747935 (+1598171us) service_pool.cc:224] Handling call
0709 12:27:21.748265 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:21.749297 (+  1032us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:21.749570 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:21.751164  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:21.751842  5708 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:21.754154  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.753055  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.744671 11663 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:21.743481 12174 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:21.797060 12182 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:21.725113 11992 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:21.801924 11597 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.452s	user 0.010s	sys 0.003s
I20260709 12:27:21.744284 11880 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:21.802533 12113 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:21.714182 12195 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:21.794945 12205 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:27:21.804436  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 107220us
I20260709 12:27:21.755273  5560 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:21.677845 12021 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.810249  5627 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:21.803092 12100 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:21.836635 12100 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:21.837572 12100 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.838665  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1248) took 1066 ms. Trace:
I20260709 12:27:21.755995 12131 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:21.815943  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2682) took 1550 ms. Trace:
I20260709 12:27:21.840590 12131 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:21.840632  5708 rpcz_store.cc:276] 0709 12:27:20.265636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:21.751277 (+1485641us) service_pool.cc:224] Handling call
0709 12:27:21.751568 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:21.813961 (+ 62393us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:21.815881 (+  1920us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:21.841279 12131 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:21.842108  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:21.816845 12120 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:21.845396  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:21.846177 11880 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:27:21.819725 12204 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.818856  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1231) took 1146 ms. Trace:
I20260709 12:27:21.824146  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:21.824185  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:21.825711  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:21.826397  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:21.814002 12132 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.147s	user 0.004s	sys 0.009s
I20260709 12:27:21.713208 12077 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.107s
W20260709 12:27:21.818858 11976 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.266s	user 0.011s	sys 0.008s
I20260709 12:27:21.834328 12192 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:21.834597 12202 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:21.835186 12189 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.835506 11992 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:21.839046  5635 rpcz_store.cc:276] 0709 12:27:20.772036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.791013 (+ 18977us) service_pool.cc:224] Handling call
0709 12:27:20.791375 (+   362us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:20.842799 (+ 51424us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:20.842841 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:20.971654 (+128813us) log.cc:844] Serialized 52 byte log entry
0709 12:27:21.225575 (+253921us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:21.401699 (+176124us) log.cc:844] Serialized 14 byte log entry
0709 12:27:21.401972 (+   273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:21.402248 (+   276us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:21.838449 (+436201us) raft_consensus.cc:1685] finished
0709 12:27:21.838468 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:21.838598 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":154879,"prepare.queue_time_us":451667,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":374556,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":15319,"thread_start_us":256912,"threads_started":2,"wal-append.queue_time_us":712053}
I20260709 12:27:21.886173  5553 raft_consensus.cc:2368] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17.
I20260709 12:27:21.869313  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.888633  6272 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:21.873567  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:21.871264  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:21.883211  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:21.887975  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.852887  5634 rpcz_store.cc:276] 0709 12:27:20.672551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.672671 (+   120us) service_pool.cc:224] Handling call
0709 12:27:20.673006 (+   335us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:20.673740 (+   734us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:20.673790 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:20.691320 (+ 17530us) log.cc:844] Serialized 52 byte log entry
0709 12:27:21.637188 (+945868us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:21.752195 (+115007us) log.cc:844] Serialized 14 byte log entry
0709 12:27:21.818129 (+ 65934us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:21.818277 (+   148us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:21.818375 (+    98us) raft_consensus.cc:1685] finished
0709 12:27:21.818393 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:21.818759 (+   366us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70019,"prepare.queue_time_us":40327,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":98444,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":29929,"spinlock_wait_cycles":23936,"thread_start_us":989123,"threads_started":2,"wal-append.queue_time_us":1061294}
I20260709 12:27:21.889325  6273 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:21.871672  6277 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:21.890863  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:21.898451  5602 leader_election.cc:400] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17.
I20260709 12:27:21.899258  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:21.900283  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:21.903355  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:21.906941  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:21.907460  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:21.908286  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:21.910969  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.891204  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.913471  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:21.913645  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:21.914544  5634 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:21.860605 12008 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.921624  5627 raft_consensus.cc:2410] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:27:21.923261  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:21.893639 11976 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.869050 11909 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:21.880052 12205 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:21.828805 11597 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:21.848204 12204 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:21.882403 11880 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.081s
I20260709 12:27:21.919637 12155 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:27:22.027479  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:21.927060 12109 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:21.980630 12180 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:21.984537 12193 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
W20260709 12:27:21.985810 12207 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260709 12:27:21.986197 12201 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:27:21.992252 11715 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:21.995782 12038 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:21.896440 11720 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.901501 12108 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:21.906045 11978 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:27:21.905850 12152 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:22.013886  5627 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:22.063302 12193 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:22.063036 12205 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:21.892300 12069 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:27:22.053435 11914 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.851s	user 0.007s	sys 0.007s
W20260709 12:27:22.056674 12134 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.339s	user 0.011s	sys 0.004s
I20260709 12:27:21.889027 12132 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:22.060392  5675 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.970s (SENT) [suppressed 46 similar messages]
I20260709 12:27:22.062418  6274 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:21.894742 12211 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.119s
W20260709 12:27:22.064507  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 141349us
I20260709 12:27:22.207662  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:21.911820 12181 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.137s
I20260709 12:27:21.983731 12172 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:22.048031 12010 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.612s	user 0.012s	sys 0.000s
I20260709 12:27:22.071472  5629 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 18: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:22.011766 12202 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:22.073083  5707 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 10. (term mismatch)
I20260709 12:27:22.214037  5707 pending_rounds.cc:85] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 9
I20260709 12:27:22.214416  5707 pending_rounds.cc:107] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260709 12:27:22.214632  5707 raft_consensus.cc:2889] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:22.079363  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2809) took 1068 ms. Trace:
I20260709 12:27:22.215348  5627 rpcz_store.cc:276] 0709 12:27:21.010852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:21.997522 (+986670us) service_pool.cc:224] Handling call
0709 12:27:22.013642 (+ 16120us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:22.078960 (+ 65318us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:22.079291 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:22.216027 12204 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:22.059811 12155 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:22.333293 12155 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:21.831916 12110 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.074s
I20260709 12:27:21.907684 12174 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:21.962955 12008 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:22.069804 11715 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:21.920153 12189 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:22.083350 11774 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.981s	user 0.008s	sys 0.008s
W20260709 12:27:22.198287  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:22.159847  5705 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
W20260709 12:27:22.381392  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:22.382102  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:22.170674  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:22.159225  6279 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:22.079671 12183 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:22.080627 12193 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:22.080004 11720 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:22.124396  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:22.203029  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 142692us
I20260709 12:27:22.205400  5702 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
W20260709 12:27:22.403004  5675 leader_election.cc:336] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.747s (SENT)
I20260709 12:27:22.405079  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:22.173312 11597 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:22.161155 11855 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:22.219425 12172 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:22.220019  5636 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:27:22.221658 11983 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.601s	user 0.013s	sys 0.000s
I20260709 12:27:22.439304  6276 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:27:22.439765 11855 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:22.223155 12113 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:22.440537 11720 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:22.020308  5634 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:27:22.441052 12161 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:22.441382  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2808) took 1437 ms. Trace:
I20260709 12:27:22.441530 12161 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:22.442039  5705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2668) took 2294 ms (client timeout 1679 ms). Trace:
I20260709 12:27:22.442348  5555 raft_consensus.cc:2368] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18.
W20260709 12:27:22.442435  5705 rpcz_store.cc:269] 0709 12:27:20.147978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.219636 (+ 71658us) service_pool.cc:224] Handling call
0709 12:27:22.441879 (+2222243us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b5400067054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:22.441979 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4080445952}
I20260709 12:27:22.441769  5634 rpcz_store.cc:276] 0709 12:27:21.004092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:21.961585 (+957493us) service_pool.cc:224] Handling call
0709 12:27:22.019773 (+ 58188us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:22.441135 (+421362us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:22.441327 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:22.443749  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:22.084470 12131 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:22.223368 12049 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.728s	user 0.007s	sys 0.004s
I20260709 12:27:22.228757 12158 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:27:22.228634 12216 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.135s	user 0.000s	sys 0.001s
I20260709 12:27:22.231281  5627 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:27:22.450022  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2810) took 1339 ms. Trace:
I20260709 12:27:22.450448  5627 rpcz_store.cc:276] 0709 12:27:21.110873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.216206 (+1105333us) service_pool.cc:224] Handling call
0709 12:27:22.230914 (+ 14708us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:22.449697 (+218783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:22.449952 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:22.233348 11663 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:22.236480 12109 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:22.454900  5557 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:27:22.455016  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.455047  5559 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:27:22.247155 12124 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:22.252440 12202 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:22.258049 11914 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:27:22.266961 12215 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.210s	user 0.003s	sys 0.000s
I20260709 12:27:22.269323  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2571) took 1007 ms. Trace:
I20260709 12:27:22.461942  5553 rpcz_store.cc:276] 0709 12:27:21.261896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:21.306933 (+ 45037us) service_pool.cc:224] Handling call
0709 12:27:22.269018 (+962085us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b54003a8ed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:22.269233 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1322962304}
I20260709 12:27:22.275777  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:22.463757  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 188131us
W20260709 12:27:22.281188  5630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 1242) took 1583 ms (client timeout 1911 ms). Trace:
I20260709 12:27:22.466529  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:22.466450  5630 rpcz_store.cc:269] 0709 12:27:20.697737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.708778 (+ 11041us) service_pool.cc:224] Handling call
0709 12:27:22.280920 (+1572142us) spinlock_profiling.cc:243] Waited 675 ms on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:22.281105 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1551971200}
W20260709 12:27:22.282250  5704 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2680) took 2017 ms (client timeout 2184 ms). Trace:
I20260709 12:27:22.283221  5628 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
W20260709 12:27:22.284746 12170 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.335s	user 0.003s	sys 0.000s
I20260709 12:27:22.285590 11999 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:22.201608 12152 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:22.288456  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2578) took 1296 ms. Trace:
I20260709 12:27:22.201371 11978 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:22.472020  5552 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:27:22.291174  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2577) took 1299 ms (client timeout 1506 ms). Trace:
I20260709 12:27:22.292924 12120 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:22.293610  5709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2661) took 2147 ms (client timeout 1547 ms). Trace:
W20260709 12:27:22.294773 12191 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.582s	user 0.009s	sys 0.000s
I20260709 12:27:22.295125 12223 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:22.482422 12191 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:22.485237  6276 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:27:22.488560 12152 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:22.160569 12134 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:22.302971  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:22.295259  5706 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:27:22.310019 12112 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:22.492316 12112 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:22.492604  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:22.493289  5708 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:22.494375  5701 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:22.303953 12022 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:22.319216  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.497256  5635 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:22.497876  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:22.498395  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.498652  5556 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:27:22.327821  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:22.331651 12084 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:27:22.342561  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:22.360796  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1250) took 1457 ms. Trace:
I20260709 12:27:22.502146  5632 rpcz_store.cc:276] 0709 12:27:20.903453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.903648 (+   195us) service_pool.cc:224] Handling call
0709 12:27:20.971317 (+ 67669us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:21.424471 (+453154us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:21.424501 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:21.424546 (+    45us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:21.424583 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:22.360530 (+935947us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b5400240754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:22.360587 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:22.360733 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1041592192}
I20260709 12:27:22.360107 12174 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:22.384532  6271 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1135) took 1850 ms (client timeout 2010 ms). Trace:
W20260709 12:27:22.510814  6271 rpcz_store.cc:269] 0709 12:27:20.534437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.538084 (+  3647us) service_pool.cc:224] Handling call
0709 12:27:22.384227 (+1846143us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b54005910d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:22.384454 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":572306176}
I20260709 12:27:22.385332  6280 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
W20260709 12:27:22.397763  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:22.396606 12180 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:22.398979 11774 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:22.408887 12220 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
W20260709 12:27:22.361605 11721 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.637s	user 0.004s	sys 0.007s
I20260709 12:27:22.411064 12189 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:22.397151 12004 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:22.420269 12132 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:22.422049 12008 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:22.369979 11715 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:22.443831  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:22.229947 12039 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.151s	user 0.016s	sys 0.000s
W20260709 12:27:22.448628  5675 leader_election.cc:336] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.784s (SENT)
W20260709 12:27:22.530695  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 126335us
W20260709 12:27:22.532765  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:22.452593  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47576.1us
I20260709 12:27:22.458474  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:22.458653  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:22.460247  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:22.536057  5559 raft_consensus.cc:2368] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16.
I20260709 12:27:22.536146  5705 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:22.468536  5628 pending_rounds.cc:85] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 13
I20260709 12:27:22.536586  5628 pending_rounds.cc:107] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:27:22.536860  5628 raft_consensus.cc:2889] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:27:22.468170  5704 rpcz_store.cc:269] 0709 12:27:20.265046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.331620 (+ 66574us) service_pool.cc:224] Handling call
0709 12:27:22.281992 (+1950372us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b54001982d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:22.282181 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1572624256}
I20260709 12:27:22.538071  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.538146  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:22.538815  5704 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:22.471024  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:22.537446  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 231574us
I20260709 12:27:22.541410  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2896) took 2629 ms. Trace:
I20260709 12:27:22.541772  5702 rpcz_store.cc:276] 0709 12:27:19.912306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.473061 (+560755us) service_pool.cc:224] Handling call
0709 12:27:20.473410 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:22.400767 (+1927357us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:22.541198 (+140431us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b5400323f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:22.541357 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3982690048}
I20260709 12:27:22.290179  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2891) took 2829 ms. Trace:
I20260709 12:27:22.542689  5703 rpcz_store.cc:276] 0709 12:27:19.461242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.436568 (+975326us) service_pool.cc:224] Handling call
0709 12:27:20.437140 (+   572us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:21.740727 (+1303587us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:22.289944 (+549217us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54002d14d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:22.290123 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2896836480}
I20260709 12:27:22.472652  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:22.543828  6273 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
W20260709 12:27:22.474844  5709 rpcz_store.cc:269] 0709 12:27:20.146357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.157110 (+ 10753us) service_pool.cc:224] Handling call
0709 12:27:22.293427 (+2136317us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b54002f5654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:22.293530 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1792521088}
I20260709 12:27:22.475857  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.545061  5630 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
W20260709 12:27:22.474583  5629 rpcz_store.cc:269] 0709 12:27:20.991363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.991504 (+   141us) service_pool.cc:224] Handling call
0709 12:27:22.290996 (+1299492us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b540018b854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:22.291117 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2289983744}
I20260709 12:27:22.472218  5631 rpcz_store.cc:276] 0709 12:27:20.991692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:21.127379 (+135687us) service_pool.cc:224] Handling call
0709 12:27:22.288276 (+1160897us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b5400438354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:22.288397 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":556438784}
I20260709 12:27:22.546535  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.547215  5631 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 18: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:22.477896  6274 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: []
W20260709 12:27:22.452044 12195 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.611s	user 0.005s	sys 0.006s
I20260709 12:27:22.480930  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:22.481132 11978 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:22.488869  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:22.317691  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2675) took 2168 ms. Trace:
I20260709 12:27:22.556541  5707 rpcz_store.cc:276] 0709 12:27:20.149552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:21.744965 (+1595413us) service_pool.cc:224] Handling call
0709 12:27:21.745269 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:22.215067 (+469798us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:22.317391 (+102324us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b5400210f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:22.317622 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":752928000}
I20260709 12:27:22.480149  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:22.506879  5526 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.717s (SENT)
I20260709 12:27:22.558432  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:22.509636  5635 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:22.457528 11983 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:27:22.520754  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:22.526793  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.011s	sys 0.059s
W20260709 12:27:22.575021  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.011s	sys 0.060s
I20260709 12:27:22.575434  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.473240 12188 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:22.473393 11914 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:22.549464  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:22.518481 12202 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:22.559634  5558 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:22.563647  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:22.567512  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:22.599285  5557 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:22.569049 12189 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:27:22.573853  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:22.600726  5709 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 21 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:27:22.601833  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:22.604447  5709 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:22.575775  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 667516us
I20260709 12:27:22.576133  5630 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:22.578106  5704 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:22.525375 12022 consensus_queue.cc:1243] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 16.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:27:22.491659 12152 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:22.584071  5559 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:22.491684 12134 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:22.586617 12162 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.478s	user 0.001s	sys 0.000s
I20260709 12:27:22.586491 11909 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:22.586859  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:22.503087 12112 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:22.587525  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:22.589226  5676 leader_election.cc:400] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16.
I20260709 12:27:22.589637 12085 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:27:22.590325 12009 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:27:22.527244 11976 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:22.593565  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:22.526243 12131 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:22.602625  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:22.672851  5629 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:22.576762  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:22.527135 12180 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:22.573683  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 66847us
I20260709 12:27:22.611171  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:22.538991 11720 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:22.625582  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:22.678758  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 67801.7us
I20260709 12:27:22.574241 11983 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:22.657469  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2664) took 2510 ms. Trace:
I20260709 12:27:22.663509  5676 leader_election.cc:403] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 12:27:22.686694 12180 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:22.690824  5706 rpcz_store.cc:276] 0709 12:27:20.147042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:21.223071 (+1076029us) service_pool.cc:224] Handling call
0709 12:27:22.294890 (+1071819us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:22.492206 (+197316us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:22.657168 (+164962us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54002f5650. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:22.657381 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2464189312}
W20260709 12:27:22.692461  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 103393us
I20260709 12:27:22.529877 12004 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:22.704154  5559 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:22.732973  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:22.665160  5709 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:22.579589 12158 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:27:22.666501  5555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2557) took 2116 ms (client timeout 2319 ms). Trace:
I20260709 12:27:22.594390 12182 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:22.670794  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.530043 11721 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:22.747186  5705 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:22.748055 11721 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:22.677855 12112 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:22.748605  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:22.608784 12202 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:22.750231 12205 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.638s	user 0.005s	sys 0.006s
I20260709 12:27:22.751195 12172 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:27:22.750666 12205 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:22.674582  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2672) took 2525 ms. Trace:
I20260709 12:27:22.752573  6277 raft_consensus.cc:2368] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19.
I20260709 12:27:22.753245  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:22.608186 12188 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:22.754617  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.754245 12188 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:22.755882  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2834) took 1035 ms. Trace:
I20260709 12:27:22.623809 12109 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.058s
I20260709 12:27:22.685273 11976 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:22.757763  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1152) took 1651 ms. Trace:
I20260709 12:27:22.768090  6280 rpcz_store.cc:276] 0709 12:27:21.106150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:21.106345 (+   195us) service_pool.cc:224] Handling call
0709 12:27:22.385064 (+1278719us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:22.513004 (+127940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:22.757584 (+244580us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54005910d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:22.757707 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2939773312}
I20260709 12:27:22.758352  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2898) took 2755 ms. Trace:
I20260709 12:27:22.769769  5701 rpcz_store.cc:276] 0709 12:27:20.002744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.678406 (+675662us) service_pool.cc:224] Handling call
0709 12:27:20.711088 (+ 32682us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:22.502422 (+1791334us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:22.758105 (+255683us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54003e6f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:22.758298 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4100639104}
I20260709 12:27:22.771166  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:22.771986  5701 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:22.689577  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:22.580921 12008 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:22.562489 12039 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:22.735586  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:22.576952 12195 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:22.667330 12152 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:22.664459 12161 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:22.740198  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:22.752043  5710 rpcz_store.cc:276] 0709 12:27:20.148885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:21.725943 (+1577058us) service_pool.cc:224] Handling call
0709 12:27:21.749576 (+ 23633us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:21.749920 (+   344us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:21.749951 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:21.757575 (+  7624us) log.cc:844] Serialized 26 byte log entry
0709 12:27:22.004629 (+247054us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:22.004687 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:22.004788 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:22.674308 (+669520us) raft_consensus.cc:1685] finished
0709 12:27:22.674338 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:22.674501 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5050,"raft.queue_time_us":149,"raft.run_cpu_time_us":6158,"raft.run_wall_time_us":54261,"thread_start_us":144385,"threads_started":1,"wal-append.queue_time_us":342387}
I20260709 12:27:22.810539  5558 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:22.756696 12132 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:27:22.741037  5555 rpcz_store.cc:269] 0709 12:27:20.543322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.553216 (+  9894us) service_pool.cc:224] Handling call
0709 12:27:21.462705 (+909489us) spinlock_profiling.cc:243] Waited 3.2 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754deee 000000000036eb39 00007f4fff5c11e0 00007f4fff68a0c4 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:27:22.659021 (+1196316us) spinlock_profiling.cc:243] Waited 859 ms on lock 0x7b54002279d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:22.659211 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1982588160}
I20260709 12:27:22.742231  5629 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:22.744374  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:22.744622  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1247) took 1981 ms. Trace:
I20260709 12:27:22.737035 12223 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:22.745841  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:22.832976  5632 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:22.823654  5628 rpcz_store.cc:276] 0709 12:27:20.763392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:20.763595 (+   203us) service_pool.cc:224] Handling call
0709 12:27:22.282705 (+1519110us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:22.537340 (+254635us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:22.744437 (+207097us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b54001dddd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:22.744570 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3492916608}
I20260709 12:27:22.751578 12172 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:22.751794 12202 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:22.755185  5631 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:22.605839  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:22.756317  5636 rpcz_store.cc:276] 0709 12:27:21.720383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:21.747684 (+ 27301us) service_pool.cc:224] Handling call
0709 12:27:22.755717 (+1008033us) spinlock_profiling.cc:243] Waited 410 ms on lock 0x7b54001f2a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:22.755830 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":941963648}
W20260709 12:27:22.682802 12038 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.606s	user 0.006s	sys 0.003s
I20260709 12:27:22.679672  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:22.627218  5556 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
W20260709 12:27:22.770351  6236 connection.cc:256] RPC call timeout handler was delayed by -1.6364864639999723s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21920000000000003s for a response
I20260709 12:27:22.759677  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.758641  5708 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:22.775797  5707 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:22.773227  5553 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:22.702143 11983 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:22.844259  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:22.693769 12100 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:22.781955  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:22.845052  5707 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:22.845798  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:22.846441  5709 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:22.700412 12009 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:22.849284  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:22.846962  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1119) took 1026 ms. Trace:
I20260709 12:27:22.836941  5554 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:22.767530  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.846298  5627 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:22.810457  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:22.811589  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:22.740288  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:22.851738  5710 raft_consensus.cc:2410] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:27:22.698091 12131 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:22.823794  5559 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
W20260709 12:27:22.853329  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 173777us
W20260709 12:27:22.795306  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 315240us
I20260709 12:27:22.831934  5558 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:22.748771  5552 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:27:22.783149 12010 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:22.736876 11720 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:22.846926  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 241158us
I20260709 12:27:22.847509  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:22.896095  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:22.700317 12085 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:22.849931  5709 rpcz_store.cc:276] 0709 12:27:21.820194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.845553 (+1025359us) service_pool.cc:224] Handling call
0709 12:27:22.846909 (+  1356us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:22.897467  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:22.859750  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:22.850127  5555 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 17: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:22.894654  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:22.899241  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:22.826972  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.900305  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.900607  5710 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:43276
I20260709 12:27:22.900844  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.867993  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:22.900967  5631 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:22.900615  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:22.902837  5629 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:22.807549 12152 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:22.783864 12008 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:22.877588 12010 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:22.872999  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:22.894516  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:22.906226  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:22.923722  5707 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 23 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:27:22.896217  5708 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:22.906701  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:22.924731  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1114) took 1108 ms. Trace:
I20260709 12:27:22.925299  5701 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
W20260709 12:27:22.907617  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:22.908442  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:22.898402  6279 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:22.909660  5702 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:27:22.927603  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2684) took 2641 ms. Trace:
I20260709 12:27:22.928009  5702 rpcz_store.cc:276] 0709 12:27:20.286149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.900436 (+2614287us) service_pool.cc:224] Handling call
0709 12:27:22.909406 (+  8970us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:22.927293 (+ 17887us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:22.927531 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:22.929036  5703 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:22.909888  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:22.929944  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2687) took 2585 ms. Trace:
I20260709 12:27:22.762639  5557 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:22.905831  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:22.861515  5675 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.958s (SENT)
I20260709 12:27:22.925156  5708 rpcz_store.cc:276] 0709 12:27:21.816466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.847276 (+1030810us) service_pool.cc:224] Handling call
0709 12:27:22.924676 (+ 77400us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:27:22.932478  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 71018.3us
I20260709 12:27:22.923175  5628 raft_consensus.cc:2376] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18. Re-sending same reply.
I20260709 12:27:22.903784  5710 raft_consensus.cc:606] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:27:22.873637 12120 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:27:22.854494 12131 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:22.835850 12223 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:22.934758 12131 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:22.935415 12010 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:22.936710 12010 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:22.939769 12223 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:22.849308 12202 raft_consensus.cc:2764] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:27:22.940912  5629 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:22.854415 12009 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:22.778064 11976 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:22.895552 12197 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.013s
I20260709 12:27:22.836529 12132 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:22.904376 12226 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:22.917680 11774 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:22.917771 11720 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:22.913717 12195 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:22.919025  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:23.014616  5631 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:22.920609  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:22.766702 12193 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:22.901739  6274 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 23: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:22.930506  5703 rpcz_store.cc:276] 0709 12:27:20.344920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.928062 (+2583142us) service_pool.cc:224] Handling call
0709 12:27:22.928645 (+   583us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:22.929605 (+   960us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:22.929875 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:22.940893 12110 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:22.901218 12103 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.014s
I20260709 12:27:22.910156  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:22.941958  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:22.943199  6277 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 19: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:22.934062  5710 raft_consensus.cc:614] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:27:22.855566 12181 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:22.948428  5557 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:27:22.965529  5555 raft_consensus.cc:1217] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Deduplicated request from leader. Original: 13.8->[15.9-16.10]   Dedup: 15.9->[16.10-16.10]
I20260709 12:27:23.027340  5555 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17. Ops: [15.9-16.10]
I20260709 12:27:22.916478 12161 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.100s
I20260709 12:27:22.988345  5628 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:27:22.990501  5632 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:23.003248  5701 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:23.034319  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2688) took 1715 ms. Trace:
I20260709 12:27:23.034849  5701 rpcz_store.cc:276] 0709 12:27:21.318811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.943939 (+1625128us) service_pool.cc:224] Handling call
0709 12:27:23.002718 (+ 58779us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.033865 (+ 31147us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.034236 (+   371us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:22.919668 12189 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:23.044255  5632 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:23.010481  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2904) took 2313 ms. Trace:
I20260709 12:27:23.045140  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2822) took 1638 ms. Trace:
I20260709 12:27:23.045610  5632 rpcz_store.cc:276] 0709 12:27:21.407138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.043588 (+1636450us) service_pool.cc:224] Handling call
0709 12:27:23.044011 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.044826 (+   815us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.045071 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:23.045524  5702 rpcz_store.cc:276] 0709 12:27:20.697037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.946340 (+2249303us) service_pool.cc:224] Handling call
0709 12:27:23.009765 (+ 63425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.010020 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:23.010062 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:23.010124 (+    62us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:23.010170 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.010258 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:23.010405 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:22.918390 12188 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:22.909014 11914 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:22.909547 12039 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:22.946905 12211 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:23.021689  5703 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:27:22.976845  5707 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:23.061605  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2690) took 1742 ms. Trace:
I20260709 12:27:23.061997  5703 rpcz_store.cc:276] 0709 12:27:21.319480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.019851 (+1700371us) service_pool.cc:224] Handling call
0709 12:27:23.020294 (+   443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:23.061310 (+ 41016us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.061543 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:23.061616  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2686) took 2730 ms. Trace:
I20260709 12:27:23.062788  5707 rpcz_store.cc:276] 0709 12:27:20.331436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.924470 (+2593034us) service_pool.cc:224] Handling call
0709 12:27:22.976592 (+ 52122us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.061349 (+ 84757us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.061543 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:23.032951  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2817) took 1626 ms. Trace:
I20260709 12:27:23.021900 11720 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:23.063987  5628 rpcz_store.cc:276] 0709 12:27:21.406111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.933620 (+1527509us) service_pool.cc:224] Handling call
0709 12:27:22.987906 (+ 54286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.032487 (+ 44581us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.032868 (+   381us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:23.069890  5628 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:23.071305  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2829) took 1471 ms. Trace:
I20260709 12:27:23.071842  5628 rpcz_store.cc:276] 0709 12:27:21.599933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.069050 (+1469117us) service_pool.cc:224] Handling call
0709 12:27:23.069456 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.070904 (+  1448us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.071229 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:22.872085  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:23.053885 12224 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.386s	user 0.002s	sys 0.000s
I20260709 12:27:23.079826  5632 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:27:23.037648 12237 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:23.094547  5631 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:23.096616  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2826) took 1604 ms. Trace:
I20260709 12:27:23.093276  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1121) took 1272 ms. Trace:
I20260709 12:27:23.098114  5632 rpcz_store.cc:276] 0709 12:27:21.492300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.047300 (+1555000us) service_pool.cc:224] Handling call
0709 12:27:23.079404 (+ 32104us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.083651 (+  4247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.096558 (+ 12907us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:23.098490  5707 rpcz_store.cc:276] 0709 12:27:21.820725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.067016 (+1246291us) service_pool.cc:224] Handling call
0709 12:27:23.092580 (+ 25564us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.092836 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:23.092875 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:23.092934 (+    59us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:23.092980 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.093050 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:23.093205 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:23.074687 12193 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:23.074118 11978 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:23.030153 12181 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:23.116575  5707 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:23.077672 12211 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:23.124608  5632 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:23.078613 11720 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:23.032936 12110 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:23.138288 12131 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:23.139725  5710 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: []
W20260709 12:27:23.125201 12155 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.791s	user 0.006s	sys 0.007s
W20260709 12:27:23.101838 11977 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.014s	sys 0.029s
I20260709 12:27:23.081898 12205 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:23.105322 12237 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:23.113862 12085 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16.
I20260709 12:27:23.134438  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2832) took 1434 ms. Trace:
W20260709 12:27:23.135720 11999 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.650s	user 0.010s	sys 0.002s
I20260709 12:27:23.123623 12186 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:23.141486  5707 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:27:23.118502  5631 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:23.150769  5632 rpcz_store.cc:276] 0709 12:27:21.699954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.104711 (+1404757us) service_pool.cc:224] Handling call
0709 12:27:23.124002 (+ 19291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.125240 (+  1238us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.134269 (+  9029us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:23.140319 12110 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:23.090996 12132 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:23.155411  6276 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:23.155848  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2831) took 1496 ms. Trace:
I20260709 12:27:23.156579  5707 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:23.138403 11720 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:23.158007  5631 rpcz_store.cc:276] 0709 12:27:21.658522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.107368 (+1448846us) service_pool.cc:224] Handling call
0709 12:27:23.108898 (+  1530us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.155041 (+ 46143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.155234 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:23.154129 12186 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:23.147521 12155 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:23.168076  5632 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18. Ops: []
W20260709 12:27:23.241372 12174 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.718s	user 0.010s	sys 0.003s
I20260709 12:27:23.162444 12021 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:23.248968 12021 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:23.250830  5635 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:23.220289  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2815) took 1814 ms. Trace:
I20260709 12:27:23.167196 12008 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:23.204406  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:23.243808  5633 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:23.254675  5629 rpcz_store.cc:276] 0709 12:27:21.405764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.922690 (+1516926us) service_pool.cc:224] Handling call
0709 12:27:22.922987 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:22.941561 (+ 18574us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.220049 (+278488us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b54001f2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:23.220226 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":40643840}
I20260709 12:27:23.255779  5636 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 18: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:23.223017  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:23.256896  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 384922us
I20260709 12:27:23.257647  5634 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:27:23.224792  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.198596  5704 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:23.260051  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:23.260855  5706 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:23.232213 11983 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:23.193190 12191 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:23.162011  5631 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:27:23.240859  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2691) took 1921 ms. Trace:
W20260709 12:27:23.204427  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.245730 12152 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:23.239142 11715 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.656s	user 0.010s	sys 0.003s
I20260709 12:27:23.224191  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:23.248456 12131 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:23.171159 11976 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:23.233944 12084 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.719s	user 0.010s	sys 0.005s
W20260709 12:27:23.262441  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 449080us
I20260709 12:27:23.262996  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:23.264190  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:23.280658  5706 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:23.266253  5710 rpcz_store.cc:276] 0709 12:27:21.319770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.037499 (+1717729us) service_pool.cc:224] Handling call
0709 12:27:23.066807 (+ 29308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.140140 (+ 73333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.240486 (+100346us) spinlock_profiling.cc:243] Waited 72.7 ms on lock 0x7b54003ad4d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:23.240753 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":167180160}
I20260709 12:27:23.247005 12155 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:23.168004  5707 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: [17.12-17.12]
W20260709 12:27:23.234280 12225 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.488s	user 0.003s	sys 0.000s
I20260709 12:27:23.263670 12039 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:23.178196 12237 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:23.272467 12184 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.012s	sys 0.045s
I20260709 12:27:23.237713 12009 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:23.287292  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:23.273342 12100 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:23.281793 11721 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:23.281015 12152 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:23.318840  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:23.181782 11720 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:23.288920 12184 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.013s	sys 0.045s
I20260709 12:27:23.217450 12211 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:23.276981 11715 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:23.329669  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49761.6us
I20260709 12:27:23.318076  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:23.218878 12186 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:23.231776 12172 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:23.282495 12205 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.041s
I20260709 12:27:23.338686  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:23.339437  5706 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:23.348219  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2685) took 3042 ms. Trace:
I20260709 12:27:23.329736 11720 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:23.348688  5709 rpcz_store.cc:276] 0709 12:27:20.305894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.921158 (+2615264us) service_pool.cc:224] Handling call
0709 12:27:22.921461 (+   303us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:22.923446 (+  1985us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:22.923478 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:22.923930 (+   452us) log.cc:844] Serialized 26 byte log entry
0709 12:27:23.263104 (+339174us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:23.263146 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.263210 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:23.347993 (+ 84783us) raft_consensus.cc:1685] finished
0709 12:27:23.348016 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:23.348151 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3988,"raft.queue_time_us":24703,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":24574,"thread_start_us":338791,"threads_started":1,"wal-append.queue_time_us":364582}
I20260709 12:27:23.324339 12100 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:23.350571 11720 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:23.325341 11721 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:23.392843  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49148.9us
I20260709 12:27:23.373832 12205 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:23.375352 12100 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:23.377285 12184 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:23.355687 12172 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:23.390904 11663 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:23.345976 12186 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:23.365238 12004 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:23.336305 12175 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.020s
I20260709 12:27:23.346545 12246 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:27:23.383277 12241 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:27:23.351909 12039 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:27:23.377288 12181 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:23.397137 12205 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:23.395592 12021 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:23.405313 11977 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:23.532723 11977 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:23.532843 11997 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:27:23.534619  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2813) took 2129 ms. Trace:
I20260709 12:27:23.535135  5633 rpcz_store.cc:276] 0709 12:27:21.405214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.277236 (+872022us) service_pool.cc:224] Handling call
0709 12:27:22.277658 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.255204 (+977546us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.534053 (+278849us) spinlock_profiling.cc:243] Waited 966 ms on lock 0x7b540032bed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:23.534414 (+   361us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2221554432}
I20260709 12:27:23.414307 11983 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:23.476856  5451 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.007s	sys 0.004s
W20260709 12:27:23.540717  5451 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.008s	sys 0.004s
I20260709 12:27:23.206039  5561 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:27:23.541287  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:23.478649  5560 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.008s	sys 0.004s
W20260709 12:27:23.543612  5560 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.759s	user 0.008s	sys 0.005s
I20260709 12:27:23.544804  6273 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:23.545337  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1205) took 1098 ms. Trace:
I20260709 12:27:23.545197  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:23.545974  5560 rpcz_store.cc:276] 0709 12:27:22.446815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.446961 (+   146us) service_pool.cc:224] Handling call
0709 12:27:22.447335 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:22.447711 (+   376us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:27:22.783944 (+336233us) log.cc:844] Serialized 14 byte log entry
0709 12:27:23.544644 (+760700us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:23.544748 (+   104us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:23.544883 (+   135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.545022 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:23.545267 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10480,"prepare.queue_time_us":590132,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":68,"thread_start_us":1092735,"threads_started":2,"wal-append.queue_time_us":557076}
I20260709 12:27:23.547631  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2779) took 1066 ms. Trace:
I20260709 12:27:23.528331  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:23.481208  5553 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 18. Current term is 19. Ops: []
W20260709 12:27:23.478683  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.275684  5627 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:23.255949  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:23.208237  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:23.591058  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:23.486110  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:23.593066  5631 raft_consensus.cc:2376] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17. Re-sending same reply.
I20260709 12:27:23.487555  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2692) took 2167 ms. Trace:
I20260709 12:27:23.593999  5701 rpcz_store.cc:276] 0709 12:27:21.320062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.035941 (+1715879us) service_pool.cc:224] Handling call
0709 12:27:23.041713 (+  5772us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:23.042312 (+   599us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:23.042382 (+    70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:23.050325 (+  7943us) log.cc:844] Serialized 26 byte log entry
0709 12:27:23.288879 (+238554us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:23.288922 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.288996 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:23.486448 (+197452us) raft_consensus.cc:1685] finished
0709 12:27:23.486661 (+   213us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:23.487230 (+   569us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":194818,"threads_started":1,"wal-append.queue_time_us":337475}
I20260709 12:27:23.595851  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:23.490974  5709 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:27:23.243949  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:23.608898  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:23.610476  5709 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:23.611157  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:23.611577  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:23.499491  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:23.500787  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:23.507515 11715 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:23.499516  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:23.514241 11587 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:27:23.516263  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1208) took 1068 ms. Trace:
I20260709 12:27:23.517168 11774 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:27:23.524006  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.531419  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:23.531478  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:23.542784  5633 raft_consensus.cc:2368] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:23.423372 12205 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:23.542886  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2789) took 1058 ms. Trace:
I20260709 12:27:23.548283  5552 rpcz_store.cc:276] 0709 12:27:22.480909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.494566 (+ 13657us) service_pool.cc:224] Handling call
0709 12:27:22.741589 (+247023us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:22.862493 (+120904us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.547158 (+684665us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b5400082850. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:23.547409 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":567119104}
W20260709 12:27:23.663712  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.542855 11721 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:23.603686  5558 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:27:23.604478  6273 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:23.547886 11997 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.069s
W20260709 12:27:23.605798  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.417748 12120 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:23.607368 12100 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:23.613737  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2788) took 1130 ms. Trace:
I20260709 12:27:23.498045  5706 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:23.615202  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:23.616008  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:23.618850  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:23.620132  5556 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:23.673498  5707 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:23.674103  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2793) took 1106 ms. Trace:
I20260709 12:27:23.675236  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:23.617796  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.621308 12246 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:27:23.682301  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 183181us
I20260709 12:27:23.683399 11663 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:23.684720  5710 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:23.684468 11663 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:23.685482  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.688763 12251 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:23.617780  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:23.629878  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2696) took 1567 ms. Trace:
I20260709 12:27:23.645318  5554 rpcz_store.cc:276] 0709 12:27:22.447686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.447794 (+   108us) service_pool.cc:224] Handling call
0709 12:27:22.449669 (+  1875us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:22.850642 (+400973us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.515440 (+664798us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b54000ebad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:23.515911 (+   471us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":890163968}
I20260709 12:27:23.628858  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:23.535758 11977 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:23.621690  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:23.661635  5635 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:23.680025  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:23.680827 12155 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:23.674932  5556 rpcz_store.cc:276] 0709 12:27:22.567246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.618847 (+1051601us) service_pool.cc:224] Handling call
0709 12:27:23.619467 (+   620us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.673630 (+ 54163us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.673985 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:27:23.660274  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 132720us
I20260709 12:27:23.682564 12246 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:23.625288  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.663724  5561 rpcz_store.cc:276] 0709 12:27:22.483845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.205186 (+721341us) service_pool.cc:224] Handling call
0709 12:27:23.205584 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.542048 (+336464us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.542650 (+   602us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:23.672075 12100 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:23.677889  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:23.671785  5553 rpcz_store.cc:276] 0709 12:27:22.483494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.030785 (+547291us) service_pool.cc:224] Handling call
0709 12:27:23.066776 (+ 35991us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.587180 (+520404us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.613273 (+ 26093us) spinlock_profiling.cc:243] Waited 414 ms on lock 0x7b54000ec4d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:23.613641 (+   368us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":951491328}
I20260709 12:27:23.689842  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2791) took 1184 ms. Trace:
W20260709 12:27:23.666851  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55959.3us
I20260709 12:27:23.701447  5707 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:23.700773  5558 rpcz_store.cc:276] 0709 12:27:22.493162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.519634 (+1026472us) service_pool.cc:224] Handling call
0709 12:27:23.520756 (+  1122us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.665525 (+144769us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.669268 (+  3743us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:23.690680  5705 rpcz_store.cc:276] 0709 12:27:22.058992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.748312 (+689320us) service_pool.cc:224] Handling call
0709 12:27:23.625980 (+877668us) spinlock_profiling.cc:243] Waited 739 ms on lock 0x7b54001982d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:23.626328 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1699858816}
I20260709 12:27:23.695048 12251 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:23.716980  5709 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:23.721247  5701 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:23.710554  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:23.719537  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:23.726356  5630 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 17: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:27:23.730786  5555 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.007s	sys 0.005s
W20260709 12:27:23.730801  5628 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.002s	sys 0.009s
I20260709 12:27:23.737267  5633 raft_consensus.cc:2368] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16.
I20260709 12:27:23.729096  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:27:23.739187  5555 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.007s	sys 0.005s
I20260709 12:27:23.741088 12181 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:23.740870 12251 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:23.742235 12181 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:23.744792  5706 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:23.744498 12155 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:23.746819  5706 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:23.747643 12155 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:23.748010  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:23.748901  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:23.749495  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:23.750133  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:23.751971 12081 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.838s	user 0.000s	sys 0.012s
I20260709 12:27:23.733726  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2792) took 1207 ms. Trace:
W20260709 12:27:23.752602 12081 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.939s	user 0.001s	sys 0.013s
W20260709 12:27:23.761580  5628 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.003s	sys 0.009s
I20260709 12:27:23.757433  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:23.756574  5560 rpcz_store.cc:276] 0709 12:27:22.525802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.585750 (+1059948us) service_pool.cc:224] Handling call
0709 12:27:23.586387 (+   637us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.586998 (+   611us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:27:23.589020 (+  2022us) log.cc:844] Serialized 14 byte log entry
0709 12:27:23.732195 (+143175us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:23.732678 (+   483us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:23.732798 (+   120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.732966 (+   168us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:23.733522 (+   556us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":769,"prepare.queue_time_us":17648,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":4062,"thread_start_us":138305,"threads_started":1}
I20260709 12:27:23.749564  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:23.736718 11715 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:23.768438  5558 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:23.730897 12236 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:23.773850  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:23.779915  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 177947us
W20260709 12:27:23.778743  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 90439.3us
I20260709 12:27:23.785900  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:23.772156 11977 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:23.781960  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 104410us
I20260709 12:27:23.799441  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:23.768899 11715 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:23.727938  5632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:23.798158  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1211) took 1010 ms. Trace:
I20260709 12:27:23.807920  5554 rpcz_store.cc:276] 0709 12:27:22.781142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.704699 (+923557us) service_pool.cc:224] Handling call
0709 12:27:23.757796 (+ 53097us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.791097 (+ 33301us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:23.791137 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:23.791225 (+    88us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:23.791292 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.791367 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:23.791514 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:23.809217  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1212) took 1001 ms. Trace:
I20260709 12:27:23.809671  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:23.809753  5553 rpcz_store.cc:276] 0709 12:27:22.807707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.707940 (+900233us) service_pool.cc:224] Handling call
0709 12:27:23.777221 (+ 69281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.808614 (+ 31393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:23.808638 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:23.808701 (+    63us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:23.808844 (+   143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.808985 (+   141us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:23.809140 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:23.810581 12193 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:23.811507 12193 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:23.812034 12193 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:23.816488 12182 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:23.818004  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.817703 12182 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:23.818800  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.820454 12165 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:23.821322 12165 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:23.821839 12165 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:27:23.824705  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 206353us
I20260709 12:27:23.825516 12103 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:27:23.826246 12103 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:23.827479 12103 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:23.829401  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:23.831838  5634 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.573s	user 0.011s	sys 0.012s
I20260709 12:27:23.833283 12246 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:23.835243  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:23.837218  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.841807  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1117) took 2022 ms. Trace:
I20260709 12:27:23.841326 12120 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:23.843324  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 129668us
W20260709 12:27:23.843724 12262 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.165s	user 0.000s	sys 0.002s
I20260709 12:27:23.843899 12155 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:23.829468  5633 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:27:23.820536  5707 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18. Ops: []
W20260709 12:27:23.851569  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.862093  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:23.845750  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:23.870887  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.840487 11715 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:23.879581  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 74596.1us
W20260709 12:27:23.880846  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 81752us
I20260709 12:27:23.833341 12081 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:27:23.882714  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53515.3us
W20260709 12:27:23.884128  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:23.886597  5553 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:27:23.891464  6280 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 20: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:23.854069  5703 rpcz_store.cc:276] 0709 12:27:21.819644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.065546 (+1245902us) service_pool.cc:224] Handling call
0709 12:27:23.095987 (+ 30441us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:23.096258 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:23.096295 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:23.096808 (+   513us) log.cc:844] Serialized 26 byte log entry
0709 12:27:23.497051 (+400243us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:23.497161 (+   110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.497306 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:23.839269 (+341963us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:23.841223 (+  1954us) raft_consensus.cc:1685] finished
0709 12:27:23.841249 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:23.841563 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6111488,"thread_start_us":375739,"threads_started":1,"wal-append.queue_time_us":609996}
I20260709 12:27:23.870887  5634 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
W20260709 12:27:23.879505  5602 leader_election.cc:336] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.319s (SENT)
I20260709 12:27:23.857079 11774 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:23.908378  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 64613.5us
I20260709 12:27:23.909529  5635 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:23.910463  5630 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:27:23.911144  5635 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:27:23.888144 12081 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:23.917800  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:23.892369 12155 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:23.920795  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1115) took 2103 ms. Trace:
I20260709 12:27:23.921553 12100 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:23.922495  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2785) took 1297 ms. Trace:
I20260709 12:27:23.923520 12182 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:23.932739 11774 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:23.934173 12174 raft_consensus.cc:2764] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:27:23.935866  5633 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 10. (term mismatch)
I20260709 12:27:23.973233  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:23.938210  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:23.938961  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:23.940847 11976 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:27:23.944770  6276 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:27:23.945561  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:23.920652  5630 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:23.915539  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:23.948846  5710 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:23.928689  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:23.950596  5702 rpcz_store.cc:276] 0709 12:27:21.816904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.063729 (+1246825us) service_pool.cc:224] Handling call
0709 12:27:23.103070 (+ 39341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:23.106093 (+  3023us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:23.106139 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:23.106751 (+   612us) log.cc:844] Serialized 26 byte log entry
0709 12:27:23.360961 (+254210us) spinlock_profiling.cc:243] Waited 4.69 ms on lock 0x7b48001701d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:27:23.489086 (+128125us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:23.489289 (+   203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.489462 (+   173us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:23.842810 (+353348us) spinlock_profiling.cc:243] Waited 44.7 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:23.911735 (+ 68925us) raft_consensus.cc:1685] finished
0709 12:27:23.911760 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:23.920576 (+  8816us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33160,"raft.queue_time_us":3526,"raft.run_cpu_time_us":3129,"raft.run_wall_time_us":253062,"spinlock_wait_cycles":113640576,"thread_start_us":356531,"threads_started":1,"wal-append.queue_time_us":557952}
W20260709 12:27:23.954408 12266 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.129s	user 0.002s	sys 0.001s
I20260709 12:27:23.954710 12155 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:23.958165  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:23.954880 11997 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.010s
I20260709 12:27:23.961867  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2787) took 1478 ms. Trace:
I20260709 12:27:23.936848  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:23.972939  5635 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
W20260709 12:27:23.984766  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 69935.8us
I20260709 12:27:23.979457  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:23.972962  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:23.991915  5709 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:23.973649  5633 pending_rounds.cc:85] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 9
I20260709 12:27:24.004822 12251 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:24.005378 12251 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:24.005802 12251 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:24.013638  5704 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 13. Current term is 14. Ops: []
I20260709 12:27:24.002182  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:23.992681  5633 pending_rounds.cc:107] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 16.10
W20260709 12:27:24.037463  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 229826us
I20260709 12:27:24.032930  6274 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 19: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:27:24.038440  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45337.4us
W20260709 12:27:24.034420  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 138712us
I20260709 12:27:24.033136  6277 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:27:24.040695  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 135372us
I20260709 12:27:24.040478  5554 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:23.990154 11663 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:24.037477  5633 raft_consensus.cc:2889] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:23.985548  5555 rpcz_store.cc:276] 0709 12:27:22.483182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.028386 (+545204us) service_pool.cc:224] Handling call
0709 12:27:23.062020 (+ 33634us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.062234 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:27:23.740220 (+677986us) log.cc:844] Serialized 14 byte log entry
0709 12:27:23.903036 (+162816us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:23.914182 (+ 11146us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:23.918437 (+  4255us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.918532 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:23.961748 (+ 43216us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":61361,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":3065,"raft.queue_time_us":75329,"thread_start_us":798966,"threads_started":2,"wal-append.queue_time_us":214870}
I20260709 12:27:24.038095 11721 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:23.996042 12100 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:24.060812  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:24.060256  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:24.063195  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:24.059664  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:24.032706 11983 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
W20260709 12:27:24.066066  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 93251.2us
I20260709 12:27:24.046805  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:24.067910 12193 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:24.068799  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:24.070170  5634 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
W20260709 12:27:24.088186  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:24.055630  6236 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:24.072170  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1209) took 1235 ms. Trace:
I20260709 12:27:24.072847  5553 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:24.073638  5560 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:23.961462  5557 rpcz_store.cc:276] 0709 12:27:22.482585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.025120 (+542535us) service_pool.cc:224] Handling call
0709 12:27:23.025509 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.025763 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:27:23.482445 (+456682us) log.cc:844] Serialized 14 byte log entry
0709 12:27:23.720501 (+238056us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:23.728238 (+  7737us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:23.754789 (+ 26551us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.758745 (+  3956us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:23.780026 (+ 21281us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15633,"prepare.queue_time_us":76750,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":4147,"raft.run_cpu_time_us":393,"raft.run_wall_time_us":36368,"thread_start_us":663559,"threads_started":2}
I20260709 12:27:24.042995 12256 raft_consensus.cc:2764] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:27:23.989244 12181 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:24.093010  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:24.097172  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:24.098140  5557 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:24.099224  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:24.100677  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2905) took 3344 ms. Trace:
I20260709 12:27:24.101255  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:24.102056  5557 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:24.101117  5708 rpcz_store.cc:276] 0709 12:27:20.697335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:22.951263 (+2253928us) service_pool.cc:224] Handling call
0709 12:27:22.951780 (+   517us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:23.314499 (+362719us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:23.314556 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:23.319579 (+  5023us) log.cc:844] Serialized 26 byte log entry
0709 12:27:23.794687 (+475108us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:27:23.849508 (+ 54821us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.868345 (+ 18837us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:23.944887 (+ 76542us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:24.035744 (+ 90857us) raft_consensus.cc:1685] finished
0709 12:27:24.035770 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:24.041256 (+  5486us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56587,"raft.queue_time_us":70719,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":10497,"spinlock_wait_cycles":26880896,"thread_start_us":659991,"threads_started":2,"wal-append.queue_time_us":495725}
I20260709 12:27:24.103561  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:24.105011  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:24.060173 12081 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:24.106204 12081 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:24.106221  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:24.108680 12186 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:24.110810 12193 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:24.097857 12103 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:24.112329  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:24.114727  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:24.119277  6277 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:24.110711 11663 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:24.112392 12264 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:27:24.124502  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:24.128454  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:24.123291  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:24.132619  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:24.129527  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 162344us
I20260709 12:27:24.138579  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:24.135179 12236 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:24.139742  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:24.094254 11715 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:27:24.090896  5559 rpcz_store.cc:276] 0709 12:27:22.570709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.622473 (+1051764us) service_pool.cc:224] Handling call
0709 12:27:23.623095 (+   622us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.623671 (+   576us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:27:23.624300 (+   629us) log.cc:844] Serialized 14 byte log entry
0709 12:27:23.801010 (+176710us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:23.801402 (+   392us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:23.801863 (+   461us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:23.802352 (+   489us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:23.806141 (+  3789us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3167,"prepare.queue_time_us":1141,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":18973,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":3106,"thread_start_us":152363,"threads_started":1,"wal-append.queue_time_us":349242}
I20260709 12:27:24.130468 12103 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:24.153057 12211 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:24.154054 12211 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:24.157135 12236 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:27:24.160928  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 100762us
I20260709 12:27:24.162897  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:24.164317  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 123216us
I20260709 12:27:24.165657  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:24.167073 12039 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:24.167266  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:24.167685 12039 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:24.134384 12193 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:27:24.169128  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 113776us
I20260709 12:27:24.169703  5558 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:24.172133 12182 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:24.172595 12182 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:24.170817  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:24.172950 12182 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:24.172957 12251 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:24.179329 12205 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:24.180821  5555 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:24.181814  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:24.182395  5555 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:27:24.183449  5560 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:24.173290  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2587) took 1004 ms. Trace:
I20260709 12:27:24.178025 12251 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:24.185397 12251 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:24.189141 12282 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:24.189776 12282 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:24.190732  5553 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:24.192137  5553 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:27:24.192595 12251 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:24.193085 12282 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:24.193737 12251 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:24.194481 12282 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:24.195438 12271 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 12:27:24.184639  5560 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: []
W20260709 12:27:24.197182  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 156724us
I20260709 12:27:24.140164 11976 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:24.184978  5631 rpcz_store.cc:276] 0709 12:27:23.168842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.617312 (+448470us) service_pool.cc:224] Handling call
0709 12:27:24.173077 (+555765us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b5400241654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:24.173230 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":722229120}
I20260709 12:27:24.127425 12181 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
I20260709 12:27:24.203262  5560 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:27:24.204530  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:24.209751  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:24.214617 11715 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:27:24.215137 12272 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:27:24.189173 11663 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:24.224740  5561 raft_consensus.cc:2410] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 21 because replica has last-logged OpId of term: 20 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
W20260709 12:27:24.224963  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:24.185109 11997 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 12:27:24.207577 11983 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:27:24.232797  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:24.139974 12120 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.073s
I20260709 12:27:24.243086  5558 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:27:24.250057  5553 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: [22.13-22.13]
W20260709 12:27:24.251987 12172 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.848s	user 0.011s	sys 0.010s
W20260709 12:27:24.255089  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 113237us
I20260709 12:27:24.259994 12237 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:24.260440 12237 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:24.260874 12237 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:24.261597 12246 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:27:24.262818 11774 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:24.239393 11983 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:24.271868  5561 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:27:24.249419 12211 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:24.273103  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 105932us
I20260709 12:27:24.277395 12276 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:24.275770 11976 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:24.278184 12276 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:24.279769 11721 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:27:24.280855  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:24.281054  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:24.280866  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:24.281658  5631 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:24.281844  5630 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:24.282290  5635 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:24.282864  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:24.283217  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:24.283417  5630 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:24.284044  5635 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 20: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:24.283234  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:24.286501  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 123757us
W20260709 12:27:24.287097  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:24.287985  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:24.291183  5633 raft_consensus.cc:2410] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:27:24.271270 11983 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:24.294906  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 197919us
I20260709 12:27:24.298907  5553 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:27:24.274688 12172 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:24.302000  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 180057us
I20260709 12:27:24.303833  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:24.319024  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44956.4us
I20260709 12:27:24.322911  5554 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:24.329698  5630 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 12:27:24.330890  5553 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18. Ops: []
W20260709 12:27:24.321643  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 150461us
I20260709 12:27:24.278201 11976 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:24.352556  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:24.344871  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:24.325811 11663 raft_consensus.cc:2764] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election decision vote started in defunct term 18: won
I20260709 12:27:24.313724 12277 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:27:24.365620  5559 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
W20260709 12:27:24.366422 12152 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.018s	user 0.005s	sys 0.018s
W20260709 12:27:24.367388 12283 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:27:24.368511 12103 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:24.371325  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2798) took 1674 ms. Trace:
I20260709 12:27:24.329488 12276 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:24.371760  5556 rpcz_store.cc:276] 0709 12:27:22.696605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.707584 (+1010979us) service_pool.cc:224] Handling call
0709 12:27:23.783035 (+ 75451us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:24.030162 (+247127us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:24.030201 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:24.030982 (+   781us) log.cc:844] Serialized 26 byte log entry
0709 12:27:24.120896 (+ 89914us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:24.120946 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:24.121016 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:24.371045 (+250029us) raft_consensus.cc:1685] finished
0709 12:27:24.371069 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:24.371251 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17864,"raft.queue_time_us":59061,"raft.run_cpu_time_us":383,"raft.run_wall_time_us":61045,"thread_start_us":254230,"threads_started":2,"wal-append.queue_time_us":206806}
W20260709 12:27:24.373875 11977 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.567s	user 0.011s	sys 0.004s
W20260709 12:27:24.374053 12281 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.323s	user 0.002s	sys 0.000s
W20260709 12:27:24.375682 12165 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.553s	user 0.008s	sys 0.004s
I20260709 12:27:24.376143 12165 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:24.363725  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 239333us
I20260709 12:27:24.328328 12186 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:24.318256 12172 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:24.295701 12039 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:24.427341  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:24.438771  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:24.356663 12184 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.141s
I20260709 12:27:24.444517  6273 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260709 12:27:24.384568 12277 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:24.448127  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2797) took 1752 ms. Trace:
W20260709 12:27:24.453531  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 69254.2us
W20260709 12:27:24.457620  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:24.457170 12039 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:27:24.475485  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:24.476439  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:24.482913  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 121455us
W20260709 12:27:24.485321  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:24.451220  5552 rpcz_store.cc:276] 0709 12:27:22.696168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.678406 (+982238us) service_pool.cc:224] Handling call
0709 12:27:23.679145 (+   739us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:23.679946 (+   801us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:23.679982 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:23.681156 (+  1174us) log.cc:844] Serialized 52 byte log entry
0709 12:27:23.852770 (+171614us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:23.866505 (+ 13735us) log.cc:844] Serialized 14 byte log entry
0709 12:27:24.038740 (+172235us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:24.038927 (+   187us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:24.372706 (+333779us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b480027e958. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:27:24.438897 (+ 66191us) raft_consensus.cc:1685] finished
0709 12:27:24.447806 (+  8909us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:24.448027 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8339,"prepare.queue_time_us":17637,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":2715,"raft.queue_time_us":74426,"raft.run_cpu_time_us":3689,"raft.run_wall_time_us":437250,"spinlock_wait_cycles":32113792,"thread_start_us":341164,"threads_started":2,"wal-append.queue_time_us":356396}
I20260709 12:27:24.492491  5636 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
W20260709 12:27:24.493809  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 179696us
I20260709 12:27:24.494846  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:24.495787 11663 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:24.504164  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:24.505192  5633 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:24.506596  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:24.507341  5633 raft_consensus.cc:2376] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18. Re-sending same reply.
I20260709 12:27:24.508807  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:24.509629  5635 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:27:24.520522  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72796.4us
I20260709 12:27:24.521486  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:24.523779 12285 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:24.525209 12285 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:24.526750 12285 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:24.528558  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 83899.9us
W20260709 12:27:24.531867  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 67545.1us
I20260709 12:27:24.534309 12172 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:24.540798  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 96763.8us
W20260709 12:27:24.542469  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:24.545634 12288 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:27:24.566651  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:24.585933  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62472.5us
W20260709 12:27:24.592585  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48835.1us
W20260709 12:27:24.596678  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:24.621790 12290 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:24.622604 12290 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:24.627926 12100 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.535s	user 0.007s	sys 0.000s
I20260709 12:27:24.455621 12277 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:24.631394 11663 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:24.633144 12181 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:27:24.634057 11774 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:24.636554 12193 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:27:24.738894  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:24.659400  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 163070us
W20260709 12:27:24.729003  5555 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.546s	user 0.010s	sys 0.001s
I20260709 12:27:24.740962  5555 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:24.735920  5633 raft_consensus.cc:2368] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18.
W20260709 12:27:24.742274  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 279801us
I20260709 12:27:24.611996 12211 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.119s
I20260709 12:27:24.548228 12288 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.187s
I20260709 12:27:24.748531 12152 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.002s
I20260709 12:27:24.751451 11715 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:24.751938 11715 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:24.753341 11715 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:24.754854 11715 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:24.734105 11977 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 12:27:24.732576 12277 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:24.770504  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:24.770629  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:24.761377  5629 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:24.775991  5632 raft_consensus.cc:606] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:24.776575  5632 raft_consensus.cc:614] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:27:24.773430 12211 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:24.795846 12211 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:24.797379  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2830) took 3172 ms. Trace:
I20260709 12:27:24.760308 12186 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:24.746997 12172 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:24.800334 12172 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:24.802732 12172 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:24.798548  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:24.803798 12100 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:24.804342 12100 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:24.805102  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:24.805341 12100 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:24.806042 12184 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:27:24.807040 12081 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.700s	user 0.009s	sys 0.001s
I20260709 12:27:24.809180 12165 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:24.809494 12184 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:27:24.814404 11997 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:27:24.815721  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:24.818527  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:24.820806  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:24.824337  5555 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:24.831208  5557 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:24.837173  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:24.851483  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 354906us
W20260709 12:27:24.853390  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60019us
W20260709 12:27:24.856635  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 112925us
I20260709 12:27:24.862097  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:24.733332 12237 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:24.858317  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:24.865248  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:24.865478  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 71784.5us
I20260709 12:27:24.870045  5635 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
W20260709 12:27:24.852144 12182 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.679s	user 0.007s	sys 0.005s
I20260709 12:27:24.871608  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:24.874480  5554 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:24.872109  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56589.2us
I20260709 12:27:24.871094  5631 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:24.876770  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:24.877380  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:24.877759  5634 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 18: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:24.879192  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:24.881060  5552 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:24.879796  5633 raft_consensus.cc:2376] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20. Re-sending same reply.
I20260709 12:27:24.834656  5628 rpcz_store.cc:276] 0709 12:27:21.624853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.074725 (+1449872us) service_pool.cc:224] Handling call
0709 12:27:23.075237 (+   512us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:23.075612 (+   375us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:27:23.816394 (+740782us) log.cc:844] Serialized 14 byte log entry
0709 12:27:24.234905 (+418511us) spinlock_profiling.cc:243] Waited 28.4 ms on lock 0x7b18004800a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:27:24.386510 (+151605us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:24.452297 (+ 65787us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:24.796976 (+344679us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:24.797125 (+   149us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:24.797309 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20184,"prepare.queue_time_us":726168,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":61,"raft.queue_time_us":25355,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":51819,"spinlock_wait_cycles":412830848,"thread_start_us":796422,"threads_started":2,"wal-append.queue_time_us":671972}
I20260709 12:27:24.860083 12193 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:24.881304  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:24.873116 12290 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:24.889958  5524 leader_election.cc:336] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.184s (SENT)
I20260709 12:27:24.890093  5634 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:24.874114 11587 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:24.877076 12276 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:24.893376  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:24.893958  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:24.932119 12246 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.670s	user 0.009s	sys 0.001s
W20260709 12:27:24.896132 12293 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
I20260709 12:27:24.898087  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:24.902963 11715 raft_consensus.cc:2764] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:27:24.881399 12100 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:24.905201 12289 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:24.909756 12237 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:24.952730  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54714us
I20260709 12:27:24.918718 12098 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:24.878904 12165 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:27:24.921722 12280 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.869s	user 0.010s	sys 0.015s
W20260709 12:27:24.926864  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 178613us
I20260709 12:27:24.930913  5554 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
W20260709 12:27:24.934326 11983 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.606s	user 0.000s	sys 0.009s
I20260709 12:27:24.886917 11774 consensus_queue.cc:1243] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 19.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
W20260709 12:27:24.937382  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:24.939802  5631 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
W20260709 12:27:24.959429 12269 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.902s	user 0.015s	sys 0.015s
W20260709 12:27:24.936993 12205 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.757s	user 0.011s	sys 0.000s
I20260709 12:27:24.955766  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:24.956074  6277 raft_consensus.cc:2410] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:27:24.878973 12182 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:24.950721 12193 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:24.952137  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 67595us
I20260709 12:27:24.953397 12289 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:24.950394 12100 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:24.965718 11983 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:24.954643 11997 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:24.969086  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:24.960103 11977 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:24.984994  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:24.997104  5634 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: [21.17-21.17]
W20260709 12:27:24.989341 12279 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.936s	user 0.011s	sys 0.001s
I20260709 12:27:24.989307 11977 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:24.989883  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:24.990588  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:24.993813  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:24.965516 12155 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:24.968490 11587 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:25.008638 12155 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:25.009797 11587 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:25.010962 12155 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:25.011386  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:25.011996 12172 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:24.963301 12165 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:25.004292  5561 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:24.992216 12287 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.479s	user 0.000s	sys 0.011s
I20260709 12:27:24.980074 12237 raft_consensus.cc:2764] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
W20260709 12:27:25.013456  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:25.013963  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:25.016187  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:25.011638 12155 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:25.007844 11977 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:24.999332 11774 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:25.039916  5555 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:25.033882  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:24.963666 12098 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:25.016322 12282 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.821s	user 0.006s	sys 0.010s
I20260709 12:27:24.972275 12289 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.016819 12152 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:27:25.046383 12300 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:27:25.052088 12175 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:25.052749 12175 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:25.060662  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:24.999464 11997 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.068315 12155 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:25.070251  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2586) took 1150 ms. Trace:
I20260709 12:27:25.070648  5559 rpcz_store.cc:276] 0709 12:27:23.936539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:24.211666 (+275127us) service_pool.cc:224] Handling call
0709 12:27:24.265188 (+ 53522us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:24.366094 (+100906us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:25.069989 (+703895us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b54003a8ed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:25.070185 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":230408576}
I20260709 12:27:25.073429  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:25.084754  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 119155us
I20260709 12:27:25.064111 12205 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.024s
W20260709 12:27:25.087134  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 135928us
I20260709 12:27:25.088554 12182 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:25.085822  5633 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
W20260709 12:27:25.091456  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46109.9us
W20260709 12:27:25.095924  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 135838us
I20260709 12:27:25.079056 12282 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:25.081050 12120 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:25.103266  6236 leader_election.cc:400] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18.
W20260709 12:27:25.105273  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 103602us
I20260709 12:27:25.095810 12256 raft_consensus.cc:2764] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
W20260709 12:27:25.117064  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 97621us
W20260709 12:27:25.115339  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:25.117789  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:25.118563  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:25.101698 12184 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:25.119405  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 101720us
I20260709 12:27:25.107991 12120 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:25.129688 12120 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:25.126629  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:25.124526  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:25.145341 11976 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:25.081019 12165 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.151520  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:25.155786  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:25.158713  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:25.096916 12193 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.140118 12304 raft_consensus.cc:2764] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:27:25.160741  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:39496 (request call id 46) took 1478 ms. Trace:
I20260709 12:27:25.187073  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:25.161845  5675 leader_election.cc:400] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:25.177474  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:25.140619 12184 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.177208 12152 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:25.178488 12275 raft_consensus.cc:2764] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:27:25.202543  5675 leader_election.cc:403] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
W20260709 12:27:25.256613  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 114785us
I20260709 12:27:25.201510  5632 rpcz_store.cc:276] 0709 12:27:23.679668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.711004 (+ 31336us) service_pool.cc:224] Handling call
0709 12:27:25.157663 (+1446659us) spinlock_profiling.cc:243] Waited 939 ms on lock 0x7b54001cfcd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa66c853 00007f4fff63a60a 00007f4ff8bc07d4 00007f4ff8bc0762 00007f4ff8bc0691 00007f4ff8bc060c 00007f4ff8bbf8c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:25.157775 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2160818944}
W20260709 12:27:25.273129  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 72975.9us
W20260709 12:27:25.306398  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:25.308096  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:25.200769  6276 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:25.157855 12285 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:25.342988 12292 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.852s	user 0.013s	sys 0.022s
I20260709 12:27:25.342890 12285 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.161159 11976 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:25.345414  5561 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:25.348070  5557 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:25.103736 12175 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:25.355686  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 155505us
W20260709 12:27:25.358860  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:25.347173  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2598) took 2166 ms. Trace:
I20260709 12:27:25.361102  5627 rpcz_store.cc:276] 0709 12:27:23.180931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.796581 (+615650us) service_pool.cc:224] Handling call
0709 12:27:23.915287 (+118706us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:24.113353 (+198066us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:24.113391 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:24.200334 (+ 86943us) log.cc:844] Serialized 26 byte log entry
0709 12:27:24.346012 (+145678us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:24.346048 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:24.346117 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:25.342171 (+996054us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:25.344127 (+  1956us) raft_consensus.cc:1685] finished
0709 12:27:25.344162 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:25.347077 (+  2915us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":36110,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":50829,"spinlock_wait_cycles":274631424,"thread_start_us":247982,"threads_started":2,"wal-append.queue_time_us":259079}
I20260709 12:27:25.364322 12172 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.364966 12172 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.365578  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:25.369462 12285 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:25.380044  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:25.382164  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:25.383285  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 233722us
I20260709 12:27:25.381003  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:25.387961  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:25.396710  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:25.401296  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:25.359930 12256 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:25.404558  5631 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:25.347942 12152 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.391036 12155 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:25.394075 12236 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.398802 12175 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:25.410125  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47414.8us
I20260709 12:27:25.411252  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:25.411741  5710 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:25.412761  5709 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
W20260709 12:27:25.414793  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 261674us
I20260709 12:27:25.423363  5704 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:25.424894  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:25.424973  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:25.425755  5702 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:25.425983  5631 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:25.426474  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:25.427155 12302 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.363s	user 0.001s	sys 0.003s
I20260709 12:27:25.427922  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:25.428252  5701 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:27:25.430128  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:25.431514  5705 raft_consensus.cc:2410] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 17 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:27:25.435781  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:25.436650  5706 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:25.427330  5703 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:27:25.408792 12282 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:25.447486  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:25.421005 12288 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:27:25.496436  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:25.437029 11997 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:25.445341 12282 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:25.439297 12317 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:27:25.552625  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 124828us
W20260709 12:27:25.555483  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 162939us
I20260709 12:27:25.561420  5709 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:27:25.537645 12288 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:25.547384 12277 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.778s	user 0.010s	sys 0.005s
I20260709 12:27:25.571507  5628 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:27:25.572945  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1166) took 1243 ms. Trace:
I20260709 12:27:25.448108 12236 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.563350  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:27:25.549440  5676 connection.cc:256] RPC call timeout handler was delayed by -1.5715020170000571s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:27:25.552608 12317 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:25.574923  6277 rpcz_store.cc:276] 0709 12:27:24.325017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:24.325170 (+   153us) service_pool.cc:224] Handling call
0709 12:27:25.567708 (+1242538us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b54002a8d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:25.567851 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1385201024}
W20260709 12:27:25.582683  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50507.5us
I20260709 12:27:25.556622 12155 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:25.579643  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:25.547488 12184 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:25.554616 12282 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:25.564069 12172 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:25.573114 12288 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:25.609581  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:25.606308  5710 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
W20260709 12:27:25.583779  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 195903us
I20260709 12:27:25.584955 12317 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:25.612288  5636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2606) took 2406 ms (client timeout 1884 ms). Trace:
I20260709 12:27:25.617949 11663 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:25.618397 11663 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:25.619643  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2589) took 2450 ms (client timeout 1733 ms). Trace:
I20260709 12:27:25.620644 12285 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:25.621277 12294 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.068s	user 0.012s	sys 0.017s
I20260709 12:27:25.621203 12285 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.622064  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:25.622689  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1217) took 1706 ms. Trace:
I20260709 12:27:25.623663  6236 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:25.624503 12039 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.623090  5560 rpcz_store.cc:276] 0709 12:27:23.916176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:24.204386 (+288210us) service_pool.cc:224] Handling call
0709 12:27:24.237827 (+ 33441us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:24.285683 (+ 47856us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:24.285733 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:24.301310 (+ 15577us) log.cc:844] Serialized 26 byte log entry
0709 12:27:24.451763 (+150453us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:24.451966 (+   203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:24.452240 (+   274us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:25.620641 (+1168401us) spinlock_profiling.cc:243] Waited 89 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:25.622373 (+  1732us) raft_consensus.cc:1685] finished
0709 12:27:25.622396 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:25.622624 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2923,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":8458,"spinlock_wait_cycles":204662144,"thread_start_us":120690,"threads_started":1,"wal-append.queue_time_us":238703}
I20260709 12:27:25.624907 12236 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:25.620011  5629 rpcz_store.cc:269] 0709 12:27:23.169483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.255707 (+ 86224us) service_pool.cc:224] Handling call
0709 12:27:25.619442 (+2363735us) spinlock_profiling.cc:243] Waited 678 ms on lock 0x7b54001cfcd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:25.619575 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1560114944}
I20260709 12:27:25.624994 12039 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.625640 12236 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:27:25.626154 12236 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:25.627270 12039 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:25.627032 12236 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.629297  5629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:25.629879  5629 raft_consensus.cc:606] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
W20260709 12:27:25.631628  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52110.8us
W20260709 12:27:25.616897  5636 rpcz_store.cc:269] 0709 12:27:23.205421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.316564 (+111143us) service_pool.cc:224] Handling call
0709 12:27:25.612052 (+2295488us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540032d7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:25.612192 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2397067136}
I20260709 12:27:25.630826  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:25.637741  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:25.638192 12312 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
W20260709 12:27:25.639189 12318 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:27:25.628280 12211 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:25.641760 12211 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:25.629508 11977 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.570s	user 0.013s	sys 0.001s
I20260709 12:27:25.644938 11977 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.647190 11977 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:25.647897 11977 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.579137 12277 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.650029  5552 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:25.650295 12277 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:25.651201 12275 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:25.651841 12275 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:25.652915 12275 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:25.631482 11976 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 12:27:25.657925 12304 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:25.658280 11976 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:27:25.659241 12304 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:27:25.587352 12186 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:25.660851  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:25.660610 12186 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.661816 12184 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:25.662230 12184 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:25.662575 12184 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:25.633224  5706 raft_consensus.cc:2410] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 22 because replica has last-logged OpId of term: 21 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 16.
I20260709 12:27:25.656857 12155 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:25.634557  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:25.623495 12285 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:25.671443 12324 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:25.672269  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:25.673904  5675 leader_election.cc:400] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16.
I20260709 12:27:25.675035  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:25.676457  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 792303us
I20260709 12:27:25.682377  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:25.683408  5560 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:27:25.665983 11997 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:25.672143 12324 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:25.706219  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:25.707118  5558 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:25.665670 12155 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:25.701282  5703 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
W20260709 12:27:25.713218  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 122935us
I20260709 12:27:25.714568  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:25.665591 12172 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 12:27:25.670655 12039 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 12:27:25.716784 12155 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:25.753013 12326 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:27:25.769033  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72503.8us
I20260709 12:27:25.694564 11977 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.770594  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:25.711148 12186 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:25.776208 12319 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:25.777297 12319 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:25.777834 12319 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:25.778894 12319 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:25.780463 12314 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 12:27:25.782846  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1270) took 1279 ms. Trace:
I20260709 12:27:25.783475  5633 rpcz_store.cc:276] 0709 12:27:24.503175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:25.093673 (+590498us) service_pool.cc:224] Handling call
0709 12:27:25.094690 (+  1017us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:25.115734 (+ 21044us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:25.115758 (+    24us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:25.158931 (+ 43173us) log.cc:844] Serialized 52 byte log entry
0709 12:27:25.389487 (+230556us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:25.393064 (+  3577us) log.cc:844] Serialized 14 byte log entry
0709 12:27:25.584599 (+191535us) log.cc:844] Serialized 14 byte log entry
0709 12:27:25.584811 (+   212us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:25.584896 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:25.782596 (+197700us) raft_consensus.cc:1685] finished
0709 12:27:25.782615 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:25.782768 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7589,"prepare.queue_time_us":7694,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":2277,"raft.queue_time_us":51082,"raft.run_cpu_time_us":439,"raft.run_wall_time_us":27836,"spinlock_wait_cycles":4987648,"thread_start_us":221675,"threads_started":2,"wal-append.queue_time_us":75415}
I20260709 12:27:25.785198 12320 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:25.785624 12320 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:25.786566 12320 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.788024 12320 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:25.782742 12334 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:27:25.795825  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:25.801833  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:25.798388 12277 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:25.799062 12100 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.704613 12174 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.059s
W20260709 12:27:25.802961  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:25.770366 12152 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:25.774087 12290 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:27:25.815920  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 67809.8us
W20260709 12:27:25.817258  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 154187us
I20260709 12:27:25.818643  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:27:25.821357  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 164398us
I20260709 12:27:25.805382 12334 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:25.809994 12100 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:25.784245 12317 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:25.826102  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 111675us
W20260709 12:27:25.816139 12120 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.686s	user 0.011s	sys 0.003s
I20260709 12:27:25.835217 12236 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:25.840204 12290 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.841235 12290 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.843662 12290 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:25.841650  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 166665us
I20260709 12:27:25.806577 12277 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:25.815973 12328 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:25.828089 12172 consensus_queue.cc:1243] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 20.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260709 12:27:25.850131  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 73118us
I20260709 12:27:25.860504 12328 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:25.862643  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:25.863024 12291 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.374s	user 0.012s	sys 0.017s
I20260709 12:27:25.863292 12328 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:25.863586  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:25.861186  5710 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:27:25.846866 12290 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:25.866146  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:25.869143  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:25.868088  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:25.870381 12277 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:25.871987  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1149) took 2110 ms. Trace:
I20260709 12:27:25.847683 12186 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.841231 12282 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:25.847591 12193 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 12:27:25.877863 12331 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:25.878327 12331 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:25.878701 12331 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:25.880807 12334 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:25.882341  6276 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:25.885485 12317 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:25.813129 12324 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:25.886577 12324 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:25.888597 12332 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:25.874366  6273 rpcz_store.cc:276] 0709 12:27:23.761953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.762135 (+   182us) service_pool.cc:224] Handling call
0709 12:27:23.781935 (+ 19800us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:24.445252 (+663317us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:25.871602 (+1426350us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b54002ce7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:25.871883 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1507470720}
I20260709 12:27:25.898375  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:25.899217  5704 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:25.845481 12120 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:25.891781 12155 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.873054 12328 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.908383  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:25.908823 12286 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:25.909392 12120 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:25.901136 12290 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:25.903941 12193 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.913154  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:25.903800 12251 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:25.914726  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:25.915673 12317 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.916268  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:25.916527 12330 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.066s
I20260709 12:27:25.911871 12184 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:25.920607 11976 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:25.920960  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:25.922883 12155 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:25.923733  6280 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:27:25.925029  6273 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:25.927364 12289 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:25.932898  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:25.933656  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1271) took 1430 ms. Trace:
I20260709 12:27:25.933612  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:25.934258 11983 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:25.935878 12098 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:25.936111  6272 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:25.938350  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:25.998819  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:25.999302  6279 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:25.937741 12282 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:25.944957  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:25.947281 11977 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:25.952780 12186 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:25.979241  5706 raft_consensus.cc:2376] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23. Re-sending same reply.
W20260709 12:27:25.980875  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:25.981485  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 79132.4us
I20260709 12:27:25.932142 11997 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:25.988526  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:26.023898  5629 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 13.
I20260709 12:27:25.948724 12181 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:26.024662 12181 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:26.025732 12181 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.026783 12181 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:25.984565  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:25.978415 12286 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:25.994392  5634 rpcz_store.cc:276] 0709 12:27:24.503521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:25.159970 (+656449us) service_pool.cc:224] Handling call
0709 12:27:25.177861 (+ 17891us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:25.350254 (+172393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:25.350303 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:25.394898 (+ 44595us) log.cc:844] Serialized 26 byte log entry
0709 12:27:25.445010 (+ 50112us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:25.565450 (+120440us) log.cc:844] Serialized 14 byte log entry
0709 12:27:25.890813 (+325363us) spinlock_profiling.cc:243] Waited 3.81 ms on lock 0x7b180008c4a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ffa228d70 00007f4ffa21c293 00007f4ffa211cea 00007f4ffa21223a 00007f4ffa67ca2a 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:27:25.905137 (+ 14324us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:25.905211 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:25.905263 (+    52us) raft_consensus.cc:1685] finished
0709 12:27:25.933477 (+ 28214us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:25.933604 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":168489,"prepare.queue_time_us":109779,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":2777,"raft.queue_time_us":19463,"raft.run_cpu_time_us":346,"raft.run_wall_time_us":28128,"spinlock_wait_cycles":8771840,"thread_start_us":29321,"threads_started":1,"wal-append.queue_time_us":145898}
W20260709 12:27:25.996448  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 80261.6us
I20260709 12:27:25.997231  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:25.999675  6277 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:25.999607  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:26.035620  5633 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:27:26.035806  5710 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:26.000748  5628 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
W20260709 12:27:26.001394  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 95301.8us
I20260709 12:27:26.001767  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:25.947980  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:25.985826 12277 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:26.004207  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:26.039018  5709 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:26.009106  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:26.039781  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:26.009841 12326 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:26.009749  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:26.013044 12316 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.442s	user 0.003s	sys 0.001s
W20260709 12:27:26.014369  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 101389us
I20260709 12:27:26.017225 12288 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.018553 12330 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.991573 11976 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:25.985616 12193 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:25.988044 12317 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.022320  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:26.024267 12186 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:26.059041  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:25.979458 12328 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.003943 12340 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:27:26.040836  5710 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:26.041059  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:26.088428  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:26.088703  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47728.2us
I20260709 12:27:26.089643  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:26.037497  6274 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
W20260709 12:27:26.044133  5526 connection.cc:256] RPC call timeout handler was delayed by -2.0233386730001257s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1778s for a response
I20260709 12:27:26.090512  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:27:26.090809  5526 connection.cc:256] RPC call timeout handler was delayed by -1.9513386730000093s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18580000000000002s for a response
W20260709 12:27:26.091338  5526 connection.cc:256] RPC call timeout handler was delayed by -1.7985251800000697s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1796s for a response
W20260709 12:27:26.045446  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 60889.5us
I20260709 12:27:26.092936  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:25.998631 12275 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:25.984588 12251 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.050519  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:26.103516  5706 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:26.021814  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1274) took 1070 ms. Trace:
I20260709 12:27:26.107191  5632 rpcz_store.cc:276] 0709 12:27:24.951225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:25.368135 (+416910us) service_pool.cc:224] Handling call
0709 12:27:25.368480 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:25.447308 (+ 78828us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:25.447342 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:25.457286 (+  9944us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:25.457334 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:26.021546 (+564212us) spinlock_profiling.cc:243] Waited 53.5 ms on lock 0x7b54001de2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:26.021600 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:26.021747 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":122988928}
I20260709 12:27:26.055819  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:26.050876  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:26.083009  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:26.109058  6275 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:26.091853  5526 connection.cc:256] RPC call timeout handler was delayed by -1.7569386730001497s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2074s for a response
W20260709 12:27:26.109948  5526 connection.cc:256] RPC call timeout handler was delayed by -1.6444386729999678s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21989999999999998s for a response
I20260709 12:27:26.050599 12172 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:27:26.111853  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:26.096004  5702 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:27:26.094449 12175 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:26.100184  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:26.113113  6276 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:26.052006 12290 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:26.101714  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:26.102560  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:26.097729  5630 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:26.111788  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:26.059428 12326 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:26.116078  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1273) took 1177 ms. Trace:
I20260709 12:27:26.117182 12275 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.111969 12304 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:26.073535 12330 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.121793 12288 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:26.087432 12098 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:26.092021 12337 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:26.100983 12039 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:26.127183 12184 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:26.168668 12039 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:26.151208 11977 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:26.096901 12331 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.169456 12039 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:26.169701 11977 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.170260 12275 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:26.127142  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1168) took 1001 ms. Trace:
I20260709 12:27:26.129930  5636 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 19: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:26.138279  5706 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:26.173187 12305 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:27:26.130745  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:26.111183 12334 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:26.174794 12334 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:26.174928 12337 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.104967 12120 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.072s
W20260709 12:27:26.172467  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:26.175190 12334 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:26.176431  5710 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:26.188043  5709 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:26.146210  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:26.160728  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 78788.6us
W20260709 12:27:26.150758  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 46523.4us
I20260709 12:27:26.172335  5631 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:26.133447  5701 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:27:26.190346  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:26.188938  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:26.189873  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:26.192497  5710 raft_consensus.cc:2410] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 20 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:27:26.196532  6275 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:26.152823 12182 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:26.175184  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:26.102687 12350 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:26.203722  5560 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:26.203424 12350 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.177240  5627 rpcz_store.cc:276] 0709 12:27:24.938469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:25.364444 (+425975us) service_pool.cc:224] Handling call
0709 12:27:25.364891 (+   447us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:25.365311 (+   420us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:25.365345 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:25.365941 (+   596us) log.cc:844] Serialized 26 byte log entry
0709 12:27:25.409569 (+ 43628us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:25.409629 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:25.409721 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:26.012455 (+602734us) spinlock_profiling.cc:243] Waited 6.49 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:26.115758 (+103303us) raft_consensus.cc:1685] finished
0709 12:27:26.115782 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:26.116011 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3255,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"spinlock_wait_cycles":14923008,"thread_start_us":43000,"threads_started":1,"wal-append.queue_time_us":166140}
I20260709 12:27:26.142499 12328 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:26.187453  5704 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:26.205350  5702 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:26.205785  5710 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:27:26.192626  6279 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:26.202644  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:26.190042 12345 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:27:26.171952 12331 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:26.180562  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:26.198554  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 52424.3us
I20260709 12:27:26.199163  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:26.199330  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.128304  6271 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:26.154139 12251 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:26.128505 12175 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:26.146828 12193 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:26.145183 12317 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:26.202117  5707 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:26.193146  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:26.173035  6280 rpcz_store.cc:276] 0709 12:27:25.125287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:25.125424 (+   137us) service_pool.cc:224] Handling call
0709 12:27:25.131599 (+  6175us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:25.992534 (+860935us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:26.126843 (+134309us) spinlock_profiling.cc:243] Waited 775 ms on lock 0x7b54002909d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:26.127052 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1782869248}
I20260709 12:27:26.201437  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.241704  5555 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:26.209450  5561 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 19: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:26.189515 12098 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.212080  5704 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:27:26.213495  5526 leader_election.cc:400] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19.
I20260709 12:27:26.214092 12337 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:26.172010 11977 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:26.221791  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:26.200167 12165 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:26.204865 12332 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:26.210791  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:26.259289  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48562.6us
I20260709 12:27:26.225617 12275 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:26.105078 12277 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.201058 12290 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:26.188725 12282 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
I20260709 12:27:26.248010  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:26.250586  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:26.144272 11663 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.252216  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.252439  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.274592  5555 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:26.189437 12305 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.046s
I20260709 12:27:26.177218 12304 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.255369  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:26.231163 12172 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:26.257467  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.260288  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:26.148000 11976 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.285254  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.288051  5555 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 12:27:26.285426  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:26.264245 12320 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:26.249596 12211 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 12:27:26.266990 12275 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:26.137764 12286 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.295316  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.305404  5561 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
W20260709 12:27:26.298445  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 60397us
W20260709 12:27:26.291952  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 78582.4us
I20260709 12:27:26.249857 12340 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 12:27:26.301834  5554 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:26.314893 12236 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:26.315352 12236 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:26.265941 12332 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.299448 12172 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:26.300382 12320 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:26.302878 12275 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:26.294294 12337 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:26.316323 12236 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.350090 12236 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:26.295361 12205 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:26.352319 12344 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.252s	user 0.001s	sys 0.000s
W20260709 12:27:26.316711  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 133557us
I20260709 12:27:26.317466 12305 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:26.368963 12305 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:26.369422 12305 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:26.370220 12326 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.317945  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:26.373052  5559 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:26.374691 12351 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:27:26.318163 11663 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:26.344050  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.348704  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:26.316512  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 56293us
I20260709 12:27:26.384441  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:26.334892  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 93483.6us
I20260709 12:27:26.363075  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:26.352586  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:26.389863  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:26.392536  5709 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:27:26.334230  5701 raft_consensus.cc:2376] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19. Re-sending same reply.
W20260709 12:27:26.396477  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:26.400780  5631 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
W20260709 12:27:26.402449  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:26.406387  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:26.332482 12339 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:26.275621 12290 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.334461 11976 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:26.319844 12098 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:26.335316 12320 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:26.430864 12334 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.414606  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:26.433418 12358 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:26.433862 12358 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:26.434226 12358 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:26.422188 12285 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.013s
I20260709 12:27:26.423995 12286 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:26.329175 12340 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:26.436862 12340 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:26.334585 12332 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:26.342671 12350 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:26.418021 12165 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.010s
I20260709 12:27:26.334084 12172 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.440858 12330 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:26.410267 12246 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:26.405035 12317 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:26.404152 12357 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:26.415994 12337 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:26.415022 12182 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:27:26.393134 12181 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
I20260709 12:27:26.353693 12205 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.456972  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:26.459429  5556 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:27:26.459532  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.465229  5704 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:27:26.329571 12175 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.409772 12193 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.045s
I20260709 12:27:26.410610 11997 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20260709 12:27:26.437835 12340 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.467173  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:26.480201  5601 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:26.480432  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:26.449923 11983 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:26.484742  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:26.459290 12165 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:26.494640  5554 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:26.460213 12357 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:26.447420 12039 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.510630 12039 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:26.497996 12174 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:26.511775 12174 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:26.512164 12174 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:26.495908 12165 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:26.513294 12165 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.514497 12165 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:26.508496 12184 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:26.516479 12155 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:26.516606 12184 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:26.516861  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:26.508616 12334 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:26.518747  5707 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:26.468336 12246 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:26.454334 12186 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:27:26.479979 12330 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:26.465713 12289 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:27:26.493255 12251 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:26.493839 12337 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:26.529979  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:26.530803  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.539294  5704 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:26.539350  5555 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 12.
W20260709 12:27:26.540287  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:26.532224  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:26.541070  6276 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:26.492988 12277 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:26.542505  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.544071  5554 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:26.544179  5559 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:26.486346 12332 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 14.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
W20260709 12:27:26.599375  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 58317.5us
I20260709 12:27:26.518181 12357 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:26.600786  5555 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 12:27:26.605422  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 109684us
I20260709 12:27:26.614318  6276 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:26.526276 11977 raft_consensus.cc:2764] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election decision vote started in defunct term 15: won
I20260709 12:27:26.517612 12184 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.518869 12324 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.080s
I20260709 12:27:26.638571  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:26.498248 11983 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.545910 12304 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:26.542662 12256 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:26.622973  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:26.595433 12251 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:26.653853  5704 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:26.665933  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:26.671520  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 48669.2us
I20260709 12:27:26.533586 12330 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.679116  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.596261 12337 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.656944 12340 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:26.543267 12361 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.135s
I20260709 12:27:26.689860  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:26.690900  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:26.691030  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:26.692297  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:26.692992  5632 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:26.693207  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:26.693895 12337 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:26.694468  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:26.695305  5633 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 15: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:26.696187  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:26.697075  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:26.700911 12251 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.707116 12290 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:26.676326 12334 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:26.710632 12334 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.681666 12282 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:26.682341 12319 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:26.722590 12319 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.723898  5630 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:27:26.727219  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45097.1us
I20260709 12:27:26.693861  5636 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:26.728750  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:26.719331 12275 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.727109  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:26.680478 12184 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:27:26.745795  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:26.750160  5554 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:26.748611  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:26.788406  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:26.789610  5560 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:26.789881  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:26.790495  5554 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:27:26.744243 11997 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:27:26.739025 12326 raft_consensus.cc:2764] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
W20260709 12:27:26.784758 12294 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:27:26.801615  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.743757 12155 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:26.805296  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:26.732204 12289 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.067s
I20260709 12:27:26.680696 12256 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.818950  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:26.720057 12174 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.814404  5559 raft_consensus.cc:2410] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 20 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:27:26.791252 11997 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:26.744554 12331 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.064s
I20260709 12:27:26.731452 12186 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.078s
I20260709 12:27:26.681847 12246 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.804574 12290 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:26.794559 12286 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:26.797750 12337 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:26.739096 12332 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 12:27:26.802629 12282 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:27:26.840142  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:26.812463 12205 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:26.865218  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 133040us
I20260709 12:27:26.801242 12305 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.839733 12339 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:26.874506  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:26.876591  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.877941  5559 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:27:26.844954 11997 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:26.817380 12155 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.847955 12286 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:26.886693  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:26.854421  5554 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:26.859098  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.887838  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.888078 12282 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:26.889008 12364 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:27:26.889730  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:26.894155 12357 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:26.895162  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:26.897754  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:26.897908 12155 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:26.897980 12320 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.898727  5629 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:26.899225  5636 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:26.888576  5559 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:26.900323 12305 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:26.900622  5630 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:26.901459 12275 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:26.888119  5561 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:26.907922  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:26.905427  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 156744us
I20260709 12:27:26.912701  5633 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:26.876425 12339 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:26.873437 12319 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:26.889040 12334 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:26.926406  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52043.5us
I20260709 12:27:26.886193 12286 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:26.909350 12324 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:26.863571 12337 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.945508  5557 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:26.951633  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:26.983333  5561 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:26.946259  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:26.984719  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:26.958943  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:26.959585  5629 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20260709 12:27:26.962684 11976 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.516s	user 0.008s	sys 0.001s
I20260709 12:27:26.963097 12098 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:26.964322 12320 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:26.931510 12289 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:27:26.966091 12368 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 12:27:26.936619 12186 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:26.926021 12358 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:26.995211  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:26.999850  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:27.000545  5554 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:27:26.991453  5633 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:26.994407  5636 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
W20260709 12:27:26.992609  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46494.6us
I20260709 12:27:27.002326  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:27.004199  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:27.006774  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1142) took 1003 ms. Trace:
I20260709 12:27:27.007332  5705 rpcz_store.cc:276] 0709 12:27:26.003487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:26.003646 (+   159us) service_pool.cc:224] Handling call
0709 12:27:26.998974 (+995328us) spinlock_profiling.cc:243] Waited 595 ms on lock 0x7b54004515d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:27.006684 (+  7710us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1368169856}
I20260709 12:27:27.040663  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:26.972092  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:27.002164 12289 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:27.046974  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45854.4us
I20260709 12:27:26.996994 11977 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:27.053594  5556 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:27.057346  5558 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:27:27.048643  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:27.050832  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:27.058676  5559 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:27.051659  5554 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 15. Current term is 16. Ops: []
W20260709 12:27:27.061546 12371 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
W20260709 12:27:27.063014 12372 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:27:27.071576  6236 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:27.074414 12039 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:27.078310 12039 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:27.080902  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:27.081161 12331 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:27.085675 12331 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:27.086040 12331 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:27.055609 12326 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:27.097383 12326 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:27.097802 12326 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:27.099845 12039 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:27.100749 12039 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:27.102016 12039 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.105775 12256 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:27.106776 12256 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:27.107262 12256 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:27.107684 12256 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:27.010373 12165 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:27:26.998370 12186 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:27.084810  5559 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:27.120121 12288 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:27.120662 12288 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:27.120698 12331 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.121956 12288 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.6, Last appended by leader: 6, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:26.995464 12324 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:26.996860 12098 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.050753 12376 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:27.059914 11977 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:27.131491  5558 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
W20260709 12:27:27.147535  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:27.114699 12165 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:27.118459 12186 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:27:27.129099 12377 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 12:27:27.178548 12357 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:27.180327 12098 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:27.181641  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:27.182562 11663 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.805s	user 0.012s	sys 0.000s
I20260709 12:27:27.183496  5555 raft_consensus.cc:2410] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:27:27.185755 12337 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:27.189507 12175 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:27.190121 12175 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:27.191049 12175 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.193372  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:27.194554 12330 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:27.195590 12358 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:27.203089  5554 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 12. (term mismatch)
I20260709 12:27:27.203722  5554 pending_rounds.cc:85] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 11
I20260709 12:27:27.204041  5554 pending_rounds.cc:107] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 17.12
I20260709 12:27:27.204380  5554 raft_consensus.cc:2889] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:27.178208 12289 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.161468 11977 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:27.162454 12165 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:27.236624  5552 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:27:27.250603  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:27.257525  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:27.258725  6271 raft_consensus.cc:2410] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:27:27.263630  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:27.273499  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:27.265323  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 220067us
I20260709 12:27:27.274320  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:27.251788 12340 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.561s	user 0.009s	sys 0.000s
I20260709 12:27:27.275096  5627 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
W20260709 12:27:27.276615  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:27.277853  6280 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:27.278555  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:27.282334  5708 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:27.279219  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:27.286455  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:27.287117  6275 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 13. (term mismatch)
I20260709 12:27:27.287498  6275 pending_rounds.cc:85] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 12
I20260709 12:27:27.287375  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:27.287822  6275 pending_rounds.cc:107] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 17.13
I20260709 12:27:27.288115  6275 raft_consensus.cc:2889] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:27.288206  6271 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:27:27.289309  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:27.289717 12369 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.457s	user 0.002s	sys 0.000s
I20260709 12:27:27.291730  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:27.292232 12181 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:27.292910  5627 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 13: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:27.293968  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:27.295183  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:27.298039  5706 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 12:27:27.298709  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:27.292810 12181 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:27.299371  6279 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:27.250726 12288 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:27.303399  5627 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:27:27.303617  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:27.209740 12174 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:27.307147  5703 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:27:27.259485 12305 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:27.310616  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:27.261157 12330 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:27.326313  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:27.327278  5552 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:27.328323  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:27.329631  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 357617us
I20260709 12:27:27.332202  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:27.332792  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:27.211915  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:27.316337  5627 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:27.208714 12339 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.343317  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:27.344180  6275 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 19: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:27.343348  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:27.357019  5710 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:27:27.314002 12193 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:27.364053  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:27.274745 12098 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.055s
I20260709 12:27:27.367267  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:27.369253  5630 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:27.321669 12275 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:27.364172  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:27.373070  5627 raft_consensus.cc:2368] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
I20260709 12:27:27.371019  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:27.370929  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:27.373972  5701 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:27:27.323374 12350 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:27.377924  5629 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:27.374931  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:27.375648  6280 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:27:27.375898  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45454us
I20260709 12:27:27.380779  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:27.382014  5706 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 22 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
W20260709 12:27:27.367635 12184 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.582s	user 0.009s	sys 0.002s
W20260709 12:27:27.384425  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:27.343295 12324 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:27:27.384970  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:27.341610 12330 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:27.378750 11663 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:27.333977 12305 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.417148  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:27.377502 12350 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:27.423717  5630 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 20: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:27.425987  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:27.433032  6280 raft_consensus.cc:2368] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
W20260709 12:27:27.422184  6234 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.729s (SENT) [suppressed 25 similar messages]
W20260709 12:27:27.439209  5735 heartbeater.cc:646] Failed to heartbeat to 127.4.65.62:38921 (4 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.4.65.62:38921 timed out after 15.000s (SENT)
I20260709 12:27:27.441835 12384 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:27:27.462625  5552 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:27.419088 12331 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:27.419369 12357 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:27.385018 12275 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:27.381525 12193 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:27.475348  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:27.409986 12376 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:27.420356 12340 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.053s
W20260709 12:27:27.497123  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 155181us
I20260709 12:27:27.428752 11663 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:27.390462 12181 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:27.519950  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 97796.1us
I20260709 12:27:27.523133  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:27.522857  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:27.485057 12184 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:27.537446  5701 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:27.533007 12376 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:27.490489 12175 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:27.540195  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:27.544442  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:27.540902  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:27.549110  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:27.549800  5556 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:27.550324  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:27.555658  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 174995us
I20260709 12:27:27.489015 12357 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.539388 12394 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:27.540163 12331 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:27:27.576215  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:27.565900  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:27.582542  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:27.551548 12361 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:27.592499  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:27.582022 12181 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:27.547258 12184 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.555130 12246 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.043s
I20260709 12:27:27.589071 12358 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:27.580133 12394 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:27.579457 12400 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:27.641757  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:27.633689 12256 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.524s	user 0.014s	sys 0.000s
I20260709 12:27:27.633548 12339 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:27.589439 12326 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.656450  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:27.654381  6280 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e69273530292453d87614627debcc13e"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:27:27.632655 12358 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:27.636545 12394 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:27.639796 12361 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:27.579937 12350 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.663602  5552 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:27.650959 12256 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:27.678858 12256 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:27.666525  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58967.1us
I20260709 12:27:27.736367  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:27.739563  5705 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
W20260709 12:27:27.750769  5524 leader_election.cc:336] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.536s (SENT)
W20260709 12:27:27.700239 11997 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.796s	user 0.009s	sys 0.000s
W20260709 12:27:27.712150 12386 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.282s	user 0.001s	sys 0.001s
I20260709 12:27:27.708714 12350 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:27.711963 12359 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:27:27.665205 12358 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:27.750283 12387 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.235s	user 0.000s	sys 0.001s
I20260709 12:27:27.742190 12415 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:27.749639 12330 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:27.751969 12256 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:27:27.778374  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 228156us
W20260709 12:27:27.779095 12186 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.605s	user 0.012s	sys 0.001s
I20260709 12:27:27.779628 12186 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:27.780563 12401 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 12:27:27.781925 12186 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:27.760982 12165 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.782246  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:27.782958 12397 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.208s	user 0.001s	sys 0.000s
I20260709 12:27:27.783576  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:27.757630 12414 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:27.760057 12275 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:27.768368 12184 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:27.784124 12165 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:27.777347 12389 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
I20260709 12:27:27.766706 12305 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:27.777726 12256 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:27.764896 11997 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.784317  5555 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:27.807822  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2817) took 1578 ms. Trace:
I20260709 12:27:27.823128  5560 rpcz_store.cc:276] 0709 12:27:26.229566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:26.240253 (+ 10687us) service_pool.cc:224] Handling call
0709 12:27:26.258537 (+ 18284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:26.790095 (+531558us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:27.807574 (+1017479us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b54003a93d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:27.807764 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1220658048}
I20260709 12:27:27.809178  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2609) took 1141 ms. Trace:
I20260709 12:27:27.825187  5554 rpcz_store.cc:276] 0709 12:27:26.667523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:27.063488 (+395965us) service_pool.cc:224] Handling call
0709 12:27:27.202599 (+139111us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:27.258958 (+ 56359us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:27.808948 (+549990us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54002288d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:27.809109 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":318824320}
I20260709 12:27:27.811232  6276 raft_consensus.cc:2368] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 13. Current term is 14.
I20260709 12:27:27.830082  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:27.785750 12403 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:27:27.784531 12414 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:27.832470 11977 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.567s	user 0.010s	sys 0.002s
I20260709 12:27:27.832931 12414 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:27.801594 12357 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:27.785885 12286 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.788s	user 0.013s	sys 0.000s
I20260709 12:27:27.798475 12326 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:27.834208 12286 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:27.833372 11977 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.835796  5704 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:27.793562 12394 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:27.808194 12256 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:27.838157 12394 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:27.838405 11983 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:27.838666 12193 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:27.785041 12396 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260709 12:27:27.841202  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:27.842414  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:27.843055  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:27.843200  5559 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:27.832383  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2814) took 1632 ms. Trace:
I20260709 12:27:27.842455  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:27.844772  5560 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:27.844105  5553 rpcz_store.cc:276] 0709 12:27:26.200388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:26.200560 (+   172us) service_pool.cc:224] Handling call
0709 12:27:26.200845 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:26.314390 (+113545us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:27:26.466892 (+152502us) log.cc:844] Serialized 14 byte log entry
0709 12:27:27.813792 (+1346900us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b18001c2d00. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a
0709 12:27:27.814038 (+   246us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:27.814103 (+    65us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:27.814168 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:27.831962 (+ 17794us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b540023f854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:27.832023 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:27.832293 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":192194,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":11997,"raft.queue_time_us":5548,"spinlock_wait_cycles":267474560,"thread_start_us":272910,"threads_started":2,"wal-append.queue_time_us":208705,"wal-append.run_cpu_time_us":1695,"wal-append.run_wall_time_us":1030488}
I20260709 12:27:27.848963  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:27.850476  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:27.851751  5553 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 12:27:27.852727 12331 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:27.853835 12331 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.860005  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:27.861052  5701 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:27.861495  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:27.862141  5707 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:27.867647  5709 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:27.868144  5709 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:27.873904  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:27.874565  5554 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:27.869024  5709 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.866283 11983 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:27.877102  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:27.810345 11997 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:27.863260 12286 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:27.877293  5709 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:27.884984 12288 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.574s	user 0.008s	sys 0.004s
W20260709 12:27:27.886041  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:27.886569  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:27.888528 12181 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:27.889150 12181 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:27.890673 12393 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.261s	user 0.002s	sys 0.000s
I20260709 12:27:27.891817 12181 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:27.894179  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:27.899255  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:27.900084  5554 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:27.903534  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:27.904893  5709 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 16. (term mismatch)
I20260709 12:27:27.906054  5709 pending_rounds.cc:85] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 15
I20260709 12:27:27.906342  5709 pending_rounds.cc:107] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 19.16
I20260709 12:27:27.907146  5709 raft_consensus.cc:2889] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:27:27.912245  5631 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.005s	sys 0.010s
W20260709 12:27:27.915083  5631 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.005s	sys 0.010s
I20260709 12:27:27.913724  5558 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:27:27.915472  5559 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:27:27.916821  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:27.917937  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:27.919625  5559 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:27:27.904875 11983 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:27.921953  5702 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.011s	sys 0.020s
W20260709 12:27:27.931521  5702 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.012s	sys 0.020s
W20260709 12:27:27.935365  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 336701us
I20260709 12:27:27.936725 12286 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:27.937544 12286 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:27.939868  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 93682.5us
W20260709 12:27:27.942358 12237 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.004s	sys 0.023s
W20260709 12:27:27.942788 12237 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.004s	sys 0.024s
I20260709 12:27:27.914837 12039 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:27.944124 12039 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:27.945848 12286 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:27.946712 12039 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:27.953559  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:27.954417  5552 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 17: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:27.955263  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:27.956032  5552 raft_consensus.cc:2410] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 18 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:27:27.957803  5559 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:27:27.939970  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:27.914894 12175 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:27.960224 11983 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:27.977257  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 551264us
I20260709 12:27:27.978325  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:27.965088  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:27.967564 12411 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.291s	user 0.000s	sys 0.002s
W20260709 12:27:27.986510  5557 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2607) took 1319 ms (client timeout 1501 ms). Trace:
W20260709 12:27:27.987005  5557 rpcz_store.cc:269] 0709 12:27:26.663056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:26.696842 (+ 33786us) service_pool.cc:224] Handling call
0709 12:27:27.981776 (+1284934us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54001c48d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:27.981939 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":532809472}
I20260709 12:27:27.968791  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:27.970108  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:27.995174 12361 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:27.995041 12174 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:27:27.994135  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:28.003249  5706 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:27:28.022394  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:28.033548  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:28.034137  5633 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:28.035006  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:28.035540  5635 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:27:28.036394 12337 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.036958 12337 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.037953  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:28.039582 12337 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.039887  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:27.988947 11977 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:28.031180  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:27.981442 12427 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:28.041718  5561 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:27:28.044019  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:28.045400  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 80396.6us
I20260709 12:27:28.046167  5627 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:27.992009 12286 raft_consensus.cc:2764] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:27:28.046933  5526 leader_election.cc:400] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
I20260709 12:27:28.046820  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:28.047789  5526 leader_election.cc:403] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:27:28.047550  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.022145 12155 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.051083  5557 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260709 12:27:28.051478  5557 pending_rounds.cc:85] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 13
I20260709 12:27:28.051859  5557 pending_rounds.cc:107] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260709 12:27:28.033006  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.052168  5557 raft_consensus.cc:2889] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:28.052595  5630 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:28.054469  5630 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:27:28.025599 12246 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.021922 12400 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:28.052284  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:27:28.045140 12411 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:27:27.990314 12324 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.055s
I20260709 12:27:28.056216  5526 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:28.046020 12193 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:28.075733  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 1283) took 1187 ms. Trace:
I20260709 12:27:28.044831 12334 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:27:28.049561  5524 leader_election.cc:336] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.995s (SENT)
I20260709 12:27:28.068138  5633 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:28.046805 12039 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.071792 12246 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.042397 12172 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:28.072558 12425 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:28.072685 12429 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:27:28.073685  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:28.000303 12181 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 12:27:28.059851  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:28.076200  5636 rpcz_store.cc:276] 0709 12:27:26.882049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:26.882228 (+   179us) service_pool.cc:224] Handling call
0709 12:27:28.069461 (+1187233us) spinlock_profiling.cc:243] Waited 8.73 ms on lock 0x7b54001ce8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:28.069584 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20076928}
I20260709 12:27:28.049921  5627 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:28.073148 12317 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.002686 12256 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.089697 12375 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:28.090622 12400 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.041654 12098 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.034s
W20260709 12:27:28.092919  5524 leader_election.cc:336] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.017s (SENT)
I20260709 12:27:28.197067  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:28.093322 12039 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.095559 12246 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.096055 12172 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.087605 12373 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.059022 12275 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:28.099069  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:28.051930 12357 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
W20260709 12:27:28.101205  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 46183.6us
I20260709 12:27:28.215147  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:28.215590 12172 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:28.103685  5635 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:27:28.218725 12256 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:28.219249 12417 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:28.219376 12256 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:28.022207 12361 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:28.220835 12392 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.693s	user 0.003s	sys 0.002s
I20260709 12:27:28.194972 12237 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.223023 12237 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.224215  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2858) took 2048 ms. Trace:
W20260709 12:27:28.224910  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 125908us
I20260709 12:27:28.225322  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:28.225648  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:28.226975  5526 leader_election.cc:400] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
W20260709 12:27:28.230270  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 137380us
W20260709 12:27:28.231374  5524 leader_election.cc:336] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.059s (SENT)
W20260709 12:27:28.156906  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:28.076032 12193 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:28.059799 12155 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.127628 12437 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:28.207945 12205 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:28.211124 12039 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:28.211257 11663 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.211871  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:28.239848  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.249563  5554 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:28.240965  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.190620 12286 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:28.196164 12334 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:28.243309  5635 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:27:28.224609  5628 rpcz_store.cc:276] 0709 12:27:26.176302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:26.648526 (+472224us) service_pool.cc:224] Handling call
0709 12:27:26.664067 (+ 15541us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:26.676987 (+ 12920us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:27:27.813128 (+1136141us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x7b18000d06c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:27:27.839194 (+ 26066us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b18000d06c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:27:27.871585 (+ 32391us) log.cc:844] Serialized 14 byte log entry
0709 12:27:28.189893 (+318308us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:28.195291 (+  5398us) log.cc:844] Serialized 26 byte log entry
0709 12:27:28.199008 (+  3717us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:28.199062 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:28.199126 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:28.224010 (+ 24884us) raft_consensus.cc:1685] finished
0709 12:27:28.224028 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:28.224160 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55996,"prepare.queue_time_us":18536,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":145564,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":11246,"spinlock_wait_cycles":83523200,"thread_start_us":226809,"threads_started":2,"wal-append.queue_time_us":325429}
W20260709 12:27:28.247092  5524 leader_election.cc:336] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.699s (SENT)
I20260709 12:27:28.247815  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:28.246362 12193 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:28.100575 12425 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:28.255133  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.122280 12429 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:28.193361 12330 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.057s
I20260709 12:27:28.260782  5627 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:27:28.248380  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:28.261696  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:28.254511 12437 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:28.265398  5631 raft_consensus.cc:2410] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 12:27:28.196019 12317 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.246377 12205 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:28.250413  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 190642us
I20260709 12:27:28.244014 12417 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:27:28.273597  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 233775us
I20260709 12:27:28.293457  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:28.294945  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:28.243031 12339 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:28.261586 12039 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:28.263353 11663 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:27:28.288110  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:28.267321  5633 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
W20260709 12:27:28.297365  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 49630.3us
I20260709 12:27:28.302444 12334 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:27:28.306557  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:28.228679 12373 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.076723 12175 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.269491 12425 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:28.302052  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:28.270751 12429 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:28.311201  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:28.316540  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:28.320900  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:28.284512 12361 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.264362 12286 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.299532 12039 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:28.284286 12414 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:28.387938  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 71509.7us
I20260709 12:27:28.348459 11983 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:28.364784 12375 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:28.312573 12324 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:28.329022 11977 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:28.328809 12282 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:28.364426 12326 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:27:28.396409  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 127691us
I20260709 12:27:28.313130 12427 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:28.401364 12427 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.380751 12039 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:28.407711  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:28.409319 12317 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:28.409919 12414 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.410429 12442 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.395582 11983 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:28.411398 11983 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:28.411116 12437 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.414806 12402 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:28.385536 12237 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.417208 12446 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.367506 12340 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20260709 12:27:28.329231 12256 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.078s
W20260709 12:27:28.430114  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:28.417869 12446 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.378665  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:27:28.411053 12399 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.699s	user 0.002s	sys 0.001s
I20260709 12:27:28.402876 12282 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:28.442350  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63842.5us
I20260709 12:27:28.443385  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:27:28.445396  5524 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.133s (SENT)
I20260709 12:27:28.449108  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:28.312933 12339 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.422865 12402 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.404522 11977 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.467128 12182 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.003s
I20260709 12:27:28.466992 12402 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.437196 12426 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.087s
I20260709 12:27:28.445868 12375 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:28.473131 12339 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.464313  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:28.473989 12446 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.474771 12446 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:28.475037  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:28.475232 12446 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:28.476022 12339 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:28.476015 12446 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:28.476547 12376 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.004s
I20260709 12:27:28.476859 12402 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:28.477522  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:28.483259  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:28.484143 12375 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:28.484632 12375 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:28.485018 12375 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:28.448407 12324 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:28.489114 12324 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:28.491788 12334 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:28.492648 12181 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.527884  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:28.528662  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:28.492909 12375 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.495074 12286 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:28.496536  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:28.497021 12446 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:28.500738 12339 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:28.445551 12282 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:28.461858 12155 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.411065 12442 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.502099 12444 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:27:28.502286 12039 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.503947  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2868) took 1232 ms. Trace:
I20260709 12:27:28.549050  5632 rpcz_store.cc:276] 0709 12:27:27.271380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:27.271563 (+   183us) service_pool.cc:224] Handling call
0709 12:27:27.272019 (+   456us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:27.279034 (+  7015us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:27.279068 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:27.296634 (+ 17566us) log.cc:844] Serialized 52 byte log entry
0709 12:27:27.451449 (+154815us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:27:27.451491 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:27.451560 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:28.410473 (+958913us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:28.503547 (+ 93074us) raft_consensus.cc:1685] finished
0709 12:27:28.503578 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:28.503866 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7366,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"spinlock_wait_cycles":7400576,"thread_start_us":153993,"threads_started":1,"wal-append.queue_time_us":338904}
I20260709 12:27:28.550705  5556 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:27:28.551014  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:28.551343 12446 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:27:28.551981  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:28.552167  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:28.552655  5627 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:28.553557  6274 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
I20260709 12:27:28.504683 12450 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:27:28.513186  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 67817.1us
I20260709 12:27:28.478968 12383 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:28.526194 11977 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:28.526994 12358 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.750s	user 0.008s	sys 0.004s
I20260709 12:27:28.530807  5630 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:28.531425  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:28.349192 12417 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.141s
I20260709 12:27:28.551560  5632 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:28.556298  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.573446  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:28.536688 12256 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:28.472993 12246 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:28.488016 11983 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.577847  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:28.597252  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:28.598551  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:28.599350  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 252042us
W20260709 12:27:28.603008  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:28.603686  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:28.604713  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:28.604243  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:28.606482  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 320494us
I20260709 12:27:28.607671  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:28.609841  6273 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:28.614291  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.615161  5634 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
W20260709 12:27:28.618182  5524 leader_election.cc:336] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.175s (SENT)
I20260709 12:27:28.618964  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:28.607044  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:28.545574 12181 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.624281  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1158) took 1310 ms. Trace:
I20260709 12:27:28.632248  5705 rpcz_store.cc:276] 0709 12:27:27.313586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:27.330361 (+ 16775us) service_pool.cc:224] Handling call
0709 12:27:27.371583 (+ 41222us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:27.740333 (+368750us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:28.624006 (+883673us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b54002f6054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:28.624218 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":784833920}
I20260709 12:27:28.625864  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:28.630208  6280 raft_consensus.cc:606] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:27:28.622156  6272 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:28.634029  6276 raft_consensus.cc:2368] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23.
I20260709 12:27:28.578161 12256 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:28.576556 12237 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:27:28.640786  5630 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:28.642537  5702 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:28.646152  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.647660  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:28.655762  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:28.656668  5706 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:28.657266  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.660391  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.658041  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:28.660971  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:28.576174 12358 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.668462  5702 raft_consensus.cc:2368] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17.
I20260709 12:27:28.670277  5552 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:27:28.670758  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:28.621277 12334 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:28.673013  5554 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:27:28.645533 12326 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:27:28.677234  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:28.679420  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:28.685688  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:28.644868 12256 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:28.686110  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:28.689857  5630 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:28.608911 12331 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.055s
I20260709 12:27:28.612978 12419 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.052s
I20260709 12:27:28.693826  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:28.698910  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:28.665405 11997 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:28.695705  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.725275  5561 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
W20260709 12:27:28.694952  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 165252us
I20260709 12:27:28.696471  5709 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:28.694950  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:28.701157  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:28.663359 12039 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.727798  5556 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:28.660847 12237 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:28.732689  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:28.733037  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:28.733986  5636 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 20: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:28.736192  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:28.736732  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:28.660518 12455 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:28.737295  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:28.737773  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.724957  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:28.736230  5560 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:28.689273 12450 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.745072  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:28.745752  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 136461us
I20260709 12:27:28.747325  6272 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:27:28.738564  5708 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:28.747303  5630 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:28.746521  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:28.748944  5709 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:27:28.751041  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 150879us
I20260709 12:27:28.731731 11997 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:28.758970  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:28.761087  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:28.761417  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 154075us
I20260709 12:27:28.734350 12383 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:28.772949  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:28.776115  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:28.776795  6271 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:28.774381  5706 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:27:28.775463  6273 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 19. Ops: []
I20260709 12:27:28.776924  5709 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:28.765633  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:28.690428 12334 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.784718 12375 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.785693 12464 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:28.786010 12334 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:28.780084  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.787020  5634 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 24 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:27:28.787845  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:28.788172 12460 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:28.788719 12460 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.788893  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.786206 12464 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:28.789958  5634 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:28.735762 12286 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 12:27:28.809219  5706 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:27:28.797726  5632 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
W20260709 12:27:28.817495  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 66751.5us
I20260709 12:27:28.767638 11997 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:28.737885 12425 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.799014 12460 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:28.864511  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 98975.8us
I20260709 12:27:28.822520  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:28.824936  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:28.825878  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:28.826262  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:28.866642 12458 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:27:28.866955  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:28.867726 12384 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.868515 12400 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:27:28.868257 12384 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.869096 12455 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:28.869488 12455 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:28.869824 12455 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:28.870471 12384 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:28.871685 12460 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.872200 12460 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.834606  5701 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:28.834714  5707 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:27:28.877215  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:28.877453  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:28.836753  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:28.877946  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:28.878073  5703 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:28.867758  5706 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 14. Current term is 15. Ops: [14.10-14.10]
W20260709 12:27:28.840366  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:28.842062  5633 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:28.844496  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:46870 (request call id 0) took 1212 ms. Trace:
I20260709 12:27:28.879868  6280 rpcz_store.cc:276] 0709 12:27:27.635242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:27.641344 (+  6102us) service_pool.cc:224] Handling call
0709 12:27:28.844289 (+1202945us) spinlock_profiling.cc:243] Waited 973 ms on lock 0x7b54000b54d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa66c853 00007f4fff63a60a 00007f4ff8bc07d4 00007f4ff8bc0762 00007f4ff8bc0691 00007f4ff8bc060c 00007f4ff8bbf8c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:28.844394 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2236957568}
I20260709 12:27:28.848161 12275 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:28.863339 12358 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.862365  5705 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:28.821880  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:28.833819  5708 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 19. Ops: []
I20260709 12:27:28.835453  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:28.892261  5710 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:28.878999  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:28.893422  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:28.893826  6273 raft_consensus.cc:2410] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:27:28.879491  6236 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:28.812952 12334 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.750650 12450 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.904877  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:28.905550  5703 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:28.904928  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:28.908457  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:28.909214  5634 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:28.910377  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:28.912173  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:28.912984  5708 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
W20260709 12:27:28.893044 12429 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.538s	user 0.011s	sys 0.000s
I20260709 12:27:28.966745 12429 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.921840  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:28.968780  5707 raft_consensus.cc:2376] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16. Re-sending same reply.
I20260709 12:27:28.969455 12429 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:28.969951  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:28.971217  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:28.972110 12455 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.972715  6275 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 12:27:28.922504  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:28.974573  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:28.975127 12455 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:28.975387  5701 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:28.977847 11983 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.978283  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:28.978904  6277 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:27:28.926416 12193 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.656s	user 0.003s	sys 0.007s
I20260709 12:27:28.903278 12442 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.893819 12275 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:28.927369  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.008491  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:28.916944 12334 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:29.012182  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:28.940277  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.012956  5554 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:28.943822  6280 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:27:28.947479 12465 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:27:28.952729 12464 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:28.958611 12181 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:28.959260  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 79840.8us
I20260709 12:27:28.959350 12457 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:28.960845 12175 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:28.961192 12450 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.960974  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:28.961678  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 200392us
I20260709 12:27:29.026695  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:28.897387 12237 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.897588 12444 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:28.887108 12400 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:28.906478 12470 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:27:28.964396 12383 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:28.965358 12425 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:28.965750 12460 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:28.976300  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:28.984512  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 194406us
I20260709 12:27:29.048254  5701 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:28.972307  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.049572  5707 raft_consensus.cc:2410] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:27:29.050410  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.051455  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:28.997049  5702 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:29.056499  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.001264  5634 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
W20260709 12:27:29.010967  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:29.008704  5708 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:29.014144  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:29.014966  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:29.015300  6277 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:27:29.013142  5706 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:29.016726  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:29.015908  5703 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:29.019760  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:29.060830  5704 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:29.024366  5553 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 12:27:29.062063  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:29.062597  5553 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:27:28.961752 12039 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:27:29.029196  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:29.027751 12457 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:29.058993  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.057313  5702 raft_consensus.cc:2410] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:27:29.057305  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2842) took 1016 ms. Trace:
I20260709 12:27:29.058921  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:28.927377 11997 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:29.037911  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:29.059942  5554 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 14.
I20260709 12:27:29.071179  5707 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:29.071519  5561 rpcz_store.cc:276] 0709 12:27:28.041294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:28.054048 (+ 12754us) service_pool.cc:224] Handling call
0709 12:27:28.054428 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:28.731201 (+676773us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:29.057078 (+325877us) spinlock_profiling.cc:243] Waited 671 ms on lock 0x7b540044e654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:29.057249 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1542317312}
I20260709 12:27:29.042622 12289 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:29.011623 12193 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:29.041877 12181 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:29.078899  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.098665  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:29.087159  5704 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:29.088673  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.078205  5558 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 18: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:27:29.118209  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 80388.6us
I20260709 12:27:29.097389  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:29.119728  5707 raft_consensus.cc:2368] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17.
I20260709 12:27:29.023479 11983 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 12:27:29.064700 12174 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:29.077252 12457 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:29.121397  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.127815  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:29.187915  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.040907 12175 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:29.144344  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1304) took 1093 ms. Trace:
I20260709 12:27:29.190330  5708 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:29.194876  5629 rpcz_store.cc:276] 0709 12:27:28.050626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:28.055895 (+  5269us) service_pool.cc:224] Handling call
0709 12:27:28.056269 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:28.128546 (+ 72277us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:28.128578 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:28.128637 (+    59us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:28.128685 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:29.143985 (+1015300us) spinlock_profiling.cc:243] Waited 72.1 ms on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:29.144055 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:29.144257 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":165773824}
I20260709 12:27:29.151829  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:29.197150  5627 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:29.162714 12373 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:29.169049 12459 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260709 12:27:29.066931 12455 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:29.080756 12460 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:29.180739 12098 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:27:29.188066  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 60320.9us
W20260709 12:27:29.216987  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:29.217039  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.047279 12444 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:29.135090  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:29.219216  5631 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:27:29.139328  5524 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.361s (SENT)
I20260709 12:27:29.148777  6279 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:27:29.110806 12429 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.104s
I20260709 12:27:29.058162 12450 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.126s
I20260709 12:27:29.197664  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:29.224138  5629 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:29.205108  6236 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:29.094264 12470 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:29.184768  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.183642  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.198271 12415 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:29.215143  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:29.234473  5627 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:29.066469 12282 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:29.058702 12400 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:29.197777  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:29.220409  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:29.111069 12289 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:29.117321 12181 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:29.229151  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:29.232014  5702 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:29.209996 12460 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:29.210593 12174 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:29.231633 12331 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:29.218557 12455 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:29.244470  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 105183us
I20260709 12:27:29.248492  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:29.249831  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.265982  5702 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:29.248281  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:29.234486 12256 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.511s	user 0.010s	sys 0.000s
I20260709 12:27:29.257735  5553 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:29.261443  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.261608  5635 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:27:29.263100  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:29.276648  5634 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:29.278057  5634 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:27:29.278498 12175 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:29.279441  5634 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:27:29.238729 12415 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:29.252565 12425 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:29.277559 12470 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:29.253938 12460 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:29.253971 12437 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:27:29.302152  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:29.280551 12350 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:29.309597 12350 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:29.310561 12350 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:29.314034  5701 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:29.293262  5634 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:29.260761 12331 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:29.275108 12193 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:29.329845 12400 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:29.331696 12334 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:29.316747 12437 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:29.283535 12256 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:29.334955 12444 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:29.336687 11997 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:29.389432  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:29.390146  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.345845  5631 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:29.352936  5629 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:29.364251  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:29.356889  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:29.392400  5636 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:29.351063 12331 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:29.397461  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:29.398394  5631 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:29.405000  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:29.405750 12174 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:29.406432  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:29.406893 12289 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:29.407099  5629 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:29.393956 12457 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:29.411636 12457 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:29.412804  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:29.414412  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:29.415259  5634 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:29.365789 12415 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:29.417066 12437 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:29.420717  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.422569  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.360049 12282 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:29.425851 12476 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 12:27:29.428524  5630 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: [16.12-18.13]
I20260709 12:27:29.430141  5630 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:29.432328  5630 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:27:29.408097  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:29.423998  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:29.442065  5708 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:29.461325  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 67640us
I20260709 12:27:29.463290  5704 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 18: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:29.441538  5629 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:29.485289  5631 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
W20260709 12:27:29.479187 12466 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.012s	sys 0.008s
I20260709 12:27:29.533887  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:29.529711  5631 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:27:29.496340  5634 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:29.525565  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:29.500640  5558 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:27:29.526361  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:29.533461 12466 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.012s	sys 0.008s
I20260709 12:27:29.554901  5631 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 10. (term mismatch)
I20260709 12:27:29.555969  5631 pending_rounds.cc:85] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 9
I20260709 12:27:29.557382  5631 pending_rounds.cc:107] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 19.10
I20260709 12:27:29.558470  5631 raft_consensus.cc:2889] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:29.533375 11977 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:29.570500  5707 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:27:29.595938 12402 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:29.596588 12402 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:29.594090  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 131427us
I20260709 12:27:29.594070  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:29.599812 12402 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:29.601178  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:29.601766  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:29.602146  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:29.539532 12326 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:27:29.536105 12282 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 12:27:29.602741 12181 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:29.615228 12256 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:29.601222 12402 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:29.623090 12425 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:29.627900 12331 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:29.602592 12455 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:29.631139 12331 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:29.641425  6272 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:27:29.643673  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:29.644747  5631 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:29.608094 12275 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.588s	user 0.012s	sys 0.000s
I20260709 12:27:29.650375  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:27:29.657308  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:29.663208  5708 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:29.651114 12460 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:29.671558  5631 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:29.671731  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:29.678323  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 288957us
I20260709 12:27:29.675930  5708 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:29.688125  5631 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:27:29.662122 12275 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:27:29.699389 12462 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.855s	user 0.002s	sys 0.000s
I20260709 12:27:29.704185  5710 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:29.768162  5557 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:29.706292  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2626) took 1170 ms. Trace:
I20260709 12:27:29.768716 12275 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:29.769202  5635 rpcz_store.cc:276] 0709 12:27:28.535525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:28.535719 (+   194us) service_pool.cc:224] Handling call
0709 12:27:28.536054 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:29.276224 (+740170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:29.705956 (+429732us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b540032d7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:29.706210 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1668203904}
I20260709 12:27:29.707011 12421 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:27:29.714177  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:29.726616 12205 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:29.733999  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.750910 12415 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:29.759115  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.759711  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:29.786980  5631 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 18: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:29.787464  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:29.757452  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:29.784312  5706 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:29.794449  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:29.795540  5631 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 19: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:29.798115 12205 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:29.805729  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.806511  5705 raft_consensus.cc:2368] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23.
I20260709 12:27:29.806973  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.785699 12421 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:29.808085  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.809319  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.792695 12415 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:29.811225  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53861.1us
I20260709 12:27:29.943435  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2738) took 1011 ms. Trace:
I20260709 12:27:29.935050  5557 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:29.943415  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:29.957406  5560 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:29.942559  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2716) took 1107 ms. Trace:
I20260709 12:27:29.953676  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:29.939340  6276 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1155) took 1411 ms (client timeout 1656 ms). Trace:
I20260709 12:27:29.956135  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:29.913564  5553 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:29.962769  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:29.963232  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:29.964684  5553 raft_consensus.cc:2410] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 20 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:27:29.915843  5703 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:27:29.966089  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1177) took 1090 ms. Trace:
I20260709 12:27:29.876641  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1174) took 1012 ms. Trace:
I20260709 12:27:29.972014  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:29.966439  5703 rpcz_store.cc:276] 0709 12:27:28.875192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:29.915034 (+1039842us) service_pool.cc:224] Handling call
0709 12:27:29.915426 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:29.965797 (+ 50371us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:29.966037 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:27:29.968151  5559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 1264) took 2006 ms (client timeout 2225 ms). Trace:
I20260709 12:27:29.886381  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:29.969226  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:29.980780  5559 rpcz_store.cc:269] 0709 12:27:27.961317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:27.969277 (+  7960us) service_pool.cc:224] Handling call
0709 12:27:29.967976 (+1998699us) spinlock_profiling.cc:243] Waited 752 ms on lock 0x7b54003a89d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:29.968095 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1730313600}
I20260709 12:27:29.956102  5704 rpcz_store.cc:276] 0709 12:27:28.932184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:29.121139 (+188955us) service_pool.cc:224] Handling call
0709 12:27:29.943207 (+822068us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b540026fa54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:29.943357 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":632915840}
I20260709 12:27:29.981793  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:29.960831  6276 rpcz_store.cc:269] 0709 12:27:28.528178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:28.528344 (+   166us) service_pool.cc:224] Handling call
0709 12:27:29.939095 (+1410751us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54002e3954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:29.939266 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":240166144}
W20260709 12:27:29.981134  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 94883.1us
I20260709 12:27:29.958166  5707 rpcz_store.cc:276] 0709 12:27:28.834865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:29.177377 (+342512us) service_pool.cc:224] Handling call
0709 12:27:29.177831 (+   454us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:29.575030 (+397199us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:29.942168 (+367138us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b54003ad4d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:29.942466 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":902395648}
I20260709 12:27:29.845039  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:29.986001  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:29.983911  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:29.991122  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:29.973809  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:29.974035  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2637) took 1346 ms. Trace:
I20260709 12:27:29.992564  5633 rpcz_store.cc:276] 0709 12:27:28.627768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:28.893220 (+265452us) service_pool.cc:224] Handling call
0709 12:27:28.893553 (+   333us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:29.043505 (+149952us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:29.043547 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:29.049865 (+  6318us) log.cc:844] Serialized 52 byte log entry
0709 12:27:29.466160 (+416295us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:29.466213 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:29.954179 (+487966us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b5400438354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:29.954247 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:29.954314 (+    67us) raft_consensus.cc:1685] finished
0709 12:27:29.954335 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:29.973959 (+ 19624us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5631,"raft.queue_time_us":366,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":12930,"spinlock_wait_cycles":343632640,"thread_start_us":352715,"threads_started":1,"wal-append.queue_time_us":533849}
I20260709 12:27:29.877297  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:29.971073  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:29.889629  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:29.995831  5709 raft_consensus.cc:2410] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 17 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:27:29.881099  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1158) took 1121 ms. Trace:
I20260709 12:27:29.887394  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:29.996872  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 109568us
I20260709 12:27:29.997681  5553 raft_consensus.cc:2410] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 17 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:27:29.932590  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:30.000046  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:30.000327  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:29.996336  6275 rpcz_store.cc:276] 0709 12:27:28.759284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:28.759639 (+   355us) service_pool.cc:224] Handling call
0709 12:27:28.799610 (+ 39971us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:28.973125 (+173515us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:29.880829 (+907704us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b5400772754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:29.881011 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":397250432}
I20260709 12:27:30.026991  5555 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:30.001080  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:29.972694 12500 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:29.978433  5710 rpcz_store.cc:276] 0709 12:27:28.864652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:29.159152 (+294500us) service_pool.cc:224] Handling call
0709 12:27:29.159501 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:29.767723 (+608222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:29.876369 (+108646us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b5400324454. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:29.876570 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1252196224}
I20260709 12:27:30.024489  6278 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:27:30.033349  5707 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:29.833021 12326 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:30.030800  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:30.034175  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:27:30.040294  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:29.853945 12437 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:30.070155  6236 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:29.917147 12358 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:30.042675  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:29.870450 12506 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:27:30.048235  5557 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:30.032176 12402 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:29.979915  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:30.081207  5707 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:30.057475  5706 raft_consensus.cc:1217] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Deduplicated request from leader. Original: 16.11->[17.12-17.12]   Dedup: 17.12->[]
W20260709 12:27:30.060070  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 69075.2us
I20260709 12:27:30.081550  5707 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:30.081868  5706 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:27:30.082502  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2722) took 1205 ms. Trace:
I20260709 12:27:29.998751 12286 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:30.000093 12425 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:30.053836 12383 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:29.976325 12485 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:30.023440 12429 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:27:29.836735 12460 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:30.065757 12326 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:30.032888 12319 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:30.045457 12469 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.049s
I20260709 12:27:30.023010 12376 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:30.023304 12499 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:30.106758 12326 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:30.079977  5675 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:29.976131 12373 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.090s
I20260709 12:27:29.972659 12251 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.093s
I20260709 12:27:30.082903  5706 rpcz_store.cc:276] 0709 12:27:28.876777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:29.998318 (+1121541us) service_pool.cc:224] Handling call
0709 12:27:30.057330 (+ 59012us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:30.082274 (+ 24944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:30.082450 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:30.100440 12500 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:27:30.082414 12466 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:30.081625 12437 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:30.080797 12289 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:27:30.033183 11997 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:27:30.134183 12478 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.814s	user 0.003s	sys 0.000s
W20260709 12:27:30.135574 12509 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
I20260709 12:27:30.102335  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1278) took 1049 ms. Trace:
I20260709 12:27:30.104846  5556 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22. Ops: []
W20260709 12:27:30.137017 12479 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.801s	user 0.003s	sys 0.000s
I20260709 12:27:30.045506 12256 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
I20260709 12:27:30.138139  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2635) took 1526 ms. Trace:
I20260709 12:27:30.093467  5707 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.139386  5707 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:30.113646 12499 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:30.141044 12499 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:30.142884 12466 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:30.051786  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:30.093976 12425 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:30.145246 12425 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:30.114457  5703 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:27:30.097806 12426 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:30.149153 12426 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:30.150272 12426 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.095920 12485 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:30.152670 12485 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:27:30.155049 12461 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.280s	user 0.000s	sys 0.004s
W20260709 12:27:30.126129 12378 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.789s	user 0.002s	sys 0.031s
W20260709 12:27:30.156749 12378 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.002s	sys 0.031s
I20260709 12:27:30.156780 12289 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:30.092309  5560 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
W20260709 12:27:30.157963  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 78098.2us
W20260709 12:27:30.159185  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:30.067198  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 177652us
I20260709 12:27:30.109012 12319 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:30.082849 12358 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:30.046093 12193 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 12:27:30.158979  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:30.120286  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:30.160317  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:30.138520  5632 rpcz_store.cc:276] 0709 12:27:28.613668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:28.816185 (+202517us) service_pool.cc:224] Handling call
0709 12:27:28.816962 (+   777us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:28.820615 (+  3653us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:28.820724 (+   109us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:28.896061 (+ 75337us) log.cc:844] Serialized 26 byte log entry
0709 12:27:29.354695 (+458634us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:29.388532 (+ 33837us) log.cc:844] Serialized 14 byte log entry
0709 12:27:29.390612 (+  2080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:29.390685 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:30.122576 (+731891us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:30.137850 (+ 15274us) raft_consensus.cc:1685] finished
0709 12:27:30.137950 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:30.138082 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63312,"prepare.queue_time_us":37940,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":52,"raft.queue_time_us":39318,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":8873,"spinlock_wait_cycles":29789440,"thread_start_us":335649,"threads_started":1,"wal-append.queue_time_us":384812}
I20260709 12:27:30.173058  5707 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 14. (term mismatch)
I20260709 12:27:30.094458 12486 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260709 12:27:30.138952  5558 rpcz_store.cc:276] 0709 12:27:29.053442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:29.571954 (+518512us) service_pool.cc:224] Handling call
0709 12:27:29.572418 (+   464us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:29.572865 (+   447us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:27:29.813254 (+240389us) log.cc:844] Serialized 14 byte log entry
0709 12:27:30.101937 (+288683us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:30.101993 (+    56us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:30.102044 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:30.102114 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:30.102256 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1292,"prepare.queue_time_us":31300,"prepare.run_cpu_time_us":194,"prepare.run_wall_time_us":1332,"thread_start_us":513295,"threads_started":2}
I20260709 12:27:30.175523  5707 pending_rounds.cc:85] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 13
I20260709 12:27:30.190886  5707 pending_rounds.cc:107] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 16.14
I20260709 12:27:30.191323  5707 raft_consensus.cc:2889] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:27:30.180827  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:30.123485 12429 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:30.191920  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:30.194175  5556 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:27:30.195501  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1179) took 1319 ms. Trace:
I20260709 12:27:30.167980  6274 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:30.199267  5555 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:30.172842 12469 raft_consensus.cc:2764] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:27:30.173236 12319 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:30.094475 12421 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.208055  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:30.208901  5554 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:27:30.169391 11997 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:30.219594  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:30.201151 12402 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:30.201251 12457 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:30.232240  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 136193us
I20260709 12:27:30.202374 12350 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:30.237874  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:30.200209  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:30.156303  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:30.251473  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:30.200801 12205 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:27:30.203328 12256 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.035s
I20260709 12:27:30.205164 11977 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:27:30.274471  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:30.281356  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 211283us
I20260709 12:27:30.257236 12175 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:27:30.290042  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 207544us
W20260709 12:27:30.291091  6274 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1185) took 1530 ms (client timeout 1660 ms). Trace:
W20260709 12:27:30.291640  6274 rpcz_store.cc:269] 0709 12:27:28.760177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:28.811798 (+ 51621us) service_pool.cc:224] Handling call
0709 12:27:30.290871 (+1479073us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54005910d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:30.291016 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2887475584}
W20260709 12:27:30.292877  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 122429us
I20260709 12:27:30.286350  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2735) took 1354 ms. Trace:
I20260709 12:27:30.293831  5703 rpcz_store.cc:276] 0709 12:27:28.931433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:29.981631 (+1050198us) service_pool.cc:224] Handling call
0709 12:27:29.981919 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:30.148627 (+166708us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:30.286008 (+137381us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b54004529d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:30.286260 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":304047360}
I20260709 12:27:30.295228  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:30.287250  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:30.231992 12286 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:27:30.267765 12415 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:30.280891  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:30.303452  6236 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:30.257680 12457 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:30.317973  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 117901us
I20260709 12:27:30.268052 12350 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.320699  5557 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:27:30.274573 12500 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
W20260709 12:27:30.321472  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 76660.9us
I20260709 12:27:30.255204 12289 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.255303 12466 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:30.337201  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:30.322759  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:30.352671  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:30.276536 12429 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.312224 11977 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
W20260709 12:27:30.371759 12511 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
W20260709 12:27:30.372959 12515 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
W20260709 12:27:30.373926 12513 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
W20260709 12:27:30.374951 12480 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 12:27:30.380991 12512 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
W20260709 12:27:30.383651 12516 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 12:27:30.384940 12514 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
I20260709 12:27:30.301185 11997 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.340862 12470 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:30.408711 12470 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:30.196085  5701 rpcz_store.cc:276] 0709 12:27:28.875695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:29.920052 (+1044357us) service_pool.cc:224] Handling call
0709 12:27:29.934764 (+ 14712us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:29.935028 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:27:29.939173 (+  4145us) log.cc:844] Serialized 14 byte log entry
0709 12:27:30.195106 (+255933us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:30.195168 (+    62us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:30.195218 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:30.195291 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:30.195416 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":345,"prepare.queue_time_us":268,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":590,"thread_start_us":254938,"threads_started":1}
I20260709 12:27:30.410022 12470 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.410964  5701 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:27:30.412174 12358 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:30.335482 12486 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.064s
I20260709 12:27:30.339977 12525 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.089s
I20260709 12:27:30.398856 12437 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:30.428037  5701 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:27:30.442581 12350 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:30.395987 12499 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:30.471707  5701 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:27:30.475674  5706 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:30.479183 12421 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:30.489861 12499 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:30.491010 12523 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:30.491627 12523 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.493754 12464 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:30.494172 12464 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:30.471539 12485 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:30.495066 12464 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:30.496825 12485 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:30.497556 12523 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:30.503947 12464 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:30.502529 12326 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.504048 12425 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:30.506451 12457 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:30.508240  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 528365us
I20260709 12:27:30.508988 12442 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:30.500713  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1305) took 1477 ms. Trace:
I20260709 12:27:30.512557 12251 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:30.516749 12251 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:30.517693 12251 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:30.516640  5630 rpcz_store.cc:276] 0709 12:27:29.023417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:29.435892 (+412475us) service_pool.cc:224] Handling call
0709 12:27:29.436272 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:29.439172 (+  2900us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:27:29.753463 (+314291us) log.cc:844] Serialized 14 byte log entry
0709 12:27:30.061742 (+308279us) log.cc:844] Serialized 14 byte log entry
0709 12:27:30.413245 (+351503us) spinlock_profiling.cc:243] Waited 9.46 ms on lock 0x7b18000aed00. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:27:30.500127 (+ 86882us) spinlock_profiling.cc:243] Waited 4.1 ms on lock 0x7b18000aed00. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:27:30.500297 (+   170us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:30.500377 (+    80us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:30.500435 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:30.500513 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:30.500651 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5587,"prepare.queue_time_us":106308,"prepare.run_cpu_time_us":320,"prepare.run_wall_time_us":325,"raft.queue_time_us":443122,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":214,"spinlock_wait_cycles":31184512,"thread_start_us":620348,"threads_started":2,"wal-append.queue_time_us":373305}
I20260709 12:27:30.519127 12251 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:30.531594  5705 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:30.534789  5559 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:30.534664  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2894) took 1217 ms. Trace:
I20260709 12:27:30.517518 12455 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:27:30.549494  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 288722us
W20260709 12:27:30.567994  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:30.572235  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:30.538347 12402 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:30.533423 12337 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:30.581936  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:30.579677  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:30.533711 12488 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:30.583181  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 288301us
W20260709 12:27:30.540439  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:30.586678  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:30.591053  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:30.607684  5561 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:30.592849  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:30.534375 12485 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:30.604743  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:30.610957  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:30.545003 12455 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:30.634977  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:30.644570  5556 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:30.631289 12526 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:27:30.534776 12319 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:30.684072 12319 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:30.611913  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:30.690064  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 78278.4us
I20260709 12:27:30.691826  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:30.676807  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:30.615845 12536 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:30.647284 12181 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:30.615645  5554 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:27:30.704453  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:30.648754 12337 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:30.613126 12488 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:30.601186 12402 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:30.722139  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:30.628327 12485 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:30.612468 12282 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.068s
W20260709 12:27:30.716104  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 69451.4us
I20260709 12:27:30.612207 12442 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.065s
W20260709 12:27:30.727432  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:30.730046  5602 leader_election.cc:336] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.501s (SENT)
W20260709 12:27:30.720461  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 469068us
W20260709 12:27:30.605325  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 299017us
W20260709 12:27:30.740939 12539 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:27:30.747238  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:30.723637 12467 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:30.755788  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:30.757215  5561 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:30.717275 12542 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:30.785058  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:30.788275  5554 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 14.
I20260709 12:27:30.714375 12536 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.723618 12181 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.785010 12319 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:30.807879  5561 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:27:30.743767 12466 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:27:30.809971 12488 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:30.811110  5555 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:30.812196 12536 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:30.753829 12455 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:30.813203 12540 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:27:30.814574 12326 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:30.815006 12425 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:30.764303 12470 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20260709 12:27:30.819563 12485 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:30.845196  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:30.830197  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 106116us
W20260709 12:27:30.833730  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45612.3us
I20260709 12:27:30.833235 12319 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:30.847529  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:30.848309  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:30.838912 12175 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:30.854575 12181 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:30.864892  5556 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:30.856817 12361 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:30.849187 12534 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.017s
I20260709 12:27:30.839404 12373 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.024s
I20260709 12:27:30.872869 12550 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:30.875703  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:30.851220 12319 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:30.877720  5556 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:30.864152 12175 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:30.870262 12361 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:30.867415 12485 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:30.892823  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:30.895026  5557 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 18: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:30.893419 12470 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:30.919770  5524 connection.cc:256] RPC call timeout handler was delayed by -1.5588356109997221s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19510000000000002s for a response
W20260709 12:27:30.920241  5524 connection.cc:256] RPC call timeout handler was delayed by -1.5424683629998981s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1696s for a response
I20260709 12:27:30.922753 12470 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.925477 12470 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:30.928807 12361 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.930020 12361 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:30.934880  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2975) took 1760 ms. Trace:
I20260709 12:27:30.935325 12455 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:30.880534 12550 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:30.935499  5705 rpcz_store.cc:276] 0709 12:27:29.174519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:29.807067 (+632548us) service_pool.cc:224] Handling call
0709 12:27:30.934576 (+1127509us) spinlock_profiling.cc:243] Waited 556 ms on lock 0x7b5400278654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:30.934769 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1278364032}
I20260709 12:27:30.941699  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:30.942545  5705 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 18: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:30.943339  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:30.943923  5705 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:30.944866  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:30.946179 12548 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:30.961277 12548 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:30.963045 12444 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:30.963598 12444 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:30.965634  5636 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:30.968328  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:30.969225  5632 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:27:30.970453  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2970) took 1797 ms. Trace:
I20260709 12:27:30.970866  5704 rpcz_store.cc:276] 0709 12:27:29.173324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:30.028463 (+855139us) service_pool.cc:224] Handling call
0709 12:27:30.028914 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:30.029192 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:27:30.213797 (+184605us) log.cc:844] Serialized 14 byte log entry
0709 12:27:30.969944 (+756147us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b180048ee00. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:27:30.970112 (+   168us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:30.970179 (+    67us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:30.970225 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:30.970289 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:30.970397 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25614,"prepare.queue_time_us":277491,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":2382,"spinlock_wait_cycles":8692608,"thread_start_us":232046,"threads_started":2,"wal-append.queue_time_us":226980}
I20260709 12:27:30.970866  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:30.972465  5636 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:30.972913  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:30.973223 12550 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:30.973417 12256 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:30.973743  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:30.974157 12256 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:27:30.974824 12256 consensus_queue.cc:1243] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 22.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:27:30.973412  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:30.975454 12256 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:30.975833 12256 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:30.976965 12521 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:30.977415 12521 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:30.978376 12521 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:30.979203 12444 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:30.975845  5632 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:30.962978  5630 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:27:30.980906  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:30.982090  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:30.973699  5705 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 18: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:27:30.986006  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:30.986516  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:30.986793  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:30.987495  5633 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:30.989740  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:30.991494  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:31.009399  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 87355.4us
I20260709 12:27:30.979737 12521 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:30.985059  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:31.017059  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:31.017545  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 694848us
W20260709 12:27:31.018137  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 665515us
W20260709 12:27:31.026285  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:31.043853  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:31.046324  5632 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:27:30.961822 12548 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:31.048447  5633 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:31.070860  5628 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
W20260709 12:27:31.053987  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:31.053391  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:31.073292  5636 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:27:31.019073 12491 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:31.045725 12488 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:31.088006  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:31.090838  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:31.092429  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:31.092494  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1180) took 2216 ms. Trace:
I20260709 12:27:31.094950  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1189) took 1013 ms. Trace:
I20260709 12:27:31.093740  5708 rpcz_store.cc:276] 0709 12:27:28.875921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:29.976566 (+1100645us) service_pool.cc:224] Handling call
0709 12:27:29.976875 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:29.977090 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:27:30.039974 (+ 62884us) log.cc:844] Serialized 14 byte log entry
0709 12:27:30.539740 (+499766us) log.cc:844] Serialized 14 byte log entry
0709 12:27:31.092021 (+552281us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:31.092100 (+    79us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:31.092179 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:31.092280 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:31.092433 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":769578,"prepare.queue_time_us":67710,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":7548,"raft.queue_time_us":695906,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"thread_start_us":461489,"threads_started":2,"wal-append.queue_time_us":384156}
I20260709 12:27:31.099895  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:31.100800  5628 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:31.047616 12378 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:31.095351  5701 rpcz_store.cc:276] 0709 12:27:30.081419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:30.473059 (+391640us) service_pool.cc:224] Handling call
0709 12:27:30.511370 (+ 38311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:30.511645 (+   275us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:27:30.527824 (+ 16179us) log.cc:844] Serialized 14 byte log entry
0709 12:27:31.094604 (+566780us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:31.094660 (+    56us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:31.094704 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:31.094771 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:31.094889 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":514744,"prepare.queue_time_us":15556,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":6245,"thread_start_us":49605,"threads_started":1,"wal-append.queue_time_us":89264}
I20260709 12:27:31.107182  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:31.108033  5557 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:27:31.070355 12522 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:31.075415 12373 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:31.126371  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:31.115399  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 194606us
I20260709 12:27:31.127223  6272 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
W20260709 12:27:31.128343  5524 connection.cc:256] RPC call timeout handler was delayed by -1.449768363000203s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1799s for a response
W20260709 12:27:31.128927  5524 connection.cc:256] RPC call timeout handler was delayed by -1.4110683630001404s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18420000000000003s for a response
I20260709 12:27:31.088090 12499 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:31.047526 12181 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.059s
I20260709 12:27:31.093554 12488 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:31.152594  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:31.092998 12456 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:31.157068  5701 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:31.084218 12533 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:31.157958  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:31.159456  5557 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:31.160879  5556 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:31.129624 12378 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:31.151325 12499 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:31.164470 12499 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:31.050689 12256 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:31.166321 12256 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:31.165524 12499 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:31.169133  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:31.170030  5701 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:31.111757 12522 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:31.190928  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 147973us
I20260709 12:27:31.109477 12532 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:27:31.162658 12540 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:27:31.202044  5631 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 12:27:31.198951  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 155174us
I20260709 12:27:31.209605  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:31.213761  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:31.214403  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:31.214984  5556 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:27:31.216869  5701 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:31.217284  5701 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 14 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:31.179044 12550 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:31.223205  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:31.182375 12488 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:31.190321 12376 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:31.186182 12475 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:31.097291 12491 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:31.150534 12373 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:31.222020  5701 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:31.268312  5701 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:31.185819 12175 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:31.276952  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 85187.5us
I20260709 12:27:31.246335 12460 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:31.265182 12528 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:31.260753 12376 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:31.197427 12486 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.071s
I20260709 12:27:31.279590  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2973) took 2105 ms. Trace:
I20260709 12:27:31.240557 12523 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 12:27:31.215499 12466 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.059s
I20260709 12:27:31.285351  5552 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:27:31.317227  5710 rpcz_store.cc:276] 0709 12:27:29.174041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:30.073037 (+898996us) service_pool.cc:224] Handling call
0709 12:27:30.073451 (+   414us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:30.280586 (+207135us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:27:30.336273 (+ 55687us) log.cc:844] Serialized 14 byte log entry
0709 12:27:31.084172 (+747899us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:31.084811 (+   639us) log.cc:844] Serialized 26 byte log entry
0709 12:27:31.085006 (+   195us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:31.085052 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:31.085158 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:31.279364 (+194206us) raft_consensus.cc:1685] finished
0709 12:27:31.279385 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:31.279528 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":684833,"prepare.queue_time_us":74516,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":637384,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":17143,"thread_start_us":391507,"threads_started":2,"wal-append.queue_time_us":408961}
I20260709 12:27:31.322930  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:31.321346  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:31.327752  6273 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:31.261628 12467 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
W20260709 12:27:31.318722 12553 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:27:31.189240 12456 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:31.277812 12485 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:31.319180  5701 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
I20260709 12:27:31.345072  5701 pending_rounds.cc:85] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 10
I20260709 12:27:31.345396  5701 pending_rounds.cc:107] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 12:27:31.345626  5701 raft_consensus.cc:2889] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:31.271672 12572 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:31.281239 12426 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:27:31.298761  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:31.349820  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:31.262202 12550 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:31.298363  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 50159.6us
I20260709 12:27:31.270015 12475 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:31.320094 12528 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:31.355127  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:31.356796 12475 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:31.356988 12528 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:31.357432 12475 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:27:31.358500 12548 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:31.360442 12373 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:31.287446 12286 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:27:31.361550 12286 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:27:31.297662 12532 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:31.363992 12475 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:27:31.363713 12532 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:31.364524 12475 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:31.368892 12550 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:31.369298 12475 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:31.346834  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:31.332690 12326 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:31.367106 12456 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:31.370540  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 161072us
W20260709 12:27:31.371062  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 179357us
I20260709 12:27:31.371381  5701 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:27:31.356822  5710 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:27:31.365098 12574 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:27:31.381134 12564 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
W20260709 12:27:31.376940 12571 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:27:31.385882 12574 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:27:31.387841 12550 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:31.388491  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:31.347280 12579 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:31.396281  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:31.350445 12572 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:31.397848 12572 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:31.398198 12337 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:31.398768 12337 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:31.350597 12485 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:31.374796 12378 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:31.401216 12337 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:31.401829  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:31.402297 12485 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:31.353246 12426 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:31.403015 12426 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:31.334151 12402 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:27:31.403990 12402 consensus_queue.cc:1243] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 16.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:27:31.395341 12579 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:31.409611 12566 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:27:31.402573  5710 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
W20260709 12:27:31.410802 12567 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
I20260709 12:27:31.410588 12579 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:31.327909 12569 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.070s
I20260709 12:27:31.447185  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:31.474609  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:30.535444  5634 rpcz_store.cc:276] 0709 12:27:29.317669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:29.553703 (+236034us) service_pool.cc:224] Handling call
0709 12:27:29.559974 (+  6271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:29.560642 (+   668us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:27:29.806334 (+245692us) log.cc:844] Serialized 14 byte log entry
0709 12:27:30.533746 (+727412us) log.cc:844] Serialized 14 byte log entry
0709 12:27:30.534121 (+   375us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:30.534225 (+   104us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:30.534305 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:30.534406 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:30.534597 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":338561,"prepare.queue_time_us":53532,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":340672,"thread_start_us":624687,"threads_started":2,"wal-append.queue_time_us":486827}
I20260709 12:27:31.412207  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:31.449301  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:31.491170  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:31.478132  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:31.433290  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2972) took 2259 ms. Trace:
I20260709 12:27:31.438165  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2990) took 1242 ms. Trace:
I20260709 12:27:31.500437  5705 rpcz_store.cc:276] 0709 12:27:30.195251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:31.021056 (+825805us) service_pool.cc:224] Handling call
0709 12:27:31.051315 (+ 30259us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:31.190788 (+139473us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:31.190829 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:31.190886 (+    57us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:31.190938 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:31.437811 (+246873us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400075154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:31.437877 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:31.438085 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":320140672}
I20260709 12:27:31.489467  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:31.506016  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:31.511904  5634 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:27:31.488765  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:31.421610 12561 log.cc:927] Time spent T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:27:31.497885  5709 rpcz_store.cc:276] 0709 12:27:29.173807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:30.027132 (+853325us) service_pool.cc:224] Handling call
0709 12:27:30.038463 (+ 11331us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:30.052843 (+ 14380us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:27:30.279036 (+226193us) spinlock_profiling.cc:243] Waited 6.49 ms on lock 0x7b180048ef80. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:27:30.611714 (+332678us) log.cc:844] Serialized 14 byte log entry
0709 12:27:31.222290 (+610576us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:31.242047 (+ 19757us) log.cc:844] Serialized 52 byte log entry
0709 12:27:31.242291 (+   244us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:31.242346 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:31.242455 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:31.432721 (+190266us) raft_consensus.cc:1685] finished
0709 12:27:31.432894 (+   173us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:31.433200 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":572990,"prepare.queue_time_us":263431,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":46,"raft.queue_time_us":564240,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":19188,"spinlock_wait_cycles":14928128,"thread_start_us":366256,"threads_started":2,"wal-append.queue_time_us":107053}
W20260709 12:27:31.525252  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:31.514370  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:31.543630  5710 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 18: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:31.543985  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:31.374401 12326 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:31.553335  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:31.550191  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:31.412163 12579 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:31.558960  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:31.424782 12491 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:31.566603  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:31.419708 12175 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:31.436619 12537 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:31.467691 12548 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:31.568476  5630 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 14.
I20260709 12:27:31.574393 12580 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:31.575037  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:31.569944  6273 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:27:31.577517  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 311075us
W20260709 12:27:31.579068  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:31.579439  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2989) took 1384 ms. Trace:
I20260709 12:27:31.583250  5704 rpcz_store.cc:276] 0709 12:27:30.194992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:31.009107 (+814115us) service_pool.cc:224] Handling call
0709 12:27:31.009436 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:31.009647 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:27:31.039240 (+ 29593us) log.cc:844] Serialized 14 byte log entry
0709 12:27:31.298565 (+259325us) log.cc:844] Serialized 14 byte log entry
0709 12:27:31.425178 (+126613us) spinlock_profiling.cc:243] Waited 9.03 ms on lock 0x7b180048f100. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:27:31.579087 (+153909us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:31.579150 (+    63us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:31.579196 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:31.579261 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:31.579383 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":166478,"prepare.queue_time_us":43562,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":78,"raft.queue_time_us":202424,"raft.run_wall_time_us":3569,"spinlock_wait_cycles":53324800,"thread_start_us":364973,"threads_started":2,"wal-append.queue_time_us":230301}
I20260709 12:27:31.475266 12485 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.099s
I20260709 12:27:31.563134 12326 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:31.565400 12193 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:31.444772 12550 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:31.570662 12275 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:31.603116  5632 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:27:31.603834  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:31.605325  5706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2987) took 1411 ms (client timeout 1699 ms). Trace:
I20260709 12:27:31.430560 12319 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:31.602388  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:31.424290 12572 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:31.620476  5706 rpcz_store.cc:269] 0709 12:27:30.194380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:30.197121 (+  2741us) service_pool.cc:224] Handling call
0709 12:27:31.605084 (+1407963us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b54002f4c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:31.605253 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":578253184}
I20260709 12:27:31.628412  5630 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:27:31.632382  5636 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:27:31.575006 12531 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.064s
I20260709 12:27:31.607581 12580 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:31.579036 12181 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.047s
I20260709 12:27:31.648286  5629 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20. Ops: []
W20260709 12:27:31.643055  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:31.613979 12337 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:31.623445 12275 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:31.631469 12455 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.695s	user 0.011s	sys 0.003s
I20260709 12:27:31.593644 12537 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:31.630919 12574 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:27:31.660528  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 306164us
I20260709 12:27:31.598479 12289 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.049s
I20260709 12:27:31.594139 12548 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:31.683619  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 924362us
I20260709 12:27:31.684327 12275 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:31.652863 12337 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:31.696470  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:31.719290  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 230628us
W20260709 12:27:31.721119  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 394849us
I20260709 12:27:31.685321 12574 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:31.724783  5524 connection.cc:256] RPC call timeout handler was delayed by -1.0681683630000407s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2223s for a response
W20260709 12:27:31.688299 12256 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.521s	user 0.009s	sys 0.005s
I20260709 12:27:31.734444  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:31.719331  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 228232us
W20260709 12:27:31.734576  5524 connection.cc:256] RPC call timeout handler was delayed by -1.0099334579999777s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2223s for a response
I20260709 12:27:31.737313  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:31.737771  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:31.738647  5704 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:31.740998  5704 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:27:31.742806  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:31.743026  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:31.743604  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:31.747010  5706 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 23 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:27:31.748306  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:31.721706 12337 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:31.752566  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:31.756636  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:31.758044  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:31.725158 12532 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:31.701720 12486 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.051s
I20260709 12:27:31.740123 12574 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:31.698211 12569 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.050s
I20260709 12:27:31.733554 12455 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:31.725555 12456 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 12:27:31.737035 11977 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:31.750547 12521 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:27:31.766249  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:31.738070 12531 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:31.754180 12256 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:31.776273  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:31.789912  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:31.761677 12425 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:31.760237 12541 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 12:27:31.756059 12491 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
W20260709 12:27:31.813128  5524 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.500s (SENT)
I20260709 12:27:31.750840 12570 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.050s
I20260709 12:27:31.813935  5675 leader_election.cc:400] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23.
I20260709 12:27:31.773979 12536 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:31.818419  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:31.817262  5524 leader_election.cc:336] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.732s (SENT)
I20260709 12:27:31.807200 12181 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:31.806185 12442 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:27:31.869647  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52425.3us
I20260709 12:27:31.882579  6273 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260709 12:27:31.885111  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2860) took 1834 ms. Trace:
W20260709 12:27:31.885847  5524 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.956s (SENT)
I20260709 12:27:31.886431  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:31.886861  5559 rpcz_store.cc:276] 0709 12:27:30.050559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:30.054827 (+  4268us) service_pool.cc:224] Handling call
0709 12:27:30.056215 (+  1388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:30.535407 (+479192us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:31.823178 (+1287771us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b54001bd354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:31.885055 (+ 61877us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1100099840}
I20260709 12:27:31.814512 12517 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:31.779522 12455 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:31.897449  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:31.899299  6271 raft_consensus.cc:2410] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 20 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 15.
W20260709 12:27:31.899663  5524 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.083s (SENT)
I20260709 12:27:31.918658  6280 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:31.921182  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:31.922056  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:31.922924  5556 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:31.923094  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:31.922111  5560 raft_consensus.cc:2410] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:27:31.923960  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:31.924525 12584 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
I20260709 12:27:31.924744  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:31.924947  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:31.925127  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:31.925854  5553 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:31.926236  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:31.926827  5557 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:31.927016 12586 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:31.927538 12586 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:31.931370 11977 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:31.932102 12581 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260709 12:27:31.931999 11977 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:31.933713  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1183) took 2171 ms. Trace:
I20260709 12:27:31.934172  5707 rpcz_store.cc:276] 0709 12:27:29.762401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:30.304213 (+541812us) service_pool.cc:224] Handling call
0709 12:27:30.316749 (+ 12536us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:31.181971 (+865222us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:27:31.199815 (+ 17844us) log.cc:844] Serialized 14 byte log entry
0709 12:27:31.505201 (+305386us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:31.517047 (+ 11846us) log.cc:844] Serialized 26 byte log entry
0709 12:27:31.517214 (+   167us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:31.517250 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:31.517316 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:31.933409 (+416093us) raft_consensus.cc:1685] finished
0709 12:27:31.933443 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:31.933638 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":864791,"prepare.queue_time_us":17665,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":869303,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":45841,"thread_start_us":282738,"threads_started":1,"wal-append.queue_time_us":486932}
I20260709 12:27:31.945417 12595 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:31.946105 12595 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:31.947732 12499 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:31.950899 12587 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.261s	user 0.001s	sys 0.000s
I20260709 12:27:31.951990 12500 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:31.952567 12500 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:31.954418 12585 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.256s	user 0.001s	sys 0.000s
I20260709 12:27:31.954772 12500 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:31.958084  5524 leader_election.cc:400] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23.
I20260709 12:27:31.958606  5524 leader_election.cc:403] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:27:31.805357 12531 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:31.945631 12464 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:31.884300 12529 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:31.878034 12599 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:31.865963 12588 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:31.891202 12541 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:27:31.974418  5599 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.679s (SENT)
I20260709 12:27:31.883896 12181 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:31.983294  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:31.958108 12491 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:31.960036 12550 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:31.965369 12572 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:31.966814 12464 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:31.948297 12499 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:31.934288 11977 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:32.018719  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:31.964259 12595 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:31.951220 12376 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:31.973343 12529 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:32.023628  5524 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.141s (SENT)
I20260709 12:27:31.977198 12599 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:31.811034 12580 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:31.979851 12541 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:31.989871 12181 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:32.034117  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:32.044849  5524 leader_election.cc:336] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.204s (SENT)
W20260709 12:27:32.017701 12426 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.602s	user 0.012s	sys 0.000s
I20260709 12:27:31.977506 12532 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:32.002243 12521 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:32.017481 12537 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:31.991168 12517 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.050s
I20260709 12:27:32.035058 12548 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:32.042353 12596 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:27:32.089210  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:32.116899  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:32.063570 12376 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:32.051471 12491 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:27:32.117942  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:32.118649  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 300305us
I20260709 12:27:32.016634 12326 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.119621  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:32.124588  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:32.125367 12609 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:27:32.125658  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 2992) took 1873 ms (client timeout 1679 ms). Trace:
I20260709 12:27:32.126768 12609 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:32.127355 12609 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:27:32.126818  5702 rpcz_store.cc:269] 0709 12:27:30.252622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:30.286832 (+ 34210us) service_pool.cc:224] Handling call
0709 12:27:32.125416 (+1838584us) spinlock_profiling.cc:243] Waited 853 ms on lock 0x7b540006efd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:32.125564 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1962674432}
I20260709 12:27:32.073448 12541 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.134491 12541 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:32.137212 12499 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.139446 12596 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:27:32.140161 12596 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:32.140457 12605 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:32.120324 12529 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.141361 12589 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23.
I20260709 12:27:32.141832 12529 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:32.141333 12596 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.140031 12609 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.154073  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:32.158567  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:32.159273 12548 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:32.161000 12522 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.114178 12537 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:32.163376 12537 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.164767  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.165650 12537 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:32.166007 12610 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:32.166560 12610 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:32.167836 12601 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:27:32.168941 12610 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.123102 12532 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.6, Last appended by leader: 6, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:32.175145  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:32.175310  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.176401  5705 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:32.177690  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.166576  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.178515  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:32.178673  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:32.179852  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:32.115221 12579 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.539s	user 0.011s	sys 0.002s
I20260709 12:27:32.051759 12426 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.179721  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:32.193364  5706 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:32.194711  5710 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:27:32.197685  5702 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:32.198645  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.199488  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.200932  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:32.141870 12607 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:27:32.205467  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.208565  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.062464 12464 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.188441 12610 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:32.187570 12286 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:32.187820 12499 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:32.231101 12326 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.232071 12286 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.236222 12586 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.237991 12455 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.218315 12595 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:32.247052  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.247953  5706 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:32.249290  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.204314 12550 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:32.253154 12550 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.213817  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.219489 12604 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:27:32.263705  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:32.237353 12286 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:32.274528 12606 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.209s	user 0.000s	sys 0.002s
I20260709 12:27:32.297137  5707 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:32.265566 12609 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:32.302240  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.320604  5557 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:32.320607  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:32.322747  5553 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:32.323760  5704 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:32.330159  5553 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
W20260709 12:27:32.333128  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 153486us
I20260709 12:27:32.265290 12499 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.265290 12610 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:32.340507  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.341873  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:32.341907  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:32.345448  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.348518  5706 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:32.349413  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:32.347536  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.350131  5561 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:32.350484  5704 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:27:32.329793 12455 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:32.202924 12599 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.355919  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:32.355458  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.339591 12531 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:32.358351  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:32.353508  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:32.362294  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:32.363219  5558 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:27:32.328743 12286 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
I20260709 12:27:32.366256  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.366577  5561 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:27:32.363992  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.355120 12625 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:32.378507  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.357156 12579 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:32.381587  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:32.380162  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:32.387084  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.389180  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.390163 12499 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:32.390735  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.396924  5704 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:32.397197 12376 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:32.397918 12376 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:32.398895  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.407536  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.408056  5706 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:32.375031 12275 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.647s	user 0.013s	sys 0.000s
I20260709 12:27:32.411944  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:32.421983  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:32.360554 12455 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.379802 12609 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:32.429308  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:32.435714 12466 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:32.436604  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:32.435873 12542 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:32.437970 12429 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:32.438057 12542 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:27:32.439523  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:32.440323 12542 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:32.440778 12542 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:32.441159 12542 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:32.441269 12193 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:32.441869 12193 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:32.442972 12193 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.444832 12457 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 12:27:32.453186  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:32.474839  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52936.4us
I20260709 12:27:32.475728  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:32.423687 12528 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:32.482666 12528 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.432399 12600 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:32.485929  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:32.486876  5558 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:27:32.487226  5706 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
W20260709 12:27:32.488320 12620 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 12:27:32.488996 12193 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:32.488897  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.490765  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.491315  5706 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:27:32.483989 12625 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:32.493770  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:32.421113 12426 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:32.434161 12618 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 12:27:32.495958  5555 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 13. (term mismatch)
I20260709 12:27:32.499060  5555 pending_rounds.cc:85] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 12
I20260709 12:27:32.499374  5555 pending_rounds.cc:107] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 18.13
I20260709 12:27:32.499672  5555 raft_consensus.cc:2889] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:32.399677 12579 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.494975  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:32.399243 12376 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.503052 12596 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:32.433305 12550 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:32.508311 12610 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:32.510457  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2642) took 1678 ms. Trace:
I20260709 12:27:32.519534 12599 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:32.512737 12521 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:32.522037 12521 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:32.522414 12521 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:32.522246 12181 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.513024 12608 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:32.524948 12608 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.526986 12608 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:32.514249 12596 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:32.515483 12579 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.481724 12612 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:32.513760 12376 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:32.644778  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.645435  6280 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:32.646021 12608 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:32.646633 12608 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:32.571133  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 149908us
I20260709 12:27:32.651896 12608 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:32.655262  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 179626us
W20260709 12:27:32.623126 12574 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.842s	user 0.016s	sys 0.002s
I20260709 12:27:32.625188 12542 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.625517 12625 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:32.659284 12574 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.659505 12625 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.660867 12542 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:32.660872  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.661701 12574 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.661859 12625 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:32.626052 12609 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:32.662603  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.662778 12609 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.663380  6280 raft_consensus.cc:2368] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22.
I20260709 12:27:32.662603  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:32.640046  5706 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:32.664672 12625 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:32.665028 12609 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.665067 12625 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:32.667328 12625 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:32.509783  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:32.627514 12596 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.628561 12600 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:32.508312 12442 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:32.630290 12455 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.631542 12181 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:32.632491 12488 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.362s	user 0.021s	sys 0.001s
I20260709 12:27:32.633865 12528 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:32.636773 12550 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:32.637348  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.416200 12275 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.501964 12588 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.046s
I20260709 12:27:32.642441  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:32.643509 12605 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:32.697553 12605 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.698326 12275 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.699829 12605 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.655689  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:32.656623  5704 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 12:27:32.643605 12521 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.653182  5707 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:27:32.701476  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:32.701782  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.702282 12521 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.702406  5707 raft_consensus.cc:2376] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19. Re-sending same reply.
I20260709 12:27:32.703029  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:32.703433  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.662040  6277 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 17: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:27:32.628381 12337 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.865s	user 0.012s	sys 0.005s
I20260709 12:27:32.692520 12488 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.705051 12337 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.706425 12488 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:32.707046 12337 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:32.674576  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:32.675086 12499 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:32.708616 12499 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.710623 12499 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:32.676834  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 167128us
I20260709 12:27:32.678524  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:32.519567  5554 rpcz_store.cc:276] 0709 12:27:30.832157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:30.832345 (+   188us) service_pool.cc:224] Handling call
0709 12:27:30.862403 (+ 30058us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:30.866708 (+  4305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:30.866752 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:30.872748 (+  5996us) log.cc:844] Serialized 26 byte log entry
0709 12:27:31.034031 (+161283us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:31.034095 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:31.034188 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:31.362501 (+328313us) raft_consensus.cc:1685] finished
0709 12:27:31.362526 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:32.510179 (+1147653us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b54001c48d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:32.510391 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3892,"raft.queue_time_us":4474,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":3742,"spinlock_wait_cycles":68567552,"thread_start_us":160824,"threads_started":1,"wal-append.queue_time_us":409830,"wal-append.run_cpu_time_us":2106,"wal-append.run_wall_time_us":1075370}
W20260709 12:27:32.668202 12590 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.868s	user 0.001s	sys 0.002s
I20260709 12:27:32.715072  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:32.715373  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.715711  6271 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:32.680474  5631 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:27:32.716611  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.717235  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.717332  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:32.680869  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.681131 12193 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:32.681638  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:32.681895  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:32.720271  5628 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 18: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:32.682152 12579 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:32.683089  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.980s (SENT) [suppressed 35 similar messages]
I20260709 12:27:32.681663  5630 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 14. Current term is 15. Ops: []
W20260709 12:27:32.721222  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:32.687530  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:32.721606  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:32.722751  6275 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:32.723153  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:32.723557  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:32.683482 12517 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:32.694833 12548 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:32.730463  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:32.730707 12548 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.731451 12475 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:32.731635  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:32.635195 12637 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:32.683734 12537 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:32.732865 12537 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:32.733256 12537 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:32.735014  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:32.736032  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.732676 12637 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.737090 12557 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:32.704486  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:32.737692 12557 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.738713 12637 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.739037  5675 leader_election.cc:400] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22.
I20260709 12:27:32.739413  5675 leader_election.cc:403] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:27:32.739652  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.740459 12586 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:32.740516 12608 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22.
I20260709 12:27:32.741462 12275 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:32.672652  5706 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:27:32.713459  5635 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:32.708220  5636 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:27:32.742745  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:32.743129  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:32.743367  5554 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:32.744287  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:32.713618  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:32.744338 12548 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.680629  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:32.745923 12638 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:27:32.696398  5552 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:32.747022  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1329) took 1653 ms. Trace:
I20260709 12:27:32.747591 12608 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:32.725244  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:32.748224 12608 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.747378  5633 rpcz_store.cc:276] 0709 12:27:31.093672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:31.093840 (+   168us) service_pool.cc:224] Handling call
0709 12:27:31.094274 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:31.165089 (+ 70815us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:27:31.165407 (+   318us) log.cc:844] Serialized 14 byte log entry
0709 12:27:31.799785 (+634378us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:31.799868 (+    83us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:31.799941 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:32.746826 (+946885us) spinlock_profiling.cc:243] Waited 70.5 ms on lock 0x7b54000c0b54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:32.746870 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:32.746973 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27087,"prepare.queue_time_us":19428,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":47,"raft.queue_time_us":316442,"raft.run_cpu_time_us":52,"raft.run_wall_time_us":52,"spinlock_wait_cycles":162260352,"thread_start_us":606246,"threads_started":2,"wal-append.queue_time_us":641253}
I20260709 12:27:32.745371  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:32.724150  5631 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:32.726394 12534 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:27:32.726914 12579 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.727065 12517 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.727670 12193 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.728111  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.729015  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.732267  5630 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:32.730476  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:32.683917 12442 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:32.696594 12550 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.738938  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:32.751055  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:32.751832  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:32.718741  5632 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:32.772104  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:32.770287  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:32.773931  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:32.750044  5553 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:27:32.749133  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:32.732646  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:32.752245  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.776721  5706 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:32.778040  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.752627  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.779729  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:32.745586  5554 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 6. Preceding OpId from leader: term: 16 index: 7. (index mismatch)
I20260709 12:27:32.745682  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:32.754735  6271 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
W20260709 12:27:32.756461  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:32.759686  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:32.784644  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:32.784716  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:32.785722  5633 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:32.769622 12425 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.017s
I20260709 12:27:32.770730 12442 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:32.786693  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:32.791451 12193 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:32.760053 12181 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:32.765560  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.767611  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:32.768615  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:32.745379  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:32.782850  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:32.786398  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.757705 12275 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:32.696231 12455 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.067s
I20260709 12:27:32.761722  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:32.794016  5557 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:32.763114 12586 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:32.840822  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2905) took 1865 ms. Trace:
I20260709 12:27:32.760185 12521 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:32.837742  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:32.811434  5706 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:32.848899  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.849073  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:32.811250  5707 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:32.841512  5629 rpcz_store.cc:276] 0709 12:27:30.975008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:30.976172 (+  1164us) service_pool.cc:224] Handling call
0709 12:27:30.976486 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:31.649302 (+672816us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:32.839725 (+1190423us) spinlock_profiling.cc:243] Waited 671 ms on lock 0x7b54000c2954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:32.840741 (+  1016us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1544175872}
I20260709 12:27:32.761272  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:32.839179  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2903) took 1868 ms. Trace:
I20260709 12:27:32.840303  5635 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:32.851046  5627 rpcz_store.cc:276] 0709 12:27:30.970493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:30.970959 (+   466us) service_pool.cc:224] Handling call
0709 12:27:30.971269 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:30.980100 (+  8831us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:30.980140 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:30.980191 (+    51us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:30.980233 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:32.838912 (+1858679us) spinlock_profiling.cc:243] Waited 8.59 ms on lock 0x7b540032d2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:32.838966 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:32.839111 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19750784}
I20260709 12:27:32.851951  5631 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:32.808964  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:32.829085  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44488.4us
I20260709 12:27:32.813555  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:32.795440 12557 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:32.811419  5630 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: [20.10-20.10]
W20260709 12:27:32.845947  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 88222.5us
I20260709 12:27:32.855279  5633 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260709 12:27:32.855773  5633 pending_rounds.cc:85] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 9
I20260709 12:27:32.855922  5635 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: [17.10-19.11]
I20260709 12:27:32.856110  5633 pending_rounds.cc:107] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260709 12:27:32.856869  5633 raft_consensus.cc:2889] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:32.857020  5628 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:27:32.849653  5706 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:32.857188  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:32.844374  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 61624.5us
I20260709 12:27:32.856485  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:32.793695 12256 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:32.793380 12610 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:32.793624 12205 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:32.863008  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.867226  5707 raft_consensus.cc:2376] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18. Re-sending same reply.
I20260709 12:27:32.860610  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:32.859983  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:32.846495 12534 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:32.818572 12608 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:32.865101  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 96573.7us
I20260709 12:27:32.868993  5706 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:32.795766 12531 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:32.847909 12286 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:32.873356 12425 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:32.864322 12557 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:32.876984  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.854763 12326 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:32.881179  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:32.877622 12550 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.878748 12361 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:32.884313  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.864705 12181 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:32.886261  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:32.862632 12541 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:32.856604 12521 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.888067  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:32.889339  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:32.848023 12586 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:32.885782  5707 raft_consensus.cc:2376] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20. Re-sending same reply.
I20260709 12:27:32.910693  6280 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:27:32.889850 12437 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:32.895699 12541 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:32.896590  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:32.914427 12442 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:32.915251 12612 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:32.917706 12586 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:33.065533  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 169017us
I20260709 12:27:33.065821 12586 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:32.918742 12181 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:33.066727 12442 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:33.067018  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:32.892102 12361 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:32.920543 12579 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.885072 12625 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:33.071382 12625 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:32.923481 12517 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:32.925287 12425 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:32.925802 12455 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:27:32.910864 12534 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:32.887351 12326 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:32.881690  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:33.073473 12326 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:33.073717  5706 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:33.074661  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 152215us
I20260709 12:27:33.075517  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:32.928668  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:33.076409 12455 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:33.077016  5707 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:27:33.078270  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:33.078666  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:33.079622  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:32.894652 12557 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:32.931658  5702 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:27:32.897243  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:32.883402 12205 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:32.882745 12256 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.082s
I20260709 12:27:32.891865 12286 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:32.896091 12537 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:33.068408 12361 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:33.068529 12610 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:27:33.068764  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:33.069133  5629 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:33.076951 12455 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:33.091928 12455 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:33.094893  5706 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:33.081666  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:33.083412  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:33.046676 12521 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:32.931605  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:33.098371  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:33.101649  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:33.103458  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:33.079659 12534 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:27:33.105567  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 174054us
I20260709 12:27:33.106374  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:33.107869  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:33.109268  5709 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:33.109349  5706 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:27:33.109449  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:33.111542  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:33.110953  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:33.132382 12595 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:33.132921 12517 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:33.166946 12193 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:33.137740 12642 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:27:33.137629 12579 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:27:33.139518  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:33.109597 12475 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:33.107923 12644 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:33.106717 12586 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:33.146104 12319 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:33.127327 12286 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:33.154910  5706 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:33.157841  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:33.208503  5628 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:33.158818  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:27:33.209621  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 94815.2us
I20260709 12:27:33.153509  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:33.211194  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 57823.7us
I20260709 12:27:33.164429  5631 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:27:33.166226 12595 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:33.171214  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 64933.9us
I20260709 12:27:33.086728 12425 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20260709 12:27:33.170593 12580 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:27:33.177446 12592 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.292s	user 0.004s	sys 0.000s
I20260709 12:27:33.109498 12647 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:33.175920  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:33.120801 12610 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:33.179183  5635 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:33.182081  5636 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:27:33.182224  5704 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:33.217273  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:33.184975 12475 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:33.239427  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:33.240548  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:33.213662 12528 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:27:33.245245  5704 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:33.235309  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:33.248731  5631 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 19: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:33.249780  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:33.211513 12570 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.309s	user 0.064s	sys 0.101s
I20260709 12:27:33.230831  5706 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
W20260709 12:27:33.260110  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:33.178005 12557 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:33.181779 12205 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:33.254024 12569 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:27:33.240074 12521 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:33.240298 12475 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:33.235850 12181 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:27:33.365051 12570 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.467s	user 0.065s	sys 0.102s
I20260709 12:27:33.379249  5706 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:33.224862 12544 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.032s
I20260709 12:27:33.241937 12579 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:33.225014 12586 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 12:27:33.365789 12531 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:33.224992 12644 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.143s
I20260709 12:27:33.389379  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2766) took 1215 ms. Trace:
I20260709 12:27:33.397994  5706 rpcz_store.cc:276] 0709 12:27:32.174421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.378527 (+1204106us) service_pool.cc:224] Handling call
0709 12:27:33.378998 (+   471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.388892 (+  9894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.389291 (+   399us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:33.400300  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3006) took 1219 ms. Trace:
I20260709 12:27:33.400782  5706 rpcz_store.cc:276] 0709 12:27:32.181039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.399490 (+1218451us) service_pool.cc:224] Handling call
0709 12:27:33.399799 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.399984 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:33.400007 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:33.400050 (+    43us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:33.400089 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.400139 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:33.400246 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:27:33.392807  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 166885us
I20260709 12:27:33.385121  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:33.253552 12402 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:33.403236  5706 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:33.403882  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2769) took 1222 ms. Trace:
I20260709 12:27:33.404228  5706 rpcz_store.cc:276] 0709 12:27:32.181292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.402189 (+1220897us) service_pool.cc:224] Handling call
0709 12:27:33.402553 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.403636 (+  1083us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.403832 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:33.386901 12521 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:33.385121 12610 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:33.406203  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3011) took 1215 ms. Trace:
I20260709 12:27:33.407482  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 2998) took 2513 ms. Trace:
I20260709 12:27:33.365288 12460 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 12:27:33.409246  5706 rpcz_store.cc:276] 0709 12:27:32.190574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.405131 (+1214557us) service_pool.cc:224] Handling call
0709 12:27:33.405708 (+   577us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.405895 (+   187us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:33.405922 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:33.405964 (+    42us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:33.405993 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.406047 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:33.406153 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:33.409296  5708 rpcz_store.cc:276] 0709 12:27:30.889887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:31.098864 (+208977us) service_pool.cc:224] Handling call
0709 12:27:31.099347 (+   483us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:31.196464 (+ 97117us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:31.196494 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:31.196984 (+   490us) log.cc:844] Serialized 26 byte log entry
0709 12:27:31.622097 (+425113us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:31.622135 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:31.622202 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:33.176897 (+1554695us) spinlock_profiling.cc:243] Waited 7.31 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:33.394799 (+217902us) raft_consensus.cc:1685] finished
0709 12:27:33.394827 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:33.403131 (+  8304us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":131332,"raft.queue_time_us":122439,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":10208,"spinlock_wait_cycles":16804224,"thread_start_us":356384,"threads_started":1,"wal-append.queue_time_us":536829}
I20260709 12:27:33.412168  5706 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 16. (term mismatch)
I20260709 12:27:33.412566  5706 pending_rounds.cc:85] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 15
I20260709 12:27:33.412814  5706 pending_rounds.cc:107] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 21.16
I20260709 12:27:33.392683 12181 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:33.413213  5706 raft_consensus.cc:2889] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:33.413992  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2773) took 1176 ms. Trace:
I20260709 12:27:33.414255  5706 rpcz_store.cc:276] 0709 12:27:32.237946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.411010 (+1173064us) service_pool.cc:224] Handling call
0709 12:27:33.411698 (+   688us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.413774 (+  2076us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.413952 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:33.415189  5706 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:33.416193  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3014) took 1132 ms. Trace:
I20260709 12:27:33.416615  5706 rpcz_store.cc:276] 0709 12:27:32.283823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.414858 (+1131035us) service_pool.cc:224] Handling call
0709 12:27:33.415080 (+   222us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.415954 (+   874us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.416144 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:33.418823  5708 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:27:33.397851 12579 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:33.422336  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2774) took 1184 ms. Trace:
I20260709 12:27:33.413091 12521 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:33.412695 12608 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:33.416601 12181 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:33.448217 12650 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 12:27:33.414731 12537 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:33.446709 12646 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.274s	user 0.001s	sys 0.000s
W20260709 12:27:33.469066 12655 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:27:33.451311 12608 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:33.470556 12608 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:33.453282 12595 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:33.472199 12608 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:33.527920  5628 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:33.483938 12193 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:33.444177  5708 rpcz_store.cc:276] 0709 12:27:32.238253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.411300 (+1173047us) service_pool.cc:224] Handling call
0709 12:27:33.418523 (+  7223us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:33.421938 (+  3415us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.422258 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:33.552596  5708 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:27:33.553408  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1206) took 1110 ms. Trace:
I20260709 12:27:33.553799  5708 rpcz_store.cc:276] 0709 12:27:32.442710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.551626 (+1108916us) service_pool.cc:224] Handling call
0709 12:27:33.552095 (+   469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.553120 (+  1025us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.553355 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:33.487912 12326 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:33.421367 12610 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:33.595079  5707 raft_consensus.cc:2368] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
I20260709 12:27:33.603336  5560 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:27:33.525807 12579 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:33.613260  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:33.613848  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:33.608737 12460 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:33.615604 12326 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:33.624161 12535 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:33.609063 12608 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:33.636938 12535 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:33.639430 12535 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:33.640646  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1190) took 3554 ms. Trace:
I20260709 12:27:33.641075 12475 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:33.641170  5703 rpcz_store.cc:276] 0709 12:27:30.085915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:30.519369 (+433454us) service_pool.cc:224] Handling call
0709 12:27:30.519711 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:31.091349 (+571638us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:31.091404 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:31.105701 (+ 14297us) log.cc:844] Serialized 26 byte log entry
0709 12:27:32.930485 (+1824784us) spinlock_profiling.cc:243] Waited 9.62 ms on lock 0x7b180039a120. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:27:33.250553 (+320068us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:33.250602 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.250724 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:33.640232 (+389508us) raft_consensus.cc:1685] finished
0709 12:27:33.640261 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:33.640568 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":658917,"raft.queue_time_us":593812,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":24004,"spinlock_wait_cycles":22122368,"thread_start_us":319833,"threads_started":1,"wal-append.queue_time_us":2309768}
I20260709 12:27:33.628324 12579 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:33.652232  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2915) took 2132 ms. Trace:
I20260709 12:27:33.653295  5634 rpcz_store.cc:276] 0709 12:27:31.515498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:31.516008 (+   510us) service_pool.cc:224] Handling call
0709 12:27:31.516428 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:31.780102 (+263674us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:27:32.049607 (+269505us) log.cc:844] Serialized 14 byte log entry
0709 12:27:32.123428 (+ 73821us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:32.124077 (+   649us) log.cc:844] Serialized 26 byte log entry
0709 12:27:32.124248 (+   171us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:32.124295 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.129402 (+1005107us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b1800297580. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419
0709 12:27:33.646990 (+517588us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b540009ecd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:33.647068 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:33.647134 (+    66us) raft_consensus.cc:1685] finished
0709 12:27:33.647154 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:33.647337 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3737,"prepare.queue_time_us":389662,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":307905,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":186,"spinlock_wait_cycles":360153088,"thread_start_us":448615,"threads_started":3,"wal-append.queue_time_us":215451,"wal-append.run_cpu_time_us":9805,"wal-append.run_wall_time_us":1252316}
W20260709 12:27:33.660868  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 253035us
I20260709 12:27:33.635219 12460 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:33.668509 12470 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.006s
I20260709 12:27:33.680850 12426 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:33.681326 12426 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:33.682180 12426 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:33.683219 12426 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:33.683321 12464 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:33.687554  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:33.688230  5634 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:33.688293  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1203) took 1942 ms. Trace:
I20260709 12:27:33.688833  5710 rpcz_store.cc:276] 0709 12:27:31.745676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:31.745828 (+   152us) service_pool.cc:224] Handling call
0709 12:27:31.761135 (+ 15307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:32.195243 (+434108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.688010 (+1492767us) spinlock_profiling.cc:243] Waited 433 ms on lock 0x7b540033fc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:33.688229 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":996317952}
I20260709 12:27:33.691306 12373 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:33.691911 12373 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:33.693950 12373 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:33.694969 12361 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.592s	user 0.009s	sys 0.001s
I20260709 12:27:33.695106  5633 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 6. Preceding OpId from leader: term: 16 index: 7. (index mismatch)
I20260709 12:27:33.695467 12361 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:33.696218  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:33.697208 12475 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:33.703315  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2764) took 1530 ms (client timeout 1830 ms). Trace:
I20260709 12:27:33.703795  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:33.705727  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:33.707290 12460 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:33.703840  5702 rpcz_store.cc:269] 0709 12:27:32.173394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:32.205259 (+ 31865us) service_pool.cc:224] Handling call
0709 12:27:33.703041 (+1497782us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b5400323f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:33.703231 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1667936512}
I20260709 12:27:33.708534  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1300) took 1295 ms. Trace:
I20260709 12:27:33.708922  5560 rpcz_store.cc:276] 0709 12:27:32.413143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:32.413329 (+   186us) service_pool.cc:224] Handling call
0709 12:27:32.430059 (+ 16730us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.603840 (+1173781us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.708298 (+104458us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540041a0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:33.708478 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2694292352}
I20260709 12:27:33.710304 12595 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:33.710189  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3016) took 1216 ms. Trace:
I20260709 12:27:33.711124  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2788) took 1034 ms. Trace:
I20260709 12:27:33.710996 12595 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:33.646320 12608 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:33.711658  5707 rpcz_store.cc:276] 0709 12:27:32.677064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.079270 (+402206us) service_pool.cc:224] Handling call
0709 12:27:33.710921 (+631651us) spinlock_profiling.cc:243] Waited 515 ms on lock 0x7b54000e01d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:33.711067 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1183587584}
I20260709 12:27:33.711134  5702 rpcz_store.cc:276] 0709 12:27:32.493893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.709090 (+1215197us) service_pool.cc:224] Handling call
0709 12:27:33.709487 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.709737 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:33.709780 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:33.709841 (+    61us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:33.709890 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.709968 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:33.710111 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:33.721163  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:33.721997  5702 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:33.722719  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3017) took 1071 ms. Trace:
I20260709 12:27:33.723011  5702 rpcz_store.cc:276] 0709 12:27:32.650939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.721525 (+1070586us) service_pool.cc:224] Handling call
0709 12:27:33.721824 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.722479 (+   655us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.722667 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:33.724699  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:33.725385  5554 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:33.724875  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3018) took 1073 ms. Trace:
I20260709 12:27:33.726099  5702 rpcz_store.cc:276] 0709 12:27:32.651220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.724067 (+1072847us) service_pool.cc:224] Handling call
0709 12:27:33.724391 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.724570 (+   179us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:33.724592 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:33.724631 (+    39us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:33.724668 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.724723 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:33.724827 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:33.728632  5553 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 17. (term mismatch)
I20260709 12:27:33.729287  5553 pending_rounds.cc:85] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 16
I20260709 12:27:33.729580  5553 pending_rounds.cc:107] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 21.17
I20260709 12:27:33.729878  5553 raft_consensus.cc:2889] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:33.739779  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:33.759611  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:33.665151 12557 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.069s
I20260709 12:27:33.798918  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:33.799863  5630 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:33.805657  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:33.806561  5636 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:33.733798 12437 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.670s	user 0.011s	sys 0.002s
I20260709 12:27:33.802067  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:33.726321 12426 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:33.818183  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:33.713135 12595 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:33.735239 12275 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:33.816038 12256 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:33.815719 12437 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:33.742177 12470 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.108s
I20260709 12:27:33.833910 12426 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:33.739244 12541 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:33.816314 12475 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 12:27:33.868276 12595 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:33.876200 12610 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:33.886266 12662 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.148s	user 0.000s	sys 0.000s
W20260709 12:27:33.886832 12664 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
W20260709 12:27:33.887333 12665 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260709 12:27:33.889103 12574 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:33.892597 12574 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:33.884902 12595 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:33.889845 12608 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:33.894029 12574 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:33.896490 12574 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:27:33.898697 12663 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.160s	user 0.002s	sys 0.000s
I20260709 12:27:33.933857  5552 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
W20260709 12:27:33.935040  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 207166us
I20260709 12:27:33.938664  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1204) took 1519 ms. Trace:
I20260709 12:27:33.939313  5706 rpcz_store.cc:276] 0709 12:27:32.419302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.418013 (+998711us) service_pool.cc:224] Handling call
0709 12:27:33.418315 (+   302us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:33.442430 (+ 24115us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:33.442467 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:33.442973 (+   506us) log.cc:844] Serialized 26 byte log entry
0709 12:27:33.611083 (+168110us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:33.816379 (+205296us) log.cc:844] Serialized 14 byte log entry
0709 12:27:33.816666 (+   287us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.816734 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:33.937784 (+121050us) raft_consensus.cc:1685] finished
0709 12:27:33.937834 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:33.938289 (+   455us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21081,"prepare.queue_time_us":272442,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":24600,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":31470,"thread_start_us":346330,"threads_started":2,"wal-append.queue_time_us":216393}
W20260709 12:27:33.944928  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 206916us
I20260709 12:27:33.943164  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2920) took 1262 ms. Trace:
I20260709 12:27:33.948982  5629 rpcz_store.cc:276] 0709 12:27:32.684561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:32.850351 (+165790us) service_pool.cc:224] Handling call
0709 12:27:32.850682 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.089699 (+239017us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.942468 (+852769us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b540018bd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:33.942869 (+   401us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":501941632}
I20260709 12:27:33.963973  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3009) took 1781 ms. Trace:
I20260709 12:27:33.964645  5709 rpcz_store.cc:276] 0709 12:27:32.182043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:32.636985 (+454942us) service_pool.cc:224] Handling call
0709 12:27:33.963298 (+1326313us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x7b5400451fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:33.963672 (+   374us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":973667712}
I20260709 12:27:33.967209  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1214) took 1225 ms. Trace:
I20260709 12:27:33.971060 11997 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:33.973587  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2874) took 1617 ms. Trace:
I20260709 12:27:33.967612  5709 rpcz_store.cc:276] 0709 12:27:32.741659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.965772 (+1224113us) service_pool.cc:224] Handling call
0709 12:27:33.966383 (+   611us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.966807 (+   424us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:33.966834 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:33.966890 (+    56us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:33.966935 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.967016 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:33.967120 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:27:33.974291  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:33.974226  5561 rpcz_store.cc:276] 0709 12:27:32.355515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:32.355724 (+   209us) service_pool.cc:224] Handling call
0709 12:27:33.973005 (+1617281us) spinlock_profiling.cc:243] Waited 6.28 ms on lock 0x7b540016ba54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:33.973318 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14454528}
I20260709 12:27:33.975328  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:33.977608  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:33.951814  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:33.863583 12256 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:34.028981 12256 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:33.885246 12541 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:34.066310  5629 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:33.854208 12275 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:34.085994  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:34.086409  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 114132us
I20260709 12:27:34.086717 12657 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 12:27:34.088594 12657 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:34.089155 12657 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:34.089685 12588 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:34.089532 12537 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.007s
I20260709 12:27:34.090767 12657 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:34.095252 12535 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
W20260709 12:27:34.106729  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:34.112776  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:34.084183  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2880) took 1288 ms. Trace:
I20260709 12:27:34.137603 12402 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:34.138187 12361 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:34.138613 12402 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.142977  5709 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:27:34.143440  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.143695  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:34.144186  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:34.144450  5560 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:34.144484  5559 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:34.147142  5553 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:34.149160  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.150290  5556 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:34.090797 12588 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:34.154083 12437 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:34.155015 12437 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.155719 12588 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:34.089537 12426 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:34.163502  5709 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:34.165570 12275 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:34.121733  5557 rpcz_store.cc:276] 0709 12:27:32.742737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:32.745453 (+  2716us) service_pool.cc:224] Handling call
0709 12:27:33.976267 (+1230814us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b54003c4bd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:34.031476 (+ 55209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27915008}
I20260709 12:27:34.177771  5559 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:34.083987 12602 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:34.131062  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 190716us
I20260709 12:27:34.181137  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1211) took 1448 ms. Trace:
I20260709 12:27:34.190383  5706 rpcz_store.cc:276] 0709 12:27:32.724790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.945263 (+1220473us) service_pool.cc:224] Handling call
0709 12:27:33.946017 (+   754us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.946641 (+   624us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:27:33.947265 (+   624us) log.cc:844] Serialized 14 byte log entry
0709 12:27:34.173007 (+225742us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:34.173103 (+    96us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:34.173188 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:34.173280 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:34.173437 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":97271,"prepare.queue_time_us":62500,"prepare.run_cpu_time_us":130,"prepare.run_wall_time_us":4817,"raft.queue_time_us":98339,"thread_start_us":122838,"threads_started":1}
I20260709 12:27:34.198832 12602 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:34.228788  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 97130.3us
I20260709 12:27:34.234212 12657 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:34.241853 12522 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:34.245249 12548 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:34.246032  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:34.247059  5557 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:34.252650  5630 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:34.269824  5706 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 14. (term mismatch)
I20260709 12:27:34.269945  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.270483  5706 pending_rounds.cc:85] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 13
I20260709 12:27:34.270759  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.271505  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.271741  5552 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
W20260709 12:27:34.270875  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52825.1us
I20260709 12:27:34.276042  5636 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:34.270794  5706 pending_rounds.cc:107] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 23.14
I20260709 12:27:34.298321  5706 raft_consensus.cc:2889] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:27:34.311126  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 77361.4us
W20260709 12:27:34.322810  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 92688.1us
I20260709 12:27:34.246066 12548 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:34.329581  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:34.339418  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:34.340801  5627 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.515s	user 0.009s	sys 0.005s
I20260709 12:27:34.341334  5627 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
W20260709 12:27:34.342259  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53311.6us
I20260709 12:27:34.343369  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:34.344611 12667 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.254s	user 0.002s	sys 0.000s
W20260709 12:27:34.347575 12671 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.258s	user 0.000s	sys 0.001s
W20260709 12:27:34.349603 12521 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.888s	user 0.004s	sys 0.018s
I20260709 12:27:34.350098 12521 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:34.352483  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:34.352869  5704 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:34.353609 12589 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:34.354350 12589 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.355209 12437 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:34.356561 12402 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:34.356760  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1207) took 1913 ms. Trace:
I20260709 12:27:34.309862 12426 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:27:34.357201  5708 rpcz_store.cc:276] 0709 12:27:32.443498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.555215 (+1111717us) service_pool.cc:224] Handling call
0709 12:27:33.555609 (+   394us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:33.594504 (+ 38895us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:33.594555 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:33.605688 (+ 11133us) log.cc:844] Serialized 26 byte log entry
0709 12:27:33.813580 (+207892us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:33.813634 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.813726 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:34.343725 (+529999us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:34.356116 (+ 12391us) raft_consensus.cc:1685] finished
0709 12:27:34.356151 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:34.356522 (+   371us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":586,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":6750,"spinlock_wait_cycles":59471104,"thread_start_us":207481,"threads_started":1,"wal-append.queue_time_us":483935}
W20260709 12:27:34.359815  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 61741us
I20260709 12:27:34.361058 12536 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:34.361505 12536 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:34.362651 12536 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.363827 12536 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:34.366295 12602 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:34.367306  6279 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:34.369055  6278 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:34.370391  5628 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:27:34.373764  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.375442  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.376317  5561 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:34.377496  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:34.356382  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58343.5us
I20260709 12:27:34.378309  5555 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:34.376925  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:34.377041  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:34.378283  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.377170  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.384097  5553 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:34.391428  5554 raft_consensus.cc:2376] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18. Re-sending same reply.
W20260709 12:27:34.348222 12579 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.689s	user 0.008s	sys 0.013s
W20260709 12:27:34.381793 12660 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.644s	user 0.004s	sys 0.000s
I20260709 12:27:34.422215  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1180) took 1183 ms. Trace:
I20260709 12:27:34.422704  6271 rpcz_store.cc:276] 0709 12:27:33.238221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.238958 (+   737us) service_pool.cc:224] Handling call
0709 12:27:33.239627 (+   669us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:33.387609 (+147982us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:33.387655 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:33.388666 (+  1011us) log.cc:844] Serialized 52 byte log entry
0709 12:27:33.592940 (+204274us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:33.593002 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:33.593109 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:34.380666 (+787557us) spinlock_profiling.cc:243] Waited 68.1 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:34.421635 (+ 40969us) raft_consensus.cc:1685] finished
0709 12:27:34.421657 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:34.421967 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4170,"raft.queue_time_us":133380,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":3921,"spinlock_wait_cycles":156624512,"thread_start_us":322200,"threads_started":2,"wal-append.queue_time_us":241468}
W20260709 12:27:34.427697 12574 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.531s	user 0.008s	sys 0.005s
W20260709 12:27:34.430099  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:34.431391  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:34.438228 12181 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.973s	user 0.013s	sys 0.007s
I20260709 12:27:34.403416  5555 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:34.402141  5557 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:34.406298 12589 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:34.406507 12548 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:34.407336  5561 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:27:34.408110  5633 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:34.408154  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:34.412595 12467 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:27:34.417119  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 73969.3us
I20260709 12:27:34.418074 12521 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:34.419265  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:34.412632 12672 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:34.439579  5556 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:34.439827 12376 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:34.404480  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:34.401034  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:34.456758  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48186.8us
I20260709 12:27:34.458824  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:34.465683  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:34.466825  5560 raft_consensus.cc:2410] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:27:34.469522  5628 raft_consensus.cc:2410] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:27:34.482848  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:34.481215  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 73227.7us
I20260709 12:27:34.468673 12376 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.487214  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:34.487807 12574 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:34.444139 12181 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:34.488484 12574 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.488739  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:34.488969 12541 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:34.489580 12541 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.490495 12574 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:34.487360 12579 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:34.491767 12689 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:34.491843 12541 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:34.490435 12181 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:34.492331 12689 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:34.587491 12689 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:34.492803  5701 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:34.587864 12181 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:27:34.589038 12181 consensus_queue.cc:1243] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 16.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:27:34.590567 12193 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:34.591019 12193 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:34.591511  5701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3003) took 3203 ms (client timeout 1868 ms). Trace:
I20260709 12:27:34.491477 12579 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:34.501809 12603 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:27:34.592888 12193 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:34.503834  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2778) took 1829 ms. Trace:
I20260709 12:27:34.506810  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:34.594520 12193 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:34.590939 12529 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.004s
I20260709 12:27:34.594663 12579 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:34.507378  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:34.507803 12256 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:34.597824  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 97497.1us
I20260709 12:27:34.599213 12256 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:34.511102  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:34.599745 12603 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:27:34.592105  5701 rpcz_store.cc:269] 0709 12:27:31.387488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:31.396024 (+  8536us) service_pool.cc:224] Handling call
0709 12:27:34.591010 (+3194986us) spinlock_profiling.cc:243] Waited 3.1 s on lock 0x7b54002d19d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:34.591286 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7119220096}
I20260709 12:27:34.515475 12610 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:34.601336 12610 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.601009  5702 rpcz_store.cc:276] 0709 12:27:32.674119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.727950 (+1053831us) service_pool.cc:224] Handling call
0709 12:27:33.728258 (+   308us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:33.737983 (+  9725us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:33.738024 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:33.739151 (+  1127us) log.cc:844] Serialized 26 byte log entry
0709 12:27:34.233797 (+494646us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:27:34.233937 (+   140us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b48001701d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:27:34.234316 (+   379us) log.cc:844] Serialized 14 byte log entry
0709 12:27:34.503169 (+268853us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x7b1800460e40. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:27:34.503359 (+   190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:34.503464 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:34.503522 (+    58us) raft_consensus.cc:1685] finished
0709 12:27:34.503544 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:34.503688 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":93923,"prepare.queue_time_us":61352,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":5413,"raft.queue_time_us":213213,"raft.run_cpu_time_us":3799,"raft.run_wall_time_us":502270,"spinlock_wait_cycles":10431744,"thread_start_us":279464,"threads_started":1,"wal-append.queue_time_us":394734}
I20260709 12:27:34.603699 12610 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:34.517479  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:34.518596  5706 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:27:34.519260  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:34.519999  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:34.521216  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:34.521667  5703 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 23. Current term is 24. Ops: [23.18-23.18]
I20260709 12:27:34.521776  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1210) took 1800 ms. Trace:
I20260709 12:27:34.609777  5707 rpcz_store.cc:276] 0709 12:27:32.721271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.735575 (+1014304us) service_pool.cc:224] Handling call
0709 12:27:33.738311 (+  2736us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:33.797398 (+ 59087us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:27:34.220931 (+423533us) log.cc:844] Serialized 14 byte log entry
0709 12:27:34.519107 (+298176us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:34.519190 (+    83us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:34.519269 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:34.521359 (+  2090us) spinlock_profiling.cc:243] Waited 58.8 ms on lock 0x7b54004515d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:34.521412 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:34.521619 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":127388,"prepare.queue_time_us":91002,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":130478,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"spinlock_wait_cycles":135322496,"thread_start_us":394219,"threads_started":2,"wal-append.queue_time_us":233407}
I20260709 12:27:33.391058  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:34.522667 12402 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:27:34.522804  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 91594.2us
I20260709 12:27:34.525079  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3020) took 1778 ms. Trace:
I20260709 12:27:34.534051  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:34.534900  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:34.538484 12672 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.540891  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.541853  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.543234  5561 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:27:34.544564  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:34.727869  5559 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:34.545292  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:34.546085  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.728685  5560 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:27:34.494904 12595 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:34.496737  5552 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:34.501550 12659 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
W20260709 12:27:34.550904 12464 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.867s	user 0.016s	sys 0.004s
I20260709 12:27:34.553321  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:34.553642  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:34.554581  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:34.554623  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:34.557885 12574 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:34.560303 12685 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:27:34.472548 12657 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.092s
I20260709 12:27:34.565178  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:34.562107 12376 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:34.584743  5602 leader_election.cc:336] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.780s (SENT)
I20260709 12:27:34.596962 12529 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:34.514281  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:34.712133  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1208) took 2268 ms. Trace:
I20260709 12:27:34.712980  5709 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:34.744132  5556 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
W20260709 12:27:34.617101 12691 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.068s	user 0.004s	sys 0.000s
W20260709 12:27:34.716578 12653 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.109s	user 0.004s	sys 0.004s
W20260709 12:27:34.705300  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 104306us
W20260709 12:27:34.724400  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:34.760113  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:34.718431 12402 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:34.768371 12402 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:34.768796  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:34.770372  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:34.743276 12529 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:34.772149  5707 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 11. (term mismatch)
I20260709 12:27:34.772671  5707 pending_rounds.cc:85] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 10
I20260709 12:27:34.772308 12529 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:34.773653  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.773916 12529 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:34.776870  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.777841  5556 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:34.778975  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.779672  5556 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:34.779568 12574 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:34.781091  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:34.781957 12574 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:34.782529 12677 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.205s	user 0.002s	sys 0.005s
I20260709 12:27:34.717553  5704 rpcz_store.cc:276] 0709 12:27:32.746712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.252453 (+505741us) service_pool.cc:224] Handling call
0709 12:27:34.516226 (+1263773us) spinlock_profiling.cc:243] Waited 950 ms on lock 0x7b54002d0fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:34.525014 (+  8788us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2184779136}
I20260709 12:27:34.784951  5705 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:34.734448  5635 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:34.785679 12467 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:27:34.785728  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 220732us
I20260709 12:27:34.786337 12467 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:34.786741  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:34.771051  5701 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:34.788055  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:34.788475  5554 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:34.789069  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1209) took 2311 ms. Trace:
I20260709 12:27:34.734655  6277 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 17: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:34.713497  5708 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:34.790258 12599 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:34.790900 12599 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:34.791299 12373 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:34.791535  5708 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:34.792035 12373 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:34.792208 12599 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.793177  5708 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: [18.11-20.12]
I20260709 12:27:34.793291 12373 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.747092 12685 consensus_queue.cc:1243] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 16.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:27:34.752709  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:34.794864 12672 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:34.743763  5703 rpcz_store.cc:276] 0709 12:27:32.444098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.643588 (+1199490us) service_pool.cc:224] Handling call
0709 12:27:34.521215 (+877627us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:34.609590 (+ 88375us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:34.711856 (+102266us) spinlock_profiling.cc:243] Waited 877 ms on lock 0x7b54002d0fd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:34.712043 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2017013632}
I20260709 12:27:34.752713  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:34.748734 12657 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:34.795686  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:34.796003 12657 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:34.796278  5628 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:34.796692  5703 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:27:34.797011 12657 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:34.790890  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:34.799343 12537 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:34.735903  5459 catalog_manager.cc:5697] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 7 to 11, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 11 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:34.799870 12537 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.801163  5555 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:34.802233  5459 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 9 to 10, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 10 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:34.805729  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.806576  5552 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:27:34.807559  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.762876  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:34.763088 12256 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:34.764271  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.754782  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:34.772961  5707 pending_rounds.cc:107] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 12:27:34.747954  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:34.797353  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:34.789479  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:34.810155  5707 raft_consensus.cc:2889] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:34.785849  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:34.808931  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:34.814414  5552 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:34.812182  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:34.819402  5709 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
W20260709 12:27:34.819911  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:34.806579  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:34.821266  5559 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:34.786763 12467 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:34.823165  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:34.823365  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:34.826551  5561 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:34.825456  5703 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:27:34.827206  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:34.827899  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:34.829500  5701 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 19: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:34.831409  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:34.835219  5709 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:34.835451  5559 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
W20260709 12:27:34.836000 12687 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.218s	user 0.000s	sys 0.009s
I20260709 12:27:34.836160  5559 pending_rounds.cc:85] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 12
I20260709 12:27:34.836839  5559 pending_rounds.cc:107] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260709 12:27:34.836699  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:34.837133  5559 raft_consensus.cc:2889] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:34.837369  5631 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:34.837540 12548 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:27:34.838124 12666 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.001s	user 0.004s	sys 0.006s
I20260709 12:27:34.838814  5559 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:34.822978 12402 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:34.839465 12690 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.220s	user 0.002s	sys 0.004s
I20260709 12:27:34.836709 12193 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:34.841295 12402 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:34.841509 12193 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.841903  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:34.842245  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:34.843266  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:34.840574  5560 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
W20260709 12:27:34.847280  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:34.789440  5710 rpcz_store.cc:276] 0709 12:27:32.477590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:33.689756 (+1212166us) service_pool.cc:224] Handling call
0709 12:27:33.690134 (+   378us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:33.718946 (+ 28812us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:33.718985 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:33.732756 (+ 13771us) log.cc:844] Serialized 52 byte log entry
0709 12:27:34.102348 (+369592us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:34.104924 (+  2576us) log.cc:844] Serialized 14 byte log entry
0709 12:27:34.194134 (+ 89210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:34.198324 (+  4190us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:34.534030 (+335706us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b18004b5a40. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:27:34.788635 (+254605us) raft_consensus.cc:1685] finished
0709 12:27:34.788731 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:34.788904 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":116791,"prepare.queue_time_us":18432,"prepare.run_cpu_time_us":161,"prepare.run_wall_time_us":6854,"raft.queue_time_us":130660,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":2521,"spinlock_wait_cycles":9876224,"thread_start_us":254664,"threads_started":1,"wal-append.queue_time_us":411439}
I20260709 12:27:34.877068  5560 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:27:34.877068  5561 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
W20260709 12:27:34.871598  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:34.868373  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:34.880026  5459 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 7 to 11. New cstate: current_term: 11 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:34.908049  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:34.820048 12256 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:34.860998 12610 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:34.913528  5635 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:27:34.824741 12689 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.922362  5554 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:34.882524 12602 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:34.915948  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 57759.3us
I20260709 12:27:34.923560 12548 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.007s
I20260709 12:27:34.893153 12574 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:34.932224 12548 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:27:34.944218  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 430063us
I20260709 12:27:34.941239  5635 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:27:34.945169  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:34.893345 12603 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 12:27:34.941432 12286 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:34.940117 12193 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:34.961807  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:34.943348 12548 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:34.965775  5459 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 9 to 12. New cstate: current_term: 12 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:34.964694  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:34.968617 12286 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:34.969935  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 385157us
I20260709 12:27:34.970290  5560 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:34.955463 12579 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:34.958271 12426 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:34.945600 12602 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.983695  5552 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:27:34.958997 12685 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:34.944674 12574 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:34.995254  5635 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:34.986790  5636 pending_rounds.cc:85] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 9
I20260709 12:27:34.965538 12402 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:34.997372  5636 pending_rounds.cc:107] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260709 12:27:35.001099  5636 raft_consensus.cc:2889] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:34.972522 12541 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:34.989217  5705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2760) took 2817 ms (client timeout 1591 ms). Trace:
W20260709 12:27:35.002514  5705 rpcz_store.cc:269] 0709 12:27:32.172006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:32.200694 (+ 28688us) service_pool.cc:224] Handling call
0709 12:27:34.972938 (+2772244us) spinlock_profiling.cc:243] Waited 2.56 s on lock 0x7b540026f054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:34.989107 (+ 16169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5887216896}
I20260709 12:27:35.002496  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:35.008420  5555 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:27:34.979099 12437 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:35.011318  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:35.016382  5602 leader_election.cc:336] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.800s (SENT)
I20260709 12:27:34.999703 12402 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:35.020486  5451 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35600 (request call id 278) took 33481 ms (client timeout 15000 ms). Trace:
W20260709 12:27:35.032322  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:35.032027  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 80144.3us
I20260709 12:27:35.034492 12475 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:27:35.035228  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 74289.5us
W20260709 12:27:35.035725 12697 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:27:35.035965 12537 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:35.020696 12376 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:35.022070 12402 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:35.045926 12376 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:35.047756  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:35.028961 12541 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:35.052973 12708 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:27:35.053306 12557 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:27:35.054075 12541 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:35.057490  5557 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18. Ops: [17.14-17.14]
W20260709 12:27:35.059522 12698 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:27:35.061080  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:35.062703  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:35.064471 12595 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:35.064605 12599 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:35.065234 12595 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:27:35.065618 12689 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:35.046011 12548 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:35.067378 12548 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:35.070422 12602 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:35.036999 12437 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:35.072232  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 72355us
I20260709 12:27:35.011281 12685 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:35.075372 12685 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:35.076416 12256 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:35.077211  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:35.078964 12574 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:35.079954  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:35.049727 12475 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:35.081658 12475 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:35.083140 12286 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:35.087971 12657 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:35.031361  5459 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:35.094444  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:35.095275  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 72911.8us
I20260709 12:27:35.105327 12464 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:35.106006 12464 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:35.108071 12464 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:35.023121  5451 rpcz_store.cc:269] 0709 12:27:01.529081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:01.529795 (+   714us) service_pool.cc:224] Handling call
0709 12:27:35.006395 (+33476600us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:27:23.351387 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:27:23.420453 (+ 69066us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:27:23.420518 (+    65us) write_op.cc:435] Acquired schema lock
0709 12:27:23.420535 (+    17us) tablet.cc:662] Decoding operations
0709 12:27:23.436659 (+ 16124us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:27:23.436793 (+   134us) write_op.cc:641] Partition lock acquired for write op
0709 12:27:23.436828 (+    35us) tablet.cc:685] Acquiring locks for 38 operations
0709 12:27:23.439729 (+  2901us) tablet.cc:701] Row locks acquired
0709 12:27:23.439780 (+    51us) write_op.cc:260] PREPARE: finished
0709 12:27:23.439960 (+   180us) write_op.cc:270] Start()
0709 12:27:23.440111 (+   151us) write_op.cc:276] Timestamp: P: 1783600043439888 usec, L: 0
0709 12:27:23.440146 (+    35us) op_driver.cc:348] REPLICATION: starting
0709 12:27:23.440952 (+   806us) log.cc:844] Serialized 14527 byte log entry
0709 12:27:24.876837 (+1435885us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:25.364466 (+487629us) op_driver.cc:464] REPLICATION: finished
0709 12:27:25.612646 (+248180us) write_op.cc:301] APPLY: starting
0709 12:27:25.612779 (+   133us) tablet.cc:1367] starting BulkCheckPresence
0709 12:27:26.012060 (+399281us) tablet.cc:1370] finished BulkCheckPresence
0709 12:27:26.012105 (+    45us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:27:26.264323 (+252218us) spinlock_profiling.cc:243] Waited 6.46 ms on lock 0x7b18002e2880. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa3b10ec 00007f4ffa3b3af7 00007f4ffa3b4fb0 00007f4ffe0c91bb 00007f4ffe1105d4 00007f4ffdfbc8da 00007f4ffdfbf3cb 00007f4ffdfbefd3 00007f4ffe0bde34
0709 12:27:27.095327 (+831004us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:27:27.095828 (+   501us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=76,key_file_lookups=76,delta_file_lookups=76,mrs_lookups=0
0709 12:27:27.095894 (+    66us) write_op.cc:312] APPLY: finished
0709 12:27:27.296802 (+200908us) log.cc:844] Serialized 324 byte log entry
0709 12:27:27.477299 (+180497us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:27:27.477886 (+   587us) write_op.cc:454] Released schema lock
0709 12:27:27.481151 (+  3265us) write_op.cc:341] FINISH: Updating metrics
0709 12:27:27.554220 (+ 73069us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:27:27.556619 (+  2399us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":171154,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":230390,"apply.run_cpu_time_us":84310,"apply.run_wall_time_us":1924399,"cfile_cache_hit":154,"cfile_cache_hit_bytes":162967,"num_ops":38,"prepare.queue_time_us":752604,"prepare.run_cpu_time_us":47511,"prepare.run_wall_time_us":574069,"raft.queue_time_us":158689,"raft.run_cpu_time_us":37758,"raft.run_wall_time_us":305431,"replication_time_us":1740270,"spinlock_wait_cycles":26927616,"thread_start_us":1561551,"threads_started":5,"wal-append.queue_time_us":796837,"wal-append.run_cpu_time_us":55707,"wal-append.run_wall_time_us":3033077}]]}
I20260709 12:27:35.115336  5459 catalog_manager.cc:5697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 7 to 12, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 12 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:35.049764 12376 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:35.118175 12437 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:35.137181  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 86717us
I20260709 12:27:35.104291 12589 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:35.139017 12535 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:35.147352 12373 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:27:35.148221  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53978.5us
I20260709 12:27:35.149502 12715 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:35.151541  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 141505us
I20260709 12:27:35.153518  5459 catalog_manager.cc:5697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 6 to 7, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 7 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:35.173688  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:35.124111 12402 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:35.180122  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 100256us
I20260709 12:27:35.179481 12535 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:35.185640 12535 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:27:35.188807  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 50673.6us
I20260709 12:27:35.160827 12373 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:35.201774  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:35.201541  5459 catalog_manager.cc:5697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 7 to 10, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 10 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:35.205595  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:35.213888  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:35.216069  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:35.200883 12373 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:35.220549 12470 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:35.221501 12470 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:35.209460 12537 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:35.173954 12715 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:35.239856  5602 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.125s (SENT)
I20260709 12:27:35.243474  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:35.208510 12286 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:35.231004 12595 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:35.207454 12376 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:35.239636 12589 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:35.259982  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:35.266224  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:35.267426  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 77338.6us
I20260709 12:27:35.270526  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:35.242933 12426 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:35.272599  5707 raft_consensus.cc:2410] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 22 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 12:27:35.255188 12256 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:35.277623  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:35.260646 12574 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:35.280815  5710 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 15.
I20260709 12:27:35.265996 12689 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:27:35.289283  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 83765.8us
W20260709 12:27:35.303787  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63964.3us
I20260709 12:27:35.286100 12589 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:35.314312 12715 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:35.343611  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:35.299563 12426 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:35.343571  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:35.315737  5555 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:35.289693 12286 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:35.347275  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 133867us
W20260709 12:27:35.348311  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:35.344736  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:35.317245 12689 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:35.319892 12529 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.545s	user 0.012s	sys 0.001s
I20260709 12:27:35.320602  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:35.322300 12467 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:35.323212  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:35.275354 12470 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:35.324265 12402 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:35.324645 12376 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:35.327248  5524 leader_election.cc:400] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
I20260709 12:27:35.392616  5524 leader_election.cc:403] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:27:35.327670 12541 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:27:35.395148  5702 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:27:35.279495 12256 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:35.332858  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:35.333786  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:35.399153  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 65459.5us
W20260709 12:27:35.334528 12696 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.361s	user 0.002s	sys 0.000s
I20260709 12:27:35.302476 12716 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 12:27:35.374611  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:35.385686  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:35.388875  5706 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:35.273448 12673 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:35.397634  5707 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:35.404560  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:35.399843  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:35.407024  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54718.7us
W20260709 12:27:35.407100  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 74362.9us
I20260709 12:27:35.408691  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:35.393507 12602 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:35.397461 12541 consensus_queue.cc:1243] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 19.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:27:35.404304 12256 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:35.423271  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 48734.6us
I20260709 12:27:35.425613  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:35.418242  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:35.409256 12537 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:35.404516 12548 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:27:35.424422  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 80930.4us
I20260709 12:27:35.431420 12286 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:35.433212 12467 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:35.421201 12376 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:35.435047  5633 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:35.438795  5707 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:35.440615 12675 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:35.415586 12529 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
I20260709 12:27:35.419466 12402 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:35.344736 12589 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:35.498773  5635 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:35.499532  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 74048.1us
I20260709 12:27:35.501775  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:35.502537 12692 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:27:35.505422  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 178352us
W20260709 12:27:35.525846  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:35.505730  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:35.499972 12675 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:35.508118  5631 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:35.509002 12727 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:35.510141 12470 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:35.513203 12286 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:35.532166  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:35.536547  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 102693us
W20260709 12:27:35.537420  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:35.524190 12402 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:35.504520 12603 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:27:35.540683 12402 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:35.541242  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:35.542282  5632 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:35.542387 12470 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:35.543141  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:35.527272 12692 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:35.544651 12692 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:35.545146 12470 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:35.547680  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:35.549214  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:35.517853 12711 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.445s	user 0.002s	sys 0.000s
I20260709 12:27:35.521468  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1362) took 1153 ms. Trace:
I20260709 12:27:35.522694  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:35.449800 12464 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:35.506512 12537 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:35.563851 12681 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:27:35.528954  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:35.566099  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:35.529852  5633 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:35.552326  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:35.556164  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:35.561676 12685 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:35.561721 12181 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:35.528478 12529 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:35.573467  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 164847us
I20260709 12:27:35.558492  5634 rpcz_store.cc:276] 0709 12:27:34.367525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:34.370683 (+  3158us) service_pool.cc:224] Handling call
0709 12:27:34.371064 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:34.371641 (+   577us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:34.371668 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:34.372250 (+   582us) log.cc:844] Serialized 26 byte log entry
0709 12:27:34.474422 (+102172us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:34.474492 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:34.475138 (+   646us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:35.333858 (+858720us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:35.520972 (+187114us) raft_consensus.cc:1685] finished
0709 12:27:35.521000 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:35.521285 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1816,"raft.queue_time_us":19784,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":4805,"spinlock_wait_cycles":27898112,"thread_start_us":94123,"threads_started":1,"wal-append.queue_time_us":561056}
I20260709 12:27:35.574056 12181 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:35.574786 12541 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:35.553248 12548 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:27:35.577956  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44440us
I20260709 12:27:35.599480  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:35.574065 12286 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:35.581501 12685 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:35.567255 12673 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:35.585114 12689 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:35.530956 12728 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:35.533999 12675 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:35.588835  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 76999.9us
I20260709 12:27:35.588418 12181 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:35.572785 12579 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:35.564314 12681 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:35.607568  5635 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:35.611922  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:35.590488 12529 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:35.590214 12602 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:35.637218  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:35.670322  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 104304us
I20260709 12:27:35.680806  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:35.681766  6277 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:35.681893 12286 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:35.682924 12470 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:27:35.673465 12610 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:35.685144 12595 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:35.685730 12610 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:35.688565  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 136376us
I20260709 12:27:35.694645 12529 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:35.669665 12733 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:27:35.696229  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:35.688138 12610 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:35.637077 12681 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:35.827525  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 258545us
I20260709 12:27:35.623298 12673 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:35.830761  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:35.827905  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 131792us
W20260709 12:27:35.836544  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 292236us
I20260709 12:27:35.832134  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:35.839069  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:35.843181 12692 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:35.845297  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:35.847160  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:35.820130 12376 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:35.848727 12376 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:35.848982  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:35.849781  5634 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:35.850986 12376 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:35.851506 12675 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:35.623027 12728 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:35.676381 12256 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:35.837846 12373 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.592s	user 0.011s	sys 0.002s
I20260709 12:27:35.834251 12716 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:35.865167  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1317) took 1087 ms. Trace:
I20260709 12:27:35.837713 12733 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:35.869130  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:35.871552  5560 rpcz_store.cc:276] 0709 12:27:34.778071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:34.878402 (+100331us) service_pool.cc:224] Handling call
0709 12:27:34.966568 (+ 88166us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:34.976838 (+ 10270us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:35.864887 (+888049us) spinlock_profiling.cc:243] Waited 87.8 ms on lock 0x7b540044dc50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:35.865076 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":201864320}
I20260709 12:27:35.859917 12541 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:35.917188  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48118.4us
I20260709 12:27:35.861881 12574 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:35.864292 12579 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:35.869519 12716 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:35.869567 12714 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:27:35.917191  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52591.6us
I20260709 12:27:35.908622 12681 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:27:35.904491 12376 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:35.929697 12541 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:35.870842 12685 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.058s
I20260709 12:27:35.868575 12738 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.059s
I20260709 12:27:35.924289 12589 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:35.923413 12579 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:35.941761 12181 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:35.949913 12421 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:27:35.952981 12737 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.258s	user 0.002s	sys 0.000s
I20260709 12:27:35.963006 12402 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:35.968854 12728 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:35.970777 12675 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:35.971701 12529 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:35.915114 12602 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:35.930922 12574 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:35.933861  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:35.869854 12373 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:35.927863 12714 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:35.928596 12716 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:35.975032 12256 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:36.049286  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:35.977746 12742 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.011354 12286 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:36.014848  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:36.077250  5701 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:36.006274 12535 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.020071  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:36.021008  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:36.021716  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:36.080075  5636 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:36.022097  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.024777  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:35.922603 12741 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.080s
I20260709 12:27:35.993490 12589 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:36.021729  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:35.998628 12579 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:35.916304 12733 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:36.036739  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 182086us
I20260709 12:27:36.042048  5559 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 14. (term mismatch)
I20260709 12:27:36.002789 12421 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:35.981274 12376 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.110942  5634 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:27:36.043084 12602 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:36.085157  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:36.081382  5633 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:36.116657  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:36.091094  5555 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:27:36.090799  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:36.090226  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:36.118098  5554 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:36.093130  5627 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:36.083649  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:36.119443  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:36.119704  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:36.100724  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:36.095911  5559 pending_rounds.cc:85] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 13
I20260709 12:27:36.084455  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.120728  5559 pending_rounds.cc:107] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 16.14
I20260709 12:27:36.121162  5559 raft_consensus.cc:2889] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:36.113408  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.047917 12714 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:36.029799 12529 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:36.090245 12589 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:36.113291  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:36.139379  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:36.172856  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:36.109666 12256 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:36.109069 12470 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.121169 12741 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:27:36.181234  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 247445us
I20260709 12:27:36.110488 12610 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:36.176697  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 127468us
I20260709 12:27:36.168836  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.181861  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.193128  5559 raft_consensus.cc:2410] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 21 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 15.
I20260709 12:27:36.193338  5710 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:27:36.091799 12535 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:36.182111  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:36.128399 12715 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:36.124209 12692 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:36.138486 12673 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:36.108837 12286 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:36.198416  5633 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:27:36.096765 12742 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:36.241597  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 59559.8us
I20260709 12:27:36.183248 12657 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.253975  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:36.200557  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.260808  5559 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:27:36.268786  5636 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:36.294047  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:36.186700 12741 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:36.252075 12675 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.200131 12685 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:36.302989  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:36.123404 12421 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.305168  5559 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:36.308459  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:36.258782 12751 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:36.323145  5554 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:36.199582 12256 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.189857 12470 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.212237 12610 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.298029 12741 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:36.304822 12685 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:36.366356  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:36.360194  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 263817us
I20260709 12:27:36.363219  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:36.283517 12657 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.351011 12402 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:36.379643  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 130609us
I20260709 12:27:36.344758 12181 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:36.307560 12675 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:36.391028  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:36.394526  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:36.394692  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:36.395254  5710 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:36.401002  5706 raft_consensus.cc:2410] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:27:36.405934  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:36.408910  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:36.409809  5704 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:36.376590 12373 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:36.337455 12751 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.418191  5702 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:36.296435 12602 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:36.392791 12759 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:36.425200  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:36.385802 12574 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:36.421278  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 167379us
I20260709 12:27:36.390931 12535 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:36.431391  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:36.435321  5559 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:36.442144  5633 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:36.447118  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.449019  5559 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:36.426048 12376 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:36.437897 12529 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:36.438342 12373 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.464685  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:36.468593 12286 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:36.427995 12426 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 12:27:36.491900  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.479417  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.480928  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:36.447305 12733 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:36.493346  5553 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:36.493791 12733 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:36.494458 12733 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:36.495169 12763 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:27:36.496667 12657 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:36.491775 12181 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:36.450068 12685 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:36.502609  5555 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:27:36.502830 12751 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:36.503793  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:36.504696  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:36.505330  5554 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:36.508122  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:36.508739 12757 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.223s	user 0.000s	sys 0.001s
I20260709 12:27:36.509819 12610 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:36.481593 12373 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:36.464052 12748 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.276s	user 0.000s	sys 0.001s
I20260709 12:27:36.502965  5561 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:27:36.504812  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:36.513550  5555 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:36.516921  5701 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:36.518174  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:36.514334  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49765.8us
I20260709 12:27:36.437940 12768 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.063s
I20260709 12:27:36.526144  5553 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 12:27:36.543725  5704 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:36.467248 12437 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:36.482339 12529 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.565361  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.482676  5710 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:36.568060  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:36.579079  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:36.579933  5561 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:36.523237 12181 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:36.587738 12579 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:36.592535 12656 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:36.593286  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.594328  5559 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:36.595779  5559 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:27:36.586923  5709 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:36.598769  5703 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:36.607825  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:36.608650  5633 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:36.519073 12373 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:36.593209 12579 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:36.621059  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53076.5us
I20260709 12:27:36.622207 12657 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:27:36.601483 12529 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.626981 12733 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:36.627769 12767 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:36.593179 12656 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.630318 12773 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:36.630914 12773 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:36.636341  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51422.9us
I20260709 12:27:36.538218 12574 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.108s
I20260709 12:27:36.661199  5555 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:36.662715  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:36.664196  5633 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:36.664842  5559 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:36.607358 12742 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:36.643970 12256 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:36.615864 12437 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:36.692008  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 178312us
W20260709 12:27:36.692744  5630 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.015s	sys 0.013s
W20260709 12:27:36.693538  5630 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.015s	sys 0.013s
I20260709 12:27:36.665912 12579 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.672348 12529 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:36.684638  5561 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:36.665201 12470 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:36.706153 12761 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:36.706178  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.715440  5555 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:36.707100 12776 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.683192  5554 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:36.708969 12718 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:27:36.711701  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:36.672971 12752 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:36.680886 12595 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:36.720073  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:36.699708 12181 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:36.723519  5561 raft_consensus.cc:2376] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18. Re-sending same reply.
I20260709 12:27:36.726828  5630 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:27:36.733158  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:36.741246  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 118515us
I20260709 12:27:36.743471  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:36.742987  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:36.746217  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:36.747493  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:36.748447  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.024s	sys 0.033s
W20260709 12:27:36.748914  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.024s	sys 0.033s
W20260709 12:27:36.749658  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 623460us
I20260709 12:27:36.751406  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.722645 12718 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:36.726739 12685 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:36.770272  5602 leader_election.cc:336] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.713s (SENT)
I20260709 12:27:36.713054 12579 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.772296  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:27:36.742106 12771 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.110s	user 0.001s	sys 0.001s
I20260709 12:27:36.754556 12733 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:27:36.770166 12376 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:36.807509  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:36.723315 12776 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:36.817884  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.809016  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.730537 12256 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.804273  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:36.820806 12256 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.823035 12256 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:36.823854 12724 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:36.824900  5561 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:36.833484  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:36.834451  5703 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:36.834652  5709 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:27:36.834950 12718 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.835986 12656 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:36.836344 12718 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:36.741227 12595 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.774055 12541 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:36.846644  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2697) took 1504 ms. Trace:
I20260709 12:27:36.807055  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:36.805285  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:36.800051 12751 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:36.863090  5555 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:36.873276  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 65841.6us
I20260709 12:27:36.874764  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:36.843348 12595 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:36.815970 12773 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:36.847131  5627 rpcz_store.cc:276] 0709 12:27:35.342451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:35.344400 (+  1949us) service_pool.cc:224] Handling call
0709 12:27:36.846405 (+1502005us) spinlock_profiling.cc:243] Waited 740 ms on lock 0x7b54001f2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:36.846540 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1701192064}
I20260709 12:27:36.849344 12738 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:36.813756  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:36.818439 12205 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:27:36.887022 12738 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:36.887496 12738 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:36.887779 12205 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:36.888198 12205 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:36.889777 12205 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.890452 12437 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:36.876222 12256 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:36.745859 12181 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.824316 12724 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:36.891254 12673 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:36.896693  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:36.809717 12286 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:27:36.892716 12681 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.913s	user 0.014s	sys 0.000s
I20260709 12:27:36.799448 12535 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.080s
I20260709 12:27:36.891844  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:36.884781  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.834496 12579 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:36.730471 12752 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.911015 12675 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:27:36.894450  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:36.914706  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:36.915545  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:36.916669 12256 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:36.917160 12256 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:36.918099 12769 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.245s	user 0.000s	sys 0.000s
I20260709 12:27:36.918320 12402 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:36.918821 12402 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:36.919173 12402 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:36.921521  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.923352  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.925777 12181 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:36.924304  5560 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:36.811327 12733 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:36.885078  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.848017 12603 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:36.879151 12751 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:36.898239 12780 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.089s	user 0.000s	sys 0.004s
I20260709 12:27:36.891206 12205 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:36.908324  6278 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:36.900548 12724 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:36.893601 12610 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:36.843068  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:36.933456  6277 raft_consensus.cc:2410] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:27:36.934294  5676 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:36.895361 12776 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:36.938211  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:36.908921 12727 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:36.934938 12752 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:36.937675 12376 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:27:36.971997  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 201756us
I20260709 12:27:36.951844  5555 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
W20260709 12:27:36.955742  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 133801us
I20260709 12:27:36.945063  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:36.959498  5553 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:36.938585 12733 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:36.967851 12181 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:36.957309 12535 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:36.976965  5602 leader_election.cc:336] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.724s (SENT)
I20260709 12:27:36.977767  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:36.937561 12603 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:36.947443  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:36.972020  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.936939 12286 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.032s
W20260709 12:27:36.971122 12766 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.428s	user 0.005s	sys 0.000s
I20260709 12:27:36.981618  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.954850 12784 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:36.994267 12784 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.994575  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.995904  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.980289  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:36.996691  5552 raft_consensus.cc:2376] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19. Re-sending same reply.
I20260709 12:27:36.997606  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:36.998641  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:36.999631  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:37.000689 12752 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:37.000837  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:37.001929 12773 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:37.002111  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:37.002542  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:37.002488 12773 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.004695 12779 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:37.004905 12776 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:37.005398 12776 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.005754 12776 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:36.976295 12579 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:27:36.987905  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:36.987568  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:36.918126 12256 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.010032  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1377) took 1174 ms. Trace:
I20260709 12:27:37.010426  5628 rpcz_store.cc:276] 0709 12:27:35.835284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:35.853366 (+ 18082us) service_pool.cc:224] Handling call
0709 12:27:35.856747 (+  3381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:36.077472 (+220725us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:27:36.337529 (+260057us) log.cc:844] Serialized 14 byte log entry
0709 12:27:36.650957 (+313428us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:36.677199 (+ 26242us) log.cc:844] Serialized 26 byte log entry
0709 12:27:36.677543 (+   344us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:36.677580 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:36.677656 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:37.009824 (+332168us) raft_consensus.cc:1685] finished
0709 12:27:37.009842 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:37.009978 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30051,"prepare.queue_time_us":279491,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":3408,"raft.queue_time_us":125671,"raft.run_cpu_time_us":355,"raft.run_wall_time_us":28654,"thread_start_us":614792,"threads_started":3,"wal-append.queue_time_us":361806}
I20260709 12:27:36.974226 12376 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:36.991815 12286 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.020494 12286 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.022915 12286 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:37.023521 12286 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:37.023849 12286 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:36.992473 12535 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.030033  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:36.979849 12741 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.598s	user 0.010s	sys 0.001s
I20260709 12:27:36.992980 12603 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:36.979063 12181 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.030654  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:36.977574 12727 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.048830  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:37.049841  5703 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:37.008936  5636 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:37.048830  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:36.943879 12537 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 12:27:37.012259  5552 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:27:37.024227 12286 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:36.991923 12589 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.809s	user 0.006s	sys 0.007s
I20260709 12:27:37.056855  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:37.014683  5632 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 23. Current term is 24. Ops: []
I20260709 12:27:37.014999  5559 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:37.013873 12599 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.004s
I20260709 12:27:37.018129 12376 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:37.018255  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:37.018889 12784 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:37.019898 12752 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:27:37.033560  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 99340.2us
I20260709 12:27:37.059821  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:37.064507  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:37.059108  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:36.987393  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:37.065896  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:37.067329  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:37.050860  6279 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:37.053987  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:37.057597  5706 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:37.071185  5703 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: [20.15-22.16]
I20260709 12:27:37.072861  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:37.073300 12603 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:27:37.074692 12716 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.987s	user 0.006s	sys 0.007s
I20260709 12:27:37.075265 12716 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.049008 12181 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:37.077785 12738 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.081809 12535 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:37.018178 12402 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.084699 12402 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:37.084455 12589 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.086851 12589 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:37.060165 12751 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.090325 12752 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:37.091044 12752 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:37.091455 12752 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:37.093204  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:37.096118  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:37.096573  5629 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:37.097859  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:37.098229 12727 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:37.100638 12773 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:37.100896 12548 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:37.056715 12373 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 12:27:37.102035 12673 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:37.068614  5702 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:37.104486  5556 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:37.100081  5555 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 12. (term mismatch)
I20260709 12:27:37.106207  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:37.108093  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:37.105630  5555 pending_rounds.cc:85] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 11
I20260709 12:27:37.113526  5555 pending_rounds.cc:107] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 20.12
I20260709 12:27:37.113844  5555 raft_consensus.cc:2889] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:37.049696 12768 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:37.100692 12762 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:37.133857  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:37.114517 12610 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:37.142728  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:37.154897  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:37.113904 12714 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.938s	user 0.010s	sys 0.002s
I20260709 12:27:37.105477 12681 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:37.102459 12738 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:37.052338 12741 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.128434 12535 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:37.128055 12373 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:37.172262  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:37.174901  5555 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:27:37.177443  5554 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:37.096609 12778 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.096s
I20260709 12:27:37.149537 12718 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:37.182569  5703 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:27:37.189359  5558 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:37.191715  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:37.194031  5706 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:37.161062 12762 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:37.125614 12733 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.169512 12373 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:37.164407 12714 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.168035 12798 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:37.207419  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:37.209733  5709 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 16: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:37.215941 12376 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.185703 12535 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:37.186323 12718 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.138132 12768 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.216332 12762 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.158798 12724 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.223258  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:37.231423  5557 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:37.235301  5710 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 21: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:37.240724 12181 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.240970 12535 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.242959 12181 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:37.249766  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:37.250178  5702 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:27:37.250535  5701 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 19: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:37.250747  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:37.251322  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:37.253510  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:37.253706 12799 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:37.254585  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:37.255525 12733 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:37.256002  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2908) took 1132 ms. Trace:
I20260709 12:27:37.255894 12599 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:27:37.251893  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:37.259071  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:37.259827  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:37.256520  5558 rpcz_store.cc:276] 0709 12:27:36.124069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:36.124195 (+   126us) service_pool.cc:224] Handling call
0709 12:27:37.255817 (+1131622us) spinlock_profiling.cc:243] Waited 994 ms on lock 0x7b54000271d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:37.255944 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2285097856}
I20260709 12:27:37.223104 12718 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:37.296370  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:37.301607  5704 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 19: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:37.299055  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:37.223819 12798 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:37.297670  5707 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:27:37.256899 12776 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.297488  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:37.257296 12599 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:37.324297  5706 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:27:37.334340  5708 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:37.335033  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:37.335475  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:37.340850  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:37.341566  5628 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:37.196635 12610 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.344588  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:37.322999 12804 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:37.301007 12537 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:37.355921  5707 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:37.302736 12548 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:37.300668 12673 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:37.297281 12733 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:37.359481  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:37.358958  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 261184us
I20260709 12:27:37.366544  5704 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:37.303546 12376 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:37.368913  6279 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:27:37.369709  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:37.371654  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:37.341727 12464 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:37.372170  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:37.378793  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:37.340869 12716 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:37.353814 12256 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:37.231442 12373 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.358769 12778 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:37.359560 12537 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.351178 12804 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:37.392547  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:37.393128  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 125295us
I20260709 12:27:37.360954 12599 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:37.396934  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:37.405055 12548 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:37.406970 12548 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.353101 12727 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 12:27:37.401129 12599 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.417543  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:37.427644  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:37.428488  5704 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:37.424073  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:37.395663 12778 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.398797 12537 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:37.370743 12286 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.442847  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:37.450367 12373 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:37.450810 12768 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:37.451881 12741 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:37.456079  6279 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:37.452178 12421 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:37.439471 12778 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:37.465679 12768 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.012s
W20260709 12:27:37.475346 12802 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:27:37.475382  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:37.487517  6276 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:27:37.485110 12714 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:37.484087 12402 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.459815 12798 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.503396 12811 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 12:27:37.504982 12786 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.041s
I20260709 12:27:37.508903  6276 raft_consensus.cc:606] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:37.532913  6276 raft_consensus.cc:614] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:27:37.535308  5628 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:37.528088  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2903) took 1498 ms. Trace:
I20260709 12:27:37.536675  5557 rpcz_store.cc:276] 0709 12:27:36.029553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:36.042104 (+ 12551us) service_pool.cc:224] Handling call
0709 12:27:37.527845 (+1485741us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54000eb5d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:37.528007 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2546478848}
W20260709 12:27:37.502137  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 119146us
I20260709 12:27:37.543706  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:37.550209  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2902) took 1504 ms. Trace:
I20260709 12:27:37.508533  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:37.551018  5556 rpcz_store.cc:276] 0709 12:27:36.029183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:36.029331 (+   148us) service_pool.cc:224] Handling call
0709 12:27:36.073304 (+ 43973us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:37.110849 (+1037545us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:37.533504 (+422655us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54001bdd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:37.533698 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2370953472}
I20260709 12:27:37.500192 12791 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:37.514156 12801 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 12:27:37.551604  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:37.518435 12286 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:37.547492 12799 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:27:37.572564  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 146574us
I20260709 12:27:37.466780 12804 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.499189 12768 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.072s
I20260709 12:27:37.566378 12776 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:37.574525 12464 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:37.518440 12402 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.591557  6278 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:37.594298  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:37.595196  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:37.595454  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:37.596493 12816 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:37.599099  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:37.599810  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:37.600777  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:37.603615  5705 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:27:37.604151  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:37.604449  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1189) took 1773 ms. Trace:
I20260709 12:27:37.604597  5708 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:37.605845  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:37.532158 12752 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.606954 12724 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:37.612838  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:37.614081  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:37.618903  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:37.622802  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:37.640520  5560 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:27:37.633368  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:37.627496  6277 rpcz_store.cc:276] 0709 12:27:35.831120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:35.831255 (+   135us) service_pool.cc:224] Handling call
0709 12:27:37.604189 (+1772934us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54002a8d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:37.604335 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2505612544}
I20260709 12:27:37.648407  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:37.668540  5552 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:37.639801  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:37.652307  5557 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 12:27:37.654109  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:37.667873  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:37.670612  5558 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:37.671067  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:37.657647  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:37.652525  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:37.672279  5554 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:37.672773  5553 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:37.661319  5702 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:37.662027  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:37.662810  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:37.664222  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:37.665307  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:37.676419  5561 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:37.572327 12791 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.643436 12751 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:37.670090  6278 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:37.660851  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:37.679416  5707 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:37.577324 12733 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.076s
I20260709 12:27:37.657035 12727 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:37.578856 12797 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.085s
I20260709 12:27:37.692219  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:37.695169  5555 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:37.685199  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:37.696882  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:37.701169  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68155.9us
I20260709 12:27:37.532099 12718 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.705775  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:37.611378 12778 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.716189  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:37.757771  6278 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
W20260709 12:27:37.759922  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 56901.8us
I20260709 12:27:37.762203  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:37.768680  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 76786.9us
I20260709 12:27:37.772761  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:37.773660  5556 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 22
W20260709 12:27:37.691917 12818 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:27:37.783684  5707 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:37.784972  5554 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:37.790189  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:37.805936  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 43837.5us
I20260709 12:27:37.681895 12754 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:37.663951 12610 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.095s
I20260709 12:27:37.687427 12727 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:37.687134 12673 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:37.813337  5633 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:27:37.769379 12760 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:27:37.814003  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 97912.1us
I20260709 12:27:37.814813  5558 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:37.789335  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:37.827992  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:37.758800 12776 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:37.819394  5555 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:37.816236  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:37.811264 12529 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:37.837714  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:37.838220 12754 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:37.812337 12714 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:37.841934  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:37.842820  6272 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 18: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:37.826829 12727 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:37.788093 12181 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 12:27:37.830443 12776 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.830791 12773 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:37.810894 12798 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 12:27:37.842161 12741 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:37.836761 12376 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.021s
I20260709 12:27:37.865478 12778 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:37.831952 12537 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:37.867566 12825 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:27:37.850768 12776 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:37.876679 12402 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:37.878010 12791 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:37.851162 12437 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:37.879473  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:37.904428  5554 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:37.879869 12773 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:37.881235 12752 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:37.882565 12718 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:37.883622  6271 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:37.885197 12610 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:37.910354 12610 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:37.910677 12773 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.911243 12610 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.912598 12610 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:37.905758 12402 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:37.913329 12402 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:37.914129 12402 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.885306 12673 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:37.891637  5553 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:37.896844  5558 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:37.896050  5556 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:37.916096  5708 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 19: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:37.888304 12529 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 12:27:37.923746 12751 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:37.872660 12754 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:37.948935  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:37.955114  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:37.946722  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:37.957384  5554 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:27:37.960855 12751 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:37.963474 12751 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.963950 12751 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:37.964560 12537 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:37.960271  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 149681us
I20260709 12:27:37.973502  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:37.961067  5552 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:27:37.995821  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:37.999514 12798 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:38.003553  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:38.008016  5636 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:27:38.003140  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 156740us
I20260709 12:27:38.004048  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:38.013788  5524 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:38.017678  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:38.020417  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:38.010099  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:38.022059  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:38.023658  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:38.026870  5676 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:38.003532 12798 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:38.014482 12537 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:38.028339 12741 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:38.095291  6273 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:38.083561 12776 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.109896 12776 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:38.107632 12836 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:38.083891 12798 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:38.082655 12537 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:38.110695 12776 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.012s
I20260709 12:27:38.182966  5636 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:38.218653  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:38.219178  5636 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:38.169008 12836 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:38.175680 12537 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:38.183538 12741 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:38.218276 12181 consensus_queue.cc:1243] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 19.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:27:38.221936 12760 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:38.241472  5636 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
W20260709 12:27:38.242816  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 216029us
I20260709 12:27:38.244911  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:38.246764  5636 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:38.229951 12834 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:38.246773  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:38.253047 12535 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:38.253624 12535 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:38.254508 12535 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:38.257157 12831 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 12:27:38.258926  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1389) took 1271 ms. Trace:
I20260709 12:27:38.260015  5635 rpcz_store.cc:276] 0709 12:27:36.987739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:36.988469 (+   730us) service_pool.cc:224] Handling call
0709 12:27:36.988739 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:36.988937 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:27:37.188107 (+199170us) log.cc:844] Serialized 14 byte log entry
0709 12:27:38.257959 (+1069852us) spinlock_profiling.cc:243] Waited 5.23 ms on lock 0x7b180018e620. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287
0709 12:27:38.258362 (+   403us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:38.258440 (+    78us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:38.258508 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:38.258638 (+   130us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:38.258829 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1661,"prepare.queue_time_us":230285,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":74,"raft.queue_time_us":54171,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":10211,"spinlock_wait_cycles":12024064,"thread_start_us":430919,"threads_started":3,"wal-append.queue_time_us":256333}
I20260709 12:27:38.272691 12835 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:38.273313 12835 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:38.273420 12804 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:38.242105 12799 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:27:38.289271  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 267248us
I20260709 12:27:38.246284 12840 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.297462  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:38.299690  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:38.306447 12714 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:38.307077 12714 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:38.313845 12760 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:38.314522 12760 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:38.318662  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3049) took 1056 ms. Trace:
I20260709 12:27:38.319200  5702 rpcz_store.cc:276] 0709 12:27:37.261801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:37.296298 (+ 34497us) service_pool.cc:224] Handling call
0709 12:27:37.297212 (+   914us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:37.673518 (+376306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:38.318319 (+644801us) spinlock_profiling.cc:243] Waited 364 ms on lock 0x7b54000e0bd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:38.318578 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":836796672}
I20260709 12:27:38.320863 12835 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:38.327194  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:38.328493 12836 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.329649 12733 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:38.330247 12733 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:38.333007 12840 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.335254 12840 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:38.338210  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:38.294010 12751 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.341163 12751 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:38.249763 12727 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.343957 12727 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:38.345008 12839 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
W20260709 12:27:38.349433  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 102771us
I20260709 12:27:38.351132  5560 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: [22.13-24.14]
I20260709 12:27:38.352784  5555 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:27:38.356896 12742 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.357488 12742 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.359617 12742 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:38.312139 12820 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.499s	user 0.005s	sys 0.000s
I20260709 12:27:38.369861  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:38.370697  5552 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:38.372201  5635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:38.373284  5635 raft_consensus.cc:606] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:38.375785  5635 raft_consensus.cc:614] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:27:38.376053  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:38.378707  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:38.379585  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:38.380411  5561 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:38.379664  5702 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:38.381736  5707 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: [20.15-22.16]
I20260709 12:27:38.383227  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:38.383998  5561 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:38.384330  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:38.385943  5555 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:27:38.387779  5560 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:27:38.389353  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:38.331344 12733 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.395979  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:38.396608  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:38.391770 12832 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:38.400038 12848 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:38.308230 12714 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.400640 12848 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:38.400975 12840 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:27:38.401535 12848 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.369601 12742 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.406659  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:38.420198  5559 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 15: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:38.422688  5709 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:38.432868  5636 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
W20260709 12:27:38.435364  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:38.413161 12846 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.420068 12776 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 12:27:38.443249 12776 consensus_queue.cc:1243] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 19.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:27:38.424023 12742 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.431926 12832 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:38.446007 12742 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:38.445778 12832 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.446954  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1239) took 1515 ms. Trace:
I20260709 12:27:38.447343  6271 rpcz_store.cc:276] 0709 12:27:36.931381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:36.931583 (+   202us) service_pool.cc:224] Handling call
0709 12:27:38.446793 (+1515210us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b5400772754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:38.446893 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2081265920}
I20260709 12:27:38.453547  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:38.454288  5707 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:38.439338 12205 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:38.455405  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:38.456081  5707 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:38.455127 12205 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.460249  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:38.448710 12846 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.439177 12834 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:38.465734 12846 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:38.470021 12256 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:38.470575 12256 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:38.471483 12256 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.473922 12256 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:38.474169 12205 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:38.473675 12798 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.476805  5706 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:38.465479 12834 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.482401  5707 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:38.483582  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:38.486956  5556 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:27:38.488612  5703 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:27:38.490595  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:38.507246  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 1391) took 1365 ms. Trace:
W20260709 12:27:38.542002  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:38.507831  5632 rpcz_store.cc:276] 0709 12:27:37.142189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:37.142409 (+   220us) service_pool.cc:224] Handling call
0709 12:27:38.498088 (+1355679us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b54001de2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:38.507144 (+  9056us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1164662528}
I20260709 12:27:38.489926 12846 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:38.546320  5703 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18. Ops: []
W20260709 12:27:38.520730  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.523s (SENT) [suppressed 20 similar messages]
I20260709 12:27:38.584848  5701 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: [20.13-20.13]
W20260709 12:27:38.586558  5710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2811) took 1556 ms (client timeout 1523 ms). Trace:
W20260709 12:27:38.587083  5710 rpcz_store.cc:269] 0709 12:27:37.029671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:37.029774 (+   103us) service_pool.cc:224] Handling call
0709 12:27:38.584366 (+1554592us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b54002d23d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:38.586467 (+  2101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":469834112}
I20260709 12:27:38.599128  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:38.545075 11997 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.098s
I20260709 12:27:38.548132 12846 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:38.500912 12767 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:27:38.609493 12829 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
W20260709 12:27:38.658614  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 138047us
I20260709 12:27:38.597028 12437 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:38.665542  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1199) took 1317 ms. Trace:
W20260709 12:27:38.667110  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:38.665946  6272 rpcz_store.cc:276] 0709 12:27:37.348149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:37.348319 (+   170us) service_pool.cc:224] Handling call
0709 12:27:38.665330 (+1317011us) spinlock_profiling.cc:243] Waited 468 ms on lock 0x7b54006383d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:38.665479 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1075854592}
I20260709 12:27:38.593612 12847 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:38.660473 12848 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:38.628767 12754 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.061s
I20260709 12:27:38.614353 12846 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:38.664927 12836 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:38.673769 12659 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.679445 12767 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:38.705633 12854 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
W20260709 12:27:38.709394 12851 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:27:38.711764  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2825) took 1088 ms. Trace:
I20260709 12:27:38.712268  5709 rpcz_store.cc:276] 0709 12:27:37.623053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:37.931653 (+308600us) service_pool.cc:224] Handling call
0709 12:27:38.711543 (+779890us) spinlock_profiling.cc:243] Waited 396 ms on lock 0x7b54002f5154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:38.711693 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":910566784}
I20260709 12:27:38.716140 12834 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:38.718639 12595 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:27:38.696341 12836 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.719113 12595 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:38.719720  5702 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:27:38.694764 12848 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:38.722600  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 1.14168e+06us
I20260709 12:27:38.726661 12768 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:38.727247 12768 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.729010 12817 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.729038  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2963) took 1127 ms. Trace:
I20260709 12:27:38.729841 12817 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.729755  5631 rpcz_store.cc:276] 0709 12:27:37.601592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:37.601712 (+   120us) service_pool.cc:224] Handling call
0709 12:27:37.602045 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:37.602258 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:27:37.602552 (+   294us) log.cc:844] Serialized 14 byte log entry
0709 12:27:38.728650 (+1126098us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:38.728716 (+    66us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:38.728775 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:38.728845 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:38.728977 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1125819,"prepare.queue_time_us":1007,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":1126021,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":100}
W20260709 12:27:38.734982  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:38.741997  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:38.743297  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:38.753916 12822 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:38.754820 12822 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.756630 12821 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:38.757249 12821 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.757992 12768 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:38.758642 12862 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:38.759538 12821 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:38.760572  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2958) took 1167 ms. Trace:
I20260709 12:27:38.760936  5628 rpcz_store.cc:276] 0709 12:27:37.592871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:37.594376 (+  1505us) service_pool.cc:224] Handling call
0709 12:27:37.595768 (+  1392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:37.595987 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:27:37.596241 (+   254us) log.cc:844] Serialized 14 byte log entry
0709 12:27:38.760188 (+1163947us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:38.760255 (+    67us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:38.760309 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:38.760395 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:38.760515 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":924018,"prepare.queue_time_us":678,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"spinlock_wait_cycles":283904,"thread_start_us":239401,"threads_started":1,"wal-append.queue_time_us":318093}
W20260709 12:27:38.763945 12858 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:27:38.764261 12834 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:27:38.764849 12834 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:38.681114 12847 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.765609 12760 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.765779 12834 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.767923 12848 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:38.768018  5706 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 12:27:38.696102 12659 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.769013  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:38.770298  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:38.770409 12548 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:38.770726 12659 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:38.771004 12548 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.771318 12673 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:38.771600 12847 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:38.772050 12673 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.772365 12689 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:38.773270 12548 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:38.773851 12548 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.774206 12673 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:38.774619 12752 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:27:38.774678 12815 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:38.769129 12742 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.773350 12689 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.776662  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:38.776950  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:38.777395  5702 raft_consensus.cc:2368] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18.
I20260709 12:27:38.759178 12862 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.778208  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:38.778784  5632 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:38.780016  5709 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:38.781169  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:38.781744  5702 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:38.775486 12752 consensus_queue.cc:1243] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 20.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:27:38.782765 12181 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:38.782582 12742 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.775610 12548 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.785022 12742 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:38.785140 12548 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:38.785382 12537 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.550s	user 0.009s	sys 0.000s
I20260709 12:27:38.785804 12537 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.782598 12848 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:38.765785 12767 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.790151 12862 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:38.751993 12836 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:38.797112  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:38.797143  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:38.776324  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:38.788060 12537 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:38.798874  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:38.807013  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:38.803838  5632 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:38.807638  5554 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:38.732093 12817 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:38.809765  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:38.810642  5710 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:27:38.805416  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:38.806686 12836 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:38.803208 12848 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:38.683314 12437 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.720103 12595 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.768174 12760 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:38.817019  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:38.824436  5636 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:38.835564  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:38.818001  5706 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 21. Current term is 22. Ops: [18.11-21.12]
I20260709 12:27:38.774693 12673 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.815534 12846 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.823812  5702 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:38.817086  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:38.854802  5632 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 14.
I20260709 12:27:38.809055 12815 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.845867  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:38.816102 12373 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:38.821589 12849 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:27:38.861478  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:38.816118 12776 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:38.861918  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:38.874524  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:38.875038  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:38.838223 12821 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:38.818348 12811 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:38.826231 12437 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:38.832170 12836 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:38.884965  5709 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:27:38.886847  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:38.884438  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:38.891083  5599 leader_election.cc:400] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18.
I20260709 12:27:38.818787 12181 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.060s
I20260709 12:27:38.857688 12537 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:38.892418  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:38.827989 12376 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:27:38.833151 12286 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 12:27:38.891486  5599 leader_election.cc:403] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:27:38.891510  5633 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:38.896956  5632 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:27:38.888132 12804 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.600s	user 0.004s	sys 0.009s
I20260709 12:27:38.871287 12373 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:38.904373  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:38.906176 12822 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:38.906775 12822 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:27:38.907145 12822 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:38.907480 12822 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:38.880812 12849 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:38.908692 12849 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:38.910152 12602 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:27:38.911275 12753 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:38.911631 12753 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:38.841166 12673 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.913715  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:38.862675 12535 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
I20260709 12:27:38.933971  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:38.891535 12811 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:38.940140  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:38.939100  5634 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:38.940858  5636 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:27:38.891585 12821 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:38.893231 12776 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:38.931948  5632 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:38.892880 12784 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:38.924208  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:38.943361  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:38.943892  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:38.862893 12815 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:38.938084 12848 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:38.968883  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:38.952759  5561 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:38.949899  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:38.953694  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:38.954548  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:38.895489 12548 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260709 12:27:38.960178 12689 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:38.962293 12846 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:38.962314 12659 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:39.020669 12659 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.023615 12659 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:39.027891  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:39.028602  5554 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:27:39.029479  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:39.030496  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:39.031489  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:39.032044  5554 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:39.033440 12848 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.036655 12548 consensus_queue.cc:1243] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 20.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:27:39.037233 12548 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:39.037552 12548 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:39.038412 12548 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.039659  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:39.042362 12717 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:39.042900 12717 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:39.043779 12717 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:39.044782  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:38.944550 12778 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18.
I20260709 12:27:39.050545 12373 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.051944 12373 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:39.052850  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:38.946638 12821 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:38.964053 12811 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.065099 12811 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:38.965342  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1349) took 1275 ms. Trace:
I20260709 12:27:39.068573  5557 rpcz_store.cc:276] 0709 12:27:37.690173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:37.788264 (+ 98091us) service_pool.cc:224] Handling call
0709 12:27:37.812878 (+ 24614us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:37.841810 (+ 28932us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:37.841838 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:37.841880 (+    42us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:37.841911 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:38.965117 (+1123206us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x7b54001c57d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:38.965160 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:38.965286 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":66097408}
I20260709 12:27:39.070854  5557 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:27:38.965391 12681 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.071949 12681 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.074704 12681 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:38.966230  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:38.968897  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.075874  5633 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 18: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:27:39.075979  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 109859us
I20260709 12:27:39.076865  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.077554  5633 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:39.078008 12767 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:39.078598 12767 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:38.952930  5552 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:38.970690  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:39.079524 12830 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:38.956323  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3048) took 1695 ms. Trace:
I20260709 12:27:39.080320  5701 rpcz_store.cc:276] 0709 12:27:37.261416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:37.261578 (+   162us) service_pool.cc:224] Handling call
0709 12:27:38.584513 (+1322935us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:38.585544 (+  1031us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:38.956006 (+370462us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54002d23d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:38.956252 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3041534848}
I20260709 12:27:38.982446  5632 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:38.973428  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:39.081717  5555 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 20: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:38.937794 12817 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:39.082655  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:38.946161 12727 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:39.039557 12548 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:39.056077  5631 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:39.056771 12836 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.913133 12753 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.058518 12776 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:38.943171 12752 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.114s
I20260709 12:27:39.055893  5636 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
W20260709 12:27:39.019193  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 209641us
I20260709 12:27:39.078545  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.079873  5561 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:38.972077  5560 raft_consensus.cc:2376] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17. Re-sending same reply.
W20260709 12:27:38.972589  5602 connection.cc:256] RPC call timeout handler was delayed by -1.5701182960001461s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1879s for a response
W20260709 12:27:39.124487  5602 connection.cc:256] RPC call timeout handler was delayed by -1.5440182960001039s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1908s for a response
I20260709 12:27:39.083642  5553 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:27:39.103855  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.059842 12822 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.112980  5555 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 12:27:39.124879  5602 connection.cc:256] RPC call timeout handler was delayed by -1.4855182960000093s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19730000000000003s for a response
W20260709 12:27:39.127300  5602 connection.cc:256] RPC call timeout handler was delayed by -1.4450182959999438s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20179999999999998s for a response
I20260709 12:27:39.129294  5632 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:39.130426  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.131487  5632 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:39.117057 12681 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:39.114143 12817 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:39.151016  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.120967 12778 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.152722  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:39.133708 12753 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:39.157927  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:39.165283  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:39.166505  5561 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:39.114398 12727 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.169912  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.079595 12767 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.171897  5630 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 25: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:39.174813  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:39.175486  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47739.4us
I20260709 12:27:39.153244 12681 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:39.189977  5602 connection.cc:256] RPC call timeout handler was delayed by -1.3280182960002094s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21480000000000002s for a response
I20260709 12:27:39.189898  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:39.201457  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:39.232010  5602 connection.cc:256] RPC call timeout handler was delayed by -1.1561005680000562s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.166s for a response
W20260709 12:27:39.232338  5602 connection.cc:256] RPC call timeout handler was delayed by -1.055300568000348s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17720000000000002s for a response
W20260709 12:27:39.278187  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:39.296618  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:39.166400 12778 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.190636 12877 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.190104 12681 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:39.196265 12837 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:39.360759 12673 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:39.190493 12879 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:39.196839 12402 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:39.196288 12727 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:39.173799 12817 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.373255  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.373533 12822 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:39.385248  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 152203us
I20260709 12:27:39.335530 12752 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:27:39.402030  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:39.428103  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:39.435204  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:39.377199 12877 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.396747 12849 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.394572 12837 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.426630 12846 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.399797 12879 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:39.441730 12848 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:39.446524 12752 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:39.459690  5709 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:27:39.456511 12846 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.463258 12846 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:39.476019 12361 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:39.476744 12361 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.487160 12714 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:39.489280  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:39.509829  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2968) took 1906 ms (client timeout 2413 ms). Trace:
W20260709 12:27:39.510298  5629 rpcz_store.cc:269] 0709 12:27:37.602955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:37.669367 (+ 66412us) service_pool.cc:224] Handling call
0709 12:27:39.509612 (+1840245us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x7b54001f2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:39.509766 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1192780160}
I20260709 12:27:39.510767  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:39.516790  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.517439  5629 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:39.496789 12846 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:39.520773  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:39.499076 12880 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:39.502552 12812 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:39.516490 12822 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:39.526466  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.511622 12886 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.465988 12848 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.518108 12837 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:39.553560  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 876478us
W20260709 12:27:39.558878  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:39.564538 12881 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.565171 12881 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.576826  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:39.526602 12846 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.540982 12778 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:39.579171 12822 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:39.583043 12361 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:39.587091 12402 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.588858 12879 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:39.539265 12886 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:39.593124 12850 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.997s	user 0.004s	sys 0.000s
I20260709 12:27:39.531317 12880 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.604714  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:39.610937  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:39.581579 12881 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:39.624754  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:39.620934 12888 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 12:27:39.614457 12849 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:39.629778  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:39.619901 12812 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:39.602136 12402 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.618288 12877 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:39.632642  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:39.631074  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:39.619015 12778 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:39.637801  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:39.633232  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:39.638022  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:39.644434  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:39.638706  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:39.644817  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:39.639810  5702 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:39.642081  5706 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:39.645599  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:39.642376  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:39.627918 12373 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.576s	user 0.014s	sys 0.000s
I20260709 12:27:39.646436  6280 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:39.648778  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:39.630892 12822 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:39.650529  5702 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:39.639886 12767 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.655165  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:39.658049  5633 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:39.658386  5702 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:39.656567  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:39.659578 12893 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.659709  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:39.660199 12893 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.661634  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:39.662997  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:39.663633  5631 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:39.664158 12887 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20260709 12:27:39.666535 12893 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:39.672307  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:39.654343 12812 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:39.678511 12741 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:27:39.649137 12811 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.583s	user 0.007s	sys 0.006s
I20260709 12:27:39.683475  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:39.683908  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:39.684166  5707 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:39.684492  5705 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:27:39.686154 12815 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:39.689827  5707 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:27:39.692003  5702 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
W20260709 12:27:39.680254  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47113.9us
I20260709 12:27:39.697692  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:39.691510 12837 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.660941 12791 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:39.679818  5631 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:39.642673 12681 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.701542  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:39.702171  6271 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:27:39.703356 12681 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:39.679024 12741 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:39.704612 12741 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.700152  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.696928 12880 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:39.699136 12791 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:39.700663 12822 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:27:39.701285 12811 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:27:39.694087 12848 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:39.687206 12402 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:39.706037 12821 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.595s	user 0.012s	sys 0.000s
W20260709 12:27:39.697211 12548 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.582s	user 0.013s	sys 0.000s
I20260709 12:27:39.700393  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:39.681898 12767 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.720014  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:39.721061  5627 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 20 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:27:39.721988 12812 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:39.721782 12821 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.723482 12812 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:39.724629 12548 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:27:39.724496 12812 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.725536  6279 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:39.725956 12812 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:39.733618  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:39.726130 12733 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:39.708657 12837 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.743266 12837 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:39.744100 12846 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:39.744879  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:39.745514  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:39.746261  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:39.748171 12548 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:27:39.748814  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:39.749627  5706 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:39.750140  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:39.732937  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:39.752485  5703 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 18: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:39.753824 12822 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:39.754364 12822 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:39.754843 12822 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:39.756531 12681 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:27:39.756579 12895 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:39.757196 12681 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:39.757586 12895 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:39.758247 12895 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:39.757768 12791 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.759028 12791 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:39.758889 12681 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.726136 12767 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:39.718851 12886 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:39.761776  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:39.764765  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:39.761783  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:39.766108  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:39.766717  5710 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:39.767238  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:39.770056  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:39.771852  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:39.772933  5705 raft_consensus.cc:2410] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:27:39.775624  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:39.770705 12675 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:39.776830  5558 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:39.777172  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:39.778291  5709 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:39.778612  5559 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: []
W20260709 12:27:39.781045 12899 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 12:27:39.785878  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:39.788769  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:39.769469 12548 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:39.789768  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:39.791605 12894 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:27:39.792374 12895 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.793248 12892 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:39.793718 12892 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:39.794902 12892 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.796191 12892 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:39.796576 12595 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:39.798448 12832 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:39.799099 12832 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.802713 12798 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:39.788944  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:39.804046  6280 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:39.789232 12741 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:39.769966 12753 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.604s	user 0.010s	sys 0.003s
I20260709 12:27:39.809180  5630 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 22 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:27:39.810011  5631 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:39.811342 12821 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:39.804220  5634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2961) took 2210 ms (client timeout 2128 ms). Trace:
I20260709 12:27:39.813510  5636 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:39.815315  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:27:39.813614  5634 rpcz_store.cc:269] 0709 12:27:37.593843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:37.593983 (+   140us) service_pool.cc:224] Handling call
0709 12:27:39.803988 (+2210005us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400241654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:39.804120 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3020579072}
I20260709 12:27:39.776259  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:39.817786  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.818580  5634 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:27:39.815568  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:39.821184  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1397) took 1084 ms. Trace:
I20260709 12:27:39.821576  5628 rpcz_store.cc:276] 0709 12:27:38.736343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:38.778411 (+ 42068us) service_pool.cc:224] Handling call
0709 12:27:38.778686 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:38.826706 (+ 48020us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:38.826735 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:38.826774 (+    39us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:38.826803 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:39.820961 (+994158us) spinlock_profiling.cc:243] Waited 47.8 ms on lock 0x7b540043a154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:39.821011 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:39.821129 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":110040448}
I20260709 12:27:39.823015  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.823593  5628 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:39.826124  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:39.830644  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:39.831754  5558 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:39.833714  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:39.833978  5701 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:27:39.835620  5710 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:27:39.835934  5561 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:39.836476  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:39.838562  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.841881  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.842916  5628 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 20 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:27:39.843864  5628 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "52cee1d09af749489de4689a67047497"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:39.844657  5628 raft_consensus.cc:606] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:27:39.845080  5628 raft_consensus.cc:614] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:27:39.846766 12895 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:39.835119 12890 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:39.854817  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:39.857925  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:39.858973  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 82805.1us
I20260709 12:27:39.841183  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:39.793277 12752 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:39.912245  5632 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:27:39.862854  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:39.904556  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 89303.5us
I20260709 12:27:39.918031  5559 raft_consensus.cc:2376] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19. Re-sending same reply.
I20260709 12:27:39.835620 12815 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:39.916646  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2730) took 1012 ms. Trace:
I20260709 12:27:39.922376  5632 rpcz_store.cc:276] 0709 12:27:38.904527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:39.911308 (+1006781us) service_pool.cc:224] Handling call
0709 12:27:39.911755 (+   447us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:39.916196 (+  4441us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:39.916557 (+   361us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:27:39.918303  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 77195.5us
I20260709 12:27:39.924185  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:39.835426 12879 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.928889  5629 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:39.929251 12879 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.931546 12879 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:39.898877 12890 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:39.932956  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:39.933177 12890 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.933851  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:39.935319 12872 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:39.935946 12872 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:39.863252  5630 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:39.862300  6280 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:27:39.920907  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:39.922386 12877 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:39.943082 12675 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:39.943656 12675 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:39.946221  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2717) took 1121 ms. Trace:
W20260709 12:27:39.947865  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 90027.2us
I20260709 12:27:39.928709  5632 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:27:39.929440 12822 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:39.935384 12437 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:39.953644  5632 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
W20260709 12:27:39.864743  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:39.897383  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2721) took 1071 ms. Trace:
I20260709 12:27:39.942018 12673 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:27:39.926438 12832 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:40.178907  5706 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:27:39.941269  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:39.878078 12895 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:39.863348 12256 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:40.172345 12464 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:40.181607 12895 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:40.182346 12895 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:39.862668 12840 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:39.854079 12846 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.998296  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:40.007717  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55687us
W20260709 12:27:40.151494  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:40.156949  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:39.982580  5633 rpcz_store.cc:276] 0709 12:27:38.826151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:39.305739 (+479588us) service_pool.cc:224] Handling call
0709 12:27:39.897207 (+591468us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b540008bbd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:39.897314 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":609992960}
W20260709 12:27:40.193040  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 194835us
I20260709 12:27:40.194094  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:40.195390 12529 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:40.195648  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:40.174546  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:40.409765  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:40.176654  5632 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:27:40.178005  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:40.178196  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:39.935969 12848 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:40.412885  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:40.413318 12848 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:40.413502  5632 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:40.180518 12675 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:40.413770 12848 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:40.171777 12835 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:39.936534 12872 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:40.181564 12464 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:40.416157 12256 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.416940 12256 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:40.418555 12464 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:40.185931 12840 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:40.419255 12840 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:39.946805  5630 rpcz_store.cc:276] 0709 12:27:38.824997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:39.862799 (+1037802us) service_pool.cc:224] Handling call
0709 12:27:39.863082 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:39.940958 (+ 77876us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:39.946124 (+  5166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:39.953310 12437 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:40.197067 12742 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:40.174286 12376 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:40.214854  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:40.200279  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:40.190315  5634 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:40.232982  5702 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:27:40.234436 12904 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:27:40.236088  5559 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 12:27:40.172084 12830 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.260705 12673 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:40.236855  5557 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:27:40.246949  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:39.949499 12877 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:40.264655 12548 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:27:40.265421  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:40.267741  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:40.271121  5708 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 17. Current term is 18. Ops: []
W20260709 12:27:40.271785  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 73194.4us
W20260709 12:27:40.271872  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:40.178925  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:40.445542  5701 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:40.272801  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:40.273317  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 332081us
I20260709 12:27:40.273975  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:40.274482  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:40.275126  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:40.262643  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2725) took 1416 ms. Trace:
I20260709 12:27:40.447182  5627 rpcz_store.cc:276] 0709 12:27:38.846736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:39.427778 (+581042us) service_pool.cc:224] Handling call
0709 12:27:40.262457 (+834679us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b540008c0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:40.262593 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":656795008}
I20260709 12:27:40.285885  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:40.287878  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:40.226102 12681 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.199220 12849 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.191236 12822 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:40.196225 12529 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:40.293067  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:40.282253 12373 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:27:40.279412 12836 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:40.300490 12847 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.466434  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.285991 12733 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
I20260709 12:27:40.280790 12846 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:40.300490 12834 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 12:27:40.348265  6279 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8c6474db7a70405694509f6976d850ce"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:27:40.476668  6279 raft_consensus.cc:606] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:40.349416  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:40.467185  5633 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:40.350306  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:40.448258 12657 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.013s
I20260709 12:27:40.352322  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:40.355263 12595 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.558s	user 0.011s	sys 0.000s
W20260709 12:27:40.356985 12812 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.624s	user 0.009s	sys 0.000s
I20260709 12:27:40.358497  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:40.357331  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:40.364915  6276 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 12:27:40.368328  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 72964.3us
W20260709 12:27:40.381201 12714 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.878s	user 0.008s	sys 0.006s
I20260709 12:27:40.405364  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:40.408906 12741 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.600s	user 0.011s	sys 0.000s
I20260709 12:27:40.419317 12256 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:39.953645 12892 raft_consensus.cc:2764] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
W20260709 12:27:40.420567  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 224972us
I20260709 12:27:40.501085  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:40.501600  5556 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 12:27:40.421213 12811 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:40.421146  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.502487  5630 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:40.428565 12437 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:40.416047 12835 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:40.436007  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 235796us
W20260709 12:27:40.444028  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 178677us
I20260709 12:27:40.450362  5707 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:27:40.450538  5702 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:27:40.450989  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:40.451145  5709 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:40.431790 12832 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:40.452719  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.436496 12376 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:40.441237 12862 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:40.459312  5632 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:27:40.528838  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:40.529675  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.530370  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:40.530965  5632 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 18: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:40.462273  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:40.532480  5559 raft_consensus.cc:2410] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:27:40.448477 12675 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:40.477339  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.478516  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:40.351497  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:40.535537  5628 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:40.480979  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:40.453917 12602 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.445678 12830 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:40.493342  5558 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:40.541534  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:40.540282  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:40.479326 12847 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:40.497658  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.420418 12773 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:40.504292  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.476697 12373 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:40.457582 12849 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:40.455225 12681 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:40.517534  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:40.521426  5627 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:40.583755  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.478938  5560 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 17 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:27:40.584861  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:40.585527  5559 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:40.513535 12714 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:27:40.503805 12673 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.476457 12836 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:40.539156  5554 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 20: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:40.502909 12811 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:40.546855  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:40.535079 12675 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:40.542371  5635 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:27:40.554822 12773 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:40.554168  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:40.604409  5558 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:40.605042  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:40.497483  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 43895.8us
I20260709 12:27:40.525717 12760 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:40.556859  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.609906  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.555498  6280 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:40.560708  6274 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 12:27:40.558343  5630 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:27:40.499545 12421 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:27:40.596036 12846 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:40.540416  5601 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:27:40.574152  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 132970us
W20260709 12:27:40.627390  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 86353.3us
W20260709 12:27:40.576663  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46285.4us
I20260709 12:27:40.628012  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.577855 12920 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:27:40.580699  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:40.537791  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:40.639436  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:40.640386  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:40.641369  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:40.588371  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.589097 12376 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:40.539069 12595 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.050s
I20260709 12:27:40.552762 12912 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:40.534806 12877 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:40.540910 12529 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:40.595355  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.654966  5633 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
I20260709 12:27:40.655441  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:40.596221  5557 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:27:40.597838  5706 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
W20260709 12:27:40.552261  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 86901.9us
W20260709 12:27:40.598973  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 94953.9us
I20260709 12:27:40.656869  5706 pending_rounds.cc:85] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 13
I20260709 12:27:40.657271  5706 pending_rounds.cc:107] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260709 12:27:40.657502  5706 raft_consensus.cc:2889] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:40.658640  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:40.525250 12832 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:40.602758  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:40.660187  5559 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:40.660296  6273 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:40.613752  5554 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:40.546700 12812 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 12:27:40.555465 12753 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:27:40.555871 12895 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:40.670077 12895 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:40.541983 12602 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:40.519693 12880 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.670886 12880 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:40.658677 12811 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:40.574862  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:40.628826  5630 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:40.672821 12815 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:40.673326 12815 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:40.630364  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.673939 12836 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:40.601692 12675 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:40.602334 12752 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.587234 12834 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:27:40.642484  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:40.599718 12762 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.606195 12773 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:40.645576  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:40.687546  5633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:40.688251  5633 raft_consensus.cc:606] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:27:40.688616  5633 raft_consensus.cc:614] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:27:40.646121  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 105796us
I20260709 12:27:40.534476 12835 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:40.660295  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 113502us
I20260709 12:27:40.601256 12673 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:40.665158  6275 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: [15.13-17.14]
I20260709 12:27:40.666186 12832 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:40.675709  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:40.689391  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:40.709400  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:40.644708 12916 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.060s
I20260709 12:27:40.651646 12548 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.047s
I20260709 12:27:40.731158  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:40.666206 12847 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:40.732298  5554 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:27:40.735888  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:40.744395  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:40.748124  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 89578.9us
I20260709 12:27:40.749194  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:40.675721 12872 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:40.652635 12373 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:40.750298  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 121442us
I20260709 12:27:40.730837 12917 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:40.763223  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:40.698150 12762 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:40.762378 12681 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:40.698333 12752 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:40.769381  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 140683us
I20260709 12:27:40.734475 12919 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:40.768563  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:40.773856  5553 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:40.763199 12846 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:40.758852  5559 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:40.777343  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:40.783831  5675 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:40.782372  5559 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:27:40.787568  5704 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: [17.10-17.10]
W20260709 12:27:40.788089  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 85075us
I20260709 12:27:40.789912  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:40.792440  5633 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:40.763259 12917 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:40.796321  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:40.783128 12836 raft_consensus.cc:2764] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:27:40.800652 12714 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:40.801957 12714 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:40.793473  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:40.805307  5627 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:40.815455 12840 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:40.816457 12913 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:40.794502 12834 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:40.808109 12830 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:40.830873  5633 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 10. (term mismatch)
I20260709 12:27:40.840400  5633 pending_rounds.cc:85] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 9
I20260709 12:27:40.840720  5633 pending_rounds.cc:107] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 19.10
I20260709 12:27:40.840968  5633 raft_consensus.cc:2889] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:40.841439  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:27:40.842136  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.842741  5633 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:40.843699  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.844965  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 1420) took 1162 ms. Trace:
I20260709 12:27:40.845352  5631 rpcz_store.cc:276] 0709 12:27:39.682722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:39.699923 (+ 17201us) service_pool.cc:224] Handling call
0709 12:27:40.844801 (+1144878us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54001cedd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:40.844912 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":237843328}
I20260709 12:27:40.846575  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.847162  5631 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:40.797777 12923 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:40.806740 12922 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:40.817101 12913 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:40.848115  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.871527  5631 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:27:40.831645  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:40.861320  6279 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:40.844225  5633 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:40.779997 12924 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:40.865587  5558 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:27:40.805681 12921 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:40.894405  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:40.820322 12912 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:40.797279 12919 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:40.822252 12846 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:40.870318  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:40.873986 12927 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:40.875229  5635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2726) took 2018 ms (client timeout 2077 ms). Trace:
I20260709 12:27:40.876657  5628 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:27:40.836344 12834 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:40.852701 12872 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:40.893054  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:40.884063  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:40.865708 12923 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:40.867554 12922 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:40.870441 12602 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:40.912904  5703 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:27:40.912575 12919 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:27:40.919108  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 87592.2us
I20260709 12:27:40.886925 12928 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.885303 12832 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:27:40.914369  5635 rpcz_store.cc:269] 0709 12:27:38.857166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:39.140047 (+282881us) service_pool.cc:224] Handling call
0709 12:27:40.875042 (+1734995us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540032d7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:40.875165 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3069804288}
I20260709 12:27:40.889082 12835 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:40.883255 12931 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:40.974858  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:40.968642  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:40.959523 12548 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:40.943411 12846 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:40.980816  5631 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
W20260709 12:27:40.981261  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 125699us
I20260709 12:27:40.962203 12923 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:40.964313 12922 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:40.966660 12930 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:40.965476 12834 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:40.985280  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:40.895474 12675 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:40.883329 12917 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:40.979408 12760 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:41.022039  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 226206us
I20260709 12:27:41.027886  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:41.041961  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 56766.2us
I20260709 12:27:40.970779 12437 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.042407  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:40.943943 12927 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.047742  5635 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 13. (term mismatch)
I20260709 12:27:41.051831  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:41.012828 12849 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.053627  5635 pending_rounds.cc:85] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 12
I20260709 12:27:41.054792  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:41.059506  5635 pending_rounds.cc:107] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260709 12:27:41.059860  5635 raft_consensus.cc:2889] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:27:41.052130  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 414463us
I20260709 12:27:41.053673  5630 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:41.060670  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:41.070628  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:41.072237  6276 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:41.075631  5558 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 12:27:41.015411 12926 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:40.978938 12928 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.047079 12840 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:41.079402  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 330319us
I20260709 12:27:41.082103  5702 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:27:41.050984 12548 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:27:41.083391  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:41.078351  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:41.053352 12835 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:41.055670 12832 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:41.046617 12919 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:41.047912 12935 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:27:41.096675 12898 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.380s	user 0.013s	sys 0.035s
W20260709 12:27:41.097035 12933 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:27:41.021035 12925 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.053s
W20260709 12:27:41.106362  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:41.074826 12849 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:40.985762 12913 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:41.022585 12930 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.117395  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:41.140084 12675 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:41.132685  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62541.7us
I20260709 12:27:41.140914  5635 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:41.150115  5635 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:27:41.147178  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:41.113309 12835 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:27:41.161470  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:41.166239 12464 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:27:41.167735  5635 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:27:41.168049 12464 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:41.168782 12464 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:41.169203  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.154786 12940 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:41.169883 12940 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:41.170264 12940 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:41.170791  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.161518 12832 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 12:27:41.171952  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.172727  5557 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:27:41.174378 12930 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.174525  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.175033  6272 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:27:41.155062 12529 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:41.176198  5557 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:41.176251 12529 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:41.178575 12928 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.179005 12817 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:27:41.184139 12872 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.184725 12872 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:41.186827 12872 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:41.186913 12921 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.187353 12872 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.187400 12921 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:41.188231 12947 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:41.188500 12912 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.189087 12912 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:41.190382 12811 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:41.191280 12912 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:41.191570  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2972) took 2066 ms (client timeout 2198 ms). Trace:
W20260709 12:27:41.191975  5629 rpcz_store.cc:269] 0709 12:27:39.125293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:39.522266 (+396973us) service_pool.cc:224] Handling call
0709 12:27:41.191406 (+1669140us) spinlock_profiling.cc:243] Waited 402 ms on lock 0x7b5400134554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:41.191513 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":923749120}
I20260709 12:27:41.193975  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:41.194777 12437 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.195171  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:41.195933  5553 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:41.141338 12938 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.075s
I20260709 12:27:41.197517 12928 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:41.198067 12928 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:41.200776 12907 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20260709 12:27:41.150782 12846 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.202051  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.203171  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.178674 12529 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:41.161633 12835 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:41.142499 12919 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:41.208261  5553 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:41.152594 12925 consensus_queue.cc:1243] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 19.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:27:41.214699  5553 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20260709 12:27:41.148341 12849 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:41.187906 12872 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.180199 12817 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:41.250820 12817 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:41.251788 12817 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:41.217489 12848 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.803s	user 0.009s	sys 0.005s
I20260709 12:27:41.222455  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.253824 12848 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:41.222004  5553 pending_rounds.cc:85] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 13
W20260709 12:27:41.226110 12944 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:27:41.256546  5552 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:41.210765 12947 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:41.258772  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2955) took 1044 ms. Trace:
I20260709 12:27:41.258539 12947 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.259235  5552 rpcz_store.cc:276] 0709 12:27:40.231147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:40.231298 (+   151us) service_pool.cc:224] Handling call
0709 12:27:40.231611 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:41.257012 (+1025401us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:41.258530 (+  1518us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54003c41d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:41.258715 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2356709632}
I20260709 12:27:41.216836 12941 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:41.235616  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.238179  5705 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260709 12:27:41.240163 12927 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.241140  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.163120 12675 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:41.127796 12760 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.200299 12928 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.250267  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:41.253180 12817 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:41.226050 12849 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:41.226614 12922 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:41.292814  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.293699  5559 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:41.259986  5553 pending_rounds.cc:107] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:27:41.262545  5703 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: []
W20260709 12:27:41.294060  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 233498us
I20260709 12:27:41.294338  5553 raft_consensus.cc:2889] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:41.267484  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:41.267902  5705 pending_rounds.cc:85] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 13
I20260709 12:27:41.295548  5705 pending_rounds.cc:107] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260709 12:27:41.295585  5555 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:41.295977  5705 raft_consensus.cc:2889] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:41.268683  5629 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:27:41.264770  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.297896  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:27:41.298336 12890 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.116s	user 0.018s	sys 0.000s
W20260709 12:27:41.299139 12890 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.117s	user 0.018s	sys 0.000s
I20260709 12:27:41.299733  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:41.301090  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2728) took 2413 ms. Trace:
I20260709 12:27:41.261435 12947 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.305078  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:41.301642  5636 rpcz_store.cc:276] 0709 12:27:38.888210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:39.905862 (+1017652us) service_pool.cc:224] Handling call
0709 12:27:39.920011 (+ 14149us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:39.941151 (+ 21140us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:39.941180 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:39.951007 (+  9827us) log.cc:844] Serialized 52 byte log entry
0709 12:27:40.335814 (+384807us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:40.978512 (+642698us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b18002753e0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:27:41.120755 (+142243us) log.cc:844] Serialized 14 byte log entry
0709 12:27:41.291251 (+170496us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:41.291371 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:41.291441 (+    70us) raft_consensus.cc:1685] finished
0709 12:27:41.300845 (+  9404us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:41.301029 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20855,"prepare.queue_time_us":56536,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":65580,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":31401,"spinlock_wait_cycles":7417088,"thread_start_us":450377,"threads_started":3,"wal-append.queue_time_us":196888}
I20260709 12:27:41.311108  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:41.306458  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:41.307585  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 229333us
I20260709 12:27:41.315493  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.313038  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:41.276310 12753 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.318260  5703 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
W20260709 12:27:41.301357 12815 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.628s	user 0.005s	sys 0.009s
I20260709 12:27:41.273926 12675 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:41.325178  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:41.293759 12927 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.317052 12954 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:41.301998 12926 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.337844  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:41.339493  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:41.340997  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:41.311578 12928 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:41.309450 12931 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:27:41.351802  5555 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:41.311416 12848 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:41.273238 12941 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.358731  5559 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: [16.9-19.11]
I20260709 12:27:41.364398  5636 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:27:41.359127  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.364913  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:41.336203 12935 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:41.335453 12675 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:41.367599  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.342609 12947 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:41.371058  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.337200 12954 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:41.375046  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 108916us
I20260709 12:27:41.357297 12952 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:41.410356  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:41.357338 12911 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:41.334280 12753 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.420194 12911 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:41.420805 12911 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:41.422701 12753 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.337327 12815 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.425809 12815 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.351004 12799 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 12:27:41.426021 12952 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:41.429731 12952 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:41.431602  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.433065  5705 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:41.431058 12922 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:41.434782  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:41.436043 12815 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:41.436731 12815 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:41.437186 12815 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:41.442276 12753 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:41.443260 12753 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:41.447480  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.354645 12832 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 12:27:41.342021 12927 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:41.374035 12947 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:41.466261  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 124962us
I20260709 12:27:41.342234 12926 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:41.366771 12529 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:41.374843 12548 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:27:41.468299  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 114320us
I20260709 12:27:41.376382 12954 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:41.412431 12931 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:27:41.457772 12773 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.742s	user 0.011s	sys 0.004s
I20260709 12:27:41.377914 12955 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:41.460543 12753 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:27:41.470331 12947 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:41.462761 12890 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.378105 12928 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.098s
I20260709 12:27:41.478921 12373 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:41.485807 12931 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:41.506453 12956 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
W20260709 12:27:41.507230 12960 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:27:41.482076 12921 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:41.488732 12955 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:41.515844  5705 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:27:41.494670 12773 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.637138  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:41.599607 12941 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.560070 12940 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.509208 12872 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.504662 12890 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.573863 12922 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:41.495656 12799 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:27:41.543390 12955 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:41.578455 12953 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:27:41.597504  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2966) took 1322 ms. Trace:
I20260709 12:27:41.513350 12373 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.674211 12373 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.482645 12529 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:41.590723 12924 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.696s	user 0.013s	sys 0.000s
I20260709 12:27:41.675962  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:41.598527  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1268) took 1915 ms. Trace:
I20260709 12:27:41.543287 12913 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 12:27:41.683045 12529 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:41.678148  5627 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:41.511207 12931 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:27:41.582973 12834 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.557s	user 0.007s	sys 0.003s
I20260709 12:27:41.623379  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:41.684049  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.609261  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:41.684334 12834 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:41.685266  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 76141us
I20260709 12:27:41.686269 12773 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.686888 12834 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:41.562502  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:41.578671  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:41.580484 12548 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.631688  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:41.634447  5709 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:27:41.653784  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:41.654887  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:41.647933  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:41.545760 12911 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:41.677002  5708 rpcz_store.cc:276] 0709 12:27:39.682709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:39.682927 (+   218us) service_pool.cc:224] Handling call
0709 12:27:40.270942 (+588015us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:40.446644 (+175702us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:41.598289 (+1151645us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x7b5400452ed0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:41.598459 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1351647104}
I20260709 12:27:41.670917  5556 rpcz_store.cc:276] 0709 12:27:40.275530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:40.278915 (+  3385us) service_pool.cc:224] Handling call
0709 12:27:41.597196 (+1318281us) spinlock_profiling.cc:243] Waited 51.7 ms on lock 0x7b54001be254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:41.597415 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":118908672}
I20260709 12:27:41.676364 12924 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:41.684934  5704 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 21: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:41.472457 12923 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.704896  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.631520 12835 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:41.697952  6279 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:41.688119  6274 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
W20260709 12:27:41.674070 12905 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.669s	user 0.013s	sys 0.000s
W20260709 12:27:41.708889  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 220336us
I20260709 12:27:41.710455  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:41.704039  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:41.704160  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.713539  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:41.705909  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.690223  5703 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:41.706460  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.707495  5554 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:41.718304  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.719651  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1406) took 2887 ms. Trace:
I20260709 12:27:41.720175  5634 rpcz_store.cc:276] 0709 12:27:38.832586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:39.863103 (+1030517us) service_pool.cc:224] Handling call
0709 12:27:39.863452 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:40.435735 (+572283us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:41.719341 (+1283606us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b54000c1a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:41.719563 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":751158528}
I20260709 12:27:41.721709  5557 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:41.702205 12904 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:41.700073 12927 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:41.728433  5708 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:41.728334  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:41.723088  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:41.723793  5704 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:41.725781  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:41.736727  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.741092  5556 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:41.722265 12799 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:41.727052 12904 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:41.731948 12753 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.749449  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.748267  5632 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:27:41.753986  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 2977) took 1938 ms. Trace:
I20260709 12:27:41.754340  5632 rpcz_store.cc:276] 0709 12:27:39.815656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:40.641359 (+825703us) service_pool.cc:224] Handling call
0709 12:27:40.641620 (+   261us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:41.748005 (+1106385us) spinlock_profiling.cc:243] Waited 35.9 ms on lock 0x7b5400120fa0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa6741c0 00007f4ffa675231 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c
0709 12:27:41.753612 (+  5607us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:41.753907 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":82525056}
I20260709 12:27:41.757109 12940 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.721799 12941 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 12:27:41.722201 12834 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.029s
W20260709 12:27:41.753270 12811 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.544s	user 0.013s	sys 0.000s
I20260709 12:27:41.767272  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.789260 12890 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:41.790609 12835 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:41.796581 12979 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:41.760718 12919 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.753054 12849 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.798448 12974 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:41.760933 12880 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.793784 12815 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.800631 12941 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:41.800800 12675 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:41.753221 12904 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:41.803007 12911 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:41.815456  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 128375us
I20260709 12:27:41.751967 12799 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:41.821158  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 97159.5us
I20260709 12:27:41.888661 12799 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:41.890836  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:27:41.800830 12965 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260709 12:27:41.824173 12923 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.825635 12752 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:41.826380 12815 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:41.827217 12817 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.532s	user 0.010s	sys 0.000s
I20260709 12:27:41.828856 12675 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:41.820760 12941 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:41.894629 12941 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:41.895277  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:41.895892 12941 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:41.833954 12849 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.897912 12922 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:41.820343 12974 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:41.899699 12974 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.822144 12947 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:41.903239 12799 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.904563 12947 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:41.761272 12753 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.906957 12753 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.908576 12984 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:27:41.910449 12984 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:41.910915 12984 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:41.912222  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:41.913677  5705 raft_consensus.cc:2368] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
I20260709 12:27:41.914393  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:41.915217  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:41.917182  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:41.918556  6276 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:27:41.918632  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:41.841854  5557 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:41.842325  5702 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:41.822397 12952 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.845847 12904 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.923539  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:41.923192  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:41.924678 12952 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:41.924803 12904 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:41.925427 12922 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:41.917276  5553 raft_consensus.cc:2376] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22. Re-sending same reply.
I20260709 12:27:41.926616  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:41.822052 12919 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.932934  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:41.844269  5630 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:41.848165 12955 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:41.858608 12983 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:27:41.925864 12922 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:41.866878  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 138733us
I20260709 12:27:41.869026 12931 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:41.874787 12961 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.320s	user 0.003s	sys 0.000s
I20260709 12:27:41.919800  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:41.950425  5634 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:41.951519  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:41.952546 12923 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:27:41.952697 12836 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:41.954500 12923 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:41.954898 12923 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:41.955268 12923 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:41.926944 12753 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 12:27:41.956569 12940 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:27:41.921830  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:41.922911  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:41.928727  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:41.933360  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:41.934379  5630 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:41.934720  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.960444  5556 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:41.960546  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:41.961493  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.961850  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.962239  5701 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 18: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:41.935119  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:41.962477  5710 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:27:41.963040  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:41.963405  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:41.935400  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:41.965241  5632 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:41.938045 12675 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:41.935735  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:41.972270  5708 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:27:41.936772  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:41.854729 12954 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.937857  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.911724 12984 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.974016  5552 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:41.862072 12762 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:41.935711  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:41.946625  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:41.976146  5558 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:27:41.961347  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:41.970011  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:41.957494 12733 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.013s
W20260709 12:27:41.970957  6271 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1250) took 2176 ms (client timeout 2222 ms). Trace:
W20260709 12:27:41.978047  6271 rpcz_store.cc:269] 0709 12:27:39.781667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:39.835993 (+ 54326us) service_pool.cc:224] Handling call
0709 12:27:41.957999 (+2122006us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54005f5d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:41.958126 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3357978496}
I20260709 12:27:41.938920 12822 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:41.944159  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:41.979481  5636 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260709 12:27:41.976325  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:41.932418 12947 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:41.945746 12922 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:41.945901 12548 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:41.975026  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:41.980659  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:41.981235  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44740.1us
I20260709 12:27:41.994675  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:41.994868  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:41.982358  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:41.977206  5630 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:41.984594  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:41.996795  5629 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:42.001124  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:42.006217  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:42.008404  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:42.011191  5632 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:42.012138  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:42.014818  5556 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:42.017192  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2743) took 1812 ms. Trace:
I20260709 12:27:42.017553  5628 rpcz_store.cc:276] 0709 12:27:40.205177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:40.674549 (+469372us) service_pool.cc:224] Handling call
0709 12:27:40.876031 (+201482us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:40.961318 (+ 85287us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:42.016973 (+1055655us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b540032d7d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:42.017133 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":462376320}
I20260709 12:27:42.010736 12890 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:41.993461 12799 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:42.010885 12979 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.025s
I20260709 12:27:42.021853 12862 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:27:42.022807  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:42.023094 12880 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:41.987610 12832 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:42.016997 12822 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:42.025164 12822 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.027638 12954 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:42.028723  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:41.984087 12980 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:41.987210 12924 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 12:27:42.033808 12955 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:42.037106 12963 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.483s	user 0.003s	sys 0.000s
I20260709 12:27:42.037693  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:42.041738  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:42.043290  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:42.043936  5552 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:42.044196  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 69268.4us
I20260709 12:27:42.043311  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:42.045617  5556 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:42.046373  5634 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:27:42.047452  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:42.047539  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:41.985340 12786 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:41.985674 12947 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:42.048389  5628 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:41.993690 12760 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:41.971583 12836 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:27:42.066030  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:42.076710  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 78129.6us
I20260709 12:27:42.050177 12799 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:42.083020  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:42.064391 12947 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:42.086546  5552 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:42.082240  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:42.097447 12673 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:42.106613  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 59137.5us
I20260709 12:27:42.107309 12830 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:42.107721  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:42.109323  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:42.110069 12376 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:42.110438  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:42.110893 12799 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:42.111152 12954 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:27:42.111639 12799 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:42.112102 12799 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:42.085032 12786 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.022s
I20260709 12:27:42.087095 12976 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:42.117656  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:42.113091 12799 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.122177  5628 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
W20260709 12:27:42.122948  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 184685us
I20260709 12:27:42.123210  5633 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 12:27:42.124876  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:42.125801 12849 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:27:42.112198 12760 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:42.107748 12830 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:42.129246 12971 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:42.119980 12256 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:42.133188  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2852) took 1933 ms. Trace:
I20260709 12:27:42.136585 12954 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:42.137032 12941 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.137986 12919 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:42.138278 12922 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.139247 12931 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:42.141788  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 146973us
I20260709 12:27:42.144791  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:42.145948  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1375) took 1751 ms. Trace:
I20260709 12:27:42.170054  5554 rpcz_store.cc:276] 0709 12:27:40.394374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:40.734217 (+339843us) service_pool.cc:224] Handling call
0709 12:27:40.750891 (+ 16674us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:41.715942 (+965051us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:42.145626 (+429684us) spinlock_profiling.cc:243] Waited 954 ms on lock 0x7b54000ebad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:42.145874 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2193101056}
I20260709 12:27:42.153118  5706 rpcz_store.cc:276] 0709 12:27:40.200192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:40.201648 (+  1456us) service_pool.cc:224] Handling call
0709 12:27:40.202057 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:40.664204 (+462147us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:42.132929 (+1468725us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x7b5400210554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:42.133121 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":909391744}
I20260709 12:27:42.148703 12918 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:42.164278 12973 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:42.179351 12973 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:42.179268 12836 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:42.180459 12256 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:42.181434 12256 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.183238  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 2967) took 1907 ms. Trace:
I20260709 12:27:42.183622  5560 rpcz_store.cc:276] 0709 12:27:40.275792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:40.285538 (+  9746us) service_pool.cc:224] Handling call
0709 12:27:42.182994 (+1897456us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b540016ba54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:42.183178 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":64561536}
I20260709 12:27:42.184600 12974 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:42.185003 12974 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:42.185246 12714 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:42.185477 12974 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:42.189237 12917 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:42.164851 12847 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:42.148480 12830 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:42.140236  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:42.161576  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:42.142524 12602 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:42.171398 12954 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:42.227344  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:42.178771 12926 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:42.217216  5704 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 19: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:42.171725 12919 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:42.221918  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:42.222738 12836 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:42.223528  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:42.226230  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:42.160274 12971 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:42.228111  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 66659.5us
I20260709 12:27:42.230444  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:42.230567  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:42.231546  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 86843.7us
W20260709 12:27:42.220559  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44774.9us
I20260709 12:27:42.235280  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:42.157413 12976 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 12:27:42.148980 12760 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.244251  5701 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:42.244818  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:42.165896 12941 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.252427  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:42.237748  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 141665us
I20260709 12:27:42.245936  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:42.246620  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:42.155704 13000 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.073s
I20260709 12:27:42.205675 12847 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:42.218304 12918 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:42.272876 12918 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.256948  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:42.192775 12923 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.280167  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:42.262346 12714 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:27:42.263072 12971 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.289356  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:42.227140 12957 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:27:42.264160 12786 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:27:42.292464  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:42.293484  6272 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 21: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:42.247368 12836 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:42.267107 13000 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:27:42.233242 12602 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.261689  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:42.298302  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:42.283222  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:42.198442 12973 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.288074  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:42.233393 13001 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:42.304631  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:42.297312  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45040.8us
I20260709 12:27:42.303058  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:42.259764 12926 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:42.256186 12911 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:42.306172  5634 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:42.304309  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:42.292389 12786 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:42.307345  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:42.320036  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:42.228036 12992 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.088s
I20260709 12:27:42.311033  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:42.324313  6273 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:42.312062  5701 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:27:42.309670  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:42.326270  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:42.327620  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:42.318228  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:42.329345  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:42.330049  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:42.330834  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:42.331496  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:42.257649 12939 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:27:42.332571  5634 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 12. (term mismatch)
I20260709 12:27:42.332894  5634 pending_rounds.cc:85] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 11
I20260709 12:27:42.333112  5634 pending_rounds.cc:107] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 20.12
I20260709 12:27:42.333343  5634 raft_consensus.cc:2889] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:42.333490 12256 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.369392  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:42.334419  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:42.371207  5634 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 12:27:42.372485  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 115621us
W20260709 12:27:42.369421  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 107851us
I20260709 12:27:42.373616  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:42.314643 12762 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.376864 12762 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.379091 12762 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:42.379575 13010 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:42.294831 12836 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.380333 13010 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:27:42.306475 12957 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:42.379220 12256 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.382449  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:42.336721  5631 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:27:42.320336 12926 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:42.406514 12926 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:42.322081 12786 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:42.256094 12919 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.394480  5633 raft_consensus.cc:2410] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:27:42.395599  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:42.412398  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:27:42.399016  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:42.336843  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:42.401211  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 151225us
I20260709 12:27:42.404263  5629 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
W20260709 12:27:42.405339  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 83777.4us
I20260709 12:27:42.407966  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:42.322569 12920 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:42.415792  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:42.446930  5559 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:42.371304 12548 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:42.393671 12753 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:42.393085 12675 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:42.322223 12862 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.088s
I20260709 12:27:42.415724  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1284) took 1127 ms. Trace:
I20260709 12:27:42.409865 12811 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:27:42.489665  5707 rpcz_store.cc:276] 0709 12:27:41.288653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:41.288949 (+   296us) service_pool.cc:224] Handling call
0709 12:27:41.289341 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:41.417244 (+127903us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:41.417293 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:41.417353 (+    60us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:41.417627 (+   274us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:42.415495 (+997868us) spinlock_profiling.cc:243] Waited 88.8 ms on lock 0x7b5400075654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:42.415552 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:42.415671 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":204169472}
W20260709 12:27:42.491255  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 77256.3us
I20260709 12:27:42.450861 12931 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:42.514642  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:42.457038 12675 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:42.457082 12834 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:42.545710 12834 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:42.454478 12957 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:42.546484 12957 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:42.546916 12957 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:42.546729 12834 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:42.550444 12993 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.483s	user 0.002s	sys 0.000s
I20260709 12:27:42.552639  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:42.411900 12847 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:42.553328  5558 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:42.554029 12760 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:42.554702 12989 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.481s	user 0.002s	sys 0.000s
I20260709 12:27:42.519755 12811 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:42.557780 12602 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:42.558125 12811 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.558857 12924 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:42.559373 12924 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:42.528527  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:42.561956  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:42.562737  5559 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:42.323220 12954 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.530418 12959 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:42.449117 13001 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.086s
W20260709 12:27:42.539546 12995 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.469s	user 0.004s	sys 0.000s
I20260709 12:27:42.540232 12786 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.541697 12923 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:42.579582 12848 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:42.541879 12830 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.581820 12830 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:42.584621  6274 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:42.542562 12947 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.587592 12947 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:42.507274 13013 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:27:42.544863  6236 connection.cc:256] RPC call timeout handler was delayed by -1.4742731399996956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
I20260709 12:27:42.588340 13013 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:42.588903  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:42.589457 13013 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.590659 13013 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:42.591117 12941 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:42.605926  5559 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:42.511314 12862 consensus_queue.cc:1243] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 17.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:27:42.568715  5627 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:27:42.569301  5552 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 11. (term mismatch)
I20260709 12:27:42.617633  5552 pending_rounds.cc:85] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 10
I20260709 12:27:42.618014  5552 pending_rounds.cc:107] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 21.11
I20260709 12:27:42.618301  5552 raft_consensus.cc:2889] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:42.539690 12974 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.570624  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:42.552539 12675 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:42.531032 12922 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:42.575258  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:42.576244  5629 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
W20260709 12:27:42.578236  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 63697us
I20260709 12:27:42.698379  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:42.699335  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:42.560245 12924 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.651436  6273 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:42.656018  5558 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:42.545651 12529 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:42.453164 12920 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.715706  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:42.750173  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:42.750298  5630 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:42.753961  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:42.700464 12862 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:42.760129  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:42.761410  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 353497us
I20260709 12:27:42.700001 12931 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:42.766294  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:42.766893  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:42.720288 12923 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:42.720295 12760 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:42.792748  5701 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:27:42.720635 12992 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:42.797729  5558 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:42.819021 12959 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:42.819809 12959 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.822651 12959 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:42.826550  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:42.764243 12924 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:42.828003  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 454471us
W20260709 12:27:42.836220  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:42.769272 12773 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:42.849596  5632 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
W20260709 12:27:42.849153  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 403349us
I20260709 12:27:42.770468 12921 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:42.853875  5630 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:27:42.762596 12529 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.866698  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1426) took 2407 ms. Trace:
I20260709 12:27:42.825016 12923 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:42.881973  5630 rpcz_store.cc:276] 0709 12:27:40.458923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:41.183038 (+724115us) service_pool.cc:224] Handling call
0709 12:27:41.183268 (+   230us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:41.844145 (+660877us) spinlock_profiling.cc:243] Waited 38.7 ms on lock 0x7b5400180c20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa6741c0 00007f4ffa675231 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c
0709 12:27:42.749699 (+905554us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b580070fdd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5ce0aa 00007f4ffa650447 00007f4ffa668fe7 00007f4ffa66a7de 00007f4ffa674b49 00007f4ffa675416 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4
0709 12:27:42.865943 (+116244us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:42.866620 (+   677us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1058791040}
I20260709 12:27:42.791280 12931 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.847793 12919 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:42.849687 12256 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:42.855978 12959 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:42.899780  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 148684us
W20260709 12:27:42.883721 12812 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 19 current term: 19 is higher.
I20260709 12:27:42.910660  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:42.921367  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 154566us
I20260709 12:27:42.922438  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:42.928692 12714 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:27:42.928887  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:42.929864 12760 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:42.810860 12992 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.904587 13014 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.942848  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:42.943158  6271 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:42.930315  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:42.946094 12974 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:42.951061 13023 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.003s
I20260709 12:27:42.954685 12949 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:42.954924 13023 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:42.955297 13023 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:42.955253 12949 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.947989 13024 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:42.956892 13024 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:42.957798 12949 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:42.958523 12529 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:42.959115 13024 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:42.959496 13012 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:27:42.931753 12773 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:27:42.960721 13016 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:27:42.961757 12992 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:42.963935 12920 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:42.926446 12919 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:42.959208 13017 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:27:42.985842  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:42.992669  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:42.994671 12931 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:42.907442 12959 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:42.947485 13007 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:43.007580  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:43.010277  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:43.014407  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:43.011183  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:43.005575  6277 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:42.949203 12760 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:43.015480  6277 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:43.016937  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:43.017530  5630 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:43.017886  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:43.039213  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:43.041833 12959 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:43.043073  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:43.023319 13007 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.047246 13007 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:43.048460 12529 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:43.049166 12529 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:43.050199 12529 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.015753 12919 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.020411 13012 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:43.051962  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:43.061740  5635 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 23: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:27:43.065289  6237 leader_election.cc:336] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.783s (SENT)
I20260709 12:27:43.022403 12931 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.009711 12992 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260709 12:27:42.949911 13014 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.070619  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:43.071672  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:43.074445  5630 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:43.056219 12919 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:43.076452  5635 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:27:43.072468  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 481009us
I20260709 12:27:43.078739  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:43.081125  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:43.082010  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:43.085760  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:43.076242 12799 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.012s	sys 0.014s
I20260709 12:27:42.999889 13023 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.084836 13012 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:43.102811 13040 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:43.086171 12714 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:43.059172 12957 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:43.101236 12799 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.012s	sys 0.014s
I20260709 12:27:43.088616 12760 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.073746 12931 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.117770 12919 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.150435 12919 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.123327  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1291) took 1082 ms. Trace:
I20260709 12:27:43.156502  5704 rpcz_store.cc:276] 0709 12:27:42.041093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:42.041213 (+   120us) service_pool.cc:224] Handling call
0709 12:27:43.122980 (+1081767us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b5400452ed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:43.123213 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":375881088}
W20260709 12:27:43.126677  5553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 1385) took 1521 ms (client timeout 1783 ms). Trace:
W20260709 12:27:43.127895  5705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2860) took 1493 ms (client timeout 1685 ms). Trace:
W20260709 12:27:43.158751  5705 rpcz_store.cc:269] 0709 12:27:41.634280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:41.654621 (+ 20341us) service_pool.cc:224] Handling call
0709 12:27:43.127664 (+1473043us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b54001991d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:43.127827 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":500941568}
I20260709 12:27:43.082288 12924 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:27:43.143838  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:43.147742  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:43.149876 12931 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:43.155988  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:43.157639  5553 rpcz_store.cc:269] 0709 12:27:41.605237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:41.749185 (+143948us) service_pool.cc:224] Handling call
0709 12:27:43.126369 (+1377184us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b540044dc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:43.126568 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":376188800}
I20260709 12:27:43.166683  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:43.153041  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:43.168510  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 63986.3us
I20260709 12:27:43.173548  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:43.134423 12992 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:43.178725  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:43.180301  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:43.185082 13021 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:43.181954  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:43.187511 12714 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:27:43.236825  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54913.7us
I20260709 12:27:43.139889 13023 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:43.130450 13018 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.062s
W20260709 12:27:43.280294  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 110956us
I20260709 12:27:43.276860  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:43.281997  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:43.194761 12752 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:27:43.255782  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:43.285221 13048 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:43.290346  6276 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:43.281761 12919 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:43.195488 12949 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:43.232389 13041 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:43.191257 12817 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.222646 12602 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:43.154981 12760 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:43.135071 13040 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.303089 13039 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.008s
I20260709 12:27:43.275877 13050 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:27:43.310235 12926 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.903s	user 0.008s	sys 0.005s
W20260709 12:27:43.310287 12848 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.730s	user 0.008s	sys 0.003s
I20260709 12:27:43.274957 13049 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:43.176851 13025 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.114s
I20260709 12:27:43.221731 12992 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.291801 13012 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:43.328876 12675 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.586s	user 0.011s	sys 0.001s
I20260709 12:27:43.329905 13047 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:43.343739  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61309.3us
I20260709 12:27:43.291761  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:43.349002  5553 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:43.325731 13041 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:43.339908 13045 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:43.346093 12848 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:43.355811 13041 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:43.349155 12675 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:43.366637 12675 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:43.367518  5629 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:27:43.368791 13026 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:43.352459 12529 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:43.376050  5556 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:27:43.334898 12817 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:43.328037 12949 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.370302 12529 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:43.353847 13014 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:43.379212  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 138022us
I20260709 12:27:43.354627 13049 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:43.386387 13052 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:43.386901 13052 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:43.387750 12834 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:43.389405  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3002) took 1080 ms. Trace:
I20260709 12:27:43.389791  5633 rpcz_store.cc:276] 0709 12:27:42.308840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:42.314607 (+  5767us) service_pool.cc:224] Handling call
0709 12:27:43.389185 (+1074578us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b540032dcd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:43.389343 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31699328}
I20260709 12:27:43.391161  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:43.350075 13050 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:43.394085 12994 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.325s	user 0.003s	sys 0.001s
I20260709 12:27:43.396822  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2995) took 1389 ms. Trace:
I20260709 12:27:43.398240 12992 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:43.397243  5557 rpcz_store.cc:276] 0709 12:27:42.007371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:42.030160 (+ 22789us) service_pool.cc:224] Handling call
0709 12:27:42.030492 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:42.030883 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:42.030918 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:42.031521 (+   603us) log.cc:844] Serialized 26 byte log entry
0709 12:27:42.068305 (+ 36784us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:42.068374 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:42.068455 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:43.393373 (+1324918us) spinlock_profiling.cc:243] Waited 4.2 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:43.396460 (+  3087us) raft_consensus.cc:1685] finished
0709 12:27:43.396487 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:43.396752 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":99,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"spinlock_wait_cycles":9670912,"thread_start_us":36322,"threads_started":1,"wal-append.queue_time_us":36668}
I20260709 12:27:43.398845 12992 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:43.399519  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:43.401108 13012 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:43.401777 12949 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:43.403872 12817 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:43.345947 12926 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.400166  5557 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:43.411064  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:43.420993  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:43.423214  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 99144.3us
W20260709 12:27:43.423600  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:43.421733  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:43.435962  5554 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 24: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:43.438476 13050 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:43.439569 13042 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:27:43.441866 13040 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:43.443617  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:43.448716 12760 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:43.387755 13052 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.451177  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:43.357026 13047 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:43.464604  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 107545us
I20260709 12:27:43.464191  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:43.371512 13045 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:43.475160 13045 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:43.420778 12924 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:43.420797 12799 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:43.483461  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49773.1us
I20260709 12:27:43.413576 13014 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.067s
I20260709 12:27:43.492100  5558 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
W20260709 12:27:43.493259  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:43.477502 12949 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:43.508576  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:43.512972 13013 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.922s	user 0.010s	sys 0.004s
I20260709 12:27:43.515066 12817 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:43.516186 12817 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:43.525640 12529 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:43.526011 13013 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:43.526736 13013 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:43.529884 13013 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:43.531790 13047 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:43.526199 12817 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:43.534399 12926 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:43.537024 13000 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:43.538074 13000 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:27:43.538766 13000 consensus_queue.cc:1243] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 18.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:27:43.545197  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:43.546231 12817 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:43.546636 12817 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:43.546973 12817 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:43.547530 13037 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:43.488072  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:43.548115 13037 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:43.589674  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 109334us
I20260709 12:27:43.507051 12949 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:43.599579  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:43.491359 12799 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.598809 12949 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:43.505973 13049 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.617812  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:43.618498  5559 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:43.614378 12714 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:43.628903 12714 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.642493 13031 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:43.647689 12815 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.650702 12822 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:43.651197 12815 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.651458 12822 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:43.617002 13041 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.628885 13037 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.643229 13031 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:43.684985  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:42.566035  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:43.729132 13037 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:43.838246 12998 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:43.838990 12998 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:43.840476 12970 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.841454 12373 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:43.842072 12949 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.842320 12373 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:43.841761 12970 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:43.844605 13069 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.150s	user 0.006s	sys 0.016s
I20260709 12:27:43.846203 13041 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:43.844630 12926 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:43.844880 13062 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.188s	user 0.000s	sys 0.002s
W20260709 12:27:43.859376  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:43.847096 12373 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:43.865520 12926 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:43.868168  5559 raft_consensus.cc:2410] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:27:43.871394 12934 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.876055  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:27:43.878161  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:43.884719  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:43.879011  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1290) took 1838 ms. Trace:
I20260709 12:27:43.880574 12799 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:43.881507 12817 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.893635  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2740) took 2704 ms. Trace:
I20260709 12:27:43.894234  5561 rpcz_store.cc:276] 0709 12:27:41.181987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:41.231266 (+ 49279us) service_pool.cc:224] Handling call
0709 12:27:42.184476 (+953210us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:42.225335 (+ 40859us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:42.225432 (+    97us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:42.226489 (+  1057us) log.cc:844] Serialized 26 byte log entry
0709 12:27:42.297189 (+ 70700us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:42.297301 (+   112us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:42.297424 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:42.380429 (+ 83005us) raft_consensus.cc:1685] finished
0709 12:27:42.380449 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:43.693677 (+1313228us) spinlock_profiling.cc:243] Waited 953 ms on lock 0x7b540016ba50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:43.886715 (+193038us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40391,"raft.queue_time_us":41099,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":270,"spinlock_wait_cycles":2191150336,"thread_start_us":70146,"threads_started":1,"wal-append.queue_time_us":98580,"wal-append.run_cpu_time_us":1803,"wal-append.run_wall_time_us":1056457}
I20260709 12:27:43.888021  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:43.889778  6276 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:43.885485  5709 rpcz_store.cc:276] 0709 12:27:42.040795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:42.040929 (+   134us) service_pool.cc:224] Handling call
0709 12:27:42.041271 (+   342us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:42.080878 (+ 39607us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:42.080915 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:43.548942 (+1468027us) spinlock_profiling.cc:243] Waited 3.83 ms on lock 0x7b2c0008bd48. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:27:43.549520 (+   578us) log.cc:844] Serialized 52 byte log entry
0709 12:27:43.629763 (+ 80243us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:43.822350 (+192587us) log.cc:844] Serialized 14 byte log entry
0709 12:27:43.823003 (+   653us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:43.823123 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:43.878611 (+ 55488us) raft_consensus.cc:1685] finished
0709 12:27:43.878784 (+   173us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:43.878944 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":84689,"raft.queue_time_us":42436,"raft.run_cpu_time_us":333,"raft.run_wall_time_us":8344,"spinlock_wait_cycles":9248896,"thread_start_us":256381,"threads_started":2,"wal-append.queue_time_us":97672}
I20260709 12:27:43.899724  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:43.901168  5559 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
W20260709 12:27:43.885813 13023 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.595s	user 0.024s	sys 0.000s
I20260709 12:27:43.904521  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:43.874953 12928 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.906214  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:43.907465  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:43.905540  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:43.868223 12753 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:43.913486  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:43.914834  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:43.915901  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:43.918313  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:43.920022  5561 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:43.921432  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:43.892076 12941 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:43.923485  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:43.914299 13044 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.730s	user 0.004s	sys 0.000s
W20260709 12:27:43.925863  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:43.925343  5559 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 18: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:43.961037  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:43.892020 12934 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:43.969826  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:43.884655 12998 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:43.973482  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:43.979027  5561 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 19: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:27:43.959692 13059 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.180s	user 0.000s	sys 0.014s
I20260709 12:27:43.974759  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:43.917711 12928 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:43.981861  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.721s (SENT) [suppressed 19 similar messages]
I20260709 12:27:43.986629  5559 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:27:43.988224  5559 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:43.917286 12753 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:43.927413 12941 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:43.994110  5559 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
W20260709 12:27:43.999606 13070 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:27:44.003505 12753 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:44.004179  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2788) took 1758 ms. Trace:
I20260709 12:27:44.022047  5561 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:44.021185  5628 rpcz_store.cc:276] 0709 12:27:42.246270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:42.246393 (+   123us) service_pool.cc:224] Handling call
0709 12:27:42.246726 (+   333us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:42.247098 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:42.247128 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:42.247637 (+   509us) log.cc:844] Serialized 26 byte log entry
0709 12:27:43.143525 (+895888us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x7b18004cc4e0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:27:43.192455 (+ 48930us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:43.192526 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:43.192985 (+   459us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:43.882697 (+689712us) spinlock_profiling.cc:243] Waited 35 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:44.003966 (+121269us) raft_consensus.cc:1685] finished
0709 12:27:44.003989 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:44.004121 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6208,"raft.queue_time_us":95,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":9362,"spinlock_wait_cycles":84372608,"thread_start_us":41372,"threads_started":1,"wal-append.queue_time_us":936600}
I20260709 12:27:44.023120  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 2998) took 1902 ms. Trace:
I20260709 12:27:44.023528  5552 rpcz_store.cc:276] 0709 12:27:42.121008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:42.121166 (+   158us) service_pool.cc:224] Handling call
0709 12:27:42.121455 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:42.618867 (+497412us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:44.022914 (+1404047us) spinlock_profiling.cc:243] Waited 446 ms on lock 0x7b540044e154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:44.023069 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1025187712}
I20260709 12:27:44.025094  5552 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:44.025410  5552 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:44.026286  5552 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:44.027295  5552 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:44.031792 12970 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:44.006990 12941 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:44.008694 13066 log.cc:927] Time spent T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.126s	user 0.002s	sys 0.000s
W20260709 12:27:44.009987 12834 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.622s	user 0.017s	sys 0.002s
I20260709 12:27:43.980922 12913 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:44.033633 12913 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:44.010874 12822 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:44.035740  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1274) took 1126 ms. Trace:
I20260709 12:27:44.036193  6273 rpcz_store.cc:276] 0709 12:27:42.909461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:42.909603 (+   142us) service_pool.cc:224] Handling call
0709 12:27:42.909904 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:44.034754 (+1124850us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:44.034789 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:44.034835 (+    46us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:44.034892 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:44.035489 (+   597us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54002909d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:44.035539 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:44.035678 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2586422528}
I20260709 12:27:44.039137  5552 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 21 index: 12. (term mismatch)
I20260709 12:27:44.039563  5552 pending_rounds.cc:85] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 11
I20260709 12:27:44.039805  5552 pending_rounds.cc:107] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:27:44.040053  5552 raft_consensus.cc:2889] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:27:44.018443 12988 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.889s	user 0.007s	sys 0.010s
I20260709 12:27:44.019418  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2771) took 2388 ms. Trace:
I20260709 12:27:43.995596 13031 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:44.010015 12928 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:44.040617  5636 rpcz_store.cc:276] 0709 12:27:41.631002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:41.632441 (+  1439us) service_pool.cc:224] Handling call
0709 12:27:41.661033 (+ 28592us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:41.979954 (+318921us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:44.019117 (+2039163us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b540008c0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:44.019326 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":731805440}
I20260709 12:27:44.035758 12913 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:44.047539 12815 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:44.050896 12817 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:44.053742 12926 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:44.053431  6277 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:44.056093  6277 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:44.054636  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:44.057312  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:44.068769  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 112563us
I20260709 12:27:44.069500  5628 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:27:44.071861  5628 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:27:44.072539  5559 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:44.072675  6277 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:27:44.073421  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1222) took 1068 ms. Trace:
I20260709 12:27:44.073769  6277 rpcz_store.cc:276] 0709 12:27:43.005023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:43.005153 (+   130us) service_pool.cc:224] Handling call
0709 12:27:43.005443 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:44.053085 (+1047642us) spinlock_profiling.cc:243] Waited 5.24 ms on lock 0x7b4c0044b700. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff893ea20 00007f4ff894fa29 00007f4ff8915644 00007f4ff893cf2d 00007f4ff893cb93 00007f4ffa66d18b 00007f4ffa66dcb5 00007f4ffa668f30 00007f4ffa66a7de 00007f4ffa674b49 00007f4ffa675416
0709 12:27:44.073146 (+ 20061us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:44.073366 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12048768}
I20260709 12:27:44.092545  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.093992  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.095357  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.096112  5701 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:44.100478  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.101462  5701 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:44.103693  5710 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:27:44.086745 12991 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 12:27:44.111238  5635 pending_rounds.cc:85] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 13
I20260709 12:27:44.113662  5635 pending_rounds.cc:107] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:27:44.114037  5635 raft_consensus.cc:2889] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:27:44.115888  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.117187  5628 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:44.129365 12890 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:27:44.130435 13078 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:44.104791 13002 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:44.101440 12928 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:44.101282 12913 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:44.133263 12893 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:44.136698 12893 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:44.138839  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.139601  5705 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:44.101069 13001 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:27:44.153436  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.154368  5707 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:44.161643  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.162307  5705 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:27:44.167178  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:44.168902  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.170847  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.171494  5707 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:44.172127  5627 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:44.173095  5705 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 16: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:44.138270 13002 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:44.198519  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.199482  5701 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:44.141016 12928 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:44.200794  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.203913 12928 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:44.208539  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.161259 12913 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:44.212620  5705 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:27:44.142270 12893 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:44.217092 12893 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:44.222150 12957 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:44.240528 13033 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.144s	user 0.004s	sys 0.000s
W20260709 12:27:44.243806  6274 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1218) took 1941 ms (client timeout 1521 ms). Trace:
W20260709 12:27:44.244288  6274 rpcz_store.cc:269] 0709 12:27:42.302386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:42.302804 (+   418us) service_pool.cc:224] Handling call
0709 12:27:44.243558 (+1940754us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b54006379d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:44.243699 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":641339648}
I20260709 12:27:44.245833  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2792) took 1393 ms. Trace:
I20260709 12:27:44.246178  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:44.247390  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:44.248029 13015 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.833s	user 0.002s	sys 0.003s
I20260709 12:27:44.249819  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1220) took 1927 ms. Trace:
W20260709 12:27:44.250451  5632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 2993) took 1943 ms (client timeout 2193 ms). Trace:
W20260709 12:27:44.251116  5632 rpcz_store.cc:269] 0709 12:27:42.306701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:42.306831 (+   130us) service_pool.cc:224] Handling call
0709 12:27:44.250266 (+1943435us) spinlock_profiling.cc:243] Waited 479 ms on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:44.250391 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1100683136}
I20260709 12:27:44.250286  6279 rpcz_store.cc:276] 0709 12:27:42.322627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:42.322767 (+   140us) service_pool.cc:224] Handling call
0709 12:27:42.334289 (+ 11522us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:42.394378 (+ 60089us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:42.394410 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:42.394984 (+   574us) log.cc:844] Serialized 52 byte log entry
0709 12:27:42.414677 (+ 19693us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:42.414723 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:42.557077 (+142354us) spinlock_profiling.cc:243] Waited 36.4 ms on lock 0x7b54002ce7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:42.557138 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:44.247346 (+1690208us) spinlock_profiling.cc:243] Waited 3.62 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:44.249512 (+  2166us) raft_consensus.cc:1685] finished
0709 12:27:44.249544 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:44.249744 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23061,"raft.queue_time_us":23674,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":4912,"spinlock_wait_cycles":92135424,"thread_start_us":5375,"threads_started":1,"wal-append.queue_time_us":5720}
W20260709 12:27:44.253141  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.253238 13018 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:44.253877 13018 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:44.254808 13018 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:44.255928 13018 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:44.256430 13002 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:44.256970  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 1394) took 1109 ms. Trace:
I20260709 12:27:44.257318  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:44.257741  5636 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:44.258057  6274 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:44.259532  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:44.246218  5631 rpcz_store.cc:276] 0709 12:27:42.852529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:42.852651 (+   122us) service_pool.cc:224] Handling call
0709 12:27:42.852976 (+   325us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:42.853353 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:42.853381 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:42.861303 (+  7922us) log.cc:844] Serialized 52 byte log entry
0709 12:27:42.996421 (+135118us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:43.016889 (+ 20468us) log.cc:844] Serialized 14 byte log entry
0709 12:27:43.017275 (+   386us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:43.017345 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:44.239975 (+1222630us) spinlock_profiling.cc:243] Waited 5.27 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:44.245548 (+  5573us) raft_consensus.cc:1685] finished
0709 12:27:44.245638 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:44.245778 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5516,"prepare.queue_time_us":23528,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":2824,"raft.queue_time_us":641,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":38923,"spinlock_wait_cycles":12110080,"thread_start_us":89148,"threads_started":1,"wal-append.queue_time_us":176106}
I20260709 12:27:44.257346  5554 rpcz_store.cc:276] 0709 12:27:43.147330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:43.147464 (+   134us) service_pool.cc:224] Handling call
0709 12:27:44.256768 (+1109304us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b54001be254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:44.256910 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":631633280}
I20260709 12:27:44.261811  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3003) took 1104 ms. Trace:
I20260709 12:27:44.262172  5553 rpcz_store.cc:276] 0709 12:27:43.157574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:43.178569 (+ 20995us) service_pool.cc:224] Handling call
0709 12:27:43.178805 (+   236us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:43.352877 (+174072us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:44.261602 (+908725us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b54001c4dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:44.261760 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":390905216}
I20260709 12:27:44.264851 12949 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:44.246186  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:44.268285  6277 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:44.270404  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:44.271289  5630 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 22: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:44.275595 12836 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:44.292317  5630 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:44.290699  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:44.297056  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:44.306502  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:44.298000 12890 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.016s
I20260709 12:27:44.316334 12949 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:44.317579 13052 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:44.319973  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:44.323586 12949 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:44.329460 13039 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:44.330331 13039 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:44.335438 12949 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:27:44.339831 12949 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:44.342283 12752 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:44.342869 12752 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:27:44.349663  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.294073 12675 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:44.309881 13023 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:44.359670 12934 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:44.311340 12959 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:44.331610 13039 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:44.367270 12675 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:44.371783  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.372938 12675 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:44.357028 12811 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:44.365736 13023 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:44.383947 12811 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:44.387534 13023 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:44.392163 12752 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:44.393514 12811 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:44.394381 12752 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:44.409363 13067 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:44.409979 13067 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:44.411113 13067 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:44.412101 13023 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:44.414116 13067 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:44.414469 13023 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:44.414878 12998 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:44.427112 13008 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:44.427937 13008 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:44.429250 13008 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:44.431155 13008 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:44.436031 13039 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:44.436280 12973 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:44.493476  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1402) took 1038 ms. Trace:
I20260709 12:27:44.494705  5552 rpcz_store.cc:276] 0709 12:27:43.454546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:44.046226 (+591680us) service_pool.cc:224] Handling call
0709 12:27:44.046563 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:44.046758 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:27:44.287790 (+241032us) log.cc:844] Serialized 14 byte log entry
0709 12:27:44.491826 (+204036us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:44.491965 (+   139us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:44.492142 (+   177us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:44.492294 (+   152us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:44.493021 (+   727us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":190416,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"thread_start_us":443769,"threads_started":2,"wal-append.queue_time_us":99228}
I20260709 12:27:44.276602  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:44.499459  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:44.287810  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:44.533064  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:44.534264  5553 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:44.534618  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3008) took 1527 ms. Trace:
I20260709 12:27:44.535454  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1395) took 1387 ms. Trace:
I20260709 12:27:44.535759  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:44.501603  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:44.536094  5555 rpcz_store.cc:276] 0709 12:27:43.147623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:43.147746 (+   123us) service_pool.cc:224] Handling call
0709 12:27:43.148025 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:43.356457 (+208432us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:43.356514 (+    57us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:43.356594 (+    80us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:43.356637 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:44.534923 (+1178286us) spinlock_profiling.cc:243] Waited 208 ms on lock 0x7b54001af254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:44.534983 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:44.535270 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":478424448}
I20260709 12:27:44.504631  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1401) took 1096 ms. Trace:
I20260709 12:27:44.535224  5627 rpcz_store.cc:276] 0709 12:27:43.006549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:43.007957 (+  1408us) service_pool.cc:224] Handling call
0709 12:27:43.008319 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:44.174045 (+1165726us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:44.533769 (+359724us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54001dd8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:44.534316 (+   547us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2676091648}
I20260709 12:27:44.538489  5553 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:44.522917  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3001) took 1365 ms. Trace:
I20260709 12:27:44.525936  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1393) took 2208 ms. Trace:
W20260709 12:27:44.526211  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 180039us
I20260709 12:27:44.526762  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1397) took 1371 ms. Trace:
I20260709 12:27:44.554024  5558 rpcz_store.cc:276] 0709 12:27:43.155198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:43.161286 (+  6088us) service_pool.cc:224] Handling call
0709 12:27:43.180848 (+ 19562us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:43.492721 (+311873us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:44.524020 (+1031299us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b54003a89d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:44.526544 (+  2524us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":715215104}
W20260709 12:27:44.567538  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.573294  5552 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:44.551033  5556 rpcz_store.cc:276] 0709 12:27:43.156888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:43.157049 (+   161us) service_pool.cc:224] Handling call
0709 12:27:43.164624 (+  7575us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:43.376515 (+211891us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:44.522140 (+1145625us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b54001c43d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:44.522690 (+   550us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":485493632}
I20260709 12:27:44.588900  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:44.592929  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.596949  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:44.596930  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.599398  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.605818  5709 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:44.608067  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.616701  5707 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 19
W20260709 12:27:44.650401  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:44.656062  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.657891 12373 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:44.659289  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.658914 12373 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:44.660270  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.537638  5561 rpcz_store.cc:276] 0709 12:27:43.407812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:44.022869 (+615057us) service_pool.cc:224] Handling call
0709 12:27:44.023161 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:44.023384 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:27:44.210224 (+186840us) log.cc:844] Serialized 14 byte log entry
0709 12:27:44.503611 (+293387us) log.cc:844] Serialized 14 byte log entry
0709 12:27:44.503920 (+   309us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:44.503985 (+    65us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:44.504072 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:44.504176 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:44.504439 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":213342,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":65,"thread_start_us":443357,"threads_started":2,"wal-append.queue_time_us":194786}
I20260709 12:27:44.684077 12373 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:44.684998  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 104351us
W20260709 12:27:44.685137  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 396182us
I20260709 12:27:44.408142 12913 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:44.688335  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.690810  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:44.692456  5704 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:27:44.693465  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:44.693895  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.694242  6274 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:44.694347  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:44.695335  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.695963  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.697124  5705 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:44.697019  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:44.697399  5710 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:27:44.698786  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.699740  5710 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:44.688666 12675 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.696143  5704 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
W20260709 12:27:44.691232  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.701186  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:44.705729  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:44.706300  5557 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:44.706543 12872 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.707576 12830 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:44.708772 12848 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.709553 12714 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:44.710075 12848 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:44.711796  5557 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2764) took 1422 ms (client timeout 1566 ms). Trace:
W20260709 12:27:44.712489  5557 rpcz_store.cc:269] 0709 12:27:43.289791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:43.420649 (+130858us) service_pool.cc:224] Handling call
0709 12:27:44.711428 (+1290779us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54001bd354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:44.711681 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2943698688}
I20260709 12:27:44.551152  5560 rpcz_store.cc:276] 0709 12:27:42.317253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:42.848192 (+530939us) service_pool.cc:224] Handling call
0709 12:27:42.848556 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:42.899779 (+ 51223us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:42.899829 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:42.900382 (+   553us) log.cc:844] Serialized 26 byte log entry
0709 12:27:42.943895 (+ 43513us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:42.943989 (+    94us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:44.525024 (+1581035us) spinlock_profiling.cc:243] Waited 51 ms on lock 0x7b54001c48d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:44.525153 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:44.525272 (+   119us) raft_consensus.cc:1685] finished
0709 12:27:44.525286 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:44.525740 (+   454us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":117236992,"thread_start_us":42984,"threads_started":1,"wal-append.queue_time_us":54849,"wal-append.run_cpu_time_us":1666,"wal-append.run_wall_time_us":1005165}
I20260709 12:27:44.717545 13011 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.720726 13011 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:44.722780 13040 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.723681 13040 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:44.724512 12786 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:44.727816 12762 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:44.731586 12675 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:44.755649  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.758234 12847 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:44.760571  5632 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 14. (term mismatch)
I20260709 12:27:44.761057  5632 pending_rounds.cc:85] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 13
I20260709 12:27:44.732723  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:44.733186  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:44.708030 12830 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:44.707576 12872 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:44.738022  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.745756 12880 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:44.746837  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.761377  5632 pending_rounds.cc:107] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 20.14
I20260709 12:27:44.755731 12762 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:44.692543  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:44.761744 12675 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:44.710112 12923 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.777186 12830 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:44.780879  5451 catalog_manager.cc:5697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 8 to 17, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 17 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:44.821902  5707 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:44.782068 12872 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:44.782155  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:44.786588  6275 raft_consensus.cc:2410] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
W20260709 12:27:44.788663  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 102060us
I20260709 12:27:44.765738  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:44.803495  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 117347us
I20260709 12:27:44.797211  5632 raft_consensus.cc:2889] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:44.717837 13010 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:44.794013 12848 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:44.808102 12762 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:44.815347  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:44.841987  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:44.848912  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.851408  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:44.855340  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.847041 12773 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:27:44.846179  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 113830us
I20260709 12:27:44.869333 12773 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:44.874660 12773 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:44.876683 12373 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:44.896865 12373 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:44.897300 12373 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:44.881117  5703 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 10. (term mismatch)
I20260709 12:27:44.848522 12880 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:44.884684  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.838495 12923 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:44.900750  6274 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:44.897912  5703 pending_rounds.cc:85] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 9
I20260709 12:27:44.906599  5553 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:44.909051  5703 pending_rounds.cc:107] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 12:27:44.909332  5703 raft_consensus.cc:2889] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:44.918959 12880 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:44.921041  5703 pending_rounds.cc:107] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 12:27:44.921778  5703 raft_consensus.cc:2889] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:44.918882 12921 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:44.893059  5451 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 17 to 20. New cstate: current_term: 20 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:44.915588  5708 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:44.926970  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:44.930433  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:44.932390  5557 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:44.925652  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:44.934054 12991 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.934746 12991 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:44.935248  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 66390.7us
I20260709 12:27:44.936305 12675 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:27:44.936534 13011 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:44.940791  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.942214  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:44.941704  5451 catalog_manager.cc:5697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 15 to 19. New cstate: current_term: 19 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:44.943296  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.945703 12830 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:44.945907 12923 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:44.943869  5451 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:44.952881  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.953410  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:44.954458  5707 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:44.954834  5710 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:44.955168  5710 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:44.848516 13010 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:44.958709 13010 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:44.960208 13076 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.960773 13076 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:44.962301 13040 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:44.964143  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 65209.6us
I20260709 12:27:44.956182  5710 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:44.971028  5710 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:44.970734  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:44.923065 12862 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:44.976724  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 87105us
I20260709 12:27:44.977685  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:44.980861 12973 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:44.981905  6278 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:27:45.007776  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:45.009122  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:45.018589  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:45.018287  5451 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:45.019450  5553 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 20: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:45.018796  5705 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:27:45.025310  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:45.026371  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:45.036391  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:45.040119  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:45.020092  5451 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 15 to 18, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 18 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:45.051133  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:44.997653 12773 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:45.051944  6275 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
W20260709 12:27:45.056696  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 72791.4us
W20260709 12:27:45.064447  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 227148us
I20260709 12:27:45.071821  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:45.072585 12548 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:45.073659  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:45.072670  5627 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:27:45.060562 12880 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.103323  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:45.104862  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:45.105871  5629 raft_consensus.cc:2410] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 24 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 16.
W20260709 12:27:45.108768 13039 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.672s	user 0.015s	sys 0.012s
I20260709 12:27:45.072885  5451 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 12 to 15, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 15 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:27:45.121373 13008 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.690s	user 0.022s	sys 0.004s
I20260709 12:27:45.122017  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:45.122210 13008 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:27:45.123113 12753 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:45.123595 12752 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:27:45.125008  5459 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 276) took 89226 ms (client timeout 15000 ms). Trace:
I20260709 12:27:45.115475  5451 catalog_manager.cc:5697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 13 to 16, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 16 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:45.075606 13067 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.502s	user 0.007s	sys 0.013s
W20260709 12:27:45.123585  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 187593us
I20260709 12:27:45.153043 12974 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.153782 12974 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:45.154570 12998 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.155171 12998 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:45.155898 12974 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:45.161240  5710 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:27:45.046869 12862 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:45.172030 12949 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:45.172976 12949 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:45.174096 13067 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.174484 12714 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:45.174751 13067 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:45.176653  5559 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: [16.9-19.11]
I20260709 12:27:45.176834  6279 raft_consensus.cc:2410] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:27:45.178079  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:45.179060  5557 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:45.178782 12548 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:45.183254  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:45.184252 13076 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:45.184944  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:45.185608  5552 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:27:45.188872  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:45.189764  5561 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:45.192543  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 106583us
I20260709 12:27:45.192945  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:45.193573  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:45.193859  5558 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:45.194897 12752 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:45.195361 12752 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:45.195845 12752 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:45.198199 12714 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.198961 12714 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:45.200623 12836 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.201006  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:45.201277 12836 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:45.201788  5635 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:45.203073  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:45.203760  5632 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:45.204008 12836 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:45.204525  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:45.204946 12998 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:45.205371  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 65466.1us
I20260709 12:27:45.207443 13023 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:45.208200 13023 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:45.178148 12991 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:45.157012  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 73312us
I20260709 12:27:45.101540 12675 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:45.171635 12880 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:45.222929  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 142171us
I20260709 12:27:45.220220  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:45.227262  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 246870us
I20260709 12:27:45.224562  6273 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:45.229497  5556 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:45.235293  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:45.125535  5459 rpcz_store.cc:269] 0709 12:26:15.898189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.898443 (+   254us) service_pool.cc:224] Handling call
0709 12:27:45.124743 (+89226300us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:27:35.492297 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:27:35.493306 (+  1009us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:27:35.493383 (+    77us) write_op.cc:435] Acquired schema lock
0709 12:27:35.493422 (+    39us) tablet.cc:662] Decoding operations
0709 12:27:35.494728 (+  1306us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:27:35.494891 (+   163us) write_op.cc:641] Partition lock acquired for write op
0709 12:27:35.494941 (+    50us) tablet.cc:685] Acquiring locks for 8 operations
0709 12:27:35.496057 (+  1116us) tablet.cc:701] Row locks acquired
0709 12:27:35.496126 (+    69us) write_op.cc:260] PREPARE: finished
0709 12:27:35.502630 (+  6504us) write_op.cc:270] Start()
0709 12:27:35.502830 (+   200us) write_op.cc:276] Timestamp: P: 1783600055496287 usec, L: 0
0709 12:27:35.502891 (+    61us) op_driver.cc:348] REPLICATION: starting
0709 12:27:35.507135 (+  4244us) log.cc:844] Serialized 3147 byte log entry
0709 12:27:35.956020 (+448885us) spinlock_profiling.cc:243] Waited 3.23 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa66de28 00007f4ffe0a8581 00007f4ffe0a7d58 00007f4ffe0ac9f1 00007f4ffe0ac9a9 00007f4ffe0ac939 00007f4ffe0ac901 00007f4ffe0abb2d 0000000000388634 0000000000388479
0709 12:27:36.631112 (+675092us) op_driver.cc:464] REPLICATION: finished
0709 12:27:36.955106 (+323994us) write_op.cc:301] APPLY: starting
0709 12:27:36.967790 (+ 12684us) tablet.cc:1367] starting BulkCheckPresence
0709 12:27:37.099529 (+131739us) tablet.cc:1370] finished BulkCheckPresence
0709 12:27:37.099570 (+    41us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:27:37.864680 (+765110us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:27:37.864991 (+   311us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=16,key_file_lookups=16,delta_file_lookups=16,mrs_lookups=0
0709 12:27:37.865051 (+    60us) write_op.cc:312] APPLY: finished
0709 12:27:37.865411 (+   360us) log.cc:844] Serialized 81 byte log entry
0709 12:27:38.182796 (+317385us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:27:38.217002 (+ 34206us) write_op.cc:454] Released schema lock
0709 12:27:38.226205 (+  9203us) write_op.cc:341] FINISH: Updating metrics
0709 12:27:38.282487 (+ 56282us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:27:38.287282 (+  4795us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":749416,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":301526,"apply.run_cpu_time_us":63796,"apply.run_wall_time_us":1310627,"cfile_cache_hit":34,"cfile_cache_hit_bytes":37567,"num_ops":8,"prepare.queue_time_us":57735,"prepare.run_cpu_time_us":24884,"prepare.run_wall_time_us":464093,"raft.queue_time_us":433475,"raft.run_cpu_time_us":10841,"raft.run_wall_time_us":135625,"replication_time_us":1127967,"spinlock_wait_cycles":7420800,"thread_start_us":700390,"threads_started":5,"wal-append.queue_time_us":528333,"wal-append.run_cpu_time_us":32418,"wal-append.run_wall_time_us":2976277}]]}
I20260709 12:27:45.235306  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:45.210073 12862 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:45.239553 12753 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:45.246367  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:45.165371 13011 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.249172  5627 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:45.221494 12675 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:45.207259 12714 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:45.157605 12773 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:45.264035  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:45.270092  5556 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:27:45.275969  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:45.265473 12675 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:45.286167  5629 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:45.289424  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:45.292531 12957 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.291819 12376 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.003s
W20260709 12:27:45.293301 12786 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.568s	user 0.011s	sys 0.005s
I20260709 12:27:45.293156 12957 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:45.294498  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 98145.7us
I20260709 12:27:45.295539  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:45.272668  5451 catalog_manager.cc:5697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:45.289427 12762 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:45.302855  5561 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:45.292042 12548 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:45.307099  5630 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:27:45.308606  6272 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:27:45.308768  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:45.309755 12935 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:27:45.310191  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:45.310540  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:45.310983  5635 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:45.312302  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:45.313259  5556 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:45.314637 13041 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.315341 13041 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:45.316951  5629 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:45.318485 13041 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:45.319373  5560 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:45.320649 12949 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:45.325011  5599 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:45.301098 12957 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:45.333119  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:45.333987  5628 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:45.334561  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:45.335217  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 51305.5us
I20260709 12:27:45.341073  5552 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:27:45.327657  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:45.343549  5555 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:27:45.302837 12762 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:45.341745 12548 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:45.342044 12935 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:45.348254  5632 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 26: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:45.358923 12376 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:27:45.363080 12548 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:45.305671 12959 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:27:45.364692  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58902.1us
I20260709 12:27:45.287599 13011 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:45.233439 13023 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:45.396593  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 462886us
I20260709 12:27:45.401295  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:45.404997  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 280032us
I20260709 12:27:45.403978  5451 catalog_manager.cc:5697] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 14 to 17. New cstate: current_term: 17 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:45.389294 12548 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:45.427382  5710 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 23. Current term is 24. Ops: []
I20260709 12:27:45.370440 13067 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:45.419480  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:45.389698 12959 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:45.504206  5558 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
W20260709 12:27:45.512154  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 92752.6us
I20260709 12:27:45.413546 12872 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:27:45.514295  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 81641.1us
I20260709 12:27:45.515163 12921 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:45.515606 12921 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:45.510975 12836 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:45.416093 13076 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:45.400229 12973 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:45.508489 12959 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:27:45.531414  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:45.512960  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:45.515957 12921 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:27:45.535187  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 200815us
I20260709 12:27:45.522385  5451 catalog_manager.cc:5697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 10 to 19, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 19 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:45.544668  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:45.545459  5701 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:45.549052  5705 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:45.549899 12752 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:45.550763  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:45.553320  5707 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:45.554359  5708 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:45.555258  5702 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:45.556418  5701 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:27:45.557103 12773 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:45.558127  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:45.559056  5555 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:45.560194  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:45.552062 12752 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:45.567260  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:45.595742  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:45.614384  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:45.615456 13039 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:45.616174 13039 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:45.615952  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:45.616618 13039 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:45.617977  5555 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20260709 12:27:45.618312  5555 pending_rounds.cc:85] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 10
I20260709 12:27:45.618646  5555 pending_rounds.cc:107] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 12:27:45.618913  5555 raft_consensus.cc:2889] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:45.620525  5558 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: []
I20260709 12:27:45.621004  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:45.621753  5704 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:45.620337  5555 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:45.623391  5552 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:27:45.519413 12872 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:45.543608 12847 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.779s	user 0.014s	sys 0.004s
W20260709 12:27:45.591368  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:45.593439  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:45.638449  5556 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:45.595716  5561 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:45.600674  5451 catalog_manager.cc:5697] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 16 to 18, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 18 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:45.604525  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:45.605141  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 89147.2us
I20260709 12:27:45.607051  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:45.608872  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:45.611057  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:45.608263  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:45.666833  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:45.668282  5629 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:45.668375 12913 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:45.669373  5635 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:45.669587  5451 catalog_manager.cc:5697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 17 to 18, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 18 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:27:45.669090 12913 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:45.670266  5627 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
W20260709 12:27:45.672683  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 425392us
I20260709 12:27:45.671523  5451 catalog_manager.cc:5697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 14 to 17, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 17 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:45.564280 13040 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:45.682981  5710 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:45.678946  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:45.684434 12913 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:45.688201 12911 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.688961 12911 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:45.685501  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:45.694024  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 83146.8us
I20260709 12:27:45.661883 12872 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:45.568085 13076 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:45.703619  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 496295us
I20260709 12:27:45.682271 13040 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:45.711576  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:45.708408  5451 catalog_manager.cc:5697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 13 to 15, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 15 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:27:45.691093 13008 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.020s
I20260709 12:27:45.699353 13010 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:45.706952 12752 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:45.710448 13040 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:45.707187 13041 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:45.721146  6235 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.024s	sys 0.038s
I20260709 12:27:45.717938 12998 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:27:45.740517  6235 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.024s	sys 0.038s
I20260709 12:27:45.718116 12913 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:45.721540 12880 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:45.749327  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 157291us
I20260709 12:27:45.721495 12786 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:45.703251 12974 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.027s
W20260709 12:27:45.762300  6272 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1276) took 2840 ms (client timeout 2076 ms). Trace:
I20260709 12:27:45.745605 12998 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:45.729861 12752 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:27:45.766602  5675 connection.cc:256] RPC call timeout handler was delayed by -1.5272581879999052s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15200000000000002s for a response
W20260709 12:27:45.782651  5675 connection.cc:256] RPC call timeout handler was delayed by -1.2249877639997067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19970000000000002s for a response
W20260709 12:27:45.783099  5675 connection.cc:256] RPC call timeout handler was delayed by -1.2176581879998594s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1864s for a response
W20260709 12:27:45.783463  5675 connection.cc:256] RPC call timeout handler was delayed by -1.1636581879997721s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19240000000000002s for a response
I20260709 12:27:45.733561 12762 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:45.773074  6272 rpcz_store.cc:269] 0709 12:27:42.910041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:42.924951 (+ 14910us) service_pool.cc:224] Handling call
0709 12:27:45.731359 (+2806408us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b5400055854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:45.750873 (+ 19514us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5409088896}
I20260709 12:27:45.751797 12913 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:45.747676 13067 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:45.779099  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:45.737689 12753 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:45.761394 12786 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:45.801064  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:45.787032  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:45.804458 13112 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.805445 13112 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:45.808425 13107 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:45.808688 12847 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:45.808997 12949 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:45.809172 12847 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:45.809679 13110 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 12:27:45.809777 12847 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:45.817421 12990 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.818126 12990 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:45.809193 13107 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:45.787433 12998 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:45.731031 12836 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:27:45.832111  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:45.832824 12753 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:45.836715 12753 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:45.842334 12911 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:45.775805 13010 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:45.847024 13010 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:45.787544 12752 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:45.848438 12880 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:45.848906 12880 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:45.849290 12880 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:45.853384 13076 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:45.843681 12921 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:45.799412 12786 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:45.795519 12913 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:45.756757 13041 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:45.727752 12973 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:45.809859 12949 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:45.778021 12714 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.085s
I20260709 12:27:45.880658  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:45.831288 13112 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:45.832706 13011 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:45.832633 12991 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.893273  5451 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 15 to 18, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 18 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:27:45.901475 13072 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.003s	sys 0.036s
I20260709 12:27:45.856232 12762 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:45.904320 12957 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:45.870036 13067 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:45.928705  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:45.935797  5627 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:45.943666  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:45.923043 13112 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:27:45.910668 13008 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.027s
W20260709 12:27:45.946115  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 155682us
I20260709 12:27:45.917146  5451 catalog_manager.cc:5697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 16 to 20. New cstate: current_term: 20 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:45.946125 13122 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.026s
I20260709 12:27:45.915132 12991 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:45.970463  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 146315us
I20260709 12:27:45.949151 13072 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:45.930522 13076 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.035s
W20260709 12:27:45.990159  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 157542us
I20260709 12:27:45.990764 13112 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:27:45.955124 13106 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:27:45.998659  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:45.999428  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:46.006923  5451 catalog_manager.cc:5697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 15 to 17. New cstate: current_term: 17 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:45.984087 12990 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:45.995946 13072 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:45.958822 12974 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.046s
W20260709 12:27:45.998473 12373 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.092s	user 0.012s	sys 0.003s
I20260709 12:27:46.018939  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:46.025875  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68362.2us
I20260709 12:27:46.006588 13010 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:45.997815 13122 consensus_queue.cc:1243] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 21.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:27:46.024453 12773 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:46.021545 13072 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:46.038437  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 66264.3us
I20260709 12:27:46.018214 13106 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:27:46.042532  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 529610us
I20260709 12:27:46.033936  5451 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:27:46.049320  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:45.972719 12949 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.053849  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:46.014130 13039 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.031777 12921 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.012532 12998 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.021181 12957 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:27:46.116688  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:46.117437  6275 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:46.118120 12911 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:46.118520 12911 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:46.036191 12990 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.066473  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:46.040329 13107 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.027530 12373 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:46.063351 12675 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.722s	user 0.013s	sys 0.000s
I20260709 12:27:46.076215 13041 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.084685 12773 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.069902 12836 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:46.103922 13067 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.066527 12752 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.074007  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:46.108117 12998 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:46.111722 13114 log.cc:927] Time spent T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.329s	user 0.004s	sys 0.000s
I20260709 12:27:46.113211  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:46.114811  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:46.173096  6274 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:46.126000  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 59621.3us
I20260709 12:27:46.044958 13106 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:46.127027 12911 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.128149  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:46.128407  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:46.130159  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77060.2us
W20260709 12:27:46.068783 12935 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.686s	user 0.013s	sys 0.000s
I20260709 12:27:46.135716  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:46.132925 12921 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:46.141215  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:46.135669 13008 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:46.141263  6273 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:27:46.141952  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:46.128427 12376 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:46.149166 12848 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:46.143277  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:46.143880 12373 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:46.145171  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 339752us
I20260709 12:27:46.197260  5636 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:46.070801 12974 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:27:46.136386 12762 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:46.179061  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 154148us
I20260709 12:27:46.141320 12675 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.177534  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:46.146445 12836 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:46.177286  6278 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:46.136238 13076 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.182008  6280 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
W20260709 12:27:46.181809 12959 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.632s	user 0.014s	sys 0.000s
I20260709 12:27:46.133678 12990 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.200449  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:46.221730  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:46.229598  5553 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:27:46.199374  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:46.207507  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:46.205513  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:46.206897  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:46.184772 13041 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.227870  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:46.197036 12848 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:46.181474 12921 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:46.148515 12847 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.223465  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:46.200201  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:46.190320 12935 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:46.232535  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:46.191790 12376 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:46.182245 13010 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:27:46.216112  6272 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 19: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:46.235031  6275 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:27:46.212929 12714 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:46.241482  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:46.215530 12836 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:46.240841  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 284281us
I20260709 12:27:46.246582 12373 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:46.251669  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:46.262249  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 156655us
I20260709 12:27:46.258270  5559 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:46.263680  6279 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:46.283675  5628 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:46.259333 12921 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:46.271973 13128 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:27:46.277099  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:46.286842  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:46.295825  5556 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:46.283624  5560 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
W20260709 12:27:46.296258  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 63853.8us
I20260709 12:27:46.233368 13076 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.265048  5636 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:46.293197  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:46.299278  6274 raft_consensus.cc:2410] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:27:46.265461 13011 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:46.233803 12959 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.333119  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1410) took 1425 ms. Trace:
I20260709 12:27:46.236833 13008 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.269065 12714 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:46.310493  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 72210.9us
I20260709 12:27:46.310395 12376 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.311534 13139 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:46.316221  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:46.316444  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:46.317503 13124 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:46.318934 13128 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:46.319674 13108 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.041s
I20260709 12:27:46.322009 12911 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:46.375415  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 224075us
I20260709 12:27:46.322184 12847 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:46.326196  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:46.380277  5636 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:46.326910  5635 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:46.327577  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:46.278470 12373 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:46.308461  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:46.338409  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:46.386646  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 78276.4us
I20260709 12:27:46.376864  6276 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:46.343133 13136 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:27:46.345865 13106 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.360332  5556 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:27:46.250136 13041 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.363469  5553 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:46.364784  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:46.333698  5554 rpcz_store.cc:276] 0709 12:27:44.907348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:44.914286 (+  6938us) service_pool.cc:224] Handling call
0709 12:27:44.920057 (+  5771us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:44.920404 (+   347us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:44.920437 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:45.049529 (+129092us) spinlock_profiling.cc:243] Waited 2.13 ms on lock 0x7b2c000fb608. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:27:45.099769 (+ 50240us) log.cc:844] Serialized 26 byte log entry
0709 12:27:45.508427 (+408658us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:45.508491 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:45.508590 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:46.110141 (+601551us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:46.332822 (+222681us) raft_consensus.cc:1685] finished
0709 12:27:46.332854 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:46.333037 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66592,"spinlock_wait_cycles":27907072,"thread_start_us":241497,"threads_started":1,"wal-append.queue_time_us":610440}
I20260709 12:27:46.332182 13011 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:46.326366  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:46.381260  5629 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:27:46.342748 13132 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:27:46.406776  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 155256us
W20260709 12:27:46.380401 12880 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.531s	user 0.015s	sys 0.000s
I20260709 12:27:46.362656 12714 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:46.368647 13124 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:46.407948  5559 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:46.393684 12376 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:46.389593 13128 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:46.416853 13126 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
I20260709 12:27:46.390857 13039 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:46.420589  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:46.421046 13106 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:27:46.423466  5629 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:27:46.368814 12848 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.423899 13008 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.424263 12848 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:46.425143 12675 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.425595  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:46.426358 12990 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.397974 13139 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:46.428189 13139 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.431797 13137 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:46.417358 13128 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:46.433125 13039 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:46.437983 13113 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:27:46.438568 13113 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:46.441592 13131 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
W20260709 12:27:46.442786  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 126634us
I20260709 12:27:46.444815  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:46.445178  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:46.446403 12376 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:46.446861 12376 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:46.447247 12376 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:46.448159 13129 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.167s	user 0.000s	sys 0.000s
I20260709 12:27:46.455353 12376 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.457587  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:46.458120 12376 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:46.459365 12872 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.745s	user 0.012s	sys 0.002s
I20260709 12:27:46.459957 12872 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.438174 12836 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.462589 12872 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.463112 12836 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:46.396541 13040 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.625s	user 0.013s	sys 0.000s
I20260709 12:27:46.463904 13040 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.465049  5552 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:46.466044 13040 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.389878 12373 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:46.469012  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:46.414206 12904 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.007s
I20260709 12:27:46.471320  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:46.426975 12990 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:27:46.413484  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:46.439594 13113 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.480578  5556 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:46.394233 13107 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:46.484647  5554 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:27:46.426986  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:46.404341 13010 raft_consensus.cc:2764] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
W20260709 12:27:46.491465  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 166907us
I20260709 12:27:46.405001 13011 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:46.499467  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:46.499467  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:46.468720 12752 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.482843 12675 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:46.467236 13041 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:46.549180  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:46.571791  5629 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:46.548679  5628 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20. Ops: []
W20260709 12:27:46.518224  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:46.573774  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:46.547135  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 80227.8us
I20260709 12:27:46.552204  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:46.575133  5701 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:27:46.553776  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 96292.8us
I20260709 12:27:46.499699 12872 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:27:46.580456  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:46.580332  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:46.560657  5636 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 23. Current term is 24. Ops: []
I20260709 12:27:46.561254  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:46.560039  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:46.561952  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:46.564778  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 153808us
I20260709 12:27:46.565618 13124 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.559723  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:46.501031 13107 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:46.612159  5710 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
W20260709 12:27:46.494364 12786 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.589s	user 0.011s	sys 0.001s
I20260709 12:27:46.506596  5559 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23. Ops: [20.11-22.12]
I20260709 12:27:46.576367  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:46.615469  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:46.615792  5704 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:46.571692  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:46.618449  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:46.619724  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 43485.8us
I20260709 12:27:46.574369  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:46.582487  5630 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:27:46.583361  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:46.583563  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:46.583608  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:46.586740  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:46.620909  5702 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:46.621002  5628 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:46.500221 13137 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.088s
I20260709 12:27:46.621694  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:46.412971 12880 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.622325 12786 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.623926 12880 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:46.494994 12376 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.095s
I20260709 12:27:46.624377 12786 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.612147  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:46.548804 13128 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.610411  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:46.625847  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 154598us
I20260709 12:27:46.626222  5561 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:46.506788 12974 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:46.626183  5556 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
W20260709 12:27:46.557615  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52912.7us
I20260709 12:27:46.629141  5557 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:46.630237  5635 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:46.628598  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:46.635116  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:46.635265  5636 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:46.635800  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:46.607376 13107 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:46.637189  5629 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:27:46.646289  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:46.647892  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 78806.2us
I20260709 12:27:46.652123  5636 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:46.628715 12974 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:46.659844  5710 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:27:46.637910 13113 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:27:46.665768  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 53761.1us
I20260709 12:27:46.637809 13108 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:27:46.649626 12548 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.140s	user 0.013s	sys 0.002s
W20260709 12:27:46.584195 13134 consensus_queue.cc:1175] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cea66aa15be3423880843974acdbf50f" responder_term: 23 status { last_received { term: 23 index: 17 } last_committed_idx: 17 last_received_current_leader { term: 23 index: 17 } }
I20260709 12:27:46.668017  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:46.656809 13040 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:46.661926 13124 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:46.662035 13041 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:46.685225  5559 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:46.664979 12974 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:46.646040 12973 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
W20260709 12:27:46.686167  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 64570.5us
I20260709 12:27:46.689291  5701 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:46.643934 13128 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.694653  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:46.669552 12872 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 12:27:46.677583 12959 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:46.684916 13040 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:46.704974  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:46.706679  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:46.675369 12548 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.720702  5561 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:27:46.697347 13076 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:46.695930 13041 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:46.710310 13040 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:46.695758 12786 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
W20260709 12:27:46.737973 13147 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 12:27:46.737881 13041 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.740669 12872 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:46.742620 12548 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.745591 12991 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:46.748394 13146 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:27:46.750854 12921 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.755036 12773 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:46.756433 13142 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:46.757123 13142 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:46.758277 13142 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.762260  5636 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:46.722455 13076 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:46.763358  5628 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:46.763335  5630 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:27:46.784211  5630 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:46.789235 12921 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.790068 13109 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:46.790524 13109 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:46.791509 13109 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.794157  5704 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
W20260709 12:27:46.794421  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 104154us
I20260709 12:27:46.767096 13106 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:27:46.737905 13145 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:46.768177 13146 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:46.799045  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 105548us
I20260709 12:27:46.766571 13076 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:46.803103  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:46.807659  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 380757us
W20260709 12:27:46.803234 12913 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.897s	user 0.006s	sys 0.008s
I20260709 12:27:46.813258 12913 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.815690 12913 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:46.806064 13076 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:46.810009 13109 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:46.821250 12373 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:46.824587 13072 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.763s	user 0.005s	sys 0.007s
I20260709 12:27:46.825274 13072 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.827502 13072 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.827579 13146 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.828835  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:46.829146  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:46.830185  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:46.831415  6271 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:46.798022 13106 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:46.837188  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:46.838989  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 150857us
I20260709 12:27:46.840132  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:46.858654  5628 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:46.856050 12675 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:46.824025 12373 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:46.855849 12714 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:46.861198 13106 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:27:46.864483  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52276.7us
I20260709 12:27:46.991161  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:46.864542 13159 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.128s
I20260709 12:27:46.993566 13159 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:27:46.994053 13159 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:46.995168 13159 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.997279  6277 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:46.868763  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:46.999688  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 131003us
I20260709 12:27:46.874863 12714 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:47.001325 12714 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:46.875221  5560 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:47.002584  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:47.003393  6274 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:46.883944 13076 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.005678  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:47.006758  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:46.892118  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:47.009308  5628 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:46.896288  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:47.017024  5630 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
W20260709 12:27:46.903133  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:47.023398  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:46.904786  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:47.024442  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:47.024572  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:47.025360  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:47.025970  5709 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:46.909893 13128 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:47.028069  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:46.905833 13011 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.030211  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:46.915758  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:46.919373  5632 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:47.035097  5555 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:46.941232 13152 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:47.037626 13152 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:47.038672 13152 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:46.950493  5631 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:46.942297  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:47.041468  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 99230.3us
I20260709 12:27:46.952636 13107 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.043702 13107 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:47.043756 12675 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:47.044263 12675 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:47.044678 12675 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:47.045056 13137 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:27:46.974246  6236 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:27:47.047046  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 72890.7us
I20260709 12:27:47.047245  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:47.048012 12762 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:47.048542 12762 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:47.049431 12762 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.051343 13113 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:27:46.883875  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:46.892117 13039 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:27:47.051446  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:47.052908  5631 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:47.052712 12675 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.053862 13134 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:27:47.054806 12675 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:47.055069 13113 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:46.920881  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:47.056157  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:47.055526 13113 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:47.052310 13039 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:47.056584  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:47.057144 12998 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.057253  5706 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:47.057729 12998 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:46.971047  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:46.973202  5635 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:47.059213  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:46.976702 13008 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:47.006781 13076 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:46.938619 13106 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.942751  6237 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:47.059737  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:47.060945  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:47.061314  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 118650us
I20260709 12:27:46.943377 12880 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:47.062245 12880 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:47.062381 13106 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:47.063055 12786 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:47.063560 12786 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:47.063169 12880 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:46.975625  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:47.057511 13039 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:47.064368  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:47.064111 13067 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:27:47.064842  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:47.065061 13039 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:47.065337 13067 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:27:47.065814  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:47.066303 13109 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:47.060294 13008 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:47.066756 13109 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:46.973270  5702 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:27:47.067894 13109 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:46.942410 13154 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.066704 13113 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.069015 13154 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.069470 13109 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:47.071079 13113 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:47.071370 13154 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:47.032297 13011 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:47.072060 13076 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:27:47.032583  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:47.067718 13008 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.073939 13008 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:47.074224  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:47.074014 13106 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:47.074906  5635 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:47.075081 13106 consensus_queue.cc:1243] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 21.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:27:47.075891 13136 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:47.076329 13136 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:47.076637 13076 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:47.077049 13076 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:47.077787  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:47.077378 13136 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.077872 13076 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.079197  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:47.079505 13076 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:47.059789  5706 raft_consensus.cc:2376] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24. Re-sending same reply.
I20260709 12:27:47.025396  6277 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 15.
I20260709 12:27:47.064254 12880 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:47.091018  5631 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:46.944382 12974 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.092340  5704 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 18: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:47.063907 12786 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:47.094980 12974 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:47.097206  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:47.098623 13107 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:47.085271  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:47.066490  5707 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
W20260709 12:27:47.103195  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:47.104421  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72555.3us
W20260709 12:27:47.111300  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:46.863646  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 68308.3us
I20260709 12:27:47.118960  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:47.122346  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 171611us
I20260709 12:27:47.130338  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:47.040307 13152 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:47.139406  5709 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:47.140018  5635 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
W20260709 12:27:47.143932  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 458965us
I20260709 12:27:47.145388  6236 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:47.160020  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:47.162259  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:47.162930  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:47.165165  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:47.166060  5635 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:27:47.171880  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:47.175513  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:47.178042  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:47.059888 12998 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:47.181218  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:47.188570  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.188927  5704 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:47.189304  5556 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:27:47.183631  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:47.198751  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:47.199544  5601 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:47.200248  5556 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:47.147185  5628 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:47.129637 12373 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:47.135571 12847 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:27:47.213806  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 94958.5us
I20260709 12:27:47.130303 13145 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:27:47.129637 13128 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:47.199723  6278 raft_consensus.cc:2376] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21. Re-sending same reply.
I20260709 12:27:47.136174 13134 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.214226  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.138670 13011 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.138716 13107 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:47.216223  5599 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:47.217762  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.253957  5552 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 19: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:47.137253 12921 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:47.143793 13139 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:47.218492  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:47.226496  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:47.233767  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:47.234290  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:47.237376  5556 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:47.225494  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:47.244782  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.248665  5635 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:27:47.250988  5704 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:27:47.137351 12714 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.083s
I20260709 12:27:47.187989 13159 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:27:47.256186  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 78291us
I20260709 12:27:47.273851  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:47.267642  6276 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:47.145074 12911 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.093s
I20260709 12:27:47.203227 13067 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:27:47.272373  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:47.275382  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:47.144250 13040 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.266662 13132 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.202337 13158 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:47.137410 13010 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.125s
I20260709 12:27:47.273275  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:47.223052 12773 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:47.287854  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53684.3us
I20260709 12:27:47.277467 12957 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:27:47.288023  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 62634.5us
I20260709 12:27:47.314546  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:47.252766 12913 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:47.269086 12872 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:47.317911  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:47.324141  5556 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:27:47.259871 12847 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:47.261807 13142 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:47.326712  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:47.293138  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 66726.7us
I20260709 12:27:47.373483  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:47.348791  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 132645us
I20260709 12:27:47.352715  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:47.313373 12990 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:27:47.341502  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:47.354576  5553 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:47.381369  5702 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 20
W20260709 12:27:47.385797  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 68023.1us
I20260709 12:27:47.387457  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:47.251861 13134 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.294087 13158 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:47.331720  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.287406 13160 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:47.272389 12921 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.264953 13011 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:47.373224 13168 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.152s	user 0.000s	sys 0.002s
I20260709 12:27:47.406428  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:47.327092 12862 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:47.380934 12990 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:47.412020  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 212579us
I20260709 12:27:47.423713 12990 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:47.425055 12836 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:47.330318 12935 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:47.426074 12935 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:47.427014 12935 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.429286  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:47.425555 12836 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:47.430320 12836 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:47.431155 13011 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:47.372810 12773 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.431849 13011 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:47.325547 13067 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:47.447799  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:47.413333 13114 log.cc:927] Time spent T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:27:47.408962 13158 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:47.315848 13132 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:47.480661  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:47.503515  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:47.348208 13145 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.127s
I20260709 12:27:47.505336  5708 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 19: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:27:47.506841  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:47.505280  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:47.465824 13126 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:27:47.407939 13144 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:27:47.480773 13129 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:27:47.455955 12862 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:47.481242  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:47.519516  5557 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:47.517367  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:47.478884 12373 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:47.406469 12913 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.482302 13139 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:47.510658 13011 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:27:47.532933 13067 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.533463 12773 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:47.535172 12836 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.536505 12872 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:47.537734 12921 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:47.418279 13142 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.482676 13107 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.539669  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:47.540767 13133 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:27:47.543699  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:47.523639  5702 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:47.546277  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:47.547430  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 118196us
I20260709 12:27:47.550038  5560 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:47.514101 13144 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:47.588017 13144 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:47.590493 13136 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:47.550519  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:47.562369  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:47.594918  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 291482us
W20260709 12:27:47.562132  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 186761us
W20260709 12:27:47.567632  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 240292us
I20260709 12:27:47.565163 13011 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:47.543211 12786 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.539673 12913 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:47.598642 13008 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:47.599129 13008 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:47.593369 12880 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.506s	user 0.012s	sys 0.000s
I20260709 12:27:47.599318 13113 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:47.599730 13008 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:47.600853  5702 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:47.586835 12836 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:47.602859 13011 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.543246 12990 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.584142  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:47.608914  5561 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:27:47.639410  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:47.640091  5557 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
W20260709 12:27:47.609212  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58869.8us
I20260709 12:27:47.600013 13113 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:47.633738  5635 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:27:47.596365 13133 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:47.598505  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:47.659420  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 61016.8us
I20260709 12:27:47.661423  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:47.645995 13040 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:47.646821 13195 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:47.647478 13171 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.298s	user 0.000s	sys 0.001s
W20260709 12:27:47.599254 13109 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.529s	user 0.013s	sys 0.000s
I20260709 12:27:47.522064 12847 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.596975 13106 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:47.657011  5524 leader_election.cc:336] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.511s (SENT)
I20260709 12:27:47.687403  5706 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:47.677958  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.687777  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:47.652156 12786 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:47.686175  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.647799 12862 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.689065  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:47.689244  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:47.691563  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 318138us
I20260709 12:27:47.663304  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:47.693008  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:47.675346 13072 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:47.721568  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.685930 12959 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.688594 13183 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:27:47.722252  5557 raft_consensus.cc:2376] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26. Re-sending same reply.
W20260709 12:27:47.724411  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 61220.1us
I20260709 12:27:47.723937  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.726856  5556 raft_consensus.cc:2368] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19.
I20260709 12:27:47.723968  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:47.727730  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:47.729967  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:47.685719 12904 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.037s
I20260709 12:27:47.737957  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:47.736246  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:47.732621  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.765061  5560 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:47.767741  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.768265  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:47.768428  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:47.768653  5557 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:47.686671 13195 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.723383 13067 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:47.777549  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.725592 13160 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:27:47.695051 13106 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.677945 13139 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.685184 12373 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.764721 12786 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:47.731207 13134 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:47.784893  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:47.799181  6276 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:47.785776  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.731935 12935 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:47.787438 13181 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:47.801779  5635 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:47.787770 12990 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:47.788689 13199 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 12:27:47.789130 13158 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:47.791219 13190 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:27:47.767375 13107 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:47.794348 13106 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:47.762904 12959 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.797765  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:47.777174 13183 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:47.782953  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:47.800890  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.803774  5560 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:47.807904  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:47.809847  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:47.816012  5706 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:47.815609  6271 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:47.818323 13200 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:47.819765 12959 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:47.781267 13113 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.825217 13113 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:47.825534  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:47.826429  5557 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:47.818003  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:47.828547  5701 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
W20260709 12:27:47.833186  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 64847.8us
I20260709 12:27:47.834398  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:47.804383 13160 consensus_queue.cc:1243] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 18.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20260709 12:27:47.836545  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:47.778445 13133 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:47.871995  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 95813.6us
W20260709 12:27:47.873176  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:47.850593 13067 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.874951  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:47.865702 13192 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
W20260709 12:27:47.879662 13151 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.641s	user 0.000s	sys 0.003s
I20260709 12:27:47.880216 13202 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:47.880535 12974 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:47.880939 13201 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:47.851183 12935 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:47.873019 12762 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:27:47.903508 13130 consensus_queue.cc:1175] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f613ba929b1046d3b5aed909414a95bf" responder_term: 21 status { last_received { term: 21 index: 17 } last_committed_idx: 17 last_received_current_leader { term: 21 index: 17 } }
W20260709 12:27:47.904810 13191 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
W20260709 12:27:47.904853 13039 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.839s	user 0.013s	sys 0.000s
I20260709 12:27:47.909849  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:47.874569 12913 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 12:27:47.918610  5555 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: []
W20260709 12:27:47.932634  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:47.894029 12762 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:27:47.942404  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 51083.3us
I20260709 12:27:47.943463  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:47.884979 13067 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:47.951967  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:47.907956 12911 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 16.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 12:27:47.940140 12762 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:47.944434 13076 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:47.966070  5707 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:47.967952  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:47.893069 13202 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.954659 13200 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:47.974915  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:47.975950  6276 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:47.981673  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:47.965095 13195 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:47.984978 13159 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.704s	user 0.012s	sys 0.000s
I20260709 12:27:47.974059 13158 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:47.989454 13067 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:47.970623 12847 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:47.992380  5557 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 16. Current term is 17. Ops: []
I20260709 12:27:47.993788  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:47.994370  5557 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:47.996730  5557 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:27:47.964366 12935 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:47.998121  6280 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:27:47.972539 13203 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:47.964532 13106 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:27:48.005311  5706 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:48.021962  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.025424  5628 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:27:48.027086  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45513.8us
I20260709 12:27:48.011518 13076 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:48.030155  5557 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:48.033218  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.033867  5706 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:48.036413 13202 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:48.037314 13132 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:27:48.038403  5555 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
W20260709 12:27:48.042102  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:48.024688 13136 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:47.974315 13008 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.053855  6236 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:48.044064  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:48.058931  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.059530  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.059948  5706 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:48.060387  5710 raft_consensus.cc:2410] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 23 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:27:48.073642  6278 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:48.055821 13072 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:48.029088 13144 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.072088 13041 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.006s
I20260709 12:27:48.096164  6274 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:27:48.098889 13128 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:48.099332 13128 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:48.106611  5635 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:27:48.108322 13134 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:27:48.066666 13067 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:48.072949 13183 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:27:48.052862 13133 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:27:48.120141  5704 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: []
W20260709 12:27:48.121831 13211 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20260709 12:27:48.123530 13144 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:48.116582 13067 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.124389 13067 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:48.125469  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.126134  5707 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:48.127089  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.127648  5707 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:48.117650 13183 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.134850  5707 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
W20260709 12:27:48.136051  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 552639us
I20260709 12:27:48.136832  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:48.128247 13203 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:48.100181 13128 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:48.134316 13183 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:27:48.163800  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 506780us
I20260709 12:27:48.165030 13011 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:27:48.193992  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 57222.8us
I20260709 12:27:48.195266  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:48.196844 13011 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:48.197316 12880 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:48.184489 13203 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.197937 13185 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:48.169289 12921 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:48.204509 13128 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:48.205289  5635 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:48.220835  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3047) took 1037 ms. Trace:
W20260709 12:27:48.206544 13213 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:27:48.207532  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:48.224506  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:48.208426  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:48.210234  5629 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:27:48.215656  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:48.216341 13067 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.216925  5627 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:48.217453  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:48.218053  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:48.198164 13203 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.219157  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:48.219234  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:48.233027  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:48.237216  6279 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:48.221486  5554 rpcz_store.cc:276] 0709 12:27:47.183358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:47.184485 (+  1127us) service_pool.cc:224] Handling call
0709 12:27:48.162078 (+977593us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:27:48.162298 (+   220us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:48.162521 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:27:48.162839 (+   318us) log.cc:844] Serialized 14 byte log entry
0709 12:27:48.220507 (+ 57668us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:48.220573 (+    66us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:48.220616 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:48.220682 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:48.220785 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":5019,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":2223,"spinlock_wait_cycles":3501440,"thread_start_us":57183,"threads_started":1}
I20260709 12:27:48.222461  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:48.214166 13219 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.058s	user 0.002s	sys 0.000s
I20260709 12:27:48.232043  6275 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 21: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:48.197363 13011 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:48.201886 12880 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:48.239228  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 74731.4us
I20260709 12:27:48.218655 13124 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.246304 13124 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.241164  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:48.184103 12762 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:48.244565  5632 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:48.245252  6273 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: [21.15-21.15]
W20260709 12:27:48.246641  5524 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.562s (SENT)
I20260709 12:27:48.249119  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:48.262332  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.198526 13185 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:48.263885  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:48.265053  5707 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:48.249022 12762 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:48.249904 13145 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.031s
I20260709 12:27:48.272858  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:48.223237 12921 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:48.292266  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:48.282205  6272 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:48.293140  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:48.296262  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:48.296636  5707 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:27:48.298323 13113 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.298621 13197 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:48.299166 13197 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:48.301635 12921 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:48.300524 13197 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 12, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.251269 12880 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:48.308168  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3043) took 1126 ms. Trace:
I20260709 12:27:48.254312 13040 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:48.303262  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:48.240847 13008 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:48.298940 13113 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.309073  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.302297 12921 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:27:48.313736  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:48.308653  5553 rpcz_store.cc:276] 0709 12:27:47.182230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:47.184667 (+  2437us) service_pool.cc:224] Handling call
0709 12:27:47.224331 (+ 39664us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:47.382455 (+158124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:48.307932 (+925477us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b540023f354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:48.308110 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":298436992}
I20260709 12:27:48.303996  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:48.249068 13124 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:48.317788  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:48.278357 13203 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:48.334599  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.334967  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.335387  5707 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:48.336586  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:48.337997 13186 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:48.338764  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.339493  5707 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:48.339874 13067 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:48.336217  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.341005  5710 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:48.343122  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.344846  6278 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:48.345824 13113 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:48.345991 13203 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:48.346632 13203 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:48.274652 13152 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.124s	user 0.009s	sys 0.004s
I20260709 12:27:48.327291  6272 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:48.348986  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.352588  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:48.354936  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:48.355683  5630 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:48.357044  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:48.357827  5635 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:48.360888  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.362004  5710 raft_consensus.cc:2410] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:27:48.364753  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.366153  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.367208  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.367908  5707 raft_consensus.cc:2410] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:27:48.368654  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.369748  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.370672  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.371295  5707 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:48.371889  6278 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:27:48.371970  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.374599  5707 raft_consensus.cc:2368] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22.
I20260709 12:27:48.375865  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.272318 13011 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:48.354568 13186 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.020s
W20260709 12:27:48.384517 13189 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:27:48.386026  5631 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:48.389204 13185 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:48.390055 13203 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:48.390638  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.393455 13160 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:48.397512  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3046) took 1214 ms. Trace:
I20260709 12:27:48.410434  5558 rpcz_store.cc:276] 0709 12:27:47.183073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:47.202337 (+ 19264us) service_pool.cc:224] Handling call
0709 12:27:47.238965 (+ 36628us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:47.261388 (+ 22423us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:47.261435 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:47.261499 (+    64us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:47.261558 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:48.397274 (+1135716us) spinlock_profiling.cc:243] Waited 22.1 ms on lock 0x7b540044f554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:48.397318 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:48.397451 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50833792}
I20260709 12:27:48.381747  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:48.419035  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:48.419605  5630 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:48.410418 13186 consensus_queue.cc:1243] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 22.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:27:48.410300 13160 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:48.417157 12973 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.447505 12973 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.447827 13160 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:48.449179 13160 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:48.449834 12973 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:48.456836  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.457365  6275 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:48.459942  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1320) took 1217 ms. Trace:
W20260709 12:27:48.460206  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:48.460413  5705 rpcz_store.cc:276] 0709 12:27:47.242285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:47.522267 (+279982us) service_pool.cc:224] Handling call
0709 12:27:47.530348 (+  8081us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:47.539990 (+  9642us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:47.540023 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:47.540067 (+    44us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:47.540100 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:48.459727 (+919627us) spinlock_profiling.cc:243] Waited 9.43 ms on lock 0x7b5400075654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:48.459774 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:48.459886 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21697280}
I20260709 12:27:48.462034  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.463168  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.463379  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.431020  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:48.435189 13142 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:48.441964 12847 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:48.445556 12913 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:27:48.428390 13183 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.464179  5707 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:48.467108  5705 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:27:48.468343  5710 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:48.470705  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:48.481968  6278 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:48.484613  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.484583 13145 consensus_queue.cc:1243] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 20.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:27:48.464130 13152 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.010s
I20260709 12:27:48.471045 12847 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:48.508750  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.509260  6279 raft_consensus.cc:2368] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
I20260709 12:27:48.513336  5710 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 12:27:48.510192  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:48.471280 12913 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 16 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:48.510192  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 263562us
I20260709 12:27:48.514348  6278 raft_consensus.cc:2368] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
W20260709 12:27:48.515744  5524 leader_election.cc:336] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.679s (SENT)
I20260709 12:27:48.515880  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.516233  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:48.530012 13183 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:48.516587  5705 raft_consensus.cc:2376] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18. Re-sending same reply.
I20260709 12:27:48.531824 12847 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:48.547307 12847 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:48.569299  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:27:48.572268  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:48.573063  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:48.573148  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:48.571607  6278 raft_consensus.cc:1217] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Deduplicated request from leader. Original: 15.12->[16.13-19.14]   Dedup: 16.13->[19.14-19.14]
I20260709 12:27:48.574170  6275 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:48.574304  6278 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: [16.13-19.14]
I20260709 12:27:48.580721  6277 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:27:48.580852  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1316) took 1339 ms. Trace:
I20260709 12:27:48.581359  5708 rpcz_store.cc:276] 0709 12:27:47.240994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:47.241151 (+   157us) service_pool.cc:224] Handling call
0709 12:27:48.580612 (+1339461us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b5400210554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:48.580758 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":414461184}
I20260709 12:27:48.582610  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.583381  5708 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 18: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:48.584009  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:48.585022 13193 raft_consensus.cc:2764] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:27:48.586229 12753 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:48.588503 13231 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.589112 13231 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:48.591069  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.591754  5703 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:48.593174  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:48.594269 13128 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:48.594691 13128 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:48.595111 13128 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:48.597254 13205 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:48.597776 13205 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:48.599018 13205 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.584142  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.600524  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:48.601023  5708 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
W20260709 12:27:48.602322  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 86618.6us
I20260709 12:27:48.602985  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1319) took 1361 ms. Trace:
I20260709 12:27:48.603467  5701 rpcz_store.cc:276] 0709 12:27:47.241986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:47.433518 (+191532us) service_pool.cc:224] Handling call
0709 12:27:48.602766 (+1169248us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b5400323054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:48.602915 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":798194048}
W20260709 12:27:48.605194  5524 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.983s (SENT)
I20260709 12:27:48.606755 13197 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:48.608528 12921 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:48.608913 13205 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:48.603266  5703 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:48.610756  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.611352  5703 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:27:48.613426  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.615120  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.612282  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.618538  5703 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:48.619580  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.620826  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:48.621475  5703 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:48.591370 13231 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:48.624678  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:48.626168  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:48.627015 12880 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:48.627316  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:48.627813 12880 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:48.628235 12880 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.628618 12880 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:48.628523 13145 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:48.629681 13145 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:48.630435 13145 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:48.630801 13145 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.631183 13145 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:48.632642 13142 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:48.633085 13142 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:48.633322  6280 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1302) took 1449 ms (client timeout 1693 ms). Trace:
W20260709 12:27:48.633735  6280 rpcz_store.cc:269] 0709 12:27:47.191603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:47.197753 (+  6150us) service_pool.cc:224] Handling call
0709 12:27:48.633108 (+1435355us) spinlock_profiling.cc:243] Waited 723 ms on lock 0x7b54004f9cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:48.633230 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1662365440}
I20260709 12:27:48.633998 13142 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.635380 13142 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:48.637156 13188 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:48.637574 13188 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:48.638468  5703 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:27:48.638351 13188 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:48.639595 13188 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:48.641794  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:48.642792  6236 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:48.642978 13183 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:48.643595 13183 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.643893 12786 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:48.646870 12786 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:48.657414 12786 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.661984  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:48.664487  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57365.4us
I20260709 12:27:48.664856  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3167) took 1030 ms. Trace:
I20260709 12:27:48.614058  5701 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:48.644063 13183 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:27:48.648968  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:48.650087 13226 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:48.652623 13214 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.653362  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:48.615789  5710 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:48.682549  5703 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 17. Current term is 18. Ops: [17.15-17.15]
I20260709 12:27:48.655259  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:48.632608 13231 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.012s
I20260709 12:27:48.692596  5561 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:27:48.692714  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 69122.6us
I20260709 12:27:48.693205  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:48.693835  5553 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:48.676281  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:48.694886  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:48.695497  5552 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:27:48.665256  5704 rpcz_store.cc:276] 0709 12:27:47.634404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:47.698367 (+ 63963us) service_pool.cc:224] Handling call
0709 12:27:47.698677 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:48.120593 (+421916us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:48.664621 (+544028us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b5400066154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:48.664798 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":917003520}
I20260709 12:27:48.696892  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.698158  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:48.698715  5704 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:27:48.684051  5703 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:27:48.690548  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:48.705492  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:48.706300  5704 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:27:48.707849  5704 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:27:48.708755  5561 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:27:48.683619  5633 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
W20260709 12:27:48.710103  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:48.710422  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:48.711632  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:48.712342  5556 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:27:48.716842  5635 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:27:48.663282 13122 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:48.821502 13144 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:48.821938 13122 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:48.822455  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:48.718639  5554 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:27:48.823216  5557 raft_consensus.cc:2368] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:48.822845 13122 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:48.823931 13122 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:48.678085 13226 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:48.825101  5554 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:48.825569 13226 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:48.826974 13226 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:48.682560 12921 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:27:48.828912  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 118599us
I20260709 12:27:48.829336 12921 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:48.830173  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:48.830408 12921 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.831602 13072 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:48.832605 13041 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:48.832155 13072 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:48.834204 13145 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.667845 13205 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:48.840649 13205 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:48.842149  5559 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:27:48.843492 13205 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:48.709514 12973 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:48.716012 13203 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.130s
I20260709 12:27:48.692041 13137 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.125s
W20260709 12:27:48.818087  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 141925us
I20260709 12:27:48.846477  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:48.821658  5560 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:48.834342 13067 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:48.734483  5703 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:27:48.834461 13072 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.708564 13214 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:48.835078  5524 leader_election.cc:336] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.249s (SENT)
W20260709 12:27:48.854591  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:48.855508 13214 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:48.855736 13010 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.837872 13011 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:48.839111 13107 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:27:48.856606 13011 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:48.857123  5703 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:27:48.858480  5703 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:27:48.836158 13133 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.876664  5710 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:48.889350  5702 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:48.879518  5553 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:48.833132  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:48.896903  5703 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:27:48.898051  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3172) took 1023 ms. Trace:
W20260709 12:27:48.899286  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 193865us
I20260709 12:27:48.900285  5707 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: [21.15-21.15]
W20260709 12:27:48.904310  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 71261.1us
I20260709 12:27:48.801805  5526 leader_election.cc:400] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
I20260709 12:27:48.886788  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:48.856289 13010 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:48.907894  5554 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:27:48.909360  5526 leader_election.cc:403] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:27:48.910710  5676 leader_election.cc:400] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:48.911347  5676 leader_election.cc:403] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
W20260709 12:27:48.919242  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 318650us
I20260709 12:27:48.883919  5556 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
W20260709 12:27:48.929605  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60124.7us
I20260709 12:27:48.925987  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:27:48.940318  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 274922us
I20260709 12:27:48.912124 13107 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:48.955513  5701 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:27:48.947518 12973 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:48.945674 13231 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
I20260709 12:27:48.996379  5524 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:48.999671  5524 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:48.888208 13011 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.945376 13067 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.044s
I20260709 12:27:48.912465 13133 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:49.001587  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:49.013659  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:48.980798 13160 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:49.031991 13011 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:49.037580 13145 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:49.038796 13231 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:27:49.044195  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:49.046409  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 103726us
I20260709 12:27:49.046550  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:49.011673 13232 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:49.014587 13124 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:49.013224 13067 consensus_queue.cc:1243] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 19.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
W20260709 12:27:49.092083  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:49.101203  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 76914.6us
W20260709 12:27:49.103011  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54897.4us
I20260709 12:27:49.092314  6279 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:49.105518  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:49.111363  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:49.104714 13220 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.084s	user 0.000s	sys 0.006s
I20260709 12:27:49.130426  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:49.105938 13144 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:49.113284 13197 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.506s	user 0.010s	sys 0.000s
I20260709 12:27:49.132865  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:49.114594 12862 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:27:49.134228  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 208329us
I20260709 12:27:49.136549  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:49.139325  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3059) took 1969 ms. Trace:
I20260709 12:27:49.120231 13232 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.139865  5631 rpcz_store.cc:276] 0709 12:27:47.170372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:47.170511 (+   139us) service_pool.cc:224] Handling call
0709 12:27:47.217319 (+ 46808us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:48.406123 (+1188804us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:49.139011 (+732888us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54001f2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:49.139244 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2687306496}
I20260709 12:27:49.108105 13109 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.027s
I20260709 12:27:49.133910 13124 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:27:49.153317 13238 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:27:49.154385 13142 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.154881 13142 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.156430 13010 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:49.157207 13142 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:49.158622 13133 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:49.145641 12862 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:49.161176 12862 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.162415 12913 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.7, Last appended by leader: 7, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.163558 12913 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:49.165633 13128 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.570s	user 0.011s	sys 0.000s
I20260709 12:27:49.166060 13128 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:48.898639  5706 rpcz_store.cc:276] 0709 12:27:47.874585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:48.625456 (+750871us) service_pool.cc:224] Handling call
0709 12:27:48.625772 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:48.625984 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:27:48.626275 (+   291us) log.cc:844] Serialized 14 byte log entry
0709 12:27:48.897566 (+271291us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:48.897648 (+    82us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:48.897713 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:48.897809 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:48.897967 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":555,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":109,"thread_start_us":270780,"threads_started":1}
I20260709 12:27:49.176565  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:49.176954 13245 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
W20260709 12:27:49.177233 13240 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:27:49.177425  5706 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:49.177942  5557 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:49.178850  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:49.178594 13144 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.179783 13041 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:49.180166 13041 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.126233 12880 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.180513 13041 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:49.181192 13040 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:49.181641 13040 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:49.182297 12880 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:49.141059 13145 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:27:49.184121  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:49.184764  6274 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:49.191257  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:49.148439  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:49.135351 13231 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:49.201973  5554 raft_consensus.cc:2410] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 23 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:27:49.196336  5701 pending_rounds.cc:85] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 14
I20260709 12:27:49.202582  5701 pending_rounds.cc:107] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260709 12:27:49.202870  5701 raft_consensus.cc:2889] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:49.193776  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:49.205643  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:49.205909 13186 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:49.206351 13186 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.206709 13186 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:49.206845 13250 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.207479 13250 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.208518  5553 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:27:49.168146 13128 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:49.217358  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:49.201036  6274 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:49.222798  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:49.223688 12880 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:49.212183  6279 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:49.220130  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:49.190265 13183 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.224222 12880 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:49.192848 13197 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.201535 13160 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:49.201049 13145 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.232508  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:49.234946  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:49.238651  5558 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:27:49.236249  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:49.244501 13145 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.249326 13250 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:49.250272 13145 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:49.234259 13197 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.251421 12973 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:49.252776 13197 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:49.254122  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:49.255406  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:49.255599  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:49.257092  5559 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:49.257218  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:49.258798 13250 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:27:49.259197 13250 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.259526 13250 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:27:49.260531  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 551459us
I20260709 12:27:49.260782 13186 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.261257  5601 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:49.265062 12973 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:49.265489 12973 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.265836 12973 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:49.145844 13232 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:49.265986  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:49.267740 13232 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:49.215862 13231 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:49.269586 13183 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:49.269599 12847 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:27:49.270314 13231 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.270748 13214 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:27:49.271026 12847 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:49.271449 12847 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:49.272038  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:49.272312 12847 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.273309  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:49.284646  5706 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:49.273473 13067 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.002s
I20260709 12:27:49.291797  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.292734  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.293449  5635 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:49.295022  5630 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:49.274960 13182 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:27:49.276279 13137 consensus_queue.cc:1243] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 17.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260709 12:27:49.276767  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 430343us
I20260709 12:27:49.277330 12762 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:49.278309 12935 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.245061 13160 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:49.278777 13193 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:49.316817 13193 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.316963 13250 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.318498 12847 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:49.319233 13193 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:49.319307 13205 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:27:49.279217  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.262010 12753 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:49.322295 12753 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.322777 12753 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:49.324838  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:49.282289  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 145823us
I20260709 12:27:49.282734  5633 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:49.285916  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:49.238843 13124 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:49.307585  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.302855 12762 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:49.347451  5632 raft_consensus.cc:2368] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20.
I20260709 12:27:49.267779 12880 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.252071 13041 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.315685 13160 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.281420  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:49.333832  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:49.286450 12862 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:49.343052  5561 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:27:49.344192  5675 leader_election.cc:400] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
W20260709 12:27:49.392910  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 52022.6us
I20260709 12:27:49.311970 12935 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.394027  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:49.364869  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.396550  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:49.269614 13040 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.359824  5706 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:49.362983  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:49.297295 13181 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.380393 13232 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:49.381083 13124 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:49.362330 12762 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:49.383137 13137 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:49.385295 13205 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:27:49.387014  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:49.388513 13041 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:49.286360 13159 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.063s
I20260709 12:27:49.403244  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:49.309376 13182 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:49.350462 13193 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:27:49.442586  6274 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:27:49.442523  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:49.443979  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:49.444460  5635 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:49.444998  6280 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 20: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:49.454455  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:49.401810 12847 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:49.413079 13181 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.412050 12957 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 12:27:49.466048 13252 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.453992 13193 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:49.409417 13011 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260709 12:27:49.491153 13264 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:49.485255  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.453459 13128 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:49.499734  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.501703 12935 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:49.502136 13181 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:49.471356 12847 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:49.509276 13124 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.512629 13186 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:49.482793 13107 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.033s
I20260709 12:27:49.470708 13250 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:49.515527  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 128857us
I20260709 12:27:49.445721 12973 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:49.533751  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 238201us
I20260709 12:27:49.552124 13226 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:49.542739  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:49.574245  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.418716  5561 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:27:49.550675  5629 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:27:49.579797  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:49.413341 13232 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.498425 13145 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.556200  5633 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:27:49.516597 12957 consensus_queue.cc:1243] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:27:49.565341  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.566479  5635 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:27:49.569164  5630 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:49.566201  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:49.422940  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:49.573266  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.588166  5702 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:49.574244  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.574936  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.590497  5703 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:27:49.498931 13139 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.079s
I20260709 12:27:49.580945  6277 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:49.556173 12847 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:49.597142  5707 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
W20260709 12:27:49.603096  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:49.604143  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:49.560492 13128 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:49.604848  6274 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:27:49.541271 12753 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:49.613451  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.620131  5557 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 15. (term mismatch)
I20260709 12:27:49.620617  5557 pending_rounds.cc:85] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 14
I20260709 12:27:49.620867  5557 pending_rounds.cc:107] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 21.15
I20260709 12:27:49.621112  5557 raft_consensus.cc:2889] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:49.615938  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:49.544418 13252 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.599452 13130 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:49.635913  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.638343  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.639113  5707 raft_consensus.cc:2368] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27.
I20260709 12:27:49.639954  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.640707  5702 raft_consensus.cc:2368] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
I20260709 12:27:49.642277  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.643626  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.643364  5702 raft_consensus.cc:2376] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19. Re-sending same reply.
I20260709 12:27:49.644533  5710 raft_consensus.cc:2410] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:27:49.645135  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.646247  5702 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:49.646938  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.647693  5707 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:49.653215  5524 leader_election.cc:400] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27.
I20260709 12:27:49.653604  5524 leader_election.cc:403] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:27:49.625486  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:49.602174 12862 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:49.622309 13266 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:49.663935 13157 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:49.754310 13266 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.754554 13157 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.625798 13144 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.756870 13144 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.757452 13157 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:49.575766 13182 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.758229 13040 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:49.759050 13144 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:49.650480 13250 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:49.760557 13265 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 12:27:49.720027 13247 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:49.650878 13214 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.761673 13247 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.761940 13214 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.763792 13247 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:49.763876 13214 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:49.664988  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:49.673326 13128 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.608230 13193 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.601759 13145 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.810221 13266 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:49.717069 13235 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:27:49.677863 13226 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:49.811197 13226 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:49.678740 12753 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:27:49.680126  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.681758  5634 raft_consensus.cc:2368] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23.
W20260709 12:27:49.688136  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 62726.3us
W20260709 12:27:49.680929  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 69614.5us
I20260709 12:27:49.621737 13139 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.693146  5524 leader_election.cc:400] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
W20260709 12:27:49.819832  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 126856us
I20260709 12:27:49.691947  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.823045  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.824046 13157 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.824501  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:49.824631  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:49.825129  5636 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:49.715237  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.829689  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:49.830245  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:49.831208  5561 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:49.833379  5703 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:27:49.834951  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:49.835762  5703 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:49.718722  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.688143  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.838970  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.839800  5701 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:49.694646 13122 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:27:49.697167 12957 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:27:49.719732  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:49.840854  5635 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:49.732569  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:49.841814  5629 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:27:49.734655 13188 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:49.734540  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.735528  5630 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:27:49.740693 13267 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:27:49.741940 13272 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.728509 13130 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:49.832533 13214 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.747747 13181 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:49.753091 12862 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.760854 13250 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.671360 13203 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:27:49.768795 13243 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
W20260709 12:27:49.769313 13258 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:27:49.714743 13142 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27.
W20260709 12:27:49.765616 13259 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.251s	user 0.000s	sys 0.002s
I20260709 12:27:49.806864  5705 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 12. (term mismatch)
I20260709 12:27:49.807754  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:49.969456  5705 pending_rounds.cc:85] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 11
I20260709 12:27:49.810766 13145 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:49.812754  5706 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:49.816913  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:49.817180  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:49.817305  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 84022.3us
I20260709 12:27:49.883530 13113 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:49.969767  6279 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:49.818746  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:49.837704  5710 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:27:49.844074  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:49.840778  5627 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:49.311543  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:49.845469  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:49.974114  5701 raft_consensus.cc:2376] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18. Re-sending same reply.
I20260709 12:27:49.975240  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:49.864053 13280 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.875535  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:49.744694 13270 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:49.882642  5553 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:49.832789 13011 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.892143  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:49.896706  5703 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:27:49.840463 12880 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.845237 13186 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:50.040226  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:50.042235  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.840440 13231 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:49.832772 13139 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.902598  5636 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:27:49.903873  5629 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:49.904944  5635 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:27:49.905766  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:49.906368  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:49.828012 12921 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
W20260709 12:27:49.908701  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:50.078496  5553 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:27:50.066880  6279 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
W20260709 12:27:50.079514  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:49.824661 13157 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:49.988365 13133 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:49.912041  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.877609 13122 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:50.091907  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:49.893286 12714 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.818149  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:50.097772  5628 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:27:50.099239 13157 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:49.969929  5705 pending_rounds.cc:107] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:27:50.100543  5705 raft_consensus.cc:2889] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:49.984472  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:50.101737  5702 raft_consensus.cc:2410] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 20 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:27:50.044198 12880 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.102918  5702 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:27:49.968410 12753 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.104246 12880 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:50.104890 12714 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.890561 13181 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:50.026613 13280 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:50.107250 13181 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:50.108532 13181 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:50.108743 12762 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.663s	user 0.005s	sys 0.007s
I20260709 12:27:50.109212 12762 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:49.900440 13214 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:50.110739  5635 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260709 12:27:49.973337  5675 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.503s (SENT) [suppressed 24 similar messages]
I20260709 12:27:50.096834 13133 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:50.005069  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 161132us
I20260709 12:27:49.995954  5701 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:50.112864  5632 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:50.027485  5558 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:49.907662 13203 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:49.986423 12957 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:50.078020  5709 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:49.886241 13272 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.085994  5706 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:50.028658 13113 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:50.049467 13249 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 12:27:50.050570  5633 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:50.145859  5633 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:50.088953  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:50.146731  5636 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:49.911201  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:50.004974 13270 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:49.885842 13197 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.034798 13276 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.162704  5634 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 16. Current term is 17. Ops: []
W20260709 12:27:50.095182  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58559.8us
I20260709 12:27:50.034709 13182 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:50.096338  5710 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 11. (term mismatch)
I20260709 12:27:50.092896  5630 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:50.107172 12714 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.111159 12762 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.112818  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:49.911372 13107 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:49.906915 13161 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.104197  5702 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:27:50.104159  5707 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:27:50.114459  5629 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
W20260709 12:27:50.091907 13243 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:27:50.139716  5706 pending_rounds.cc:85] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 9
I20260709 12:27:50.179284  5706 pending_rounds.cc:107] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 18.10
I20260709 12:27:50.179721  5706 raft_consensus.cc:2889] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:50.141840  5631 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:50.170548 13122 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:50.090108 13232 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.037431 13011 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:50.147812  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 103986us
I20260709 12:27:50.149808  5555 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:50.149878  5628 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:27:50.165822  6280 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:27:50.143108 12957 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:50.168536  5710 pending_rounds.cc:85] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 10
I20260709 12:27:50.141255 12990 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:50.166913 13197 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:50.089823 13231 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:50.181720 13107 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.176164  5709 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:27:50.204598  5710 pending_rounds.cc:107] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 12:27:50.209779  5710 raft_consensus.cc:2889] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:27:50.181679  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 67486.7us
I20260709 12:27:50.210577  5703 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:50.078519 12973 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:50.138556 12753 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.185132  5558 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:27:50.083146 13130 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.188475 13113 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:50.189232 13277 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 12:27:50.206880 13231 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.227674 12935 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:27:50.228042  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:50.206259 13197 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:50.228840  5561 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 22: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:50.230899 12753 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:50.232091  5635 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:27:50.189582 13011 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.182237 13203 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:50.191696 13274 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.510s	user 0.009s	sys 0.000s
I20260709 12:27:50.186009 13287 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.008s
I20260709 12:27:50.193496 13139 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:50.200311 13282 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:27:50.198758 13226 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:50.198616 13270 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.203923 13250 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:50.204394 13214 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:50.266817 13270 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:50.205706 13133 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.201248 13272 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.169241 13276 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.168490 13182 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:50.268538 13182 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:50.177392  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:27:50.269832  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 92509.1us
I20260709 12:27:50.204867 12990 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.272648 13267 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:27:50.273520 13267 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:50.184329 13161 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.274531 13267 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.277678 13008 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:27:50.280653 13214 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.281242 13214 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:50.283344 13214 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:50.285704 13276 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:50.208511 13271 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:50.286849 13271 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.289013 13271 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:50.289779 13203 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:50.290516 13280 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:50.299573  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:50.300380  5554 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:50.300743  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:50.301461  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:50.195973 13266 raft_consensus.cc:2764] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:27:50.302088  5553 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:50.303296  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:50.305064  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:50.305742  5557 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:50.306731  5633 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:27:50.307255  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:50.308605  5561 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:27:50.306190 13272 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.315205  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:50.316145  5561 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:50.317802  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:50.318547  5556 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:50.321360  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:50.280316 12847 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.665s	user 0.009s	sys 0.000s
I20260709 12:27:50.309791 13011 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.325273 13107 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.314065 13272 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.313971 12957 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:50.357831  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 384497us
I20260709 12:27:50.331344 13011 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.325757 12847 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.372700 13252 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:50.375241 13011 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:50.382416 13252 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:27:50.393817 13161 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.367502 13280 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.398245 13280 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:50.398535  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3067) took 1143 ms. Trace:
I20260709 12:27:50.406170  5561 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:27:50.416428  5704 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:50.432181  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:50.399380  5552 rpcz_store.cc:276] 0709 12:27:49.255002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:49.255131 (+   129us) service_pool.cc:224] Handling call
0709 12:27:50.398258 (+1143127us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54003a93d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:50.398450 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2553474432}
I20260709 12:27:50.435209  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:50.428862  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:50.445614  5707 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 22: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:50.446326  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:50.433168  5702 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:50.447445  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:50.448127  5709 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:50.450428  5702 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:27:50.451511  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:50.423774  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:50.458348  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:50.458614  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:50.459306  5557 raft_consensus.cc:2376] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22. Re-sending same reply.
I20260709 12:27:50.460264  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:50.461297  5701 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:50.463111  6277 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:27:50.455529  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:50.475541  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:50.476969  6280 raft_consensus.cc:2410] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 21 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:27:50.475528  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:50.478650  5703 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:50.478350  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:50.479460  5552 raft_consensus.cc:2376] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19. Re-sending same reply.
I20260709 12:27:50.479513  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:50.480322  5703 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:27:50.482756  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:50.482960  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:50.449481 13144 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:50.483498  5710 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:50.484859  5710 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:27:50.485977  5710 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:50.489135  5552 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:27:50.489135  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:50.489624  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:50.490108  5557 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 18: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:50.492789 13272 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.445101 13264 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.496776 13264 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.498920 13264 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:50.499517 12762 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:27:50.500633  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:50.501761  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:50.503062  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:50.503679  5703 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:50.505652  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:50.505733  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:50.507791  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:50.444912 13122 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:50.510404  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:50.508587  5701 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:27:50.509523  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:50.511725  5602 connection.cc:256] RPC call timeout handler was delayed by -1.1995251020002797s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15500000000000003s for a response
I20260709 12:27:50.511205  5703 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:50.456883 13214 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:50.511874  5705 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:50.468788 13137 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:27:50.504953  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:50.518752  5702 raft_consensus.cc:2410] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:27:50.519425  5557 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:50.516654  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:50.524753  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:50.521559  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:50.526208  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:50.526239  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:50.528193  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:50.507376 13109 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.506882 13072 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.016s
I20260709 12:27:50.513252 13113 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:50.530804  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:50.535979  5702 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:27:50.537609  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:50.517026 13124 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:50.527766 13188 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:50.545946  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:50.545928  5709 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:50.548403  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:50.528681 13264 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.524127 13280 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:50.509018 13144 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:50.528605 13226 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.542128 13109 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.583034 13109 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.541801 13231 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:50.586874  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:50.587154 13285 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 12:27:50.518133 13122 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:50.589252 12847 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:50.513659 13159 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 12:27:50.596925 12957 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:50.532892 13142 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:50.598186 13142 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:50.589159  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:50.599119 13142 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.532776 13272 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:50.600602 13272 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:50.600958  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:50.591369 13182 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:50.601612  5630 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:50.560009 13252 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:27:50.523476 13067 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.030s
I20260709 12:27:50.603623  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:50.541836 13113 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:50.566622 13107 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.007s
I20260709 12:27:50.604463 13183 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:50.604662 13144 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:50.604967 13183 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:50.567875 13250 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:50.605386 13183 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:50.605667 13250 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:50.606127 13250 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:50.606276 12990 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.569411 13264 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:50.601230 13272 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:50.604712 13113 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:50.611697  5703 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:50.544118 13137 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:50.529461 13011 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:50.612321  5703 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:50.613441 13137 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:50.618348 13183 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.621507  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:50.622082  5630 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:50.551558 13188 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:50.624890 13188 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:50.627628  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3188) took 1054 ms. Trace:
I20260709 12:27:50.628075  5704 rpcz_store.cc:276] 0709 12:27:49.573262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:49.574020 (+   758us) service_pool.cc:224] Handling call
0709 12:27:50.627401 (+1053381us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b540006f754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:50.627553 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1899890560}
I20260709 12:27:50.524788 13214 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.623885  5703 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.634276  5703 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:50.637362  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 111224us
W20260709 12:27:50.630879  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 75512.7us
I20260709 12:27:50.638392  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:50.641637 12935 raft_consensus.cc:2764] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election decision vote started in defunct term 26: won
W20260709 12:27:50.642405  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 111838us
I20260709 12:27:50.569667  5709 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
W20260709 12:27:50.644330  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:50.644954  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:50.645496  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:50.645412  5630 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
W20260709 12:27:50.644867  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 98084.1us
W20260709 12:27:50.648381  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:50.648416  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:50.649595  5707 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:50.651073  6277 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:27:50.654067  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:50.666458  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:50.668849  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:50.669900  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:50.670598  6274 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:50.685799  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 139928us
I20260709 12:27:50.689905  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:50.699311  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:50.665314  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:50.632026 13182 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:50.761014  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 122682us
I20260709 12:27:50.673506 13144 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:50.755926 12847 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:50.779183 13144 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:50.762912  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:50.780094  6273 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:27:50.671422 13010 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.115s
I20260709 12:27:50.780653 13010 consensus_queue.cc:1243] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 20.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:27:50.773114  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:50.773459  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:50.780411  6277 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:50.782116  5628 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 12. (term mismatch)
I20260709 12:27:50.781962  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:50.783435  5628 pending_rounds.cc:85] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 11
I20260709 12:27:50.701359  5703 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260709 12:27:50.783799  5628 pending_rounds.cc:107] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260709 12:27:50.778559  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:50.785157  6275 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:27:50.718353  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59958.5us
I20260709 12:27:50.784207  5628 raft_consensus.cc:2889] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:50.787870 13193 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:27:50.775338  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:50.662251 13122 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:50.784121  5703 pending_rounds.cc:85] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 14
I20260709 12:27:50.813439  5703 pending_rounds.cc:107] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 19.15
W20260709 12:27:50.813627  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 126251us
I20260709 12:27:50.771589  5702 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:27:50.819823  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:50.812203  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:50.658677 13264 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.664484 13285 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.094s
W20260709 12:27:50.806797  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 141599us
I20260709 12:27:50.765404  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:50.763854  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:50.834165  5636 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:50.813764  5703 raft_consensus.cc:2889] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:50.826948  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:50.780342 13144 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:50.803974  5706 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
W20260709 12:27:50.842649  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55191.6us
I20260709 12:27:50.844161  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:50.755075 13186 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:50.658632 12753 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.165s
I20260709 12:27:50.806285 13288 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:50.824424 13231 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:50.825253 12862 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.870425  6274 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:50.835736 13280 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:27:50.877343  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:50.824993 13260 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:50.807974 13010 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.051s
I20260709 12:27:50.806102 13226 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.059s
I20260709 12:27:50.847999 13276 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:50.877175 13144 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:50.875996 13231 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:50.923174 13183 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.932958 13183 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:50.934294  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:50.935297  5706 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:50.937894 13183 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:50.957635 13280 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:50.998052 13280 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:50.960781  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:27:50.977645 13298 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.113s	user 0.002s	sys 0.000s
W20260709 12:27:50.989183  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68796.2us
I20260709 12:27:51.005128  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:51.001704 13280 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:51.006474 13280 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.007201 13280 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.012041  5456 catalog_manager.cc:5697] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 10 to 12, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 12 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:51.017205 13183 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.015600  5456 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:27:51.032824 13280 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:50.856081 13157 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.055s
I20260709 12:27:50.881672 13124 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:50.886559 12973 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:27:50.892153 12913 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:27:51.054138  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 221145us
I20260709 12:27:51.057178  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:51.058421 12913 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:51.058867 12913 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:27:51.059795 13203 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.750s	user 0.014s	sys 0.006s
I20260709 12:27:50.906817 13260 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.061219  6280 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:51.062680 13260 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:50.893294 12862 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.065753 12862 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:51.044128 13278 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
I20260709 12:27:50.912868 13197 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.154s
I20260709 12:27:51.068235 12913 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.068192 13124 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.070127 13280 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:51.070582 13280 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 19 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:51.070667 13124 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:51.071710 13280 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.073422 13280 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:51.040622 13231 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:51.087445  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 126781us
I20260709 12:27:51.097194 12935 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:51.097874 12935 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:51.098801 12935 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:51.103390  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 98353.3us
I20260709 12:27:51.100240 13124 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.010s
W20260709 12:27:51.115425  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 333519us
I20260709 12:27:51.124058  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:51.112274  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 292650us
I20260709 12:27:51.133350  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:51.131256 13285 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:51.144253 13285 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.146944 13285 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:51.147625 13285 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.149240 13318 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:51.150708 13157 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:51.151132 13157 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:51.152143 13157 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:51.153431 13306 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:27:51.154366 13157 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:51.154469  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:51.160880  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:51.162093 12847 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:27:51.148305 13285 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.166492 13285 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:51.169436  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:51.170454 12847 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:51.170873 12847 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:51.171187 12847 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:51.113828 13011 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:51.172683 13011 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:51.173099 13011 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:51.114861 13203 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:51.185007  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:51.186748 13113 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.577s	user 0.008s	sys 0.000s
I20260709 12:27:51.187107  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:51.187275 13113 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.189011  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:51.189657  5557 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
W20260709 12:27:51.191715  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 58712us
W20260709 12:27:51.193521  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:51.138446  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:51.195607  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 71728.5us
I20260709 12:27:51.194625  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:51.221071  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 363519us
W20260709 12:27:51.223260  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 466773us
I20260709 12:27:51.205217 13139 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:51.224671  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:51.222261  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:51.252103  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:51.140084 12762 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.077s
I20260709 12:27:51.262435  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:51.274183  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:51.276228  5708 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:51.257377 13285 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:51.279043 13285 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:51.280014 13285 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:51.292038 13186 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:51.293167  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 480871us
W20260709 12:27:51.296128  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 437394us
I20260709 12:27:51.308390  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:51.309494  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:51.319334  5702 pending_rounds.cc:85] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 10
I20260709 12:27:51.320336  5702 pending_rounds.cc:107] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20260709 12:27:51.321919  5710 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:27:51.323346  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:51.327071  5707 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:27:51.342602  5702 raft_consensus.cc:2889] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:51.327999  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:51.332710  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:51.306463  5634 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 15. (term mismatch)
I20260709 12:27:51.343925  5634 pending_rounds.cc:85] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 14
I20260709 12:27:51.344260  5634 pending_rounds.cc:107] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 21.15
I20260709 12:27:51.344601  5634 raft_consensus.cc:2889] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:27:51.381251  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 107198us
I20260709 12:27:51.303258  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:51.384574  5634 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:27:51.221513 13290 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:51.393184  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:51.396059  6275 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:51.181754 12847 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.254222 13252 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.407112 13252 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.408694  5706 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 12:27:51.409878 13252 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:51.411671  5628 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:27:51.395853 13290 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:51.412894 13290 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:51.418730 13327 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:51.419358 13327 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:51.419780 13327 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:51.423027 13157 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:51.424742 13314 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:27:51.426649 13314 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:27:51.427991 13314 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:27:51.430079 13314 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:51.403041  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:51.431960  6279 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 17: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:51.273324 13159 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:51.433142 13159 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.405100 13113 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:51.399164 13041 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.437201 13041 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.395135 13315 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:51.440722 13315 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:51.441252 13315 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:51.444572 13041 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:51.430954 12847 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:51.446382 13159 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:51.446856  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 160029us
I20260709 12:27:51.276641 13139 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:27:51.450521  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53876.7us
I20260709 12:27:51.449872 13139 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:51.457288 13188 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.832s	user 0.007s	sys 0.004s
I20260709 12:27:51.458309 13188 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.393314 13124 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:27:51.461031 13124 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:51.462309 13124 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.399554 13186 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.470830 13010 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:51.472681  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:51.401881  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:51.483271  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:51.519483 13137 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.675s	user 0.045s	sys 0.097s
W20260709 12:27:51.520010 13137 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.046s	sys 0.097s
W20260709 12:27:51.520287  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 210907us
I20260709 12:27:51.521749  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:51.530491  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:51.500900 13314 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:51.541049 13314 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:51.542142 13272 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.933s	user 0.011s	sys 0.000s
I20260709 12:27:51.542649 13272 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.543809  6278 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:51.544636 12847 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.545224 12847 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.551050 13197 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:51.551697 13197 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:51.552380 13197 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:51.554363 12847 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:51.552711 13157 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:51.555769 13157 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:51.556223 13157 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:51.561233  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:51.569096  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:51.571913 12847 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:51.573351 12847 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:51.573935 12847 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:27:51.577435  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 175639us
I20260709 12:27:51.579012  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:51.564128 13214 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:51.497977 13250 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.891s	user 0.006s	sys 0.006s
W20260709 12:27:51.593587  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:51.594183  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:51.478601 13312 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:27:51.607898  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 131934us
I20260709 12:27:51.608457  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:51.613046  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:51.624220  5629 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
W20260709 12:27:51.615787 13331 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.067s	user 0.003s	sys 0.000s
I20260709 12:27:51.617654 13314 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.617861 13313 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:27:51.478138 13266 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.119s
I20260709 12:27:51.618391 13226 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:51.593930 12973 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:51.619016  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:51.619457 13011 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.606308 13250 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:51.629297  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 297215us
I20260709 12:27:51.478454 13142 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.118s
W20260709 12:27:51.634374  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50115.7us
I20260709 12:27:51.636226  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:51.616151 13280 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.542s	user 0.006s	sys 0.000s
I20260709 12:27:51.478513 13010 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:51.699239  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63116.4us
I20260709 12:27:51.597644 12762 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:27:51.597096 13264 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:27:51.633737 13193 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.822s	user 0.012s	sys 0.000s
I20260709 12:27:51.674034 13290 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.521018 13260 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.113s
I20260709 12:27:51.676959 13314 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:51.664676 13226 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:51.608014  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:51.687624  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:51.740272  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 52726.9us
I20260709 12:27:51.522208 13263 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.112s
I20260709 12:27:51.709057 13142 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:51.709280 12847 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.704056  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:27:51.752463  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48510.9us
W20260709 12:27:51.752831  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 144917us
I20260709 12:27:51.710462 13010 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:51.710928 13325 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:51.663884 13183 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:51.760955  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:51.595865 13285 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.711607 13320 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:27:51.763243  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:51.711730 13011 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:51.719645  5703 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:51.720404  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:51.658695 12913 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:51.672251 13272 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:51.679881 13318 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.624876 12957 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.081s
I20260709 12:27:51.663695 13041 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
W20260709 12:27:51.746790  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:51.724645 13203 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:51.750488  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 447312us
I20260709 12:27:51.724623 13266 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.775168 13203 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.775358 13266 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.744444 13188 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:51.752607 13252 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:51.776299  5627 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:27:51.752465 13271 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.777374 13010 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:51.777508 13271 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.777971 13266 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:51.778419  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:51.749656 13142 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:51.776664 13252 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:51.779743 13313 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:51.757803 13325 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:51.781760  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:51.758211 13193 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:51.783991  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:51.795876  5628 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:27:51.796891  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:51.759891 13159 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:27:51.771728  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51437.5us
I20260709 12:27:51.760665 12753 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:51.785535 13266 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.008s
I20260709 12:27:51.784089  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:51.813220  5629 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:51.793773  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:51.781000 13231 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.670s	user 0.000s	sys 0.012s
I20260709 12:27:51.764621 13320 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:51.794323 13272 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.783712 13312 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:51.784015 13287 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:51.815716  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:51.847908  5557 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:27:51.772307 13276 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.031s
I20260709 12:27:51.786027 13252 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:51.814388  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:51.800248  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:51.821049  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:51.794013 13334 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:51.817440 13271 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:51.821188  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:51.816313  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:51.855615  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:51.796133 13315 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.882732  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:51.784963 13226 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:51.785195 13318 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.896306  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:51.869333 13287 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:51.803334 13159 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:27:51.822829  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 560467us
I20260709 12:27:51.931370  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:51.803604 13137 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:27:51.906064  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:51.907171 13329 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:27:51.818076 13313 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:51.824986 13320 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:51.913771  5556 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
W20260709 12:27:51.914183  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 97955.5us
I20260709 12:27:51.945120  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:51.949532  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:51.832741  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:51.818048 13321 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.826745 13186 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.917196  5552 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:51.885560  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:51.918777  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 104463us
I20260709 12:27:51.834666 13342 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:51.838884  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:51.925274  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:51.932574  5557 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:51.900831 13312 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:51.823611 13231 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.869305 13272 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:51.925855 13011 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:51.930541 13287 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:51.958284 13264 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:27:51.973182  5559 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:51.975261 13137 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:27:51.976747  5702 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.013s	sys 0.016s
I20260709 12:27:51.973481  5708 raft_consensus.cc:2410] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:27:51.932881 13250 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:51.970872  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:51.984108  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:51.984681  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:51.986747 13332 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.417s	user 0.002s	sys 0.000s
I20260709 12:27:52.001919  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:52.002367  5675 leader_election.cc:336] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.503s (SENT)
I20260709 12:27:52.002770  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.001969  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.006079  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:51.976207 13144 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.936s	user 0.012s	sys 0.000s
I20260709 12:27:51.916605 13334 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.974992 13321 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:52.017341  5627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 1529) took 1680 ms (client timeout 1671 ms). Trace:
I20260709 12:27:52.017438  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3212) took 1068 ms. Trace:
I20260709 12:27:51.904261 13142 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:51.982851 13330 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.210s	user 0.001s	sys 0.001s
I20260709 12:27:51.932224 13159 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:52.006717 13333 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
W20260709 12:27:52.025501  5702 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.013s	sys 0.016s
I20260709 12:27:52.047290  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:52.174686  5708 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:52.175927  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:52.178561  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:51.967834 13340 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:52.189163  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:52.051620  6274 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260709 12:27:52.054452 13326 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.510s	user 0.000s	sys 0.003s
I20260709 12:27:51.992292 13010 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:52.016391 13011 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:52.027899 13329 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:27:51.972659 13193 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:27:52.091353 13310 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.187s	user 0.000s	sys 0.000s
I20260709 12:27:51.979275 13290 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.043892 12847 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:51.981024 13186 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:51.971766 13226 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:52.045887 13324 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.110s
I20260709 12:27:51.982990 13342 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:52.120855  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:52.170522  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 168205us
I20260709 12:27:52.080505  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1532) took 1498 ms. Trace:
I20260709 12:27:52.078732  5706 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 21: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:27:52.170522  5627 rpcz_store.cc:269] 0709 12:27:50.317839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:50.317972 (+   133us) service_pool.cc:224] Handling call
0709 12:27:51.998152 (+1680180us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b540032c8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:51.998355 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3344372992}
I20260709 12:27:52.173624  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.214720 13193 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:52.189762  5709 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:52.281008  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:52.281852  5707 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:52.170583  5703 rpcz_store.cc:276] 0709 12:27:50.924690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:50.927162 (+  2472us) service_pool.cc:224] Handling call
0709 12:27:50.927547 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:51.764381 (+836834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:51.993223 (+228842us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b5400074c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:51.993397 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1820929920}
I20260709 12:27:52.202450  5629 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:52.198884  5552 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:52.042913 13157 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:52.189221 13348 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:27:51.982620 13188 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.070s
I20260709 12:27:52.118865 13288 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.094632 13285 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.190371 13340 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:52.204114 12786 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:52.204366 13322 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:52.216408 13011 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:27:52.287751 13350 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:27:52.306006  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.306658  5627 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:52.181061  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:52.276546  5633 rpcz_store.cc:276] 0709 12:27:50.582257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:50.588169 (+  5912us) service_pool.cc:224] Handling call
0709 12:27:50.588529 (+   360us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:50.862102 (+273573us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:50.862130 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:50.862715 (+   585us) log.cc:844] Serialized 52 byte log entry
0709 12:27:51.292652 (+429937us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:51.292701 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:51.292792 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:52.080232 (+787440us) raft_consensus.cc:1685] finished
0709 12:27:52.080264 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:52.080437 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":90090,"raft.queue_time_us":111884,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":50777,"thread_start_us":569410,"threads_started":2,"wal-append.queue_time_us":927732}
I20260709 12:27:52.285059 13193 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:52.285703  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3208) took 1781 ms. Trace:
W20260709 12:27:52.286698  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 315894us
I20260709 12:27:52.209134 13010 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.291945  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3088) took 1529 ms. Trace:
I20260709 12:27:52.129382 13312 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.291146  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:52.304167  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 298205us
I20260709 12:27:52.322378 13010 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.323254 13010 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:27:52.323992  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:52.279035  5635 raft_consensus.cc:2376] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27. Re-sending same reply.
W20260709 12:27:52.326277 13349 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.223s	user 0.000s	sys 0.001s
I20260709 12:27:52.294396 13318 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:52.328812  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:52.329945  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:52.330513  5703 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 15.
I20260709 12:27:52.330838  5559 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:52.331382  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:52.332393  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.333073 13188 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.333608 13188 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.335664 13188 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.336597 13358 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.337137 13358 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.337949 13186 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.339325 13358 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:52.340888 13351 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.137s	user 0.000s	sys 0.000s
I20260709 12:27:52.341524  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.342437  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:52.343209 13342 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:52.343566  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:52.344159  5710 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:52.305142  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1530) took 1723 ms. Trace:
I20260709 12:27:52.277630 13139 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:52.348559 13318 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.348886 13139 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.349375  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 2976) took 1022 ms. Trace:
I20260709 12:27:52.349283 13318 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:52.349895  5709 rpcz_store.cc:276] 0709 12:27:51.327285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:51.327654 (+   369us) service_pool.cc:224] Handling call
0709 12:27:52.349202 (+1021548us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b54001991d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:52.349306 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1946059136}
I20260709 12:27:52.351239  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:52.351557 13318 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:52.287572 13124 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.009s	sys 0.016s
I20260709 12:27:52.285054 13315 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:52.348107  5631 rpcz_store.cc:276] 0709 12:27:50.581618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:50.585906 (+  4288us) service_pool.cc:224] Handling call
0709 12:27:50.586272 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:50.659546 (+ 73274us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:50.659587 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:50.671999 (+ 12412us) log.cc:844] Serialized 26 byte log entry
0709 12:27:51.218944 (+546945us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:51.219006 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:51.219111 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:52.304934 (+1085823us) raft_consensus.cc:1685] finished
0709 12:27:52.304955 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:52.305087 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":86221,"raft.queue_time_us":85827,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":105527,"thread_start_us":439692,"threads_started":1,"wal-append.queue_time_us":1008398}
I20260709 12:27:52.312548  5704 rpcz_store.cc:276] 0709 12:27:50.503810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:50.633204 (+129394us) service_pool.cc:224] Handling call
0709 12:27:50.633555 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:50.636820 (+  3265us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:27:50.913037 (+276217us) log.cc:844] Serialized 14 byte log entry
0709 12:27:51.284964 (+371927us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:51.285629 (+   665us) log.cc:844] Serialized 26 byte log entry
0709 12:27:51.285794 (+   165us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:51.285839 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:51.285933 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:52.285464 (+999531us) raft_consensus.cc:1685] finished
0709 12:27:52.285491 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:52.285642 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":163078,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":142530,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":200,"thread_start_us":620477,"threads_started":2,"wal-append.queue_time_us":840380}
I20260709 12:27:52.354800  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:52.313277  5524 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:52.357497  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44297.3us
I20260709 12:27:52.359669 13315 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:52.360307 13315 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 12:27:52.361431  5451 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35600 (request call id 279) took 25505 ms (client timeout 15000 ms). Trace:
I20260709 12:27:52.361774 13315 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:27:52.314311  5708 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 19: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:52.362700 13314 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:52.363138 13314 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:52.315543  5632 rpcz_store.cc:276] 0709 12:27:50.762155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:50.764041 (+  1886us) service_pool.cc:224] Handling call
0709 12:27:50.764499 (+   458us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:51.119936 (+355437us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:51.119980 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:51.134206 (+ 14226us) log.cc:844] Serialized 26 byte log entry
0709 12:27:51.518879 (+384673us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:51.518918 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:51.518993 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:52.291739 (+772746us) raft_consensus.cc:1685] finished
0709 12:27:52.291762 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:52.291889 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41861,"raft.queue_time_us":77525,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":10788,"thread_start_us":593961,"threads_started":2,"wal-append.queue_time_us":536308}
I20260709 12:27:52.321251 13324 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:27:52.335933  5456 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 10 to 13. New cstate: current_term: 13 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:27:52.306943 13346 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.324s	user 0.000s	sys 0.001s
I20260709 12:27:52.351910  5709 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:52.183106  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:52.308286  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.365336  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.367772  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:52.382704  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 199730us
I20260709 12:27:52.382328  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.368736  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:52.385200  5707 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
W20260709 12:27:52.369495  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 362963us
I20260709 12:27:52.370023  5561 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 12:27:52.362365 13344 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.383s	user 0.000s	sys 0.002s
I20260709 12:27:52.371692  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:52.348697 13357 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:27:52.368487  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.399718  5552 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:52.369030  5635 raft_consensus.cc:2410] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 24 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:27:52.385524  5703 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:52.385799  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.387220  5553 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 18: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:52.301198 13340 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:52.303865 13290 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.448774 13290 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.451035 13290 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.389014  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:52.452381  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:52.453250  5633 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:52.391512  5710 raft_consensus.cc:2410] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:27:52.455061  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:52.455832  5710 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:52.456010 12973 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:52.456652 12973 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.457494  5710 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:27:52.459169 12973 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:52.459458  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.393061  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:52.460687  5556 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:27:52.461766 13352 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.358s	user 0.000s	sys 0.003s
I20260709 12:27:52.351400 13139 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.462970 13122 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:52.463577 13122 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:52.465662 13122 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:52.470908 13183 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.706s	user 0.009s	sys 0.000s
I20260709 12:27:52.471040 13334 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.471446 13183 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:52.407474  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:52.391186  5707 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:27:52.408921  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:52.473402  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:52.474277  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.474742  5627 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
W20260709 12:27:52.361847  5451 rpcz_store.cc:269] 0709 12:27:26.864780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:35.145969 (+8281189us) service_pool.cc:224] Handling call
0709 12:27:52.361339 (+17215370us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:27:46.418354 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:27:46.419377 (+  1023us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:27:46.419457 (+    80us) write_op.cc:435] Acquired schema lock
0709 12:27:46.419476 (+    19us) tablet.cc:662] Decoding operations
0709 12:27:46.586132 (+166656us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:27:46.586256 (+   124us) write_op.cc:641] Partition lock acquired for write op
0709 12:27:46.586288 (+    32us) tablet.cc:685] Acquiring locks for 19 operations
0709 12:27:46.617996 (+ 31708us) tablet.cc:701] Row locks acquired
0709 12:27:46.635888 (+ 17892us) write_op.cc:260] PREPARE: finished
0709 12:27:46.663804 (+ 27916us) write_op.cc:270] Start()
0709 12:27:46.684374 (+ 20570us) write_op.cc:276] Timestamp: P: 1783600066649585 usec, L: 0
0709 12:27:46.689335 (+  4961us) op_driver.cc:348] REPLICATION: starting
0709 12:27:46.699695 (+ 10360us) log.cc:844] Serialized 7366 byte log entry
0709 12:27:47.780734 (+1081039us) op_driver.cc:464] REPLICATION: finished
0709 12:27:47.969706 (+188972us) write_op.cc:301] APPLY: starting
0709 12:27:47.971009 (+  1303us) tablet.cc:1367] starting BulkCheckPresence
0709 12:27:48.099100 (+128091us) tablet.cc:1370] finished BulkCheckPresence
0709 12:27:48.099226 (+   126us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:27:48.194640 (+ 95414us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:27:48.195077 (+   437us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=38,mrs_lookups=0
0709 12:27:48.195151 (+    74us) write_op.cc:312] APPLY: finished
0709 12:27:48.195661 (+   510us) log.cc:844] Serialized 172 byte log entry
0709 12:27:48.195857 (+   196us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:27:48.196050 (+   193us) write_op.cc:454] Released schema lock
0709 12:27:48.196145 (+    95us) write_op.cc:341] FINISH: Updating metrics
0709 12:27:48.217936 (+ 21791us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:27:48.217974 (+    38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":129198,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":163205,"apply.run_cpu_time_us":29293,"apply.run_wall_time_us":249690,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83547,"num_ops":19,"prepare.queue_time_us":143986,"prepare.run_cpu_time_us":167906,"prepare.run_wall_time_us":824885,"raft.queue_time_us":160985,"raft.run_cpu_time_us":18675,"raft.run_wall_time_us":184935,"replication_time_us":1036914,"thread_start_us":670270,"threads_started":4,"wal-append.queue_time_us":692459,"wal-append.run_cpu_time_us":43082,"wal-append.run_wall_time_us":1752293}]]}
I20260709 12:27:52.410557  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:52.410280  5631 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:27:52.415132  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:52.371083 13124 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.689s	user 0.010s	sys 0.016s
I20260709 12:27:52.392760 13358 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:52.419603 12753 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:27:52.512619 13354 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
I20260709 12:27:52.303551 13288 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:52.513571 12753 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.514880 12753 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:52.475785 13350 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
W20260709 12:27:52.520063 13197 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.967s	user 0.010s	sys 0.001s
I20260709 12:27:52.423309  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.360024 13325 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:52.425663 13337 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.628s	user 0.003s	sys 0.000s
I20260709 12:27:52.426080  5456 catalog_manager.cc:5697] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 6 to 12. New cstate: current_term: 12 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:52.426759  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.396556 13266 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:52.428052  6237 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:52.546497  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:52.547086  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 119114us
I20260709 12:27:52.430660 12786 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:27:52.432920  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3086) took 1671 ms. Trace:
I20260709 12:27:52.363937 13314 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.434098  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:52.435252 13231 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.436041  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:52.436774  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:52.437109 13203 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:52.439461 13327 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.019s	user 0.005s	sys 0.006s
I20260709 12:27:52.380594 13188 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.060s
I20260709 12:27:52.440697  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:52.446738  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.449057  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:52.475822  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.575755  5552 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 15.
W20260709 12:27:52.551254 13252 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.636s	user 0.005s	sys 0.006s
I20260709 12:27:52.476593  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.476306  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:52.478046  5553 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:27:52.479053  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 64093.3us
I20260709 12:27:52.478259  5710 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:27:52.413702  5709 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:52.651546  5709 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:52.628337 13263 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.493108  5561 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:52.435758 13287 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:52.495260  5707 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
I20260709 12:27:52.505806  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.654843  5707 pending_rounds.cc:85] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 13
I20260709 12:27:52.626791 13288 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:52.413892 13272 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:52.527491 13320 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.548s	user 0.007s	sys 0.001s
I20260709 12:27:52.420019 13321 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.308135 13322 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.540803  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.541649  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.539251 13183 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:27:52.660542  5710 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:27:52.559768  6280 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:27:52.554375  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.546241  5456 catalog_manager.cc:5697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 12 to 15, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 15 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:52.514313 12957 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:27:52.558854  5630 rpcz_store.cc:276] 0709 12:27:50.761229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:50.762651 (+  1422us) service_pool.cc:224] Handling call
0709 12:27:50.763058 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:51.298740 (+535682us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:51.300312 (+  1572us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:51.574155 (+273843us) spinlock_profiling.cc:243] Waited 79.2 ms on lock 0x7b2c00000de8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:27:51.574616 (+   461us) log.cc:844] Serialized 26 byte log entry
0709 12:27:51.758790 (+184174us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:51.758831 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:51.758902 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:52.361726 (+602824us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:52.432671 (+ 70945us) raft_consensus.cc:1685] finished
0709 12:27:52.432694 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:52.432846 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43875,"raft.queue_time_us":67238,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":4760,"spinlock_wait_cycles":217853952,"thread_start_us":670926,"threads_started":2,"wal-append.queue_time_us":319565}
I20260709 12:27:52.664847  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:52.565385  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:52.568823  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:52.667709  6278 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: [22.18-22.18]
W20260709 12:27:52.573613  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 350953us
I20260709 12:27:52.626593 13252 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:52.481591  5556 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:27:52.675509  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:52.575069  5706 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:27:52.580514  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.554203 13319 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:52.550060 13266 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:52.679481 13319 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:52.679626 13266 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:52.679998 13319 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:52.540661 13358 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.682014  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:52.682668 13358 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.569972 13325 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:52.684930 13130 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:52.685585 13130 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:52.686487 13130 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:52.631492  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:52.687866  5703 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:27:52.688838  5560 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:27:52.576048 13203 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.579310 13157 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:52.654009  5709 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 12:27:52.512367  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:52.655121  5707 pending_rounds.cc:107] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260709 12:27:52.696215  5707 raft_consensus.cc:2889] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:52.655049  5557 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:52.696491  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:52.697440  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.697887  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.698784  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.699690  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.659210  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:52.702669  5553 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:52.655002  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:52.644593 13314 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:52.663367  5710 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:52.576696  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.707798  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.576754 13327 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.671276  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.716548  5633 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:52.676784  5636 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:52.660068 13320 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:52.678017  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:52.660800 13183 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:52.678689  5456 catalog_manager.cc:5697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 10 to 13, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 13 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:52.548167 13197 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:52.689429  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 213163us
I20260709 12:27:52.699757  5557 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 24: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:52.703629  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:52.727629  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:52.727998  5553 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:27:52.728744  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 256255us
W20260709 12:27:52.729430  5675 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.538s (SENT)
I20260709 12:27:52.729784  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:52.669694  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 94297.6us
I20260709 12:27:52.728629  5636 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:27:52.733577  5701 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:52.711915  5630 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:27:52.734526  5628 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:52.659837 13365 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:52.718504  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:52.718999  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:52.745689  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.746973  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:52.748018  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:52.748445  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:52.751581  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:52.754026  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:52.705497 13358 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:27:52.755215  5675 leader_election.cc:336] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.696s (SENT)
W20260709 12:27:52.761739  5675 leader_election.cc:336] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.716s (SENT)
W20260709 12:27:52.762799  5675 leader_election.cc:336] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.793s (SENT)
I20260709 12:27:52.763181  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:52.660928 13263 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:52.750288  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:52.726765 13183 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:52.766235  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:52.749333  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.784559  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.776232 13370 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:52.777318  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:52.777536  5630 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:27:52.777637  5705 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:52.688889 13319 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:52.759207  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.776389  5553 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:52.746690 13157 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:52.798121  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:52.789868  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:52.758371 13159 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.547s	user 0.010s	sys 0.000s
I20260709 12:27:52.750597 13318 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.798986  5631 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:27:52.793121  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.799093  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:52.799788  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:52.804705  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:52.805171  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 123238us
I20260709 12:27:52.713518 13203 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:52.802289  5553 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:52.805866  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.758168 12973 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:52.805953  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:52.809688  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.778796 13122 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:52.778949 13124 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:27:52.818151  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:52.827778  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:52.758838 13365 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.799751 12786 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:27:52.789403 13347 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.817245  5630 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 23: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:52.808043 13358 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:52.819520 13313 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:52.810648 12935 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:27:52.817710 13290 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:52.877974  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 209237us
I20260709 12:27:52.879825  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.848531 13157 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:27:52.883056  5557 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:27:52.880695  5630 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:52.812885 13318 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:52.885525  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:52.807122 13159 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:52.886181  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 120058us
I20260709 12:27:52.822688 12973 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.871606 13122 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:52.906741  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.869298 13370 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:27:52.878997 13287 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.925240  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:52.927250  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:52.933815  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:52.886142 12935 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:27:52.938745 13375 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:27:52.915585 13231 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:27:52.939507 13371 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:27:52.955477  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:52.926326 13370 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:52.958933  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:52.959373 13370 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:27:52.961700 12935 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.7, Last appended by leader: 7, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.941324 13159 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.941246 13260 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:27:52.919477 13383 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.038s
W20260709 12:27:52.945885  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 146875us
I20260709 12:27:52.938946 13319 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.874337 13347 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:52.940660 13384 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:52.958431 13157 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:52.967676  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:52.977496  5707 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:52.885970 13290 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:52.989962  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 108988us
W20260709 12:27:52.957867 13193 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.585s	user 0.006s	sys 0.004s
I20260709 12:27:53.003320  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:52.983157 13369 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.264s	user 0.001s	sys 0.000s
I20260709 12:27:52.954438 13231 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:52.929853 13122 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:52.984941 12786 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:52.998571 13322 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:52.989351 13384 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:52.968061  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:53.067435 13390 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:27:52.995746 13377 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:27:53.076797  5630 raft_consensus.cc:606] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
W20260709 12:27:53.078671  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 144966us
I20260709 12:27:53.079607  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:53.007884 13388 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.022076 13157 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:53.021366 13231 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:53.008914 13181 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:53.006604 13182 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:27:53.234220  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:53.007763 13252 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:53.023692 13276 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.192s
I20260709 12:27:53.073626 13263 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:53.072224 13320 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:53.252031  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:53.081058 13197 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:53.082086 13384 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:53.278383 13231 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:53.289803 13231 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:53.298019 13386 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:27:53.298022 13263 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.299266 13263 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.259912  5630 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:27:53.303902  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:53.260440 13252 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:27:53.260986 13391 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.239s	user 0.001s	sys 0.000s
I20260709 12:27:53.007907 13193 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.218300 13327 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:53.244645 13388 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.263998 13157 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:53.264968 13376 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.383s	user 0.000s	sys 0.001s
W20260709 12:27:53.266651 13378 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.341s	user 0.000s	sys 0.001s
I20260709 12:27:53.330739  5632 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:27:53.267184 13182 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.333457  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:53.333745 13182 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:53.334239 13325 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.651s	user 0.008s	sys 0.001s
I20260709 12:27:53.333493 13193 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:53.335117  5634 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:27:53.334340  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2860) took 1096 ms. Trace:
I20260709 12:27:53.335813 13182 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:53.256294 13398 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:27:53.267448 13380 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.265s	user 0.000s	sys 0.001s
I20260709 12:27:53.256170 13181 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:53.221644 12786 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.245687 13203 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:53.215260 13264 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 12:27:53.245301 13290 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:53.264165 13360 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.257s
I20260709 12:27:53.333108 13263 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:53.336325  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2833) took 1335 ms. Trace:
I20260709 12:27:53.342167  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2972) took 2028 ms. Trace:
I20260709 12:27:53.384989  5630 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:53.383687  5702 rpcz_store.cc:276] 0709 12:27:51.315010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:51.317913 (+  2903us) service_pool.cc:224] Handling call
0709 12:27:51.318790 (+   877us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:52.174486 (+855696us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:52.174533 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:52.175261 (+   728us) log.cc:844] Serialized 52 byte log entry
0709 12:27:52.576541 (+401280us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:52.796454 (+219913us) log.cc:844] Serialized 14 byte log entry
0709 12:27:53.040540 (+244086us) log.cc:844] Serialized 14 byte log entry
0709 12:27:53.341696 (+301156us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:53.341812 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:53.341874 (+    62us) raft_consensus.cc:1685] finished
0709 12:27:53.341893 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:53.342067 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":361558,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":6671,"raft.queue_time_us":450181,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":13570,"thread_start_us":1850449,"threads_started":5,"wal-append.queue_time_us":446200}
I20260709 12:27:53.337738  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2977) took 1404 ms. Trace:
I20260709 12:27:53.386435  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:53.386756  5708 rpcz_store.cc:276] 0709 12:27:51.933226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.363216 (+429990us) service_pool.cc:224] Handling call
0709 12:27:52.363611 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:52.363970 (+   359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:52.364002 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:52.364497 (+   495us) log.cc:844] Serialized 26 byte log entry
0709 12:27:52.744704 (+380207us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:52.744763 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:52.744861 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:53.337532 (+592671us) raft_consensus.cc:1685] finished
0709 12:27:53.337552 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:53.337678 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46360,"raft.queue_time_us":17167,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":26531,"thread_start_us":379760,"threads_started":1,"wal-append.queue_time_us":498519}
I20260709 12:27:53.382710  5634 rpcz_store.cc:276] 0709 12:27:52.000659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.001630 (+   971us) service_pool.cc:224] Handling call
0709 12:27:53.336130 (+1334500us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540018c254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:53.336270 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2672995200}
I20260709 12:27:53.335698  5629 rpcz_store.cc:276] 0709 12:27:52.237925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.453738 (+215813us) service_pool.cc:224] Handling call
0709 12:27:53.333301 (+879563us) spinlock_profiling.cc:243] Waited 5.84 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754de9c 00007f4ff754da5b 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:27:53.334279 (+   978us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13442176}
I20260709 12:27:53.392787  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:53.323186 12762 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:53.337661 13318 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:53.290741 13372 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.433s	user 0.000s	sys 0.001s
I20260709 12:27:53.337343 12973 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:53.328656 13183 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.530s	user 0.007s	sys 0.004s
I20260709 12:27:53.326896 13370 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:53.348836 13325 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:53.336742 13398 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:53.375108 13011 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.059s	user 0.013s	sys 0.000s
I20260709 12:27:53.400698  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2861) took 1118 ms. Trace:
I20260709 12:27:53.414214  5634 rpcz_store.cc:276] 0709 12:27:52.282122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:53.399525 (+1117403us) service_pool.cc:224] Handling call
0709 12:27:53.399975 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:53.400201 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:53.400229 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:53.400269 (+    40us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:53.400301 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:53.400459 (+   158us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:53.400611 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:27:53.390334 13391 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:27:53.374626 13365 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:53.396988 13231 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:53.416752  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 407035us
W20260709 12:27:53.416393  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.007s	sys 0.009s
W20260709 12:27:53.441439  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.008s	sys 0.009s
W20260709 12:27:53.426062  6234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.009s	sys 0.010s
W20260709 12:27:53.444742  6234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.009s	sys 0.010s
W20260709 12:27:53.445272  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 601430us
I20260709 12:27:53.427448 12973 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.440716  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:53.428146  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:53.484373  5633 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.016s	sys 0.028s
I20260709 12:27:53.485785  5554 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 28: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:27:53.488775  5633 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.017s	sys 0.028s
I20260709 12:27:53.410344 13186 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:53.405915 12762 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.498725  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:53.496851  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:53.503068  6274 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
W20260709 12:27:53.492424  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 411937us
W20260709 12:27:53.492621  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:53.417064 13398 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:27:53.451372 13340 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.945s	user 0.009s	sys 0.006s
I20260709 12:27:53.493709 13339 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:53.418294 13325 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:53.487977 13365 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:53.507220  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:53.522433  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 94773.1us
I20260709 12:27:53.522593  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:53.523756  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:53.509493  5702 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:27:53.425000 13011 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.505391  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1356) took 1652 ms. Trace:
I20260709 12:27:53.511606  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:53.507251  6275 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:27:53.494841 13390 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.447571 13181 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:53.505949  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:53.556825  5708 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:53.507982  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:53.523813  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:53.526054  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:53.510279  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:53.488283 13182 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:27:53.496446 13276 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.528982 12762 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:53.499171 13290 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:53.531216  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:53.532694  5704 rpcz_store.cc:276] 0709 12:27:51.852881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.354646 (+501765us) service_pool.cc:224] Handling call
0709 12:27:52.354960 (+   314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:52.355346 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:52.355376 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:52.355845 (+   469us) log.cc:844] Serialized 26 byte log entry
0709 12:27:52.669381 (+313536us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:27:52.669419 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:52.669481 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:53.505144 (+835663us) raft_consensus.cc:1685] finished
0709 12:27:53.505179 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:53.505329 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15858,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":6560,"thread_start_us":313146,"threads_started":1,"wal-append.queue_time_us":420354}
I20260709 12:27:53.504298 13347 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:53.493181 13263 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 12:27:53.502667 13186 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.566465  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:53.518211 13340 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:53.571614  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:53.413295 13183 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.617089  5708 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 17.
I20260709 12:27:53.571789  5707 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 19: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:53.618314  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:53.618773  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:53.620800  6279 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:53.621407  5704 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:27:53.487978 12973 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.563849  5559 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:53.567433  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:53.623231  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:53.568306  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44626.9us
I20260709 12:27:53.567159  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:53.488561 13272 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.033s
I20260709 12:27:53.483834 13231 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.624305  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:53.557394  5709 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 17.
I20260709 12:27:53.629215  5702 raft_consensus.cc:2410] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:27:53.629422  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:53.630647  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:53.630923  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:53.631583  5702 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:53.632470 13231 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:53.623973 12973 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:53.627174  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:53.633541  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:53.627345  5706 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:53.628334  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:53.634739  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:53.635900 13347 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:27:53.636489 12973 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:53.636490  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:53.623793 13183 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:53.639248  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:53.635998  5709 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:53.640244 13347 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:53.640707  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:53.641567 13347 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:53.642007  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:53.562575 13313 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.683s	user 0.010s	sys 0.002s
I20260709 12:27:53.620573  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:53.649120 13404 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:53.652227 13231 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:53.653307 13370 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:53.649436 13271 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:53.654814 13271 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.652858 13231 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:53.657107  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 91737.9us
I20260709 12:27:53.513089 13395 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:27:53.665840  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:53.573616 13142 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:53.650221  5708 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:53.560338 13182 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.526316 13339 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.530387 13365 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.664104  5556 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:53.696615  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:53.647977 13388 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:53.650283 13285 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:53.660784 13313 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:53.710736  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:27:53.708568  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 187521us
I20260709 12:27:53.662235 13183 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:53.571167 13276 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.556164 13390 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.645036 13290 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.656570 13406 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:53.666844 13395 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:53.645090 13409 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:27:53.734237 13400 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.250s	user 0.000s	sys 0.001s
W20260709 12:27:53.735615 13401 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
W20260709 12:27:53.737696 13415 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.207s	user 0.002s	sys 0.000s
I20260709 12:27:53.700632 13403 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:27:53.739670 13340 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:53.740854 13271 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:53.885478 13313 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:53.702198 13182 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:27:53.742349  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:53.692237 13384 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.744609 13231 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:53.750788 13139 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.753870 13288 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.756400  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2835) took 1389 ms. Trace:
I20260709 12:27:53.759318  5706 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
W20260709 12:27:53.759487  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:53.763635 13011 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:53.701602 12762 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:53.767097  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1480) took 1929 ms. Trace:
W20260709 12:27:53.807006  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 96405.3us
I20260709 12:27:53.937570  5555 rpcz_store.cc:276] 0709 12:27:51.837400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:51.837848 (+   448us) service_pool.cc:224] Handling call
0709 12:27:51.838450 (+   602us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:52.630460 (+792010us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:52.630491 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:52.646844 (+ 16353us) log.cc:844] Serialized 26 byte log entry
0709 12:27:52.793340 (+146496us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:27:52.793379 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:53.766360 (+972981us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b540041a0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:53.766447 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:53.766537 (+    90us) raft_consensus.cc:1685] finished
0709 12:27:53.766558 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:53.766992 (+   434us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":41745,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":1047960320,"thread_start_us":464879,"threads_started":2,"wal-append.queue_time_us":285274}
I20260709 12:27:53.938858  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:53.939604  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:53.739365 13308 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:27:53.858929 13266 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.179s	user 0.007s	sys 0.009s
I20260709 12:27:53.940407  5555 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 20: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:53.940521 13406 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:53.940657 13266 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:53.941345  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:53.941954  5555 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:53.942024 13406 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:27:53.699435 13410 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.181s
I20260709 12:27:53.865068 13398 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.885346  5556 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:27:53.718026 13318 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:53.886833  5708 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:27:53.947981  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:53.724174 13285 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:53.858505 13314 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.146s	user 0.015s	sys 0.001s
I20260709 12:27:53.869203  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3223) took 1846 ms. Trace:
I20260709 12:27:53.950987  5705 rpcz_store.cc:276] 0709 12:27:52.022911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.463592 (+440681us) service_pool.cc:224] Handling call
0709 12:27:52.463909 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:52.790667 (+326758us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:53.773029 (+982362us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b54004529d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:53.869112 (+ 96083us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":721038720}
W20260709 12:27:53.876592  5701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1358) took 1908 ms (client timeout 1645 ms). Trace:
I20260709 12:27:53.871366 13183 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:53.898360  5631 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:27:53.882774 13347 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:53.741793 13186 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:53.919912  5560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2844) took 1534 ms (client timeout 1851 ms). Trace:
I20260709 12:27:53.944047 13308 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:53.897852 13395 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:27:53.925024  5554 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:53.887599 13370 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:53.891820 13409 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:53.923753  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3115) took 1068 ms. Trace:
I20260709 12:27:53.928587 13313 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.898962 13157 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:53.931469  5559 rpcz_store.cc:276] 0709 12:27:52.367104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.371366 (+  4262us) service_pool.cc:224] Handling call
0709 12:27:53.755717 (+1384351us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54000267d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:53.756210 (+   493us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2678855040}
I20260709 12:27:53.935364  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:53.943691  5634 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:53.942910  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:53.927284 13403 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:53.943399 13406 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:27:53.952301  5701 rpcz_store.cc:269] 0709 12:27:51.960933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:51.964423 (+  3490us) service_pool.cc:224] Handling call
0709 12:27:53.775033 (+1810610us) spinlock_profiling.cc:243] Waited 652 ms on lock 0x7b5400323054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:53.868856 (+ 93823us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1499831040}
I20260709 12:27:53.995049  5706 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:27:53.932024 13203 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 12:27:53.954532  5561 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 21 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 12:27:53.955391  5702 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
W20260709 12:27:53.961083  5560 rpcz_store.cc:269] 0709 12:27:52.385334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.385477 (+   143us) service_pool.cc:224] Handling call
0709 12:27:53.919698 (+1534221us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b540016ba54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:53.919850 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":662064384}
I20260709 12:27:53.997795  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:53.998765  5560 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:53.920315  5553 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:53.944242 13419 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:53.967628 13390 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:53.968102 13384 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:53.960630 13413 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:53.971185  5630 rpcz_store.cc:276] 0709 12:27:52.855581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:53.396692 (+541111us) service_pool.cc:224] Handling call
0709 12:27:53.404018 (+  7326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:53.407755 (+  3737us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:27:53.616304 (+208549us) log.cc:844] Serialized 14 byte log entry
0709 12:27:53.923287 (+306983us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:53.923363 (+    76us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:27:53.923437 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:53.923528 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:53.923677 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47927,"prepare.queue_time_us":472039,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":4728,"raft.queue_time_us":46817,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":3072,"thread_start_us":396010,"threads_started":2,"wal-append.queue_time_us":324448}
I20260709 12:27:53.898800 13388 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:53.935385 13139 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:53.946202 12762 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:53.936657 13288 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.009660  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:53.975692 13157 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:53.973767 13409 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:54.018325  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:54.013657 12973 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:27:54.020491  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 124981us
I20260709 12:27:54.022258  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:54.010344  5706 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:54.013345 13413 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:54.001050 13313 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.014652  5559 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:53.990319 13285 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.000828 13403 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.024888  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:54.031725  5630 raft_consensus.cc:606] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:27:54.033939 13322 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.034499 13322 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.031912  5560 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:27:54.035658  5555 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:54.036832  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:54.037535  5560 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:54.012756 13276 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.014655 13339 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.041320 12753 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:54.042083 13396 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:54.042536 13396 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:54.043460 13396 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.045753  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1554) took 1981 ms. Trace:
I20260709 12:27:54.046190  5633 rpcz_store.cc:276] 0709 12:27:52.064578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.730346 (+665768us) service_pool.cc:224] Handling call
0709 12:27:52.730769 (+   423us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:53.489425 (+758656us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:53.489477 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:53.495699 (+  6222us) log.cc:844] Serialized 26 byte log entry
0709 12:27:53.886885 (+391186us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:53.886921 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.045274 (+158353us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54001cf2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:54.045336 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:54.045396 (+    60us) raft_consensus.cc:1685] finished
0709 12:27:54.045410 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:54.045681 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":27478,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":4257,"spinlock_wait_cycles":275700096,"thread_start_us":905234,"threads_started":2,"wal-append.queue_time_us":416115}
I20260709 12:27:54.047739  5560 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 16.
I20260709 12:27:54.048708  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:54.049925  5635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2853) took 2005 ms (client timeout 1999 ms). Trace:
W20260709 12:27:54.050740  5635 rpcz_store.cc:269] 0709 12:27:52.044272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.457618 (+413346us) service_pool.cc:224] Handling call
0709 12:27:54.048543 (+1590925us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b58001e2798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:27:54.049822 (+  1279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47777664}
I20260709 12:27:54.037022 13203 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:54.053012 13203 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:54.053457 13203 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:54.018182 13157 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:54.061784 13139 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:54.063030  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1555) took 1998 ms. Trace:
I20260709 12:27:54.063447  5636 rpcz_store.cc:276] 0709 12:27:52.064948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.732369 (+667421us) service_pool.cc:224] Handling call
0709 12:27:52.738366 (+  5997us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:52.738849 (+   483us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:52.738878 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:52.757369 (+ 18491us) log.cc:844] Serialized 26 byte log entry
0709 12:27:52.877714 (+120345us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:52.877772 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:52.879791 (+  2019us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:52.945258 (+ 65467us) raft_consensus.cc:1685] finished
0709 12:27:52.945278 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:54.062748 (+1117470us) spinlock_profiling.cc:243] Waited 5.58 ms on lock 0x7b54000c0b50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:54.062964 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6631,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":7913,"spinlock_wait_cycles":12827392,"thread_start_us":76701,"threads_started":1,"wal-append.queue_time_us":157972,"wal-append.run_cpu_time_us":1567,"wal-append.run_wall_time_us":1002610}
I20260709 12:27:54.066996  5554 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:54.048776  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.070791  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3110) took 1081 ms. Trace:
I20260709 12:27:54.070986  5560 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:54.071213  5554 rpcz_store.cc:276] 0709 12:27:52.989714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:53.741977 (+752263us) service_pool.cc:224] Handling call
0709 12:27:54.070670 (+328693us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.070009 12762 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:53.976862 13370 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.083619 13188 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:27:54.060082 13399 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.762s	user 0.000s	sys 0.003s
I20260709 12:27:54.084218 13188 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.084986 13370 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.085281  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.086328 13188 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.086591  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:54.086704 13370 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:27:54.071962  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.068186 13288 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:54.017997 13365 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.088686 13290 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.088939  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:54.090159  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.011487 13419 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.091444  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.092044  5560 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:27:54.087919  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.027174  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:54.092888  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:54.078544  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:54.080567  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 140637us
I20260709 12:27:54.017547 13308 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.024428 13420 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:54.098739  5704 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 16 index: 7. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
I20260709 12:27:54.116119 13308 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:54.098989  5706 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
W20260709 12:27:54.100183  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 73094us
I20260709 12:27:54.117720  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:54.102311  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.121098  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.122426  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.123585  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:54.124979  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:54.103057  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:54.017666 13388 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.096948  5554 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:54.116137 13420 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.132978 13339 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:54.133257  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:54.078217 13368 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:54.136480  5635 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:54.141489  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.142923  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.109757 13396 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:54.154762  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.158854  5708 raft_consensus.cc:2410] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 22 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:27:54.027513 13183 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.158282  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2846) took 1766 ms. Trace:
I20260709 12:27:54.135473 13276 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:27:54.160554  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:54.163735  5557 rpcz_store.cc:276] 0709 12:27:52.392232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.789113 (+396881us) service_pool.cc:224] Handling call
0709 12:27:52.789464 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:52.886655 (+ 97191us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.158010 (+1271355us) spinlock_profiling.cc:243] Waited 93.3 ms on lock 0x7b54000eb5d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:54.158222 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":214526720}
I20260709 12:27:54.164901  5554 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:27:54.165261  5557 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 12:27:54.165673  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3099) took 1758 ms. Trace:
I20260709 12:27:54.165839  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2847) took 1731 ms. Trace:
I20260709 12:27:54.166033  5554 rpcz_store.cc:276] 0709 12:27:52.407783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.163308 (+1755525us) service_pool.cc:224] Handling call
0709 12:27:54.164683 (+  1375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:54.165433 (+   750us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.165624 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.166203  5557 rpcz_store.cc:276] 0709 12:27:52.434849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.164783 (+1729934us) service_pool.cc:224] Handling call
0709 12:27:54.165054 (+   271us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:54.165627 (+   573us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.165795 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.146904 13139 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.155525  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3109) took 1188 ms. Trace:
I20260709 12:27:54.170436 13420 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:54.171017 13388 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:54.172603 13403 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.173249 13182 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.173659  5559 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:27:54.153331 13368 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.192272  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 1503) took 1253 ms. Trace:
I20260709 12:27:54.174561 13395 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.175038 13419 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:54.208343  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0969148699996367s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19990000000000002s for a response
I20260709 12:27:54.125527 13413 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:54.179658 13429 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:27:54.195036 13368 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.212814  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.213807 13395 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:54.214289  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:54.215790  5627 raft_consensus.cc:2368] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27.
I20260709 12:27:54.181630  5560 rpcz_store.cc:276] 0709 12:27:52.967605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.102413 (+1134808us) service_pool.cc:224] Handling call
0709 12:27:54.155458 (+ 53045us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.217617 13410 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:54.218273 13410 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:54.218672  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2850) took 1783 ms. Trace:
I20260709 12:27:54.219359 13410 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.220345  6236 leader_election.cc:400] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27.
I20260709 12:27:54.220788  6236 leader_election.cc:403] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260709 12:27:54.186214  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:54.190209  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 150869us
I20260709 12:27:54.221854 13267 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:27:54.222223  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:54.223275  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:54.193131  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:54.194556 13276 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:54.224289  5704 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
W20260709 12:27:54.225270 13411 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.670s	user 0.002s	sys 0.000s
W20260709 12:27:54.225751 13430 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:27:54.225334 13276 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.194669  5559 rpcz_store.cc:276] 0709 12:27:52.939343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.009175 (+1069832us) service_pool.cc:224] Handling call
0709 12:27:54.192214 (+183039us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.197137 13182 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.189877 13139 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.226907  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1262) took 2422 ms. Trace:
I20260709 12:27:54.201804  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:54.225409  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:54.228528 13182 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:27:54.228987 13182 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:54.229502 13182 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:54.228035  5559 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:27:54.230496 13181 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:27:54.232164 13334 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:54.205137  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.163560 13398 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.147639 13339 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:54.209748 13313 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
W20260709 12:27:54.180572  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:54.227341  6278 rpcz_store.cc:276] 0709 12:27:51.804867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:51.805482 (+   615us) service_pool.cc:224] Handling call
0709 12:27:51.805881 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:52.668121 (+862240us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.226627 (+1558506us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b54005eef54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:54.226828 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1981501696}
I20260709 12:27:54.228901  5704 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:54.192813 12762 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.218166  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.254103  5627 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:27:54.225749  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:54.199167 13285 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.256269  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:54.257788  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:54.182461  5554 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
I20260709 12:27:54.219166  5560 rpcz_store.cc:276] 0709 12:27:52.435642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.217663 (+1782021us) service_pool.cc:224] Handling call
0709 12:27:54.218048 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.218263 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:54.218303 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:54.218356 (+    53us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:54.218401 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.218469 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:54.218606 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.261381  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2848) took 1826 ms. Trace:
I20260709 12:27:54.262050  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.261839  5554 rpcz_store.cc:276] 0709 12:27:52.435139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.166935 (+1731796us) service_pool.cc:224] Handling call
0709 12:27:54.182242 (+ 15307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.261067 (+ 78825us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.261311 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.264186  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.266163  5554 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: []
I20260709 12:27:54.224249  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:54.210503  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:54.230912  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2852) took 1794 ms. Trace:
I20260709 12:27:54.267552  5559 rpcz_store.cc:276] 0709 12:27:52.436156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.227471 (+1791315us) service_pool.cc:224] Handling call
0709 12:27:54.227743 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.230661 (+  2918us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.230850 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.268782  5559 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:54.269318  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2853) took 1794 ms. Trace:
I20260709 12:27:54.258423  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:54.269910  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:54.269771  5559 rpcz_store.cc:276] 0709 12:27:52.475138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.268302 (+1793164us) service_pool.cc:224] Handling call
0709 12:27:54.268621 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.269091 (+   470us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.269264 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.271667  5559 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:54.272379  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2855) took 1796 ms. Trace:
I20260709 12:27:54.271497  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:54.242615 13313 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:54.262735  5560 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:27:54.282356  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:54.272804  5559 rpcz_store.cc:276] 0709 12:27:52.475746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.271225 (+1795479us) service_pool.cc:224] Handling call
0709 12:27:54.271493 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.272116 (+   623us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.272304 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.280740  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.276473  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:54.289160  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.291584  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:54.308463  5635 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 20: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:54.309731  5704 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:54.273963  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3101) took 1832 ms. Trace:
I20260709 12:27:54.255493 12935 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:54.277694 13365 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:54.336305 13203 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.336596 12935 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.343173 12935 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.337337 13420 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:27:54.346606 13139 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:54.320817 13386 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:27:54.347488  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1563) took 1277 ms. Trace:
I20260709 12:27:54.348901  5630 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:54.337914 13157 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.350914  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1562) took 1281 ms. Trace:
W20260709 12:27:54.352860  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:54.301004  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:54.348006  5633 rpcz_store.cc:276] 0709 12:27:53.069776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.048017 (+978241us) service_pool.cc:224] Handling call
0709 12:27:54.075374 (+ 27357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.092549 (+ 17175us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:54.092589 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:54.092636 (+    47us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:54.092664 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.347125 (+254461us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b54001cedd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:54.347215 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:54.347407 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38838528}
I20260709 12:27:54.357151  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.354087  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:54.359424  5710 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:27:54.359701  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58786.5us
I20260709 12:27:54.277575 13313 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:54.289079 13410 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:54.314152 13419 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:27:54.363360  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 105691us
I20260709 12:27:54.352329 13157 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.275142  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:54.232244 13181 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.057s
I20260709 12:27:54.321610  5554 rpcz_store.cc:276] 0709 12:27:52.441898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.265652 (+1823754us) service_pool.cc:224] Handling call
0709 12:27:54.265976 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.267174 (+  1198us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.273905 (+  6731us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:27:54.372097  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 97048us
I20260709 12:27:54.323685  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.271945 13424 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:54.333809 13339 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:54.351581  5630 rpcz_store.cc:276] 0709 12:27:53.069484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.035948 (+966464us) service_pool.cc:224] Handling call
0709 12:27:54.079016 (+ 43068us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.349507 (+270491us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.350348 (+   841us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b54001de2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:54.350718 (+   370us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":619928448}
I20260709 12:27:54.325310  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.381965  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.278913  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.370018  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:54.280618  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:54.384814  5706 raft_consensus.cc:2368] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22.
I20260709 12:27:54.364331  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:54.279495 13398 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:54.373903  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:54.361814 13203 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.273742 12762 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.361666 13403 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:54.376120 13424 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:54.388563 13181 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:27:54.428488  5561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3097) took 2026 ms (client timeout 2071 ms). Trace:
I20260709 12:27:54.428956  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:54.428969  5561 rpcz_store.cc:269] 0709 12:27:52.402258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.654768 (+252510us) service_pool.cc:224] Handling call
0709 12:27:54.428276 (+1773508us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54001bdd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:54.428432 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2874538880}
I20260709 12:27:54.428773  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2840) took 2056 ms. Trace:
I20260709 12:27:54.278745 13285 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.377957 13267 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27.
I20260709 12:27:54.378491 13313 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.380710  5710 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:27:54.372611  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:54.370707  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:54.386313  5559 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:54.387115  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:54.388769 13182 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.398034  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3112) took 1069 ms. Trace:
I20260709 12:27:54.391431  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.437631  5555 rpcz_store.cc:276] 0709 12:27:53.328267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:53.942689 (+614422us) service_pool.cc:224] Handling call
0709 12:27:54.397814 (+455125us) spinlock_profiling.cc:243] Waited 41.4 ms on lock 0x7b54000271d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:54.397962 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":95151360}
I20260709 12:27:54.400228 13183 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
W20260709 12:27:54.400686  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 144493us
I20260709 12:27:54.400923  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:54.442212  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:54.401074  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:54.401631  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:54.403451  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:54.402369 13339 raft_consensus.cc:2764] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:27:54.365556 13419 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:54.437208  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:54.375481 13365 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:54.443256  5704 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:27:54.439992  5559 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:27:54.446336  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1497) took 1778 ms. Trace:
I20260709 12:27:54.446728  5559 rpcz_store.cc:276] 0709 12:27:52.667764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.439485 (+1771721us) service_pool.cc:224] Handling call
0709 12:27:54.439765 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.446125 (+  6360us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.446294 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:27:54.440483  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53513.2us
I20260709 12:27:54.441043  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.426285 13424 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:54.427229 13181 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.449236  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:54.443065  5560 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:54.442979  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:54.444773  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1498) took 1776 ms. Trace:
I20260709 12:27:54.431838  5558 rpcz_store.cc:276] 0709 12:27:52.372159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.372311 (+   152us) service_pool.cc:224] Handling call
0709 12:27:52.413650 (+ 41339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:52.626004 (+212354us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:27:52.645706 (+ 19702us) log.cc:844] Serialized 14 byte log entry
0709 12:27:52.839522 (+193816us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:52.839593 (+    71us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:52.839654 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.428534 (+1588880us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54003c41d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:54.428597 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:54.428718 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":135305,"prepare.queue_time_us":19944,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":126115,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":29402,"spinlock_wait_cycles":287154176,"thread_start_us":153963,"threads_started":1,"wal-append.queue_time_us":284648,"wal-append.run_cpu_time_us":2566,"wal-append.run_wall_time_us":1429865}
I20260709 12:27:54.456276  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1499) took 1673 ms. Trace:
I20260709 12:27:54.465186  5561 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 12:27:54.466226  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1500) took 1683 ms. Trace:
I20260709 12:27:54.453058  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2856) took 1977 ms. Trace:
I20260709 12:27:54.456878  5554 rpcz_store.cc:276] 0709 12:27:52.668001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.443958 (+1775957us) service_pool.cc:224] Handling call
0709 12:27:54.444265 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.444493 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:54.444509 (+    16us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:54.444539 (+    30us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:54.444559 (+    20us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.444603 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:54.444718 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.446769 13276 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:54.446244 13419 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:54.436939 13384 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:54.478174  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:54.478844  5710 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:54.466727  5561 rpcz_store.cc:276] 0709 12:27:52.782828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.453860 (+1671032us) service_pool.cc:224] Handling call
0709 12:27:54.454166 (+   306us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:54.465861 (+ 11695us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.466151 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.466904  5555 rpcz_store.cc:276] 0709 12:27:52.476008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.443907 (+1967899us) service_pool.cc:224] Handling call
0709 12:27:54.452710 (+  8803us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.452850 (+   140us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:54.452865 (+    15us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:54.452892 (+    27us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:27:54.452916 (+    24us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.452947 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:54.453011 (+    64us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.464996  5559 rpcz_store.cc:276] 0709 12:27:52.782488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.455635 (+1673147us) service_pool.cc:224] Handling call
0709 12:27:54.455911 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.456064 (+   153us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:54.456079 (+    15us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:54.456107 (+    28us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:54.456126 (+    19us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.456165 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:54.456232 (+    67us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.457348 13203 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:54.472097 13142 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.003s
I20260709 12:27:54.463418 13358 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:54.474233  5554 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:27:54.489632  5710 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:27:54.489975  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1505) took 1042 ms. Trace:
I20260709 12:27:54.449012 13188 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:27:54.485287  5561 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:54.487998  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:54.490339  5554 rpcz_store.cc:276] 0709 12:27:53.447533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.473588 (+1026055us) service_pool.cc:224] Handling call
0709 12:27:54.473901 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.489700 (+ 15799us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.489915 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.490813  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1501) took 1707 ms. Trace:
I20260709 12:27:54.492034  5704 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:27:54.445761 13365 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:54.492285  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.463986 13313 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:54.491743  5560 rpcz_store.cc:276] 0709 12:27:52.783136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.456879 (+1673743us) service_pool.cc:224] Handling call
0709 12:27:54.490330 (+ 33451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.490498 (+   168us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:54.490522 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:54.490564 (+    42us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:54.490605 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.490669 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:54.490763 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.460846 12935 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:27:54.464705 13181 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:27:54.494149 13448 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:27:54.499771  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1504) took 1054 ms. Trace:
I20260709 12:27:54.489656 13203 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:54.461807 13182 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.485384 13276 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.503495 13319 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:54.453536 13454 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:27:54.507668  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44658.8us
I20260709 12:27:54.508654  5558 rpcz_store.cc:276] 0709 12:27:53.445580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.473459 (+1027879us) service_pool.cc:224] Handling call
0709 12:27:54.498877 (+ 25418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.499136 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:54.499163 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:54.499212 (+    49us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:27:54.499269 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.499494 (+   225us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:54.499691 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:54.473152 13124 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:54.492888 13188 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.492981 13358 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:54.460021 13312 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:27:54.501729 13365 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:27:54.506489  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48900.8us
I20260709 12:27:54.503710 12935 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:54.519433  5524 leader_election.cc:400] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22.
I20260709 12:27:54.528990  5524 leader_election.cc:403] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:27:54.520253  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.513055 13203 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:54.492707 13384 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.525687  5555 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:54.517010 13319 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:54.530371  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:27:54.526194 13450 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 12:27:54.516947 13276 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:54.520244 12762 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:54.539290  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:54.540198 13447 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:27:54.511387 13409 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.570181 13409 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.525905 13188 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:54.530683 13398 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:54.572873 13398 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.574460 13384 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:54.576792  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.577909  5558 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 20: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:54.579540 13398 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:54.536963 13370 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22.
I20260709 12:27:54.580647 12935 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.540930 13157 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:54.582286 12935 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:27:54.582639 13452 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260709 12:27:54.583453 13404 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:54.583956 13404 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:54.572459  5633 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:27:54.587242  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:54.591105  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1558) took 2365 ms. Trace:
I20260709 12:27:54.591516  5632 rpcz_store.cc:276] 0709 12:27:52.226200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.982474 (+756274us) service_pool.cc:224] Handling call
0709 12:27:52.982842 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:53.341242 (+358400us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.590722 (+1249480us) spinlock_profiling.cc:243] Waited 348 ms on lock 0x7b5400439c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:54.591000 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":799260416}
I20260709 12:27:54.593883  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.592837  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.595057  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:54.597385  5632 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:54.598565  5633 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:54.599308  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:54.600572 12973 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:54.600889  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:54.601730  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1374) took 1081 ms. Trace:
I20260709 12:27:54.602034  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.602262 12973 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:54.602108  5701 rpcz_store.cc:276] 0709 12:27:53.520374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:53.997090 (+476716us) service_pool.cc:224] Handling call
0709 12:27:53.997452 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.237747 (+240295us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:54.237797 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:54.237892 (+    95us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:27:54.237943 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.601470 (+363527us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b54003ad4d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:54.601534 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:54.601678 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":551968512}
I20260709 12:27:54.603566 12973 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:54.550678  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.604601  5632 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:54.605767  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:54.605970  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:54.607043  5556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3106) took 1938 ms (client timeout 1578 ms). Trace:
W20260709 12:27:54.607443  5556 rpcz_store.cc:269] 0709 12:27:52.668235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.675128 (+  6893us) service_pool.cc:224] Handling call
0709 12:27:54.606856 (+1931728us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b540044f554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:54.606947 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2272154880}
I20260709 12:27:54.561832 13420 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:27:54.563758  5635 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:27:54.601009 13122 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:54.607828  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:54.613320  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.614683  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.615403  5633 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:27:54.620939  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:54.622113  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:54.563009 13285 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.623831  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:54.615018  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:54.631736  5633 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:54.631212  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:54.643204  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:54.610877 13334 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:54.650192  5633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:54.653422  5633 raft_consensus.cc:606] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:54.628221 13288 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:27:54.655514  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 123900us
I20260709 12:27:54.630179 13398 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:54.593295 13458 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.089s
I20260709 12:27:54.634096 13122 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:54.633471 13157 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:54.647663  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:54.658355  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.658658  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.661358 13122 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.664850  5558 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 18: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:54.665786  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:54.666827  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.667843  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.668535  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:54.660066  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.672729 13435 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:54.673063  5560 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:54.673326 13435 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.675561 13435 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.640014 13433 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:54.676115 13433 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.676679 13433 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:54.678261  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:54.679025  5705 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:54.600565 13384 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.051s
I20260709 12:27:54.662248  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1557) took 2596 ms. Trace:
I20260709 12:27:54.672732  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.661763  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.681411  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:54.682052  5710 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:54.664773 13122 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:54.688498  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.680670  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:54.681613  5559 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:54.689414  5556 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:54.655150 13396 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:54.680995  5628 rpcz_store.cc:276] 0709 12:27:52.066001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.735409 (+669408us) service_pool.cc:224] Handling call
0709 12:27:52.735808 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:52.736212 (+   404us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:52.736239 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:52.739233 (+  2994us) log.cc:844] Serialized 26 byte log entry
0709 12:27:52.943735 (+204502us) spinlock_profiling.cc:243] Waited 5.74 ms on lock 0x7b48002727d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:27:54.553095 (+1609360us) spinlock_profiling.cc:243] Waited 7 ms on lock 0x7b180002d880. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:27:54.618173 (+ 65078us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:54.618243 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.618378 (+   135us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:54.662033 (+ 43655us) raft_consensus.cc:1685] finished
0709 12:27:54.662053 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:54.662197 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10728,"raft.queue_time_us":347,"raft.run_cpu_time_us":2342,"raft.run_wall_time_us":197365,"spinlock_wait_cycles":29306368,"thread_start_us":64802,"threads_started":1,"wal-append.queue_time_us":1919018}
I20260709 12:27:54.697243  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:54.681723  5561 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:54.685531  5632 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:27:54.695521  5554 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:54.701967  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:54.703495  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:54.695317  5630 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
W20260709 12:27:54.692719  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45150.4us
I20260709 12:27:54.708632  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:54.712453  5632 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:27:54.722236  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:54.723274  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1570) took 1059 ms. Trace:
I20260709 12:27:54.723532  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:54.722606  5632 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:27:54.722515  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:54.654815 13388 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 12:27:54.696218 13396 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.639086 13319 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.723856  5636 rpcz_store.cc:276] 0709 12:27:53.663418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.066179 (+402761us) service_pool.cc:224] Handling call
0709 12:27:54.066429 (+   250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.066617 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:27:54.093588 (+ 26971us) log.cc:844] Serialized 14 byte log entry
0709 12:27:54.675058 (+581470us) spinlock_profiling.cc:243] Waited 6.06 ms on lock 0x7b1800453220. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:27:54.722855 (+ 47797us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:54.722933 (+    78us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:54.722989 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.723051 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:54.723206 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":26678,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":3251,"raft.queue_time_us":115,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"spinlock_wait_cycles":13947136,"thread_start_us":47537,"threads_started":1}
I20260709 12:27:54.661571 13419 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.688783 12973 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.696753 13398 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:54.730708  5636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:54.716528 13334 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.716750 13186 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.719872 13157 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:27:54.734400  5636 raft_consensus.cc:606] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:27:54.735220  5627 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
W20260709 12:27:54.736712  5631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2866) took 1982 ms (client timeout 1578 ms). Trace:
I20260709 12:27:54.744628  5636 raft_consensus.cc:614] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:27:54.744837  5631 rpcz_store.cc:269] 0709 12:27:52.754343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.801790 (+ 47447us) service_pool.cc:224] Handling call
0709 12:27:54.736340 (+1934550us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b540008c0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:54.736493 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2118213888}
I20260709 12:27:54.746908  5631 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:27:54.721092 13460 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:54.730695 13370 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:54.739814 13435 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:27:54.760511 13435 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:54.760788 13370 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:54.761299 13370 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:54.761443 13435 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.762240 13370 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.762728 13435 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:27:54.721272 13458 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:54.750150 13460 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:54.767277 13460 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.768556 13460 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:54.769161  5631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:27:54.756513  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:54.772832  5554 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:27:54.758710 12786 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:27:54.759604 13317 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:27:54.773736  5631 raft_consensus.cc:606] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:54.727581 13124 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:54.730178 13142 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:54.786624  5636 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:54.793762  5631 raft_consensus.cc:614] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:27:54.732331 13396 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:54.739331 13398 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:54.749841 13388 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:54.840157  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3116) took 1969 ms. Trace:
I20260709 12:27:54.841209  5634 rpcz_store.cc:276] 0709 12:27:52.870408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:53.415737 (+545329us) service_pool.cc:224] Handling call
0709 12:27:53.416143 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:53.994410 (+578267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.839839 (+845429us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b540018bd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:54.840080 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1212765056}
W20260709 12:27:54.806867  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:54.811007 13188 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.812460 13339 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.752856 13334 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.775480 13319 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:54.828506 13398 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:54.774472 13186 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.805608 13183 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:54.801203 13142 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.851787  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:27:54.863329 13142 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:54.873301 13419 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.852280 13339 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.878679 13186 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:54.879914  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:54.882413  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:54.880861  5702 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
W20260709 12:27:54.885836  6279 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1269) took 2450 ms (client timeout 1916 ms). Trace:
I20260709 12:27:54.887192  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:54.888046  5706 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:54.878281 13188 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:54.886372  6279 rpcz_store.cc:269] 0709 12:27:52.435498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.440450 (+  4952us) service_pool.cc:224] Handling call
0709 12:27:54.885578 (+2445128us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400711954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:54.885739 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2405867264}
I20260709 12:27:54.898048  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2849) took 2462 ms. Trace:
I20260709 12:27:54.861011 13183 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:54.899837  5557 rpcz_store.cc:276] 0709 12:27:52.435401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.167057 (+1731656us) service_pool.cc:224] Handling call
0709 12:27:54.181213 (+ 14156us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:54.181639 (+   426us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:54.181673 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:54.191553 (+  9880us) log.cc:844] Serialized 26 byte log entry
0709 12:27:54.590251 (+398698us) spinlock_profiling.cc:243] Waited 2.17 ms on lock 0x7b18002c9740. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:27:54.717992 (+127741us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:27:54.719187 (+  1195us) log.cc:844] Serialized 14 byte log entry
0709 12:27:54.887637 (+168450us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.887824 (+   187us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:54.887886 (+    62us) raft_consensus.cc:1685] finished
0709 12:27:54.897825 (+  9939us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:54.897979 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8155,"prepare.queue_time_us":3568,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":45852,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":9050,"spinlock_wait_cycles":5957248,"thread_start_us":287484,"threads_started":2,"wal-append.queue_time_us":587447}
W20260709 12:27:54.903954  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 200564us
I20260709 12:27:54.913834  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:54.889055 13368 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:54.898218 13186 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:27:54.916971  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:54.917779  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:54.929886 13188 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:27:54.869788 13433 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.932253  5706 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:27:54.887192 13424 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.930413 13460 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.011s
I20260709 12:27:54.919188 13186 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.910735 13365 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.931058 13463 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:27:54.941655 13183 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.952678 13183 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:54.954262 13424 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.933552 13276 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.957006 13339 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:54.957427 13276 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:54.944911 13186 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:54.963084 13433 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:54.948101 13463 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:54.965932 13358 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.964524 13334 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:54.970404 13288 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:54.986135 13322 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:54.989635 13122 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.992026 13122 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:54.959954 13181 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:54.979396 13463 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:27:54.968116 13404 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:54.999715 13325 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:55.004268 13396 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.005087 13365 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:55.006757 13404 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:55.044275  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:55.049029  5557 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:27:55.049364  5552 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
W20260709 12:27:55.049980  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:55.051128  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.053879  5631 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
W20260709 12:27:55.061306  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.061903  5704 raft_consensus.cc:2410] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 24 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:27:55.062861 13181 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:55.061183  5633 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: [18.14-21.15]
I20260709 12:27:55.066987  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:54.978503  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.071969 13322 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:27:55.076867 13322 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:55.077919 13287 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:27:55.078773 13287 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:55.080999  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:55.081496 13142 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.089923 13404 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.002s
I20260709 12:27:55.094419  5554 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:55.095988  6279 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
I20260709 12:27:55.090016  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:55.079520 13322 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.107679  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:55.092344  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 118900us
I20260709 12:27:55.116987 13266 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:55.005333 13388 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:55.120086  5705 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:55.122375 13142 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:55.122525  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:55.123523  5634 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:55.124505  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.125412  6278 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:27:55.126055  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.126765  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.127557  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.128113  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.131738  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.133111  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.134344  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:55.097117 13398 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.134478 13435 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:55.140929  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:55.143219  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.147089  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.147181  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:55.150342  5632 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:27:55.151461  6273 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:27:55.152801  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.153331 13435 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.155501 13383 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:27:55.156793  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1572) took 1458 ms. Trace:
I20260709 12:27:55.157419  5633 rpcz_store.cc:276] 0709 12:27:53.698634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.661064 (+962430us) service_pool.cc:224] Handling call
0709 12:27:54.661648 (+   584us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:55.067183 (+405535us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:55.152251 (+ 85068us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b5400439254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:55.156697 (+  4446us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":916933504}
I20260709 12:27:55.162068 13468 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:55.162940 13468 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:55.164675  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:55.166464  5706 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
W20260709 12:27:55.167230  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.134349 13287 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.167459  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.156020 13383 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:55.168566  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:55.148528  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.175238  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3098) took 2771 ms. Trace:
I20260709 12:27:55.176988  5553 rpcz_store.cc:276] 0709 12:27:52.402574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.810535 (+407961us) service_pool.cc:224] Handling call
0709 12:27:53.920060 (+1109525us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:53.999889 (+ 79829us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:55.174054 (+1174165us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540016ba50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:55.174320 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2551028096}
I20260709 12:27:55.181915  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3114) took 2336 ms. Trace:
I20260709 12:27:55.184463  5702 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:55.185798  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2987) took 1677 ms. Trace:
I20260709 12:27:55.189294  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.190443  6273 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:55.191720  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.182549  5629 rpcz_store.cc:276] 0709 12:27:52.834399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:53.392182 (+557783us) service_pool.cc:224] Handling call
0709 12:27:53.392626 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:53.555255 (+162629us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:27:53.898624 (+343369us) log.cc:844] Serialized 14 byte log entry
0709 12:27:53.995599 (+ 96975us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:27:53.995688 (+    89us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:53.995771 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.921645 (+925874us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b180008c140. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419
0709 12:27:55.118874 (+197229us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b540008bbd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:55.129434 (+ 10560us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:55.170490 (+ 41056us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3545,"prepare.queue_time_us":362777,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":110,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":8889,"spinlock_wait_cycles":381691648,"thread_start_us":427521,"threads_started":2,"wal-append.queue_time_us":210199}
I20260709 12:27:55.193454  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.194507  6279 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:27:55.195078  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:55.163443 13203 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.609s	user 0.014s	sys 0.001s
W20260709 12:27:55.196456  6237 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.754s (SENT) [suppressed 28 similar messages]
I20260709 12:27:55.196880 13203 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.199582 13386 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:55.200847 13386 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.202601 13122 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:27:55.204221 13157 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:55.204721 13157 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:55.206537  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:55.207543  6274 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:55.208914  6280 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 16: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:55.193349  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 2998) took 1679 ms. Trace:
I20260709 12:27:55.186409  5703 rpcz_store.cc:276] 0709 12:27:53.507863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:53.508425 (+   562us) service_pool.cc:224] Handling call
0709 12:27:53.508732 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:54.034120 (+525388us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:54.034153 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:54.034630 (+   477us) log.cc:844] Serialized 26 byte log entry
0709 12:27:54.202919 (+168289us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:27:54.202972 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:55.184814 (+981842us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b5400324454. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:55.184932 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:55.185024 (+    92us) raft_consensus.cc:1685] finished
0709 12:27:55.185042 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:55.185512 (+   470us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22920,"raft.queue_time_us":23276,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":6677,"spinlock_wait_cycles":847910784,"thread_start_us":301062,"threads_started":2,"wal-append.queue_time_us":167910}
I20260709 12:27:55.175257  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.212203  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:55.213792  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:55.213513  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:55.200528 13203 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:55.214629  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.226233  5552 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:27:55.228811  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:55.226122  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:55.232095  5629 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:55.230732  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:55.228636 13267 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:55.234011  6274 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:55.234310 13267 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:55.234644  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:55.235491  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.237525  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.230183  6275 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:27:55.206156 13157 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.238816  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:55.243439  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:55.243842  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:55.244385  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:55.240427  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:55.248636  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52384.4us
I20260709 12:27:55.250849  5631 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:55.255897  6280 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:55.261221  5561 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:55.264294  6278 raft_consensus.cc:2410] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 21 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:27:55.267994  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:55.271369  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:55.269167  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:55.210203  5707 rpcz_store.cc:276] 0709 12:27:53.513858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:53.694311 (+180453us) service_pool.cc:224] Handling call
0709 12:27:53.869709 (+175398us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:53.899619 (+ 29910us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:53.899672 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:53.901662 (+  1990us) log.cc:844] Serialized 52 byte log entry
0709 12:27:54.094522 (+192860us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:27:54.106374 (+ 11852us) log.cc:844] Serialized 14 byte log entry
0709 12:27:54.106646 (+   272us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:54.106748 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:54.232809 (+126061us) raft_consensus.cc:1685] finished
0709 12:27:54.232830 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:55.192831 (+960001us) spinlock_profiling.cc:243] Waited 83.2 ms on lock 0x7b5400210550. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:55.193111 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29337,"prepare.queue_time_us":11648,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":30917,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":12310,"spinlock_wait_cycles":191259904,"thread_start_us":150280,"threads_started":1,"wal-append.queue_time_us":267027}
W20260709 12:27:55.275094  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:55.278357  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.275782 13157 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:55.280246  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:55.281611  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2862) took 1372 ms. Trace:
I20260709 12:27:55.282053  5555 rpcz_store.cc:276] 0709 12:27:53.900720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.481360 (+580640us) service_pool.cc:224] Handling call
0709 12:27:54.494844 (+ 13484us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:54.535245 (+ 40401us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:55.272690 (+737445us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x7b54003a8ed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:55.273145 (+   455us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":70467840}
I20260709 12:27:55.289099  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3122) took 1206 ms. Trace:
I20260709 12:27:55.289604  5630 rpcz_store.cc:276] 0709 12:27:54.082875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.605436 (+522561us) service_pool.cc:224] Handling call
0709 12:27:55.288713 (+683277us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b540032d7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:55.288912 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":53019392}
I20260709 12:27:55.290318 13396 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:55.232414 13468 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.291224 13398 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:55.292958  5627 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:27:55.293076 13383 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.294665  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2881) took 1090 ms. Trace:
I20260709 12:27:55.295113 13383 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:27:55.296221  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.297147  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:55.289191  5628 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.228663  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:55.301807 13358 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:55.302881 13267 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.303635 13468 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:55.304733  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.307425  5633 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:55.310586  5633 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:27:55.314842  5631 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:27:55.315306  5707 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:27:55.318476  5706 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:27:55.319835  5561 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:27:55.320207 13288 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:27:55.263051 13435 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:55.295125  5627 rpcz_store.cc:276] 0709 12:27:54.204048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.752198 (+548150us) service_pool.cc:224] Handling call
0709 12:27:55.292392 (+540194us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:55.293550 (+  1158us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:55.294422 (+   872us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b540032d7d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:55.294575 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1240715136}
I20260709 12:27:55.304189 13267 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:55.269273 13386 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:55.335134  5707 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
I20260709 12:27:55.335714  5554 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:27:55.338002  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:55.340683  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 71127.1us
I20260709 12:27:55.386088  5556 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:55.386636  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:55.387876 13271 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:27:55.388530 13203 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:27:55.388831 13404 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:55.389570 13404 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:55.390292 12935 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:27:55.390714 12935 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:55.340920  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:55.392428 13203 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:55.392820  5706 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:55.392879 13203 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:55.393955 13203 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.395474 13203 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:55.342072 13288 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:55.397189 13288 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.399197  5629 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:55.400568  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2882) took 1193 ms. Trace:
I20260709 12:27:55.400619  5631 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:55.401010  5632 rpcz_store.cc:276] 0709 12:27:54.207245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.752229 (+544984us) service_pool.cc:224] Handling call
0709 12:27:55.147769 (+395540us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:55.150784 (+  3015us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:55.400316 (+249532us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b540018d650. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:55.400495 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":864473984}
I20260709 12:27:55.391868 12935 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.403673 12935 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:55.388074 13266 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.017s
I20260709 12:27:55.405546 13287 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:55.387979  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:55.409755  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:55.409869  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:55.410715 13271 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:55.348937  5630 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:55.352805  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:55.411804 13271 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:55.348690  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:55.354259  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:55.413789  5703 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:55.354677  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:55.356592  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.368137  5701 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:55.414938  5708 raft_consensus.cc:2368] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24.
I20260709 12:27:55.368870  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:55.415813  5705 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:55.353652  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:55.416813  5702 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:55.373025 13413 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:55.424654 13386 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:27:55.425138  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.382247  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:55.426041  5557 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:55.428308  5706 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:27:55.435478  6276 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:27:55.442353  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:55.443856  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:55.444327  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:55.444800  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:55.445266  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:55.447490  6276 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:27:55.449417  5708 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 12:27:55.449824  5701 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: [18.14-21.15]
I20260709 12:27:55.448625  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:55.412932 13271 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:55.456691 13271 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:55.464570 13404 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.484627 13287 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260709 12:27:55.485285  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 91782.3us
I20260709 12:27:55.486716 13395 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:55.487180 13395 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:55.486907 12786 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:27:55.488170 12786 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:55.488241 13395 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.489573 12786 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:55.490056 12786 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:55.490191 13317 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:55.490711 13317 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:27:55.491820  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.491346  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:55.491401 12786 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.492198  6237 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:55.493108 13475 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:55.491755 13317 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.493185 13395 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:55.488616  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:55.410805  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:55.497565 13260 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:27:55.493124  5705 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:55.488041  5555 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:55.519346  6274 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
W20260709 12:27:55.520588  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 109908us
I20260709 12:27:55.524129  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:55.526702  5557 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:27:55.533205  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:55.520071 13475 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:27:55.534209 13358 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:55.534821 13358 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.536991  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:55.537839  5630 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:55.540017 12786 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:55.540174  5627 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:27:55.540635  5705 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:27:55.545467 13358 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:55.550630  6278 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:27:55.552562 13383 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:27:55.553422 13319 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:55.558147  6273 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:55.561873 13435 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:27:55.562058 13463 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.563s	user 0.018s	sys 0.000s
I20260709 12:27:55.562512 13435 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:55.563009 13435 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:55.562700 13463 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.565039 13317 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:55.565215 13463 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:55.570065  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:55.576511  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:55.531096 13203 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:55.557785 13404 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:55.556031 13475 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:55.580155 13203 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:55.590988  5557 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:27:55.590991 13203 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:27:55.596732  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.597719  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:55.600592  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:27:55.607686  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 116497us
I20260709 12:27:55.610396  5705 raft_consensus.cc:2368] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27.
I20260709 12:27:55.610438  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:55.613965  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:55.615156 13398 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:27:55.615914 13398 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:55.617228 13475 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.619143 13475 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:27:55.621938  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 94124.8us
I20260709 12:27:55.601329 13404 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:55.631388  5557 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:27:55.631111  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:55.622184 13463 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:55.644271  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:55.660048  5702 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:27:55.639338 13386 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:27:55.672577  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.645292 13463 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:55.681109  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:55.683751  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 147539us
I20260709 12:27:55.665493 13319 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:55.675917 13386 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:55.677395 13396 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:27:55.684778 13183 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.632s	user 0.011s	sys 0.011s
W20260709 12:27:55.712301  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 190967us
I20260709 12:27:55.697630 13386 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:55.729071  6271 raft_consensus.cc:2410] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 23 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 16.
I20260709 12:27:55.730556 13489 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:55.731256 13489 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:55.732604 13489 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.733770 13489 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:27:55.736227  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 247745us
I20260709 12:27:55.712102 13271 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:55.736809  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 189451us
I20260709 12:27:55.736941 13463 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.738241  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:55.738780 13271 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:55.739452 13325 raft_consensus.cc:2764] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:27:55.741003 13403 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:55.741894 13403 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.742025 13404 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.744761 12762 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:27:55.745383 12762 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:55.746387 12762 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.747773 13398 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:27:55.748486 12753 raft_consensus.cc:993] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:27:55.749255 12753 raft_consensus.cc:1081] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Signalling peer f613ba929b1046d3b5aed909414a95bf to start an election
W20260709 12:27:55.749564 13186 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.778s	user 0.018s	sys 0.005s
W20260709 12:27:55.724093  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.721475  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:55.751461  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 192476us
I20260709 12:27:55.718014 13157 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:55.756644  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:55.759055  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:55.762485  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:55.763193  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.722635 13396 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:55.750092 13186 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.773300 13468 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:27:55.782524 13482 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:27:55.783867  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:55.784054  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 96342.4us
I20260709 12:27:55.782625 13468 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:55.791081 13468 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:55.791446  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:55.792598 12935 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:55.792753 13368 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:27:55.795389 13370 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:27:55.798624 13325 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:55.799336 13325 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:55.799834 13325 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:55.800510  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:55.806488  5459 catalog_manager.cc:5697] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 12 to 20. New cstate: current_term: 20 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:55.824461  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 85230.9us
I20260709 12:27:55.771953 13403 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:55.830369  5634 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
W20260709 12:27:55.830994  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 138771us
I20260709 12:27:55.834700  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:55.835067  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 73826.3us
I20260709 12:27:55.835737 13487 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:27:55.836238 13487 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:55.836611 13487 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:55.841958 13267 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:55.842487 13267 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:55.843462 13267 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.827205 13398 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:55.853017 13398 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.855399 13398 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:55.857215  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:55.857952  5557 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:27:55.859479  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:55.860450 13398 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:55.860894 13398 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:55.861331 13398 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:55.819706 12935 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:27:55.767416 13260 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.058s
W20260709 12:27:55.872792  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.880455 13288 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:55.880743  5635 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: []
I20260709 12:27:55.881846  5555 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
W20260709 12:27:55.885073  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 50534.9us
I20260709 12:27:55.887632  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:27:55.893849 13481 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260709 12:27:55.901539  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:55.902316  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:55.823948 13157 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:27:55.903103  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 69814us
W20260709 12:27:55.900763  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 108614us
I20260709 12:27:55.910741  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:55.916666  5555 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 25. Current term is 26. Ops: []
I20260709 12:27:55.921245 12935 raft_consensus.cc:993] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:27:55.922047 12935 raft_consensus.cc:1081] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Signalling peer cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:27:55.828948 13319 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.882741 13490 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 12:27:55.929421  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:55.932677  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45127us
I20260709 12:27:55.938982 13264 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:27:55.943943  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60719.1us
I20260709 12:27:55.948098 13186 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:55.948822  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:55.946938 13403 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:55.961704 13403 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.962738 13266 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:55.950475 13203 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.950385  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:55.964293  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:55.942170 13454 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:55.926566 13487 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.936884  5459 catalog_manager.cc:5697] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 11 to 19, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 19 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:55.964339 13266 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.024s
I20260709 12:27:55.952714 13388 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:55.942293 13396 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.991560 13388 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:55.944640 13492 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:27:55.991587  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:55.952208  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:55.997217  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:55.915786 13325 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:55.952925 13267 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:55.967119 13203 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:55.943460 13490 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:27:55.964304 13403 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:55.996855 12753 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:27:56.032102  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:56.031878  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:56.032704  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:56.036304  5556 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:27:55.995954 13463 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:55.952409 13413 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.082s
I20260709 12:27:56.006749 13186 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:56.018240 13157 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:55.986617 13435 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:55.939544 13287 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.095s
I20260709 12:27:56.058554  5557 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:27:55.966431 13398 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:56.044853 13260 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:27:56.052275 13490 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:56.074015 13325 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:56.084479 13288 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:27:56.087447 13468 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.102461  5706 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:56.015781 13368 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:56.088295 13454 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:27:56.090227  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 126094us
I20260709 12:27:56.091970 13203 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:56.090664  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 119703us
I20260709 12:27:56.090756 13487 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:56.091285  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 94188.1us
W20260709 12:27:56.092306  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68339.3us
I20260709 12:27:56.091763 13157 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.063283 13403 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:56.095335 13370 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.119107  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:56.120343  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 168181us
W20260709 12:27:56.121873  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:56.130045  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:56.119141 13395 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.610s	user 0.007s	sys 0.007s
I20260709 12:27:56.101781 13186 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.143931  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:56.123140 13454 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:56.150287  5556 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:27:56.144645 13403 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.152324 13368 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.153187 13403 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:56.153960 13398 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:56.113687 13492 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.029s
I20260709 12:27:56.154887 13492 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:27:56.155258 13492 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:56.107450 13494 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:27:56.139853 12753 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:27:56.176524  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:56.114547 13203 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.116205 13319 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
I20260709 12:27:56.110838 13463 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.121627 13157 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.194011  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:56.205722  5556 pending_rounds.cc:85] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 11
I20260709 12:27:56.206173  5556 pending_rounds.cc:107] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:27:56.206473  5556 raft_consensus.cc:2889] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:56.142226 13496 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.108s
I20260709 12:27:56.268867  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:56.246394  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:56.270632  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:27:56.272598  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 131213us
W20260709 12:27:56.252578  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 117137us
W20260709 12:27:56.277432  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 327150us
I20260709 12:27:56.271780  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:56.281087  5561 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:27:56.253639  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 59737.1us
I20260709 12:27:56.258926  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:56.274552  5558 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:56.287953  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:56.143844 12935 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 12:27:56.138056 13370 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.293255  5601 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:56.271003  5459 catalog_manager.cc:5697] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 13 to 17, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 17 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:27:56.295287  5705 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
W20260709 12:27:56.295950  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:56.309469  5703 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:56.208266 13487 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:56.253126 13508 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.318432  5557 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:27:56.249192 13404 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:56.317297  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:56.247825 13203 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:56.253016 13396 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:56.259941 13495 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.253415 13322 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.342219 13264 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:27:56.352722 13386 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.592s	user 0.006s	sys 0.005s
I20260709 12:27:56.353361 13386 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:56.354322 13435 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:56.323043 13487 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:56.355463 13487 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:56.255694  5552 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:49044
I20260709 12:27:56.356503  5552 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:56.356889  5552 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:56.360327 13157 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:56.366664 13498 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.448s	user 0.003s	sys 0.000s
I20260709 12:27:56.202270 13492 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:56.184490 13325 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:27:56.368373 12786 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:56.369024  5707 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:56.368906 12786 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:56.370707 13404 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.370199 12786 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:56.372367 13487 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:56.372901 13404 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:56.373456 12786 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:56.374257  5459 catalog_manager.cc:5697] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 11 to 20. New cstate: current_term: 20 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:56.375602  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:56.376467 13396 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.379484 13396 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:56.318888  5702 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:38468
I20260709 12:27:56.380671  5702 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:56.201289 13490 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:56.381242  5702 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:56.380872 13487 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:56.203222 12753 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:56.383072 12753 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:56.258267 13266 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.071s
I20260709 12:27:56.400569  5553 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:56.376042  5459 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 12 to 17, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 17 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:56.405351  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:56.553548  5709 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:27:56.341636 13508 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:56.411289  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 118092us
W20260709 12:27:56.551599  5456 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 277) took 71872 ms (client timeout 15000 ms). Trace:
W20260709 12:27:56.513058  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:56.552814  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:56.370282 13454 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.546712 13495 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:56.582461  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 439252us
I20260709 12:27:56.554292 13322 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:56.531932  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:56.343415 12935 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.677815 13271 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:27:56.696051  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:56.724830  5635 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:27:56.711745 13388 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.719s	user 0.010s	sys 0.015s
W20260709 12:27:56.706586  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 59601us
I20260709 12:27:56.645071  5702 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:56.711040  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:56.718672  5459 catalog_manager.cc:5697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 4 to 19, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 19 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:27:56.720906  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:56.697569  5552 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.675643 13494 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:56.725396  5635 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:56.568605 13264 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:27:56.736397  5556 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.013s	sys 0.021s
W20260709 12:27:56.737393  5556 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.013s	sys 0.021s
W20260709 12:27:56.738374  5526 leader_election.cc:336] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 2.081s (SENT)
I20260709 12:27:56.738878  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:56.741492  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:56.745733  5552 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:56.746352 13516 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:56.747089 13516 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:56.734838 13264 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:56.750238 13516 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:56.749778 13264 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.745687 13370 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:56.745905  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 424190us
I20260709 12:27:56.756083  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1381) took 1186 ms. Trace:
I20260709 12:27:56.784194  5706 rpcz_store.cc:276] 0709 12:27:55.575502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:55.576319 (+   817us) service_pool.cc:224] Handling call
0709 12:27:55.576815 (+   496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:56.102869 (+526054us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:56.755855 (+652986us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b5400067054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:56.756024 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1208436096}
I20260709 12:27:56.729081 13260 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:27:56.723538 12935 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.788200  5459 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 11 to 20, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 20 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:56.733124 13388 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:27:56.799377  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 469704us
I20260709 12:27:56.736775 13404 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:56.752140 13516 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:56.811358 13510 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:27:56.805759  5459 catalog_manager.cc:5697] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 14 to 18, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 18 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:56.831256 13515 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:27:56.831415 13358 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:27:56.831259 13266 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:56.794306 13475 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.838582  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:56.836967  5459 catalog_manager.cc:5697] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 12 to 17. New cstate: current_term: 17 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:27:56.841861 13505 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:56.833038 13266 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.842505 13505 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.796618 13386 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:56.803905 13260 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:56.835448 13186 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:56.839574 13463 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:56.754735 13494 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.845348 13157 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:56.877102 13506 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:56.877592 13523 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.833269 13516 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.878551 13404 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:27:56.878010 13519 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.857785 13388 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:56.890515  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 157706us
I20260709 12:27:56.895021  5459 catalog_manager.cc:5697] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 19 to 20, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 20 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:27:56.898893  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:56.845721 13475 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:56.869419 13505 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:56.902050  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:56.947924  5459 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 9 to 15. New cstate: current_term: 15 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:56.952732 12786 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:27:56.952956  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 119515us
I20260709 12:27:56.953478 13203 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.954037 13203 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:56.684759  5456 rpcz_store.cc:269] 0709 12:26:44.515559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:44.528650 (+ 13091us) service_pool.cc:224] Handling call
0709 12:26:45.256219 (+727569us) spinlock_profiling.cc:243] Waited 54.3 ms on lock 0x7b5400002490. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754e0b2 00007f50004c7759 00007f500045fa76 00007f5000430d58 00007f500052f04e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc
0709 12:27:56.340743 (+71084524us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:27:53.297540 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:27:53.298130 (+   590us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:27:53.298167 (+    37us) write_op.cc:435] Acquired schema lock
0709 12:27:53.298183 (+    16us) tablet.cc:662] Decoding operations
0709 12:27:53.298888 (+   705us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:27:53.298952 (+    64us) write_op.cc:641] Partition lock acquired for write op
0709 12:27:53.298979 (+    27us) tablet.cc:685] Acquiring locks for 6 operations
0709 12:27:53.299544 (+   565us) tablet.cc:701] Row locks acquired
0709 12:27:53.299563 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:27:53.299707 (+   144us) write_op.cc:270] Start()
0709 12:27:53.299839 (+   132us) write_op.cc:276] Timestamp: P: 1783600073299656 usec, L: 0
0709 12:27:53.299859 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 12:27:53.396471 (+ 96612us) log.cc:844] Serialized 2399 byte log entry
0709 12:27:54.111891 (+715420us) op_driver.cc:464] REPLICATION: finished
0709 12:27:54.202961 (+ 91070us) write_op.cc:301] APPLY: starting
0709 12:27:54.203065 (+   104us) tablet.cc:1367] starting BulkCheckPresence
0709 12:27:54.207778 (+  4713us) tablet.cc:1370] finished BulkCheckPresence
0709 12:27:54.207889 (+   111us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:27:54.336105 (+128216us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:27:54.359015 (+ 22910us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=12,key_file_lookups=12,delta_file_lookups=12,mrs_lookups=0
0709 12:27:54.362562 (+  3547us) write_op.cc:312] APPLY: finished
0709 12:27:54.386392 (+ 23830us) log.cc:844] Serialized 65 byte log entry
0709 12:27:54.402891 (+ 16499us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:27:54.432202 (+ 29311us) write_op.cc:454] Released schema lock
0709 12:27:54.434562 (+  2360us) write_op.cc:341] FINISH: Updating metrics
0709 12:27:54.451331 (+ 16769us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:27:54.454164 (+  2833us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3010599,"reported_tablets":49,"spinlock_wait_cycles":124850560,"child_traces":[["op",{"apply.queue_time_us":90759,"apply.run_cpu_time_us":15258,"apply.run_wall_time_us":242529,"cfile_cache_hit":26,"cfile_cache_hit_bytes":29207,"num_ops":6,"prepare.queue_time_us":435384,"prepare.run_cpu_time_us":25018,"prepare.run_wall_time_us":266744,"raft.queue_time_us":161497,"raft.run_cpu_time_us":10483,"raft.run_wall_time_us":91294,"replication_time_us":811854,"thread_start_us":391986,"threads_started":4,"wal-append.queue_time_us":127062,"wal-append.run_cpu_time_us":7907,"wal-append.run_wall_time_us":1909551}]]}
W20260709 12:27:56.953128  5601 leader_election.cc:336] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.570s (SENT)
I20260709 12:27:56.959123  5459 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 9 to 17. New cstate: current_term: 17 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:27:56.968341  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:56.968667  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:56.969782  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:56.958456 13203 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:56.952023 13495 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:56.971241  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:56.972587 13495 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:56.973167  5459 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 13 to 19. New cstate: current_term: 19 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:27:56.985280  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:56.894619 13524 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.077s
I20260709 12:27:56.904994 13511 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:56.897529 13157 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:56.968367 12786 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.900480 13506 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:56.980258 13495 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:56.959355 13404 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:56.901149 13523 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:57.009333 13526 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:27:57.009874 13463 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.910531 13519 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:56.959936 13388 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.021699  5459 catalog_manager.cc:5697] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:56.981787 13260 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:56.988775 13435 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.023929 13322 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:57.025799 13157 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:57.027915  5702 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:57.029913  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.076972  5556 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:57.030298 13494 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.086670  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.031798  5560 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:57.087684  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:57.088331  5636 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:57.032945  5561 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:27:57.035048  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:57.023945 12786 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.036629  5559 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 16 index: 7. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:27:57.039047 13508 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.039486 13396 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:56.990849 13386 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:57.049751 13475 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.010357 13518 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.008083 13509 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.066972  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:57.067657  5706 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:27:57.058462 13505 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:57.080571  5708 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:27:57.080998 13325 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.082269  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:57.083343  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:57.084811  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.085587 13317 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.085923  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.086279 13528 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.087364 13266 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.057852 13495 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:57.088845  5636 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:57.143721  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.060855 13398 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.079193  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:57.148382  5634 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:57.102135  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.114785  5628 raft_consensus.cc:2368] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:57.117558  5627 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:57.118816  5630 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 19: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:27:57.121522  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:57.147214  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:57.149281  5706 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:27:57.133777  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.138160  5459 catalog_manager.cc:5697] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:57.060745 13271 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.055s
I20260709 12:27:57.097829 13183 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.149290  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.157626  5702 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:27:57.088765  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:57.154825  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:57.159071  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:57.162920  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:57.074262 13463 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:57.169255  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:57.171525  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:57.162303  5636 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.172904  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:57.166103  5459 catalog_manager.cc:5697] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 18 to 19, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 19 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:57.177271  5636 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:57.177999  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:57.178364  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:57.142915 13468 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.180899 13508 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:57.196504 13157 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.196835 13508 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:57.196563 13532 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:57.197572 13532 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.197870 13508 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.182512  5560 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:57.197906 13532 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:57.203244 12786 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:57.183429  5702 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: []
I20260709 12:27:57.109386 13386 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.184288 13486 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.134423 13435 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.184638  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:57.185099 13506 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:57.186386 13388 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:57.187196  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.187520 13523 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.143443 13528 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.188133 13516 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.188813  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.239214  5707 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:27:57.189937  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.190752  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.191890 13534 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:27:57.166069 13266 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:57.160962 13398 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.193267  5459 catalog_manager.cc:5697] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 7 to 15, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 15 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:57.155802 13487 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.244707 13487 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.245333  5459 catalog_manager.cc:5697] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 14 to 18, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 18 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:57.246891 13487 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.179056  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:57.247162  5459 catalog_manager.cc:5697] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 12 to 19. New cstate: current_term: 19 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:27:57.115527 13396 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.209378  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.249866 13396 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:57.215005  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0937851749999936s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1938s for a response
I20260709 12:27:57.144608 13317 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.250339 13516 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:57.250919 13516 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:57.187546 13525 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 12:27:57.237022  5627 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:27:57.251962 13516 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.252615 13317 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.137786 13325 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.253434  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.253634 13494 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.254076  5627 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:27:57.254148 13494 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.254832 13325 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:57.236970  5631 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:27:57.256089  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:57.257217  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.258172  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.259184  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.242008 13528 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:57.247707  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68735.5us
I20260709 12:27:57.250871  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:57.255117  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.261947  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:57.262585  5627 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:27:57.262915  5627 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:57.263499 13506 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.264017 13398 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:57.249780  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.264072 13506 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.267091 13494 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.215750 13541 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:57.263759  5627 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.295279  5627 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:57.260164  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.153614 13505 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.273262  5459 catalog_manager.cc:5697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 15 to 22. New cstate: current_term: 22 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:57.270829 13398 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:27:57.276618 13404 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.156255 13509 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.115794 13518 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:57.311385  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:57.312069 13505 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:57.312988 13518 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.314599 13386 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:57.313889  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:57.284868  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.315995  5560 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:57.237370  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.319864  6276 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:27:57.270653  5675 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:57.325919  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:57.290568  5634 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
W20260709 12:27:57.297534  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 107121us
I20260709 12:27:57.309002 13435 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:57.309273  5636 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:27:57.311058 13463 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:57.183023 13524 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.111s
I20260709 12:27:57.284193  5710 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 23 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:27:57.315299  5627 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
W20260709 12:27:57.328794  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:57.338824  5459 catalog_manager.cc:5697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 7 to 17. New cstate: current_term: 17 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:57.329787  5631 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:27:57.175662 13183 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:57.337265  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:57.343006  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.337836  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:57.340225  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.340370  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:57.340179  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:57.340883  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.342986  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.348862  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.350301  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:57.345031  5632 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:57.350507  5629 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:57.353924  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:57.356127  5634 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:57.357748  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:57.363191  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:57.340006 13494 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:27:57.213269 13157 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.360750  5459 catalog_manager.cc:5697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 13 to 18, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 18 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:27:57.372165  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 101599us
I20260709 12:27:57.343397 13475 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.375674  5459 catalog_manager.cc:5697] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 12 to 14, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 14 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:57.225584 13486 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.383108  5560 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:27:57.377836 13266 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:27:57.384765 13266 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.385263 13266 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:27:57.390113 13183 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.360239 13325 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:57.392894 13325 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.393412 13325 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:57.359715 13322 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:27:57.398337 13322 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.398860 13322 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:57.401340 13506 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:57.405488  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.350783 13524 consensus_queue.cc:1243] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 19.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:27:57.406574  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.406898 13454 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:27:57.407512  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.407837 13494 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:57.408169  5561 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:57.408686  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.409403  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.410081  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.410660  5557 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:27:57.411728  5557 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:27:57.360826 13398 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:27:57.408915 13494 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.414582  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.420742  5702 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 12:27:57.372996 13370 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:27:57.373175 12786 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:57.424228  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.431454  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.439405  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.440060  5555 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:27:57.445320  5556 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 22 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 12:27:57.430867  5459 catalog_manager.cc:5697] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 15 to 18, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 18 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:27:57.416297 13509 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.447047  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.418467 13524 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.449954  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.413527 13475 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.455457  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.441654 13398 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:57.368723 13541 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.442652 13370 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:57.458606  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:27:57.444458 12786 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.468238 13487 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.476363 13540 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:27:57.504011 13487 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.505543 13541 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:57.506064 13475 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:57.506307 13487 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.492769 12786 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:57.507436 13463 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:57.507992 13463 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:57.508961 13463 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.494479 13398 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.494753 13539 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.005s
I20260709 12:27:57.477259  5560 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
W20260709 12:27:57.522576  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45359.7us
I20260709 12:27:57.531731  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:57.533790  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:57.538497  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 80015.8us
I20260709 12:27:57.517481 13509 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.457885 13524 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.548900  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.555024  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.556082  5705 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:57.536927 13519 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:57.568313  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.569953  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.543336 13395 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:57.571585 13395 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.571951 13395 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:57.572701  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:57.573434  5701 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 12:27:57.574254 13542 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 12:27:57.574550 13541 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20260709 12:27:57.575282 13546 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 12:27:57.577538  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.578521  5555 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:27:57.580327 13370 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.580765 13492 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:27:57.572186  5703 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:57.570827  5632 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:57.583748  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:57.550405 13487 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.542778 13496 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.597334  5557 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:57.595427  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:57.583380 13395 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.607878  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:57.607291  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:57.608336  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.608747  5703 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:27:57.559618 13509 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.603776 13435 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:57.606216 13541 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.010s
I20260709 12:27:57.607872 13516 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.632671  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.628412  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.612565 13260 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:57.625749  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:57.603475 13386 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.023s
I20260709 12:27:57.655200  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.619683 13157 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:57.643116  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 741055us
I20260709 12:27:57.647409 13524 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.656279  5704 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:57.607905 13505 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.025s
I20260709 12:27:57.633522  5632 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:57.664209  5703 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:27:57.606976 13487 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.629670 13435 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:57.626678 13486 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:27:57.672545  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:57.680886  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:57.641925 13260 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.683210  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:57.683991  5709 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:57.683140  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.692023  5703 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:27:57.695012  5709 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:27:57.683233  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:57.623462 13495 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:57.703754  5526 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.580s (SENT)
I20260709 12:27:57.706393  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:57.705089  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:57.680909 13435 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:57.693135 13260 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:57.717381  5635 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.648684 13516 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.726965 13555 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:57.724431  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.718124 13386 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:57.738536  5560 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:57.704871 13486 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:57.629534 13496 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.752025 12935 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.753242 13509 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.757280 13395 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.760691  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.761551 13487 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:57.740442 13386 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.762545 13386 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:57.763868 13264 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:27:57.765411  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.766129  5557 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:57.768049  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.736730 13322 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.769440  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.770226  5559 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:27:57.770438 13322 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.771477  5561 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:57.773108  5709 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:27:57.733536  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:57.741113  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.776532  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.741737  5635 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:27:57.705410 13532 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.784124  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.791687  5558 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:27:57.792759  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59358.6us
W20260709 12:27:57.801137  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:57.721647 13325 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.797950  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:57.805361  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.805985  5556 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:27:57.807322  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:57.808075  5634 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:57.810580  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.813627  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:57.814796  5561 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:57.820575  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.826287  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:57.805025 13396 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:57.828476  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:57.831730  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 160166us
I20260709 12:27:57.839233  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.817132 13454 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:57.839895  5557 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:27:57.826526 13551 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:57.852387 13551 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.852784 13551 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:57.805565 13541 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.037s
I20260709 12:27:57.838650 13496 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:57.857451 13266 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.858352  5708 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:27:57.859731 13266 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:57.859774  5704 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:27:57.838490 13396 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:57.861364 13396 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:57.861858  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.862681  5704 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:57.863520  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:57.864053  5704 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:27:57.847095 13487 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:57.864516 13460 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:27:57.864773 13487 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.865002 13460 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:57.865295 13487 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:57.866050 13460 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.867105 13460 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:57.868757  5632 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 25. Current term is 26. Ops: []
I20260709 12:27:57.869613  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:57.874555  5704 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:27:57.877370  5635 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 12. (term mismatch)
I20260709 12:27:57.879758  5635 pending_rounds.cc:85] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 11
I20260709 12:27:57.880025  5635 pending_rounds.cc:107] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260709 12:27:57.880275  5635 raft_consensus.cc:2889] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:27:57.885957  5526 leader_election.cc:336] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.591s (SENT)
I20260709 12:27:57.886634  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:57.890956  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.892973  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3147) took 1922 ms. Trace:
I20260709 12:27:57.894761  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:57.898919  5634 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:27:57.899868  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:57.893626  5635 rpcz_store.cc:276] 0709 12:27:55.970354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:55.970512 (+   158us) service_pool.cc:224] Handling call
0709 12:27:55.988849 (+ 18337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:56.724731 (+735882us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b5400120aa0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa6741c0 00007f4ffa675231 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c
0709 12:27:57.679864 (+955133us) spinlock_profiling.cc:243] Waited 5.45 ms on lock 0x7b4c00313100. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff893ea20 00007f4ff894fa29 00007f4ff8915644 00007f4ff893cf2d 00007f4ff893cb93 00007f4ffa66d18b 00007f4ffa66dcb5 00007f4ffa668f30 00007f4ffa66a7de 00007f4ffa674b49 00007f4ffa675416
0709 12:27:57.892651 (+212787us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:57.892903 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":353800320}
I20260709 12:27:57.825901 13388 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 12:27:57.847163 13505 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.847502 13454 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:57.938463 13454 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:57.939554 13532 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.935806 13505 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:57.948918 13505 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:57.902992  5630 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
W20260709 12:27:57.913179  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:57.913482 12762 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 19 current term: 19 is higher.
I20260709 12:27:57.917332  5628 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
W20260709 12:27:57.918164  5554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3138) took 1915 ms (client timeout 2051 ms). Trace:
I20260709 12:27:57.923879  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:57.995898  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.996564  5553 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:27:57.928745 13559 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:57.929626 13317 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.826889 13540 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:27:57.949743 13325 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:57.960966  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 75052.3us
I20260709 12:27:57.905889  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:58.022840  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:57.963088 13549 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:57.969758 13495 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.971179 13516 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:57.969758  5627 raft_consensus.cc:2410] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 19 because replica has last-logged OpId of term: 17 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 7.
I20260709 12:27:57.962944 13435 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:57.976711  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:58.030529  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53948.7us
I20260709 12:27:58.031476  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:58.032560  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:58.021762  5526 leader_election.cc:336] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.707s (SENT)
I20260709 12:27:57.979241 13203 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:27:57.992796  5554 rpcz_store.cc:269] 0709 12:27:56.002401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:56.007273 (+  4872us) service_pool.cc:224] Handling call
0709 12:27:57.917879 (+1910606us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b54003a98d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:57.918065 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4245639040}
I20260709 12:27:58.033442  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:58.007859 13157 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:27:58.008670  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 169391us
I20260709 12:27:58.013403  5561 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:27:58.009663  6271 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:27:57.993114 13186 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:58.041018  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.033453  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:58.027597  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:58.036707  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:58.034463 13203 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:58.037700  6234 leader_election.cc:400] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27.
I20260709 12:27:58.037788 13358 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:27:58.039146  5702 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:58.035706  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:58.044471  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 16 index: 7 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:57.963207 13386 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:58.045720  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:58.054126  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:58.054854  5703 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:58.056183 13368 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:58.057997  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:58.062055 13368 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:58.051489  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:58.056912 13495 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:58.049335  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:58.063431  5709 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:58.064520  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:58.048342  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:58.043893 13288 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:58.072932 13288 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:58.073320 13288 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:58.075412  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:58.076903  5704 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:58.075537  5524 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:58.036518 13534 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:58.078446  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:58.081032  5561 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:58.080991  5709 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:27:58.035331 13496 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:58.035027 13157 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:27:58.051491 12786 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.539s	user 0.009s	sys 0.003s
W20260709 12:27:58.084486  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 51098.7us
I20260709 12:27:58.085186  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:58.051239 13559 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:27:58.020942 13317 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:58.009085 13260 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:58.090592  5710 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:27:58.043632 13463 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:58.091017  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:58.079818  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:58.054348 13518 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:27:58.101817  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:58.039721 13549 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:58.099538  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:58.106400  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58156.1us
I20260709 12:27:58.107247  5561 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:58.107636  5675 leader_election.cc:400] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
W20260709 12:27:58.100972  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:58.097291  5706 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:58.104956  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.051316 13494 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.046s
I20260709 12:27:58.109589  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:58.110472  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 75033.9us
I20260709 12:27:58.109784  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:58.115316  5704 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:27:58.116852  5704 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 12:27:58.088596 13508 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:27:58.122073  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.121953  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.124096  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.099188 13186 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:58.126098  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:27:58.126104  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 94307.1us
I20260709 12:27:58.126796  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.126847  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:58.143486  5630 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:27:58.144255  5561 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:58.102795 13388 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:58.145828  5631 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:58.146250  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.105877 13486 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:58.105599 13559 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:58.153342  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:58.157473 13568 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:27:58.118103 13404 raft_consensus.cc:2764] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:27:58.088055 13534 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:58.121551 13509 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:58.172724  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.173065 13509 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:58.173477 13509 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:58.175740  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.178103  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.178941  5627 raft_consensus.cc:2368] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:58.093153 12786 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:58.162262  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.180707  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.161593  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.181811  6278 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:27:58.164443  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54904.8us
I20260709 12:27:58.099953 13496 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:58.185847 13496 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:58.162367  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.173449  5635 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:58.179769  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.197392  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:58.198520  5704 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:27:58.149114 13521 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:27:58.199837  5561 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:27:58.200456  5554 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:27:58.201393  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.202009  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.202354  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.202606  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.204263  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.205083  6279 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:27:58.159557 13435 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:58.165462 13508 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:58.216456  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.111482 13203 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:58.169862 13541 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:58.217410  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.218415  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:27:58.221719  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 111976us
I20260709 12:27:58.217375  6275 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:58.224578  6278 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:27:58.223793 13386 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:58.230197  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:58.150549 13157 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:58.234346  6280 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:58.162545 13486 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:58.234175  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.238629  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.239867  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.242959  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.242892 13559 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:58.244616  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.245831  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.246497  6280 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:27:58.247634  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.249359  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.250888  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:58.252707  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.254462  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.255246  6279 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:58.256201  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.256924  6278 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:27:58.258307  6280 raft_consensus.cc:2410] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 20 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 15.
I20260709 12:27:58.258787  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.259372  6275 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:58.263674  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:58.264034 13570 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:27:58.264408 13487 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:58.264988  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.266582  6273 raft_consensus.cc:2368] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21.
W20260709 12:27:58.263907 13567 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:27:58.234442 13203 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:27:58.280467  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:58.294965  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.296028  6278 raft_consensus.cc:2368] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
W20260709 12:27:58.297381  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:58.298586  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:58.301255  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:58.286043 13530 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.214s	user 0.005s	sys 0.000s
W20260709 12:27:58.321657  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 93010.1us
I20260709 12:27:58.322633  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:58.327924  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:58.324069  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:58.334204  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.334329  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:58.334815  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 191198us
I20260709 12:27:58.335381  6278 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 16.
I20260709 12:27:58.234735 13508 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:58.346369  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:58.346947  5704 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: []
W20260709 12:27:58.352638  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:58.351894  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:58.354535  6278 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 16. (term mismatch)
I20260709 12:27:58.357249  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:58.341820 13383 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:27:58.359897  6278 pending_rounds.cc:85] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 15
I20260709 12:27:58.359928  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:58.333715 13523 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:58.331054 13541 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:58.334461 13388 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:58.366950  6278 pending_rounds.cc:107] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 19.16
I20260709 12:27:58.366233  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:58.370146  6278 raft_consensus.cc:2889] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:58.371685  5708 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 25. Current term is 26. Ops: []
I20260709 12:27:58.355464 12762 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:58.368228  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:58.369689  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1286) took 1211 ms. Trace:
I20260709 12:27:58.371651  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:58.367336  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 85376.2us
I20260709 12:27:58.359371 13386 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:58.377288  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.377863  6272 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:27:58.374053  6271 rpcz_store.cc:276] 0709 12:27:57.153601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:57.154599 (+   998us) service_pool.cc:224] Handling call
0709 12:27:58.365127 (+1210528us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b54002e3954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:58.365277 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1792114944}
I20260709 12:27:58.363927 13463 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:58.381556  5710 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:27:58.387553  6278 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:58.369022 13523 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:58.330485 13518 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.038s
I20260709 12:27:58.359237 13260 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:58.364861 13574 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:27:58.398411  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.374861  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:27:58.389659  5709 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:58.391274  5675 leader_election.cc:400] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
W20260709 12:27:58.391789  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:58.393839  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:58.359181 13404 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.027s
I20260709 12:27:58.397982  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.388682  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:58.401463  5710 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:58.388300 13435 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:58.383885 13264 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:58.388646 12762 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:58.402034  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:58.403776  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2891) took 1646 ms. Trace:
I20260709 12:27:58.404830  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:58.396050 13523 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:58.416489 13572 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:27:58.405819 13574 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:58.419435 13509 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:58.413267  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:58.420037 13574 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:58.423090  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:58.423833  6271 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:58.412763  5552 rpcz_store.cc:276] 0709 12:27:56.751381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:56.751548 (+   167us) service_pool.cc:224] Handling call
0709 12:27:56.751883 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:56.752266 (+   383us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:56.752295 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:56.752896 (+   601us) log.cc:844] Serialized 26 byte log entry
0709 12:27:56.891886 (+138990us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:27:56.891929 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:56.892000 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:58.285300 (+1393300us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:27:58.392689 (+107389us) raft_consensus.cc:1685] finished
0709 12:27:58.392722 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:58.398067 (+  5345us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":45211,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":30507,"spinlock_wait_cycles":24198912,"thread_start_us":138554,"threads_started":1,"wal-append.queue_time_us":228487}
I20260709 12:27:58.415858 13383 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:27:58.432760 13463 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:58.394840 13388 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:27:58.433883 13486 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:58.434545  5555 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:27:58.435487 13508 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:58.436005  5555 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 17. (term mismatch)
I20260709 12:27:58.437454  5552 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:27:58.425057  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:58.437618  5555 pending_rounds.cc:85] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 16
I20260709 12:27:58.439769  5555 pending_rounds.cc:107] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 22.17
I20260709 12:27:58.440083  5555 raft_consensus.cc:2889] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:27:58.440619 13551 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.587s	user 0.005s	sys 0.013s
I20260709 12:27:58.441057 13551 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:58.438546  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:58.443289  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:58.443662  6276 raft_consensus.cc:2368] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24.
I20260709 12:27:58.450441  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:58.451223  5555 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:27:58.453722  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:58.457015  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:58.466017  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 61555.2us
I20260709 12:27:58.443189  5708 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:27:58.479341  6271 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:58.429260 12762 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:58.430431 13288 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:58.510033  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:27:58.409493 13396 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.547s	user 0.012s	sys 0.000s
I20260709 12:27:58.513037  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:58.432833 13383 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:58.420574 13575 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:58.447991  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:58.412019  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:58.518412  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:58.406109 13435 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:58.413266  5557 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:27:58.523973  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 80593.2us
W20260709 12:27:58.525601  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 71997.5us
I20260709 12:27:58.526274  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:58.515627  6271 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 18: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:58.518445  5552 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:27:58.525782  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:58.530603  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:27:58.581271  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55617.5us
I20260709 12:27:58.583491  5631 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:58.543699  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:58.590363  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:58.606630  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:58.651150  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.651893  6271 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 16.
I20260709 12:27:58.529570 13435 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:58.507112 13555 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:58.646672  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:58.511957 13404 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:27:58.514375 13463 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:58.621043  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:58.559463 13358 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:58.525193 13511 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:58.579385 13516 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:58.524895 13383 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:58.660234  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.674381  6271 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
W20260709 12:27:58.645129 13454 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.706s	user 0.008s	sys 0.004s
I20260709 12:27:58.666457  5708 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:27:58.524856 13494 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:58.606956 13266 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:27:58.589826 13540 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:58.676218  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:58.668804  5707 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:27:58.675431  5601 leader_election.cc:336] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.582s (SENT)
I20260709 12:27:58.689783  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:58.578763 13532 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.107s
I20260709 12:27:58.527863 13396 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:58.694680  5708 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 16. (term mismatch)
I20260709 12:27:58.695132  5708 pending_rounds.cc:85] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 15
I20260709 12:27:58.695348  5708 pending_rounds.cc:107] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 27.16
I20260709 12:27:58.695559  5708 raft_consensus.cc:2889] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:27:58.665738 13404 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:58.680692  5555 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:58.674517 13511 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:58.690001 13494 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:58.703321  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:58.677994 13518 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:58.702780  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:58.670120 13575 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:27:58.705178  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:58.670440 13460 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:58.705814  5552 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:58.706852 13551 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:58.707597 13396 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:58.707809 13490 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:58.708002  5703 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:58.676642 13463 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:58.711654 13523 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:58.702181 13404 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:27:58.713145 13524 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:58.714195 13494 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:58.715167 13509 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:58.716159  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:58.758572  5561 raft_consensus.cc:2410] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 24 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260709 12:27:58.718386  6279 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: [22.18-22.18]
W20260709 12:27:58.722783  5526 leader_election.cc:336] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.800s (SENT)
I20260709 12:27:58.737182  5555 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:58.738626  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.740195  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:27:58.760598  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:58.741468  5708 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 25. Current term is 26. Ops: []
I20260709 12:27:58.743125  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.701840 13508 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:27:58.744131  5635 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:27:58.756408 13551 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:27:58.684852 13454 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:58.750283  5557 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:27:58.720530 13573 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:27:58.750911  5707 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:27:58.753561  5627 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
W20260709 12:27:58.754762 13589 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:27:58.696074 13540 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:58.745255 13460 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:58.764623  5709 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:27:58.768682  5703 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
W20260709 12:27:58.768477  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 162566us
I20260709 12:27:58.761178 13383 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:58.746459  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70342.4us
I20260709 12:27:58.784569  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1291) took 1617 ms. Trace:
I20260709 12:27:58.785897 13317 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:58.787983 13468 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:27:58.769263 13486 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:27:58.810448  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 410347us
I20260709 12:27:58.809561  6272 rpcz_store.cc:276] 0709 12:27:57.166880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:57.167011 (+   131us) service_pool.cc:224] Handling call
0709 12:27:58.368656 (+1201645us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:58.386360 (+ 17704us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:58.784199 (+397839us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54002e3950. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:58.784453 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2714393984}
W20260709 12:27:58.803015 13203 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.525s	user 0.013s	sys 0.000s
I20260709 12:27:58.842061  5633 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:58.801898 13494 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:27:58.797850 13508 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:58.898669  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1608) took 1148 ms. Trace:
I20260709 12:27:58.898821  5628 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:27:58.839977  5555 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:27:58.899489  5632 rpcz_store.cc:276] 0709 12:27:57.750207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:57.750389 (+   182us) service_pool.cc:224] Handling call
0709 12:27:57.750719 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:57.881361 (+130642us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:58.898396 (+1017035us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b540009e7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:27:58.898600 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":271007488}
I20260709 12:27:58.849532  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3307) took 1089 ms. Trace:
I20260709 12:27:58.900779  5706 rpcz_store.cc:276] 0709 12:27:57.759894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:57.760389 (+   495us) service_pool.cc:224] Handling call
0709 12:27:58.849270 (+1088881us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b540026fa54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:58.849420 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":753206016}
I20260709 12:27:58.850944  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:58.853117  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3163) took 1331 ms. Trace:
I20260709 12:27:58.855644  6273 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: [19.14-20.15]
I20260709 12:27:58.858223 13487 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:58.860981  5556 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:58.862049  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3158) took 1455 ms. Trace:
I20260709 12:27:58.807590 13551 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:58.865811  5553 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:58.881604  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:27:58.903340  5560 rpcz_store.cc:276] 0709 12:27:57.521642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:57.530907 (+  9265us) service_pool.cc:224] Handling call
0709 12:27:58.852850 (+1321943us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b5400025dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:58.853034 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":455590656}
I20260709 12:27:58.885272  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:58.890262 13157 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:58.909318  5636 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 24: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:27:58.896569  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56269.8us
I20260709 12:27:58.909598  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:58.836594 13468 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:58.913312  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:58.913964  5632 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:58.914988  5701 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:27:58.907627  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 364007us
I20260709 12:27:58.905329  5558 rpcz_store.cc:276] 0709 12:27:57.406208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:57.406334 (+   126us) service_pool.cc:224] Handling call
0709 12:27:58.861771 (+1455437us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b540044f554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:58.861955 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":840682240}
I20260709 12:27:58.911381  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:58.917297  5627 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:58.833768 13288 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:58.915701  6234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.013s	sys 0.002s
W20260709 12:27:58.918202  6234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.014s	sys 0.002s
W20260709 12:27:58.918815  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:58.905234 13564 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
W20260709 12:27:58.938076  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 407590us
I20260709 12:27:58.939356  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:58.953248  6271 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 12:27:58.953711  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 43695.2us
I20260709 12:27:58.960157  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:27:58.964605  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:58.911690 13486 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:58.908758 13549 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:58.899519 13518 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 12:27:58.970213  6272 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 15. Current term is 16. Ops: []
W20260709 12:27:58.971805  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 249075us
W20260709 12:27:58.982807  5526 leader_election.cc:336] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.843s (SENT)
I20260709 12:27:58.983198  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:58.836794 13511 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:27:58.983950  6279 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:27:58.974579  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:58.954859  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:58.976369  6276 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:27:58.923444 13203 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:58.909473 13388 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.050s
I20260709 12:27:58.978931  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:58.980650  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:58.955830 13288 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:58.989526  6278 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:27:59.009485  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 70268.8us
W20260709 12:27:59.009888  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55322.2us
I20260709 12:27:59.010308  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:59.017143  5627 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:59.027983  5553 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:27:59.029815  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:58.956840 13395 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.049s
I20260709 12:27:58.991221 13496 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:27:58.992305 13266 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:59.038048  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:27:59.044579  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 76912.9us
I20260709 12:27:59.045078  5627 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:27:59.046590  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:58.994205 13203 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:59.049296  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:27:59.024253 13573 raft_consensus.cc:2764] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election decision vote started in defunct term 23: won
I20260709 12:27:59.029161 13516 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:27:59.053381  6271 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:27:59.053982  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:27:59.083056  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 100271us
I20260709 12:27:59.080564  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:59.044317 13505 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:59.038798 13518 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:59.082473  6278 raft_consensus.cc:2376] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25. Re-sending same reply.
I20260709 12:27:59.090363  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:59.096249  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:59.096851  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:59.097975  5709 raft_consensus.cc:2410] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 24 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260709 12:27:59.099619  6278 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:27:59.085215  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:27:59.087648  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57104.4us
I20260709 12:27:59.101202  6271 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:27:59.101367  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:59.103039  5703 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 23. Current term is 24. Ops: []
I20260709 12:27:59.103196  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:59.103829  5704 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:27:59.024108 13288 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:59.004304 13603 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.091s
I20260709 12:27:59.051386 13463 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:59.142256  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 132020us
I20260709 12:27:59.109869  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:59.055455 13496 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:27:59.057837 13524 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:59.031361 13460 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:59.149224  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:59.113785  5627 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
W20260709 12:27:59.157152  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:59.084322 13398 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:59.010519 13551 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:59.008880 13508 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:59.113006 13475 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:27:59.148694 13605 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:59.027769 13468 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:59.171169  5709 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:59.171609  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:27:59.163654  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53864.2us
I20260709 12:27:59.189817 13468 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:59.169127 13549 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:27:59.151969 13383 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:59.158478 13505 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:59.202135  5632 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:27:59.261399  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:59.262264  5559 raft_consensus.cc:2368] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:59.264283  5559 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:27:59.265686 13534 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:59.266902  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:59.267338  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:59.105551 13454 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:59.205026  6275 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:27:59.209035  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2912) took 1105 ms. Trace:
I20260709 12:27:59.112879 13532 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.074s
I20260709 12:27:59.268823  5631 rpcz_store.cc:276] 0709 12:27:58.103657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:58.180445 (+ 76788us) service_pool.cc:224] Handling call
0709 12:27:59.208800 (+1028355us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b54000c1f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:59.208961 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":881160448}
I20260709 12:27:59.219154 13288 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:59.220817  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:59.222092  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:59.056219 13486 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:59.271047  5627 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:59.188915  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:27:59.273041  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 84213us
I20260709 12:27:59.224146 12786 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:59.274073  6275 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:59.244294  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2921) took 1123 ms. Trace:
I20260709 12:27:59.249650 13523 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:59.254608  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:59.257321  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:27:59.276856  5601 leader_election.cc:336] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.690s (SENT)
I20260709 12:27:59.270710  6280 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:59.270727  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:59.277931  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:59.259073  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:27:59.242100 13613 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:59.291110  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:27:59.273073  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:59.274986  5628 rpcz_store.cc:276] 0709 12:27:58.120893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:58.175466 (+ 54573us) service_pool.cc:224] Handling call
0709 12:27:59.243852 (+1068386us) spinlock_profiling.cc:243] Waited 669 ms on lock 0x7b540018d154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:59.244160 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1539524480}
I20260709 12:27:59.290058  5709 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:27:59.241281 13604 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:27:59.302306  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1287) took 1095 ms. Trace:
I20260709 12:27:59.294088  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:59.296202  5627 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
W20260709 12:27:59.303450  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 204364us
I20260709 12:27:59.302892  6280 rpcz_store.cc:276] 0709 12:27:58.206426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:59.220497 (+1014071us) service_pool.cc:224] Handling call
0709 12:27:59.302237 (+ 81740us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:27:59.310484  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:59.116307 13518 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:59.314512  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:27:59.224257 13598 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.189s	user 0.002s	sys 0.000s
I20260709 12:27:59.324800  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:59.192008 13404 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:59.169440 13524 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:59.349229  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:27:59.262148 13505 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:59.350909  5556 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 12:27:59.302448 13604 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:59.315114 13454 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:27:59.356045  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:59.334996 13395 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:27:59.369921  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:59.198879 13605 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:59.354928 13511 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:27:59.321682 13539 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.044s
I20260709 12:27:59.312744 13613 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:59.211930 13266 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:59.341743 12786 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
I20260709 12:27:59.352749 13368 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.036s
W20260709 12:27:59.392905 13607 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.347s	user 0.000s	sys 0.001s
I20260709 12:27:59.404493  5627 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 17 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:27:59.355597 13317 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 12:27:59.418228  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:59.410173  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:59.383266 13521 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.078s
I20260709 12:27:59.449201  5635 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: [20.12-20.12]
W20260709 12:27:59.450840  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 156877us
I20260709 12:27:59.459849  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3172) took 1475 ms. Trace:
I20260709 12:27:59.441540  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:59.464432  5676 leader_election.cc:400] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:59.462285  5632 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:27:59.464784  5676 leader_election.cc:403] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
W20260709 12:27:59.450347  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 177345us
I20260709 12:27:59.463532  5556 rpcz_store.cc:276] 0709 12:27:57.984026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:58.329369 (+345343us) service_pool.cc:224] Handling call
0709 12:27:58.860807 (+531438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:59.351574 (+490767us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:59.459643 (+108069us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b540044e150. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:59.459782 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1221144960}
I20260709 12:27:59.468036  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:27:59.420571  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 49027.2us
I20260709 12:27:59.366909 13549 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:27:59.470902  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:59.442016  5634 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:59.441771  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:59.342343 13496 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:59.451295 13596 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:27:59.396438 13395 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:59.418787 13454 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:59.418143 13383 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:27:59.488041  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:59.446882 12762 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:27:59.490140 13404 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:59.422886 13532 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:59.494447 13613 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:59.496800 13157 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.580s	user 0.007s	sys 0.004s
W20260709 12:27:59.546556  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 78632us
I20260709 12:27:59.417131 13534 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:59.548141  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:27:59.423223 13523 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:59.442166 13594 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:27:59.441351 13599 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:27:59.539832 13495 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:27:59.418536 13203 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:59.481707 13600 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.393s	user 0.001s	sys 0.000s
I20260709 12:27:59.423050 13610 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:59.474076 13549 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:27:59.544601  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 111803us
W20260709 12:27:59.545737  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 81460.6us
I20260709 12:27:59.584100  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:59.453073 13572 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:27:59.558075  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:27:59.424467 13603 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:27:59.570427  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 99769.2us
I20260709 12:27:59.569742 13614 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:59.541586 13612 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:59.579206  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 302397us
I20260709 12:27:59.539901 12762 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:27:59.559926 13388 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:27:59.498260 13288 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.076s
I20260709 12:27:59.501202 13509 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.075s
I20260709 12:27:59.567723 13534 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:59.723172  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:27:59.581529 13606 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.403s	user 0.000s	sys 0.001s
I20260709 12:27:59.565824 13541 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:27:59.575978 13435 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.900s	user 0.005s	sys 0.012s
I20260709 12:27:59.614514  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:27:59.615444  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 57503.8us
I20260709 12:27:59.540143 13454 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:59.602952  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:27:59.579494 13495 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:27:59.602252 13610 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:59.586424 13549 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:27:59.715658 13157 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:27:59.571594 13523 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:59.720759  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:59.559700 13532 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:59.724395  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:59.811417  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:59.548890 13596 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:27:59.745292 13599 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:27:59.741029 13541 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:59.843173  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:59.754565  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:27:59.720960 13322 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:59.787779  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 184910us
I20260709 12:27:59.721457 13404 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:59.788450  5628 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:27:59.723104 13539 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:59.742455 13611 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.382s	user 0.000s	sys 0.001s
I20260709 12:27:59.723556 13521 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:59.615814 13203 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:27:59.737795 13388 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:27:59.770203 13495 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:27:59.848788 13541 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:27:59.752563  5559 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:27:59.870995 13628 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:27:59.837746 13619 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.276s	user 0.000s	sys 0.004s
I20260709 12:27:59.874667 13523 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:59.877110 13610 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:59.878422 13602 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.724s	user 0.009s	sys 0.025s
I20260709 12:27:59.844301 13572 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:27:59.880828 13616 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.477s	user 0.001s	sys 0.000s
I20260709 12:27:59.882783  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:27:59.888937  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:59.846773  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:27:59.770843 13518 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:59.784001 13524 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:59.786923 13511 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.076s
I20260709 12:27:59.787894 13370 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:27:59.771374 13435 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:59.789691 13157 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:27:59.724200 13614 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:59.791213 13505 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.073s
I20260709 12:27:59.722738 13612 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:59.863453 13521 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:59.861449 13404 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:27:59.902570 13454 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:27:59.864228 13203 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:59.861318 13599 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:27:59.861687 13539 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:59.857981 13468 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.668s	user 0.008s	sys 0.004s
I20260709 12:27:59.857811 13322 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:27:59.865521 13388 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:27:59.861294 13625 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260709 12:27:59.865906 13532 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:59.854489 13398 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.678s	user 0.010s	sys 0.000s
I20260709 12:27:59.863876  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3322) took 1023 ms. Trace:
I20260709 12:27:59.790511 13615 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:59.920876  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 306406us
I20260709 12:27:59.997887  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:27:59.926188 13605 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:27:59.859954 13621 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.246s	user 0.000s	sys 0.000s
I20260709 12:27:59.903553 13603 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:27:59.999853 13603 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:00.000500 13322 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:27:59.856887 13490 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.109s	user 0.013s	sys 0.000s
I20260709 12:27:59.927055  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3170) took 1896 ms. Trace:
I20260709 12:28:00.005126  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:00.005772 13404 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:00.006490  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:00.007771 13540 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:27:59.960963 13573 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:00.009477 13521 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:00.010020 13468 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:00.010563 13468 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:00.010918 13603 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.014899 13539 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:00.015738  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:00.015992 13368 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:28:00.016513 13368 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:27:59.974992  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.008s	sys 0.064s
W20260709 12:28:00.017412  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.009s	sys 0.064s
W20260709 12:28:00.017755  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 557826us
I20260709 12:28:00.018355 13612 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:00.019021 13615 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.020794  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.021266 13615 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:00.021413  5701 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:00.021908 13398 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:00.022228  6279 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:28:00.022593  5706 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:00.023005 13599 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.023770  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:59.975106 13203 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:00.025920  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.026520  5709 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:27:59.960314 13596 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:27:59.976250  5705 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20. Ops: []
I20260709 12:27:59.995173  5707 rpcz_store.cc:276] 0709 12:27:58.840800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:58.841001 (+   201us) service_pool.cc:224] Handling call
0709 12:27:58.841451 (+   450us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:58.898001 (+ 56550us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:58.898039 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:58.906614 (+  8575us) log.cc:844] Serialized 52 byte log entry
0709 12:27:59.042018 (+135404us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:27:59.042073 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:59.042165 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:27:59.863524 (+821359us) raft_consensus.cc:1685] finished
0709 12:27:59.863555 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:27:59.863798 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":134811,"raft.queue_time_us":64879,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":3354,"thread_start_us":87415,"threads_started":1,"wal-append.queue_time_us":243400}
W20260709 12:27:59.987118 13317 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.001s	sys 0.016s
I20260709 12:28:00.005311  6277 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:00.004900  5633 rpcz_store.cc:276] 0709 12:27:58.030288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:58.032250 (+  1962us) service_pool.cc:224] Handling call
0709 12:27:59.926747 (+1894497us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b540032bed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:27:59.926960 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1843266944}
I20260709 12:28:00.007699  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:59.927110 12762 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:27:59.952800  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 106094us
I20260709 12:28:00.043391  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.044049  5703 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:00.038677 13183 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:00.045203 13183 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.046697 12762 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:00.048593  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.049211  5701 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:00.012442 13396 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:00.050393 13396 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.028750  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.051654  5704 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:00.046717  5710 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:28:00.050403 13596 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.041589  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:27:59.962275  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:00.055562  5556 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:00.029534  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.045218  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:00.057154  5554 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:27:59.988762 13505 consensus_queue.cc:1243] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 19.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:28:00.057072 13604 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.058611 13604 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:00.058760 13534 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:00.060328 13435 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:00.061020 13614 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:00.061930  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:00.063068  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:27:59.964506 13157 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:00.045504  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.072131  5707 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:28:00.064172  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:00.035796  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 157244us
I20260709 12:28:00.065837  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.058171  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:00.073210  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:00.069449  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:00.076586  5601 leader_election.cc:336] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 2.004s (SENT)
I20260709 12:28:00.079787  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:00.083949  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:00.039111 13603 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:00.098701  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:00.042251 13521 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:00.096108  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:00.040884 13317 consensus_queue.cc:1243] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 22.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:28:00.100195  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:00.098347  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:00.101202  5708 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:28:00.087646  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:00.086637  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.106859  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.118923  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:00.128717  5704 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:28:00.123032  5706 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
W20260709 12:28:00.087839 13508 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.017s	sys 0.008s
I20260709 12:28:00.134307  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:00.134964  5707 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:00.082168 13540 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:00.137636  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:00.138339  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:00.138448  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:00.142241  5556 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:00.139707  5630 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:28:00.142908  5552 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:00.145478  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:00.107411 13266 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.060s	sys 0.099s
W20260709 12:28:00.150589  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 50458.9us
I20260709 12:28:00.098407 13370 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:00.086248 13534 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:00.149617  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:00.138132 13508 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.017s	sys 0.009s
I20260709 12:28:00.157471  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:28:00.151216 13616 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:28:00.129050 13572 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:00.085670 13386 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260709 12:28:00.182252  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:00.183066  6279 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:28:00.167976  5556 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:28:00.117930 13264 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:28:00.152205 13540 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:00.157938  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:00.171259  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:00.176687 13586 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:00.179558 12753 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:00.189618  5707 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
W20260709 12:28:00.149091 13579 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:28:00.129380 13532 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.051s
I20260709 12:28:00.211946  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:00.213577  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:00.178840 13518 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:00.160499 13636 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:28:00.157965 13266 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.062s	sys 0.101s
W20260709 12:28:00.191994  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:00.155076 13643 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.074s
I20260709 12:28:00.174612 13534 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:00.209236 13540 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:00.224153  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:00.212806 13586 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:00.227479  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:00.186398 13468 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:00.227814  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 229987us
W20260709 12:28:00.227582  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63169.8us
I20260709 12:28:00.191135 13596 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:00.234790  5635 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:28:00.170197 13370 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:00.214828 12753 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:00.190845 13572 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:00.239866  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:00.225864 13183 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:00.224407 13518 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:00.303862  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 65244.3us
W20260709 12:28:00.331004  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:00.333102  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:00.236246 13586 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:00.224756 13636 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:00.346987  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:00.336239 13518 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:00.356746 13638 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
W20260709 12:28:00.357805 13644 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20260709 12:28:00.359814 13523 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:00.360251 13523 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:00.360419 13322 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 12:28:00.361047  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 121292us
W20260709 12:28:00.361390 13549 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.551s	user 0.010s	sys 0.000s
I20260709 12:28:00.361812 13549 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.361948 12762 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:28:00.362259 13534 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:00.368743 13629 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:28:00.370132 13637 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 12:28:00.419189  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:00.374788 13370 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:00.376744 13584 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.725s	user 0.002s	sys 0.002s
I20260709 12:28:00.426930  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:00.427667  5707 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:00.382212  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:00.360669 13523 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:00.364043 13549 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:00.414376  5710 raft_consensus.cc:2410] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:28:00.239892 13605 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:00.303521 12753 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:00.237675 13317 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:00.330564 13505 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:00.299237 13264 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:28:00.441434  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:00.439086  5552 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:28:00.305473 13495 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.418292 13396 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:00.450750  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 68670.2us
I20260709 12:28:00.478901  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:00.414949 13463 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:00.356767 13468 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.101s
I20260709 12:28:00.421895 13636 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:00.447362  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2928) took 1030 ms. Trace:
I20260709 12:28:00.476643  5707 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:28:00.460866 13630 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:28:00.490494 13463 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:00.502009 13524 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:00.445077 13264 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:00.447903 13534 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:00.439880 13605 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:00.500335  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:00.457811 13370 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:00.448176 13532 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:00.507817  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 66515.3us
I20260709 12:28:00.495448  5629 rpcz_store.cc:276] 0709 12:27:59.416852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:59.425634 (+  8782us) service_pool.cc:224] Handling call
0709 12:27:59.441591 (+ 15957us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:59.620592 (+179001us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:59.620636 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:59.721745 (+101109us) log.cc:844] Serialized 52 byte log entry
0709 12:28:00.086040 (+364295us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:00.422551 (+336511us) log.cc:844] Serialized 14 byte log entry
0709 12:28:00.447004 (+ 24453us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:00.447110 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:00.447155 (+    45us) raft_consensus.cc:1685] finished
0709 12:28:00.447166 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:00.447302 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22561,"prepare.queue_time_us":268652,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":139689,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":15531,"thread_start_us":745289,"threads_started":3,"wal-append.queue_time_us":456184}
I20260709 12:28:00.499256  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:00.450129 13505 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:00.476668 13266 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:00.482201  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:00.479408 13317 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:00.515120  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:00.506637 13264 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:00.506388 13463 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:00.528638  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:00.529608  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:00.530967  5552 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:00.517115 13605 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:00.525817 13549 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:28:00.533880  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:00.524609 13266 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.534688 13572 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:00.518224 13532 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:00.476888 13203 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:00.544816  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62738.2us
W20260709 12:28:00.524564 13541 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.656s	user 0.013s	sys 0.000s
I20260709 12:28:00.558560  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3319) took 2407 ms. Trace:
I20260709 12:28:00.567377  5558 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:28:00.526383 13624 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:28:00.566742  5702 rpcz_store.cc:276] 0709 12:27:58.151270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:58.184864 (+ 33594us) service_pool.cc:224] Handling call
0709 12:27:58.185228 (+   364us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:27:58.185660 (+   432us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:58.185692 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:27:58.186390 (+   698us) log.cc:844] Serialized 52 byte log entry
0709 12:27:58.366952 (+180562us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:27:58.366998 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:58.367068 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:00.186140 (+1819072us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:00.539891 (+353751us) raft_consensus.cc:1685] finished
0709 12:28:00.539913 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:00.558461 (+ 18548us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3898,"raft.queue_time_us":24334,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":15910,"spinlock_wait_cycles":6648064,"thread_start_us":180150,"threads_started":1,"wal-append.queue_time_us":251691}
I20260709 12:28:00.573693  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:00.516285 13534 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:00.581280  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:00.584218  5707 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:28:00.551288 13463 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:00.560696 13645 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:00.555887 13549 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:00.613776 13549 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.588086 13386 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
I20260709 12:28:00.573161 13541 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.600755 13463 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:00.618939 13617 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:28:00.619439 13617 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:00.620198 13463 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:00.622867 13604 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.564s	user 0.011s	sys 0.001s
I20260709 12:28:00.558101 13317 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:00.600992 13622 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.886s	user 0.000s	sys 0.003s
I20260709 12:28:00.602648 13494 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:28:00.603425 13488 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:00.605371 13521 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:00.609594 13518 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.583150 13629 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:28:00.611423 13495 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:00.612723  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 97730.1us
I20260709 12:28:00.623011  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:00.638561 13659 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.077s	user 0.000s	sys 0.002s
I20260709 12:28:00.623584 13386 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:28:00.557566 13540 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:00.625123  5553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 2909) took 2651 ms (client timeout 1868 ms). Trace:
I20260709 12:28:00.590076  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:00.629771  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3186) took 1164 ms. Trace:
I20260709 12:28:00.631438  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:00.641188  6278 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:28:00.566622 13396 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 12:28:00.651141  5629 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:00.621743  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:00.671412  5555 rpcz_store.cc:276] 0709 12:27:59.465390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:59.465552 (+   162us) service_pool.cc:224] Handling call
0709 12:27:59.465884 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:27:59.466093 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:28:00.454850 (+988757us) log.cc:844] Serialized 14 byte log entry
0709 12:28:00.629415 (+174565us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:00.629472 (+    57us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:00.629518 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:00.629587 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:00.629714 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":598208,"prepare.queue_time_us":694895,"prepare.run_cpu_time_us":4204,"prepare.run_wall_time_us":25399,"raft.queue_time_us":2603,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":6249,"thread_start_us":553500,"threads_started":2}
W20260709 12:28:00.670984  5553 rpcz_store.cc:269] 0709 12:27:57.973881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:58.064243 (+ 90362us) service_pool.cc:224] Handling call
0709 12:28:00.624881 (+2560638us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b54000eb5d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:00.625004 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1841244544}
W20260709 12:28:00.673187  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51557.8us
I20260709 12:28:00.652010  5526 leader_election.cc:400] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
W20260709 12:28:00.675143  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 85185.5us
I20260709 12:28:00.675516  5526 leader_election.cc:403] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:28:00.676107  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:00.654520  6274 pending_rounds.cc:85] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 13
I20260709 12:28:00.677301  6274 pending_rounds.cc:107] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:28:00.677610  6274 raft_consensus.cc:2889] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:00.677949  6274 pending_rounds.cc:107] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 21.15
I20260709 12:28:00.678208  6274 raft_consensus.cc:2889] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:00.655072  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:00.637378 13368 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.684528 13368 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:00.640419 13494 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:00.630825 13643 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:28:00.643599 13488 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:00.664759  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1368) took 1715 ms. Trace:
I20260709 12:28:00.665483 13532 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:00.666656 13518 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:00.666730 13203 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:00.667850 13505 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:00.639088 13157 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.539s	user 0.009s	sys 0.002s
W20260709 12:28:00.680805  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49435us
I20260709 12:28:00.683564  6275 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:28:00.701501  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:00.702509  6272 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:00.588464 13506 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 12:28:00.668377 13534 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:00.688383  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:00.690323  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:00.692911  6271 rpcz_store.cc:276] 0709 12:27:58.949739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:59.113154 (+163415us) service_pool.cc:224] Handling call
0709 12:27:59.113515 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:27:59.113921 (+   406us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:27:59.113965 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:27:59.114514 (+   549us) log.cc:844] Serialized 26 byte log entry
0709 12:27:59.463824 (+349310us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:27:59.463864 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:27:59.463925 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:00.600203 (+1136278us) spinlock_profiling.cc:243] Waited 3.15 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:00.664418 (+ 64215us) raft_consensus.cc:1685] finished
0709 12:28:00.664440 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:00.664688 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":37274,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"spinlock_wait_cycles":7238016,"thread_start_us":312051,"threads_started":1,"wal-append.queue_time_us":561090}
I20260709 12:28:00.706974  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:00.707886  6276 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:00.671073 13496 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:00.714645  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:00.722443 13518 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:28:00.725252 13559 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:00.726179 13505 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:00.726056 13559 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.726824 13505 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:00.711560 13157 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.732432  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:00.733175  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3034) took 3177 ms. Trace:
W20260709 12:28:00.733855  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:00.734385  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:00.734800 13266 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:00.734941 13494 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.685853 13629 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:00.736280 13494 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:00.736430  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:00.709431 13601 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:28:00.693410 13183 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:28:00.738274 13183 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:00.739207 13183 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.693046 13627 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:00.740957 13627 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:00.741314 13627 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:00.741643 13627 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:00.742383  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:00.733592  5705 rpcz_store.cc:276] 0709 12:27:57.556403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:58.851818 (+1295415us) service_pool.cc:224] Handling call
0709 12:27:58.852217 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:00.034315 (+1182098us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:00.732931 (+698616us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54002f4254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:00.733102 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2584440064}
I20260709 12:28:00.745311  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1621) took 1087 ms. Trace:
I20260709 12:28:00.745682  5635 rpcz_store.cc:276] 0709 12:27:59.657673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:59.659826 (+  2153us) service_pool.cc:224] Handling call
0709 12:27:59.723364 (+ 63538us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:00.328970 (+605606us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:00.745060 (+416090us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b540009e7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:00.745255 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1175342848}
I20260709 12:28:00.746929  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.747571  5702 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:00.743039  6278 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 16.
I20260709 12:28:00.748445  5709 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
W20260709 12:28:00.750396  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62184.4us
W20260709 12:28:00.752691  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 62474.9us
I20260709 12:28:00.755357  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:00.696394  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:00.757650  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 61575.1us
I20260709 12:28:00.694574 13495 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20.
I20260709 12:28:00.776549  5709 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:00.700493 13463 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:00.750182  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:00.718299 13540 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:00.762992  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:00.781729  5554 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:28:00.783241  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.782049 13534 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:28:00.792639  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:00.704315 13643 consensus_queue.cc:1243] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 22.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:28:00.781756  5708 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: [17.15-20.16]
I20260709 12:28:00.783329  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:00.795729  5702 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:28:00.796025  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.797082  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.798055  6278 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:00.714710 13506 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:28:00.792810  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:00.789252  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:00.806141  5704 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:28:00.799255  5701 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:00.773876 13521 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:28:00.790112 13664 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 12:28:00.774966 13604 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:00.775847 13614 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:28:00.818481  5702 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:00.792863 13532 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:00.834733  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:00.752542 13496 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:00.778412 13603 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.045s
I20260709 12:28:00.818223 13157 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:00.816679 13370 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:00.845832  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:00.797696 13463 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.809849 13506 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:00.819450 13534 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:28:00.793162 13596 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.040s
W20260709 12:28:00.840580 13595 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.729s	user 0.013s	sys 0.004s
I20260709 12:28:00.819653 13521 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:00.798919 13629 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.041s
I20260709 12:28:00.793982 13572 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.040s
W20260709 12:28:00.854278  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 61724.2us
I20260709 12:28:00.822108 13368 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:00.823576 13541 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:00.887326 13368 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:00.887773 13629 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:00.889550 13368 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:00.833753 13604 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.891814 13604 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:00.873241 13595 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.785s	user 0.014s	sys 0.004s
I20260709 12:28:00.832990 13532 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:00.873066  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 126598us
I20260709 12:28:00.916261  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:00.846608 13559 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:00.849587 13454 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:28:00.832855 13645 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 12:28:00.866173 13524 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:00.851213  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:00.895326  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:00.910650 13601 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:00.920208  6272 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:00.888192 13629 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:00.914800 13368 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:00.853649 13636 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:00.857465 13468 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:28:00.881654 13521 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:28:00.872944 13460 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 12:28:00.869153 13518 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260709 12:28:00.943774  5635 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:28:00.944054  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:00.951681  5636 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
W20260709 12:28:00.948895 13665 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 12:28:00.921682 13532 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:00.921177 13524 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:00.919095 13322 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:00.871106 13667 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.074s
I20260709 12:28:00.921391 13454 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:00.968106  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:00.950963 13398 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:00.924724 12753 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:00.969230 13524 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:28:01.002852  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 148640us
I20260709 12:28:01.006631  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:01.007325  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:01.007900  6278 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:28:00.938184 13559 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 12:28:00.962070 13604 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:00.951323 13601 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.939886 13368 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:00.979303 13454 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:00.969762 13370 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:28:00.979590 13505 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:01.030702  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 133408us
I20260709 12:28:01.037050  5630 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
I20260709 12:28:01.037559  5630 pending_rounds.cc:85] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 10
I20260709 12:28:01.037786  5630 pending_rounds.cc:107] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 12:28:01.037808  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.038317  5630 raft_consensus.cc:2889] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:00.991374 13398 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:00.990513 13549 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:01.035214 13614 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:00.974975 13643 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.059s
I20260709 12:28:00.954429 13506 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:01.057672  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 206582us
I20260709 12:28:01.051801 13398 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:01.030046 13604 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:00.979791 13322 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.064416  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.065954  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:01.066792  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:01.066874  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:01.100786  5557 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:28:01.100879  5555 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:28:01.033910 13370 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:01.067448  5553 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:28:01.051573 13549 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:01.106060  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:01.106689  5558 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:01.110245  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:01.111115  5552 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:01.116194  5636 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:01.061543 13468 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:28:01.119802  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:01.080525 13604 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:01.119653 13682 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:01.102450 13667 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:01.125916  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:01.131172  5557 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
W20260709 12:28:01.121559 13264 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.560s	user 0.010s	sys 0.000s
I20260709 12:28:01.108548 13549 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:01.061143 13614 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.122961 13681 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:01.153698  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.154305  6276 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:28:01.122153 13468 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:01.124377 13682 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:01.154534  6278 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 17.
I20260709 12:28:01.118844 13370 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:01.167567  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 134848us
I20260709 12:28:01.170084  5558 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:01.139093 13496 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.174405  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:01.163551 13586 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.748s	user 0.013s	sys 0.001s
W20260709 12:28:01.179296 13672 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260709 12:28:01.179096 13586 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.150278 13264 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.181782 13586 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.172317 13368 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:01.182412 13264 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:01.183526 13368 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:01.184018 13368 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:01.184427 13368 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:01.184752  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.185446  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:01.185890  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:01.187378  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:01.189589 13575 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:28:01.190506 13679 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:28:01.191332  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:01.191898 13676 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260709 12:28:01.178323  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.176942 13496 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:01.185993  6278 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:28:01.229600 13649 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.561s	user 0.003s	sys 0.001s
I20260709 12:28:01.229131 13496 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.192948 13157 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:28:01.193712 13594 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 12:28:01.234697 13523 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.785s	user 0.007s	sys 0.005s
I20260709 12:28:01.235148 13594 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:01.235314 13523 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.235903 13594 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:01.194391 13575 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:01.195374 13368 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.237248 13594 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.238528 13594 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:01.239133 13368 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.196836  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.240177  5631 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:28:01.241333  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:01.242012  5631 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:28:01.242763 13488 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.244047 13488 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:01.200826 13663 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:01.254894 13663 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.257244 13663 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.201839  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:01.183435 13468 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.204732 13627 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:01.258035  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56318.4us
I20260709 12:28:01.258208 13468 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:01.259516 13627 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:01.208478 13524 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.260439 13266 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:28:01.260917 13266 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:01.261340 13266 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:28:01.262253 13524 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.208798 13617 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.263260 13617 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:01.209997 13521 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:01.265467 13575 consensus_queue.cc:1175] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "38736cc545eb4389b220a3bd21d3b466" responder_term: 21 status { last_received { term: 21 index: 16 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: [20.15-20.15]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:28:01.211822 13532 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.216655  5628 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 18. Current term is 19. Ops: []
I20260709 12:28:01.266927 13521 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:01.267160  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:01.267987  5634 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:01.268700 13532 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.270689 13511 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:28:01.271179 13511 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:01.271555 13511 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:01.271740 13463 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:01.272619 13627 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:28:01.220546 13636 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.140039 13667 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.275331  5706 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:28:01.276917 13667 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:01.277334  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 1552) took 1244 ms. Trace:
I20260709 12:28:01.277681  5554 rpcz_store.cc:276] 0709 12:28:00.032892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:00.057925 (+ 25033us) service_pool.cc:224] Handling call
0709 12:28:01.277186 (+1219261us) spinlock_profiling.cc:243] Waited 707 ms on lock 0x7b54001bd854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:01.277282 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1626179840}
I20260709 12:28:01.279047  5631 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:28:01.280055  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:01.177521 13677 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260709 12:28:01.281297 13508 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:01.281785 13508 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:01.178408 13680 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:28:01.173273  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.232064 13496 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.198778  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:01.245885 13322 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:01.250226  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:01.284046  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:28:01.284067  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 85380.1us
I20260709 12:28:01.286249  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:01.287380 13264 raft_consensus.cc:2764] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:28:01.287596  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:01.250594  5703 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:01.288962 13322 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:01.289112 13496 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:28:01.288606  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:01.289736 13322 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:01.282595 13508 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:01.290721  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:01.291262 13508 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:01.291755 13322 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:01.164948 13666 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:01.287555  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3342) took 1218 ms. Trace:
I20260709 12:28:01.309156  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:01.332608  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44080.5us
I20260709 12:28:01.292096  6278 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:28:01.162827 13682 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:01.382659  5706 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:28:01.383471  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 85652.4us
W20260709 12:28:01.386941  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 89128us
I20260709 12:28:01.309465  5710 rpcz_store.cc:276] 0709 12:28:00.069130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:00.070544 (+  1414us) service_pool.cc:224] Handling call
0709 12:28:01.287368 (+1216824us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x7b540026f054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:01.287499 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":670017408}
I20260709 12:28:01.390581  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:01.390571  5633 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: []
I20260709 12:28:01.395260  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:01.227137 13612 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:01.402486  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.406050  5633 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:28:01.404227  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.163319 13681 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.393399 13524 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:01.418386  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.418996  5706 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:01.272341 13463 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.137s
I20260709 12:28:01.398808 13183 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.021s
I20260709 12:28:01.436975 13463 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:28:01.437291 13183 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:01.438097 13183 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:01.399258 13370 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:01.442499 13523 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:01.421121  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:01.392762 13521 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.025s
I20260709 12:28:01.413268 13487 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:01.444571 13487 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.446089 13681 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.446624 13487 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.415122 13612 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.422046 13398 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.448696 13612 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.428499 13524 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:01.450775  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:01.420934 13601 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:01.473712  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:01.389130 13322 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.028s
W20260709 12:28:01.424088 13605 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.858s	user 0.011s	sys 0.002s
I20260709 12:28:01.410183 12753 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.437474 13463 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:01.452703  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.502704  5633 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:01.452703  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:01.453215  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:01.505292  5553 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:28:01.506716  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:01.507643  5553 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
W20260709 12:28:01.453801  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:01.454097  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.507519  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:01.510476  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:01.455183  5558 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:01.465453  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:01.469630 13524 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.490702  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:01.516556  5554 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:28:01.504204  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.395815 13599 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.025s
I20260709 12:28:01.494932  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:01.404785 13595 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:28:01.528704 13595 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:01.529685 13595 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:01.490697  6277 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1277) took 3249 ms (client timeout 2209 ms). Trace:
I20260709 12:28:01.499127  5552 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: [24.12-24.12]
W20260709 12:28:01.531774  6277 rpcz_store.cc:269] 0709 12:27:58.201700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:58.202372 (+   672us) service_pool.cc:224] Handling call
0709 12:28:01.451165 (+3248793us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b5400772754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:01.451262 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4144444800}
W20260709 12:28:01.490386 13613 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.775s	user 0.033s	sys 0.016s
W20260709 12:28:01.532959 13613 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.817s	user 0.033s	sys 0.017s
I20260709 12:28:01.497480 13605 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.535148  5630 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 26: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:01.537182 13398 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:01.538175 13511 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.549082 13614 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:01.549693 13183 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.551160 13511 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:01.511893  5631 raft_consensus.cc:1813] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Handling vote request from an unknown peer cea66aa15be3423880843974acdbf50f
I20260709 12:28:01.393473 13666 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.552259 13468 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:01.552855  5631 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:28:01.554301  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.555099  5631 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:01.555701 13666 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.556286  5636 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:01.518625 13496 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 12:28:01.563469 13496 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:28:01.563831 13496 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:01.564284  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:01.565006  6273 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:01.564621 13496 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.566069 13496 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:28:01.567011 13662 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.029s	user 0.000s	sys 0.002s
I20260709 12:28:01.569761  5553 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:28:01.572849  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:01.573747  6271 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:01.581068  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:01.581972  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:01.590416  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:01.591751  5631 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:28:01.593323  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.594111  5631 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:01.595186  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.598320  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:01.599128  6276 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:01.599995  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:01.600802  6276 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:01.611752  5636 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:28:01.613400  6276 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 18. Current term is 19. Ops: [17.12-18.13]
I20260709 12:28:01.610332  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:01.620257  6276 pending_rounds.cc:85] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 17
I20260709 12:28:01.620712  6276 pending_rounds.cc:107] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 25.18
I20260709 12:28:01.620965  6276 raft_consensus.cc:2889] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:01.624657  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:01.624953 13488 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:01.625654 13488 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:01.626641 13488 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.627341  6279 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:28:01.628396  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:01.628265  6273 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:01.551383 13183 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:01.634887 13682 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.611318 13396 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:01.636826 13396 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.639534 13396 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:01.640336  6275 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: [20.16-20.16]
I20260709 12:28:01.553262 13468 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.643397 13468 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:01.647240  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:01.648240  6271 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:28:01.613523  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:01.631675 13549 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:01.520522  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:01.523257  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:01.515971 13266 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.525928 13463 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:01.503507 13676 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:28:01.527477  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:01.658809  5558 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 17: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:01.514117  5628 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:01.517273  5633 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:01.540367  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:01.536051 13605 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.519068 13681 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:28:01.543190 13454 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.645900  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2929) took 1508 ms. Trace:
I20260709 12:28:01.652180  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:01.673273  5553 raft_consensus.cc:2410] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 25 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:28:01.652956  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:01.674558  6276 raft_consensus.cc:2410] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 25 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:28:01.654755  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:01.655447  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:01.662287  5632 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:01.676311  6272 raft_consensus.cc:2368] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21.
I20260709 12:28:01.676745  6273 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:01.663146  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:01.663486  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:01.615545 13203 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:01.668949  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 267378us
I20260709 12:28:01.685242  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.512231  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:01.692575  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:01.692690  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:01.693125  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:01.694001  5555 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 24: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:01.671489  5560 rpcz_store.cc:276] 0709 12:28:00.138002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:00.224892 (+ 86890us) service_pool.cc:224] Handling call
0709 12:28:00.229230 (+  4338us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:00.231605 (+  2375us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:00.231643 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:00.309002 (+ 77359us) log.cc:844] Serialized 52 byte log entry
0709 12:28:00.486154 (+177152us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:00.486215 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:00.486320 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:01.566317 (+1079997us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:01.574708 (+  8391us) raft_consensus.cc:1685] finished
0709 12:28:01.574739 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:01.645834 (+ 71095us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":643,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":8880,"spinlock_wait_cycles":27217664,"thread_start_us":55127,"threads_started":1,"wal-append.queue_time_us":122490}
I20260709 12:28:01.654983 13396 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:01.697332  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.699152  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1305) took 1021 ms. Trace:
I20260709 12:28:01.659132 13468 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:01.699635  6274 rpcz_store.cc:276] 0709 12:28:00.677467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:00.759998 (+ 82531us) service_pool.cc:224] Handling call
0709 12:28:00.795374 (+ 35376us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:00.809023 (+ 13649us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:00.809054 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:00.809527 (+   473us) log.cc:844] Serialized 26 byte log entry
0709 12:28:00.871940 (+ 62413us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:00.910634 (+ 38694us) log.cc:844] Serialized 14 byte log entry
0709 12:28:00.910885 (+   251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:00.910966 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:01.698914 (+787948us) raft_consensus.cc:1685] finished
0709 12:28:01.698946 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:01.699084 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14967,"prepare.queue_time_us":195,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":37,"raft.queue_time_us":14251,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":18915,"thread_start_us":62063,"threads_started":1,"wal-append.queue_time_us":156051}
W20260709 12:28:01.665723 13604 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.527s	user 0.007s	sys 0.005s
I20260709 12:28:01.724676  5701 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:28:01.726843  5635 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:28:01.703171  5631 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:28:01.666138 13681 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:01.670382 12753 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:01.702968  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:01.734292  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:01.703578  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 191461us
I20260709 12:28:01.704493  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:01.738072  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:01.741250  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:01.683111 13614 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:01.727608 13494 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:28:01.702400 13396 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:01.748456  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45579.2us
I20260709 12:28:01.749748  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:01.751459  6273 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:01.738258 13681 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:01.699831 13203 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.748876 13690 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:01.756698 13505 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.005s
W20260709 12:28:01.760107  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:01.761065  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:01.784875  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:01.786517  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:01.761299  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:01.764062  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:01.762545  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:01.732724 13468 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.769438  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:01.772147 13617 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:01.744268 13601 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:01.780262  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:01.788300  5710 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:01.788978  5707 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:01.757670  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:01.809044  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 51464.8us
I20260709 12:28:01.729235 13604 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:01.755295 13396 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:01.727301 13486 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.031s
I20260709 12:28:01.789026  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:01.684408 13511 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.095s
I20260709 12:28:01.813601  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:01.818588  5708 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:01.746315 12753 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:01.863541  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:01.787562 13495 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:01.837662  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:01.781900 13690 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:01.756234 13614 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.835821  6277 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 22. Ops: []
I20260709 12:28:01.933787  5561 raft_consensus.cc:2410] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:28:01.892261 13524 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:01.950337 13524 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:01.951385 13266 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:01.831530 13572 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:01.779961 13322 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:01.877875 13460 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:01.809257 13682 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:01.894397  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 81082.4us
I20260709 12:28:01.805548 13494 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:01.807260 13617 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:01.757247 13688 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:02.006923  5632 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:28:01.990932  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:02.000792  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:02.011099 13454 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:01.952056 13524 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:28:02.011806 13695 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:28:02.012092 13627 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:02.012993 13627 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:02.013949 12753 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:02.013887 13627 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.11, Last appended by leader: 11, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.879571 13157 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.174s
I20260709 12:28:02.008960 13549 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:02.016988 13614 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:02.006800 13494 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:02.017849 13494 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:02.008888 13617 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:02.023140 13506 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:01.988657 13460 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:02.025835 13460 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:01.955721 13690 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:01.987768 13495 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:01.988734 13572 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:02.034461 13468 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:02.032269 13668 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.149s	user 0.000s	sys 0.003s
W20260709 12:28:02.042884  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 224333us
I20260709 12:28:02.067860  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:02.040619  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 205096us
W20260709 12:28:02.048151 13700 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:28:02.072880  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1379) took 1834 ms. Trace:
W20260709 12:28:02.077081  6237 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.716s (SENT) [suppressed 28 similar messages]
I20260709 12:28:02.041901  5636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:28:02.078828  5636 raft_consensus.cc:606] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:02.079221  5636 raft_consensus.cc:614] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:28:02.054225  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:02.062711  5710 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:02.012209 13511 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:02.066680 13572 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:02.082195  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:02.091328  5710 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 17: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:02.073349  6280 rpcz_store.cc:276] 0709 12:28:00.238987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:00.239489 (+   502us) service_pool.cc:224] Handling call
0709 12:28:00.305867 (+ 66378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:00.306239 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:00.306273 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:00.360775 (+ 54502us) log.cc:844] Serialized 26 byte log entry
0709 12:28:01.253668 (+892893us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b18000229c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:28:01.694269 (+440601us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:01.727348 (+ 33079us) log.cc:844] Serialized 14 byte log entry
0709 12:28:02.067779 (+340431us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b18000229c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:28:02.072531 (+  4752us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:02.072638 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:02.072680 (+    42us) raft_consensus.cc:1685] finished
0709 12:28:02.072693 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:02.072819 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38634,"prepare.queue_time_us":34529,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":27776,"raft.run_cpu_time_us":240,"raft.run_wall_time_us":7951,"spinlock_wait_cycles":52561536,"thread_start_us":422553,"threads_started":1,"wal-append.queue_time_us":1401182}
I20260709 12:28:02.085057  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:02.092645  6273 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:02.086222  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:02.089331  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:02.098191  6279 raft_consensus.cc:2368] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23.
I20260709 12:28:02.098325  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:01.903760 13687 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.205s
I20260709 12:28:02.083691  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:02.105254  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:02.107213  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:02.108932  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1549) took 2235 ms. Trace:
I20260709 12:28:02.113186  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:02.113750  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:02.113811  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 295925us
I20260709 12:28:02.095085 13468 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:02.105225  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:02.108299  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:02.114040  5706 raft_consensus.cc:2410] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 25 because replica has last-logged OpId of term: 24 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
I20260709 12:28:02.114750  5636 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 22: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:02.103281  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:02.109292  5559 rpcz_store.cc:276] 0709 12:27:59.874098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:59.874482 (+   384us) service_pool.cc:224] Handling call
0709 12:27:59.874960 (+   478us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:00.096484 (+221524us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:28:00.616088 (+519604us) spinlock_profiling.cc:243] Waited 6.97 ms on lock 0x7b18003e5120. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf
0709 12:28:00.616295 (+   207us) log.cc:844] Serialized 14 byte log entry
0709 12:28:00.802304 (+186009us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:00.812411 (+ 10107us) log.cc:844] Serialized 26 byte log entry
0709 12:28:00.812591 (+   180us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:00.812624 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:00.812686 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:02.030956 (+1218270us) spinlock_profiling.cc:243] Waited 67.2 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:02.108730 (+ 77774us) raft_consensus.cc:1685] finished
0709 12:28:02.108752 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:02.108877 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61767,"prepare.queue_time_us":210664,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":95,"raft.queue_time_us":68531,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":7946,"spinlock_wait_cycles":170635008,"thread_start_us":547063,"threads_started":3,"wal-append.queue_time_us":224138}
I20260709 12:28:02.100956  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:02.081367 13575 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:02.122167  5636 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:28:02.115518  5676 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:02.124311  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:02.119428  5524 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:02.119457  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:02.126228  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:02.129386  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:02.130779  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:02.143929  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:02.145532  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:02.146934  5553 raft_consensus.cc:2410] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:28:02.120101 13614 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:28:02.117928 13488 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:02.181754  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:02.132077  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:02.117727 13468 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:02.096737 13511 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.176623  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:02.183450  5557 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:28:02.138422 13660 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.625s	user 0.006s	sys 0.000s
I20260709 12:28:02.188084  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:02.138084 13701 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
W20260709 12:28:02.186257  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:02.195699  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:02.205930  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 128855us
I20260709 12:28:02.135270 13708 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:02.134666 13604 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:02.180733 13203 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:02.127638 13575 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.137575 13624 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:02.176769 13495 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:02.188786 13468 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:02.187822 12762 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:02.192183 13157 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:28:02.236420  5557 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:02.249567  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:02.250790  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:02.242306  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:02.223872 13697 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:02.226014 13370 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:28:02.193984 13488 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:02.239758 13702 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.285s	user 0.000s	sys 0.001s
W20260709 12:28:02.260435  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 72428us
I20260709 12:28:02.193475 13699 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.082s
I20260709 12:28:02.265102  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:02.267835  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:02.269788 13714 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:02.270560 13575 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:02.273341 13488 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:02.278175 13681 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.248083 13624 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:02.256170 13698 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.214s	user 0.002s	sys 0.000s
I20260709 12:28:02.282476 13624 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:02.282192 13524 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.283361  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3086) took 1171 ms. Trace:
I20260709 12:28:02.284628  5634 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:02.283731  5705 rpcz_store.cc:276] 0709 12:28:01.111547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:01.113493 (+  1946us) service_pool.cc:224] Handling call
0709 12:28:01.113913 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:01.227510 (+113597us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:01.227545 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:01.228046 (+   501us) log.cc:844] Serialized 26 byte log entry
0709 12:28:01.734907 (+506861us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:01.734957 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:01.735004 (+    47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:02.283147 (+548143us) raft_consensus.cc:1685] finished
0709 12:28:02.283170 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:02.283302 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6596,"raft.run_cpu_time_us":50,"raft.run_wall_time_us":50,"thread_start_us":442656,"threads_started":2,"wal-append.queue_time_us":526066}
I20260709 12:28:02.285637  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:02.289335 13496 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.723s	user 0.012s	sys 0.003s
W20260709 12:28:02.290323 13705 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
I20260709 12:28:02.253968 12762 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.262879 13697 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.232038 13709 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 12:28:02.306718  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2928) took 2221 ms. Trace:
I20260709 12:28:02.231818 13708 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.307304  5556 rpcz_store.cc:276] 0709 12:28:00.085221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:00.219831 (+134610us) service_pool.cc:224] Handling call
0709 12:28:00.220311 (+   480us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:00.236722 (+ 16411us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:28:00.236756 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:00.301268 (+ 64512us) log.cc:844] Serialized 26 byte log entry
0709 12:28:00.428444 (+127176us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:28:00.428586 (+   142us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:00.428723 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:02.106403 (+1677680us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:02.306363 (+199960us) raft_consensus.cc:1685] finished
0709 12:28:02.306387 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:02.306646 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3051,"raft.queue_time_us":1780,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":12962,"spinlock_wait_cycles":5376384,"thread_start_us":72228,"threads_started":1,"wal-append.queue_time_us":147604}
I20260709 12:28:02.311896  5558 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:02.293200  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:02.324294  5633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:28:02.286696  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:02.330768  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:02.331393  5708 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:02.334033  5633 raft_consensus.cc:606] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:02.334429  5633 raft_consensus.cc:614] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:02.339502  5561 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:28:02.340780  5705 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:28:02.345890  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:02.346733  5705 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:02.342195  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 74446us
I20260709 12:28:02.349574  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:02.366950  5558 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:28:02.367450  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:02.370823  5526 leader_election.cc:400] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23.
I20260709 12:28:02.371485  5526 leader_election.cc:403] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:28:02.374914  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:02.375795  5702 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:02.265018 13650 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:02.414758  5633 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:28:02.401803  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:02.417974  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:02.378212 13645 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:28:02.333408 13203 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:02.453624 13460 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.454337 13203 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.410224 13496 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:28:02.361467 13575 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:02.379026 13370 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:28:02.436676 13715 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:28:02.436967  5633 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:02.453768  5708 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:28:02.442499 13690 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.423761 13506 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:02.424803 13183 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.792s	user 0.013s	sys 0.000s
I20260709 12:28:02.502322  5633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:28:02.511093  5633 raft_consensus.cc:606] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:02.437451 12762 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:02.430231 13617 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:02.506705  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 104987us
I20260709 12:28:02.391371 13719 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:02.459096 13454 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:02.569552  5702 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:02.569749  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:02.579368  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:02.465708 13370 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:02.568846  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:02.579453  5633 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:02.584439  5705 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 29 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
W20260709 12:28:02.587437  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:02.506377 13460 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:02.592826  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:02.601317  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:02.602236  5705 raft_consensus.cc:2410] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:28:02.604084  5703 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
W20260709 12:28:02.605051  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:02.607594  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:02.610589 13726 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:02.611737 13726 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.614641 13726 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:02.615675 13716 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:02.616243 13716 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.586869 13454 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.620249  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:02.621461 13454 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:02.510555 13614 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:02.623214  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:02.628110  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:02.629547  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:02.630384  5705 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:02.632160  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:02.463886 13575 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.577479 13712 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:02.633635 13708 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:02.586215 13370 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:02.505944 13494 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.574625 13724 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:02.638330  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:02.607143 13396 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.729s	user 0.009s	sys 0.005s
I20260709 12:28:02.644981  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:02.645327  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:02.645723  5708 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:02.527427 13506 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.643328  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2953) took 1055 ms. Trace:
I20260709 12:28:02.649073  5628 rpcz_store.cc:276] 0709 12:28:01.588414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:01.881324 (+292910us) service_pool.cc:224] Handling call
0709 12:28:01.924985 (+ 43661us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:01.925489 (+   504us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:01.925534 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:01.926201 (+   667us) log.cc:844] Serialized 26 byte log entry
0709 12:28:02.177789 (+251588us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:28:02.177835 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:02.177908 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:02.643071 (+465163us) raft_consensus.cc:1685] finished
0709 12:28:02.643100 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:02.643257 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3450,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":3650,"thread_start_us":209561,"threads_started":1,"wal-append.queue_time_us":376580}
I20260709 12:28:02.618917 13716 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:02.591354 13719 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:02.653157  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:02.652585 13722 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.007s
I20260709 12:28:02.666931  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:02.667956  6274 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:02.670723  5705 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 24 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:28:02.644066 13663 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:28:02.606570 13650 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.050s
I20260709 12:28:02.643545 13511 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:02.656324 13595 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:28:02.660786 13460 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:02.665930 13716 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:02.660629 13690 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:02.662844 13697 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:02.706427  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:02.677654 13494 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:02.647732 13712 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.654654 13396 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:02.717278  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62555us
I20260709 12:28:02.681165 13729 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:02.722882 13729 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.726055 13729 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:02.726940  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:02.727895  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:02.728245 13595 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:28:02.728845 13663 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:02.729242 13719 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.729972  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:02.737080 13575 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:02.737977  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1312) took 1157 ms. Trace:
I20260709 12:28:02.738406  6271 rpcz_store.cc:276] 0709 12:28:01.580286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:01.586247 (+  5961us) service_pool.cc:224] Handling call
0709 12:28:01.647977 (+ 61730us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:01.648689 (+   712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:02.737735 (+1089046us) spinlock_profiling.cc:243] Waited 61.3 ms on lock 0x7b54002a8350. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:02.737905 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":140929280}
I20260709 12:28:02.759956  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:02.760659  5628 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:02.741428 13722 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:02.728843 13595 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:02.713598 13460 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.768577 13617 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:02.780967 13730 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 12:28:02.717204 13716 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.764173 13454 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:02.788326  5633 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:28:02.763388 13722 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:02.792281  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:02.795256  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:02.717825 13614 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.807001 13716 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:02.789883 13617 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:02.790654 13575 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:02.803954 13697 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:02.806874  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:02.800868 13539 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:02.817121  5633 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:02.816288 13496 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23.
I20260709 12:28:02.821986  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:02.818723 13572 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.714s	user 0.015s	sys 0.000s
I20260709 12:28:02.803222 13722 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.820173 13595 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.843286 13712 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:02.846055 13572 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.847474  5709 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:02.850025 13572 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:02.821064 13468 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.565s	user 0.011s	sys 0.001s
I20260709 12:28:02.857285 13460 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:02.862043  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1629) took 1721 ms. Trace:
I20260709 12:28:02.863080 13396 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:02.864511 13685 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.746s	user 0.001s	sys 0.007s
I20260709 12:28:02.864886 13572 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:02.834473 13697 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:02.822707 13539 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.867115 13601 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:02.812310 13617 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.867601 13722 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:02.868295  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 193392us
I20260709 12:28:02.821986 13506 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:02.870791  5635 rpcz_store.cc:276] 0709 12:28:01.140719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:01.140892 (+   173us) service_pool.cc:224] Handling call
0709 12:28:01.141268 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:01.727348 (+586080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:02.861762 (+1134414us) spinlock_profiling.cc:243] Waited 585 ms on lock 0x7b54001cf7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:02.861973 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1346276224}
I20260709 12:28:02.823145 13743 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:02.881618 13743 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.881996  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:02.883939 13743 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:02.814783 13575 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.885421  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:02.886315  6278 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:02.887235 13739 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:02.887878 13739 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.891371 13739 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:02.893309  5635 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:28:02.899284 13575 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:02.876169 13663 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:02.901899 13681 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:02.903420  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 173513us
I20260709 12:28:02.903559  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:02.904680  5635 raft_consensus.cc:2410] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:28:02.904944  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:02.905632  5633 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 19: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:02.905579  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:02.903030 13663 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.908002 13629 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.909125 13629 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:02.911787  5635 raft_consensus.cc:2410] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 25 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 16.
I20260709 12:28:02.915212  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:02.875561 13468 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.917719  5635 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:28:02.930445  5635 raft_consensus.cc:606] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:02.930976  5635 raft_consensus.cc:614] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:02.877756 13572 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:02.933358  5635 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 23 index: 11. (index mismatch)
I20260709 12:28:02.934013  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:28:02.914387 13540 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.995s	user 0.003s	sys 0.011s
I20260709 12:28:02.880326 13601 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.947991  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3080) took 2142 ms. Trace:
W20260709 12:28:02.938664 13540 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.026s	user 0.003s	sys 0.011s
I20260709 12:28:02.948637  5704 rpcz_store.cc:276] 0709 12:28:00.805601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:00.807138 (+  1537us) service_pool.cc:224] Handling call
0709 12:28:00.807495 (+   357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:00.813451 (+  5956us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:00.813482 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:00.830372 (+ 16890us) log.cc:844] Serialized 26 byte log entry
0709 12:28:01.062007 (+231635us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:01.062130 (+   123us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:01.062247 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:02.863898 (+1801651us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:02.938363 (+ 74465us) raft_consensus.cc:1685] finished
0709 12:28:02.938395 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:02.947916 (+  9521us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8862,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":6158,"spinlock_wait_cycles":7555968,"thread_start_us":187821,"threads_started":1,"wal-append.queue_time_us":257553}
I20260709 12:28:02.947957 13370 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:02.988173  5704 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:02.990599 13572 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:02.996606  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:03.009138  5627 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:02.937595 13697 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.015903  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1309) took 1451 ms. Trace:
I20260709 12:28:03.010082  5628 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:28:03.016286  6277 rpcz_store.cc:276] 0709 12:28:01.564085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:01.575089 (+ 11004us) service_pool.cc:224] Handling call
0709 12:28:01.652206 (+ 77117us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:01.948120 (+295914us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.015670 (+1067550us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b5400773154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:03.015844 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":410265472}
W20260709 12:28:03.030472  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 93798.1us
I20260709 12:28:03.033387  5628 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:03.026747 13617 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:03.031476 13673 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 2.138s	user 0.014s	sys 0.000s
W20260709 12:28:03.038240 13749 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:28:03.032505 13719 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:03.039252 13719 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:03.042868 13322 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:03.043308 13322 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:03.044044 13601 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:03.044164 13322 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:03.044811 13468 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:03.041769 13719 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:03.045382 13322 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:03.048818 13724 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:03.049216 13724 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:03.050036 13724 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:03.051247 13724 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:03.061213 13370 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:03.054219 13539 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:03.065115  5630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2937) took 2010 ms (client timeout 1884 ms). Trace:
I20260709 12:28:03.065315  5636 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: [25.14-25.14]
W20260709 12:28:03.065577  5630 rpcz_store.cc:269] 0709 12:28:01.055059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:01.064136 (+  9077us) service_pool.cc:224] Handling call
0709 12:28:03.064915 (+2000779us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b54001dd8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:03.065049 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":999156608}
I20260709 12:28:03.067085  5630 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:28:03.067499  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3190) took 1406 ms. Trace:
I20260709 12:28:03.065280 13612 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:03.067879  5636 rpcz_store.cc:276] 0709 12:28:01.661561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:02.127580 (+466019us) service_pool.cc:224] Handling call
0709 12:28:03.065045 (+937465us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:03.065916 (+   871us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.067318 (+  1402us) spinlock_profiling.cc:243] Waited 937 ms on lock 0x7b54001dd8d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:03.067443 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2155182336}
I20260709 12:28:03.080765  5630 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 17. Current term is 18. Ops: []
I20260709 12:28:03.083563  5630 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:28:03.088229 13663 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:03.097373 13663 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:03.097918  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:03.098563  5558 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:28:03.099745 13663 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:28:03.101055  5558 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:03.102101  5561 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 18. Current term is 19. Ops: []
I20260709 12:28:03.102484  5558 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 16. (term mismatch)
I20260709 12:28:03.102787  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:03.103307  5561 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:03.104327  5555 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:28:03.104830  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:03.105926  5560 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 27 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:28:03.106941  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:03.102816  5558 pending_rounds.cc:85] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 15
I20260709 12:28:03.107985  5558 pending_rounds.cc:107] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 20.16
I20260709 12:28:03.107744  5557 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:28:03.108202  5558 raft_consensus.cc:2889] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:03.109107  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:03.100261 13663 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:03.115005  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:03.115780  5558 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:03.112916  5556 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:03.119396  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:03.131059 13398 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:03.131844 13398 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:03.132798  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3364) took 1003 ms. Trace:
I20260709 12:28:03.133266  5705 rpcz_store.cc:276] 0709 12:28:02.128888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:02.652715 (+523827us) service_pool.cc:224] Handling call
0709 12:28:03.132515 (+479800us) spinlock_profiling.cc:243] Waited 7.4 ms on lock 0x7b5400323a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:03.132717 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17026304}
I20260709 12:28:03.088652 13612 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:03.135015  5705 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:03.139227  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2950) took 1551 ms. Trace:
I20260709 12:28:03.139678  5632 rpcz_store.cc:276] 0709 12:28:01.587532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:01.726199 (+138667us) service_pool.cc:224] Handling call
0709 12:28:01.726531 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:02.009239 (+282708us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.138908 (+1129669us) spinlock_profiling.cc:243] Waited 280 ms on lock 0x7b540018b854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:03.139159 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":644046720}
I20260709 12:28:03.135309  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:03.140625  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3362) took 1023 ms. Trace:
I20260709 12:28:03.136811  5558 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:28:03.137089  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:03.141309  5628 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:28:03.138880  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:03.142107  5561 raft_consensus.cc:2410] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:28:03.142021  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:03.141142  5704 rpcz_store.cc:276] 0709 12:28:02.117388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:03.057470 (+940082us) service_pool.cc:224] Handling call
0709 12:28:03.057737 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:03.057922 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:28:03.058190 (+   268us) log.cc:844] Serialized 14 byte log entry
0709 12:28:03.140096 (+ 81906us) log.cc:844] Serialized 14 byte log entry
0709 12:28:03.140254 (+   158us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:03.140310 (+    56us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:03.140340 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.140446 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:03.140568 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":161,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":829,"spinlock_wait_cycles":146048,"thread_start_us":80496,"threads_started":1}
W20260709 12:28:03.161104  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:03.321713  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 81966.2us
I20260709 12:28:03.333166  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:03.338349  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:03.353114  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:03.144615 13629 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:03.368150  5556 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:03.369299  5555 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:28:03.379323  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:03.132915 13663 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.329828 13708 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:03.329624 13322 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:03.331130 13690 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:03.323922 13724 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:03.327397 13505 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:03.370496 13517 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 12:28:03.417734 13463 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:03.422561 13663 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:03.407719 13539 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.017s
I20260709 12:28:03.426187  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 2955) took 1834 ms. Trace:
I20260709 12:28:03.426735  5634 rpcz_store.cc:276] 0709 12:28:01.591486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:02.121954 (+530468us) service_pool.cc:224] Handling call
0709 12:28:02.122318 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:02.285009 (+162691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.425857 (+1140848us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b540018d654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:03.426091 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":372787968}
I20260709 12:28:03.405066 13317 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260709 12:28:03.434682  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3368) took 1175 ms. Trace:
I20260709 12:28:03.435719 13317 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:03.410554 13755 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 12:28:03.408509 13708 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.443949 13508 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:03.365144 13617 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:28:03.400805 13722 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:03.444602 13508 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:03.445330 13722 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.446907 13508 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:03.449141 13684 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 2.052s	user 0.000s	sys 0.006s
I20260709 12:28:03.408694 13322 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:03.450632  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1627) took 2507 ms. Trace:
I20260709 12:28:03.451001  5629 rpcz_store.cc:276] 0709 12:28:00.943298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:00.943752 (+   454us) service_pool.cc:224] Handling call
0709 12:28:00.970330 (+ 26578us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:00.970622 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:00.970661 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:00.971279 (+   618us) log.cc:844] Serialized 26 byte log entry
0709 12:28:01.128598 (+157319us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:01.128660 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:01.128755 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:03.390637 (+2261882us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:03.450297 (+ 59660us) raft_consensus.cc:1685] finished
0709 12:28:03.450322 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:03.450537 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8402816,"thread_start_us":96750,"threads_started":1,"wal-append.queue_time_us":158387}
I20260709 12:28:03.453773 13612 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:03.383426 13667 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:03.409044 13690 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.455256 13667 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.457357 13690 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:03.457726 13667 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:03.460279 13756 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:28:03.461733  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3199) took 1059 ms. Trace:
I20260709 12:28:03.462309  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3373) took 1019 ms. Trace:
I20260709 12:28:03.462791  5702 rpcz_store.cc:276] 0709 12:28:02.443350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:02.579121 (+135771us) service_pool.cc:224] Handling call
0709 12:28:03.462096 (+882975us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b54002f4254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:03.462248 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":719472512}
I20260709 12:28:03.462675 13724 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:03.469152 13708 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:03.414189 13505 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.401245 13398 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.475055 13398 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:03.478440 13505 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:03.479508 13505 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:03.395551  6237 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:03.473308  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3351) took 1412 ms. Trace:
I20260709 12:28:03.436161 13317 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:03.486423 13720 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.089s	user 0.011s	sys 0.000s
I20260709 12:28:03.485608 13203 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:03.489953 13322 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:03.490501  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 95208.1us
I20260709 12:28:03.521083  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:03.490727 13395 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:28:03.522534 13395 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:03.523878 13395 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.491216 13722 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:03.494791 13711 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:28:03.529692 13711 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 17 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:03.530762 13711 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:03.462172  5635 rpcz_store.cc:276] 0709 12:28:02.402031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:02.935868 (+533837us) service_pool.cc:224] Handling call
0709 12:28:02.936156 (+   288us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:02.936314 (+   158us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:02.936328 (+    14us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:02.945211 (+  8883us) log.cc:844] Serialized 26 byte log entry
0709 12:28:03.034144 (+ 88933us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:03.034178 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.034242 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:03.459667 (+425425us) spinlock_profiling.cc:243] Waited 52.7 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:03.461354 (+  1687us) raft_consensus.cc:1685] finished
0709 12:28:03.461387 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:03.461660 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":121153920,"thread_start_us":81890,"threads_started":1,"wal-append.queue_time_us":196754}
I20260709 12:28:03.435228  5707 rpcz_store.cc:276] 0709 12:28:02.259201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:03.433554 (+1174353us) service_pool.cc:224] Handling call
0709 12:28:03.433994 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:03.434234 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:03.434279 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:03.434340 (+    61us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:03.434386 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.434460 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:03.434611 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:03.535691  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:03.499359  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:03.502391  5708 rpcz_store.cc:276] 0709 12:28:02.064833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:02.666199 (+601366us) service_pool.cc:224] Handling call
0709 12:28:02.671739 (+  5540us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:02.868208 (+196469us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:02.868240 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:02.868288 (+    48us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:28:02.868320 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.473020 (+604700us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54003e6f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:03.473093 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:03.473241 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":451285248}
I20260709 12:28:03.538156  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:03.538637  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:03.539214  5707 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:28:03.539777  5708 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:03.532084 13711 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:28:03.505316 13317 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.542623 13697 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 12:28:03.585908  6272 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1310) took 2021 ms (client timeout 1550 ms). Trace:
W20260709 12:28:03.586634  6272 rpcz_store.cc:269] 0709 12:28:01.580547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:01.580708 (+   161us) service_pool.cc:224] Handling call
0709 12:28:03.585598 (+2004890us) spinlock_profiling.cc:243] Waited 94.3 ms on lock 0x7b54004c7a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:03.585811 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":216837504}
I20260709 12:28:03.546911  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:03.547649  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:03.549350  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:03.589107  5636 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
I20260709 12:28:03.549607  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:03.549989 13524 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:03.550729  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:03.551219  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1634) took 1042 ms. Trace:
I20260709 12:28:03.597990  5628 rpcz_store.cc:276] 0709 12:28:02.508792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:03.036547 (+527755us) service_pool.cc:224] Handling call
0709 12:28:03.139215 (+102668us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:03.141838 (+  2623us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.550957 (+409119us) spinlock_profiling.cc:243] Waited 55.3 ms on lock 0x7b540018b850. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:03.551135 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":127107968}
I20260709 12:28:03.558377  5631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:28:03.600962  5631 raft_consensus.cc:606] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
W20260709 12:28:03.564529  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:03.565380  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:03.508059  5560 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:28:03.592597  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:03.593376  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 94111us
I20260709 12:28:03.596316  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:03.596786  5629 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 22: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:28:03.611450  5703 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3090) took 1848 ms (client timeout 1956 ms). Trace:
I20260709 12:28:03.613052  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:03.612092  5703 rpcz_store.cc:269] 0709 12:28:01.763104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:01.779787 (+ 16683us) service_pool.cc:224] Handling call
0709 12:28:03.611229 (+1831442us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b54004529d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:03.611372 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1385161984}
I20260709 12:28:03.615888  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:03.617441  5703 raft_consensus.cc:2410] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:28:03.624809  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:03.627130  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:03.628026  5558 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:03.630358  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:03.631719  5553 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 23: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:28:03.634572  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:03.605123  5701 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:03.639325  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3378) took 1054 ms. Trace:
I20260709 12:28:03.653861  5704 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19. Ops: []
I20260709 12:28:03.655158  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3097) took 1470 ms. Trace:
I20260709 12:28:03.655777  5704 rpcz_store.cc:276] 0709 12:28:02.184860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:03.399347 (+1214487us) service_pool.cc:224] Handling call
0709 12:28:03.604785 (+205438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:03.654695 (+ 49910us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.655045 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:03.639763  5701 rpcz_store.cc:276] 0709 12:28:02.584556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:03.638453 (+1053897us) service_pool.cc:224] Handling call
0709 12:28:03.638777 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:03.638988 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:03.639033 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:03.639077 (+    44us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:03.639110 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.639168 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:03.639269 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:03.663434  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1428) took 1066 ms. Trace:
I20260709 12:28:03.508637 13724 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:03.776727  5706 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:03.674492  5703 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 12. (term mismatch)
I20260709 12:28:03.678359  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:03.777875  5552 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:03.580832 13370 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:03.596199 13468 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:03.672942 13521 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:28:03.618485 13722 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:28:03.731981 13667 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 12:28:03.585909 13690 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:03.614220 13617 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:03.660655 13692 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:03.715211 13755 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:28:03.719020  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:03.721801 13317 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:03.724753  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:03.818958  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 94365.8us
I20260709 12:28:03.820102  5599 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:03.821754  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:03.744323  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:03.746203  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:03.827448  5707 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:03.828621  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:03.829444  5707 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
W20260709 12:28:03.750877  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:03.761660  5627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2945) took 2558 ms (client timeout 1803 ms). Trace:
I20260709 12:28:03.480301 13505 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.771785  5708 rpcz_store.cc:276] 0709 12:28:02.597561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:03.649381 (+1051820us) service_pool.cc:224] Handling call
0709 12:28:03.662639 (+ 13258us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:03.662912 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:03.662952 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:03.663014 (+    62us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:03.663067 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.663145 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:03.663327 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:03.775645  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3093) took 2004 ms. Trace:
I20260709 12:28:03.778558  5703 pending_rounds.cc:85] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 11
W20260709 12:28:03.832168  5627 rpcz_store.cc:269] 0709 12:28:01.178111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:01.185656 (+  7545us) service_pool.cc:224] Handling call
0709 12:28:03.736535 (+2550879us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b540032c8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:03.736713 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4174390016}
I20260709 12:28:03.833365  5703 pending_rounds.cc:107] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 24.12
I20260709 12:28:03.778059  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:03.833694  5703 raft_consensus.cc:2889] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:03.830773  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:03.618871 13322 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:03.751323 13708 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:03.748566 13601 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:03.783109  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:03.885632  5704 raft_consensus.cc:2410] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:28:03.834831  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3379) took 1233 ms. Trace:
I20260709 12:28:03.886205  5703 rpcz_store.cc:276] 0709 12:28:02.601197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:03.652839 (+1051642us) service_pool.cc:224] Handling call
0709 12:28:03.653293 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:03.834568 (+181275us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.834766 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:03.887378  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:03.888027  5703 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:03.832597  5709 rpcz_store.cc:276] 0709 12:28:01.771356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:01.787912 (+ 16556us) service_pool.cc:224] Handling call
0709 12:28:03.775391 (+1987479us) spinlock_profiling.cc:243] Waited 894 ms on lock 0x7b54002d19d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:03.775571 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2056734976}
I20260709 12:28:03.836158  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:03.837476  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59563.5us
I20260709 12:28:03.893707  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:03.894294  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:03.784775 13724 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:03.796739 13468 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.861528 13663 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:28:03.879271  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:03.802703 13692 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:03.811290 13722 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:03.786417  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:03.888536  5552 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:03.842896 13532 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:28:03.842263 13508 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:03.843181 13624 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:03.896832 13517 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:03.793442 13370 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.956413  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3350) took 1895 ms. Trace:
I20260709 12:28:03.972319  5706 rpcz_store.cc:276] 0709 12:28:02.060532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:02.120367 (+ 59835us) service_pool.cc:224] Handling call
0709 12:28:03.776545 (+1656178us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:03.777174 (+   629us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.956069 (+178895us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54002d19d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:03.956292 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3808124288}
I20260709 12:28:03.894608  5703 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:03.896701  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:03.898161  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:03.813321 13690 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:03.819758 13617 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.904872  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:03.904111  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:03.921343  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:03.974997  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:03.907379 13157 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:03.902715 13741 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:03.908190 13692 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:03.884976 13322 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:03.895795 13183 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:28:03.886885 13708 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:03.979434  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:03.981817  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 83897.6us
I20260709 12:28:04.001664  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:03.909415 13532 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:03.984580  5703 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:28:03.864880  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:03.916193 13508 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:04.005777 13508 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:03.891347 13601 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:03.986569  5704 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:03.989535 13612 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:28:03.957221  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:03.915750 13624 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:03.917742 13719 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:28:03.991204  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 87227us
I20260709 12:28:03.902000 13724 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:03.910920 13524 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 12:28:04.008949  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:04.015848  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:04.011775  5708 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:28:03.994990 13386 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:03.966271 13539 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:04.006832 13532 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:28:04.021447 13612 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:04.027007  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:04.012779 13741 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:04.015798 13322 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:04.028023  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:04.023965 13688 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:04.016387 13719 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:28:04.025470 13398 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.550s	user 0.013s	sys 0.002s
I20260709 12:28:04.045701 13690 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:04.046705 13719 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:04.050328  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 429036us
I20260709 12:28:04.050681  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:04.016115 13624 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:04.009747  5703 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:28:04.028529 13742 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:28:04.054291  5561 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.016s	sys 0.059s
I20260709 12:28:04.055213  5708 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
W20260709 12:28:04.059163  5561 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.016s	sys 0.059s
I20260709 12:28:04.051992  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:04.066867  5703 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:28:04.068217  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:04.027052 13539 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.047971 13629 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:28:04.069893  5630 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 26: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:04.070590  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:04.070904  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:04.074263  5627 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 24: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:04.079629  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.081480  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.072053  5634 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:28:04.062074 13322 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:28:04.086794  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.019670 13722 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:04.064313 13741 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:28:04.069617 13719 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:04.074012  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:04.096542  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.017s	sys 0.024s
W20260709 12:28:04.100976  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.017s	sys 0.025s
W20260709 12:28:04.117501  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 573585us
I20260709 12:28:04.089080 13742 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:04.131717  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.104523  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.095674 13687 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:04.096139 13322 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:04.120571  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:04.136309  5634 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:28:04.114912  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:04.128849  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.129678  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.138453  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:04.139108  6274 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
W20260709 12:28:04.119100 13711 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.523s	user 0.009s	sys 0.004s
I20260709 12:28:04.134593  6275 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:28:04.099103 13468 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:04.139204  5627 raft_consensus.cc:2376] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28. Re-sending same reply.
I20260709 12:28:04.134593  6271 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:28:04.160079  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.162317  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.166260  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.168010  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.177644  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:04.177831  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.178571  5627 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:04.099825 13741 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:04.180213  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.180450  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:04.180938  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:04.181149  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.185560  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:04.119511 13697 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:04.187003  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:04.186825  6280 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:04.189447  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.122715 13370 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:04.191174  6275 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:28:04.187028  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:28:04.132436 13617 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:04.191892  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:04.131652 13742 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:04.114290 13612 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:04.194912  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.225497  6280 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:04.195421  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.227953  6275 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:04.200225  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:04.198112  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:04.230554  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:04.235136  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.010s	sys 0.018s
W20260709 12:28:04.235685  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.010s	sys 0.018s
I20260709 12:28:04.201112  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:04.227946  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:04.236326  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 715329us
I20260709 12:28:04.200717  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:04.237902  6277 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:04.239411  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:04.183092 13517 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:04.241108  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:04.194020 13763 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:28:04.212869 13742 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:04.217849 13370 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:04.224856 13724 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:04.192838 13601 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:04.217626 13468 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:28:04.204720 13708 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:04.253410  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:04.196033 13505 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:04.234766 13627 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:04.273173 13322 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:04.283634 13627 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:04.284044 13627 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:04.232259 13317 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:04.284672 13317 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.206363 13697 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.286772 13317 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:04.287292 13317 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:04.287688 13697 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:04.287940 13627 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:04.272003 13370 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:04.275460 13508 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:04.290377 13370 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:04.291697  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3096) took 2107 ms. Trace:
W20260709 12:28:04.293085 13758 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.949s	user 0.005s	sys 0.000s
I20260709 12:28:04.295111  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1631) took 1787 ms. Trace:
I20260709 12:28:04.292097  5705 rpcz_store.cc:276] 0709 12:28:02.184570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:03.139618 (+955048us) service_pool.cc:224] Handling call
0709 12:28:03.140056 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:03.140418 (+   362us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:03.140445 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:03.144753 (+  4308us) log.cc:844] Serialized 26 byte log entry
0709 12:28:03.762256 (+617503us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:28:03.762295 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.762369 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:04.291467 (+529098us) raft_consensus.cc:1685] finished
0709 12:28:04.291489 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:04.291640 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":406356,"threads_started":1,"wal-append.queue_time_us":744821}
I20260709 12:28:04.295503  5633 rpcz_store.cc:276] 0709 12:28:02.508013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:02.946991 (+438978us) service_pool.cc:224] Handling call
0709 12:28:02.953093 (+  6102us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:02.966180 (+ 13087us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:28:02.987384 (+ 21204us) log.cc:844] Serialized 14 byte log entry
0709 12:28:03.037195 (+ 49811us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:03.055172 (+ 17977us) log.cc:844] Serialized 26 byte log entry
0709 12:28:03.055284 (+   112us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:03.055305 (+    21us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:03.055351 (+    46us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:04.271881 (+1216530us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:04.294755 (+ 22874us) raft_consensus.cc:1685] finished
0709 12:28:04.294863 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:04.295026 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41251,"prepare.queue_time_us":6454,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":29758,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":3809,"spinlock_wait_cycles":302887808,"thread_start_us":22986,"threads_started":1,"wal-append.queue_time_us":143418}
I20260709 12:28:04.288264 13697 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:04.297544 13697 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.299612 13697 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:04.256245  5525 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.019s	sys 0.009s
I20260709 12:28:04.257194  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:04.292097 13508 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:28:04.245226 13539 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:04.280642  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:04.300573  5525 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.019s	sys 0.009s
I20260709 12:28:04.300417  5634 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:28:04.248482  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:04.305771  6279 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:04.300107 13697 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:04.306766  5635 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:04.307729  5629 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:28:04.308315  5556 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:28:04.310657  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.313838  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.323491  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.324462  5558 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:04.325960 13697 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.247889 13517 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.342000  5553 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:04.360687  5552 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:04.273185 13468 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:04.368656  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:04.368494  5558 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:04.370504  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.370805  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:04.372035  5558 raft_consensus.cc:2410] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:28:04.372802  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:04.365092  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.374176  5554 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:04.375747  5555 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:04.377944  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:04.207098 13741 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.390741  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:04.392092  5558 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:04.393409  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 607045us
I20260709 12:28:04.329283 13370 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:04.395826  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.401561  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.402144  5560 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
W20260709 12:28:04.397320 13475 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.708s	user 0.016s	sys 0.011s
I20260709 12:28:04.328933 13708 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:04.404958 13766 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:04.405738 13755 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:28:04.405743 13697 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:04.340343 13532 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.435529  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.437006  5553 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
W20260709 12:28:04.413586 13475 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.016s	sys 0.011s
I20260709 12:28:04.450819  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.451958  5552 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:04.452425  5561 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
W20260709 12:28:04.396867  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:04.287732 13317 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.408942 13551 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:28:04.417774  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.421370  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.379575 13468 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:04.466846  5553 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260709 12:28:04.410048  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.014s	sys 0.009s
W20260709 12:28:04.536388  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.014s	sys 0.009s
W20260709 12:28:04.536746  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 639366us
I20260709 12:28:04.537045  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.537674  5553 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:04.538494  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.540853  5553 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 22: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:04.487063  5558 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:28:04.542155  5553 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 25 index: 12. (index mismatch)
I20260709 12:28:04.543754  5553 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:28:04.428512 13755 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:04.434195 13395 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.746s	user 0.006s	sys 0.019s
I20260709 12:28:04.510475  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.548089  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:04.551796  5703 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:04.549214  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.417259 13370 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:04.381863 13508 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.087s
I20260709 12:28:04.402678 13627 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:04.571565 13521 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.008s	sys 0.023s
W20260709 12:28:04.587141 13521 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.008s	sys 0.024s
I20260709 12:28:04.473359 13667 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:04.420223 13708 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:04.579209  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.010s	sys 0.038s
I20260709 12:28:04.591681  5708 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
W20260709 12:28:04.593950  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:04.534996 13551 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:04.616473  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.011s	sys 0.038s
W20260709 12:28:04.564847 13395 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.009s	sys 0.023s
I20260709 12:28:04.422405 13692 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:04.622052  5561 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
W20260709 12:28:04.610029  5707 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.024s	sys 0.027s
I20260709 12:28:04.629421  5561 pending_rounds.cc:85] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 11
I20260709 12:28:04.573269  5560 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
W20260709 12:28:04.617403  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 594261us
W20260709 12:28:04.629765  5707 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.025s	sys 0.028s
I20260709 12:28:04.630463  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:04.629820  5561 pending_rounds.cc:107] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 12:28:04.631474  5561 raft_consensus.cc:2889] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:28:04.580250 13463 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.150s	user 0.014s	sys 0.002s
W20260709 12:28:04.619311  5706 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.027s	sys 0.024s
I20260709 12:28:04.625744  5557 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:04.625669  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.642300  5558 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
W20260709 12:28:04.618053  5704 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.000s	sys 0.050s
W20260709 12:28:04.643383  5704 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.000s	sys 0.051s
W20260709 12:28:04.644490  5706 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.028s	sys 0.024s
I20260709 12:28:04.644750  5555 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:28:04.646534  5710 raft_consensus.cc:2410] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
W20260709 12:28:04.646641  5676 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.020s	sys 0.032s
I20260709 12:28:04.618402 13768 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:04.622599  5701 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.014s	sys 0.017s
W20260709 12:28:04.652815  5701 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.014s	sys 0.017s
I20260709 12:28:04.652519  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:04.648582  5676 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.020s	sys 0.033s
W20260709 12:28:04.654590  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 579400us
W20260709 12:28:04.649163  5527 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.018s	sys 0.017s
I20260709 12:28:04.657219  5556 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:28:04.658111  5527 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.019s	sys 0.017s
I20260709 12:28:04.661150  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.619597 13772 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:04.643654 13505 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:28:04.675242  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.016s	sys 0.017s
W20260709 12:28:04.675942  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.016s	sys 0.017s
I20260709 12:28:04.665040  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:04.648940 13506 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:04.675282  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 432317us
W20260709 12:28:04.663971 13629 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.002s	sys 0.028s
I20260709 12:28:04.677764  6277 raft_consensus.cc:2410] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:28:04.620548 13667 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:04.688752  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 565000us
I20260709 12:28:04.699801  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:04.660300 13743 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.003s	sys 0.033s
I20260709 12:28:04.673018 13617 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:28:04.705861 13743 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.003s	sys 0.035s
I20260709 12:28:04.698017  5703 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
W20260709 12:28:04.707985  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.015s	sys 0.025s
W20260709 12:28:04.708093  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:04.708384  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.015s	sys 0.025s
W20260709 12:28:04.708755  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 634795us
W20260709 12:28:04.710160  5710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3361) took 2584 ms (client timeout 1918 ms). Trace:
W20260709 12:28:04.697275 13629 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.003s	sys 0.029s
I20260709 12:28:04.714017  5627 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:04.720758  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:04.710662  5710 rpcz_store.cc:269] 0709 12:28:02.114597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:02.115591 (+   994us) service_pool.cc:224] Handling call
0709 12:28:04.698734 (+2583143us) spinlock_profiling.cc:243] Waited 2.52 s on lock 0x7b540006f754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:04.698886 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5792464768}
I20260709 12:28:04.705976 13617 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:04.751587 13612 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:04.704103  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:04.691715 13779 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:04.753407  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:04.763305  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.010s	sys 0.028s
W20260709 12:28:04.763891  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 59709.8us
I20260709 12:28:04.664394 13768 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:04.766870  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.010s	sys 0.028s
I20260709 12:28:04.768719  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.706461 13505 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:04.771703  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 226243us
W20260709 12:28:04.771718  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 533891us
W20260709 12:28:04.777163  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:04.692389 13778 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.083s
I20260709 12:28:04.710569 13709 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:04.752651 13725 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:04.783679  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:04.710861 13517 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:04.785378  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:04.676076 13772 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.756954 13617 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:04.804677  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.800868  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.805503  5558 raft_consensus.cc:2376] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21. Re-sending same reply.
W20260709 12:28:04.788128  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.008s	sys 0.015s
W20260709 12:28:04.810855  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.009s	sys 0.015s
I20260709 12:28:04.710016 13395 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.069s
I20260709 12:28:04.785053  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:04.799820  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:04.812248  5561 raft_consensus.cc:2376] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22. Re-sending same reply.
I20260709 12:28:04.813794  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.815582  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:04.789435 13322 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.012s	sys 0.012s
I20260709 12:28:04.767444 13612 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.835150 13612 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:04.836488 13782 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:04.668923 13755 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.802006  5553 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:28:04.837518 13782 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.838330 13779 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:28:04.820552  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:04.839910  5558 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:04.826435  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:04.760931 13743 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.031s
I20260709 12:28:04.700677 13551 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.836776 13508 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:04.697031 13506 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:04.845438  6234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.024s	sys 0.037s
I20260709 12:28:04.795606 13317 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:04.826862 13774 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.361s	user 0.000s	sys 0.001s
W20260709 12:28:04.844167 13322 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.012s	sys 0.012s
I20260709 12:28:04.818327 13788 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:04.857564  6234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.025s	sys 0.037s
I20260709 12:28:04.796489 13725 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:04.792220 13709 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:04.876919  5702 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 13. (term mismatch)
I20260709 12:28:04.886109  5702 pending_rounds.cc:85] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 12
I20260709 12:28:04.886379  5702 pending_rounds.cc:107] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 18.13
I20260709 12:28:04.886634  5702 raft_consensus.cc:2889] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:04.791887 13697 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 12:28:04.818502 13769 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:04.882076  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:04.882311  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:04.882859  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 654993us
I20260709 12:28:04.846282 13724 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:04.891443  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:04.845494 13627 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:04.898415  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:04.915599  5629 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
W20260709 12:28:04.898928 13776 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.137s	user 0.000s	sys 0.000s
I20260709 12:28:04.901911  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:04.928398  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:04.893697 13624 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.824s	user 0.010s	sys 0.003s
I20260709 12:28:04.883827 13370 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:04.887843 13692 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:04.894661 13782 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:04.905831 13322 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:04.908458 13769 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:04.994076  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52790.2us
I20260709 12:28:04.929422 13183 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:04.941283 13317 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:04.945299 13551 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:05.013963  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2945) took 2885 ms. Trace:
I20260709 12:28:05.031997  5559 rpcz_store.cc:276] 0709 12:28:02.128684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:02.134024 (+  5340us) service_pool.cc:224] Handling call
0709 12:28:02.134433 (+   409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:02.226850 (+ 92417us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:02.226918 (+    68us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:03.605721 (+1378803us) spinlock_profiling.cc:243] Waited 6.58 ms on lock 0x7b2c000fb608. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:28:03.659331 (+ 53610us) log.cc:844] Serialized 26 byte log entry
0709 12:28:04.093933 (+434602us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:28:04.093991 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:04.094179 (+   188us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:04.995514 (+901335us) raft_consensus.cc:1685] finished
0709 12:28:04.995549 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:05.013894 (+ 18345us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":121650,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":8660,"spinlock_wait_cycles":15125760,"thread_start_us":493339,"threads_started":2,"wal-append.queue_time_us":742733}
I20260709 12:28:05.011060 13770 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
I20260709 12:28:05.010620 13322 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.011206 13643 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:28:05.047078 13643 consensus_queue.cc:1243] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 20.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:28:04.977996 13624 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.051201 13624 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.058800  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:04.945887 13789 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:04.946168 13687 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:05.011219 13800 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 12:28:05.055331 13322 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:05.010708 13769 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:05.066977  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:05.011682  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 346738us
I20260709 12:28:04.884215 13505 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.944505 13724 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:04.895254 13788 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.013788 13782 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.071946 13687 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:05.083772 13612 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:05.067267 13765 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:28:05.066996 13667 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:05.093547 13719 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.996s	user 0.013s	sys 0.000s
W20260709 12:28:05.120703  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 53843.7us
I20260709 12:28:05.123737  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:05.092095 13770 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.091245 13317 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.148054  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:05.159135  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 263118us
I20260709 12:28:05.161427  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:05.164608  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:05.106305 13782 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:05.167706  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44075.3us
I20260709 12:28:05.169296 13782 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:05.169395  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:05.170132 13782 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:05.170661 13782 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:05.171132  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:05.171337 13782 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:05.174196 13719 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:05.175230 13719 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:28:05.176898 13322 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:05.177678  5634 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 18: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:05.178184  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:05.178776  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:05.130492 13743 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:05.181892  5631 raft_consensus.cc:2368] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
I20260709 12:28:05.183207  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:05.188045  5629 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
W20260709 12:28:05.188493  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:05.187336  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:05.130695 13524 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.160567 13806 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:28:05.098317 13797 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.092s
I20260709 12:28:05.136444 13768 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:05.200032  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:05.201771  6272 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:05.202633  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:05.184872 13322 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:05.132179 13612 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.182902 13794 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:05.185842 13725 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:05.186003 13743 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:05.178160 13508 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.025s
I20260709 12:28:05.207448  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3382) took 1724 ms. Trace:
I20260709 12:28:05.166153 13317 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.192265 13772 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.208552 13709 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:05.229733 13743 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.197024 13524 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.233179 13813 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.233983 13813 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.238962 13826 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:28:05.239336 13627 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:28:05.207327 13322 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:05.242066  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3377) took 2666 ms. Trace:
I20260709 12:28:05.242467  5702 rpcz_store.cc:276] 0709 12:28:02.575361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:03.464077 (+888716us) service_pool.cc:224] Handling call
0709 12:28:04.698882 (+1234805us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:04.887083 (+188201us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:05.241047 (+353964us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b540006f754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:05.241873 (+   826us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540006f750. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:05.242008 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3123438336}
I20260709 12:28:05.249114 13819 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:05.205148 13784 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.417s	user 0.002s	sys 0.000s
I20260709 12:28:05.211128  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:05.206984 13742 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.933s	user 0.008s	sys 0.007s
W20260709 12:28:05.221354 13792 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
W20260709 12:28:05.222105 13805 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
W20260709 12:28:05.222777 13799 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
W20260709 12:28:05.224248 13802 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:28:05.233404 13524 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.194597 13157 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:05.214991  5701 rpcz_store.cc:276] 0709 12:28:03.482733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:03.780208 (+297475us) service_pool.cc:224] Handling call
0709 12:28:03.780616 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:03.992229 (+211613us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:28:04.655128 (+662899us) log.cc:844] Serialized 14 byte log entry
0709 12:28:05.013863 (+358735us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:05.013918 (+    55us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:05.013964 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:05.207182 (+193218us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b5400066654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:05.207244 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:05.207390 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1944,"prepare.queue_time_us":32540,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":1505,"spinlock_wait_cycles":486161280,"thread_start_us":1003256,"threads_started":2,"wal-append.queue_time_us":410608}
I20260709 12:28:05.254845 13506 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.236013 13708 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:05.136396 13770 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.254137 13709 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:28:05.256852  5635 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260709 12:28:05.257570  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:05.257768  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:05.244688 13795 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:28:05.259284  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 485852us
I20260709 12:28:05.259398  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:05.260187  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 433823us
W20260709 12:28:05.262547  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 260273us
I20260709 12:28:05.293161  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:05.298712  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:05.303910  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 111040us
W20260709 12:28:05.305241  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:05.338317  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3383) took 1855 ms. Trace:
I20260709 12:28:05.308431  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:05.312582  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:05.338162  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:05.326748  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 139484us
I20260709 12:28:05.341814  5630 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:28:05.339385  5706 rpcz_store.cc:276] 0709 12:28:03.483005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:03.976829 (+493824us) service_pool.cc:224] Handling call
0709 12:28:03.979325 (+  2496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:03.979544 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:28:04.644972 (+665428us) log.cc:844] Serialized 14 byte log entry
0709 12:28:05.337380 (+692408us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:05.337434 (+    54us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:28:05.337484 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:05.337706 (+   222us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:05.338229 (+   523us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":34777,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":100175,"raft.run_cpu_time_us":61,"raft.run_wall_time_us":61,"thread_start_us":1352149,"threads_started":3,"wal-append.queue_time_us":562901}
I20260709 12:28:05.351032  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:05.357800  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:05.333879 13506 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:05.340090 13463 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:05.368762  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:05.301735 13742 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.377000 13203 raft_consensus.cc:993] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:28:05.378749 13521 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:28:05.379227 13521 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:05.380136 13612 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:05.380919 13803 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 12:28:05.382748 13755 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:05.379695 13687 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.383677 13812 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:05.384168 13812 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:05.384615 13812 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:05.385421 13687 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:05.388504 13766 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.969s	user 0.014s	sys 0.000s
I20260709 12:28:05.389043 13766 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:05.361734 13813 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:05.340512 13817 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:05.382789 13742 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.391734  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:05.392568 13813 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:05.393191 13810 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
W20260709 12:28:05.390395 13468 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.839s	user 0.008s	sys 0.004s
I20260709 12:28:05.394591 13468 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.365185 13767 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:05.396232 13767 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:05.366039 13506 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:05.396631 13767 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:05.397464 13506 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.398989 13506 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:05.301963 13819 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.401568 13468 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:05.380184 13521 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.402936 13819 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.404309 13788 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.405566 13524 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:28:05.370131 13724 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.346585 13697 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.028s
W20260709 12:28:05.410372 13617 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.587s	user 0.009s	sys 0.004s
I20260709 12:28:05.410315 13821 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:05.411029 13782 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.413867 13612 raft_consensus.cc:1081] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Signalling peer cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:28:05.365476 13505 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
I20260709 12:28:05.443071 13505 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:05.365577 13157 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.061s
I20260709 12:28:05.390723 13532 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.426678 13834 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:05.444975 13834 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:05.444842 13709 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:28:05.445469 13834 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:05.429217 13743 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:05.429451 13770 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:05.405908 13783 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.521s	user 0.003s	sys 0.001s
I20260709 12:28:05.416054 13826 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:05.447539 13826 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:05.447947 13826 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:05.443622 13505 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:05.449954 13505 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.451148 13505 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:05.451684 13317 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:05.437973 13697 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:05.437626 13617 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.452567 13697 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:05.453707 13813 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:05.453433 13697 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.454136 13617 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.435585 13794 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:05.439101 13821 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:05.456650 13769 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.443782 13322 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.460336 13769 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.461128 13322 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.461603 13697 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:05.465735 13521 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:05.461874 13767 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.469852 13782 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:05.457098 13821 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.473835 13826 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.475476 13821 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:05.436326 13812 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.478243 13812 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:05.456604 13834 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.483346 13826 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.487829 13834 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.489113 13767 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:05.343873  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:05.533054  5554 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 18. Current term is 19. Ops: []
I20260709 12:28:05.534521  5631 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
W20260709 12:28:05.644652  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 300782us
I20260709 12:28:05.535647  5561 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:28:05.649515  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 470918us
I20260709 12:28:05.405218  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:05.545732  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:05.665688  5560 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 25
W20260709 12:28:05.666080  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 260770us
I20260709 12:28:05.547868  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1431) took 1857 ms. Trace:
I20260709 12:28:05.668865 13834 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:05.550504  5555 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:28:05.669752 13834 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:05.670454 13834 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:05.552192  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:05.556434  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:05.672325  6277 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:05.558575  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:05.673921 13322 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:05.674512 13322 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:05.675024 13322 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:05.559731  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:05.677261  6279 raft_consensus.cc:2410] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 25 because replica has last-logged OpId of term: 24 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
I20260709 12:28:05.560177  6272 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:28:05.677809 13370 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:05.678720 13370 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:28:05.563184  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 185558us
I20260709 12:28:05.680404  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:05.681311 13468 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.576689  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:05.682348 13468 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.581768  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:05.585657  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:05.684810  6275 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:05.561141  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:05.686059 13826 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:05.687911  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3228) took 1313 ms. Trace:
I20260709 12:28:05.686164  6274 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:28:05.688498  5554 rpcz_store.cc:276] 0709 12:28:04.374967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:04.376173 (+  1206us) service_pool.cc:224] Handling call
0709 12:28:04.376535 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:05.644646 (+1268111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:05.687496 (+ 42850us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b5400026cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:05.687824 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2658238336}
I20260709 12:28:05.579770  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:05.683116  6278 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:05.693922 13743 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:05.668536  5707 rpcz_store.cc:276] 0709 12:28:03.690573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:04.010514 (+319941us) service_pool.cc:224] Handling call
0709 12:28:04.010977 (+   463us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:04.011454 (+   477us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:04.011518 (+    64us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:04.018492 (+  6974us) log.cc:844] Serialized 52 byte log entry
0709 12:28:04.630558 (+612066us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:04.664904 (+ 34346us) log.cc:844] Serialized 14 byte log entry
0709 12:28:04.665172 (+   268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:04.665246 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:05.405246 (+740000us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:05.546838 (+141592us) raft_consensus.cc:1685] finished
0709 12:28:05.547040 (+   202us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:05.547501 (+   461us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":520,"prepare.queue_time_us":131317,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":4734,"raft.queue_time_us":6565,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":10199,"spinlock_wait_cycles":43476352,"thread_start_us":582097,"threads_started":1,"wal-append.queue_time_us":787357}
I20260709 12:28:05.695004 13743 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:05.696244  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 144730us
I20260709 12:28:05.696565 13741 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:05.686195  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 2976) took 1112 ms. Trace:
I20260709 12:28:05.667754  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:05.697913  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:05.696691 13837 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:28:05.716434  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:05.723176 13532 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.724172 13532 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.727294 13532 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:05.727923 13532 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.729415 13524 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:05.730409 13524 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.730985  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:05.733667 13697 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:05.735337 13612 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:05.734540 13475 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:28:05.735989  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:05.736683 13847 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.737709  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:05.738143 13819 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:05.739130 13819 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.741178 13843 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:05.740876 13817 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:05.741695 13843 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:05.742172 13843 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:05.742447 13817 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:28:05.742614  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 400036us
I20260709 12:28:05.743120 13817 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:05.743570 13817 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:05.744652 13742 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:05.745225 13742 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:05.746650 13742 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:05.744668  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:05.750933  6275 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:28:05.737741 13847 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.754492 13847 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:05.754585 13837 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:28:05.755142  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57479.6us
I20260709 12:28:05.756737  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:05.755187 13837 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:05.758203 13837 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:05.760661  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:05.698661  5552 rpcz_store.cc:276] 0709 12:28:04.574144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:04.574334 (+   190us) service_pool.cc:224] Handling call
0709 12:28:04.680496 (+106162us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:04.680994 (+   498us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:28:05.019385 (+338391us) log.cc:844] Serialized 14 byte log entry
0709 12:28:05.164948 (+145563us) log.cc:844] Serialized 14 byte log entry
0709 12:28:05.645620 (+480672us) spinlock_profiling.cc:243] Waited 1.29 ms on lock 0x7b180016e4c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:28:05.646172 (+   552us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:05.646330 (+   158us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:05.646526 (+   196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:05.646693 (+   167us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:05.685722 (+ 39029us) spinlock_profiling.cc:243] Waited 63.9 ms on lock 0x7b54001bdd50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:05.685914 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14315,"prepare.queue_time_us":110781,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":2640,"raft.queue_time_us":511592,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":252,"spinlock_wait_cycles":149919232,"thread_start_us":462536,"threads_started":2,"wal-append.queue_time_us":217274}
I20260709 12:28:05.763639  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:05.764108  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:05.765000  5558 raft_consensus.cc:2410] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 25 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
W20260709 12:28:05.766407  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 82937.2us
I20260709 12:28:05.767675  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:05.760136 13843 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:05.804775  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 50620.6us
I20260709 12:28:05.765324  5559 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:05.770123  5560 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:05.762909  6274 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:28:05.760207  6271 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:05.773509  5452 catalog_manager.cc:5697] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 16 to 17, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 17 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:05.692840  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:05.777160 13468 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:05.778617 13743 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.780078 13697 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:28:05.786273  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 118851us
I20260709 12:28:05.799923  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:05.805562  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 105412us
W20260709 12:28:05.804975  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51178.7us
W20260709 12:28:05.784240  5459 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 279) took 33109 ms (client timeout 15000 ms). Trace:
I20260709 12:28:05.785204 13506 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:28:05.866950  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:05.787678 13742 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:05.805992 13816 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:05.798257 13819 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:05.879776  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:05.883594  5552 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:28:05.883507  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 116094us
I20260709 12:28:05.805378 13820 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:05.872831 13816 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:05.874446 13532 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.894816 13771 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:05.895532 13771 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:05.805202 13627 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.082s
I20260709 12:28:05.871833 13370 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.804878 13795 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.087s
I20260709 12:28:05.914305  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:05.895555 13532 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:05.916216 13821 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:28:05.916949 13767 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.917502 13767 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.920029 13767 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:05.907195  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:05.921775  5452 catalog_manager.cc:5697] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 16 to 17, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 17 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:05.925228 13773 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:05.929275  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 172757us
I20260709 12:28:05.890004 13506 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:05.919248 13771 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.939785 13820 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:05.901479 13816 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:05.891378 13612 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:05.990417 13612 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:05.907401  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:05.991343  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 122297us
I20260709 12:28:05.993541 13627 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:05.993171 13506 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.994211 13627 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:05.995335 13627 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:05.942117 13601 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:05.996759 13524 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:05.997010 13601 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:05.997448 13524 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:28:05.997478 13821 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:05.998772 13821 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:05.999271  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:06.000022  5707 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:28:05.999841 13821 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.998090 13601 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:06.002406  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:06.002962 13601 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:06.003659 13322 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:05.908414  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1577) took 1097 ms. Trace:
I20260709 12:28:05.947649  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:06.019604  5555 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:28:05.948678  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:06.018921  5557 rpcz_store.cc:276] 0709 12:28:04.793935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:04.794889 (+   954us) service_pool.cc:224] Handling call
0709 12:28:04.795318 (+   429us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:05.112586 (+317268us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:05.112655 (+    69us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:05.125833 (+ 13178us) log.cc:844] Serialized 52 byte log entry
0709 12:28:05.673932 (+548099us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:05.675121 (+  1189us) log.cc:844] Serialized 14 byte log entry
0709 12:28:05.675415 (+   294us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:05.675544 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:05.884611 (+209067us) raft_consensus.cc:1685] finished
0709 12:28:05.884637 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:05.891173 (+  6536us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13695,"prepare.queue_time_us":402,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":208045,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":12266,"thread_start_us":797201,"threads_started":2,"wal-append.queue_time_us":741062}
W20260709 12:28:06.021081  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 72725.9us
I20260709 12:28:06.022234 13475 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:06.022707 13475 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:06.023078 13475 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:05.962796  5452 catalog_manager.cc:5697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 19 to 20, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 20 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:06.024883  5452 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 7 to 14, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 14 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:06.004863  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:06.006500  5704 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 12:28:06.031574  5704 pending_rounds.cc:85] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 11
I20260709 12:28:06.031967  5704 pending_rounds.cc:107] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 20.12
I20260709 12:28:06.032279  5704 raft_consensus.cc:2889] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:06.008008  5707 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:28:06.008337  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.008481 13773 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:05.997128  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 90615us
I20260709 12:28:05.944481  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:06.001782  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:06.044049  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 178289us
I20260709 12:28:06.046181  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:06.048623  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 79051.7us
W20260709 12:28:06.052376  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:06.053990  5675 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:28:06.055258  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 176901us
I20260709 12:28:06.058362 13505 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:06.059723  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:06.060437  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:06.066071  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.069815  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:06.065120  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:06.073061  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:06.074666  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:06.075209  5704 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:06.075824 13521 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:06.075598 13817 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:06.076587 13521 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:06.079033 13521 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:06.079675 13820 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:06.080783  5706 raft_consensus.cc:2368] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23.
I20260709 12:28:06.077782  5452 catalog_manager.cc:5697] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 13 to 14, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 14 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:06.082145 13820 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:06.082309  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:06.084673  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:06.084838  5452 catalog_manager.cc:5697] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 9 to 18, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 18 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:06.087518  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:06.088598 13814 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:06.089686 13814 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:06.090088 13814 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:06.090448 13814 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:06.017886 13834 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:06.088497  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:06.092326 13812 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:28:06.093403 13834 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:06.094264 13771 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:06.088559 13505 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:06.096949 13505 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:06.097489 13505 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:06.091167  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:06.091123 13773 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:06.065120  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:06.080179 13819 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:06.105954  5703 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: [25.12-25.12]
I20260709 12:28:06.109719  6271 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:28:06.051996 13826 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:06.112460  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:06.115157  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:06.119159  5702 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
W20260709 12:28:06.123565  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 63299.5us
I20260709 12:28:06.129473  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.068207 13601 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:06.128235  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:06.113566 13820 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:28:06.124035  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:06.140394  5710 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:28:06.113984 13819 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:06.055545 13837 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:06.115480 13524 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:06.125034 13812 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:06.139983  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.126227 13826 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:06.180619  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:06.185549  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48334.8us
I20260709 12:28:06.185832  5636 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:28:06.186563  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:06.187136  5708 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:28:06.186849 13524 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:06.185451  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.188668  5709 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:06.188880 13826 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:06.199649  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:06.199898  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.201498  5709 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:06.201647 13812 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:06.200747  5634 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:06.202641 13834 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:28:06.203990  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:06.204797  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:06.205845  6271 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 18: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:06.202203 13812 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:06.206562 13812 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:06.204674  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:05.864624  5459 rpcz_store.cc:269] 0709 12:27:32.674744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:45.315225 (+12640481us) service_pool.cc:224] Handling call
0709 12:28:05.783871 (+20468646us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:27:57.860632 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:27:57.861357 (+   725us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:27:57.861408 (+    51us) write_op.cc:435] Acquired schema lock
0709 12:27:57.861425 (+    17us) tablet.cc:662] Decoding operations
0709 12:27:57.863248 (+  1823us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:27:57.863360 (+   112us) write_op.cc:641] Partition lock acquired for write op
0709 12:27:57.863386 (+    26us) tablet.cc:685] Acquiring locks for 24 operations
0709 12:27:58.073098 (+209712us) tablet.cc:701] Row locks acquired
0709 12:27:58.073152 (+    54us) write_op.cc:260] PREPARE: finished
0709 12:27:58.073363 (+   211us) write_op.cc:270] Start()
0709 12:27:58.073523 (+   160us) write_op.cc:276] Timestamp: P: 1783600078073297 usec, L: 0
0709 12:27:58.073563 (+    40us) op_driver.cc:348] REPLICATION: starting
0709 12:27:58.162410 (+ 88847us) log.cc:844] Serialized 9186 byte log entry
0709 12:27:58.437090 (+274680us) op_driver.cc:464] REPLICATION: finished
0709 12:27:58.712302 (+275212us) write_op.cc:301] APPLY: starting
0709 12:27:58.712453 (+   151us) tablet.cc:1367] starting BulkCheckPresence
0709 12:27:59.948099 (+1235646us) spinlock_profiling.cc:243] Waited 6.58 ms on lock 0x7b180001a2c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa3b10ec 00007f4ffa3b3af7 00007f4ffa3b4fb0 00007f4ffe0c91bb 00007f4ffe0c9532 00007f4ffe110b93 00007f4ffdfbdde3 00007f4ffdfbd6ab 00007f4ffdfbee77
0709 12:28:00.514024 (+565925us) tablet.cc:1370] finished BulkCheckPresence
0709 12:28:00.520276 (+  6252us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:28:01.551019 (+1030743us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:28:01.551991 (+   972us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=48,key_file_lookups=48,delta_file_lookups=48,mrs_lookups=0
0709 12:28:01.552160 (+   169us) write_op.cc:312] APPLY: finished
0709 12:28:01.558502 (+  6342us) log.cc:844] Serialized 212 byte log entry
0709 12:28:01.881345 (+322843us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:28:02.001190 (+119845us) write_op.cc:454] Released schema lock
0709 12:28:02.010977 (+  9787us) write_op.cc:341] FINISH: Updating metrics
0709 12:28:03.069739 (+1058762us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b180001a2c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:28:03.088409 (+ 18670us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:28:03.089111 (+   702us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":391314,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":241243,"apply.run_cpu_time_us":93465,"apply.run_wall_time_us":3317806,"cfile_cache_hit":98,"cfile_cache_hit_bytes":104447,"num_ops":24,"prepare.queue_time_us":144977,"prepare.run_cpu_time_us":19874,"prepare.run_wall_time_us":521667,"raft.queue_time_us":55095,"raft.run_cpu_time_us":3997,"raft.run_wall_time_us":136811,"replication_time_us":327682,"spinlock_wait_cycles":72347648,"thread_start_us":464818,"threads_started":5,"wal-append.queue_time_us":603150,"wal-append.run_cpu_time_us":6862,"wal-append.run_wall_time_us":2107341}]]}
I20260709 12:28:06.220964  6274 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:06.222126  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:06.221998  6278 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:06.142457 13843 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:06.217610  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 110112us
W20260709 12:28:06.260815  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:06.227563  5558 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:06.229547 13612 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:06.230233 13794 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:28:06.238524 13816 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:06.239039  6274 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:06.204530 13782 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:06.192675  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.188609 13601 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:06.242827  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.191679 13826 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:06.268712  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:06.246341  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.246909  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.247103  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:06.248570  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 127097us
I20260709 12:28:06.249117 13773 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:06.249065  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:06.253487 13817 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:06.253614  5632 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:28:06.254163 13475 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:06.254601 13322 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:06.282050  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:06.247700 13814 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:06.245766 13819 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:06.282282  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:06.282466  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:06.419127  5703 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:28:06.423669  6274 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:06.256331 13837 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:06.284543  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:06.285837  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:06.289494 13524 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:06.293709  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:06.433916  5627 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:06.268314 13782 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:06.416775  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:06.283236 13743 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:28:06.301180  5705 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 19: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:06.432446  5706 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:06.281941 13532 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:06.301434  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:06.275211  5704 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:28:06.272881 13773 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:06.459792  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 176608us
I20260709 12:28:06.420639  5628 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:06.446967  5557 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:06.448037  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.465034  5702 raft_consensus.cc:2410] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 20 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:28:06.442729  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:06.431408 13794 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:06.457830  5710 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:06.473879 13524 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:06.474895  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:06.474843 13524 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:06.476496  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:06.443321 13782 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:06.479969  5554 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:06.480628 13764 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:28:06.482640  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.483449 13814 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:06.484444  5706 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:06.488842  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:06.494242  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 56346.3us
W20260709 12:28:06.474017  5459 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 280) took 16375 ms (client timeout 15000 ms). Trace:
W20260709 12:28:06.495908  5459 rpcz_store.cc:269] 0709 12:27:50.098399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:06.473730 (+16375331us) service_pool.cc:211] Skipping call since client already timed out
0709 12:28:06.473926 (+   196us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:28:06.434831 13322 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:28:06.482134  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 303498us
I20260709 12:28:06.464016 13773 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:06.508965  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.509722  5703 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:06.510313  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:06.527179  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:06.530362  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68189.2us
I20260709 12:28:06.530948 13601 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:06.532518  5599 leader_election.cc:400] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23.
I20260709 12:28:06.489135 13532 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:06.534910 13532 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:06.534906  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:06.536762 13794 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:06.537808 13532 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:06.538512 13532 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:06.539045 13532 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:06.542181 13505 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:06.543223 13833 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:28:06.514242 13370 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:28:06.565775  5702 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:06.570919  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.544945  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:06.588819  5627 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
I20260709 12:28:06.544569 13627 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:06.515026 13612 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:06.591519  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:06.598222  5708 pending_rounds.cc:85] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 14
I20260709 12:28:06.613104  5708 pending_rounds.cc:107] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 12:28:06.613435  5708 raft_consensus.cc:2889] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:06.594684 13627 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:06.607224 13322 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:28:06.647769  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:06.651402 13764 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:06.652089 13764 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:28:06.658327 13859 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:28:06.662607 13773 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:06.666314 13532 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:06.668212 13524 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:06.672631  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 127769us
I20260709 12:28:06.673628  5526 leader_election.cc:400] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
I20260709 12:28:06.673911  5526 leader_election.cc:403] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:28:06.674646  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:06.675513 13771 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.581s	user 0.003s	sys 0.007s
I20260709 12:28:06.676050 13370 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:06.676470 13370 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:06.676846 13370 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:06.676939 13627 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 25.12, Last appended by leader: 12, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:06.678071 13627 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:06.675446 13814 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
I20260709 12:28:06.611270 13505 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:06.623031 13816 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:06.685252  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 392372us
I20260709 12:28:06.692287  5709 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: [25.12-25.12]
I20260709 12:28:06.615343 13612 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:06.699826  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.701014  5702 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:06.704999  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.710085  5706 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
W20260709 12:28:06.722932 13857 log.cc:927] Time spent T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.229s	user 0.002s	sys 0.002s
I20260709 12:28:06.724771  5705 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
W20260709 12:28:06.715518 13742 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.846s	user 0.012s	sys 0.002s
I20260709 12:28:06.755262  5635 raft_consensus.cc:2410] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 25 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:28:06.763227  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:06.748720 13834 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.028s	sys 0.009s
I20260709 12:28:06.737629 13475 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:06.782559  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 163900us
W20260709 12:28:06.764390 13834 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.029s	sys 0.009s
I20260709 12:28:06.789734  5709 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 18.
I20260709 12:28:06.789613 13767 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:06.787379 13532 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:06.794580 13506 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.009s	sys 0.041s
I20260709 12:28:06.788760 13505 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:28:06.790531 13773 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:06.828265  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.011s	sys 0.031s
W20260709 12:28:06.810423 13506 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.009s	sys 0.041s
W20260709 12:28:06.830649  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.011s	sys 0.031s
W20260709 12:28:06.821377 13821 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.804s	user 0.032s	sys 0.031s
I20260709 12:28:06.809504 13767 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:06.838155 13821 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.033s	sys 0.031s
W20260709 12:28:06.863019  5633 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.013s	sys 0.021s
W20260709 12:28:06.863507  5633 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.014s	sys 0.022s
W20260709 12:28:06.870784  5701 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.017s	sys 0.020s
W20260709 12:28:06.871587  5701 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.018s	sys 0.021s
I20260709 12:28:06.870872 13697 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:06.872467 13697 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:06.875367 13612 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:06.876021 13697 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:06.877566 13860 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:28:06.881721  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:06.882418  5704 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:06.876045 13782 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:06.885757 13858 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
I20260709 12:28:06.886142  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:06.886826  5561 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:06.887146 13782 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:06.888258  5561 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:28:06.888831  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:06.889525  5553 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:06.890492  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:06.891161  5556 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:28:06.892087  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:06.892427  5710 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:28:06.892727  5558 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:06.897604  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:06.898284 13812 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.691s	user 0.014s	sys 0.000s
I20260709 12:28:06.898518  5557 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:06.898432  5704 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:06.899569  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:06.900100  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:06.901748  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:06.902383  5552 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:06.905772 13767 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:06.906606  6235 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.023s	sys 0.022s
W20260709 12:28:06.906980  6235 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.024s	sys 0.023s
I20260709 12:28:06.912964  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:06.913928  5559 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 26: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:06.915010 13322 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:28:06.914899 13814 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:28:06.915871 13612 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:06.916497 13612 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:06.917186 13322 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:28:06.900440  5704 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:06.921746 13812 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:06.925050 13812 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:06.931572 13816 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:06.934646  5704 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:06.935400  5556 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:06.932116 13816 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:06.945556  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:06.947925  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:06.936436 13764 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:06.948520  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:06.954885  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:06.975371  5555 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:28:06.984102  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:06.975714  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:06.972298 13773 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.047s
I20260709 12:28:07.022289 13764 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:07.022535 13773 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:07.022927 13773 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:07.023746 13826 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:28:07.024245 13826 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:07.025213 13826 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:07.041522  5552 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:07.052981 13506 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:07.053650 13506 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:07.054176 13506 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:07.057735  5705 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
W20260709 12:28:07.062595  6234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.011s	sys 0.008s
W20260709 12:28:07.064114  6234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.011s	sys 0.008s
I20260709 12:28:07.067018  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:07.067756  6271 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:07.024035 13773 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:07.083070  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:07.085711  6278 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:07.084735  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 115351us
W20260709 12:28:07.093293 13853 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.606s	user 0.002s	sys 0.002s
W20260709 12:28:07.093888 13766 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.042s	sys 0.038s
W20260709 12:28:07.094230 13766 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.042s	sys 0.038s
I20260709 12:28:07.094944 13837 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:07.095342 13837 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:07.098287  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1658) took 1232 ms. Trace:
I20260709 12:28:07.100631 13861 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:07.101384 13861 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:07.028936 13816 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:07.093367  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 1657) took 1235 ms. Trace:
I20260709 12:28:07.110401  5558 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
W20260709 12:28:07.069450 13794 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.006s	sys 0.017s
I20260709 12:28:07.098785  5633 rpcz_store.cc:276] 0709 12:28:05.865444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:05.870955 (+  5511us) service_pool.cc:224] Handling call
0709 12:28:05.910922 (+ 39967us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:06.051892 (+140970us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:06.051929 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:06.141173 (+ 89244us) log.cc:844] Serialized 26 byte log entry
0709 12:28:06.863912 (+722739us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:06.863962 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:06.864045 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:07.097983 (+233938us) raft_consensus.cc:1685] finished
0709 12:28:07.098012 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:07.098196 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8060,"raft.queue_time_us":34136,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":102,"spinlock_wait_cycles":810880,"thread_start_us":631110,"threads_started":1,"wal-append.queue_time_us":796019}
I20260709 12:28:07.113845  5555 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:07.112536  5635 rpcz_store.cc:276] 0709 12:28:05.860934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:05.865622 (+  4688us) service_pool.cc:224] Handling call
0709 12:28:07.093122 (+1227500us) spinlock_profiling.cc:243] Waited 873 ms on lock 0x7b54001f2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:07.093282 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2007596928}
I20260709 12:28:07.137844  6271 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:07.140456  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:07.104151 13861 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:07.113680 13794 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.007s	sys 0.017s
W20260709 12:28:07.116540  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.021s	sys 0.038s
W20260709 12:28:07.148893  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.022s	sys 0.038s
I20260709 12:28:07.167778  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:07.168501  5554 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:07.170236 13866 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:07.170794 13866 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:07.096308 13837 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:07.173326 13866 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:07.173753 13837 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:07.173949  5636 raft_consensus.cc:1813] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Handling vote request from an unknown peer cea66aa15be3423880843974acdbf50f
I20260709 12:28:07.177696 13861 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:07.148955 13612 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:07.149382 13773 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:07.185235  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 79225.8us
I20260709 12:28:07.185999  6278 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:28:07.186086  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:07.187481  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:07.188071  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 966919us
W20260709 12:28:07.189738  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:07.194891  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:28:07.191721  6234 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.616s (SENT) [suppressed 12 similar messages]
I20260709 12:28:07.199411 13861 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:07.210625 13812 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:28:07.217211 13861 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:07.220280  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:07.230320  5552 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:07.230887  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:07.231455  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:07.237073  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:07.239300  5556 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:07.240617  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:07.242218  5602 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.015s	sys 0.024s
W20260709 12:28:07.242573  5602 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.015s	sys 0.024s
W20260709 12:28:07.243268  5707 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.016s	sys 0.027s
W20260709 12:28:07.247458  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 523323us
W20260709 12:28:07.247774  5707 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.016s	sys 0.027s
I20260709 12:28:07.247973  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:07.243948  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:07.222903 13861 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:07.253526  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:07.252663  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3229) took 1058 ms. Trace:
I20260709 12:28:07.242347 13627 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:07.261678  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:07.260742  5636 rpcz_store.cc:276] 0709 12:28:06.194820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:06.216182 (+ 21362us) service_pool.cc:224] Handling call
0709 12:28:07.250372 (+1034190us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b540018d154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:07.252599 (+  2227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2080233216}
I20260709 12:28:07.266491  5552 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:28:07.248138 13771 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.023s	sys 0.016s
I20260709 12:28:07.249372 13782 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:28:07.272763  5556 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:07.252838 13833 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:28:07.252629 13843 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:07.249190 13524 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 12:28:07.269587 13532 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:07.261543 13795 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:07.294296 13795 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:07.262553 13834 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:07.264765 13861 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:07.275616 13771 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.024s	sys 0.016s
I20260709 12:28:07.300817  5552 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
W20260709 12:28:07.304463  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60560.9us
I20260709 12:28:07.266265 13627 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:07.306903  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 504201us
I20260709 12:28:07.257947 13697 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 12:28:07.278614 13782 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:07.277884 13833 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:07.306936 13532 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:07.315043  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 446524us
I20260709 12:28:07.310734  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:07.297976 13601 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:07.313858 13370 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.633s	user 0.014s	sys 0.000s
I20260709 12:28:07.310315 13627 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:07.337986  5554 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:28:07.323393 13782 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:07.306070 13812 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.026s
I20260709 12:28:07.296550 13843 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:07.300722 13612 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 12:28:07.325398 13833 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:07.342757 13867 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:07.342947 13873 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:28:07.342787  5708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.021s	sys 0.025s
W20260709 12:28:07.387382  5708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.682s	user 0.021s	sys 0.025s
I20260709 12:28:07.343415 13870 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:07.369060 13532 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:07.372609  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:07.383184  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:07.384950  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:07.380932 13819 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.901s	user 0.008s	sys 0.041s
I20260709 12:28:07.372303 13524 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:07.333603 13601 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:07.412740  5600 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.030s	sys 0.013s
W20260709 12:28:07.414327  5600 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.030s	sys 0.013s
W20260709 12:28:07.415397  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 82730.8us
I20260709 12:28:07.417229 13812 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:07.424791 13877 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:28:07.425489 13877 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:07.425865  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:07.400241 13819 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.920s	user 0.008s	sys 0.041s
I20260709 12:28:07.333092 13370 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:07.386055 13867 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:07.428341  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3408) took 1306 ms. Trace:
I20260709 12:28:07.456907  5701 rpcz_store.cc:276] 0709 12:28:06.121766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:06.126270 (+  4504us) service_pool.cc:224] Handling call
0709 12:28:06.173482 (+ 47212us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:06.872189 (+698707us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:06.872221 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:06.872796 (+   575us) log.cc:844] Serialized 26 byte log entry
0709 12:28:07.296735 (+423939us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:07.296780 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:07.296866 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:07.428084 (+131218us) raft_consensus.cc:1685] finished
0709 12:28:07.428110 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:07.428274 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5018,"raft.queue_time_us":83105,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":123,"thread_start_us":1094186,"threads_started":2,"wal-append.queue_time_us":520524}
W20260709 12:28:07.460088  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 268375us
W20260709 12:28:07.461279  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:07.407811 13764 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:07.463421  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:07.464217  5635 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:28:07.426977 13524 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:07.487236  5559 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:28:07.491796  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:07.493292  5629 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:07.492168  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:07.495733  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:07.399866 13870 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:07.579116  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 83474.6us
W20260709 12:28:07.522303  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 191926us
I20260709 12:28:07.539072  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:07.663121  5633 raft_consensus.cc:2410] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 26 because replica has last-logged OpId of term: 25 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
I20260709 12:28:07.523442  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:07.594383  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:07.552340  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:07.498243  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:07.551273  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 135016us
I20260709 12:28:07.575904 13601 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:07.561800 13370 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:07.669471 13895 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:07.669767 13896 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.104s	user 0.003s	sys 0.001s
I20260709 12:28:07.563066 13524 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:07.670071 13895 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:07.613413 13782 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:07.564213 13843 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:07.673381 13816 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:07.673557 13898 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.095s	user 0.001s	sys 0.001s
I20260709 12:28:07.560529  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:07.674501  5630 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:07.591794 13743 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:07.679356 13627 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:07.552882 13877 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:07.676115 13743 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:07.681874 13627 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:07.664175  5632 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:07.482149 13866 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:07.600558  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 224930us
W20260709 12:28:07.564844 13885 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:28:07.687156  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:07.537119 13506 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:07.718746 13506 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:07.609620  5635 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
W20260709 12:28:07.707624  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:07.703717  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:07.722697 13601 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:07.487165 13795 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:07.536952 13812 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:07.714663  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:07.521286 13697 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:07.726151  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:07.727988 13743 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:07.737475  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 181353us
I20260709 12:28:07.739380  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:07.730719  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1445) took 1486 ms. Trace:
I20260709 12:28:07.486824 13764 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:07.723459 13601 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:07.611339 13826 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:28:07.745743  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:07.754138  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:07.577122 13900 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:07.739081 13505 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:07.746945  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:07.758188  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:07.560283 13897 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.164s
I20260709 12:28:07.752453  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:07.740113  5708 rpcz_store.cc:276] 0709 12:28:06.244318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:06.494006 (+249688us) service_pool.cc:224] Handling call
0709 12:28:06.545723 (+ 51717us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:07.392677 (+846954us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:07.392717 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:07.432609 (+ 39892us) log.cc:844] Serialized 52 byte log entry
0709 12:28:07.684914 (+252305us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:07.684972 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:07.685142 (+   170us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:07.730395 (+ 45253us) raft_consensus.cc:1685] finished
0709 12:28:07.730427 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:07.730636 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":159154,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":5936,"thread_start_us":881351,"threads_started":2,"wal-append.queue_time_us":262734}
I20260709 12:28:07.758936  5556 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:07.758833  5701 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:07.605338 13873 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:07.765995  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:07.766496  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:07.767184  5560 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:28:07.704779 13322 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:07.724234 13871 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:07.729566 13834 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:07.771194  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:07.771584  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:07.729887 13872 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:07.616744 13861 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:07.822229  5559 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:07.822230  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:07.753316 13767 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:28:07.826077  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 95848.5us
I20260709 12:28:07.712277 13866 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:07.828126  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:07.757206 13895 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:07.822410  5702 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:28:07.774930  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:07.823952  5704 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:07.785635  5555 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:07.823035  5558 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:07.828676  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:07.859462  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:07.772886  5553 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:07.860297  5708 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:07.763180 13505 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:07.861860  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 89864.7us
I20260709 12:28:07.862102  5709 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:28:07.863523 13900 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:07.869920 13764 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:07.850894  5676 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:07.776611 13766 raft_consensus.cc:2764] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:28:07.829377 13834 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:07.878206  5602 leader_election.cc:336] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.698s (SENT)
I20260709 12:28:07.877421  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:07.949653  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:07.942023 13873 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:07.924595  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 171660us
W20260709 12:28:07.941249 13901 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
W20260709 12:28:07.949396 13833 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.541s	user 0.015s	sys 0.000s
I20260709 12:28:07.936457 13755 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:28:07.540362 13867 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:07.932633 13862 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:28:07.918582  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 179412us
I20260709 12:28:07.825383 13872 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:07.920624 13601 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:07.917217 13908 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:28:07.785979 13627 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.091s
I20260709 12:28:07.766716 13506 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.094s
I20260709 12:28:07.872593 13782 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:07.877748 13900 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:07.879472  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:07.830695 13370 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.055s
I20260709 12:28:08.000213  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3417) took 1744 ms. Trace:
I20260709 12:28:08.013211  5556 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
W20260709 12:28:08.039489  5524 connection.cc:256] RPC call timeout handler was delayed by -1.6328816019999977s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1835s for a response
I20260709 12:28:07.876329 13743 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:07.876014 13870 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:08.014446  5558 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:07.880999 13902 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:28:08.015388  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:07.851578 13322 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:08.093771 13912 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:28:08.019323  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:08.022176  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 169667us
I20260709 12:28:07.949978 13505 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.001706 13895 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:08.053754  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:08.073596  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 378248us
I20260709 12:28:08.076725  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:08.074836  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:08.080632  5524 connection.cc:256] RPC call timeout handler was delayed by -1.5329816019998361s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1946s for a response
I20260709 12:28:08.097543 13902 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:08.089991 13861 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:08.096752  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:08.046921 13872 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:08.097802  5701 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:28:08.046984 13755 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:08.021116 13524 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.061s
W20260709 12:28:08.101830  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 91788.3us
W20260709 12:28:08.087867 13885 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:28:08.080327  5703 rpcz_store.cc:276] 0709 12:28:06.256048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:06.529964 (+273916us) service_pool.cc:224] Handling call
0709 12:28:06.844183 (+314219us) spinlock_profiling.cc:243] Waited 9.95 ms on lock 0x7b5800053198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754deee 000000000036eb39 00007f4fff5c11e0 00007f4fff68a0c4 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:28:06.844495 (+   312us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:06.858457 (+ 13962us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:06.858489 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:06.898940 (+ 40451us) log.cc:844] Serialized 26 byte log entry
0709 12:28:07.325103 (+426163us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:07.523180 (+198077us) log.cc:844] Serialized 14 byte log entry
0709 12:28:07.753460 (+230280us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:07.753536 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:07.995281 (+241745us) raft_consensus.cc:1685] finished
0709 12:28:07.999961 (+  4680us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:08.000142 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":279709,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":67111,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":64749,"spinlock_wait_cycles":22900480,"thread_start_us":828684,"threads_started":3,"wal-append.queue_time_us":504154}
I20260709 12:28:08.023216 13833 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.094625 13322 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:08.071190 13812 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.030s
W20260709 12:28:08.120180  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 104869us
I20260709 12:28:08.085862 13900 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:08.079627 13521 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:08.098718 13867 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:08.158244 13521 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.125239  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:08.161271  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:08.125432 13834 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.161818  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:08.163468 13834 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:08.126247 13877 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:08.164165 13877 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:08.164585 13877 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:08.126868  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:08.168581 13521 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:08.168869  5705 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:08.169191 13862 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:08.169867 13862 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:08.131101 13866 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:08.123039  5524 connection.cc:256] RPC call timeout handler was delayed by -1.4807816019997517s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20040000000000002s for a response
I20260709 12:28:08.132607  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:08.170884  5524 connection.cc:256] RPC call timeout handler was delayed by -1.4015816020000784s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20920000000000002s for a response
I20260709 12:28:08.171123  5701 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 18.
W20260709 12:28:08.171267  5524 connection.cc:256] RPC call timeout handler was delayed by -1.3268816019999576s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2175s for a response
W20260709 12:28:08.171660  5524 connection.cc:256] RPC call timeout handler was delayed by -1.0261980340001173s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1744s for a response
I20260709 12:28:08.172065  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:08.173302  5556 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
W20260709 12:28:08.124125  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49372.3us
W20260709 12:28:08.173877 13914 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:28:08.174392  5553 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:08.174815  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:08.102432 13895 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:08.175272  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:08.175539 13895 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:08.117275 13814 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:08.176191 13814 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:08.177163 13814 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.122005 13819 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:28:08.156006 13899 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.054s
I20260709 12:28:08.169734  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:08.173445  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:08.189000  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 316560us
I20260709 12:28:08.087334 13743 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.133219 13902 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:28:08.329978  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 154804us
I20260709 12:28:08.123445 13612 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:08.289494  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:08.147634  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:08.323546  5705 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:08.337549  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:08.339144  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.340232  5553 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:08.324509  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:08.135396 13909 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:28:08.330742  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:08.289652  5701 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:08.326272  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:08.149499 13773 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:08.161902  5709 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:08.194232 13755 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.150703 13764 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:08.339370  5554 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:08.346249  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:08.329607  5559 raft_consensus.cc:2376] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22. Re-sending same reply.
I20260709 12:28:08.325215 13872 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.341617  5703 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
W20260709 12:28:08.326339  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:08.363054  5560 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:28:08.364369  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:08.374806  5701 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:08.366204  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:08.375727  6271 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:08.364952  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:08.475636  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:08.331295 13834 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:28:08.393939  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 64816.8us
I20260709 12:28:08.398988  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:08.339862 13612 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:08.396435  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:08.405387  5559 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: [25.12-25.12]
W20260709 12:28:08.492000  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:08.427781  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:28:08.472033  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 375360us
I20260709 12:28:08.508980  5554 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:08.540385  5555 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 12:28:08.529943  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:08.357395 13782 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:28:08.426023  5628 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:08.514611  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:08.367376 13767 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.026s	sys 0.030s
I20260709 12:28:08.511883  5558 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 26
W20260709 12:28:08.519191  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 91562.1us
I20260709 12:28:08.543448  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:08.356915 13909 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:08.388581 13764 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.475296 13866 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:08.368209 13861 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:08.400966 13875 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:08.376084 13877 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.511283 13834 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:08.512919 13612 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:08.396485 13870 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:08.401101 13872 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:08.584627 13870 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.584627  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:08.585356  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:08.586453  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:08.587139 13870 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:08.578562  5560 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:49044
I20260709 12:28:08.588503  5560 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:08.588837  5560 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:08.589671 13826 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.818s	user 0.007s	sys 0.007s
I20260709 12:28:08.590076  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:08.591463 13867 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:08.592015 13867 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.593333  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:08.594164  5634 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:08.594468 13867 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:08.597093  5552 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:28:08.598829  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.580691 13895 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.600840  5561 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:08.602433  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.602674  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:08.553608 13866 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.605365 13866 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:08.606035  5552 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:08.613005  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:08.508116 13905 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.388s	user 0.001s	sys 0.009s
I20260709 12:28:08.529778  5559 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:28:08.614642  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.615757  5554 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:28:08.531654  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:08.543394 13767 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:08.616444  5627 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:08.617421 13767 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.530344  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:08.538767  5556 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:08.556818 13861 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.621423  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.622782 13861 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:08.515192  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:08.526612 13917 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:08.623744 13917 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:08.624018  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 108922us
I20260709 12:28:08.625162  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:08.624872 13917 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.625530  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:08.625947 13897 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:08.626197  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:08.626752 13897 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:28:08.627234 13897 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:08.627381  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:08.408078 13505 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:08.628022  5710 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:08.552876  5629 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:08.568571 13921 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:08.566164  6271 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:08.643021  6271 raft_consensus.cc:606] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:08.556308  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:08.643947  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 87750.2us
I20260709 12:28:08.570015 13834 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.572372 13743 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:08.576687  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 46839us
I20260709 12:28:08.576781  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:08.578655  5553 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:28:08.572253 13764 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:08.662205  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:08.668706  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:08.669425  5710 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:08.672060 13370 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:08.672783 13370 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.674563  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:08.663246  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:08.577775 13909 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.582029  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:08.583551  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:08.668746  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.685175  6274 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:28:08.685680  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:08.553215 13875 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:08.397285 13918 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:28:08.604231  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.698144  5557 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:08.642450 13921 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:08.657087 13861 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:08.609457 13862 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.373812 13506 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:08.611181  6280 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:08.592804  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:08.368464 13812 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:08.615362  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.622902 13819 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:08.628168 13897 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.629477  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:08.602715 13895 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:08.631037 13521 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:08.398883 13833 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:08.629511  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:08.634907  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.622180  5552 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:08.736255  5554 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:28:08.529448  5702 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:08.675746  5558 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:08.685946  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.737676  5553 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:08.737859  5556 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:08.687944  6275 raft_consensus.cc:2410] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:28:08.687268  5634 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
W20260709 12:28:08.689126  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 350036us
I20260709 12:28:08.691438  5561 raft_consensus.cc:2368] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20.
W20260709 12:28:08.687143  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:08.700739  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:08.608225 13782 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:08.704288 13764 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:08.744498 13866 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:08.744638  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:08.744956 13370 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:08.745168 13866 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:08.745622 13866 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:08.745857 13505 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:08.745558 13875 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.746744 13875 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:08.711361  5701 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:08.747803  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:08.691203 13900 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.716415  5709 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 19.
W20260709 12:28:08.718297  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 43816.9us
I20260709 12:28:08.641213  5560 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.686852 13834 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:08.721519  5627 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:08.724282  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.728614  6272 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:08.729774  6277 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:28:08.720422  5630 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:08.708292 13921 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:08.706736 13826 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:08.607486  6279 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:08.729207 13909 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:28:08.743850  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 114459us
I20260709 12:28:08.746515 13505 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.742408  5703 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:08.748538  5701 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:08.744493  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:08.750144  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:08.788964  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:08.750710  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.749186  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.750296  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.749737  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:08.788134  6271 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
W20260709 12:28:08.748788  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 119363us
I20260709 12:28:08.768113  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:08.787115  5710 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:08.804567  6274 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:08.793466  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:08.807333  5552 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:08.812830  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:08.815061  5710 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:08.817405  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:08.816783 13814 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:08.829689 13814 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.831658  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:08.825186 13322 raft_consensus.cc:2764] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:28:08.833148  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:08.833871 13322 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:08.834378 13322 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:08.834828 13322 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:08.835794  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.836648  5557 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:08.801640 13895 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:08.838889 13782 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.839181 13532 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:28:08.816162 13370 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:28:08.837824  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.776392 13833 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:08.842268  5557 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:08.855638  5706 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:08.744516 13870 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:08.803606 13873 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:08.859061  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:08.733129 13812 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:08.883597  5675 connection.cc:256] RPC call timeout handler was delayed by -2.2492077959996095s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
W20260709 12:28:08.884442  5675 connection.cc:256] RPC call timeout handler was delayed by -2.1457077959994422s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17170000000000002s for a response
I20260709 12:28:08.885404  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:08.886782  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:08.887514  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:08.867487  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:08.879303  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:08.829952 13928 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:28:08.862560  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:08.738332 13819 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.888595  5561 raft_consensus.cc:2376] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25. Re-sending same reply.
I20260709 12:28:08.748251 13612 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:08.889454  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:08.891230  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:08.911415  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 94115.7us
I20260709 12:28:08.848467 13826 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:08.912547  5557 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 17.
I20260709 12:28:08.878293 13814 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:08.931921  5707 raft_consensus.cc:2410] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:28:08.934759  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:08.940042  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 54805.5us
W20260709 12:28:08.941792  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 79365.1us
I20260709 12:28:08.941933 13826 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.942696  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:08.934140 13743 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.009s
I20260709 12:28:08.943933 13743 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:28:08.944288 13743 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:08.944661  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.945294  5557 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:08.945045 13743 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.946249  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:08.946432 13743 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:08.946799  5557 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:08.946947 13697 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:08.952003  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:08.953583  5559 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:08.954416  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:08.955013  5559 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:28:08.955212  5707 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 18: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:08.966275 13819 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:08.967439 13877 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:08.968534  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3445) took 1208 ms. Trace:
I20260709 12:28:08.969038  5705 rpcz_store.cc:276] 0709 12:28:07.760500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:08.967426 (+1206926us) service_pool.cc:224] Handling call
0709 12:28:08.967867 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:08.968106 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:08.968140 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:08.968198 (+    58us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:28:08.968243 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:08.968312 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:08.968468 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:08.969704 13612 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:08.881913 13817 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.088s
I20260709 12:28:08.970973 13612 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:08.971333 13612 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:08.971987 13505 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:08.972213 13612 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.973536 13612 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:08.975760  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:08.911844 13871 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:08.976683  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 46032.9us
I20260709 12:28:08.971344  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3446) took 1206 ms. Trace:
I20260709 12:28:08.975835 13524 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.005s
W20260709 12:28:08.981946 13925 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:28:08.915701 13861 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:08.838907 13895 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:08.986374 13532 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:28:08.883001 13870 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:08.993113 13862 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:08.993678 13601 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:08.888732 13873 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:08.995074  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 227076us
W20260709 12:28:08.995357  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:09.030601  6279 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:08.995592  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:09.001014  5561 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:09.000699  5705 rpcz_store.cc:276] 0709 12:28:07.765030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:08.970402 (+1205372us) service_pool.cc:224] Handling call
0709 12:28:08.970776 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:08.970967 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:08.971004 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:08.971059 (+    55us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:28:08.971103 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:08.971165 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:08.971282 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:09.001888  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:08.860843 13916 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.071s
W20260709 12:28:09.009480  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 185302us
I20260709 12:28:09.035531  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:09.015367  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:09.018383  5701 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:28:08.944962 13936 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:28:08.994238  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 346276us
I20260709 12:28:09.003835 13910 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:08.876791 13833 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.036237  5553 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:09.036501  5558 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:28:09.038795  6279 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:28:09.039057  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:09.040572  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:09.040980  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45467.9us
I20260709 12:28:09.046473  6280 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:09.048907  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:09.049283  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:09.049580  6271 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:09.050097  6278 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 23. Ops: []
I20260709 12:28:09.025760 13601 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:09.055163  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:09.029101 13755 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:09.058281  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:09.026925 13506 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:09.040970 13742 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:09.062263  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:09.068946  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:09.068951  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:09.076545 13773 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:09.076278 13942 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.053s
I20260709 12:28:09.076990 13773 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:09.077410 13773 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:09.078605 13766 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:09.079574 13766 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:09.080088 13900 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:09.080683 13766 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:09.081547 13902 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.712s	user 0.011s	sys 0.000s
I20260709 12:28:09.084575 13755 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.054523 13532 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:09.118851 13755 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:09.121099  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:09.121930  5701 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:09.122725  5708 raft_consensus.cc:2410] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:28:09.120883  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:09.124045 13900 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:28:09.067584 13872 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:09.124694 13532 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:09.124797 13872 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:09.125523  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:09.126151  5705 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:09.125825 13872 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.127028 13872 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:09.128266 13521 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:09.128712 13521 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:09.048815 13627 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:09.131721 13627 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:09.133401 13524 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:09.133939 13524 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:09.135057 13524 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.056195 13601 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:09.082008 13907 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 12:28:09.112797  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:09.139001  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:09.122728  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:09.140618  5557 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:09.110656  6271 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:09.132143 13627 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:09.148007  5701 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:09.191685  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:09.193053  5701 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:09.148945  5705 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:09.151075  5553 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:09.196991  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:09.204546  5703 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:09.209538  5701 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:09.210601  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3448) took 1198 ms. Trace:
I20260709 12:28:09.211315  5701 rpcz_store.cc:276] 0709 12:28:08.012419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.208571 (+1196152us) service_pool.cc:224] Handling call
0709 12:28:09.209225 (+   654us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:09.210214 (+   989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:09.210520 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:28:09.214226  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 145340us
I20260709 12:28:09.226701  5703 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 12:28:09.229059  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3447) took 1349 ms. Trace:
W20260709 12:28:09.229462  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 182544us
W20260709 12:28:09.229943  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77115.2us
I20260709 12:28:09.229574  5703 rpcz_store.cc:276] 0709 12:28:07.879549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.225068 (+1345519us) service_pool.cc:224] Handling call
0709 12:28:09.226473 (+  1405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:09.228788 (+  2315us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:09.229003 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:28:09.227473  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 212134us
I20260709 12:28:09.232057  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:09.230825  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:09.233754  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 121043us
I20260709 12:28:09.233662  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.148029 13524 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:09.136459 13861 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.082023 13902 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.225193 13834 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:09.251957  5709 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:09.225728 13900 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:09.254638  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:09.259373  6271 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:09.259806  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:09.261672  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:09.277817  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.279489  5709 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:28:09.292250  5630 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 24: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:09.293134  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.297295  6271 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:09.302495  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.303514  5634 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:28:09.304908  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:09.283083 13938 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:09.310397  5702 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:09.314291  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.250445 13742 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:09.319074  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:09.320294  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.267129 13834 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:09.326475  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:09.279863 13826 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:09.327601 13834 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:09.328071 13945 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:09.348184  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:09.349145  5629 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:09.342072  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 85137.3us
I20260709 12:28:09.342847  5628 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:28:09.344269  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:09.322666 13875 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.575s	user 0.009s	sys 0.003s
I20260709 12:28:09.346675  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:09.310272 13900 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:09.352537  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:09.352262  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:09.356696  6271 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:28:09.350589  5628 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:09.331159 13812 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:09.362803  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:09.363718  5627 raft_consensus.cc:2410] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 26 because replica has last-logged OpId of term: 25 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
I20260709 12:28:09.365600  5632 raft_consensus.cc:2410] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:28:09.365649  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:09.332422 13742 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:09.374498 13742 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:09.376804 13944 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
W20260709 12:28:09.378304 13932 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.235s	user 0.000s	sys 0.000s
W20260709 12:28:09.381063 13322 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.545s	user 0.013s	sys 0.000s
I20260709 12:28:09.381575 13322 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.385016 13773 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.386518  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:09.387265 13773 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:09.338107 13880 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:09.366178  5628 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.350632  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:09.392732  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.393455  5627 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:28:09.403582  5630 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:09.407184  5635 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:09.409926  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:09.297871 13506 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.348488 13945 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:09.410821  5635 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 24: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:09.417075  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.412528  5628 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:09.415575  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:09.418977  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.370262 13867 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:09.384191 13322 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:09.419608  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 69053.8us
I20260709 12:28:09.424041 13771 raft_consensus.cc:993] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: : Instructing follower f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:28:09.426327 13953 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:28:09.427027  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.427891 13951 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:28:09.418576 13826 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:09.428900  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 76739.3us
I20260709 12:28:09.429683  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3441) took 1673 ms. Trace:
I20260709 12:28:09.431885  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:09.454811  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:09.455117  5631 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:28:09.455643  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:28:09.433329 13370 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.577s	user 0.010s	sys 0.001s
I20260709 12:28:09.454499  5707 rpcz_store.cc:276] 0709 12:28:07.756008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:08.957010 (+1201002us) service_pool.cc:224] Handling call
0709 12:28:08.957640 (+   630us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:09.427725 (+470085us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:09.427772 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:09.427828 (+    56us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:09.427874 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:09.429412 (+  1538us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b54002d14d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:09.429465 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:09.429620 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1080548736}
I20260709 12:28:09.436229 13945 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.436609 13627 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.394169 13880 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:09.451347  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:09.448197  5629 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:09.459513  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:09.445153  5627 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:28:09.432336 13866 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.686s	user 0.010s	sys 0.002s
I20260709 12:28:09.433825 13895 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:09.411720 13870 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:09.459854  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:09.469367  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.464638  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:09.477761  5632 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:09.478909  5629 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:28:09.444315 13909 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.626s	user 0.010s	sys 0.000s
I20260709 12:28:09.470583 13627 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:09.469027 13370 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:28:09.485442 13920 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.879s	user 0.005s	sys 0.003s
I20260709 12:28:09.469373 13322 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:09.486114 13322 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:09.489660  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:09.490877 13322 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:09.470700 13953 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:09.494453  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:09.492863  5710 raft_consensus.cc:2410] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:28:09.460626 13880 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:09.495514  5627 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:09.498855  5627 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:09.480394 13875 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:09.461457 13771 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:09.462497 13917 raft_consensus.cc:1081] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Signalling peer f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:28:09.480083 13867 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:28:09.459483 13743 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.513s	user 0.008s	sys 0.005s
I20260709 12:28:09.500530  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:09.495119 13945 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:09.480260 13951 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:09.483548 13872 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:09.499740  5627 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.510959  5707 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:28:09.500872  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.512022  5560 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:09.516930  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:09.498072 13833 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:09.510072 13897 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.009s	sys 0.004s
W20260709 12:28:09.559634  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 319408us
I20260709 12:28:09.480804 13866 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.521193  5627 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:09.525401  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:28:09.538607  5706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3440) took 1782 ms (client timeout 1676 ms). Trace:
W20260709 12:28:09.566627  5706 rpcz_store.cc:269] 0709 12:28:07.755722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:07.756648 (+   926us) service_pool.cc:224] Handling call
0709 12:28:09.538295 (+1781647us) spinlock_profiling.cc:243] Waited 966 ms on lock 0x7b540026e654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:09.538465 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2222708480}
I20260709 12:28:09.568177  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:09.568723  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:09.516579 13872 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:09.507361 13895 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:09.461617 13826 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.502398 13875 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:09.568883  5706 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:09.540108  5705 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:09.506206 13834 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.587368  5707 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:09.578233  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:09.591013  6274 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:09.578317  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60141.5us
I20260709 12:28:09.591910  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:09.565089 13897 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.010s	sys 0.004s
I20260709 12:28:09.599258  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:09.601282  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:09.486972 13900 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.603263  6279 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:09.555835 13902 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:09.669224  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:09.588239 13955 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.093s	user 0.004s	sys 0.000s
I20260709 12:28:09.577512 13872 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:09.672187  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:09.672572  5627 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:28:09.676810  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.673063  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:09.678246  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:09.678825  5707 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:09.592582 13875 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:09.678966  6274 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:09.597692 13895 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:09.681504  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.695719  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:09.673775 13950 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:09.705647  5635 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
W20260709 12:28:09.709760  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:09.709925  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3452) took 1365 ms. Trace:
W20260709 12:28:09.711318  5708 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3114) took 1844 ms (client timeout 1946 ms). Trace:
W20260709 12:28:09.711724  5708 rpcz_store.cc:269] 0709 12:28:07.866752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:08.016196 (+149444us) service_pool.cc:224] Handling call
0709 12:28:08.184261 (+168065us) spinlock_profiling.cc:243] Waited 1.44 ms on lock 0x7b5800053198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:28:08.602610 (+418349us) spinlock_profiling.cc:243] Waited 2.46 ms on lock 0x7b5c00178e8c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffdfdc161 00007f4ffe17da79 00007f4fff6379b6 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:09.700302 (+1097692us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x7b5400210554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:09.711256 (+ 10954us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1202857728}
I20260709 12:28:09.681985 13947 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:09.710359  5709 rpcz_store.cc:276] 0709 12:28:08.344206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.281332 (+937126us) service_pool.cc:224] Handling call
0709 12:28:09.294886 (+ 13554us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:09.295209 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:28:09.417757 (+122548us) log.cc:844] Serialized 14 byte log entry
0709 12:28:09.709596 (+291839us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:09.709654 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:09.709699 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:09.709762 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:09.709870 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11102,"prepare.queue_time_us":205987,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"thread_start_us":366226,"threads_started":2}
I20260709 12:28:09.689337 13897 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:09.701426 13909 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:09.713063 13902 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:09.733764  5708 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:09.725349  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:09.718345 13843 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
W20260709 12:28:09.729143 13871 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.725s	user 0.009s	sys 0.002s
I20260709 12:28:09.740173 13951 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.753458 13866 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:09.757187 13959 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:28:09.725821 13960 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:09.806161  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:09.759759 13771 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:09.763727 13921 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.960s	user 0.005s	sys 0.003s
I20260709 12:28:09.768417 13902 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.771005  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1332) took 1415 ms. Trace:
I20260709 12:28:09.684602 13953 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.779379  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3113) took 1913 ms. Trace:
I20260709 12:28:09.818785  5705 rpcz_store.cc:276] 0709 12:28:07.866443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.221846 (+1355403us) service_pool.cc:224] Handling call
0709 12:28:09.539822 (+317976us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:09.590123 (+ 50301us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:09.779120 (+188997us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b540026e650. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:09.779296 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":730417024}
I20260709 12:28:09.781600  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:09.781637  6279 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:28:09.782603  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:09.783798  6275 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:09.784946 13826 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:09.785590 13834 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:09.738770  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3141) took 1384 ms. Trace:
I20260709 12:28:09.788597  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:09.853609  5556 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:09.853385  5708 rpcz_store.cc:276] 0709 12:28:08.354346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.716061 (+1361715us) service_pool.cc:224] Handling call
0709 12:28:09.732197 (+ 16136us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:09.738410 (+  6213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:09.738697 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:09.789742  5628 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:28:09.790799  5560 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:09.854997  5701 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
W20260709 12:28:09.792219  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 193070us
I20260709 12:28:09.797109  5557 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:28:09.797864  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:09.798740  5558 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:28:09.856863  5553 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 18.
I20260709 12:28:09.799865  6271 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:09.858063  6271 raft_consensus.cc:606] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
W20260709 12:28:09.800911  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 67836.4us
I20260709 12:28:09.724308 13950 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:09.803049  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:09.805994  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:09.861198  5554 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:09.731078 13897 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:09.737317 13909 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.775453  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:09.817696  6278 rpcz_store.cc:276] 0709 12:28:08.355908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:08.356087 (+   179us) service_pool.cc:224] Handling call
0709 12:28:08.360045 (+  3958us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:09.050490 (+690445us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:09.770748 (+720258us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b5400343ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:09.770917 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1586513664}
W20260709 12:28:09.851059  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 249810us
I20260709 12:28:09.828475  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:09.804941 13871 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:09.803848 13895 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.814391  5706 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:09.868167  5705 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 26
W20260709 12:28:09.870394  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:09.870272  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:09.876662  5708 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
W20260709 12:28:09.867698 13817 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
I20260709 12:28:09.889398  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3455) took 1456 ms. Trace:
I20260709 12:28:09.880137  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:09.889878  5706 rpcz_store.cc:276] 0709 12:28:08.426427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.881574 (+1455147us) service_pool.cc:224] Handling call
0709 12:28:09.881860 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:09.882027 (+   167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:09.882052 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:09.882090 (+    38us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:28:09.882120 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:09.882169 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:09.882267 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:09.902271  5703 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:09.902949  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3127) took 1565 ms. Trace:
W20260709 12:28:09.903373 13521 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.774s	user 0.004s	sys 0.008s
I20260709 12:28:09.903957 13521 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:09.904511  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:09.904564 13899 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:09.905303 13899 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:09.905705 13899 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:09.907092  5555 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:09.908145 13521 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:09.827955 13921 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.909380 13843 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:09.910029  6280 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:09.909933 13843 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.910550 13921 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:09.817495 13771 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.912829 13771 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:09.913005 13873 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:09.876749 13950 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.913890  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:09.914777  5628 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:09.916463 13950 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:09.917039  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.920651  5631 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:28:09.912153 13843 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:09.923020  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:09.923610 13834 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:28:09.924643 13834 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:28:09.926193 13834 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:09.926116 13782 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:09.926682 13834 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:09.920718  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:09.929237  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.903263  5703 rpcz_store.cc:276] 0709 12:28:08.337467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.257836 (+920369us) service_pool.cc:224] Handling call
0709 12:28:09.901868 (+644032us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b5800053198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:28:09.902093 (+   225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:09.902720 (+   627us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:09.902895 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4664704}
W20260709 12:28:09.930836  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 61218.5us
I20260709 12:28:09.931174  5628 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:09.931607  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:09.938126  5628 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:09.932703  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:09.943125  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:09.943831  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:09.945159  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3133) took 1593 ms (client timeout 2105 ms). Trace:
I20260709 12:28:09.943958  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3450) took 1793 ms. Trace:
I20260709 12:28:09.946180  5701 rpcz_store.cc:276] 0709 12:28:08.150355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.225885 (+1075530us) service_pool.cc:224] Handling call
0709 12:28:09.226156 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:09.855373 (+629217us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:09.943770 (+ 88397us) spinlock_profiling.cc:243] Waited 629 ms on lock 0x7b540026eb54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:09.943911 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1445765504}
I20260709 12:28:09.948621  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3458) took 1113 ms. Trace:
I20260709 12:28:09.949405  5707 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:28:09.948938  5701 rpcz_store.cc:276] 0709 12:28:08.835255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.947811 (+1112556us) service_pool.cc:224] Handling call
0709 12:28:09.948120 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:09.948293 (+   173us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:09.948318 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:09.948391 (+    73us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:09.948424 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:09.948474 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:09.948570 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:09.950573  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:09.951190  6271 raft_consensus.cc:2376] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29. Re-sending same reply.
I20260709 12:28:09.940493  5703 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:09.954864  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:09.945546  5702 rpcz_store.cc:269] 0709 12:28:08.351741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:08.812086 (+460345us) service_pool.cc:224] Handling call
0709 12:28:09.944974 (+1132888us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b5400075154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:09.945105 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1098672896}
I20260709 12:28:09.923668 13872 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.962795  5701 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:28:09.972874  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3459) took 1060 ms. Trace:
I20260709 12:28:09.973219  5701 rpcz_store.cc:276] 0709 12:28:08.912701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.962195 (+1049494us) service_pool.cc:224] Handling call
0709 12:28:09.962477 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:09.972630 (+ 10153us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:09.972820 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:09.974737  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:09.949973 13958 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:28:09.982138  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:09.964556  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:09.964884  5554 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:28:09.936633 13322 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.968115  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:09.963387  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 85488.2us
I20260709 12:28:09.984560  5702 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:28:09.970453 13921 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:28:09.971696  5636 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:28:09.975883  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:09.986761  5632 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:09.855486 13960 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:09.975378  6277 raft_consensus.cc:2376] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22. Re-sending same reply.
I20260709 12:28:09.968657 13873 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:09.916090 13875 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:09.979301  5703 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260709 12:28:10.001120  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:09.996905  5676 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:10.017198  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:10.018097  5628 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:28:10.020061  5703 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:42568
I20260709 12:28:10.021013  5703 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
W20260709 12:28:10.021107  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 77355.7us
I20260709 12:28:10.021509  5703 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:10.022073  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:10.024180  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:10.026145  5675 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:09.982846 13945 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:09.992056 13532 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:10.036391  5702 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:10.041801  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:09.989850 13812 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:10.044833  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:10.047158  5628 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:10.000226 13900 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:28:10.060981  5628 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:10.029734 13936 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:10.064868  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:10.042773 13743 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.023s
I20260709 12:28:10.066126 13743 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:10.066474 13743 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:10.066618  5552 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:28:10.067317 13743 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.068876 13960 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:10.070221  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:10.070346  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.071254  5559 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:28:10.073364 13322 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:10.074254  5556 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 12:28:10.074745  5556 pending_rounds.cc:85] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 14
I20260709 12:28:10.075033  5556 pending_rounds.cc:107] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 23.15
W20260709 12:28:10.075220 13880 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.569s	user 0.011s	sys 0.000s
I20260709 12:28:10.075361  5556 raft_consensus.cc:2889] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:28:10.076715 13601 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.868s	user 0.006s	sys 0.006s
I20260709 12:28:10.077503 13601 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:10.079981  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:10.050628 13873 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:10.083488  5556 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:10.031896 13897 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:10.086385  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 85333.2us
I20260709 12:28:10.075785 13880 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:10.095682 13871 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:28:10.097404 13871 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:10.097873 13871 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:10.098027 13880 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:10.099186 13871 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.086752  5703 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.101702  5702 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:10.103502  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:10.035131 13826 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:28:10.033599 13771 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:10.051838 13532 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:10.108827  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:10.106546  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:10.084019 13812 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:10.098254 13946 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.089s
I20260709 12:28:10.089633 13945 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:09.998234 13627 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.091s
I20260709 12:28:10.127127  5705 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:10.130548  5703 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:10.125187  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:10.101609 13921 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:09.993103 13834 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.119529 13826 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:10.173460 13826 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:10.121351 13973 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
I20260709 12:28:10.123615 13532 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:10.119282 13902 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:10.185818  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:28:10.129066 13961 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.266s	user 0.001s	sys 0.000s
I20260709 12:28:10.108310 13771 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:10.166648 13947 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:10.109411 13969 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:10.118811 13812 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:10.154670 13880 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:10.169445 13871 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:10.141064 13322 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:10.201625  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:10.230929  5633 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:10.132650 13873 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:10.122442 13866 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.063s
I20260709 12:28:10.196614 13947 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:10.198553 13771 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:10.212086  5632 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:10.223904 13875 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:10.224898 13872 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:10.227303 13971 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.258s	user 0.001s	sys 0.000s
I20260709 12:28:10.245076 13873 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.250605  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:10.251256  5634 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:10.228668 13969 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:10.230211  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:10.261077  5701 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:10.129210 13942 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.076s
I20260709 12:28:10.178486 13743 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:10.233364  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 202241us
I20260709 12:28:10.170661 13921 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.245452  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:10.236205 13627 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:10.140481 13945 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.230996 13834 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:10.231940 13322 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:10.239115  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:10.276762  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:28:10.277247  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:10.298271  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 59279.6us
I20260709 12:28:10.246757 13947 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:10.229995 13880 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:10.303422  5710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3140) took 1949 ms (client timeout 2170 ms). Trace:
W20260709 12:28:10.303880  5710 rpcz_store.cc:269] 0709 12:28:08.354007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:08.834481 (+480474us) service_pool.cc:224] Handling call
0709 12:28:10.303142 (+1468661us) spinlock_profiling.cc:243] Waited 604 ms on lock 0x7b54002f4254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:10.303348 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1389611392}
I20260709 12:28:10.303785  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3453) took 1944 ms. Trace:
I20260709 12:28:10.319676  5707 rpcz_store.cc:276] 0709 12:28:08.359326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.724998 (+1365672us) service_pool.cc:224] Handling call
0709 12:28:09.949158 (+224160us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:09.962765 (+ 13607us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:10.303574 (+340809us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b5400075150. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:10.303724 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":481357568}
I20260709 12:28:10.330283  5634 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:10.302042 13322 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:28:10.322613 13978 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:28:10.299780  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:10.300015  5630 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:28:10.348961  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49250.6us
I20260709 12:28:10.350257  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:10.344846  5701 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:10.300836 13951 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:10.295791  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:10.303684 13627 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:10.363215  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 337195us
W20260709 12:28:10.367143  5602 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.829s (SENT)
W20260709 12:28:10.368964  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:10.370224 13843 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:10.370865  5675 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:10.370755 13843 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:10.373204 13812 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:10.373731 13812 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:10.374146 13843 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:10.374637 13812 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 23.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:10.375531  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:10.376950 13875 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:10.377969  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:10.378680  5701 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:10.380752 13755 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:28:10.382745 13875 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:28:10.346102 13970 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:28:10.381021 13817 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:28:10.384945 13946 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:10.385434 13946 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:10.386412 13946 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:10.387591 13946 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:10.272192  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:10.391997  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:10.382900  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:10.400506  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 261246us
I20260709 12:28:10.402266  6237 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:10.400645  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 128421us
I20260709 12:28:10.406966  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:10.355975 13873 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:10.401394  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:10.358687 13895 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:10.355209 13521 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:10.425117 13506 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:10.432611  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:10.441430  5556 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:28:10.425704  5704 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:10.426317  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3457) took 1678 ms. Trace:
I20260709 12:28:10.357766 13370 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:10.467777  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:10.437953  5561 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:28:10.442651  5708 rpcz_store.cc:276] 0709 12:28:08.748441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.891438 (+1142997us) service_pool.cc:224] Handling call
0709 12:28:09.895371 (+  3933us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:09.924180 (+ 28809us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:09.924218 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:09.925572 (+  1354us) log.cc:844] Serialized 506 byte log entry
0709 12:28:10.043106 (+117534us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:28:10.043160 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:10.043224 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:10.426000 (+382776us) raft_consensus.cc:1685] finished
0709 12:28:10.426020 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:10.426237 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17223,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94,"thread_start_us":115914,"threads_started":1,"wal-append.queue_time_us":274481}
W20260709 12:28:10.462538  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 60365.5us
I20260709 12:28:10.415619 13843 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:28:10.463019  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47734.9us
I20260709 12:28:10.445541  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.501263  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:10.504065  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:10.505949  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:10.357079 13899 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:10.464831  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.465199  5554 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:28:10.412295 13902 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:28:10.432001  5634 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:10.358506 13897 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 23.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.079s
I20260709 12:28:10.473268  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.473860  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:10.473472  5629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
W20260709 12:28:10.477144  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:10.437193 13866 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:10.475500  5635 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:28:10.530680  5559 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:10.421090 13936 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.056s
I20260709 12:28:10.431878 13968 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:10.543742  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.492664  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.509699  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:10.492555  5556 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
W20260709 12:28:10.512897  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:10.531476  5553 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:10.545226  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.545435  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.545696  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.546149  5556 raft_consensus.cc:2368] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24.
I20260709 12:28:10.546907  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.547441  5556 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:10.427070 13960 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:10.531474  5629 raft_consensus.cc:606] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:10.515430  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:10.544502  5554 raft_consensus.cc:2376] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25. Re-sending same reply.
I20260709 12:28:10.473006 13895 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:10.559881  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:10.550101  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:10.560891  5705 raft_consensus.cc:2410] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 21 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:28:10.550225  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.548237  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.561728  5554 raft_consensus.cc:2376] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19. Re-sending same reply.
W20260709 12:28:10.550450 13981 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:28:10.437803 13826 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.094s
I20260709 12:28:10.556874  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: true dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:10.564896  5559 raft_consensus.cc:2368] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26.
W20260709 12:28:10.532241 13977 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 12:28:10.571677  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:10.563344  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:10.563160 13968 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.570819  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:10.573266  5710 raft_consensus.cc:2410] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:28:10.519627 13755 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:10.572139  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.574518  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:10.578795  5556 raft_consensus.cc:2376] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25. Re-sending same reply.
I20260709 12:28:10.562165  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:10.572425  6280 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:10.577188  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:10.577342  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.580070  5561 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:28:10.580235  6280 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:10.580776  5554 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:10.581503  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:10.582916  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:10.479271 13370 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:10.581240  6280 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.586335  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:10.591014  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:10.591279  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:10.591503  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:10.593964  5631 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:28:10.569274 13506 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:10.593851  6276 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:10.594640  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:10.594887  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:10.595211  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 85807.4us
W20260709 12:28:10.590499  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 156568us
I20260709 12:28:10.595595  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:10.573552 13866 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:10.590409  6280 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:10.576419 13897 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:10.600679  5701 raft_consensus.cc:2376] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21. Re-sending same reply.
I20260709 12:28:10.577430 13992 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:10.590727  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:10.602193  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:10.432305 13969 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.607578  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:10.610921  5710 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:10.612891  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:10.613547  5629 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
W20260709 12:28:10.613696  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:10.613554  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:10.541342 13627 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.615862 13899 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:10.616885  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:10.618808  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:10.619331 13902 raft_consensus.cc:2764] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:28:10.620834  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:10.621493  5553 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:10.622179 13870 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:10.622893 13870 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:10.605618 13826 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:10.626350  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:10.607692 13880 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:10.605199 13506 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.629777 13506 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:10.632210  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:10.632705  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:10.636528  5708 raft_consensus.cc:2368] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19.
I20260709 12:28:10.630059 13897 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:10.640988 13907 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:10.641500 13907 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:10.623903 13870 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.642473 13907 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.645803  5560 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:28:10.648176  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:10.648787  5560 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:10.648509 13992 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.620110 13902 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.021s
I20260709 12:28:10.652451  5561 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:28:10.652629  5705 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:28:10.601317 13921 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:10.572459 13960 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.657982  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:10.661098  5554 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:10.661201 13999 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:28:10.662784  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:10.671312  5710 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:10.670295  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:10.667634 13907 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:10.687865 13897 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:10.688146  5702 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:28:10.688323 13370 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:10.689430 13968 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:10.689659 13866 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.699404  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:10.700258  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:10.666416 13991 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.173s	user 0.000s	sys 0.003s
W20260709 12:28:10.708042  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45910.1us
W20260709 12:28:10.712996  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 80866.8us
I20260709 12:28:10.723835  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:10.724248  6280 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:10.661912 13506 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:10.725106  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:10.726200  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:10.734645  5560 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:10.650327 13987 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.074s
I20260709 12:28:10.736876  5553 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:10.735987  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:10.739538  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:10.722059 13975 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.327s	user 0.001s	sys 0.000s
I20260709 12:28:10.653003 13532 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.748610  5553 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:10.754464  5710 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:10.759469  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:10.737177 13899 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:10.771044  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:10.773262  5630 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:10.737852 13936 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:28:10.741496 13812 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:10.745141 13755 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:10.783701  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:10.747993 13506 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:10.725402 13902 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.050s
I20260709 12:28:10.785730  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:10.737293 13921 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.043s
W20260709 12:28:10.787827  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:10.670787 13895 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:10.791531  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:10.868182  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 76734.9us
W20260709 12:28:10.868317  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:10.869254  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:10.794907  5557 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:28:10.777920 13826 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:10.800642  5634 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:28:10.878408  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:10.878345  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:10.881775  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:10.883908 13872 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:10.786813 13997 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:10.792012 13873 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:10.922493  5630 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:28:10.868448 13755 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:10.781855 13899 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.884547 13872 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.877918 13370 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:10.920154  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:10.876631 13866 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:10.877776 13861 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:10.947022  5630 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:28:10.951601  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:10.915930 13812 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:10.919545 13946 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20260709 12:28:10.966538 13986 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:28:10.970916 13989 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.258s	user 0.000s	sys 0.001s
W20260709 12:28:10.979456 14001 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:28:10.984639 13506 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:10.989331  5636 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 2990) took 1687 ms (client timeout 2242 ms). Trace:
W20260709 12:28:10.990350  5636 rpcz_store.cc:269] 0709 12:28:09.301954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.310404 (+  8450us) service_pool.cc:224] Handling call
0709 12:28:10.989114 (+1678710us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b540032d7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:10.989256 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1317936384}
W20260709 12:28:10.946810 13947 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.625s	user 0.008s	sys 0.004s
I20260709 12:28:10.997059  5627 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:28:10.953795 13861 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:11.003185 13812 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:11.012871 13976 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.264s	user 0.001s	sys 0.000s
I20260709 12:28:11.013908 13812 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.002295 13968 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:11.016011 13968 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:10.956451 13370 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.018934 13968 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:11.019068 13370 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:11.021482  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:11.027668  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 328174us
I20260709 12:28:11.034389  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:11.035136  6271 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
W20260709 12:28:11.029917 13843 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.006s	sys 0.008s
W20260709 12:28:11.038507 13843 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.006s	sys 0.008s
I20260709 12:28:11.038846 13843 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:11.039846 13843 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.043828 13897 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:11.044552 13897 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:11.049607  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 180484us
I20260709 12:28:11.047768  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:11.049760  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:11.022142 13861 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:11.051290  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:11.046021  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:11.001163 13946 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 12:28:11.052325  5636 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:10.961380 13866 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.003278 13947 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.057796 13946 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:11.063046 13946 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:28:11.063459 13946 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:11.062100 13907 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:11.065466 13907 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.063109  5552 raft_consensus.cc:2368] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24.
I20260709 12:28:11.068310  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3280) took 1268 ms. Trace:
I20260709 12:28:11.072006  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:11.068809  5555 rpcz_store.cc:276] 0709 12:28:09.799988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.800422 (+   434us) service_pool.cc:224] Handling call
0709 12:28:09.813628 (+ 13206us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:09.907571 (+ 93943us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:11.068031 (+1160460us) spinlock_profiling.cc:243] Waited 93.3 ms on lock 0x7b54003c4bd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:11.068251 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":214486912}
I20260709 12:28:11.036996 13960 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:11.064682 13947 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:11.074815  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:11.054598 13370 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:11.064476 13946 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.068691 13627 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:11.066151 13909 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:11.066816 13992 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:11.092664  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 166503us
I20260709 12:28:11.089749  6276 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: [28.21-28.21]
I20260709 12:28:11.033310 13755 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.085762 13370 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:11.108654 13370 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:28:11.110127 14002 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.245s	user 0.002s	sys 0.000s
I20260709 12:28:11.078512 13532 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:11.121964 14013 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:11.132411  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:11.133349  5553 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:11.133733 13866 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:11.134156  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:11.134889  5553 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:11.135576  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:11.136240 13370 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.137725 13899 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:11.138523 13771 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.878s	user 0.015s	sys 0.000s
I20260709 12:28:11.139245 13907 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:11.125269 13506 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.014s
W20260709 12:28:11.122058 14004 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 12:28:11.131124  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:11.122604  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:11.141778  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:11.142856  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:11.143270 14009 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:28:11.143949  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:11.140056 13897 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:11.150111 13987 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:28:11.140092 13506 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:11.151594  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:11.152370  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:11.107568 13946 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:11.155953  5630 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:28:11.157317  5555 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
W20260709 12:28:11.101256  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 149763us
W20260709 12:28:11.179929  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47319.4us
I20260709 12:28:11.162273  5553 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:28:11.185189  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:11.183746  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:11.150992 13601 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:11.184793  5602 leader_election.cc:400] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24.
I20260709 12:28:11.188290  5632 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:28:11.189122  6277 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:11.189682  6277 raft_consensus.cc:606] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:28:11.189795  5602 leader_election.cc:403] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:28:11.138923 13834 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.822s	user 0.013s	sys 0.000s
W20260709 12:28:11.159710 13743 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.883s	user 0.009s	sys 0.003s
I20260709 12:28:11.138885 13370 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:11.227342  5629 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.007s	sys 0.022s
W20260709 12:28:11.209089  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 56853.9us
W20260709 12:28:11.227833  5629 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.007s	sys 0.022s
I20260709 12:28:11.208813  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:11.158977 13506 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:11.184926 13899 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:11.186940 13627 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:11.180388 13953 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:11.238356  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:11.374811  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 136510us
I20260709 12:28:11.376055  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:11.356271  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:11.317876 14017 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24.
W20260709 12:28:11.351255 14023 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:28:11.324404 13988 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.830s	user 0.007s	sys 0.006s
I20260709 12:28:11.289284 13370 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.018s
I20260709 12:28:11.256850 13743 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:11.257323 13899 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.272446 13861 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:11.408826  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 180484us
I20260709 12:28:11.188812 13960 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.067s
I20260709 12:28:11.424916  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3442) took 3665 ms. Trace:
I20260709 12:28:11.300139 13506 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:11.354868 13990 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.813s	user 0.004s	sys 0.010s
I20260709 12:28:11.267661 13942 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:11.229282 14008 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.078s
W20260709 12:28:11.410192 13322 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.061s	user 0.014s	sys 0.000s
I20260709 12:28:11.206961 13909 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:28:11.240068  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 165563us
I20260709 12:28:11.364986  5628 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:11.204916 13872 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:11.159821 13771 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:11.352128 14024 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:28:11.472242 13861 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:11.391166  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:11.291646  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3013) took 1765 ms. Trace:
W20260709 12:28:11.477030  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 86004.4us
I20260709 12:28:11.407788 13902 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:11.427225  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 217832us
I20260709 12:28:11.425426  5704 rpcz_store.cc:276] 0709 12:28:07.758372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:08.962619 (+1204247us) service_pool.cc:224] Handling call
0709 12:28:08.962951 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:10.442449 (+1479498us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:11.423241 (+980792us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54001a9354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:11.423476 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3363595648}
I20260709 12:28:11.270727 13897 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 23.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.030s
I20260709 12:28:11.430461  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:11.405603 13875 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:28:11.468468 13974 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.473s	user 0.012s	sys 0.000s
I20260709 12:28:11.468973  5560 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:28:11.469300 13771 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:11.487339  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:11.474941  6278 raft_consensus.cc:2368] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23.
I20260709 12:28:11.487926  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:11.475018  6277 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:11.478058  5629 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:28:11.488641  6277 raft_consensus.cc:606] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:11.408128 13953 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:11.484452 13939 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.364s	user 0.008s	sys 0.028s
W20260709 12:28:11.490350 13939 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.484s	user 0.008s	sys 0.029s
I20260709 12:28:11.477471  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1693) took 1240 ms. Trace:
I20260709 12:28:11.460083 13942 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:11.477113  5627 rpcz_store.cc:276] 0709 12:28:09.526158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.683223 (+157065us) service_pool.cc:224] Handling call
0709 12:28:10.989655 (+1306432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:10.999762 (+ 10107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:11.291388 (+291626us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540032d7d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:11.291566 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3003850112}
I20260709 12:28:11.464263 13506 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:11.486351  5602 leader_election.cc:400] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26.
I20260709 12:28:11.497525  5602 leader_election.cc:403] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:28:11.489009  6277 raft_consensus.cc:614] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:11.491261  5633 rpcz_store.cc:276] 0709 12:28:10.237046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:10.237181 (+   135us) service_pool.cc:224] Handling call
0709 12:28:10.237520 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:10.237862 (+   342us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:10.237889 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:10.238349 (+   460us) log.cc:844] Serialized 26 byte log entry
0709 12:28:10.394450 (+156101us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:10.394493 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:10.394566 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:11.354125 (+959559us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:11.477134 (+123009us) raft_consensus.cc:1685] finished
0709 12:28:11.477162 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:11.477400 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":885,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":3953,"spinlock_wait_cycles":33692032,"thread_start_us":155471,"threads_started":1,"wal-append.queue_time_us":201382}
I20260709 12:28:11.493080  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:11.410421 13743 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:11.493003  5706 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:11.493129  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:11.500810  5560 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:11.495558  5557 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:11.502233 13743 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:11.502803 13999 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:28:11.482367 13907 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:11.484157 14021 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:28:11.460590 13322 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.485308 14032 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:28:11.525012  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:11.527102  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45351.8us
I20260709 12:28:11.527288  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3031) took 1056 ms. Trace:
I20260709 12:28:11.527901  5556 rpcz_store.cc:276] 0709 12:28:10.471328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:10.587050 (+115722us) service_pool.cc:224] Handling call
0709 12:28:10.587391 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:10.598705 (+ 11314us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:10.598742 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:10.606332 (+  7590us) log.cc:844] Serialized 26 byte log entry
0709 12:28:10.776817 (+170485us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:10.920655 (+143838us) log.cc:844] Serialized 14 byte log entry
0709 12:28:10.920914 (+   259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:10.921015 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:11.475817 (+554802us) spinlock_profiling.cc:243] Waited 29.7 ms on lock 0x7b240013f538. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff740b9c4 00007f4ff7409acb 00007f4ff75c4dc9 00007f4ff75c4a55 00007f4ff75c3fb0 00007f4ffa23a488 00007f4ffa23abe8 00007f4ffa212398 00007f4ffa20d76a 00007f4ffa20cbb6
0709 12:28:11.518113 (+ 42296us) raft_consensus.cc:1685] finished
0709 12:28:11.527065 (+  8952us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:11.527224 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":776,"prepare.queue_time_us":259400,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":7102,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":28450,"spinlock_wait_cycles":68315008,"thread_start_us":286201,"threads_started":2,"wal-append.queue_time_us":291432}
I20260709 12:28:11.521600  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:28:11.506834 14008 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:11.537678  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:11.513175  5557 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:28:11.490720 13601 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.007s
I20260709 12:28:11.467316 13909 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:11.520558 14043 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:11.541205  5559 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:11.488739 13902 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.510246 13939 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.032s
I20260709 12:28:11.546674  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:11.537596 14008 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:11.567644  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:11.551780  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58771.5us
I20260709 12:28:11.546552 13861 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:28:11.574885  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:11.575094  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:11.522715 14032 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:28:11.558868  5628 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 1690) took 1373 ms (client timeout 1745 ms). Trace:
I20260709 12:28:11.544006 13909 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:11.582156  5628 rpcz_store.cc:269] 0709 12:28:10.185405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:10.185573 (+   168us) service_pool.cc:224] Handling call
0709 12:28:11.558681 (+1373108us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540009e7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:11.558787 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2388991872}
W20260709 12:28:11.569497  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 67551.2us
I20260709 12:28:11.570554  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:11.564734 13872 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:11.571906  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:11.576138  5701 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:28:11.589421  5557 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
W20260709 12:28:11.590812  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53219.9us
I20260709 12:28:11.543798 14043 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:11.556929 13812 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.035s
I20260709 12:28:11.570379 14008 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:28:11.579380  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:11.568843 13833 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:28:11.544327 14030 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.048s
I20260709 12:28:11.587751  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:11.599506  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:11.628439  5707 raft_consensus.cc:2410] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 24 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:28:11.599822  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:11.600107  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:11.603536  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:11.573916 14021 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:11.653614 14031 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:28:11.583294  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 153164us
I20260709 12:28:11.605125  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:11.619204  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:11.655061  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:11.620239  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:11.655767  6272 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:11.584790 13953 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:11.585580 13866 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:11.656672 14031 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:11.656803 13866 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.657599 14031 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:11.658870 14031 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.585729 13895 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:11.625248  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:11.653252  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:11.661269 14040 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
I20260709 12:28:11.661355  5709 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:28:11.661900 13946 raft_consensus.cc:2955] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Committing config change with OpId 15.11: config changed from index 3 to 11, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:11.630282  5710 raft_consensus.cc:2410] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 26 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
I20260709 12:28:11.588953 13861 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:11.660017 13866 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:11.665304 13861 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.667415 13861 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:11.635635 13899 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:11.636581 14032 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.668340 14032 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:11.670837  6272 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:11.604434 13907 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:11.672703 13907 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:11.592831 14033 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:11.673741 13907 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.674156 14033 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.675184  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1353) took 1165 ms. Trace:
I20260709 12:28:11.675647  6278 rpcz_store.cc:276] 0709 12:28:10.509282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:10.509424 (+   142us) service_pool.cc:224] Handling call
0709 12:28:11.674951 (+1165527us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b54006383d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:11.675101 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2138712960}
I20260709 12:28:11.609315 14013 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:11.617388 13872 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.677332 13939 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:11.676877 14013 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.677742 13939 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:11.678939 13872 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:11.678653 13939 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.641908 13996 raft_consensus.cc:993] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: : Instructing follower cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:28:11.642586 14016 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:11.584619 13942 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.644873 13322 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:11.645753 13902 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:11.619815 14044 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:11.648990  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61321.1us
I20260709 12:28:11.698802  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:11.649900 13999 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:28:11.638443 14037 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
W20260709 12:28:11.653229  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49797.7us
I20260709 12:28:11.710821  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:11.655522 14043 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.655999  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:11.679425 14013 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:11.681111  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 76087.5us
I20260709 12:28:11.720408  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:11.679919  6277 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:11.685325 14016 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.681774  5707 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:28:11.685317 13953 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:11.685716 13996 raft_consensus.cc:1081] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Signalling peer cea66aa15be3423880843974acdbf50f to start an election
W20260709 12:28:11.690333  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 64578.3us
I20260709 12:28:11.731596 13833 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.002s
I20260709 12:28:11.625171 13992 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:11.736477 13992 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.738595 13992 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:11.689347  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:11.739801  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:11.694739  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:11.740723  5555 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:11.741223  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:11.742066  5635 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:11.742286  5632 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:28:11.742967  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:11.743597  5635 raft_consensus.cc:2376] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30. Re-sending same reply.
I20260709 12:28:11.743726  5632 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:28:11.709817 14044 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.699842  5627 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 26: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:11.748279  5635 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:11.714184  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:11.717937  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 62039.5us
I20260709 12:28:11.751614  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:11.715781 13322 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:11.717962  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:11.727734  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:11.730866  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 106910us
I20260709 12:28:11.734545 14016 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:11.755159  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:11.756498  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:11.717464  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:11.762938  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:11.695927 13812 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 12:28:11.770179  5555 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:11.730055 14030 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:11.730175 14021 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:11.775259  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:11.776450  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:11.788511  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 80928.3us
W20260709 12:28:11.791949  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 74642.9us
I20260709 12:28:11.794651  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:11.794203  5555 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:28:11.796857  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:11.801735  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:11.803150  5707 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:28:11.805960  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:11.806659  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:11.808449  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52038.8us
I20260709 12:28:11.762034 13370 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:11.809454  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:11.810245  5555 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:28:11.820312 13370 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:11.766561 13960 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:11.811229  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:11.795979 14021 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:11.773849 13895 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:11.819078  6272 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:11.742226 14008 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.774935 13953 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:11.789681 13897 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:11.811041 14045 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:11.763653 13902 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.057s
I20260709 12:28:11.825834  5555 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:28:11.806710 13812 consensus_queue.cc:1243] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 24.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:28:11.745697 13743 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.791611 13999 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:11.830461  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:11.811425  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 96466.6us
I20260709 12:28:11.822899 13506 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:11.821555 13370 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:11.808149 14049 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:11.796372 14016 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.031s
I20260709 12:28:11.821900 13322 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:11.828938 14030 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:28:11.848722 14063 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:28:11.827231 14021 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:11.832830 13899 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:11.858073  5524 leader_election.cc:400] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19.
I20260709 12:28:11.851369  5704 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:11.861593  5524 leader_election.cc:403] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:28:11.861479  5630 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
W20260709 12:28:11.862998  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61334.4us
I20260709 12:28:11.860978 14063 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:11.865525  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:11.865916 14063 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:11.866614 13895 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 18. Current term is 19.
I20260709 12:28:11.867492  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:11.868902  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:11.869663 14049 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:11.870350 14049 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:11.858819 13322 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:11.870847  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:11.870947 13322 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:11.871040 14049 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:28:11.841351 14045 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.871826 14049 consensus_queue.cc:1243] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 22.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:28:11.872592 14038 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 12:28:11.873432 14045 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:11.850167 13812 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.024s
I20260709 12:28:11.873314 13895 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:28:11.874915 14050 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:11.875409 14050 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:11.877544 14050 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:11.877604  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:11.878302  6276 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:11.879215  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:11.880764  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:11.881428  6280 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:11.883958  5632 raft_consensus.cc:2410] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:28:11.841918 13996 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:11.909554  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:11.851343 13897 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:11.913479  5627 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:11.914513  6280 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:11.924616 13322 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.916438  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:11.889612 14030 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:11.906950  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:11.883286 13899 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:11.925752  5633 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20260709 12:28:11.928637  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 239401us
I20260709 12:28:11.911273 14016 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:11.908715 13996 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:11.874486 14056 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:11.933518  6272 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
W20260709 12:28:11.932934  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 138317us
I20260709 12:28:11.937319  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:11.917271 13895 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:28:11.944690 14070 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:11.946184  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:11.940235  6276 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:11.929085 13946 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:41448 (request call id 0) took 17976 ms. Trace:
I20260709 12:28:11.969290 14056 raft_consensus.cc:2764] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:28:11.939814 13899 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:11.951318 13322 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:11.939636 13992 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:11.955653 14016 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:11.957078 14048 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:11.957808 13909 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:11.958624 14047 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:28:11.934185 13968 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:11.947969 13996 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:11.935437 14061 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:28:11.999560 14016 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:11.984731  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3021) took 1474 ms. Trace:
W20260709 12:28:11.988420  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 79006.6us
I20260709 12:28:11.951397  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:11.973176 14045 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:12.003485  5634 rpcz_store.cc:276] 0709 12:28:10.509961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:10.627463 (+117502us) service_pool.cc:224] Handling call
0709 12:28:10.627820 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:10.872561 (+244741us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:11.984456 (+1111895us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b5400438854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:11.984663 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":395616128}
I20260709 12:28:12.018994  6275 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
W20260709 12:28:12.020521  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 70431.8us
I20260709 12:28:12.023532  5675 leader_election.cc:400] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23.
I20260709 12:28:12.023986  5675 leader_election.cc:403] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:28:12.025645  5703 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:11.977916 13992 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:11.935647 13953 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 25.12, Last appended by leader: 12, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.037060  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:12.041745  6280 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:28:12.042356  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:12.052312  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:12.053607  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:11.992604 14044 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:11.932502 13999 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.065476  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:12.059103 14031 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:12.038090 13506 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23.
I20260709 12:28:12.039861 13992 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:12.073822 13322 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 12, Time since last communication: 0.004s
I20260709 12:28:12.073660 14031 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:12.036991 14045 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:12.085839 13983 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.424s	user 0.003s	sys 0.003s
I20260709 12:28:12.089422 13909 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:12.094256  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1648) took 1104 ms. Trace:
I20260709 12:28:12.089293 14045 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:12.098502  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1358) took 1019 ms. Trace:
I20260709 12:28:12.094810  5561 rpcz_store.cc:276] 0709 12:28:10.989954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:10.992264 (+  2310us) service_pool.cc:224] Handling call
0709 12:28:10.994208 (+  1944us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:11.024124 (+ 29916us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:11.024169 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:11.026758 (+  2589us) log.cc:844] Serialized 26 byte log entry
0709 12:28:11.283728 (+256970us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:11.283786 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:11.674135 (+390349us) spinlock_profiling.cc:243] Waited 5.54 ms on lock 0x7b480027e958. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:28:12.093810 (+419675us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b540044dc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:12.093883 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:12.093944 (+    61us) raft_consensus.cc:1685] finished
0709 12:28:12.093960 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:12.094173 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5719,"raft.run_cpu_time_us":3074,"raft.run_wall_time_us":643781,"spinlock_wait_cycles":80580608,"thread_start_us":256580,"threads_started":1,"wal-append.queue_time_us":396874}
I20260709 12:28:12.101166 14049 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:12.101680 14049 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:12.102670 14049 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.004997 13968 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.106145  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:12.098963  6271 rpcz_store.cc:276] 0709 12:28:11.078727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:11.079345 (+   618us) service_pool.cc:224] Handling call
0709 12:28:11.079674 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:11.083557 (+  3883us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:11.083588 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:11.084015 (+   427us) log.cc:844] Serialized 26 byte log entry
0709 12:28:12.097315 (+1013300us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b54001e8aa0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5dc090 00007f4ffa5e96b8 00007f4ffa67292b 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c
0709 12:28:12.097405 (+    90us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:12.097801 (+   396us) log.cc:844] Serialized 14 byte log entry
0709 12:28:12.098055 (+   254us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:12.098140 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:12.098182 (+    42us) raft_consensus.cc:1685] finished
0709 12:28:12.098278 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:12.098430 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4208,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":4409,"spinlock_wait_cycles":51127168,"thread_start_us":267571,"threads_started":1,"wal-append.queue_time_us":265078}
I20260709 12:28:12.072588 14008 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:12.072953 14062 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:12.074512 14057 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:12.076445 13743 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:12.123375  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:12.124783  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:12.125767  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:12.126533  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:12.130434  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:12.131022  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:12.131805  5560 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:12.043110 14070 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.063s
I20260709 12:28:12.136711  5553 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:12.139333  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:12.140883  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:12.137349  5553 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:12.142722  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:12.143388  5709 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:12.143348  5708 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:12.104532 14049 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:12.149101  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:12.151597  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:12.143340  5553 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.026793 13897 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.087857 14031 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.157649  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:12.158788  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 105368us
I20260709 12:28:12.185042  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:12.155202  5553 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:12.157462 14045 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:28:12.154639 13861 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:28:12.164054 13985 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.663s	user 0.008s	sys 0.008s
I20260709 12:28:12.164466 13900 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:28:12.165469 13968 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:11.966476 13946 rpcz_store.cc:276] 0709 12:27:53.944844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:54.696647 (+751803us) service_pool.cc:224] Handling call
0709 12:27:55.189833 (+493186us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b54001db620. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5ef520 00007f4ffa678b42 00007f4ffa677800 00007f4ffa6709fa 00007f4fff6384c9 00007f4ff8bb4c54 00007f4ff8bb4be2 00007f4ff8bb4b11 00007f4ff8bb4a8c 00007f4ff8bb3d42
0709 12:27:55.301233 (+111400us) log.cc:844] Serialized 480 byte log entry
0709 12:27:55.734115 (+432882us) spinlock_profiling.cc:243] Waited 1.39 ms on lock 0x7b18003ff820. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa5cbb0b 00007f4ffa650b63 00007f4ffa678752 00007f4ffa6709fa
0709 12:28:11.910711 (+16176596us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":216010,"raft.queue_time_us":226541,"raft.run_cpu_time_us":1507,"raft.run_wall_time_us":9509,"spinlock_wait_cycles":11767040,"thread_start_us":204466,"threads_started":1,"wal-append.queue_time_us":368772,"wal-append.run_cpu_time_us":3918,"wal-append.run_wall_time_us":1274600}
I20260709 12:28:12.172765  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:12.129709 13909 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:12.157152 13875 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26.
I20260709 12:28:12.104539 14021 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.191005  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:12.191304  5709 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:12.199559  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:12.200306  5561 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:12.192986  5560 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:12.059746 14030 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.201323  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:12.153964 14008 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:28:12.204984  5553 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 23 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 12:28:12.207299 13861 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:12.213236  5561 raft_consensus.cc:2376] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26. Re-sending same reply.
I20260709 12:28:12.201546  5560 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:58854
I20260709 12:28:12.214509  5709 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 25 index: 19. (term mismatch)
I20260709 12:28:12.216724  5560 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:12.217434  5560 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:12.216892  5709 pending_rounds.cc:85] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 18
I20260709 12:28:12.220332  5709 pending_rounds.cc:107] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 24.19
I20260709 12:28:12.139971 14062 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.144215 14057 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.195592 14061 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:28:12.230892  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 67754.3us
I20260709 12:28:12.181372 13506 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.033s
I20260709 12:28:12.234853  5709 raft_consensus.cc:2889] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:12.213330 13968 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:12.228085 13861 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.240728  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3296) took 1131 ms. Trace:
W20260709 12:28:12.250839  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:12.265808  5709 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
W20260709 12:28:12.266641  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:12.268139  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:12.250110  5560 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:12.252641 13951 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260709 12:28:12.214011 14065 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:12.262274  6280 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:28:12.265920  5707 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:12.214792 13909 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:12.274605  6279 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:12.271886  5553 rpcz_store.cc:276] 0709 12:28:11.109159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:11.183913 (+ 74754us) service_pool.cc:224] Handling call
0709 12:28:11.184163 (+   250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:12.136600 (+952437us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b54002fc9a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa6741c0 00007f4ffa675231 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c
0709 12:28:12.215352 (+ 78752us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:12.240665 (+ 25313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105515392}
I20260709 12:28:12.257969 13900 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:12.281150 14080 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:28:12.281508 13900 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.282960  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1701) took 1056 ms. Trace:
I20260709 12:28:12.280458 13947 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:28:12.281767 14068 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 12:28:12.283700 14062 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:12.265362 14071 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:12.283708 13900 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:12.283360  5636 rpcz_store.cc:276] 0709 12:28:11.226888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:11.227107 (+   219us) service_pool.cc:224] Handling call
0709 12:28:11.255864 (+ 28757us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:11.258206 (+  2342us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:28:11.405231 (+147025us) log.cc:844] Serialized 14 byte log entry
0709 12:28:12.091240 (+686009us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b1800257920. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:28:12.147819 (+ 56579us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:12.155144 (+  7325us) log.cc:844] Serialized 26 byte log entry
0709 12:28:12.155279 (+   135us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:12.155312 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:12.155389 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:12.174572 (+ 19183us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b1800257920. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:28:12.282744 (+108172us) raft_consensus.cc:1685] finished
0709 12:28:12.282767 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:12.282902 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":158745,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":11062,"raft.queue_time_us":6074,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":11851,"spinlock_wait_cycles":40428288,"thread_start_us":174485,"threads_started":2,"wal-append.queue_time_us":727393}
I20260709 12:28:12.288719  5553 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:28:12.290616 14016 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:28:12.292171 14063 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.294836 14063 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:12.302619  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:12.307049  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3456) took 3565 ms. Trace:
I20260709 12:28:12.308302  5709 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260709 12:28:12.307734  5706 rpcz_store.cc:276] 0709 12:28:08.742142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:09.890902 (+1148760us) service_pool.cc:224] Handling call
0709 12:28:09.958922 (+ 68020us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:11.498817 (+1539895us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:12.306781 (+807964us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b54002f5154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:12.306986 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3527556352}
W20260709 12:28:12.324606  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:12.271278 13370 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.279630 14061 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:28:12.283841 13909 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:12.285969 13861 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:12.351557  5707 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:12.311673 14065 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:12.373517 14071 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:12.352015 13945 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:12.352097 14032 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:12.388019 14061 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:12.335840  5706 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:12.337298  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:12.402483  5706 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:12.403123  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 65929.8us
I20260709 12:28:12.333900 13953 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:12.443917  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:12.403337  5706 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.446782  5706 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:12.448952  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:12.442657 14085 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
W20260709 12:28:12.443111 14079 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:28:12.343096 14057 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:12.449286 14081 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 12:28:12.443327 14013 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:12.443105 14070 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:12.408622 13945 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:12.335357 13996 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.448700 13817 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 12:28:12.472981 14013 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:12.449129 13953 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:12.478624 14077 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.180s	user 0.002s	sys 0.000s
I20260709 12:28:12.480250 13992 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.466295  5706 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 16. (term mismatch)
I20260709 12:28:12.443042 14032 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.494489 14030 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:12.485066 13899 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.500110 14065 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.501524 14061 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.502569 13953 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:12.504658 13996 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:12.505335  6272 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 24: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:12.486758 13370 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:12.492599 14013 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:12.506711  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:12.508522 13945 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.509259  5703 pending_rounds.cc:85] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 13
I20260709 12:28:12.475656 14070 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.512784  6271 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: [28.21-28.21]
I20260709 12:28:12.515034  5706 pending_rounds.cc:85] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 15
I20260709 12:28:12.522413 13899 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:12.523074  5705 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:12.532225  5704 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:12.538748  5704 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:28:12.533382  5703 pending_rounds.cc:107] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 22.14
W20260709 12:28:12.512465  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1467) took 1813 ms (client timeout 2169 ms). Trace:
I20260709 12:28:12.535874  5706 pending_rounds.cc:107] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 23.16
W20260709 12:28:12.536962  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:12.541674  5703 raft_consensus.cc:2889] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:28:12.541730  5702 rpcz_store.cc:269] 0709 12:28:10.698660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:10.699141 (+   481us) service_pool.cc:224] Handling call
0709 12:28:12.512194 (+1813053us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b5400067054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:12.512397 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":937499008}
I20260709 12:28:12.543573  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:12.541893  5706 raft_consensus.cc:2889] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:12.580221 14032 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:12.519608 14030 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.584566  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:12.543694 14021 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:12.552807 13945 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:12.604264  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 155507us
I20260709 12:28:12.592756 13782 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:12.598863  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:12.613427  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:12.594691 13996 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:12.595343 14058 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.022s
W20260709 12:28:12.629002 14087 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20260709 12:28:12.630015 14084 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 12:28:12.609470 13782 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:12.640827 14021 raft_consensus.cc:993] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:28:12.641757 14021 raft_consensus.cc:1081] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Signalling peer f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:28:12.607187 13870 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 12:28:12.646256 14033 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:12.622906 13996 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.652072 13999 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:12.652863 13996 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:12.652719 13999 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.635079 13909 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:12.653764 13909 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.654940 13999 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:12.655907 13909 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:12.617061 14070 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:12.636602  5709 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 17. (term mismatch)
I20260709 12:28:12.659919  5709 pending_rounds.cc:85] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 16
I20260709 12:28:12.660244  5709 pending_rounds.cc:107] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 24.17
I20260709 12:28:12.660512  5709 raft_consensus.cc:2889] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:12.660847 13782 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.661875 13782 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:12.662783  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3022) took 2193 ms. Trace:
I20260709 12:28:12.663161  5559 rpcz_store.cc:276] 0709 12:28:10.468940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:10.583326 (+114386us) service_pool.cc:224] Handling call
0709 12:28:10.590866 (+  7540us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:11.557601 (+966735us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:12.662555 (+1104954us) spinlock_profiling.cc:243] Waited 950 ms on lock 0x7b540023fd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:12.662722 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2185024384}
I20260709 12:28:12.664505  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:12.665709  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:12.666603  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:12.669253  6279 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:28:12.671630  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:12.677867  6279 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:12.696146  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:12.695719  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:12.701642  6273 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 26: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:12.701958  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:12.706064  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:12.708956  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:12.738077  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:12.738862  5709 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:12.689082 14050 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:12.695667 13953 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:12.745538 14050 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:12.746193 14050 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:12.749956  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:12.708140 14070 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.043s
I20260709 12:28:12.751291  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:12.752980 14030 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:12.708009 13506 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:12.755383 13506 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.756690 13992 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:12.757853 13506 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:12.679198 13909 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:12.778651 13909 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.698319 13875 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.083s
I20260709 12:28:12.744138 13861 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:12.677016 13871 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:12.751382 14049 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.597s	user 0.006s	sys 0.006s
I20260709 12:28:12.743543 13870 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:12.746078 13953 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:12.772156 14021 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.010s
I20260709 12:28:12.698731 14013 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.742770  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:12.829450  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 86766.1us
I20260709 12:28:12.832541  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:12.840122  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:12.842460  5709 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:12.791424  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:12.846048  5555 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:12.860668  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:12.868206  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:12.875002  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:12.967691 14061 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:12.875831  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:12.968580 14061 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.880851  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:12.970091  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 89424.2us
I20260709 12:28:12.971438 13866 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:12.972035 13866 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:12.972182 14061 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:12.973143 13866 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.881269  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:12.821331 13942 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:12.981892 13942 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:12.836272 13872 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:12.989387 13942 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:12.877240 14021 consensus_queue.cc:1243] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 23.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20260709 12:28:12.904666  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:12.811365 13871 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:12.991587  6279 raft_consensus.cc:2955] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Committing config change with OpId 15.11: config changed from index 3 to 11, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:28:12.995528  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.098s (SENT) [suppressed 31 similar messages]
I20260709 12:28:12.811615 13861 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.004033 13861 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:12.822513 13870 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.008934 13870 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:12.963553  5554 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:13.009935 14021 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:13.010546 14021 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.012732 14021 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:13.014281 13871 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:13.017869  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:13.026072  5709 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
W20260709 12:28:12.963490  5552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3020) took 2526 ms (client timeout 1626 ms). Trace:
I20260709 12:28:12.901623 13506 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:13.028837 13506 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.032697 13506 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:13.020119  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:13.040114  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:28:13.033986  5552 rpcz_store.cc:269] 0709 12:28:10.436852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:10.445023 (+  8171us) service_pool.cc:224] Handling call
0709 12:28:12.963151 (+2518128us) spinlock_profiling.cc:243] Waited 561 ms on lock 0x7b54002288d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:12.963375 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1291173120}
I20260709 12:28:13.072927  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:12.934729 13909 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:12.915905 13953 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.038981 13812 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:13.043987  5555 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:13.072710  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:13.078992  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1660) took 1140 ms. Trace:
I20260709 12:28:13.073062  6274 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:13.080533  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:13.079365  5561 rpcz_store.cc:276] 0709 12:28:11.938551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:12.216161 (+277610us) service_pool.cc:224] Handling call
0709 12:28:12.219988 (+  3827us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:12.285237 (+ 65249us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:28:12.301252 (+ 16015us) log.cc:844] Serialized 14 byte log entry
0709 12:28:12.457300 (+156048us) log.cc:844] Serialized 14 byte log entry
0709 12:28:12.457996 (+   696us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:12.458343 (+   347us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:12.458447 (+   104us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:12.578102 (+119655us) spinlock_profiling.cc:243] Waited 9.77 ms on lock 0x7b180027dd20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:28:12.754041 (+175939us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b180027dd20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:28:13.078742 (+324701us) spinlock_profiling.cc:243] Waited 65 ms on lock 0x7b5400082854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:13.078787 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:13.078936 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15643,"prepare.queue_time_us":20134,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":80,"raft.queue_time_us":336,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":18540,"spinlock_wait_cycles":177262464,"thread_start_us":153017,"threads_started":1,"wal-append.queue_time_us":230917}
I20260709 12:28:13.084299  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:13.066154  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70663.1us
I20260709 12:28:13.085919  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:13.086565  5701 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 18.
I20260709 12:28:13.087785  5701 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:28:13.073824  5709 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:13.090193  6280 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:13.087765  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:13.091583  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:13.047124 13871 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:13.085218 14021 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:13.103948  5710 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:13.104231 14021 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:13.068580 14013 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:13.105299 14021 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.105877 13998 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:28:13.108610 13945 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.501s	user 0.008s	sys 0.004s
I20260709 12:28:13.080554 13861 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:13.116384  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:13.091153  5555 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:13.091171  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:13.117801  5559 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 24: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:28:13.100515  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:13.114530  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:13.115366  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:13.091456  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:13.118821  5561 raft_consensus.cc:2410] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:28:13.120549  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:13.122668  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1654) took 1567 ms. Trace:
I20260709 12:28:13.122979  5709 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:13.123052  5557 rpcz_store.cc:276] 0709 12:28:11.555434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:11.590779 (+ 35345us) service_pool.cc:224] Handling call
0709 12:28:11.606268 (+ 15489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:12.155628 (+549360us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:12.155660 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:12.155698 (+    38us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:12.155729 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:13.122449 (+966720us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b54001c57d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:13.122498 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:13.122613 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1262938880}
I20260709 12:28:13.121192  5559 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:13.127830  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:13.131151  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:13.136868 14101 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.203s	user 0.001s	sys 0.000s
I20260709 12:28:13.137521 13939 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:13.115798 13871 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:13.148939  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:13.150219  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:13.147080  5559 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:28:13.085440 13812 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.154361 13782 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.010s
I20260709 12:28:13.155675 14065 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:13.155669 13909 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:13.166175 13909 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:13.166757 13909 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:13.155010 13871 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:28:13.179641 14046 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260709 12:28:13.121474 13861 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:13.194173 14096 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.273s	user 0.000s	sys 0.004s
I20260709 12:28:13.212826 14092 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:13.222589 14092 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.225085 14092 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:13.226589 13812 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:13.238562  5706 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
W20260709 12:28:13.242796  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:13.246651  5560 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:13.247231  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:58854 (request call id 1659) took 1309 ms. Trace:
I20260709 12:28:13.247485  5560 rpcz_store.cc:276] 0709 12:28:11.938238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:12.201429 (+263191us) service_pool.cc:224] Handling call
0709 12:28:13.230376 (+1028947us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b240013fad8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff7408cb9 00007f4ff7406dcb 00007f4ff75052b1 00007f4ff7500848 00007f4ff75007ad 00007f4ff89419a7 00007f4ffa5ce85c 00007f4ffa5ce699 00007f4ffa61f012 00007f4ffa669a55
0709 12:28:13.247180 (+ 16804us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3887488}
I20260709 12:28:13.254225  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1652) took 1699 ms. Trace:
I20260709 12:28:13.254678 13861 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:13.254293  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3039) took 2591 ms. Trace:
I20260709 12:28:13.254725  5556 rpcz_store.cc:276] 0709 12:28:11.554776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:11.554894 (+   118us) service_pool.cc:224] Handling call
0709 12:28:13.253403 (+1698509us) spinlock_profiling.cc:243] Waited 5.52 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:28:13.253620 (+   217us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:13.253911 (+   291us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:13.253936 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:13.253976 (+    40us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:13.254007 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:13.254062 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:13.254169 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12706560}
I20260709 12:28:13.256232  5554 rpcz_store.cc:276] 0709 12:28:10.663344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:10.722338 (+ 58994us) service_pool.cc:224] Handling call
0709 12:28:12.963332 (+2240994us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:13.009739 (+ 46407us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:13.254099 (+244360us) spinlock_profiling.cc:243] Waited 2.22 s on lock 0x7b54002288d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:13.254217 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5107117952}
I20260709 12:28:13.275995  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:13.277168  5556 raft_consensus.cc:2368] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25.
W20260709 12:28:13.278347  6277 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1359) took 1698 ms (client timeout 2119 ms). Trace:
I20260709 12:28:13.282836  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3025) took 1555 ms. Trace:
W20260709 12:28:13.278906  6277 rpcz_store.cc:269] 0709 12:28:11.580133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:11.580418 (+   285us) service_pool.cc:224] Handling call
0709 12:28:13.278126 (+1697708us) spinlock_profiling.cc:243] Waited 50.3 ms on lock 0x7b54002e3954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:13.278272 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":115632000}
I20260709 12:28:13.283473  5630 rpcz_store.cc:276] 0709 12:28:11.727262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:11.727413 (+   151us) service_pool.cc:224] Handling call
0709 12:28:13.282578 (+1555165us) spinlock_profiling.cc:243] Waited 95.1 ms on lock 0x7b54001de7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:13.282757 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":218753920}
W20260709 12:28:13.376330  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:13.387436  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:13.426072  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:13.447531  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 153034us
I20260709 12:28:13.447713  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:13.443638  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:13.458050  5560 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:28:13.459874  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1650) took 2456 ms. Trace:
I20260709 12:28:13.458033 13953 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:13.470892 13953 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:13.472736 13627 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:13.473626 13627 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.442381 13872 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:13.476543 14013 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:13.476989 13872 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.477370 14013 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.446821 13861 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 12:28:13.483436 14013 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:13.441956 13950 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:13.484320 13950 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.478039 14046 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
W20260709 12:28:13.510680  5602 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.4.65.62:38921, user_credentials={real_user=slave}} blocked reactor thread for 51141.2us
W20260709 12:28:13.511617  5602 leader_election.cc:336] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.626s (SENT)
I20260709 12:28:13.486828 13950 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:13.293500 13960 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:13.460261  5558 rpcz_store.cc:276] 0709 12:28:11.003999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:11.004228 (+   229us) service_pool.cc:224] Handling call
0709 12:28:11.004775 (+   547us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:11.022976 (+ 18201us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:28:11.074331 (+ 51355us) log.cc:844] Serialized 14 byte log entry
0709 12:28:11.258308 (+183977us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:11.266590 (+  8282us) log.cc:844] Serialized 26 byte log entry
0709 12:28:11.266744 (+   154us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:11.266778 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:12.957888 (+1691110us) spinlock_profiling.cc:243] Waited 1.98 ms on lock 0x7b18003c7f00. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419
0709 12:28:13.459615 (+501727us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b54000267d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:13.459666 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:13.459715 (+    49us) raft_consensus.cc:1685] finished
0709 12:28:13.459725 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:13.459829 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":103083,"prepare.queue_time_us":114671,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":84077,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":14376,"spinlock_wait_cycles":45338880,"thread_start_us":112864,"threads_started":1,"wal-append.queue_time_us":190220,"wal-append.run_cpu_time_us":14991,"wal-append.run_wall_time_us":1258547}
W20260709 12:28:13.478955 14050 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.732s	user 0.008s	sys 0.005s
I20260709 12:28:13.483817 13871 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.442749 13998 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:13.442669 13771 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:13.479302 14092 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:13.543653 13743 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:13.547945  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:13.587551  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:13.603560  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3173) took 1481 ms. Trace:
I20260709 12:28:13.604425  5707 rpcz_store.cc:276] 0709 12:28:12.121988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:12.125423 (+  3435us) service_pool.cc:224] Handling call
0709 12:28:13.603384 (+1477961us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54001991d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:13.603497 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":311419648}
I20260709 12:28:13.606206  5707 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:28:13.608822  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:13.550414 13627 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:13.610304  6278 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: []
W20260709 12:28:13.617590  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 155330us
I20260709 12:28:13.617897  5707 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:28:13.611989  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3185) took 1477 ms. Trace:
W20260709 12:28:13.633060 14095 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.715s	user 0.000s	sys 0.008s
I20260709 12:28:13.637630 13872 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:13.639277 13871 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:13.549829 13960 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:13.566425 13998 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.646206  5703 rpcz_store.cc:276] 0709 12:28:12.134816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:12.508632 (+373816us) service_pool.cc:224] Handling call
0709 12:28:12.509015 (+   383us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:12.550663 (+ 41648us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:12.550698 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:12.554401 (+  3703us) log.cc:844] Serialized 52 byte log entry
0709 12:28:12.666460 (+112059us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:12.705094 (+ 38634us) log.cc:844] Serialized 14 byte log entry
0709 12:28:13.577636 (+872542us) spinlock_profiling.cc:243] Waited 79.6 ms on lock 0x7b180005db20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:28:13.603118 (+ 25482us) log.cc:844] Serialized 14 byte log entry
0709 12:28:13.603414 (+   296us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:13.603492 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:13.611784 (+  8292us) raft_consensus.cc:1685] finished
0709 12:28:13.611804 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:13.611938 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":871279,"prepare.queue_time_us":930420,"prepare.run_cpu_time_us":242,"prepare.run_wall_time_us":241,"raft.queue_time_us":34676,"raft.run_cpu_time_us":391,"raft.run_wall_time_us":53333,"spinlock_wait_cycles":183117696,"thread_start_us":57864,"threads_started":1,"wal-append.queue_time_us":296028}
I20260709 12:28:13.673180  5707 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:13.571158 13771 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.687372 13960 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:13.562116 14050 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:13.667774 13939 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.530s	user 0.012s	sys 0.001s
I20260709 12:28:13.660931 13951 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:13.600606 13909 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.653870  5561 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 27 index: 13. (index mismatch)
I20260709 12:28:13.572770 14092 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:13.611282 13743 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:13.684015 13998 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:13.745947  5702 raft_consensus.cc:2410] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 27 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 16.
I20260709 12:28:13.746028  5707 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
W20260709 12:28:13.752329  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 229186us
W20260709 12:28:13.662995  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 97089.7us
W20260709 12:28:13.824054  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:13.835538  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1476) took 1215 ms. Trace:
I20260709 12:28:13.842252  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:13.845719 13871 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:13.850286  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:13.816051 14114 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:28:13.858795  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1663) took 1613 ms. Trace:
I20260709 12:28:13.859225  5553 rpcz_store.cc:276] 0709 12:28:12.245683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:12.519800 (+274117us) service_pool.cc:224] Handling call
0709 12:28:12.520178 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:12.609392 (+ 89214us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:12.609422 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:12.609462 (+    40us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:12.609491 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:13.858540 (+1249049us) spinlock_profiling.cc:243] Waited 89 ms on lock 0x7b54000eb0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:13.858604 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:13.858736 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":204750208}
I20260709 12:28:13.864902  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3193) took 1207 ms. Trace:
I20260709 12:28:13.865413  5710 rpcz_store.cc:276] 0709 12:28:12.657658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:13.106381 (+448723us) service_pool.cc:224] Handling call
0709 12:28:13.106726 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:13.207970 (+101244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:13.208017 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:13.208058 (+    41us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:13.208087 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:13.864573 (+656486us) spinlock_profiling.cc:243] Waited 98.7 ms on lock 0x7b54004529d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:13.864659 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:13.864826 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":227003136}
I20260709 12:28:13.846411 13960 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:13.875133  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3187) took 1618 ms. Trace:
I20260709 12:28:13.876467  5704 rpcz_store.cc:276] 0709 12:28:12.256880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:12.539631 (+282751us) service_pool.cc:224] Handling call
0709 12:28:12.539938 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:12.540397 (+   459us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:12.540430 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:12.541102 (+   672us) log.cc:844] Serialized 26 byte log entry
0709 12:28:12.836231 (+295129us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:12.836316 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:12.836473 (+   157us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:13.607480 (+771007us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:13.861050 (+253570us) raft_consensus.cc:1685] finished
0709 12:28:13.861071 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:13.875054 (+ 13983us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":201,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"spinlock_wait_cycles":6432000,"thread_start_us":292645,"threads_started":1,"wal-append.queue_time_us":357292}
I20260709 12:28:13.891454  5707 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:13.836139  5701 rpcz_store.cc:276] 0709 12:28:12.619793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:13.088918 (+469125us) service_pool.cc:224] Handling call
0709 12:28:13.089199 (+   281us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:13.089961 (+   762us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:13.089994 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:13.102698 (+ 12704us) log.cc:844] Serialized 26 byte log entry
0709 12:28:13.216580 (+113882us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:13.834859 (+618279us) log.cc:844] Serialized 14 byte log entry
0709 12:28:13.835206 (+   347us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:13.835284 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:13.835326 (+    42us) raft_consensus.cc:1685] finished
0709 12:28:13.835337 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:13.835472 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":402908,"prepare.queue_time_us":403927,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":439,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":20598,"thread_start_us":309330,"threads_started":2,"wal-append.queue_time_us":189117}
I20260709 12:28:13.892910  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:13.851276 13601 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:13.851600 13953 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:13.895239  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:13.855965 14046 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:13.894742  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:13.858497 13939 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:13.848940 13871 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.853919 13861 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:13.851157 13899 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:13.903342  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:13.863888 13812 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:13.914917  5704 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:13.906618  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:13.907686  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:13.896340 13872 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:13.927201 13872 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.929425 13872 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:13.760799 13951 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.931413 13951 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:13.920984  5710 raft_consensus.cc:2368] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25.
I20260709 12:28:13.934405  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:13.926288  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:13.904950 13601 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:13.938464  5710 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:13.941414  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:13.931914 13951 raft_consensus.cc:476] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Not starting pre-election -- already a leader
I20260709 12:28:13.917958 13812 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:13.901376 13960 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.947666  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:13.948439  5707 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:13.949616  5704 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:13.954034 13601 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:13.954595 13601 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:13.954811 13871 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:13.955040 13601 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:13.944221  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:13.958863  5710 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:13.957654 13909 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:13.963279 14050 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:13.964543 13743 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:13.964701 14092 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:13.965646 13771 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:13.934044 13899 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.911350 13939 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.968466 13875 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:13.969127 13875 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:13.969214 13601 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:13.970093 13939 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:13.970814 14099 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.051s	user 0.004s	sys 0.004s
I20260709 12:28:13.946588 13951 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:13.972767  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:13.970110 13875 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.917178 13861 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:13.972180 13951 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:13.977537  5710 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:28:13.977366  5707 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:13.978005  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:13.910367 14046 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:13.988771 14046 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:13.989956  5701 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 28 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:28:13.991508  5701 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 29 index: 21. (term mismatch)
I20260709 12:28:13.991927  5701 pending_rounds.cc:85] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 20
I20260709 12:28:13.992230  5701 pending_rounds.cc:107] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 28.21
I20260709 12:28:13.992624  5701 raft_consensus.cc:2889] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:13.994706  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:13.995935 14104 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.883s	user 0.003s	sys 0.005s
I20260709 12:28:13.996188  5704 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:14.000002  5703 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:38468
I20260709 12:28:14.001981  5703 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:14.002404  5703 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:13.968111 13899 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:13.958844 13812 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:28:14.005232  5602 connection.cc:256] RPC call timeout handler was delayed by -1.129950178999934s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1676s for a response
I20260709 12:28:13.998667  5704 raft_consensus.cc:2955] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Committing config change with OpId 15.11: config changed from index 3 to 11, VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:14.004038 13875 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:14.021263 14050 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.019961 13743 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.022058 14050 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.022281 13743 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.024595 13743 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:14.025586 13951 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:14.016934 14092 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:14.026165 13951 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:14.026543 14092 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.027627 14050 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:14.026659 13951 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.013896 13843 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.028865 13843 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.030253 13951 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:14.030855 13843 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:14.031059  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:14.031621 14050 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:14.031952 14092 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:14.032219 14050 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.034705  5630 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 23 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 12:28:14.037544 14050 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:14.040931  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:14.042236  5557 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.784s	user 0.033s	sys 0.079s
W20260709 12:28:14.042590  5557 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.033s	sys 0.079s
I20260709 12:28:14.043213  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.043623 13861 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:14.043571  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1364) took 1879 ms. Trace:
W20260709 12:28:14.044538  5705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3176) took 1921 ms (client timeout 1963 ms). Trace:
W20260709 12:28:14.044940  5705 rpcz_store.cc:269] 0709 12:28:12.122879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:12.122997 (+   118us) service_pool.cc:224] Handling call
0709 12:28:14.044320 (+1921323us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x7b54002d23d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:14.044475 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":903559424}
I20260709 12:28:14.045468 13960 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:14.046146  5701 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:28:14.047962  5445 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 26edf338d41e42f1a5ce0d4416ec67e1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:28:14.048645 13880 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:28:14.044275  6271 rpcz_store.cc:276] 0709 12:28:12.164184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:12.165080 (+   896us) service_pool.cc:224] Handling call
0709 12:28:12.512461 (+347381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:12.537814 (+ 25353us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:14.043379 (+1505565us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b5400591ad0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:14.043502 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":798057728}
I20260709 12:28:14.055975 13843 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.056067 14092 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:14.056730 13843 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.056854 14092 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.059015 13843 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:14.059158 14092 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:14.060585 14043 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:14.060814  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3329) took 1250 ms. Trace:
I20260709 12:28:14.061439  5557 rpcz_store.cc:276] 0709 12:28:12.810872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:13.217431 (+406559us) service_pool.cc:224] Handling call
0709 12:28:13.217791 (+   360us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:13.218300 (+   509us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:13.218345 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:13.234343 (+ 15998us) log.cc:844] Serialized 52 byte log entry
0709 12:28:14.060373 (+826030us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:14.060432 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:14.060516 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:14.060567 (+    51us) raft_consensus.cc:1685] finished
0709 12:28:14.060579 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:14.060742 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":111,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":101,"thread_start_us":803550,"threads_started":1,"wal-append.queue_time_us":428137}
I20260709 12:28:12.588064  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:14.063808  5635 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:14.065295  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:14.065908  5630 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:14.069176  5635 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:14.072731  5635 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: [28.21-28.21]
I20260709 12:28:14.073252  5703 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:14.075048  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:14.017184 14125 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:14.077693  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.085343  5635 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
W20260709 12:28:14.079841  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:14.086196  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:14.081955  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.086957  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:14.083593  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:14.087706  5629 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:14.088001  5636 raft_consensus.cc:2410] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:28:14.094429  5710 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:14.097582  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.100546  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:14.101310  5633 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:14.100941  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.102872  5635 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:14.104038  5703 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:14.107519  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.108083  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.108693  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.108940  5635 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:14.109555  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:14.113603  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.114773  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:14.084198 13843 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.119441  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:14.122210  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:14.138281  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.139411  5634 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:28:14.142846  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.143966  5634 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:14.145221  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.145892  5634 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:14.146754  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.141649  5630 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:14.149502  5629 raft_consensus.cc:2410] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 22 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:28:14.150655  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.148947  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.161103  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.161211  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:14.150949  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:14.164171  5634 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:14.164116  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:14.165674  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.167165  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:14.151599  5635 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:28:14.170959  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:14.171708  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:14.145032 14050 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:14.182663  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.177007  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:14.184387  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:14.185472  5632 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 24: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:14.172879 14126 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:14.186141 14126 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:14.186558 14126 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:14.186651  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.173053 14055 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:14.107923 14125 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.187808 14013 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:28:14.187765 14055 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.188305  6236 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:14.189076 13627 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:28:14.189451 13627 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:14.189508 14000 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:14.189989 14055 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:14.189671  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.190537  5630 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:14.179962 13945 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:14.187830  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.191139 13945 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:14.191601  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.191687 13945 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:14.193286  5632 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:28:14.194828  5632 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:28:14.190483 13627 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.197774 13627 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:14.197988  5630 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:14.180711  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.200537  5634 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:14.203577  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:14.203874  5634 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:14.204520 14055 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:14.204967 14055 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:14.205173  5634 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:14.205377 14055 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:14.199833 13945 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.210239 13945 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:14.210956 13945 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:14.211367 13945 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:14.211787 13945 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:14.214926 14056 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:14.215529 14056 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.219558 14056 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:14.222134 14055 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.136482 13843 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.226004 13960 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:14.226501 13960 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:14.226848 13960 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:14.227681 13843 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:14.169392 14070 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.059s
I20260709 12:28:14.229206 14070 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:14.229620 14070 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:14.230562 14070 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.236940 14070 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:14.238638  5636 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
W20260709 12:28:14.247541  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 63258us
I20260709 12:28:14.248597  5633 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:14.249472  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:28:14.253882  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:14.180322 13950 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:14.305357 14125 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:14.307507  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:14.279733 13950 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:14.313014 13950 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:28:14.332682  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:14.336920 13755 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:14.337787 13755 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.341364 13755 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:14.341694 13950 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.250227 14133 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:14.256747 14130 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:14.366561  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:14.367883 14055 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:14.369081  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:14.388845  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 192528us
I20260709 12:28:14.354897 14130 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:14.396178  5552 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:14.389478 13950 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:14.400116  5602 leader_election.cc:400] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25.
I20260709 12:28:14.411289  5602 leader_election.cc:403] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:28:14.398401 14130 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:14.403928 14098 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.441s	user 0.006s	sys 0.008s
I20260709 12:28:14.400481 14125 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.005s
I20260709 12:28:14.397725 14049 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.406330 13951 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:14.431324  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:14.458953  5636 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:14.402045 13950 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:28:14.467469  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52380.3us
W20260709 12:28:14.483161  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:14.440824  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:14.475313  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1367) took 1644 ms. Trace:
I20260709 12:28:14.470584  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:14.419564 13743 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25.
W20260709 12:28:14.504420  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63731.9us
I20260709 12:28:14.519505  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:14.488765  6275 rpcz_store.cc:276] 0709 12:28:12.830839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:12.831035 (+   196us) service_pool.cc:224] Handling call
0709 12:28:12.874399 (+ 43364us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:12.986067 (+111668us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:12.986117 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:12.986968 (+   851us) log.cc:844] Serialized 52 byte log entry
0709 12:28:13.069075 (+ 82107us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:13.069114 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:13.995363 (+926249us) spinlock_profiling.cc:243] Waited 4.3 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:14.474857 (+479494us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54005eef54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:14.474944 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:14.475007 (+    63us) raft_consensus.cc:1685] finished
0709 12:28:14.475027 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:14.475230 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1595,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":133,"spinlock_wait_cycles":265278592,"thread_start_us":81627,"threads_started":1,"wal-append.queue_time_us":124399}
W20260709 12:28:14.560819  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55029us
I20260709 12:28:14.503044 14132 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.500222 13755 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:14.520124 14056 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:14.470177 14049 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.482327 13950 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:14.520882 14131 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:14.581221 13782 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:14.477483 13951 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.503629 13812 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.585546 13743 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.113s
I20260709 12:28:14.653834 13950 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:14.711370 13743 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:28:14.713531 13951 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:14.714731 13951 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:14.715225 13951 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.719027 13951 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:14.719569 13951 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.600437 14131 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:14.601054 14008 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:14.601727 14043 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:14.655925 13782 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:14.732925  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:14.735438  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:14.598052 13755 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.707058 14056 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:14.733807 14126 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.547s	user 0.011s	sys 0.002s
W20260709 12:28:14.728912 13945 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.516s	user 0.010s	sys 0.001s
I20260709 12:28:14.594192 14132 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:14.743817  5635 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.024s	sys 0.060s
W20260709 12:28:14.752717  5635 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.025s	sys 0.061s
I20260709 12:28:14.576045  5636 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:14.730767 14131 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:14.736945 13782 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:14.751389  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:14.739353 14136 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.337s	user 0.000s	sys 0.001s
W20260709 12:28:14.740640 14050 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.509s	user 0.011s	sys 0.001s
I20260709 12:28:14.747777  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:14.766506  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:14.732015 13861 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:14.745157  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 437726us
I20260709 12:28:14.767295  5701 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:14.770217  5636 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:28:14.770334  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:14.731096 13986 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:14.764606  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:14.777997  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:14.736650 14008 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.736884 14043 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.761082 14133 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:14.779372  6275 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:14.783742  6277 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 26 because replica has last-logged OpId of term: 24 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:28:14.784821  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:14.770955  6273 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:28:14.778479 14050 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:14.778960 13986 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:14.805451 13755 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:14.792712  6271 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:14.808626 14148 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:28:14.810210 14148 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:28:14.811484 14148 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:28:14.812129 13969 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:28:14.753803 13945 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.815541  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1461) took 2123 ms. Trace:
I20260709 12:28:14.767822 13880 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:14.817065  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3191) took 2274 ms. Trace:
I20260709 12:28:14.799389 14043 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:14.815964  6279 rpcz_store.cc:276] 0709 12:28:12.692132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:12.692555 (+   423us) service_pool.cc:224] Handling call
0709 12:28:12.692910 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:12.991458 (+298548us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:28:13.075686 (+ 84228us) log.cc:844] Serialized 14 byte log entry
0709 12:28:14.066735 (+991049us) spinlock_profiling.cc:243] Waited 2.12 ms on lock 0x7b18003a6240. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:28:14.273494 (+206759us) log.cc:844] Serialized 14 byte log entry
0709 12:28:14.729733 (+456239us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:14.729797 (+    64us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:14.729856 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:14.815227 (+ 85371us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b5400711954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:14.815282 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:14.815436 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1283587,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":58,"raft.queue_time_us":195,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":8845,"spinlock_wait_cycles":690936960,"thread_start_us":680076,"threads_started":3,"wal-append.queue_time_us":620000,"wal-append.run_cpu_time_us":1365,"wal-append.run_wall_time_us":1043252}
I20260709 12:28:14.817530  5702 rpcz_store.cc:276] 0709 12:28:12.542932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:12.543215 (+   283us) service_pool.cc:224] Handling call
0709 12:28:14.816838 (+2273623us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400067054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:14.816979 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2762668416}
W20260709 12:28:14.817782  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:14.801107 13960 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.574s	user 0.013s	sys 0.000s
I20260709 12:28:14.819185 13960 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.802273  5629 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
W20260709 12:28:14.805229 14070 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.567s	user 0.011s	sys 0.000s
I20260709 12:28:14.820989  5709 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 17. (term mismatch)
I20260709 12:28:14.747949 14056 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.753051 14126 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.838492  5709 pending_rounds.cc:85] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 16
I20260709 12:28:14.838891  5709 pending_rounds.cc:107] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20260709 12:28:14.839205  5709 raft_consensus.cc:2889] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:14.779933 13861 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.841012 13945 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:14.842036 14133 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.842809 13861 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:14.843377 13861 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:14.720039 13951 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:14.844974 14140 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:28:14.848804 14008 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:14.849862 13986 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.851012 13866 raft_consensus.cc:2764] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:28:14.853789 14132 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:14.805289 14141 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.221s	user 0.000s	sys 0.000s
I20260709 12:28:14.856534  5628 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:14.861145  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:14.843858 13861 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.868321 13969 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:14.868484  6279 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:28:14.870842  5636 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:14.872171  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:14.872912  6279 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
W20260709 12:28:14.870940  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 310787us
W20260709 12:28:14.868752 14138 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 12:28:14.779011  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:14.866201 13960 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:14.885512  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:14.889176  6277 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:14.817310 13880 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.899612  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:14.901372  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 154458us
W20260709 12:28:14.902980  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 341213us
W20260709 12:28:14.903277  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:14.870623 13969 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:14.870641 14043 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:28:14.917233  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 138284us
W20260709 12:28:14.923902  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:14.984400  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:14.991458  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 69820.8us
I20260709 12:28:14.991909  6279 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:28:14.916674 14000 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.726s	user 0.011s	sys 0.001s
I20260709 12:28:14.995702 14000 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:28:14.996273 14147 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:28:14.995509 14152 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:14.997350  6273 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:28:14.996714 13969 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:14.997771 14152 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.998294 13861 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:14.998718 13969 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:28:14.996757  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 76238.5us
I20260709 12:28:15.003412  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.005493 14152 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:15.009914 13880 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:14.923296  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:14.865710 13950 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:14.996052 14148 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:28:15.023923  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:14.924324  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 440772us
I20260709 12:28:15.015017  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:28:14.991461  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:15.026549  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:14.919981 14043 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:15.028734  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.038457  5628 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:15.039615  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:15.047331  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:15.057286  5628 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:28:15.022468 14138 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:28:14.983457 14150 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:15.112010  6279 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:28:15.016839 13812 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:15.016907 13880 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:15.017343 14008 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:15.126506  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:15.129220  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 81990.8us
I20260709 12:28:15.147274  6275 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:15.021126 14152 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:15.043874 13945 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:15.147639  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:15.146680 14130 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.670s	user 0.011s	sys 0.000s
I20260709 12:28:15.169296  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.171218 14126 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:15.174229 14157 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:15.174819 14149 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
I20260709 12:28:15.175032 14157 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:15.164660 14102 log.cc:927] Time spent T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 2.057s	user 0.011s	sys 0.020s
I20260709 12:28:15.146404 13996 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:15.176252 13950 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:15.164801 13968 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:15.179723  5628 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
W20260709 12:28:15.179986 14151 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
I20260709 12:28:15.181555 14056 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:15.181141  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.152463 14008 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:15.162937 14152 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:15.185755 14132 raft_consensus.cc:2764] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:28:15.166018 13945 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:15.172142 14130 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:15.143926 14159 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:15.178308  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:15.179312  5525 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.008s	sys 0.008s
I20260709 12:28:15.204062  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.210798  5627 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
W20260709 12:28:15.213229  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 328194us
I20260709 12:28:15.150342 13880 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:15.214463  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 307995us
W20260709 12:28:15.234011  5525 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.009s	sys 0.008s
I20260709 12:28:15.206292 14157 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:15.239590  6277 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:15.242951  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:15.243676  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:15.233729  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55501.2us
I20260709 12:28:15.262595  5553 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:15.264093  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:15.265374  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:15.244551  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:15.266589  5558 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:15.267843  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:15.269146  5558 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 26 because replica has last-logged OpId of term: 24 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:28:15.235404  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:15.269867  5628 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:15.238337  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.233278 14159 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:15.237879  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.238682  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.236018  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:15.281395  6272 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:15.282420  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.211496 13996 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:15.285830  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.250712  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:15.251623  5557 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 23. Current term is 24. Ops: []
W20260709 12:28:15.273308  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:15.288054  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:15.271238  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:15.276615  5559 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:15.164916 14043 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:15.288219  5553 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:15.289252  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:15.292568  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:15.294627  6275 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:28:15.294921  6275 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:15.297669  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:15.299957  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:15.308678  5628 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:15.310005  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:15.311252  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 156770us
I20260709 12:28:15.311796  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.315339  5628 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:15.310005  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:15.211000 13968 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:15.358403  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.359658  5628 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:15.314635  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:15.321229  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:15.360379  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45836.6us
I20260709 12:28:15.361171  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.311942  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:15.232195 14021 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.020s	sys 0.018s
W20260709 12:28:15.356398 14165 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20260709 12:28:15.362392  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 50530.7us
I20260709 12:28:15.382133  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.248805 13950 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:15.378796  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:15.313830  6275 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:15.374339 14021 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.021s	sys 0.018s
I20260709 12:28:15.387269  6275 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:15.382625  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:15.373806  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:15.389498  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:15.391271  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.390206  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:15.287348 14159 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:15.396390  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:15.397642  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.307339 14167 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:15.405452  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.398411  6276 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:15.402659  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.403028  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:15.403657  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.409402  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.417040  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:15.418015  5560 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:15.420605  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.426035  5627 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:15.432174  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:15.433102  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.433104  5560 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:15.302163 14133 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.100s
I20260709 12:28:15.386639 13950 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:15.307536 14132 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 12, Time since last communication: 0.092s
I20260709 12:28:15.441869  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:15.444339 14152 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:15.445221  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:15.390385 14043 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:15.458822  5561 raft_consensus.cc:2410] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 27 because replica has last-logged OpId of term: 26 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:28:15.459299  6280 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:28:15.461045 14008 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:15.461462 14160 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.245s	user 0.001s	sys 0.000s
I20260709 12:28:15.461697 14008 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:15.462141 14008 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:15.462244 14152 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:28:15.462567 14008 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:15.462852 14152 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:28:15.464620  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:15.465540 13968 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:15.466140 13968 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:15.466473 13968 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:15.466857 13968 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:15.467818  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3295) took 1315 ms. Trace:
W20260709 12:28:15.468330 14168 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.275s	user 0.000s	sys 0.001s
I20260709 12:28:15.468231  5635 rpcz_store.cc:276] 0709 12:28:14.152419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:14.202341 (+ 49922us) service_pool.cc:224] Handling call
0709 12:28:14.202660 (+   319us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:14.753204 (+550544us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:14.753235 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:14.762450 (+  9215us) log.cc:844] Serialized 26 byte log entry
0709 12:28:15.210668 (+448218us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:15.210704 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:15.210770 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:15.467574 (+256804us) raft_consensus.cc:1685] finished
0709 12:28:15.467594 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:15.467758 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9829,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"thread_start_us":966329,"threads_started":2,"wal-append.queue_time_us":612405}
I20260709 12:28:15.469903  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:15.470181  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.471056  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:15.471223  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.451144 13950 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:15.451720 14161 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.259s	user 0.001s	sys 0.001s
I20260709 12:28:15.468027 14169 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:28:15.474613 14169 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:15.477058  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:15.474151  5561 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:28:15.475596 14169 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:15.495549  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3040) took 1391 ms. Trace:
I20260709 12:28:15.497033  5632 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:15.417218 14167 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:15.454866  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.456486  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.456882  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.446738  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:15.481663  5635 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:28:15.457916  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.485574  6275 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:15.514462  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.493973  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:15.478433  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:15.519055  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.478071 14000 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:15.496025  5629 rpcz_store.cc:276] 0709 12:28:14.104316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:14.150356 (+ 46040us) service_pool.cc:224] Handling call
0709 12:28:15.495297 (+1344941us) spinlock_profiling.cc:243] Waited 651 ms on lock 0x7b540032bed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:15.495469 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1496770304}
W20260709 12:28:15.516337  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 133824us
W20260709 12:28:15.516381  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 593150us
I20260709 12:28:15.489073  5553 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:15.507215  5560 raft_consensus.cc:2410] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
W20260709 12:28:15.522684  5631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3019) took 5009 ms (client timeout 2875 ms). Trace:
I20260709 12:28:15.556252  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:15.529031  5631 rpcz_store.cc:269] 0709 12:28:10.507068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:10.650398 (+143330us) service_pool.cc:224] Handling call
0709 12:28:15.508220 (+4857822us) spinlock_profiling.cc:243] Waited 3.29 s on lock 0x7b54001ce8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:15.516735 (+  8515us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7578412800}
I20260709 12:28:15.557258  5560 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:15.561045  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:15.561847  5560 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:15.556134  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:15.565500  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.463851 14152 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:15.508311 13627 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:15.582659  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:15.518536 14038 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:15.577231  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:15.581828  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.588122  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.523739 14000 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:15.515808 14167 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:15.556145 13999 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:15.607447  5554 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:15.609645  6275 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
W20260709 12:28:15.608290  5554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3059) took 2331 ms (client timeout 1609 ms). Trace:
W20260709 12:28:15.618484  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:15.618492  5554 rpcz_store.cc:269] 0709 12:28:13.277186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:13.472418 (+195232us) service_pool.cc:224] Handling call
0709 12:28:15.493897 (+2021479us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5c0020248c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffdfdc161 00007f4ffe17da79 00007f4fff6379b6 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:15.608193 (+114296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":263513856}
W20260709 12:28:15.611289  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 215456us
I20260709 12:28:15.621053  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:15.613003  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 87192.2us
I20260709 12:28:15.458213 13996 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:15.558977 13771 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.042s
I20260709 12:28:15.626573  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:15.597570 14000 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:15.627434  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:15.627868  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:15.639577  5704 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:15.639647  5708 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:15.628611  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:15.629825  5599 connection.cc:256] RPC call timeout handler was delayed by -1.9434720650001509s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
W20260709 12:28:15.640625  5599 connection.cc:256] RPC call timeout handler was delayed by -1.921872065000116s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1524s for a response
I20260709 12:28:15.635561  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:15.641345  5599 connection.cc:256] RPC call timeout handler was delayed by -1.7904720649999035s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.167s for a response
I20260709 12:28:15.641609  5707 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:15.627247  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:15.623147  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:15.641948  5599 connection.cc:256] RPC call timeout handler was delayed by -1.7373720650002724s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
W20260709 12:28:15.642334  5599 connection.cc:256] RPC call timeout handler was delayed by -1.6608720650001487s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1814s for a response
W20260709 12:28:15.642644  5599 connection.cc:256] RPC call timeout handler was delayed by -1.4970720649998839s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1996s for a response
I20260709 12:28:15.642560  5710 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
W20260709 12:28:15.643287  5599 connection.cc:256] RPC call timeout handler was delayed by -1.4180751160001819s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1748s for a response
W20260709 12:28:15.643755  5599 connection.cc:256] RPC call timeout handler was delayed by -1.0979667840001639s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16090000000000002s for a response
W20260709 12:28:15.644411  5599 connection.cc:256] RPC call timeout handler was delayed by -1.0643751160000647s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2141s for a response
I20260709 12:28:15.644975  5704 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:28:15.644908  5705 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:15.645985  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:15.647827 14021 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:15.650174  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:15.657619  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.659582  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:15.609431 14173 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:15.660065  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.600366 13999 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:15.661693  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.664081 14155 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:15.665530  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:15.666467  5705 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 26 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:28:15.600533 14038 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:15.590934 13627 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:15.671602  5701 raft_consensus.cc:2410] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 20 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 12:28:15.669035  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:15.655020 13782 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.881s	user 0.008s	sys 0.004s
I20260709 12:28:15.610406 14148 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.052s
W20260709 12:28:15.683327  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:15.717422  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:15.689069  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:15.719677  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:15.719985  5628 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:28:15.720582  5628 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:15.702181  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:15.703136  6276 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:28:15.710021  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51680.2us
I20260709 12:28:15.669274 13999 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:15.736856  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 112586us
I20260709 12:28:15.738416  5628 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:15.740319  5628 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:15.741837  5634 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 13. (term mismatch)
I20260709 12:28:15.742446  5634 pending_rounds.cc:85] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 12
I20260709 12:28:15.742748  5634 pending_rounds.cc:107] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 22.13
I20260709 12:28:15.743076  5634 raft_consensus.cc:2889] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:15.633323 14172 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.107s
I20260709 12:28:15.700011 14175 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:15.669339 14188 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:15.766460  5599 leader_election.cc:400] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25.
I20260709 12:28:15.721508 14133 consensus_queue.cc:1243] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 24.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
W20260709 12:28:15.773155  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48126.4us
I20260709 12:28:15.723776 14130 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:15.782066  5701 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 27 index: 13. (index mismatch)
I20260709 12:28:15.688964 14174 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.087s
I20260709 12:28:15.721735 13782 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:15.806478  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:15.685930 14173 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:15.808260  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.816195  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.773587 14188 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:15.821825  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:15.823442  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 43721.6us
I20260709 12:28:15.824970  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:15.826335  5628 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:28:15.828670  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:15.830235  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.778875 14175 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:15.831431  5628 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
W20260709 12:28:15.832619  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 111179us
I20260709 12:28:15.835214  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:15.836019  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:15.837365  5628 raft_consensus.cc:2368] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27.
I20260709 12:28:15.839398  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:15.840943  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:15.844136  5628 raft_consensus.cc:2410] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 22 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:28:15.847170  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:15.848017  5628 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:15.824290 14188 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:15.839125 14156 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:15.836758 14182 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:15.829926 14157 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.029s
I20260709 12:28:15.842154 14132 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:15.845113 13909 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:15.848624 14193 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:15.864882 14156 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:15.867491 14185 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
W20260709 12:28:15.868047 14184 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
W20260709 12:28:15.869294 14187 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.200s	user 0.001s	sys 0.000s
I20260709 12:28:15.845114 14179 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:15.936960 14179 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:15.848811 14166 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.272s	user 0.001s	sys 0.000s
I20260709 12:28:15.842531 14126 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.023s
I20260709 12:28:15.845373 14181 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.061s
I20260709 12:28:15.801039 14038 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:15.863162 14200 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:15.949508 14200 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:15.950448 14200 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:15.951696 13969 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.952s	user 0.013s	sys 0.000s
I20260709 12:28:15.921955 14156 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:15.880918 14132 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:15.881896 13627 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:15.843911 14155 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
I20260709 12:28:15.910550 13880 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:15.910496 14153 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.830s	user 0.001s	sys 0.001s
I20260709 12:28:15.909454 14049 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:15.940181 14179 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:15.913101 14193 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:15.981241 14131 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.215s	user 0.013s	sys 0.001s
I20260709 12:28:15.870455 14182 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:15.982681 14043 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.515s	user 0.020s	sys 0.003s
I20260709 12:28:15.980962 14208 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:28:15.989503 14208 consensus_queue.cc:1243] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 22.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:28:15.989480 14131 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:15.991194 14209 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:15.991045 14167 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:15.991619 14209 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:15.992713 14173 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:15.992552 14209 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:15.992872 14131 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:15.972232 14179 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 12:28:15.911237 14175 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:15.998464 14159 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.575s	user 0.008s	sys 0.002s
I20260709 12:28:15.998553 14167 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:15.998922 14159 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:16.000604 14159 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:16.003604 14209 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:16.003408 14188 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:16.005165 14193 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:16.006830 14188 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:16.006966 14182 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:16.046347  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 266229us
I20260709 12:28:16.047950  5560 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
W20260709 12:28:16.051024  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 210772us
I20260709 12:28:16.053364  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3313) took 3395 ms. Trace:
I20260709 12:28:16.053892  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:28:16.054615  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:16.053874  5555 rpcz_store.cc:276] 0709 12:28:12.657446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:13.145956 (+488510us) service_pool.cc:224] Handling call
0709 12:28:13.157260 (+ 11304us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:13.159417 (+  2157us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:13.159454 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:13.161927 (+  2473us) log.cc:844] Serialized 26 byte log entry
0709 12:28:14.419945 (+1258018us) spinlock_profiling.cc:243] Waited 8.54 ms on lock 0x7b18002ab6e0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:28:14.905448 (+485503us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:14.905508 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:14.905606 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:15.909922 (+1004316us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:16.052658 (+142736us) raft_consensus.cc:1685] finished
0709 12:28:16.052709 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:16.053085 (+   376us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1737,"raft.queue_time_us":2236,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":2276,"spinlock_wait_cycles":45404928,"thread_start_us":485261,"threads_started":1,"wal-append.queue_time_us":1759136}
I20260709 12:28:16.070999  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:16.073642  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:16.074990  5628 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:16.076102  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:16.077776  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:16.079291  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:16.082489  5676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.009s	sys 0.001s
W20260709 12:28:16.082967  5676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.009s	sys 0.001s
I20260709 12:28:16.091960  5702 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:16.092231  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:16.092931  5701 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:28:16.093655  5704 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:16.094578  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:16.094803  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:16.094976  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:15.956025  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 388465us
I20260709 12:28:15.882807 13909 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:16.093606  5705 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:28:16.097474  5703 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:15.966871 13950 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.098083  5707 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:16.097206  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:16.119561  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3287) took 2003 ms. Trace:
I20260709 12:28:16.120141  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:16.120078  5632 rpcz_store.cc:276] 0709 12:28:14.115856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:14.195534 (+ 79678us) service_pool.cc:224] Handling call
0709 12:28:15.496779 (+1301245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:15.515149 (+ 18370us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.118835 (+603686us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540032bed0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:16.119180 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2991875968}
I20260709 12:28:16.121536  5706 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:16.122105  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:16.122697  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:16.108297 13782 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:16.096212  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:15.958976 13968 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.125689 13782 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:16.127480 14132 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:15.994167 14125 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.494s	user 0.021s	sys 0.017s
I20260709 12:28:16.096149  5704 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:16.118721  5559 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:16.096606  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:16.128000 13968 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:16.163064  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:16.224395  5629 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.004s	sys 0.007s
W20260709 12:28:16.225067  5629 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.004s	sys 0.007s
W20260709 12:28:16.224871  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 176485us
I20260709 12:28:16.226341  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:16.228617  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:16.229460  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:16.230315  5706 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:16.116465  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:16.114989  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:16.237255  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3288) took 2110 ms. Trace:
I20260709 12:28:16.238543  5630 rpcz_store.cc:276] 0709 12:28:14.126307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:14.150677 (+ 24370us) service_pool.cc:224] Handling call
0709 12:28:16.236577 (+2085900us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x7b5400241154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:16.236819 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":63527680}
W20260709 12:28:16.222730 14125 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.634s	user 0.022s	sys 0.017s
W20260709 12:28:16.243086  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 185722us
I20260709 12:28:16.148887 13950 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:16.126566 13782 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.100842 13812 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:16.250813  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:16.269734  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 43700us
I20260709 12:28:16.270629  5633 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:15.973744 13999 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:16.275475  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:16.276856  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1376) took 1480 ms. Trace:
I20260709 12:28:16.277609  6273 rpcz_store.cc:276] 0709 12:28:14.796236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:14.796502 (+   266us) service_pool.cc:224] Handling call
0709 12:28:14.814135 (+ 17633us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:14.997780 (+183645us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.259735 (+1261955us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54006121d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:16.276541 (+ 16806us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":420745600}
I20260709 12:28:16.280712 13812 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.281659 13950 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:16.282460 13950 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:16.282984 13950 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:16.284097  5706 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260709 12:28:16.284730  5627 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.001s	sys 0.009s
W20260709 12:28:16.285363  5627 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.001s	sys 0.009s
I20260709 12:28:16.265744  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:16.254078 14125 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.007s
I20260709 12:28:16.298651  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1357) took 4605 ms. Trace:
W20260709 12:28:16.308743  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:16.300390  5557 raft_consensus.cc:2410] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:28:16.298870 13999 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:16.301262  5558 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:28:16.301446  5552 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3322) took 3598 ms (client timeout 2963 ms). Trace:
W20260709 12:28:16.322705  5552 rpcz_store.cc:269] 0709 12:28:12.703188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:13.119605 (+416417us) service_pool.cc:224] Handling call
0709 12:28:16.300691 (+3181086us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b5400025dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:16.301092 (+   401us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2907657600}
I20260709 12:28:16.106400 13627 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.200s
I20260709 12:28:16.305126 14021 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:28:16.292994  5635 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.000s	sys 0.010s
I20260709 12:28:16.300061 14154 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.020s
W20260709 12:28:16.325611  5635 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.000s	sys 0.019s
I20260709 12:28:16.325922  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 1678) took 1081 ms. Trace:
I20260709 12:28:16.326335  5553 rpcz_store.cc:276] 0709 12:28:15.244319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:15.263781 (+ 19462us) service_pool.cc:224] Handling call
0709 12:28:16.324232 (+1060451us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b54002283d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:16.325720 (+  1488us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":53244800}
W20260709 12:28:16.327299  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:16.328177  5704 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:16.328569 13812 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:16.334630  6273 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
W20260709 12:28:16.332244 14008 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.868s	user 0.013s	sys 0.002s
I20260709 12:28:16.337175  5630 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 12:28:16.338248  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3045) took 1052 ms. Trace:
I20260709 12:28:16.338977  5630 rpcz_store.cc:276] 0709 12:28:15.285551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:16.253114 (+967563us) service_pool.cc:224] Handling call
0709 12:28:16.336671 (+ 83557us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:16.337760 (+  1089us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.338016 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:28:16.340699  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 464210us
I20260709 12:28:16.343921 13627 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:28:16.344551 13627 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:16.314844  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3334) took 1033 ms. Trace:
I20260709 12:28:16.307972 13909 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:16.308640  6278 rpcz_store.cc:276] 0709 12:28:11.692039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:11.692189 (+   150us) service_pool.cc:224] Handling call
0709 12:28:13.279926 (+1587737us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:13.617148 (+337222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.259379 (+2642231us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x7b54002e3954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:16.260383 (+  1004us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b54002e3950. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:16.297453 (+ 37070us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4409467392}
I20260709 12:28:16.348412 14013 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:16.345804  5557 rpcz_store.cc:276] 0709 12:28:15.284146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:16.114162 (+830016us) service_pool.cc:224] Handling call
0709 12:28:16.314746 (+200584us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:16.349638 13782 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:16.351702  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 76539.9us
W20260709 12:28:16.353245  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:16.353865  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1720) took 2197 ms. Trace:
W20260709 12:28:16.366593  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 71505.6us
W20260709 12:28:16.357021  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 85207.6us
I20260709 12:28:16.357821  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3305) took 1188 ms. Trace:
I20260709 12:28:16.345165  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3304) took 1175 ms. Trace:
I20260709 12:28:16.360854  5628 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:28:16.348109  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58335.6us
I20260709 12:28:16.325153  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3333) took 1060 ms. Trace:
I20260709 12:28:16.356557 14021 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:16.375622 13771 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:16.368510  5636 rpcz_store.cc:276] 0709 12:28:14.156819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:14.771105 (+614286us) service_pool.cc:224] Handling call
0709 12:28:14.771365 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:14.871571 (+100206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.353316 (+1481745us) spinlock_profiling.cc:243] Waited 99.3 ms on lock 0x7b54001de2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:16.353650 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":228331904}
I20260709 12:28:16.376606 14043 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:16.377341 13627 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.375941 14021 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.379176  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:16.377674 14043 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.369755  5558 rpcz_store.cc:276] 0709 12:28:15.280732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:16.115796 (+835064us) service_pool.cc:224] Handling call
0709 12:28:16.325084 (+209288us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:16.395685  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1721) took 2195 ms. Trace:
I20260709 12:28:16.397778  5634 rpcz_store.cc:276] 0709 12:28:14.157162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:14.772934 (+615772us) service_pool.cc:224] Handling call
0709 12:28:15.508501 (+735567us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:15.743689 (+235188us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.350908 (+607219us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b54001ce8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:16.351920 (+  1012us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b54001ce8d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:16.352218 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2226456704}
I20260709 12:28:16.401569  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3313) took 1137 ms. Trace:
I20260709 12:28:16.402316  5629 rpcz_store.cc:276] 0709 12:28:15.264166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:15.565669 (+301503us) service_pool.cc:224] Handling call
0709 12:28:15.566085 (+   416us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:15.576566 (+ 10481us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:15.576684 (+   118us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:15.589767 (+ 13083us) log.cc:844] Serialized 52 byte log entry
0709 12:28:16.225958 (+636191us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:16.226098 (+   140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.226249 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:16.382002 (+155753us) raft_consensus.cc:1685] finished
0709 12:28:16.382029 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:16.401448 (+ 19419us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":113,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":18347,"thread_start_us":567004,"threads_started":1,"wal-append.queue_time_us":745924}
I20260709 12:28:16.405910 14043 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:16.364168 14008 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:16.410744  5553 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:16.411481 14008 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:16.414105  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:16.385540 13812 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:16.369094  5627 rpcz_store.cc:276] 0709 12:28:15.169742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:15.524653 (+354911us) service_pool.cc:224] Handling call
0709 12:28:15.562273 (+ 37620us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:15.562762 (+   489us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:15.562793 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:15.581754 (+ 18961us) log.cc:844] Serialized 26 byte log entry
0709 12:28:16.286061 (+704307us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:16.286149 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.286272 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:16.357470 (+ 71198us) raft_consensus.cc:1685] finished
0709 12:28:16.357496 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:16.357739 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6443,"raft.queue_time_us":108,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":13591,"thread_start_us":648892,"threads_started":1,"wal-append.queue_time_us":621198}
I20260709 12:28:16.428107  5708 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:28:16.433142  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:16.425455  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:16.439307  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:16.369434  5635 rpcz_store.cc:276] 0709 12:28:15.169406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:15.538397 (+368991us) service_pool.cc:224] Handling call
0709 12:28:15.561199 (+ 22802us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:15.561756 (+   557us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:15.561829 (+    73us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:15.574343 (+ 12514us) log.cc:844] Serialized 52 byte log entry
0709 12:28:16.326239 (+751896us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:16.326635 (+   396us) log.cc:844] Serialized 14 byte log entry
0709 12:28:16.344449 (+ 17814us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.344637 (+   188us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:16.344692 (+    55us) raft_consensus.cc:1685] finished
0709 12:28:16.344707 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:16.345032 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17433,"prepare.queue_time_us":415,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":23805,"raft.run_cpu_time_us":417,"raft.run_wall_time_us":13528,"thread_start_us":663405,"threads_started":1,"wal-append.queue_time_us":628615}
I20260709 12:28:16.450610  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:16.453109 14156 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.455227  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:16.458647  5552 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:16.462507  5560 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:16.462247 14046 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:28:16.466810  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45594.1us
I20260709 12:28:16.462247  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:16.469069  6271 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:16.415022 13771 consensus_queue.cc:1243] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 27.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
W20260709 12:28:16.445381  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.026s	sys 0.015s
I20260709 12:28:16.478338  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:16.483502  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:16.484084 14038 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:28:16.485026  6275 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 19: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:16.487376  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:16.483209  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:16.489522 14215 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:16.490411 14155 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:16.490264 14043 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:28:16.491123 14155 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.491952 14216 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:28:16.494114 14155 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:16.494377 14176 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:28:16.495790 13945 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:16.490295 14215 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:16.498713 14202 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:16.500133 14202 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:16.500635 14202 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:16.501266 14183 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:16.501739 14183 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:16.503026 14183 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:16.507543 14194 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.642s	user 0.000s	sys 0.000s
I20260709 12:28:16.500108 14174 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.010s
I20260709 12:28:16.487596  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:16.446645 13627 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:16.470115 14046 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:16.512773  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:16.512918 14046 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:16.513080  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:16.498035 14215 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.508010  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:16.517222  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1714) took 2363 ms. Trace:
I20260709 12:28:16.517956 13812 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:16.517947 14183 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:16.519044 13953 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:28:16.519539 13996 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.010s	sys 0.012s
I20260709 12:28:16.519698 13953 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:16.502019  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:16.518945  5675 leader_election.cc:336] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.557s (SENT)
W20260709 12:28:16.521525  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.862s	user 0.027s	sys 0.016s
W20260709 12:28:16.517608  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61339us
I20260709 12:28:16.517781  5633 rpcz_store.cc:276] 0709 12:28:14.153998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:14.249489 (+ 95491us) service_pool.cc:224] Handling call
0709 12:28:16.240699 (+1991210us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:16.271190 (+ 30491us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.516932 (+245742us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b5400241150. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:16.517119 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4578396032}
I20260709 12:28:16.455658 14156 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:16.534385  5560 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:16.536382  6271 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:16.551992  5704 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:28:16.555567  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:16.550961  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:16.564124 14117 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:28:16.564321 14218 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:16.566090  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:16.569295  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:16.564859  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:16.501995 14202 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:16.573283  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60599.3us
I20260709 12:28:16.576491 13812 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:28:16.567775  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:16.580164  5635 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:16.577661 13953 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.586777 13953 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:16.587292 13951 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:16.588985  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:16.597034  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:16.599010  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:16.599478 14125 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:16.599606  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:16.616530  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:16.634258  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:16.628132  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 131906us
I20260709 12:28:16.640906  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3043) took 1355 ms. Trace:
I20260709 12:28:16.590427 14008 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:16.577980 14218 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.605911  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1477) took 3937 ms. Trace:
I20260709 12:28:16.648699  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:16.607791  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:16.625258 14228 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
W20260709 12:28:16.611999 14209 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.607s	user 0.005s	sys 0.011s
I20260709 12:28:16.682497  5629 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:16.676127  5709 rpcz_store.cc:276] 0709 12:28:12.668271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:13.132174 (+463903us) service_pool.cc:224] Handling call
0709 12:28:14.820609 (+1688435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:14.839850 (+ 19241us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.605474 (+1765624us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b5400067050. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:16.605708 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3850944384}
I20260709 12:28:16.615551 13969 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:16.608021 13996 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:16.622304 13743 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:16.667618  5631 rpcz_store.cc:276] 0709 12:28:15.284997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:15.591228 (+306231us) service_pool.cc:224] Handling call
0709 12:28:15.621910 (+ 30682us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:15.622311 (+   401us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:28:15.819268 (+196957us) log.cc:844] Serialized 14 byte log entry
0709 12:28:16.336974 (+517706us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:16.338054 (+  1080us) log.cc:844] Serialized 52 byte log entry
0709 12:28:16.338617 (+   563us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:16.345953 (+  7336us) log.cc:844] Serialized 14 byte log entry
0709 12:28:16.384345 (+ 38392us) log.cc:844] Serialized 14 byte log entry
0709 12:28:16.416667 (+ 32322us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.432850 (+ 16183us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:16.592635 (+159785us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b180013f120. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:28:16.624367 (+ 31732us) spinlock_profiling.cc:243] Waited 35.8 ms on lock 0x7b180013f120. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:28:16.640476 (+ 16109us) raft_consensus.cc:1685] finished
0709 12:28:16.640509 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:16.640711 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":73701,"prepare.queue_time_us":190743,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":122,"raft.queue_time_us":106822,"raft.run_cpu_time_us":770,"raft.run_wall_time_us":39901,"spinlock_wait_cycles":98014592,"thread_start_us":558779,"threads_started":2,"wal-append.queue_time_us":467863}
I20260709 12:28:16.609154 14172 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:28:16.691340 14000 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.926s	user 0.012s	sys 0.016s
I20260709 12:28:16.692003 14125 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:16.691877 14000 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:16.692524 14125 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:16.692945 14125 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:16.694756 14000 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:16.600795 14117 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.096s
I20260709 12:28:16.696799  6276 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:16.685714 13969 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.673949 14156 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:16.684814 14202 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:16.704246  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:16.711683 13969 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:16.706982  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 122643us
I20260709 12:28:16.714310  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:16.689774 13743 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:16.700335 14172 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:16.723418  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 174095us
I20260709 12:28:16.703840 14021 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:28:16.730846  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 211948us
I20260709 12:28:16.733875 13986 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:16.735293 13743 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.736892 13743 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:16.716701 13996 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:16.745442 14226 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
I20260709 12:28:16.747882  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:16.749414 14142 raft_consensus.cc:2764] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:28:16.749665 14008 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:28:16.749404  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:16.754678  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3055) took 1444 ms. Trace:
I20260709 12:28:16.750566 14008 consensus_queue.cc:1243] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 22.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:28:16.750533 14142 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 12:28:16.762835  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:16.763293 14200 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:16.763377 14008 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:16.764648 14209 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:28:16.764751  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 165260us
I20260709 12:28:16.765220 14209 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:16.765652 14209 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:16.770850 14212 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260709 12:28:16.773443 14046 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.764150 14008 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:16.776912 14008 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:16.785414  5629 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:28:16.747859 13996 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:16.798134  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3044) took 1512 ms. Trace:
I20260709 12:28:16.797333  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:16.797993  5675 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.562s (SENT)
I20260709 12:28:16.800027  5632 rpcz_store.cc:276] 0709 12:28:15.285301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:16.123479 (+838178us) service_pool.cc:224] Handling call
0709 12:28:16.124277 (+   798us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:16.339636 (+215359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:16.339680 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:16.341213 (+  1533us) log.cc:844] Serialized 26 byte log entry
0709 12:28:16.396259 (+ 55046us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:16.396311 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.396444 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:16.797864 (+401420us) raft_consensus.cc:1685] finished
0709 12:28:16.797881 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:16.798004 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30839,"raft.queue_time_us":31104,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":135,"thread_start_us":54350,"threads_started":1,"wal-append.queue_time_us":173122}
I20260709 12:28:16.757686  5636 rpcz_store.cc:276] 0709 12:28:15.306514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:16.377100 (+1070586us) service_pool.cc:224] Handling call
0709 12:28:16.377715 (+   615us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:16.424462 (+ 46747us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:16.424498 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:16.425159 (+   661us) log.cc:844] Serialized 26 byte log entry
0709 12:28:16.550610 (+125451us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:16.570914 (+ 20304us) log.cc:844] Serialized 14 byte log entry
0709 12:28:16.578620 (+  7706us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.578671 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:16.750992 (+172321us) raft_consensus.cc:1685] finished
0709 12:28:16.751009 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:16.751123 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13873,"prepare.queue_time_us":20042,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":41624,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":4383,"thread_start_us":125058,"threads_started":1,"wal-append.queue_time_us":258357}
W20260709 12:28:16.805241  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44471us
I20260709 12:28:16.807723  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:28:16.812811  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48880.1us
I20260709 12:28:16.802090 14046 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:16.816078 14215 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:28:16.822757 14150 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.560s	user 0.004s	sys 0.013s
W20260709 12:28:16.834632  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:16.831990 14224 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.263s	user 0.000s	sys 0.001s
W20260709 12:28:16.833652 14150 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.661s	user 0.005s	sys 0.014s
W20260709 12:28:16.865470  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 107102us
I20260709 12:28:16.872727 14126 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:16.873282 14126 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:16.874886 14227 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.298s	user 0.003s	sys 0.000s
I20260709 12:28:16.874532 14126 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.875649 14126 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:16.854426 14218 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:16.824842 14172 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:16.858026 14150 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:16.866547 13953 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.020s
I20260709 12:28:16.887051 13861 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.011s
W20260709 12:28:16.891896  5675 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.629s (SENT)
I20260709 12:28:16.894861  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:16.896299  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:16.897122 13950 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.613s	user 0.014s	sys 0.003s
I20260709 12:28:16.889621 14233 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
I20260709 12:28:16.904511 13950 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:16.906859 13950 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:16.907081 14046 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:28:16.910058 14209 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:16.912052  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:16.912766 14232 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:28:16.887683 14218 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.010s
I20260709 12:28:16.913870  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:16.912812  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:16.894196 14142 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.030s
I20260709 12:28:16.921547 14046 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:16.923574 14046 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:16.923981 14046 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:16.924641  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:16.925292  5633 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:28:16.927893  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3505) took 1274 ms. Trace:
I20260709 12:28:16.929836  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:16.915905 13861 consensus_queue.cc:1243] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 21.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:28:16.928304  5707 rpcz_store.cc:276] 0709 12:28:15.653520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:15.653670 (+   150us) service_pool.cc:224] Handling call
0709 12:28:15.653966 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:16.118205 (+464239us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.925995 (+807790us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b5400066154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:16.927804 (+  1809us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1019593472}
I20260709 12:28:16.933066  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:16.920534 14172 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:16.949002  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3054) took 1659 ms. Trace:
I20260709 12:28:16.935765  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:16.949561  5630 rpcz_store.cc:276] 0709 12:28:15.289512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:16.340184 (+1050672us) service_pool.cc:224] Handling call
0709 12:28:16.361093 (+ 20909us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:16.361939 (+   846us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:16.361993 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:16.362773 (+   780us) log.cc:844] Serialized 26 byte log entry
0709 12:28:16.743607 (+380834us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:16.755853 (+ 12246us) log.cc:844] Serialized 14 byte log entry
0709 12:28:16.756171 (+   318us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.756233 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:16.924618 (+168385us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b180045ebc0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:28:16.948608 (+ 23990us) raft_consensus.cc:1685] finished
0709 12:28:16.948739 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:16.948922 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6009,"prepare.queue_time_us":12136,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":4646,"raft.run_cpu_time_us":491,"raft.run_wall_time_us":11803,"spinlock_wait_cycles":49187200,"thread_start_us":368430,"threads_started":1,"wal-append.queue_time_us":451343}
I20260709 12:28:16.958236  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1681) took 1713 ms. Trace:
I20260709 12:28:16.958724  5556 rpcz_store.cc:276] 0709 12:28:15.245090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:16.956939 (+1711849us) service_pool.cc:224] Handling call
0709 12:28:16.957446 (+   507us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:16.957721 (+   275us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:16.957744 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:16.957790 (+    46us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:28:16.957817 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.957871 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:16.958003 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:28:16.960424 14235 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:28:16.944634 14150 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:16.972679  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 43813.2us
I20260709 12:28:16.973953 14119 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:28:16.976886  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:16.977092  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:16.977854  5707 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
W20260709 12:28:16.978966  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 46982.3us
I20260709 12:28:16.979900  5709 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 24. Ops: []
I20260709 12:28:16.956806  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1388) took 1130 ms. Trace:
I20260709 12:28:16.980285  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:16.980504 14154 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:28:16.981817 14154 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:16.982226  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:28:16.984529  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70430.8us
I20260709 12:28:16.985440  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:16.985569  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:16.986608  5630 raft_consensus.cc:2410] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 25 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:28:16.987033  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:16.988400  6276 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: []
I20260709 12:28:16.989042 14142 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:28:16.973765  5526 leader_election.cc:400] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27.
I20260709 12:28:16.989840  5526 leader_election.cc:403] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260709 12:28:16.990039 14142 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:16.990269  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:16.990474 14142 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:16.990989  5710 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:16.991415 13950 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:16.992007 13861 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:16.992448 14233 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27.
I20260709 12:28:16.992656  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:16.994412  5559 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:16.995221  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:16.991497 14142 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:16.991415  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:16.989923  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:16.974197 14021 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:16.980288  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:16.999485 14233 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:28:16.998692  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:16.984289 14156 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:17.007701  5705 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:16.999339 13917 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:16.999920  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:17.005151  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:17.008980  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:17.009352  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:17.012449  6276 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 12:28:16.973798 13960 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:17.098944  5557 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:16.980552  6272 rpcz_store.cc:276] 0709 12:28:15.812000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:15.812145 (+   145us) service_pool.cc:224] Handling call
0709 12:28:15.812643 (+   498us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:28:16.330902 (+518259us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b18004902a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffa66a060 00007f4ffa66abfb 00007f4ffa67564b
0709 12:28:16.331221 (+   319us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:28:16.331734 (+   513us) log.cc:844] Serialized 14 byte log entry
0709 12:28:16.477471 (+145737us) log.cc:844] Serialized 14 byte log entry
0709 12:28:16.525007 (+ 47536us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:28:16.579482 (+ 54475us) log.cc:844] Serialized 78 byte log entry
0709 12:28:16.579618 (+   136us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:16.579857 (+   239us) log.cc:844] Serialized 14 byte log entry
0709 12:28:16.580038 (+   181us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:16.591916 (+ 11878us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:16.873437 (+281521us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b18004902a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:28:16.874068 (+   631us) spinlock_profiling.cc:243] Waited 3.89 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:16.936798 (+ 62730us) raft_consensus.cc:1685] finished
0709 12:28:16.936818 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:16.942574 (+  5756us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26928,"prepare.queue_time_us":2368,"prepare.run_cpu_time_us":156,"prepare.run_wall_time_us":158,"raft.queue_time_us":84681,"raft.run_cpu_time_us":655,"raft.run_wall_time_us":38819,"spinlock_wait_cycles":20472960,"thread_start_us":203839,"threads_started":2,"wal-append.queue_time_us":244576}
W20260709 12:28:17.101822  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 169069us
I20260709 12:28:17.001183 14021 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:17.059034  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 57053.1us
I20260709 12:28:17.008231 13861 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:17.007843 14156 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:16.982259 14154 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:17.123908  5710 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:17.017385 13917 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.127187  5559 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:17.126935 14008 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.014715 14209 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:17.127539  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 65417.9us
I20260709 12:28:17.128070 14142 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:17.057868 13951 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:28:17.131124  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 203798us
I20260709 12:28:17.109918 13960 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:17.109236 14233 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:17.013541 13627 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.119s
W20260709 12:28:17.154181  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:17.167008 13960 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:17.122908 13996 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:17.013782 14125 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.146960  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:17.171456  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 184559us
I20260709 12:28:17.140390 13812 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:17.174316  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:17.164177 14215 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:17.186024  5707 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:17.204676  5553 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:17.206423  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:17.207540  5635 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: [27.19-27.19]
W20260709 12:28:17.208186  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 216950us
I20260709 12:28:17.214951  5705 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:17.218674 14046 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:17.219996  5553 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:17.222752 14209 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:28:17.224138 13861 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:17.224812 13861 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:17.226374 13861 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:17.228966 13861 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:17.229123 14240 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:28:17.248472  5707 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: [25.12-27.13]
I20260709 12:28:17.178188 14158 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.064s
W20260709 12:28:17.263021  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:17.284803  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 78449.1us
I20260709 12:28:17.261687  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:17.293792  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:17.293627  5554 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 17. (term mismatch)
W20260709 12:28:17.287588  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 117337us
W20260709 12:28:17.296526  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 123324us
I20260709 12:28:17.203106 13996 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.256688 13743 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:17.295699  5554 pending_rounds.cc:85] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 16
I20260709 12:28:17.311260  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:17.302110  5554 pending_rounds.cc:107] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 23.17
I20260709 12:28:17.325083  5554 raft_consensus.cc:2889] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:17.304831 13743 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:17.258133 14156 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:17.379828  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 67433.3us
I20260709 12:28:17.382416  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:17.342523 13743 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:17.409274 14253 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:17.409528 14251 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:17.409729 13627 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:28:17.410071 14249 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:17.410583 14249 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:17.410914 14249 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:17.411161 14244 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:17.411919 14244 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:17.413125 14244 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.414693 14244 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:17.404130 14008 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:17.427529  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 94704.3us
W20260709 12:28:17.422919  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45649.1us
I20260709 12:28:17.417253 14253 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:17.446462  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:17.336786 14233 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.428879 14125 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:17.463765  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:17.443243 13812 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:17.464712  5633 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:17.450736 14046 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:17.473990  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:17.476642  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:17.477155  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:17.452448 14253 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:28:17.463917 14202 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.754s	user 0.008s	sys 0.007s
I20260709 12:28:17.462023 13917 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:17.552999  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:17.521977  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:17.523645  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 218394us
I20260709 12:28:17.524039  5635 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:28:17.553810  5629 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:17.526597 14246 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:28:17.527873 14241 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:28:17.516431  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:17.568271 14208 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:28:17.548235 14202 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:28:17.578591  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 105698us
W20260709 12:28:17.574155  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 127695us
I20260709 12:28:17.561102  5633 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
W20260709 12:28:17.554030 13950 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.557s	user 0.012s	sys 0.000s
I20260709 12:28:17.598435  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:17.561537 14246 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:17.613590  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.615209  5707 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 12:28:17.575402 14208 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:17.625046  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:17.620710  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:17.626557  5710 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:17.603626 13950 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:17.611584 13951 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:17.635607 14208 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:17.621234 14202 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:17.668211 13951 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.669317 14202 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.669819 14205 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:17.670415 14205 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.670785 13951 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:17.670920  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:17.671612  6277 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:17.673763 14021 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:17.674837 14021 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:17.677201 14172 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.738s	user 0.009s	sys 0.003s
I20260709 12:28:17.677520  6272 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: []
W20260709 12:28:17.678421 14255 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 12:28:17.678747  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:17.679811  6274 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:17.681006  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:17.683411 14013 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:28:17.683889 14013 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:17.684258  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:17.611791 14150 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:17.691260 14150 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.691603 14249 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:17.642693  5710 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:17.647010 14254 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:17.609083 14175 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.198s
I20260709 12:28:17.655980  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.656562  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.657307  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.658203  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.659945  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.660907  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.662083  5705 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260709 12:28:17.667532  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:17.688393 14156 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:17.688596 14013 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.640676  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:17.692732  5452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 275) took 67339 ms (client timeout 15000 ms). Trace:
I20260709 12:28:17.693233 14205 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:17.636519 14256 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:28:17.679823 14263 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:28:17.695617 14150 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:17.695784 14208 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:17.663635 14154 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.698814 13872 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:17.701112  6272 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: []
W20260709 12:28:17.705066 14200 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.941s	user 0.012s	sys 0.004s
I20260709 12:28:17.769338  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:17.768021  5704 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 23. Current term is 24. Ops: []
W20260709 12:28:17.770126  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:17.770501  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.770869  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:17.771560  5703 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:17.771658  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.705754  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.773126  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:17.773957  5703 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:17.723716  5559 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:17.775228  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.776383  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:17.776683 14000 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:17.777586  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.777346 14000 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.778960  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:17.779156 13743 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.723942  5558 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:17.781121  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:17.781598 13743 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:17.781775  5558 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:17.782692  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:17.725035  5552 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.568s	user 0.005s	sys 0.006s
W20260709 12:28:17.727991  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 466420us
I20260709 12:28:17.784588  5552 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:17.730062  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:17.786113  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:17.727710  5561 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:17.738106  5555 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 25 because replica has last-logged OpId of term: 24 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 11.
I20260709 12:28:17.716986 14218 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:17.716547 13996 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:17.788378  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:17.788775 14218 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:17.789147 13996 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:28:17.789891 14268 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:17.790143 14218 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:17.791527 13996 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:17.792002 13996 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:17.747993 13872 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:17.794673  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:17.756531  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:17.796885  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:17.797302  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44481us
W20260709 12:28:17.727941  5452 rpcz_store.cc:269] 0709 12:27:10.352749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:10.352907 (+   158us) service_pool.cc:224] Handling call
0709 12:27:11.449961 (+1097054us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5dc090 00007f4ffa67af3a 00007f5000430e9c 00007f500052f04e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc 00007f4ffe9ef392 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:17.692417 (+66242456us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:28:07.300410 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:28:07.353297 (+ 52887us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:28:07.353371 (+    74us) write_op.cc:435] Acquired schema lock
0709 12:28:07.353399 (+    28us) tablet.cc:662] Decoding operations
0709 12:28:07.404325 (+ 50926us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:28:07.460717 (+ 56392us) write_op.cc:641] Partition lock acquired for write op
0709 12:28:07.464204 (+  3487us) tablet.cc:685] Acquiring locks for 6 operations
0709 12:28:07.576651 (+112447us) tablet.cc:701] Row locks acquired
0709 12:28:07.576701 (+    50us) write_op.cc:260] PREPARE: finished
0709 12:28:07.576900 (+   199us) write_op.cc:270] Start()
0709 12:28:07.577071 (+   171us) write_op.cc:276] Timestamp: P: 1783600087576824 usec, L: 0
0709 12:28:07.577126 (+    55us) op_driver.cc:348] REPLICATION: starting
0709 12:28:07.577994 (+   868us) log.cc:844] Serialized 2418 byte log entry
0709 12:28:08.735568 (+1157574us) op_driver.cc:464] REPLICATION: finished
0709 12:28:08.994404 (+258836us) write_op.cc:301] APPLY: starting
0709 12:28:08.994511 (+   107us) tablet.cc:1367] starting BulkCheckPresence
0709 12:28:09.119966 (+125455us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa66f32a 00007f4ffa5f4f14 00007f4ffa5f4ec3 00007f4ffa5f4e38 00007f4ffa5f4df3 00007f4ffa5f40dc 00007f4ffa61de16 00007f4ffa605cbb 00007f4ffa5f039d 00007f4ffa5f3d75
0709 12:28:09.136312 (+ 16346us) tablet.cc:1370] finished BulkCheckPresence
0709 12:28:09.141052 (+  4740us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:28:09.437154 (+296102us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:28:09.437454 (+   300us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=12,key_file_lookups=12,delta_file_lookups=12,mrs_lookups=0
0709 12:28:09.437514 (+    60us) write_op.cc:312] APPLY: finished
0709 12:28:09.437861 (+   347us) log.cc:844] Serialized 65 byte log entry
0709 12:28:09.469539 (+ 31678us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:28:09.469671 (+   132us) write_op.cc:454] Released schema lock
0709 12:28:09.469741 (+    70us) write_op.cc:341] FINISH: Updating metrics
0709 12:28:09.470227 (+   486us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:28:09.470258 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1519321,"reported_tablets":50,"spinlock_wait_cycles":405904128,"child_traces":[["op",{"apply.queue_time_us":250070,"apply.run_cpu_time_us":15227,"apply.run_wall_time_us":475829,"cfile_cache_hit":26,"cfile_cache_hit_bytes":29207,"num_ops":6,"prepare.queue_time_us":779141,"prepare.run_cpu_time_us":19093,"prepare.run_wall_time_us":555017,"raft.queue_time_us":499641,"raft.run_cpu_time_us":47947,"raft.run_wall_time_us":435228,"replication_time_us":1145487,"spinlock_wait_cycles":23681920,"thread_start_us":1169391,"threads_started":4,"wal-append.queue_time_us":423259,"wal-append.run_cpu_time_us":34441,"wal-append.run_wall_time_us":2452708}]]}
I20260709 12:28:17.756857 14154 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:17.799441 14205 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:17.800110 14205 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:17.800588  5636 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:28:17.796314 13872 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.801996  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:17.802105 13872 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:17.802196 14125 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:28:17.756091 13861 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:17.804543 13861 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:17.805572  5635 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:17.759743 13812 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:17.766067 14266 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:17.766443  5710 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:17.766824  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:17.811496  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:17.812024  5705 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:17.767069  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:17.767987  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:17.729110 14244 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:28:17.768851  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:17.768792  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:17.773213  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.706632 14246 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.750691  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.816679  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:17.789151  5552 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:17.780169  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:17.796051  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.796129  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:17.797397  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:17.818558  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:17.799345  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:17.801385  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:17.805495  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:17.806818  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.762694 14256 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:17.763671  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:17.764444 14142 consensus_queue.cc:1175] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "38736cc545eb4389b220a3bd21d3b466" responder_term: 23 status { last_received { term: 21 index: 16 } last_committed_idx: 16 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: [22.17-22.17]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:28:17.827353  5701 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:17.762108 14249 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:17.802030 14261 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:17.815752  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:17.815961  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:17.816545  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:17.836309  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 54905.7us
I20260709 12:28:17.837885  5710 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:17.838300  5704 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:17.836726  5708 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:17.815519 13861 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:17.824895  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:17.823839  5556 raft_consensus.cc:2376] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22. Re-sending same reply.
I20260709 12:28:17.827509  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:17.759935 13960 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.582s	user 0.004s	sys 0.004s
I20260709 12:28:17.843521  5555 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:17.820871  5557 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:17.805296 13996 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:28:17.812299  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 186888us
W20260709 12:28:17.840775  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:17.840124  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:17.841030  5706 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:17.847383  5557 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:17.843786  5561 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:17.851255  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:17.852217  5705 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:17.832270 14172 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:17.854876  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.855424  5701 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:17.828154 14266 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.841581 14125 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:17.872759  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:17.882160 14125 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:17.874786 13986 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.008s
I20260709 12:28:17.876046  5557 pending_rounds.cc:85] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 15
I20260709 12:28:17.883216  5557 pending_rounds.cc:107] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 22.16
I20260709 12:28:17.873618  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.866134  5556 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
W20260709 12:28:17.883988  5526 connection.cc:256] RPC call timeout handler was delayed by -1.1711053359999823s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31420000000000003s for a response
I20260709 12:28:17.885583  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:17.883643  5557 raft_consensus.cc:2889] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:17.887265 13969 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:28:17.876930  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:17.886380 14150 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:17.872982  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.889586  5561 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:17.860426 14261 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:17.890617  5561 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:17.878451  5634 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 12:28:17.891767  5634 pending_rounds.cc:85] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 14
I20260709 12:28:17.892093  5634 pending_rounds.cc:107] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 23.15
I20260709 12:28:17.868834  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:17.892206  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:17.863991  5560 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 25 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:28:17.889174 14150 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:17.894698  5552 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:17.892383  5634 raft_consensus.cc:2889] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:17.874552  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:17.903132  5553 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:17.891013  5561 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:17.888413  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:17.910043  5704 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:17.911634  5706 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:17.883307 13986 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:17.920435  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.928654  5556 raft_consensus.cc:2410] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:28:17.847488 14119 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.067s
I20260709 12:28:17.920933  5707 raft_consensus.cc:2410] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 25 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:28:17.869925 13861 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:17.922314  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:17.937042  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:17.920323  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 112455us
I20260709 12:28:17.823138 14205 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.914644 14200 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:17.914131 14246 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:17.850921 13960 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:17.949640  5629 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:28:17.943178  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:17.917315 14126 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:17.909165 14254 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.035s
I20260709 12:28:17.964011  5708 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 19.
I20260709 12:28:17.927968 13986 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:17.970278  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:17.971210  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 82947us
I20260709 12:28:17.972062  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:17.890509 14261 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.973505  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:17.974339 13969 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:28:17.975024 13969 raft_consensus.cc:2764] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:28:17.950791 14265 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.032s
I20260709 12:28:17.976406  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:17.977171 14233 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:17.977906 14000 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:17.978700  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:17.876605 14243 consensus_queue.cc:1175] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f613ba929b1046d3b5aed909414a95bf" responder_term: 23 status { last_received { term: 21 index: 16 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: [22.17-22.17]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:28:17.979502 14233 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:17.980484 14126 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:17.982882 14126 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:17.982586  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:17.985327 14233 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:28:17.985491  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51989.4us
I20260709 12:28:17.986304  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:17.987622  5526 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:17.989864 14233 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:17.990659 14269 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:17.990760 14265 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:17.992624 14259 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:17.992733 14233 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:28:17.993005 14262 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:17.994037 14259 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:17.883227 14125 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 27.13, Last appended by leader: 13, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:17.996829 14150 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:17.988060  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.021051  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.022779  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.024158 13866 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:18.025125 13866 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:18.025882 13866 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:17.951224 14021 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.017450  5556 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:17.947219 14038 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:17.995728  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:17.949453 14200 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:17.929162 14172 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.921751 14257 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.046s
W20260709 12:28:18.017352  5602 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.607s (SENT) [suppressed 48 similar messages]
I20260709 12:28:17.946171  5561 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:17.994366 14262 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:18.042719  5561 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:18.038861  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:17.994416 13969 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:17.999634 14259 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:18.040856  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.034497 14200 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:17.879504 14256 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:28:18.046677  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 51744us
I20260709 12:28:18.052541  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:18.054255  5627 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:18.055374 14125 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:18.056108 14266 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:18.057192 14262 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.058502 14262 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:18.060398  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 94534.9us
I20260709 12:28:18.061967  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.063325  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.044445 13969 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:18.069037 14181 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:18.069497 14181 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.069834 14181 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:18.070777 14021 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.073047 14021 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:18.077181 13960 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:18.086216 14272 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:28:18.089377  5558 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:18.090107  5555 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
W20260709 12:28:18.093307  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:18.093751  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:18.101819  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.097065  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:18.103274  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:18.109647  5555 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:18.119617  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:18.052783 14119 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:18.105679  5558 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:28:18.122429  5559 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:18.125392  5560 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:28:18.123409  5558 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: [25.12-27.13]
I20260709 12:28:18.124141  5555 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:28:18.126729  5560 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 14. (term mismatch)
I20260709 12:28:18.127055  5560 pending_rounds.cc:85] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 13
I20260709 12:28:18.127290  5560 pending_rounds.cc:107] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 21.14
I20260709 12:28:18.127550  5560 raft_consensus.cc:2889] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:28:18.132889 14253 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.584s	user 0.011s	sys 0.000s
I20260709 12:28:18.133360 14253 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.135599 14253 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:18.119630  5629 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:18.092495 14246 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.137748 14246 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.138301 14172 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:18.119920 14266 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:18.139189  5554 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 24. Ops: []
I20260709 12:28:18.102897 14175 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:18.142690 14246 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:18.143268 14273 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.173s	user 0.001s	sys 0.001s
I20260709 12:28:18.145089  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1745) took 1044 ms. Trace:
I20260709 12:28:18.145624  5633 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: [21.11-21.11]
W20260709 12:28:18.149092  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 85842.9us
I20260709 12:28:18.150298  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:18.151981  5630 rpcz_store.cc:276] 0709 12:28:17.100692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:17.100838 (+   146us) service_pool.cc:224] Handling call
0709 12:28:17.122658 (+ 21820us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:17.141057 (+ 18399us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:17.141130 (+    73us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:17.685706 (+544576us) spinlock_profiling.cc:243] Waited 4.36 ms on lock 0x7b2c00000de8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:28:17.687446 (+  1740us) log.cc:844] Serialized 26 byte log entry
0709 12:28:17.836762 (+149316us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:17.836819 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:17.836935 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:18.144792 (+307857us) raft_consensus.cc:1685] finished
0709 12:28:18.144813 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:18.145029 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7235,"raft.queue_time_us":11424,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":7533,"spinlock_wait_cycles":10030848,"thread_start_us":148811,"threads_started":1,"wal-append.queue_time_us":364493}
I20260709 12:28:18.104946  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:28:18.161841  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 56969.3us
I20260709 12:28:18.162573  5561 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
W20260709 12:28:18.166235  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 134002us
W20260709 12:28:18.171198  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:18.172686  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.174151  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.176913  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:18.177772  5704 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:18.166100 14246 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:18.187043  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:18.161060 13996 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.180775 14268 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:18.189509 14246 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.193667 14266 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:28:18.212890  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:18.215229  5561 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:28:18.217370  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:18.223723  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 56623.7us
I20260709 12:28:18.224910  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.226403  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:18.207254 14269 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:18.215829 14246 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:18.236065  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:18.243913  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 69849.7us
I20260709 12:28:18.235662 14119 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:18.256086  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.243064 14126 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.242859 14253 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:18.246279 14150 raft_consensus.cc:2764] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:28:18.245987 13953 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:18.262039 14244 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:28:18.262228 13953 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:18.263005  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 161246us
I20260709 12:28:18.263753  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.263494 13986 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.264729 14021 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.265257 14021 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.265808  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:18.266629 14142 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.266692  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:18.267252 14021 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:18.267210 14142 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.267956  5708 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 18. Current term is 19. Ops: []
I20260709 12:28:18.259683  5709 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:18.250077 14172 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:28:18.268875  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:18.258580 14283 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
W20260709 12:28:18.284636  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55040.8us
I20260709 12:28:18.276664  5558 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:18.282322  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.305980  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.271425  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:18.307473  5706 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:28:18.280062  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:18.308071  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:18.308337  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:18.271944  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:18.241876 14200 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.320739 14156 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:18.321326 14156 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:18.322110 14142 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:18.322356 14156 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.323096 14142 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.323661 14142 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.283947  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:18.306097  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:18.275573 14277 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.280s	user 0.000s	sys 0.002s
I20260709 12:28:18.305450  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:18.285897  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:18.268393 14265 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:18.316919 14202 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:28:18.318506  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:18.320626  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:18.309921  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:18.282786 14253 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.353310 14233 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:18.353368 14119 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:18.353966 14119 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.354566 14119 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:18.354681  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:18.355315 14253 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:18.353586  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:18.357507 14205 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:28:18.358052 14205 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:18.358178  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 94495.4us
I20260709 12:28:18.359004  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.359045 14205 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.327049 13986 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:18.361855 14013 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:18.259498 14126 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.362181 14142 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:18.366472 14142 raft_consensus.cc:2764] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:28:18.276229 13960 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.049s
W20260709 12:28:18.341614  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:18.280264 14243 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.270341 13953 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:18.368979  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:18.270332 14117 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:18.369318  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:18.369865  6277 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:18.270464 14154 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:18.274436 14259 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.347294  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:18.371287  5629 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:18.368374 14243 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.364392  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:18.367765  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:18.370684 14154 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:18.269703  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:18.345834  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:18.278023 13866 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:28:18.352279  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 66491.4us
W20260709 12:28:18.352771  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47413.8us
W20260709 12:28:18.271067 14282 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 12:28:18.354061  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:18.366546 14233 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:18.369922 14117 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:18.368840  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:18.350600  5628 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 17. (term mismatch)
I20260709 12:28:18.401257  5628 pending_rounds.cc:85] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 16
I20260709 12:28:18.401517  5628 pending_rounds.cc:107] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 24.17
I20260709 12:28:18.273677 13969 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.364189  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.344769  5554 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:18.406275  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:18.374055  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 104923us
I20260709 12:28:18.388299  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:18.407585  6271 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:18.351538 14256 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:18.281216 13996 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:18.390592  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44794.2us
I20260709 12:28:18.366981 14126 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:18.396461  5706 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:18.394841  5632 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:28:18.401597  6274 raft_consensus.cc:2410] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:28:18.401674  6277 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:28:18.401762  5628 raft_consensus.cc:2889] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:18.417806  5628 pending_rounds.cc:107] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 26.18
I20260709 12:28:18.418107  5628 raft_consensus.cc:2889] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:18.416536  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:18.402562  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:18.415907  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:18.418828  6276 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:28:18.372198  6272 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:18.408797 13969 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:18.408842 14117 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:18.410290  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 128057us
I20260709 12:28:18.410822 14038 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.005s
I20260709 12:28:18.275868 14181 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.419477  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:18.394150 13743 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:18.389326 14266 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.400521 14175 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:18.424795  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:18.403308 14233 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:18.426849  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:18.429157  5559 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:28:18.429962  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:18.432490  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.431424  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.418510 14268 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:18.418926 14142 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:18.438967  5707 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 22 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:28:18.426491 14208 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:18.425840 13866 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:18.437136  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:18.413609 14183 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.021s
I20260709 12:28:18.443737  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:18.443495  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:18.444864  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 80792.9us
I20260709 12:28:18.416980 14265 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.021s
I20260709 12:28:18.432103 13812 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:18.430990 13743 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:18.448462  5706 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:18.426615 14150 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.017s
I20260709 12:28:18.457933  5710 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:18.443535 13953 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:18.458662  5703 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:18.440994 14126 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:18.468919 14243 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:18.469022 14126 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:18.435484 13861 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.014s
I20260709 12:28:18.470603 14200 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:18.426086 14154 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.471935 13953 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.472461 13953 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:18.460314 13743 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:18.454669 13866 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.476933 13866 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:18.461844  5708 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:18.463568  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:18.462234 14265 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:18.467296  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:18.479138 14265 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:18.480227 14265 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.470535 14126 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.468281 14150 consensus_queue.cc:1243] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 25.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:28:18.437767 14266 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.485165  5634 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:28:18.494153 13917 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.494725 13917 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.437872 13996 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.020s
W20260709 12:28:18.461488  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 66928.7us
I20260709 12:28:18.505389  5553 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:18.506125  5629 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
W20260709 12:28:18.500150  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:18.504451  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53749us
I20260709 12:28:18.519744  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.520167  5632 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:28:18.514467  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:18.523711  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 91349.1us
I20260709 12:28:18.530987 14119 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.531352  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.498472 14266 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:18.500044 14265 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:18.535498  6279 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 28: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:18.496317  5459 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 20 to 24. New cstate: current_term: 24 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:18.542232 14202 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.008s
I20260709 12:28:18.551086 14202 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:18.551470 14202 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:18.552384 14202 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.553205  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:18.550326 13917 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:18.552444  6277 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: [27.20-27.20]
W20260709 12:28:18.546780  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:18.528847 13812 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:18.565711  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:18.567467  6275 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:18.511806 14038 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:28:18.593488  5632 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:18.595373 14183 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:18.595870 14183 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:18.596460  5635 raft_consensus.cc:1813] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Handling vote request from an unknown peer cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.597185  5635 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260709 12:28:18.592182  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 46959.9us
I20260709 12:28:18.591344  5459 catalog_manager.cc:5697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 19 to 27. New cstate: current_term: 27 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.600344  5459 catalog_manager.cc:5697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 19 to 22. New cstate: current_term: 22 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:18.542763 14297 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:18.564886 14175 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:18.618911 14246 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.613188  5459 catalog_manager.cc:5697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 14 to 20, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 20 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
W20260709 12:28:18.622768  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:18.621162  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:18.608186 14202 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:18.588768 14269 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.043s
I20260709 12:28:18.623369  6273 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:18.630456  5459 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 16 to 22, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 22 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:28:18.636288  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:18.650810  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:18.652676  5459 catalog_manager.cc:5697] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 14 to 17, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 17 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.656587 14208 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:18.659642 14280 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.466s	user 0.000s	sys 0.006s
I20260709 12:28:18.661952 14208 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:18.664740 14291 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 12:28:18.659967  5459 catalog_manager.cc:5697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 18 to 23, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 23 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:28:18.665438  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:18.666322  5459 catalog_manager.cc:5697] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 14 to 20. New cstate: current_term: 20 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:18.641762 14119 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:18.668816 14299 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:18.667692  5459 catalog_manager.cc:5697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 13 to 17, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 17 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:28:18.669112  6274 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:18.669747 14175 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 12, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.670171  5459 catalog_manager.cc:5697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 13 to 19. New cstate: current_term: 19 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:28:18.671617 14268 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:18.671813  5459 catalog_manager.cc:5697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 16 to 18, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 18 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:28:18.674888 14259 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:18.674986  5459 catalog_manager.cc:5697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 16 to 19. New cstate: current_term: 19 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.646996 14181 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:18.694350  5459 catalog_manager.cc:5697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 19 to 24. New cstate: current_term: 24 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:28:18.704516  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:18.705212  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.652319 13953 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.716560  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.733911  5459 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 17 to 18, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 18 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.710969 14304 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:18.737231  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 296888us
I20260709 12:28:18.740252  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.737169  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:18.742202 14200 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:28:18.743742 14295 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 12:28:18.737810 14208 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:18.745337  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:18.746183  5557 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 26: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:18.736341 14251 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:18.746970 14251 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.747566 14251 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:18.751291 14256 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.751870 14256 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.754011 14256 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:18.753887  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.757148  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.758816 14296 raft_consensus.cc:2764] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:28:18.759609 14296 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:28:18.761289 14276 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.788s	user 0.004s	sys 0.002s
I20260709 12:28:18.762492  5459 catalog_manager.cc:5697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 16 to 19, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 19 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:18.743963 13953 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:18.653360 13812 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.766747 14246 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:18.767992  5459 catalog_manager.cc:5697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 10 to 18. New cstate: current_term: 18 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:18.772921  5631 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:18.779227  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:18.780859  5710 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:28:18.782689  5702 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 12:28:18.784195  5701 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:18.785302  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:18.789363 13743 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.790992  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:18.791893  5556 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:18.769485 14304 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.792994 14257 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.793507 14257 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.769585 14299 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.795233 14299 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.795727 14257 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:18.797765  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:18.798008  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:18.802296  6274 raft_consensus.cc:606] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.802886  6274 raft_consensus.cc:614] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:18.810295 14256 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:18.773736 14208 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.819968  5459 catalog_manager.cc:5697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 15 to 18, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 18 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.835772  6271 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:18.843830  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.845727  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.849488  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:18.850716  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 71631.8us
I20260709 12:28:18.791898 13743 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:18.854211  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:18.858946  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 220663us
I20260709 12:28:18.852264  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:18.860630  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.855157  5459 catalog_manager.cc:5697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 15 to 21. New cstate: current_term: 21 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:18.861909  5459 catalog_manager.cc:5697] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 18 to 21, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 21 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:18.815846 14233 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.858562  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:18.863687  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.854650  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:18.864292  5560 raft_consensus.cc:2410] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:28:18.864745  5710 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
W20260709 12:28:18.866092  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 334796us
I20260709 12:28:18.870028  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:18.868857  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 355711us
I20260709 12:28:18.858832 14253 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:18.873724  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:18.856302 14000 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:18.870958  5599 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:18.874372  5459 catalog_manager.cc:5697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 17 to 19, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 19 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:28:18.863961 14265 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:18.851074 14309 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:18.875108 13861 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:18.865481 14256 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:28:18.878319  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.880609  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.880687  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:18.879487  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:18.885936  5628 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:18.886016  6279 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:18.867909 14154 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.869130 14202 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:28:18.880043 14257 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:18.888200 14013 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:18.888273 14257 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.888764 14013 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:18.888875 14257 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:18.889222 14013 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:18.881369 13861 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.891229 13861 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:18.896215  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:18.896767 14294 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:18.896515  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:18.899497 14294 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:18.900817  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:18.855664 13866 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.905107  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:18.877403 14296 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:18.866858 14244 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.905961  5635 raft_consensus.cc:2410] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 24 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:28:18.883088  5459 catalog_manager.cc:5697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 13 to 22. New cstate: current_term: 22 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:18.897475 14251 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.900249 13861 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.885118  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:18.877650 14261 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:18.908195  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.915954 14261 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:18.911183  5633 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:28:18.916445 14261 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:18.916906 14304 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:18.906121 14244 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:18.919009 14244 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:18.913074 14246 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:28:18.915038 14308 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.219s	user 0.002s	sys 0.000s
I20260709 12:28:18.911705  5459 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:18.937171  5628 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:18.864972 14218 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.041s
I20260709 12:28:18.942138  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.934530  5459 catalog_manager.cc:5697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 20 to 25, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 25 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:28:18.944571  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 59524.1us
W20260709 12:28:18.946679  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:18.948848  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 48108.5us
I20260709 12:28:18.945547  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:18.873430 14208 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.883772 14175 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260709 12:28:18.956503  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:18.957978  5459 catalog_manager.cc:5697] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 17 to 19, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 19 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:18.922827 14117 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:18.952888 14265 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.009s
I20260709 12:28:18.961621 13969 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:28:18.961880  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53770.8us
I20260709 12:28:18.962143 13969 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:18.962591 13969 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:18.962734  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:18.963531  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 443854us
I20260709 12:28:18.942497 14296 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:18.968271  5459 catalog_manager.cc:5697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 17 to 20. New cstate: current_term: 20 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.886300 14309 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.957671 13953 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:18.966081 13743 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:18.910996 14154 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:18.961057 14262 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:18.986927  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:18.977344  6274 pending_rounds.cc:85] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 18
I20260709 12:28:18.988960  6274 pending_rounds.cc:107] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 27.19
I20260709 12:28:18.989352  6274 raft_consensus.cc:2889] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:18.998773  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:18.999632  5554 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:28:18.971455 14268 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.971097 14304 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:18.980113 13953 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:19.009629  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:18.980046 14294 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:28:19.010023  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45568.2us
I20260709 12:28:18.981695 14117 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:18.981823 14296 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:19.012661  5554 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:28:19.012100  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:19.025290  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:18.992986 14218 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:19.027349  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.027987  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:19.028196  5702 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:19.007901 14243 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:19.037549  6272 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:19.017259 14294 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:19.064759  6272 raft_consensus.cc:606] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:28:19.065173  6272 raft_consensus.cc:614] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:19.065446 14294 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:19.025960  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:19.067272 14294 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:19.067390  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44280.7us
I20260709 12:28:19.068281 13917 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:19.017913 14268 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.068877 13917 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:19.069023 14296 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.069768 13917 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.070612 14268 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:19.039947  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.071329  5707 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:19.072252  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:19.038760  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 51918.7us
I20260709 12:28:19.019706 14117 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:19.073135 14117 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:19.073331  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:19.076685  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:19.077100  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:19.043015 14309 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:19.046084  5702 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:28:19.049384 14208 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:19.051651  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:19.083959  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.084815  5705 raft_consensus.cc:2368] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
I20260709 12:28:19.051898  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.007197 14262 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:19.053258  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.054165 14154 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:19.055140 14233 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:19.056736  6275 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:28:19.057574  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:19.017185 13953 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:19.076520 14218 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:19.078234  5710 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: [27.20-27.20]
I20260709 12:28:19.073451  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:19.076685  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 131245us
I20260709 12:28:19.090195  6273 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:19.097522  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:19.097891  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:19.098274  6280 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:19.084573 14293 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:19.085289  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.092056  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.061862 14261 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:19.105865  6279 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:19.110217  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:19.110440  6277 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:19.110867  5702 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:19.109642  6276 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:19.100471  5709 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:19.068171 14257 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.111055  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:28:19.120893  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 43853.2us
I20260709 12:28:19.124864  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:19.125771  6278 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 26 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:28:19.126422  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:19.086088 14243 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 12:28:19.128011  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.129678  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:19.113919 14318 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:19.089869 14013 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:19.137919  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.117238 13743 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:19.148538  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.148883  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:19.150866  5707 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:19.151324  5524 leader_election.cc:400] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
I20260709 12:28:19.153606  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.154162  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:19.154946  6278 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 26 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:28:19.139003 14269 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:19.157377  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:19.155241  5704 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:19.147397 14038 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:19.166244  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.165140  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:19.167163  5703 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:19.140518 14319 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:19.174850  5707 raft_consensus.cc:2368] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
I20260709 12:28:19.170333  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:19.179195  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:19.142647 14233 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 12:28:19.182644  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:19.146651 14293 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
I20260709 12:28:19.150111 14218 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.029s
I20260709 12:28:19.184477  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:19.149436 14150 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.029s
W20260709 12:28:19.188445  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48195.7us
W20260709 12:28:19.193529  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 82590us
I20260709 12:28:19.177246 14319 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:19.155155 13743 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.197978  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.194677  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:19.198313  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:19.181620 14331 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:19.199787  6278 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:19.201741  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.215970  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:19.174724 14269 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.164976 14262 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:19.236018 14323 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20260709 12:28:19.236994  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.237641  5703 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:28:19.197810 14319 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:19.254554 14335 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:19.259416 14335 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:19.259908 14335 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:19.200733 14327 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:28:19.277534  6278 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8c6474db7a70405694509f6976d850ce"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:19.278155  6278 raft_consensus.cc:606] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:19.278506  6278 raft_consensus.cc:614] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:19.236070 14257 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:19.282397  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:19.290288 14326 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:28:19.291913 14324 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:28:19.292249 14013 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:19.293234 13743 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:19.293638 14335 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:19.297394 14325 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:28:19.298008  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:19.298840  6277 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:19.301635  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:19.302790 14013 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:19.303233 14013 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:19.303572 14013 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:19.302366 13969 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.276819 14253 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:28:19.331005  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 147611us
I20260709 12:28:19.331588 14330 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:28:19.350055 14330 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:19.347262  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:19.286912 14261 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:19.350237  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 192945us
I20260709 12:28:19.349481  5702 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:19.357110  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:19.370280  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.371735  5702 raft_consensus.cc:1813] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Handling vote request from an unknown peer cea66aa15be3423880843974acdbf50f
I20260709 12:28:19.372444  5702 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 19: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:28:19.389914  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 195291us
I20260709 12:28:19.376957 14013 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.396291 14013 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:19.397274 14233 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:19.393910  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:19.401937  5701 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:19.407408  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:19.408030  6277 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:19.409300  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:19.412624  5702 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:28:19.394198 14331 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:19.417266  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:19.420531 14183 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:19.422858 13969 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:19.424731 14330 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:19.453526  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 59873.1us
I20260709 12:28:19.414103 14338 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:19.398825 13812 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:19.403626 14341 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:19.439574  5702 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:28:19.449020 14253 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:19.449446  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:19.450171  5629 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:19.427939 14117 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:19.455271  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:19.461822  5634 raft_consensus.cc:2410] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 28 because replica has last-logged OpId of term: 27 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
I20260709 12:28:19.456846  6274 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:28:19.462576 14342 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:19.462883  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:19.463165 14342 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.463966 14117 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:19.464468 14299 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:19.465909  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:19.466508  5636 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:19.467059  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:19.458011 14327 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:19.467864  5631 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:19.468789  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:19.469142  5633 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:28:19.469465  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:19.458151 14332 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:19.470052  5627 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:19.470350  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:19.470041  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:19.470567 14332 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.471035  5635 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:19.461771 14183 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:19.473568  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:19.474161  5634 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:19.474107  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:19.474802  5703 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:19.477061  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:19.478514 14253 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.480858 14253 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:19.481971  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:19.459703 14341 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:19.484633 13866 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:19.489143 14150 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:19.489715 14150 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:19.490453 14297 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:19.491523  5634 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:19.492754 14175 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:19.492650 14309 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:19.493505 14175 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:19.493907 14175 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:19.494912 14309 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:19.495451 14309 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:19.495867  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.496680 14343 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:28:19.498682  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:19.498284 14309 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.499266  5706 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:19.501453 14332 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:19.503499  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:19.503273 13812 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.505854 14331 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:19.507534 14142 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:28:19.507948 14142 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:19.508847 14142 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.513134 14269 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:19.513543  6278 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:19.514087  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.514410 14297 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:19.514760  5702 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:19.515048 14297 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:19.484302 14341 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.514115  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:19.516880  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 51363.5us
W20260709 12:28:19.517664  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68298.1us
I20260709 12:28:19.522823 14299 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:19.525897  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:19.525909  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:19.527208  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:19.528297  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:19.529635  6272 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
W20260709 12:28:19.533054  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:19.514616 14181 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:19.499948 14309 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:19.545189  6277 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:19.558687  5706 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
W20260709 12:28:19.567477  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 109095us
I20260709 12:28:19.618605  5634 raft_consensus.cc:1217] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Deduplicated request from leader. Original: 22.15->[24.16-24.16]   Dedup: 24.16->[]
I20260709 12:28:19.621665  5634 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:28:19.619104  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:19.622592  5629 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:19.622721  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:19.624150  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:28:19.615267  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 89419.1us
I20260709 12:28:19.627231  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:19.627753  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:19.628083  5554 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 29 because replica has last-logged OpId of term: 27 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 19.
W20260709 12:28:19.629261  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:19.626382  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:19.528652 14117 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:19.635468  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:19.543105 14181 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:19.638684  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:19.641619  5553 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:19.545073 14013 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:19.593616 14304 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:19.630295 14299 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:19.643630  5554 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20260709 12:28:19.673957  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 316962us
W20260709 12:28:19.639205 13953 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.545s	user 0.010s	sys 0.001s
I20260709 12:28:19.674353  5629 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:28:19.460397 14338 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.632819 14243 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:19.529835 14335 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.111s
I20260709 12:28:19.634734 14117 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:19.703578  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:19.683257 14294 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.615s	user 0.010s	sys 0.006s
I20260709 12:28:19.675104 14304 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:19.633447 14332 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:19.643424 14336 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:19.686465  5554 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:28:19.561290 14266 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.113s
W20260709 12:28:19.687997  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 61696.3us
I20260709 12:28:19.688778 14243 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:19.689154  5552 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:28:19.719839  5599 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:19.692813  5706 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:28:19.657506 14256 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:19.680141  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:19.706436 14117 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:19.709463  5560 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 28 index: 12. (index mismatch)
I20260709 12:28:19.713023  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:19.713603  5707 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:28:19.713668  5708 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:19.715845  5634 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:19.717636  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 93608.1us
I20260709 12:28:19.490612 14150 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.723044  5553 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:19.724501  5556 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: []
I20260709 12:28:19.692804  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:19.681325  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49158.9us
I20260709 12:28:19.735877  5702 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:19.736119  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:19.736708  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:19.713406 14304 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:19.680794 14013 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:19.742169 14150 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:19.723979 14296 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:19.688287 13953 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:19.725044 14336 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:19.735215 13743 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:28:19.747184 14349 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:28:19.747473 14347 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:19.749943  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:19.787801  5710 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:19.752835  5627 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:28:19.740243 14256 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:19.752717 14126 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:19.753207 14327 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:28:19.754132  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:19.754833 13969 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:19.756713 13861 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:19.758682 14342 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:19.764421 14338 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:19.798952  5632 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 13. Preceding OpId from leader: term: 28 index: 12. (term mismatch)
I20260709 12:28:19.799355  5632 pending_rounds.cc:85] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 11
I20260709 12:28:19.799679  5632 pending_rounds.cc:107] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 25.12
I20260709 12:28:19.800014  5632 raft_consensus.cc:2889] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:19.800441  5632 pending_rounds.cc:107] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 27.13
I20260709 12:28:19.800870  5632 raft_consensus.cc:2889] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:19.764667 13960 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:28:19.766008  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:19.767700  5703 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:19.769687  5636 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:19.696112 14297 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.806521  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:19.729825 14243 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:19.774866  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.776111  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:19.779237  5634 raft_consensus.cc:2410] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 24 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:28:19.770349 14266 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:28:19.763834 14181 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:19.728626 14332 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.744556 14317 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
I20260709 12:28:19.771243 14175 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.793818 14126 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:19.805143  5705 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260709 12:28:19.807739  5635 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:28:19.810444  5707 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 19.
I20260709 12:28:19.812002  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:19.830003  5627 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:19.822786  5524 leader_election.cc:400] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
I20260709 12:28:19.860395  5524 leader_election.cc:403] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:28:19.781875 14294 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:28:19.798583 13743 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:19.805335 14322 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:19.869971  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.876590  5709 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:19.804594 14327 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:28:19.866731 14349 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:28:19.843321 14299 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:28:19.771952  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 68437.5us
I20260709 12:28:19.813588 14320 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:28:19.835117 14319 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.582s	user 0.011s	sys 0.000s
I20260709 12:28:19.872650 14322 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:19.813433 14251 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:19.827459 14342 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:19.884214  5710 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:19.793460 14347 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:19.887826  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 65171.4us
W20260709 12:28:19.890161  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:19.859817 14266 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:19.816632 14358 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:19.812260 14013 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:28:19.868301 14296 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:19.804152 13969 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:19.822036 13953 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:19.879458 14317 consensus_queue.cc:1243] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 26.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:28:19.823526 14336 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:19.901980  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:19.846648 14343 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:28:19.901875  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:19.926216  5627 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:28:19.926893 14294 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:28:19.926671 14266 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.927320  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:19.896631 14183 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
I20260709 12:28:19.927948 14266 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:28:19.914831  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 102930us
I20260709 12:28:19.913956 14256 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.916869  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:19.935295  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:19.935899 14183 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:28:19.936337 14317 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:19.927973  5627 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:19.937089  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:19.937212 14317 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:19.898260 14320 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:19.906029 14322 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:19.938802 14320 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:19.938301 14317 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.905282 14361 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260709 12:28:19.924170  5707 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:19.947557 14183 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:19.947619 14318 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:28:19.910959  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.948300 14183 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:19.948417 13743 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:19.949199  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.949466 14183 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:19.950281  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:19.951148 14318 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:28:19.918059 14342 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:19.965785  5707 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:19.904918 14319 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:19.952184 14233 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:19.905972 14126 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:19.974148  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:19.976227  6278 raft_consensus.cc:606] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:19.984552  6278 raft_consensus.cc:614] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:19.990283  5636 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:19.967002 13960 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:19.993569  5557 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:19.974144 13861 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:20.004908  5706 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:19.974746 14233 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:20.010697  5707 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: []
W20260709 12:28:20.008318 14330 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.010s	sys 0.010s
I20260709 12:28:20.016912  5706 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:28:19.997388 14126 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:19.997337 14172 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.026050  5707 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:20.006314 14150 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:20.032858  6272 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:28:20.008102 14343 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:19.994150 14304 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.035480  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:20.022994 14251 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.021859 14332 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:19.998536 14354 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.165s
I20260709 12:28:20.044077  5706 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:28:20.043887  6276 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:20.052997  5707 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
W20260709 12:28:20.024410 14330 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.011s	sys 0.010s
I20260709 12:28:20.032401 14175 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:20.068006  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.010466 13960 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.065092  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:20.072927  5633 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:19.973901 14327 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:20.087291  5707 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:28:20.077061  5627 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:28:20.070209 14344 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:20.078970 14368 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:28:20.080199 13969 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:20.081004 14347 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:20.082187 14304 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:20.071022 14233 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:20.093008 14233 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:20.095358 14117 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.093052 14320 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:20.097819 14117 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:20.039229 14342 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:20.099210 14320 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:20.044308 14343 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.101001 14319 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:20.084811 14374 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:28:20.084939  5630 raft_consensus.cc:2368] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21.
I20260709 12:28:20.066059 14251 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.086403  5706 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:28:20.034789 14156 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:20.109275  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:20.063313 14183 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.035370 14172 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.106756  5707 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
W20260709 12:28:20.107159  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.107651  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:20.072192 14354 consensus_queue.cc:1243] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 22.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:28:20.038995 14150 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.034963 13866 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
W20260709 12:28:20.117766  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.114065  6276 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:20.115844  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:20.118578  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 99886.1us
I20260709 12:28:20.119371  5707 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:20.113168  5627 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:28:20.126740  6277 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:20.146509  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:28:20.116798 14244 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.145s	user 0.018s	sys 0.035s
W20260709 12:28:20.152021  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.153537  5706 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
W20260709 12:28:20.147433 14334 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.750s	user 0.014s	sys 0.014s
I20260709 12:28:20.108783 14344 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:20.152199 14244 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.182s	user 0.018s	sys 0.036s
I20260709 12:28:20.180441 13953 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.181071 13953 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:20.184948 14371 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
W20260709 12:28:20.185949 14367 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20260709 12:28:20.194216  5635 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:20.152787 13917 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:20.203722  5633 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:20.194841  5627 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
W20260709 12:28:20.161646 14334 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.764s	user 0.015s	sys 0.015s
I20260709 12:28:20.161638 14293 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:20.208142  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:20.118700 14183 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.125496 14156 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.192715 14181 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:20.203078 13917 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:20.254696  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 46645.1us
I20260709 12:28:20.255087  5635 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:28:20.299048  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:20.299852  5635 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 27: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:28:20.306455  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 95842.7us
I20260709 12:28:20.296530 13917 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:20.207219 14317 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.095s
I20260709 12:28:20.295964 13812 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:20.299945 14208 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.298596 13953 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:20.299963 14380 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:20.304638 14343 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:20.325409  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.336138  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.327121  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.327638  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.310602 14251 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:20.338487  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.339344  5557 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:20.339884  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.341459  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.216145 14293 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.338265  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.344179 14385 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.344974 14385 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:20.347409 14344 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:20.348693  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:20.349830  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.348905 14385 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:20.350837  5556 raft_consensus.cc:2368] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25.
I20260709 12:28:20.355142 14251 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.355708 14251 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.357937 14251 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:20.358446 14380 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:20.359386 14380 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.362933  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:20.363073 14322 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:20.344326  5555 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:20.351999  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.370380  5602 leader_election.cc:400] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25.
I20260709 12:28:20.370129  5552 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:28:20.414297  5602 leader_election.cc:403] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:28:20.380344  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.374095  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:20.376044  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:20.390038  5561 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
W20260709 12:28:20.405342  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.417237  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:20.303656 14205 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.030s
I20260709 12:28:20.335319 14150 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:20.418565  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:20.369118 14377 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
W20260709 12:28:20.428391  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58138.3us
I20260709 12:28:20.434227  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:20.428666  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.435343  5552 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:20.429988  5557 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:20.431246  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:20.431205  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.378368 14317 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:20.437414  5554 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 22: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:20.437193 14317 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:20.440665  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.441361  5560 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:20.441083 14317 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:20.442312 14317 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:20.418880 14208 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:20.454710 14208 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:20.364132 13953 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:20.456094 14183 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:20.457970  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3226) took 1401 ms. Trace:
I20260709 12:28:20.458516  5702 rpcz_store.cc:276] 0709 12:28:19.056405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:19.529205 (+472800us) service_pool.cc:224] Handling call
0709 12:28:19.628274 (+ 99069us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:19.736292 (+108018us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:20.457591 (+721299us) spinlock_profiling.cc:243] Waited 85.1 ms on lock 0x7b5400324454. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:20.457883 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":195817728}
I20260709 12:28:20.379225 14172 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:20.377825 14156 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:20.460686  5555 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:20.461220  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.420643 13866 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:20.484928  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 50847.1us
I20260709 12:28:20.485108 13866 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.485968 13861 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:20.486454 13861 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:20.486598 14126 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:20.487519 13866 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:20.487366 14126 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:28:20.479885  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.482087  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3231) took 1422 ms. Trace:
I20260709 12:28:20.482616  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:20.483333  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.483444 14322 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:20.491140  5710 rpcz_store.cc:276] 0709 12:28:19.059908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:19.951059 (+891151us) service_pool.cc:224] Handling call
0709 12:28:19.951352 (+   293us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:19.951770 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:28:19.951800 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:19.952302 (+   502us) log.cc:844] Serialized 26 byte log entry
0709 12:28:20.063136 (+110834us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:20.063195 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:20.063255 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:20.481837 (+418582us) raft_consensus.cc:1685] finished
0709 12:28:20.481863 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:20.482017 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":693,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":5024,"spinlock_wait_cycles":1362688,"thread_start_us":93937,"threads_started":1,"wal-append.queue_time_us":185185}
I20260709 12:28:20.492268  5557 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:20.487393  5555 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:20.495924  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.475871 14205 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.517343 14205 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.519167  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:20.484215 14331 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:20.520532 14331 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:20.521785 14331 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:20.523093 14331 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:20.523322  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.424837 14329 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25.
I20260709 12:28:20.496667  5555 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:20.484153  5558 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 22 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:28:20.495064  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.525931  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:20.526186  5557 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:28:20.498322 14339 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:20.529664  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:20.537302  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:20.529786  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:20.530602  6274 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:28:20.542470  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:20.551133  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:20.552452  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:20.560442  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:20.561904  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:20.566861  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:20.568591  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:20.527350 14339 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:20.527657 14319 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:20.581632  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:20.549356 14243 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.678s	user 0.008s	sys 0.004s
I20260709 12:28:20.587102  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:20.587510  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:20.598482  6277 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:20.583904  5559 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:20.592067  6276 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:28:20.561162 13953 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:20.595712  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:20.603933 14253 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:20.604265 14233 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:20.604440 14253 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:20.604851 14233 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:20.604946 14253 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:20.605279 14233 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:20.594759 14154 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:20.607214 14154 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:20.607587 14154 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:20.597260 14332 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:20.610219  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:46870 (request call id 10) took 1096 ms. Trace:
I20260709 12:28:20.611321  6278 rpcz_store.cc:276] 0709 12:28:19.513232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:19.513379 (+   147us) service_pool.cc:224] Handling call
0709 12:28:20.609585 (+1096206us) spinlock_profiling.cc:243] Waited 462 ms on lock 0x7b54002ce7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa66c853 00007f4fff63a60a 00007f4ff8bc07d4 00007f4ff8bc0762 00007f4ff8bc0691 00007f4ff8bc060c 00007f4ff8bbf8c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:20.609920 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1062713216}
W20260709 12:28:20.614459  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:20.615257  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.600239  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:20.574317 14205 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:20.582966 14244 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:20.600344  6272 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: [27.18-27.18]
I20260709 12:28:20.601419  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:20.605682  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.626278  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:20.617923  6274 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:20.624938 14332 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.629596 14332 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.600834 13953 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:20.633082 13953 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:20.637823  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:20.638556  6278 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:20.643417  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:20.644254  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:20.645023 14337 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:20.645535 14337 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:20.645938 14337 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:20.648912 14320 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:20.597474 14342 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:20.592409 14329 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:20.650189 14117 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:20.660797 14329 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:20.661216 14117 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:20.661480 14329 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:20.661724 14117 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:28:20.663275  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.666734  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:20.655417  6278 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:20.596740 14319 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.656481 14342 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:20.675673 14294 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:20.599864 14243 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:20.704692  6272 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:20.705525  6272 raft_consensus.cc:606] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:28:20.697676  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1769) took 1080 ms. Trace:
I20260709 12:28:20.683071 14172 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:20.706864  5631 rpcz_store.cc:276] 0709 12:28:19.617501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:19.619672 (+  2171us) service_pool.cc:224] Handling call
0709 12:28:19.619967 (+   295us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:19.634204 (+ 14237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:19.634237 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:19.641261 (+  7024us) log.cc:844] Serialized 26 byte log entry
0709 12:28:20.689791 (+1048530us) spinlock_profiling.cc:243] Waited 5.55 ms on lock 0x7b1800345f40. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1
0709 12:28:20.690018 (+   227us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:20.690077 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:20.690181 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:20.690238 (+    57us) raft_consensus.cc:1685] finished
0709 12:28:20.690249 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:20.697595 (+  7346us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13858,"raft.queue_time_us":87035,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":6228,"spinlock_wait_cycles":12768512,"thread_start_us":22720,"threads_started":1,"wal-append.queue_time_us":144753}
I20260709 12:28:20.715456  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:20.716096 14322 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:20.721556 14322 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.726712  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:20.695344 14205 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:20.729656  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.696393 14342 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:20.678859 14156 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:28:20.732069  5629 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:28:20.738232  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.734347  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:28:20.745187  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:20.747807  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:20.724275 14322 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:20.688021 13917 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.742251  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:20.684211 14339 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.764732  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3227) took 1708 ms. Trace:
I20260709 12:28:20.770787  5705 rpcz_store.cc:276] 0709 12:28:19.056683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:19.531022 (+474339us) service_pool.cc:224] Handling call
0709 12:28:19.804801 (+273779us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:19.859450 (+ 54649us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:20.764480 (+905030us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54002d0ad0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:20.764667 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":628846976}
I20260709 12:28:20.772006  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 1771) took 1146 ms. Trace:
I20260709 12:28:20.772420  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:20.772392  5634 rpcz_store.cc:276] 0709 12:28:19.625780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:19.627522 (+  1742us) service_pool.cc:224] Handling call
0709 12:28:20.771816 (+1144294us) spinlock_profiling.cc:243] Waited 87.5 ms on lock 0x7b540018b854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:20.771947 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":201250560}
I20260709 12:28:20.768711  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:20.737234 14142 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:20.769882  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:20.773262  5705 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:20.766220  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:20.774298  5710 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:20.761319 13743 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:20.775871  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:20.778308  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:20.718637 14294 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.725576 14172 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.781786  5704 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:20.784065  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:20.802398  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:20.802977  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 136327us
I20260709 12:28:20.737933 14205 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.804512  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:20.807613  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:20.806571  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:20.776755 14208 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.794866  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:20.816076  5702 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:20.789593 14380 raft_consensus.cc:2764] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:28:20.810310  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:20.803797 14200 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:20.829438  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:20.779604 13743 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:20.801252 14000 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.825089  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:20.809414 14175 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.776314 14038 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.037s
I20260709 12:28:20.809198 14341 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:20.823458 14262 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:20.741036 14244 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:20.854128  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:20.867964  5630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3090) took 1412 ms (client timeout 1585 ms). Trace:
W20260709 12:28:20.871229  5630 rpcz_store.cc:269] 0709 12:28:19.455486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:19.455674 (+   188us) service_pool.cc:224] Handling call
0709 12:28:20.839724 (+1384050us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b5400134f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:20.867867 (+ 28143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1428694912}
I20260709 12:28:20.873687 14175 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.815155  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:20.828735 14200 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:20.884452  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 71992.3us
I20260709 12:28:20.885432  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:20.832732 14156 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:20.833212 14398 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.833478 13743 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:20.898324  5635 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:28:20.870064 14262 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:20.866217 14322 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:20.826567 14208 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:20.833885 13812 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.057s
I20260709 12:28:20.886827 14200 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:20.921567  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.845129 14000 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:20.907892 14262 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:20.907416 14383 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.909003 14380 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:28:20.943862 14294 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:20.948725  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:20.950549  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.927397 14243 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:20.944550 14294 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:20.929191 14319 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:20.905694 14398 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:20.976733 14394 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.199s	user 0.001s	sys 0.000s
W20260709 12:28:20.978585 14393 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20260709 12:28:20.978772 14398 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:20.967969  5708 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3214) took 1913 ms (client timeout 1933 ms). Trace:
W20260709 12:28:20.979811  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 169602us
I20260709 12:28:20.937882 14399 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:20.974498 14172 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:20.980154  5708 rpcz_store.cc:269] 0709 12:28:19.051429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:19.051581 (+   152us) service_pool.cc:224] Handling call
0709 12:28:20.964916 (+1913335us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b540026eb54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:20.965053 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1444466560}
I20260709 12:28:20.973243 14380 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:28:21.005074  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:20.982969 14319 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:20.955935 14383 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.023510  5599 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:21.028605 14332 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.030874  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:20.995572 14317 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.553s	user 0.014s	sys 0.001s
I20260709 12:28:20.990556 13917 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:21.044874  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:20.979924 14294 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.050076  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:20.989125 14329 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.021701 14156 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:20.965341 14244 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.069713  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:28:21.105176  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:21.109712  5634 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 23: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:21.052619 14181 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:28:21.115115  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:21.118466  6277 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:21.120414 14403 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:28:21.127199  6280 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:21.128156  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:21.059697 14354 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.131795  5635 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: []
W20260709 12:28:21.072715 14396 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260709 12:28:21.139864 14410 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.004s
I20260709 12:28:21.142819  5635 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:21.132236  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:21.062954 14408 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:21.071151 14322 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.114473 14172 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.065161 14409 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:28:21.134716  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 65482.5us
I20260709 12:28:21.128080  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:21.136695 14354 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.139055  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.258787  5702 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:21.259933  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.125245 14175 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.260746  5702 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 29: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:21.145638  5704 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:21.057109 14317 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.070s
I20260709 12:28:21.160516  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:21.133940  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:21.165191  5635 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260709 12:28:21.186560 14181 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:21.197187  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:21.124827 14243 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:21.056336 14319 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.101926 14405 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:21.130316 14327 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:21.272213  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:21.283993  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.286295  6274 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:21.287323  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:21.289816  5635 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:28:21.297369  6277 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:21.303578  5704 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:21.304731  5702 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:21.305677  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.306203  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3554) took 1853 ms. Trace:
I20260709 12:28:21.306598  5709 rpcz_store.cc:276] 0709 12:28:19.452754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:19.503242 (+ 50488us) service_pool.cc:224] Handling call
0709 12:28:21.306017 (+1802775us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b5400278654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:21.306142 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":856270592}
I20260709 12:28:21.306875  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.307964  5704 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:21.249557 14406 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.260098 14172 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.161577 14398 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.135s
I20260709 12:28:21.260666 14409 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:21.293557 14154 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.679s	user 0.013s	sys 0.002s
I20260709 12:28:21.313309  5702 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:28:21.159447 14418 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.152s
I20260709 12:28:21.314991  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.318056  5628 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:21.323326  5635 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:28:21.326918  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.284569 14399 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:21.295140 14421 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:21.311192 14366 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:28:21.294986 14000 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.344326  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:21.345203  5702 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:28:21.314754 14402 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
I20260709 12:28:21.332664  5706 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:21.349781  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.349849  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3350) took 1615 ms. Trace:
I20260709 12:28:21.307181 14208 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:21.309365 14243 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:21.293869 14181 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.257263 14322 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.390451 14322 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.391786 14383 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.357347  5709 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:28:21.378217  5636 rpcz_store.cc:276] 0709 12:28:19.734302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:19.803871 (+ 69569us) service_pool.cc:224] Handling call
0709 12:28:19.829173 (+ 25302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:19.990965 (+161792us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:21.349571 (+1358606us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b540018d654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:21.349778 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":369886976}
I20260709 12:28:21.378288  5704 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:21.249003 14408 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.387989 14181 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.388507 14422 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:21.296151 14410 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
W20260709 12:28:21.393291 14412 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:28:21.316186 14359 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.079s
I20260709 12:28:21.317312 14398 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:21.315999 14172 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:21.307763 14327 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.330355 13969 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:21.340294 14337 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.694s	user 0.012s	sys 0.001s
I20260709 12:28:21.314188 14175 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.046s
W20260709 12:28:21.377296  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:21.292809 14156 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.315441 14336 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.067s
I20260709 12:28:21.422329 14322 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.493320 14322 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:21.496965 14342 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.745s	user 0.013s	sys 0.000s
I20260709 12:28:21.497503 14342 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.328661 14406 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:21.500979 14142 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.717s	user 0.011s	sys 0.001s
I20260709 12:28:21.409538  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3104) took 1770 ms. Trace:
I20260709 12:28:21.394124 14399 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.328685 14409 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:28:21.422358 14233 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.817s	user 0.014s	sys 0.000s
I20260709 12:28:21.332072 14154 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.317842 14415 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.115s
I20260709 12:28:21.469753 13969 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.517624 14415 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:21.518060 14415 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:21.519019 14415 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.519729 13969 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.488204 14317 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.536105 14317 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.536610 14243 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.460021 14294 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:21.540800  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 397340us
W20260709 12:28:21.438594  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 121027us
I20260709 12:28:21.502625  5632 rpcz_store.cc:276] 0709 12:28:19.639107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:19.728442 (+ 89335us) service_pool.cc:224] Handling call
0709 12:28:19.728764 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:19.879138 (+150374us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:21.409230 (+1530092us) spinlock_profiling.cc:243] Waited 62.2 ms on lock 0x7b54000c1054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:21.409462 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":143162112}
I20260709 12:28:21.512969 14233 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.502249 14419 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.474244  5702 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
W20260709 12:28:21.523350 14331 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.000s	user 0.007s	sys 0.007s
W20260709 12:28:21.524441 14117 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.862s	user 0.009s	sys 0.006s
W20260709 12:28:21.525426 13953 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.892s	user 0.015s	sys 0.000s
I20260709 12:28:21.527460 14327 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.453167 14398 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:21.504205 14253 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.867s	user 0.011s	sys 0.004s
I20260709 12:28:21.496126 14322 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:21.530769  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:21.447507 14422 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.534725  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1783) took 1074 ms. Trace:
I20260709 12:28:21.516834 14154 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.548435 14336 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:28:21.549450  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:21.454252 14208 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.380611 14421 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.392343 13743 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.570922  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:21.571448  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3106) took 1243 ms. Trace:
W20260709 12:28:21.574077  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:21.557037 14329 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.550926 14379 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.539822 14317 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.576619  5630 rpcz_store.cc:276] 0709 12:28:20.459821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:20.890451 (+430630us) service_pool.cc:224] Handling call
0709 12:28:20.907438 (+ 16987us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:20.982440 (+ 75002us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:20.982489 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:20.994139 (+ 11650us) log.cc:844] Serialized 26 byte log entry
0709 12:28:21.302026 (+307887us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:21.302059 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:21.302112 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:21.534347 (+232235us) raft_consensus.cc:1685] finished
0709 12:28:21.534390 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:21.534624 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":66996,"raft.run_cpu_time_us":54,"raft.run_wall_time_us":53,"thread_start_us":321807,"threads_started":2,"wal-append.queue_time_us":497540}
I20260709 12:28:21.562137 13953 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.550989 14342 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:21.552802 14412 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 12:28:21.587162  5559 rpcz_store.cc:276] 0709 12:28:20.327565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:20.327780 (+   215us) service_pool.cc:224] Handling call
0709 12:28:21.571267 (+1243487us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b54001be754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:21.571385 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":559010048}
I20260709 12:28:21.598985  5631 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:21.469602 14337 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.569264 14426 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:28:21.575778 14415 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:21.556049 14299 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:28:21.587142 14358 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:21.594887 13861 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.108s	user 0.013s	sys 0.001s
I20260709 12:28:21.588936 14142 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:28:21.592437 14320 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.942s	user 0.011s	sys 0.006s
W20260709 12:28:21.605288  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 208697us
I20260709 12:28:21.586619 14208 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.621057  5632 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:21.569911 14253 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:21.621634  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:21.622184  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 80317.1us
I20260709 12:28:21.558159 14419 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.604883 14354 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.652616 13960 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.629216 13861 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.653370  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:21.653951  5559 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:21.654650 13861 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:21.654965  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.655539  5552 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:28:21.655678 14408 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.596812 14379 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.657074 14419 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:21.657681  5635 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:28:21.630548  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:21.629182 14319 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.610966 14426 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:21.661432 14421 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.663307 14253 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:28:21.664191  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:21.665535  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:21.666571  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:21.667156 14233 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:21.631872 14200 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.703s	user 0.011s	sys 0.001s
I20260709 12:28:21.668738  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3100) took 1343 ms. Trace:
I20260709 12:28:21.669092  5561 rpcz_store.cc:276] 0709 12:28:20.325585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:20.325735 (+   150us) service_pool.cc:224] Handling call
0709 12:28:21.668574 (+1342839us) spinlock_profiling.cc:243] Waited 47.6 ms on lock 0x7b540044dc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:21.668685 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":109564288}
I20260709 12:28:21.670111  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.670650  5561 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:21.671495  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:21.672052  5561 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:21.674695  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.675874  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.604657 14413 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.676904  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.677243 14413 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.564101 14117 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.626093 14317 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.629801 14358 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.668035 14200 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.651839 14406 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.650599 14422 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:21.680326 14413 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:21.682299 14117 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.681207  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:21.705454 13953 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.671591 14409 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.709450 14409 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:21.625916 14320 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.976s	user 0.012s	sys 0.006s
I20260709 12:28:21.711755 14320 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:21.713842  5559 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:21.714867  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.625123 14434 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.715842  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:21.716866 14434 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.718539 14409 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:21.719553 14434 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.717202  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:21.720396  5555 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 11. (term mismatch)
I20260709 12:28:21.720598  5559 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:21.720775  5555 pending_rounds.cc:85] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 10
I20260709 12:28:21.721117  5555 pending_rounds.cc:107] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 21.11
I20260709 12:28:21.721360  5555 raft_consensus.cc:2889] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:21.717918  5632 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:21.729207  5636 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 19. Current term is 20. Ops: []
I20260709 12:28:21.730638  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:21.736816 13743 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.737665 13743 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:21.738153 13743 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.738389  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:21.741214 14117 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:21.741649 14117 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:21.742198 14117 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:21.743001 14293 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.743597 14293 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:21.751799 14390 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.103s	user 0.000s	sys 0.003s
I20260709 12:28:21.761075 14339 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:21.763550  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3359) took 1172 ms. Trace:
I20260709 12:28:21.764072  5634 rpcz_store.cc:276] 0709 12:28:20.591493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:20.802054 (+210561us) service_pool.cc:224] Handling call
0709 12:28:21.763293 (+961239us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b5400438354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:21.763470 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":704768768}
I20260709 12:28:21.766336 14320 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.768787 14337 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.769320  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.665560 14205 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.770237  5553 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:21.684922 14317 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.686322 14208 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:21.688408 14358 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.689028 14426 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.773715 14406 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:28:21.773811  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.774196 14426 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:21.774511 14406 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:28:21.774597  5560 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:21.775056 14406 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:21.694072 14379 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.695943  5557 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:21.776037 14406 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.776955  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:21.777597 14406 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:21.678170  5554 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:21.667148  5630 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:21.702720 14200 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.700825 14359 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:21.703425 14430 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.781149 14359 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.684763 14431 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:21.782281 14431 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:21.781200 14430 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.782596  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3114) took 1451 ms. Trace:
W20260709 12:28:21.688190  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 56327.5us
I20260709 12:28:21.695603 14038 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.772856  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3103) took 2139 ms. Trace:
I20260709 12:28:21.697888  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:21.785153  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 87355.8us
W20260709 12:28:21.696969  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:21.703209 14398 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.786687 14403 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.777608  5557 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:21.784175  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:21.693948  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3094) took 2227 ms. Trace:
I20260709 12:28:21.788597 14294 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:21.699237  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 1784) took 1026 ms. Trace:
I20260709 12:28:21.789554  5631 rpcz_store.cc:276] 0709 12:28:20.673078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:20.775518 (+102440us) service_pool.cc:224] Handling call
0709 12:28:21.699063 (+923545us) spinlock_profiling.cc:243] Waited 818 ms on lock 0x7b5400240754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:21.699156 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1881979264}
I20260709 12:28:21.790673  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:21.787400 14403 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.791667  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:21.784256 14358 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.008s
I20260709 12:28:21.792681  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.792970 14358 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.792785  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.784108 14038 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.794047 14435 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:21.794345 14208 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:21.794492 14435 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:21.794770 14208 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:21.794943 14435 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:21.700947 14336 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:21.783108 14431 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.698238  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3099) took 1373 ms. Trace:
I20260709 12:28:21.788816  5628 rpcz_store.cc:276] 0709 12:28:19.466194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:19.466309 (+   115us) service_pool.cc:224] Handling call
0709 12:28:20.840011 (+1373702us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:21.317891 (+477880us) spinlock_profiling.cc:243] Waited 5.89 ms on lock 0x7b54001d5720. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa6741c0 00007f4ffa675231 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c
0709 12:28:21.332609 (+ 14718us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:21.693747 (+361138us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400134f50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:21.693886 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3172052992}
I20260709 12:28:21.798907  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.797538  5558 rpcz_store.cc:276] 0709 12:28:20.325304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:20.333308 (+  8004us) service_pool.cc:224] Handling call
0709 12:28:21.698019 (+1364711us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b540023fd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:21.698180 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":309049088}
I20260709 12:28:21.799505  5554 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:21.800074  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.800699  5558 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:21.785498 14418 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:21.801466 14418 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:21.821856 14418 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:21.744200 14293 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:21.843050  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:21.848755  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3107) took 1914 ms. Trace:
I20260709 12:28:21.849352  5627 rpcz_store.cc:276] 0709 12:28:19.934370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:20.211603 (+277233us) service_pool.cc:224] Handling call
0709 12:28:20.211837 (+   234us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:20.213261 (+  1424us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:20.213292 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:20.215403 (+  2111us) log.cc:844] Serialized 26 byte log entry
0709 12:28:20.488764 (+273361us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:20.488808 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:20.488889 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:21.750900 (+1262011us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:21.848338 (+ 97438us) raft_consensus.cc:1685] finished
0709 12:28:21.848390 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:21.848677 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":750,"raft.queue_time_us":850,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":39525,"spinlock_wait_cycles":40839552,"thread_start_us":181077,"threads_started":1,"wal-append.queue_time_us":290039}
W20260709 12:28:21.857218  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:21.857846  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:21.858110 14434 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:21.858572  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:21.859351 14337 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.731866  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:21.860507 14322 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:21.860713  5636 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 20.
I20260709 12:28:21.858671 14434 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.859807 14337 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:21.682894 14341 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.813s	user 0.010s	sys 0.000s
I20260709 12:28:21.862895  5556 raft_consensus.cc:2410] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 23 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:28:21.863250 14318 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.863720 14318 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.864058  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3129) took 1231 ms. Trace:
I20260709 12:28:21.864467  5556 rpcz_store.cc:276] 0709 12:28:20.632951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:21.020712 (+387761us) service_pool.cc:224] Handling call
0709 12:28:21.863861 (+843149us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b54000ebfd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:21.864004 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1867068160}
I20260709 12:28:21.865880  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.866205 13743 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.800298  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:21.866926  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:21.867630  5556 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 29: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:21.867887 14318 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:21.867586  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:21.867671  5559 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:21.796804 14336 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.869642  5554 raft_consensus.cc:2368] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23.
I20260709 12:28:21.871263  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.871896 14336 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.872288  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:21.784848  5629 rpcz_store.cc:276] 0709 12:28:19.638799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:19.692602 (+ 53803us) service_pool.cc:224] Handling call
0709 12:28:21.772636 (+2080034us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b5400438d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:21.772785 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2290953472}
I20260709 12:28:21.805051  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:21.873222  5552 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:21.795682 14208 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:21.874078 14208 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:21.813925  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.775846  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.876772 14444 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.877421  5560 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:28:21.877579 14444 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.878391  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:21.879648  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.783100  5555 rpcz_store.cc:276] 0709 12:28:20.330699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:20.544153 (+213454us) service_pool.cc:224] Handling call
0709 12:28:21.698307 (+1154154us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:21.721818 (+ 23511us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:21.781452 (+ 59634us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b540023fd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:21.782327 (+   875us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540023fd50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:21.782494 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2635403904}
I20260709 12:28:21.869918  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:21.872437 14336 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.793466 14358 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:21.872046  5554 raft_consensus.cc:2410] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:28:21.801836  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:21.891534  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 153466us
I20260709 12:28:21.929869  5558 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:21.880210  5560 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
W20260709 12:28:21.891992  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 100380us
W20260709 12:28:21.898514  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 114398us
I20260709 12:28:21.897346  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:21.932683  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:21.933555  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:21.917570  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:21.935300  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:21.935995  6278 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:21.938575  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:21.938853  5561 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:21.929926  5556 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:21.930372  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:21.938601  5635 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:21.947266  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.947290  5635 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:28:21.947928  5705 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:21.948163  5552 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:21.948886  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:21.949072  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.949316  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.948887  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:21.952003  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:21.953531  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 81313.2us
I20260709 12:28:21.954753  5552 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:21.954558  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:21.956035  5710 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:28:21.951380  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:21.957149  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.957412  5635 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:21.958251  5553 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:28:21.958961  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:21.966765  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:21.967391  6273 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 22: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:21.960726  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.957713  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:21.969488  5709 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:21.961320  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:21.970062  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:21.961454  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.971746  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.972378  5706 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:21.972769  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:21.973299  5708 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:21.974216  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:21.964522  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:21.880014 14444 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:21.959486  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:21.961601  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:21.975488  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:21.983634  6277 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:21.980072  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:21.977095  5675 leader_election.cc:336] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.585s (SENT)
I20260709 12:28:21.977370  5704 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:28:21.985409  5558 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:21.970484  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:21.964427  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:21.880906 14038 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:21.973218  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:21.989827  5706 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:21.980345  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:21.980932  5602 leader_election.cc:400] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23.
I20260709 12:28:21.991007  5602 leader_election.cc:403] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:28:21.991108  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:21.990665  5560 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:21.963017  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:21.993122  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:21.993925  5634 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 22: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:21.994563  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:21.994992  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:21.995581  6279 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:21.995733  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:21.937678 14205 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:21.996367  5632 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:21.981613  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:21.979627  5701 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:28:21.945482 13866 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:21.997795  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:21.992149  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:22.008392  5552 raft_consensus.cc:2410] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 27 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
I20260709 12:28:22.002632  6273 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:28:22.009377  5631 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:22.009893  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:22.004096  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:22.006985  5629 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: []
I20260709 12:28:21.957300 14431 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:22.012751  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:22.015256  5704 pending_rounds.cc:85] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 15
I20260709 12:28:22.015576  5704 pending_rounds.cc:107] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 20.16
W20260709 12:28:21.968855 14262 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.014s	user 0.012s	sys 0.000s
I20260709 12:28:22.015849  5704 raft_consensus.cc:2889] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:28:21.981834 14397 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.054s	user 0.005s	sys 0.007s
I20260709 12:28:22.017699  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:21.959965 14413 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:21.983464 14172 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:21.979611 14233 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:28:22.014922  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:22.055668  5560 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:21.956812 14445 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:21.975453 14200 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:22.026918  5554 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 13. (term mismatch)
I20260709 12:28:22.020044  5636 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:22.095369  5554 pending_rounds.cc:85] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 12
I20260709 12:28:22.032924  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:22.035778  5631 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:28:22.095996  5706 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:22.021693  6271 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:28:21.974153 14208 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:21.989063 14299 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:22.050168  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.010160  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:22.106627  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:22.107237  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 97183us
I20260709 12:28:21.976390 13953 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:28:21.997588 14421 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:21.945771 14336 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:22.063355  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:22.065064  5555 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:22.093248  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.121698  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3118) took 1771 ms. Trace:
I20260709 12:28:22.095856  5554 pending_rounds.cc:107] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 25.13
I20260709 12:28:22.122169  5555 rpcz_store.cc:276] 0709 12:28:20.350891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.024514 (+1673623us) service_pool.cc:224] Handling call
0709 12:28:22.064830 (+ 40316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.121155 (+ 56325us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.121609 (+   454us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:22.122905  5554 raft_consensus.cc:2889] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:28:22.122488  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 161243us
I20260709 12:28:22.123957  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3119) took 1772 ms. Trace:
I20260709 12:28:22.099195  5561 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:28:22.125594  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3124) took 1772 ms. Trace:
I20260709 12:28:22.126150  5561 rpcz_store.cc:276] 0709 12:28:20.352695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.053472 (+1700777us) service_pool.cc:224] Handling call
0709 12:28:22.098948 (+ 45476us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:22.125313 (+ 26365us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.125540 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:22.097529  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:21.975783 14183 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.076s
I20260709 12:28:22.024621 13960 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:28:22.029958 14397 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.121s	user 0.005s	sys 0.007s
W20260709 12:28:22.093181  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 116086us
I20260709 12:28:22.105253 14269 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:22.024005 14444 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:21.979024 14318 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.071s
I20260709 12:28:21.957840 14322 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.089s
I20260709 12:28:22.037158 14439 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:22.109983  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:22.110412 14446 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:28:22.025355 13866 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.047300 14383 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:22.041458 14419 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:22.049114 14142 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:22.111569 14337 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:22.112196 14359 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:22.112849 14293 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:22.040249 14354 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:22.047005 13743 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:22.100180 14445 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:22.113821  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 81171.8us
I20260709 12:28:22.232295  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3400) took 1168 ms. Trace:
I20260709 12:28:22.058753 14413 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:22.024271 14243 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:28:22.115700  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 164362us
I20260709 12:28:22.106736  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.024267 14422 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:28:22.098975  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51687.8us
I20260709 12:28:22.244300  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:22.097716  5708 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:22.130570  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:22.124316  5554 rpcz_store.cc:276] 0709 12:28:20.351192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.026145 (+1674953us) service_pool.cc:224] Handling call
0709 12:28:22.026494 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.123719 (+ 97225us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.123904 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:22.095042 14200 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:22.157563  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:22.158792  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.114238 14126 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:22.037559 14253 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:22.025509 14205 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.051559 14172 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:22.117204 14208 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:22.119421 14233 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:22.273494 14233 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:22.274094 14233 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:22.178354  5636 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:22.277521  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.278404  5636 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:22.279718  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:22.178431  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54102.8us
I20260709 12:28:22.281631 14336 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:22.043752 14262 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.282347 14358 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:22.282582  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:22.284006 14262 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:22.284720 14262 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:22.285501  6272 raft_consensus.cc:2368] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
I20260709 12:28:22.285282 14262 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.287711 14262 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:22.141930 14299 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:22.147416 13960 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:22.114517 14409 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:22.288838 13960 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:22.163318 14000 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:22.289623 14000 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:22.289383 14299 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.290542 14000 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.185421  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:22.190174 14419 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:22.195417  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.257015 14126 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:22.293836 14413 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:22.294162 14126 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:22.294309 14419 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:22.251540 14422 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:22.194001  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.294953 14422 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:22.295244  5627 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:22.288177 14208 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:22.157421 14444 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:22.202512  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:22.296564 14444 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:22.257270 14200 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:22.296870  5630 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:22.200227  5556 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:28:22.141940 13969 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:22.201458  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:22.215160  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:22.181730 14439 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:22.234835  5553 rpcz_store.cc:276] 0709 12:28:21.064304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.230811 (+1166507us) service_pool.cc:224] Handling call
0709 12:28:22.231593 (+   782us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.231840 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:22.231877 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:22.231935 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:22.232007 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.232078 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:22.232220 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:22.167323 14319 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 12:28:22.186622 14317 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:22.182627 14383 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:22.238946  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:22.248030  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:22.329048  5557 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.541s	user 0.016s	sys 0.000s
W20260709 12:28:22.329370  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 90531.2us
I20260709 12:28:22.329509  5557 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:22.330377  5558 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: [26.14-26.14]
I20260709 12:28:22.253413  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.254774  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:22.256230  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 98871.7us
I20260709 12:28:22.168054 14294 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.070s
I20260709 12:28:22.257287  5559 raft_consensus.cc:2410] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 29 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 17.
I20260709 12:28:22.258651  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.259332  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3402) took 1194 ms. Trace:
I20260709 12:28:22.334573  5554 rpcz_store.cc:276] 0709 12:28:21.067945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.258372 (+1190427us) service_pool.cc:224] Handling call
0709 12:28:22.258750 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.258974 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:22.259001 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:22.259050 (+    49us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:22.259083 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.259141 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:22.259265 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:22.336419  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1724) took 1209 ms. Trace:
I20260709 12:28:22.336865  5554 rpcz_store.cc:276] 0709 12:28:21.131392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.335614 (+1204222us) service_pool.cc:224] Handling call
0709 12:28:22.335917 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.336081 (+   164us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:22.336116 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:22.336164 (+    48us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:22.336197 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.336247 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:22.336366 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:22.253014  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:22.256466 13812 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:22.265057  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:22.270558  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:22.272650 14339 raft_consensus.cc:2764] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:28:22.281292 14421 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:22.251361 14397 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.043s
I20260709 12:28:22.299113  5632 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:28:22.163877 13953 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:22.215147  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 252159us
W20260709 12:28:22.344826  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 74398.9us
I20260709 12:28:22.345263  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:22.240180 13743 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:22.317143  5553 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:22.298309 13969 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:22.318941 14439 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:22.362038 14453 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:22.363183 14453 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:22.364550 14379 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 12:28:22.365499 14443 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.272s	user 0.003s	sys 0.000s
I20260709 12:28:22.367481 14452 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:22.367408 13743 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.348232 14409 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:22.369180 14409 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:22.369664 14409 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:22.371644 14435 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.576s	user 0.010s	sys 0.000s
I20260709 12:28:22.347580 14317 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:22.372838 14317 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:22.373307 14317 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:22.373755 14117 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.631s	user 0.020s	sys 0.005s
I20260709 12:28:22.374199 14117 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.376663  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:22.377344  5636 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:28:22.377844  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:22.378916  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:22.380483 14337 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:22.381170 14337 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:22.382191 14294 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:22.341588 14383 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:22.383106 14294 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.323463 13960 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:22.344466 14447 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:28:22.390139 13960 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:22.390539 14205 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:22.390945  5627 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:22.249713 14445 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:22.209554 14142 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:22.265949 14449 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:28:22.328473  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:22.339484  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 86552.1us
I20260709 12:28:22.341148  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:22.346902  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1722) took 1238 ms. Trace:
I20260709 12:28:22.409118  5553 rpcz_store.cc:276] 0709 12:28:21.108166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.315976 (+1207810us) service_pool.cc:224] Handling call
0709 12:28:22.316324 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.346612 (+ 30288us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.346851 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:22.410408  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:22.338775  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1726) took 1178 ms. Trace:
I20260709 12:28:22.411271  5554 rpcz_store.cc:276] 0709 12:28:21.159967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.337748 (+1177781us) service_pool.cc:224] Handling call
0709 12:28:22.338059 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.338432 (+   373us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:22.338458 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:22.338503 (+    45us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:28:22.338533 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.338602 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:22.338718 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:22.412823  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:22.298708 14172 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:22.341229  5709 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:22.416567  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:22.418303  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:22.357367 14426 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.583s	user 0.019s	sys 0.007s
I20260709 12:28:22.416141  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:22.355060  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:22.392642  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47460.4us
I20260709 12:28:22.393059  5561 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:28:22.392710  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.394364  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:22.426928  5708 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:22.356149 13953 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:22.429569  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 101154us
I20260709 12:28:22.433207  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:22.434916  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:22.437575  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:22.379977 13969 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:22.419986  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:22.440058  5554 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:22.436335  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:22.338275  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1725) took 1178 ms. Trace:
I20260709 12:28:22.390931 14447 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:28:22.426108  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 71187.1us
I20260709 12:28:22.382270 14337 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:22.428588 14156 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:22.441983  5557 rpcz_store.cc:276] 0709 12:28:21.159645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.337073 (+1177428us) service_pool.cc:224] Handling call
0709 12:28:22.337569 (+   496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.337833 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:22.337872 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:22.337938 (+    66us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:22.337993 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.338066 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:22.338198 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:22.455770  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:22.460263  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:22.451490  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:22.376684 14117 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:22.441458  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:22.462996  5554 raft_consensus.cc:2368] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 21. Current term is 22.
I20260709 12:28:22.344765 14336 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.084s
I20260709 12:28:22.467168  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:22.403868 13812 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:22.467463  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:22.469581  5553 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 16. (term mismatch)
I20260709 12:28:22.368251 14452 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.372164 14435 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.415653 14142 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:22.498850  5554 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:28:22.499722  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1729) took 1250 ms. Trace:
I20260709 12:28:22.500001  5554 rpcz_store.cc:276] 0709 12:28:21.249326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.463605 (+1214279us) service_pool.cc:224] Handling call
0709 12:28:22.498516 (+ 34911us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.499388 (+   872us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.499639 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:22.490736  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:22.457204 14339 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:22.503224  5553 pending_rounds.cc:85] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 15
I20260709 12:28:22.503885  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:22.507655  5636 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:22.504665  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.448266 14447 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:22.476507 14318 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:22.510846  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:22.478139 13861 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:22.494778 14426 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:22.513833  5705 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:22.515087  5553 pending_rounds.cc:107] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 21.16
I20260709 12:28:22.531643  5553 raft_consensus.cc:2889] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:22.509438  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:22.518875  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1731) took 1188 ms. Trace:
I20260709 12:28:22.465202 14456 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:22.521776  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:22.507047 14339 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:22.506954 14406 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.729s	user 0.010s	sys 0.004s
I20260709 12:28:22.464895 14359 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.050s
I20260709 12:28:22.364300 14453 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.533993  5554 rpcz_store.cc:276] 0709 12:28:21.330628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.500764 (+1170136us) service_pool.cc:224] Handling call
0709 12:28:22.518219 (+ 17455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.518437 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:22.518473 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:22.518532 (+    59us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:22.518579 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.518649 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:22.518803 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:22.487002 14293 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.040s
I20260709 12:28:22.487622 14408 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.033s
I20260709 12:28:22.625929 14408 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:22.626366 14408 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:22.627326 14408 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.487236 14379 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 12:28:22.510056 14208 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:22.528268  5636 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:22.465220 14413 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.050s
W20260709 12:28:22.581933  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 60298.8us
I20260709 12:28:22.631976  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:22.532759  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1730) took 1241 ms. Trace:
I20260709 12:28:22.519711 14142 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:22.633051  5553 rpcz_store.cc:276] 0709 12:28:21.291009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.468663 (+1177654us) service_pool.cc:224] Handling call
0709 12:28:22.469086 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.532379 (+ 63293us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.532684 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:28:22.592908  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 83530.7us
W20260709 12:28:22.535441 14415 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.907s	user 0.006s	sys 0.008s
I20260709 12:28:22.508297 14156 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:22.646971  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:22.697479  5631 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:22.537130 14117 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:28:22.699633  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 67652.8us
W20260709 12:28:22.616250 14418 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.794s	user 0.015s	sys 0.000s
W20260709 12:28:22.704979  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 352308us
I20260709 12:28:22.696481  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:22.611402 14320 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:22.617118 14426 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:22.704267  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:22.724442  5554 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:28:22.725427  5602 leader_election.cc:400] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 21. Current term is 22.
I20260709 12:28:22.725860  5602 leader_election.cc:403] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:28:22.618892 14339 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:22.701330  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54459.9us
I20260709 12:28:22.725350  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3132) took 1072 ms. Trace:
I20260709 12:28:22.611974 14456 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:22.703106 14431 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.663s	user 0.007s	sys 0.005s
W20260709 12:28:22.704052 14454 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.192s	user 0.000s	sys 0.002s
I20260709 12:28:22.729683  5554 rpcz_store.cc:276] 0709 12:28:21.653086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.646465 (+993379us) service_pool.cc:224] Handling call
0709 12:28:22.723490 (+ 77025us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.724952 (+  1462us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.725278 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1094400}
I20260709 12:28:22.623982 14439 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:22.534242 13861 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.738221  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:22.699492 14294 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:22.706655 14208 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:22.702399 14142 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:22.720906 14408 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:22.758327  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:22.720625 14379 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:22.727517 14117 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:22.781698  5706 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:22.782382  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.739522 14456 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:22.620324 14444 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.731424 14397 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:22.531319 14318 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:22.731206 13969 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:22.663031 14126 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:22.762104  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 479582us
I20260709 12:28:22.784986  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.787217  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.784101 14294 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:22.731313 14322 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.051s
I20260709 12:28:22.764931 14431 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:22.730502 14205 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.052s
I20260709 12:28:22.776507 13960 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:22.844265 14126 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:22.855895 14431 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:22.856521 14431 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:22.844985  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:22.800657  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:22.844867 14142 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.11, Last appended by leader: 13, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:22.845975  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3130) took 2187 ms. Trace:
I20260709 12:28:22.860575  5560 rpcz_store.cc:276] 0709 12:28:20.658651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.097603 (+1438952us) service_pool.cc:224] Handling call
0709 12:28:22.122675 (+ 25072us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:22.151702 (+ 29027us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:22.151737 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:22.152323 (+   586us) log.cc:844] Serialized 26 byte log entry
0709 12:28:22.206433 (+ 54110us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b18003c8ec0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:28:22.550490 (+344057us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:22.550541 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.550623 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:22.845688 (+295065us) raft_consensus.cc:1685] finished
0709 12:28:22.845713 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:22.845916 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":967,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":10637,"spinlock_wait_cycles":9051904,"thread_start_us":343837,"threads_started":1,"wal-append.queue_time_us":631840}
I20260709 12:28:22.863684  5628 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: []
W20260709 12:28:22.864157  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63619.2us
I20260709 12:28:22.864480 13743 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:22.747274 14398 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 21. Current term is 22.
I20260709 12:28:22.831123 14269 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:22.848985  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.867339  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:22.833365 14458 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:28:22.833441  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.834376  5636 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:28:22.835160  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:22.838104 14444 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:22.764968 14457 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 12:28:22.838279  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:22.873986  6279 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:22.873935  5553 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:28:22.839125  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:22.840322  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:22.841145  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:22.843918 14439 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:22.723613 14418 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.864717  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3120) took 2513 ms. Trace:
I20260709 12:28:22.802620 14293 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:22.851402 14253 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.588s	user 0.007s	sys 0.004s
I20260709 12:28:22.804908  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:22.871295  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:22.865579 13743 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.846319  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:22.791917 14336 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.083s
I20260709 12:28:22.886142  5552 rpcz_store.cc:276] 0709 12:28:20.351478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.036570 (+1685092us) service_pool.cc:224] Handling call
0709 12:28:22.052722 (+ 16152us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:22.060027 (+  7305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:22.060075 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:22.060698 (+   623us) log.cc:844] Serialized 26 byte log entry
0709 12:28:22.455698 (+395000us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:22.779256 (+323558us) log.cc:844] Serialized 14 byte log entry
0709 12:28:22.779481 (+   225us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.779549 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:22.864495 (+ 84946us) raft_consensus.cc:1685] finished
0709 12:28:22.864518 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:22.864662 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6836,"raft.queue_time_us":20249,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":5545,"thread_start_us":566487,"threads_started":2,"wal-append.queue_time_us":600421}
I20260709 12:28:22.883818  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:22.862190 14341 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:22.802261 14397 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:22.869457 14269 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:22.862596 13969 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:22.908311  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:22.935483  5707 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 27: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:22.923523  6279 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:22.873327 14183 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:28:22.932981  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44435.6us
I20260709 12:28:22.943886  5703 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:28:22.944303  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:22.944736  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:22.953001  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:22.976862  6278 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:28:22.977377  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:22.981110  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:22.896814 14415 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:22.848063 14383 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:22.995884  6277 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:28:22.937016 14397 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:22.900071 14253 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23.
W20260709 12:28:23.003242  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 214885us
I20260709 12:28:23.003765  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3582) took 1053 ms. Trace:
W20260709 12:28:22.977837 14460 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:28:22.904583 14341 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:22.994349  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:23.007267  6280 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:28:23.030520  5703 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:23.031486  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:23.030525  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:23.032955  6274 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:23.008693  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:22.979226 14000 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.004s	sys 0.012s
I20260709 12:28:22.941859 14435 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:22.815271 14156 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:22.896746 14444 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.109s
I20260709 12:28:23.021217  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:23.022655  5560 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: [28.12-28.12]
I20260709 12:28:23.013232  5701 rpcz_store.cc:276] 0709 12:28:21.950172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:21.997606 (+ 47434us) service_pool.cc:224] Handling call
0709 12:28:22.037391 (+ 39785us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:22.051183 (+ 13792us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:22.051214 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:22.099826 (+ 48612us) log.cc:844] Serialized 26 byte log entry
0709 12:28:22.452738 (+352912us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:22.452797 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.452897 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:23.003490 (+550593us) raft_consensus.cc:1685] finished
0709 12:28:23.003510 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:23.003685 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13304,"raft.queue_time_us":13726,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":42274,"thread_start_us":289428,"threads_started":1,"wal-append.queue_time_us":485366}
I20260709 12:28:23.029724 13743 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:23.036689  6279 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:22.903338 14293 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.043962  5634 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:23.045469  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:23.063956  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3407) took 1398 ms. Trace:
I20260709 12:28:23.066150  5560 rpcz_store.cc:276] 0709 12:28:21.665816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.994919 (+1329103us) service_pool.cc:224] Handling call
0709 12:28:23.022369 (+ 27450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:23.063556 (+ 41187us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.063880 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:22.904752 14117 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:28:23.044667 14000 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.004s	sys 0.012s
I20260709 12:28:23.076759  6280 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:28:23.079445  5703 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:28:23.090804 13969 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:23.092103 13861 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:23.094015  5560 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:28:23.094652  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3412) took 1391 ms. Trace:
I20260709 12:28:23.094969  5560 rpcz_store.cc:276] 0709 12:28:21.703164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.067250 (+1364086us) service_pool.cc:224] Handling call
0709 12:28:23.093695 (+ 26445us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:23.094389 (+   694us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.094582 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:23.096577  5560 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:28:23.097184  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1738) took 1223 ms. Trace:
I20260709 12:28:23.088685  5701 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
W20260709 12:28:23.099128 14172 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.651s	user 0.008s	sys 0.003s
I20260709 12:28:23.053062 14208 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:23.102098 14172 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.103286  6275 raft_consensus.cc:2368] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23.
I20260709 12:28:23.104842 14293 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:23.098948  5560 rpcz_store.cc:276] 0709 12:28:21.873930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.095907 (+1221977us) service_pool.cc:224] Handling call
0709 12:28:23.096233 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:23.096931 (+   698us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.097126 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:28:23.076344 14409 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.706s	user 0.011s	sys 0.002s
I20260709 12:28:23.106657  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3416) took 1231 ms. Trace:
I20260709 12:28:23.106841 14409 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:23.107000  5560 rpcz_store.cc:276] 0709 12:28:21.875371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.105847 (+1230476us) service_pool.cc:224] Handling call
0709 12:28:23.106130 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:23.106304 (+   174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:23.106333 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:23.106383 (+    50us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:23.106431 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.106490 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:23.106604 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:23.108620 14172 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:23.109543 14422 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.814s	user 0.018s	sys 0.000s
I20260709 12:28:23.109623  5560 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:28:23.110069 14422 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:23.097047 14200 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.800s	user 0.014s	sys 0.000s
I20260709 12:28:23.111303 14200 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.118932 14383 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.120796 14200 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.121968  6276 raft_consensus.cc:2368] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31.
W20260709 12:28:23.114096 14467 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:28:23.087927  5703 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:28:23.133997  5703 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:28:23.086707 14317 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.713s	user 0.008s	sys 0.003s
I20260709 12:28:23.137341  5710 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: [22.11-22.11]
I20260709 12:28:23.139590  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 1740) took 1265 ms. Trace:
I20260709 12:28:23.140031  5559 rpcz_store.cc:276] 0709 12:28:21.874617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:21.931424 (+ 56807us) service_pool.cc:224] Handling call
0709 12:28:23.139408 (+1207984us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b54000267d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:23.139533 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":662597504}
W20260709 12:28:23.142551 14461 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.313s	user 0.001s	sys 0.000s
W20260709 12:28:23.146660  6237 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.170s (SENT) [suppressed 6 similar messages]
I20260709 12:28:23.146744  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:23.147483  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1796) took 1275 ms. Trace:
I20260709 12:28:23.147516  5632 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:28:23.147913  5628 rpcz_store.cc:276] 0709 12:28:21.871899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.293778 (+421879us) service_pool.cc:224] Handling call
0709 12:28:22.294082 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.864121 (+570039us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.147266 (+283145us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b5400439c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:23.147428 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1309544192}
I20260709 12:28:23.169070  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:23.180064  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:23.186604  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:23.190814  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:23.192811  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1728) took 2029 ms. Trace:
I20260709 12:28:23.200176  5557 rpcz_store.cc:276] 0709 12:28:21.161957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.458398 (+1296441us) service_pool.cc:224] Handling call
0709 12:28:22.458835 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.459178 (+   343us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:28:22.794929 (+335751us) log.cc:844] Serialized 14 byte log entry
0709 12:28:23.190261 (+395332us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:23.190317 (+    56us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:23.190362 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.191597 (+  1235us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:23.191723 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36146,"prepare.queue_time_us":593561,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"thread_start_us":679195,"threads_started":2}
I20260709 12:28:23.205765  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:23.209667  5630 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: [28.12-28.12]
I20260709 12:28:23.211475  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3404) took 2114 ms. Trace:
I20260709 12:28:23.211871  5556 rpcz_store.cc:276] 0709 12:28:21.100035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.297869 (+1197834us) service_pool.cc:224] Handling call
0709 12:28:22.298125 (+   256us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:22.298483 (+   358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:22.298511 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:22.319990 (+ 21479us) log.cc:844] Serialized 26 byte log entry
0709 12:28:22.768303 (+448313us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:22.768384 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.768489 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:23.211261 (+442772us) raft_consensus.cc:1685] finished
0709 12:28:23.211282 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:23.211417 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1240,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":2771,"thread_start_us":447848,"threads_started":1,"wal-append.queue_time_us":625484}
I20260709 12:28:23.168380 14317 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:23.231724  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 85074.6us
I20260709 12:28:23.245360 14403 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:23.247187  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1418) took 1274 ms. Trace:
I20260709 12:28:23.247601  6276 rpcz_store.cc:276] 0709 12:28:21.973188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:21.973987 (+   799us) service_pool.cc:224] Handling call
0709 12:28:23.246940 (+1272953us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400591ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:23.247093 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2637614464}
I20260709 12:28:23.250367  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3585) took 1299 ms. Trace:
I20260709 12:28:23.250751  5710 rpcz_store.cc:276] 0709 12:28:21.950872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.010988 (+ 60116us) service_pool.cc:224] Handling call
0709 12:28:22.011387 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:23.142231 (+1130844us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.250119 (+107888us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54003e6f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:23.250312 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2588701568}
I20260709 12:28:23.252897  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3398) took 2191 ms. Trace:
I20260709 12:28:23.205641 14337 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:23.253726  5561 rpcz_store.cc:276] 0709 12:28:21.060996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.147984 (+1086988us) service_pool.cc:224] Handling call
0709 12:28:22.173367 (+ 25383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:22.426128 (+252761us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.252650 (+826522us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b5400025dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:23.252838 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":504657152}
I20260709 12:28:23.252144  5710 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: [24.18-26.19]
I20260709 12:28:23.171645 14341 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:23.215130 14418 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:23.286335  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 117352us
I20260709 12:28:23.243963 14409 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.260156 14337 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:23.299952 14490 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:28:23.302431  5627 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
W20260709 12:28:23.311317 14486 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:28:23.316805  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:23.320528 14172 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:28:23.320801  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:23.297634 14337 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:23.345111 14445 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.927s	user 0.002s	sys 0.016s
I20260709 12:28:23.345512  6274 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:23.346097 14202 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:23.352835 13960 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:23.353320 13960 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:23.355304 14431 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:23.360903 14496 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:28:23.363909 14469 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:23.364398 14469 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:23.370038 14183 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:23.371857 13866 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.365597 14469 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.372706  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3595) took 1035 ms. Trace:
I20260709 12:28:23.346926 14406 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.029s
I20260709 12:28:23.375349  5706 rpcz_store.cc:276] 0709 12:28:22.337045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.340827 (+  3782us) service_pool.cc:224] Handling call
0709 12:28:23.372440 (+1031613us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b54002d19d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:23.372634 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":856706432}
W20260709 12:28:23.346761 14491 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.057s	user 0.007s	sys 0.000s
I20260709 12:28:23.347101 14000 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:23.346522 14466 raft_consensus.cc:2764] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:28:23.370723 14183 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:23.406881 14183 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.385716 14000 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:23.417312 14000 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.418725 14000 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:23.395231 14469 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:23.397660 14408 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.616s	user 0.010s	sys 0.000s
W20260709 12:28:23.400560 14233 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.126s	user 0.016s	sys 0.001s
I20260709 12:28:23.348034 14422 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:23.386660 13960 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:23.439553 13960 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.428953 14317 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.343976 14200 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.040s
I20260709 12:28:23.346501 14172 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.076s
I20260709 12:28:23.345672 14445 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:23.442960  5701 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:23.444519  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3596) took 1106 ms. Trace:
W20260709 12:28:23.444999  5633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3355) took 2900 ms (client timeout 2262 ms). Trace:
W20260709 12:28:23.445402  5633 rpcz_store.cc:269] 0709 12:28:20.544828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:20.835869 (+291041us) service_pool.cc:224] Handling call
0709 12:28:23.444790 (+2608921us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54000c2954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:23.444911 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3832106752}
I20260709 12:28:23.446597  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:23.446774 14445 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:23.444850  5701 rpcz_store.cc:276] 0709 12:28:22.337648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.101162 (+763514us) service_pool.cc:224] Handling call
0709 12:28:23.101455 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:23.443353 (+341898us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.444239 (+   886us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b5400451ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:23.444460 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":784986368}
I20260709 12:28:23.447722  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:23.449062  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3396) took 2388 ms. Trace:
I20260709 12:28:23.449477  5558 rpcz_store.cc:276] 0709 12:28:21.060329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.146805 (+1086476us) service_pool.cc:224] Handling call
0709 12:28:22.330151 (+183346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:22.330735 (+   584us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.448856 (+1118121us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54003c41d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:23.449000 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":420734336}
W20260709 12:28:23.452446 14339 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.694s	user 0.005s	sys 0.007s
I20260709 12:28:23.452907 14339 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:23.455214 14339 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.453401  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:23.460830  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:23.468247  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 145103us
I20260709 12:28:23.469027  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:23.475762  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:23.477774  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:23.479172  6276 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:23.475759  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:23.479489 14339 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:23.481695  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:23.490975  5633 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:23.491339  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:23.479188  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:23.492420  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:23.493964  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:23.495522  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:23.496229  6274 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 29: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:23.496274  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:23.497169  5558 raft_consensus.cc:2410] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:28:23.497076  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:23.499891  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:23.500541  5552 raft_consensus.cc:2376] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22. Re-sending same reply.
I20260709 12:28:23.501410  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:23.502470  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:23.503485  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:23.505467 14430 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.506443 14434 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.508975 14447 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:23.510107 14447 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:28:23.510754 14269 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.531s	user 0.005s	sys 0.005s
I20260709 12:28:23.511168 14269 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.512825  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3134) took 1859 ms. Trace:
I20260709 12:28:23.513175  5553 rpcz_store.cc:276] 0709 12:28:21.653700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.633880 (+980180us) service_pool.cc:224] Handling call
0709 12:28:22.634192 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.874581 (+240389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.512587 (+638006us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b54001be754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:23.512770 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":550653184}
I20260709 12:28:23.514786 14269 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:23.517010  6272 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1413) took 1556 ms (client timeout 1742 ms). Trace:
W20260709 12:28:23.517402  6272 rpcz_store.cc:269] 0709 12:28:21.960975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:21.961389 (+   414us) service_pool.cc:224] Handling call
0709 12:28:23.516823 (+1555434us) spinlock_profiling.cc:243] Waited 302 ms on lock 0x7b54006379d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:23.516952 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":694182144}
W20260709 12:28:23.518168 14456 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.717s	user 0.011s	sys 0.000s
I20260709 12:28:23.518642 14456 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.498167  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:23.521113  5557 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 24: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:23.521936  5561 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:23.524605  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:23.525225  6272 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:28:23.526546  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:23.533174  5552 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:23.489719 13960 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:23.494617 14339 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:23.540879 14339 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:23.542199 14452 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.534266 14200 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:23.544629 14200 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.461934 14233 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:23.548372  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:23.549757  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:23.537139  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:23.551231 14233 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.552040 14337 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.552762  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:23.558640  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:23.560842  5555 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:23.563122  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3397) took 2502 ms. Trace:
I20260709 12:28:23.563567  5555 rpcz_store.cc:276] 0709 12:28:21.060705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.146225 (+1085520us) service_pool.cc:224] Handling call
0709 12:28:22.146587 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:23.561274 (+1414687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.562812 (+  1538us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540044f054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:23.563046 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3252078976}
I20260709 12:28:23.550968  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:23.534901  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:23.566147  5558 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:28:23.537139  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:23.566649  5552 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 24
W20260709 12:28:23.565433  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 86337.3us
I20260709 12:28:23.569162  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:23.573071  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:23.576076  5633 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:28:23.577291  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:23.582938  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:23.583587  5555 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:23.479769 14317 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:23.534837  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:23.587778  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 53040.5us
I20260709 12:28:23.540751 14431 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:23.560874 14397 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.541s	user 0.009s	sys 0.003s
I20260709 12:28:23.592068  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:23.557291 14200 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.594455  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:23.594313  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:23.598688  5633 raft_consensus.cc:2410] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 25 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
W20260709 12:28:23.597282  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:23.600440  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:23.600682  5555 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:23.598632  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:23.564293 14456 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.602188  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:23.606902  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:23.582294 14499 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:23.602851  5526 leader_election.cc:400] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
I20260709 12:28:23.582185 14430 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:28:23.586951 14269 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:23.591948 14342 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:23.593529 13960 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:23.591678 14434 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:23.606549 14409 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:28:23.616783  5558 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:23.609892  5526 leader_election.cc:403] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:28:23.603866 14038 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:28:23.612618  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:23.625749  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:23.626242  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:23.628070  5633 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:23.615774 14456 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:23.628185  5552 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:23.633093 14456 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:23.634590 14456 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.604120 14244 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.021s
I20260709 12:28:23.630263 14038 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
I20260709 12:28:23.630228 14126 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:28:23.602665 14397 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:23.640682  5632 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 28: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:23.643031  6276 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:23.602852 14233 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.018s
I20260709 12:28:23.628280  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:23.632153  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:23.603479 14117 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.024s
I20260709 12:28:23.671427  5559 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:23.640132 14397 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.624382 14337 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:23.682286 13960 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:23.629048 14342 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:23.684561  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 52479.9us
I20260709 12:28:23.645385 14406 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:23.624789 14434 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.694905  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:23.696599  6278 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:23.702950  5633 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:23.690474 14233 consensus_queue.cc:1243] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 27.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:28:23.719043  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:23.741756  5628 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:23.723039  6278 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:23.725188  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:23.726310  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:23.726890 14342 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.728706  6279 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:28:23.748639  5630 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:23.700083 14406 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:23.738384  5559 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:23.740016  5601 leader_election.cc:400] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23.
I20260709 12:28:23.746124 14156 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:23.747771  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3152) took 1017 ms. Trace:
I20260709 12:28:23.750612  5560 rpcz_store.cc:276] 0709 12:28:22.730189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.149283 (+419094us) service_pool.cc:224] Handling call
0709 12:28:23.149584 (+   301us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:23.273456 (+123872us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:23.273505 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:23.274134 (+   629us) log.cc:844] Serialized 26 byte log entry
0709 12:28:23.693298 (+419164us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:23.693345 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.693427 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:23.747555 (+ 54128us) raft_consensus.cc:1685] finished
0709 12:28:23.747577 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:23.747714 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18286,"raft.queue_time_us":33757,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":4094,"thread_start_us":518685,"threads_started":2,"wal-append.queue_time_us":471557}
I20260709 12:28:23.750105  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:23.728957 14434 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:23.752318  6275 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:23.739656  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:23.759739  5633 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:28:23.760710  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:23.761574  5628 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:23.740531 14317 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:23.804941  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3391) took 1161 ms. Trace:
I20260709 12:28:23.805441  5636 rpcz_store.cc:276] 0709 12:28:22.643817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.644980 (+  1163us) service_pool.cc:224] Handling call
0709 12:28:22.645372 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:22.872146 (+226774us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.804631 (+932485us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b5400438854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:23.804869 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":434047360}
I20260709 12:28:23.807260  5636 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:23.808516 14406 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.749745 14409 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:23.799316 13960 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:23.827803 13960 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:23.828001 14447 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:23.828778 14447 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:23.828887 13960 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:23.831467 14447 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:23.764415 14434 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:23.826706 14409 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.835350 14339 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:23.909164  5630 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 21.
I20260709 12:28:23.910106  5633 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
W20260709 12:28:23.858269  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 1.43596e+06us
W20260709 12:28:23.858363  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1537) took 1925 ms (client timeout 2470 ms). Trace:
W20260709 12:28:23.898651  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:23.761762 14156 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:23.913810  5707 rpcz_store.cc:269] 0709 12:28:21.932752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:21.933335 (+   583us) service_pool.cc:224] Handling call
0709 12:28:23.858004 (+1924669us) spinlock_profiling.cc:243] Waited 952 ms on lock 0x7b54002f6054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:23.858235 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2189594880}
I20260709 12:28:23.762532  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:23.927938  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 165520us
W20260709 12:28:23.928211  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:23.928696  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:23.928922  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:28:23.934321  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:23.943527  5630 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:23.967653  5630 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 14. (term mismatch)
I20260709 12:28:23.968106  5630 pending_rounds.cc:85] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 13
I20260709 12:28:23.968312  5630 pending_rounds.cc:107] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260709 12:28:23.968536  5630 raft_consensus.cc:2889] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:28:23.989797  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:23.995599  5557 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:24.007413  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1799) took 1661 ms. Trace:
I20260709 12:28:24.007927  5629 rpcz_store.cc:276] 0709 12:28:22.345988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.372697 (+ 26709us) service_pool.cc:224] Handling call
0709 12:28:22.373042 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.408530 (+ 35488us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:22.408559 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:22.408601 (+    42us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:22.408632 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:24.007198 (+1598566us) spinlock_profiling.cc:243] Waited 35.3 ms on lock 0x7b54001f2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:24.007241 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:24.007359 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":81190656}
W20260709 12:28:24.012666  6237 leader_election.cc:336] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.771s (SENT)
I20260709 12:28:23.919898 14126 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:23.973649 14434 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:23.953168 14415 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:24.013713 14447 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:24.019802 14342 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:24.020963 14434 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:24.030987 14415 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:24.050836  5448 maintenance_manager.cc:419] P b3df7d767dc641b7bb01c3435d842dd5: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033379
I20260709 12:28:24.039042 14342 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:24.055326  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 126482us
I20260709 12:28:24.018218 14126 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.045316 14434 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:24.070569  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 122625us
I20260709 12:28:24.041392 14447 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.064427 14342 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:24.074208  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3258) took 1927 ms. Trace:
I20260709 12:28:24.073951  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:24.077344  5709 rpcz_store.cc:276] 0709 12:28:22.146994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.474975 (+327981us) service_pool.cc:224] Handling call
0709 12:28:22.475386 (+   411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.475648 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:28:22.784577 (+308929us) log.cc:844] Serialized 14 byte log entry
0709 12:28:23.074422 (+289845us) log.cc:844] Serialized 14 byte log entry
0709 12:28:24.073699 (+999277us) spinlock_profiling.cc:243] Waited 9.24 ms on lock 0x7b18003f1360. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:28:24.073855 (+   156us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:24.073910 (+    55us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:24.073969 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:24.074040 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:24.074154 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1079902,"prepare.queue_time_us":643550,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":81,"raft.queue_time_us":1043654,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":6149,"spinlock_wait_cycles":67046016,"thread_start_us":491984,"threads_started":2,"wal-append.queue_time_us":321509}
I20260709 12:28:24.087158  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3259) took 1939 ms. Trace:
I20260709 12:28:24.089732  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:24.090797  5630 raft_consensus.cc:2368] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31.
I20260709 12:28:24.093107  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:24.087798  5702 rpcz_store.cc:276] 0709 12:28:22.147241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.479534 (+332293us) service_pool.cc:224] Handling call
0709 12:28:22.479769 (+   235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:22.479967 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:28:22.898263 (+418296us) log.cc:844] Serialized 14 byte log entry
0709 12:28:24.086678 (+1188415us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:24.086760 (+    82us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:24.086821 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:24.086918 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:24.087082 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":934022,"prepare.queue_time_us":582305,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"thread_start_us":648641,"threads_started":2,"wal-append.queue_time_us":399090}
I20260709 12:28:24.098330  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:24.094378 14403 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:24.108433 14403 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.110430 14403 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:24.099328 14415 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.078065 13960 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:28:24.098141 14319 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.008s
I20260709 12:28:24.107820 14339 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:24.115602  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:24.117107  5709 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:24.115654  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1539) took 2145 ms. Trace:
I20260709 12:28:24.118728  5705 rpcz_store.cc:276] 0709 12:28:21.957700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:21.961229 (+  3529us) service_pool.cc:224] Handling call
0709 12:28:24.102304 (+2141075us) spinlock_profiling.cc:243] Waited 542 ms on lock 0x7b54002e81d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:24.102633 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1246237568}
I20260709 12:28:24.119921  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:24.120208  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:24.120885  6273 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:24.105507 14514 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:24.096536 14233 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:28:24.128796  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:24.131574  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:24.113723 14439 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:24.113940 13960 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:24.144515 13960 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:24.144299 14439 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.145466 13960 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.112553 14422 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:24.121716  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:24.134397  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:24.153519  5706 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:24.135285  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:24.112779 14156 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:24.138908 14318 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:24.139390 14403 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:28:24.111964 14507 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.029s
I20260709 12:28:24.179584 14403 consensus_queue.cc:1243] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 29.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:28:24.114871 14339 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:24.141860  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1804) took 1071 ms. Trace:
I20260709 12:28:24.142828  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.140183 14418 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.025s
I20260709 12:28:24.132441 14445 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:24.143311 13861 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:28:24.113586 14476 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:24.148509  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:24.148558  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.148530  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:24.157500  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:24.149467 14233 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:24.157351 14422 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.166217 14413 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:24.191728  5675 leader_election.cc:400] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31.
I20260709 12:28:24.192054  5706 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:28:24.192277  5675 leader_election.cc:403] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:28:24.168056 14499 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:24.132862 14514 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.172245  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:24.176715 14342 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.146741 14439 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:24.146741 13960 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:24.187243  5710 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:24.180310 14403 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:24.164397  5709 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:24.195843  5706 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:28:24.205130  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:24.186280  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:24.182891  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:24.181835  5636 rpcz_store.cc:276] 0709 12:28:23.070388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.849953 (+779565us) service_pool.cc:224] Handling call
0709 12:28:23.850410 (+   457us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:23.879953 (+ 29543us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:23.879999 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:23.921031 (+ 41032us) log.cc:844] Serialized 26 byte log entry
0709 12:28:24.069473 (+148442us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:24.069535 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:24.069635 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:24.141604 (+ 71969us) raft_consensus.cc:1685] finished
0709 12:28:24.141626 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:24.141795 (+   169us) inbound_call.cc:177] Queueing success response
0709 12:28:24.182836 (+ 41041us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x7b48002727d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
Metrics: {"mutex_wait_us":29012,"raft.queue_time_us":29171,"raft.run_cpu_time_us":2254,"raft.run_wall_time_us":271306,"spinlock_wait_cycles":62852352,"thread_start_us":55724,"threads_started":1,"wal-append.queue_time_us":110228}
I20260709 12:28:24.210914  5706 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:24.211297  5705 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 27: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:24.199918 14208 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31.
I20260709 12:28:24.211982 14358 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:24.184859 13861 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:24.212667 14419 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:24.212641 14358 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.213387 14419 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.188131 14422 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:24.215041 14358 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:24.189164 14413 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.181663  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:24.183630 14418 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:24.249207  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:24.251031  5709 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:24.252825  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47783.5us
I20260709 12:28:24.253720  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:24.180684 14339 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:24.255702  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.260056  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.260766  5636 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:28:24.261736  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.262599  5629 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 26: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:24.229266  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:24.264056 14337 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:24.219791 14403 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:24.208621 14444 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:24.212747 13861 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:24.249672 14418 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:24.184116 14445 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.268090  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:24.217271 14413 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:24.216233 14419 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:24.221524 14514 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:24.209585 14430 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.059s
I20260709 12:28:24.187005 14233 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.291298  5709 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:24.268916 14426 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:24.301703  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:24.315055  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:24.326980  5709 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:24.332314  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:24.336177  5709 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:24.278331 14439 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:28:24.338627  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 131753us
I20260709 12:28:24.278745 14507 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:24.325345 13861 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.314549 14419 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
I20260709 12:28:24.350363 13861 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:24.343931 14439 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:24.351312 14439 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:24.313189 14430 consensus_queue.cc:1243] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 23.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:28:24.326392 14408 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:24.275914 14444 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.353664 14418 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.354528  5702 raft_consensus.cc:2410] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:28:24.352985 14408 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.356575 14418 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:24.376315  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 86580.5us
I20260709 12:28:24.357002  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:24.290797 14434 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.384083  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:24.184784 14476 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.203366 14499 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.389495  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3261) took 2241 ms. Trace:
I20260709 12:28:24.384943  5709 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:28:24.396150  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:24.388234  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:24.400337  5709 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:24.401556  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:24.409061  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:24.390012  5704 rpcz_store.cc:276] 0709 12:28:22.147738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.863170 (+715432us) service_pool.cc:224] Handling call
0709 12:28:22.886415 (+ 23245us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:22.891297 (+  4882us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:22.891327 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:22.932552 (+ 41225us) log.cc:844] Serialized 52 byte log entry
0709 12:28:23.139409 (+206857us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:23.140320 (+   911us) log.cc:844] Serialized 14 byte log entry
0709 12:28:24.388636 (+1248316us) log.cc:844] Serialized 14 byte log entry
0709 12:28:24.389060 (+   424us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:24.389165 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:24.389225 (+    60us) raft_consensus.cc:1685] finished
0709 12:28:24.389244 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:24.389433 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":956292,"prepare.queue_time_us":393,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":966825,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":36551,"thread_start_us":419378,"threads_started":2,"wal-append.queue_time_us":252793}
I20260709 12:28:24.375975 14444 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:24.404556  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:24.417604  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:24.418488  5701 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:24.419744  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:24.419968  5709 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:24.328866 14339 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.366339 14430 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.047s
I20260709 12:28:24.430351  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:24.426237  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 125115us
W20260709 12:28:24.432456  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 208500us
I20260709 12:28:24.393978 14208 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:24.365084 14507 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.381752 14511 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.038s
I20260709 12:28:24.410948 14431 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:24.346575 14403 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 28.12, Last appended by leader: 12, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.428180 14358 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:24.447497 14499 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:24.446092  6272 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:28:24.455400  6280 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:24.456825  5553 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:28:24.457799  5555 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:28:24.447307 14434 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:24.407419 14426 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 12:28:24.430500 14408 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:24.463970  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.417065 14525 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 12:28:24.484741  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:24.485831  5555 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:24.465819  5704 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:28:24.465680  5709 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:24.486649  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:24.487057  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:24.452729 14419 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:24.447961 14476 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:24.456890 14445 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:24.482961  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 101230us
I20260709 12:28:24.485719  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:24.433817 14444 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:24.488238  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:24.455459 13812 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:24.457046 14318 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:24.490370  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:24.501389  6274 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:28:24.461529 14339 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:24.491597  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:24.485299 14409 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:24.493319  5560 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:28:24.483016  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:24.493988  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:24.496603  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:24.480337  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:24.496932  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:24.496218  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:24.510435  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:24.510735  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:24.511777  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:24.504577  5555 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:24.493860 14408 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:24.492801 14342 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:24.440783 14540 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.060s
I20260709 12:28:24.465051 14413 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 12:28:24.515518  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:24.566144  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 132831us
W20260709 12:28:24.566668  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70524.4us
I20260709 12:28:24.569047  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:24.526039  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:24.571100  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:24.572542  5552 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:24.573916  5552 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:28:24.574293  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:24.524039  5447 maintenance_manager.cc:643] P b3df7d767dc641b7bb01c3435d842dd5: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.403s	user 0.027s	sys 0.012s Metrics: {"bytes_written":5585,"delete_count":0,"lbm_write_time_us":954,"lbm_writes_lt_1ms":6,"reinsert_count":0,"update_count":209}
I20260709 12:28:24.578485  5448 maintenance_manager.cc:419] P b3df7d767dc641b7bb01c3435d842dd5: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.286372
I20260709 12:28:24.514827 14319 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:24.580435 14549 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.015s
I20260709 12:28:24.581076  5706 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:24.489503 14499 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:28:24.571214  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:24.584080  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:24.591387  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:24.592455 14430 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:24.593036 14430 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:24.594130 14430 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.596846 14507 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:24.546370  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:24.596964 14552 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:24.599102 14552 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:24.599182  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53000.6us
I20260709 12:28:24.599586 14552 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:24.602701 14341 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:24.565979 14233 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:24.606012 13960 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:24.606496 14233 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.606945 13960 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:28:24.607367  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:24.608675 14470 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.594s	user 0.000s	sys 0.016s
I20260709 12:28:24.589102 14156 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:24.610195 14156 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:24.610384  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3387) took 2415 ms. Trace:
I20260709 12:28:24.611171 14156 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.613905 14244 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:24.614315 14244 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:24.612015  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:24.615206 14244 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.616339  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 1744) took 1120 ms. Trace:
I20260709 12:28:24.616911  5556 rpcz_store.cc:276] 0709 12:28:23.496343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.496484 (+   141us) service_pool.cc:224] Handling call
0709 12:28:24.611848 (+1115364us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:28:24.616237 (+  4389us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":32411904}
I20260709 12:28:24.618945  5556 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:28:24.620508  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:24.621726  5709 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 26 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
I20260709 12:28:24.623163  5709 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:28:24.623888  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3428) took 1095 ms. Trace:
I20260709 12:28:24.624311  5557 rpcz_store.cc:276] 0709 12:28:23.528815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.558339 (+ 29524us) service_pool.cc:224] Handling call
0709 12:28:24.623703 (+1065364us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b54001af254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:24.623827 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1002638592}
I20260709 12:28:24.594020 14511 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:28:24.627082 14511 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:24.627525 14511 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:24.626463 14439 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.630270 14439 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:24.523145 13969 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:24.526163 14408 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:24.632261  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1540) took 1029 ms. Trace:
I20260709 12:28:24.633067 14408 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:28:24.624320  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:24.634455 14408 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:24.632728  6279 rpcz_store.cc:276] 0709 12:28:23.602381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.603045 (+   664us) service_pool.cc:224] Handling call
0709 12:28:23.603323 (+   278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:23.748780 (+145457us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:24.632001 (+883221us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b5400055854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:24.632197 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":287531392}
I20260709 12:28:24.518975 14415 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:24.586251  5555 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
W20260709 12:28:24.634586 14520 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.567s	user 0.000s	sys 0.001s
I20260709 12:28:24.640229  5710 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20. Ops: []
W20260709 12:28:24.640298  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 73138.7us
I20260709 12:28:24.610834  5635 rpcz_store.cc:276] 0709 12:28:22.194615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.340619 (+146004us) service_pool.cc:224] Handling call
0709 12:28:22.340945 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:22.347893 (+  6948us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:22.347939 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:22.356088 (+  8149us) log.cc:844] Serialized 26 byte log entry
0709 12:28:22.743324 (+387236us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:22.743397 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:22.743487 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:24.582354 (+1838867us) spinlock_profiling.cc:243] Waited 8.46 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:24.610040 (+ 27686us) raft_consensus.cc:1685] finished
0709 12:28:24.610069 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:24.610317 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6520,"raft.queue_time_us":6677,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":16830,"spinlock_wait_cycles":19449088,"thread_start_us":386618,"threads_started":1,"wal-append.queue_time_us":527408}
I20260709 12:28:24.643170  5710 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:24.632130  5706 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:28:24.644706  5710 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:28:24.591238  6274 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
W20260709 12:28:24.620826  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:24.572537  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:24.649128  6274 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:28:24.503706 13812 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.656167  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.657621  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.658413  5630 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:24.660060  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.662060  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.640249 14415 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:24.649261 14156 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:28:24.670462  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 69187.7us
W20260709 12:28:24.671550  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 80252.1us
I20260709 12:28:24.649893 14244 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:24.513492 14318 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.654202 14499 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:24.648727 14183 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:24.673477  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:24.672307 14415 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:24.673858  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:24.678340  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.678702  6274 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:28:24.647110 13960 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.032s
I20260709 12:28:24.684288  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.679877  5630 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:28:24.686378  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:24.693188  5552 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 29 index: 12. (index mismatch)
I20260709 12:28:24.696465  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:24.698030 14430 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:28:24.681278 14183 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.701086  5710 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:24.681641 14317 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:24.679863 14233 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:24.669906 14550 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.036s
I20260709 12:28:24.693959 14549 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:28:24.712872  5634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3379) took 2600 ms (client timeout 1920 ms). Trace:
W20260709 12:28:24.730024  5634 rpcz_store.cc:269] 0709 12:28:22.112289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.185127 (+ 72838us) service_pool.cc:224] Handling call
0709 12:28:24.712677 (+2527550us) spinlock_profiling.cc:243] Waited 734 ms on lock 0x7b5400240754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:24.712779 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1688225152}
I20260709 12:28:24.720137 14233 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:24.682036 14499 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.737284 14409 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:24.737890 14409 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:24.738123 14499 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:24.720922 14318 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:24.720937  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:24.742179  5710 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:24.714403 14317 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.726356 14183 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:24.727510 14507 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:28:24.734751 14549 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:24.706584 13960 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:24.739012 14409 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.754096  5706 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:24.719120 14552 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:24.764094  6276 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1417) took 1283 ms (client timeout 1609 ms). Trace:
W20260709 12:28:24.774794  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 88486.2us
W20260709 12:28:24.803135  6276 rpcz_store.cc:269] 0709 12:28:23.481144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.481360 (+   216us) service_pool.cc:224] Handling call
0709 12:28:24.763896 (+1282536us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54000b54d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:24.764025 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":345442048}
I20260709 12:28:24.830443  5710 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:24.839293  5706 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:28:24.841836 14317 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:24.845628  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:24.847777 14550 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:24.848528 14550 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.847251  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:24.854532 13812 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:24.855381  5710 pending_rounds.cc:85] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 14
I20260709 12:28:24.855727  5710 pending_rounds.cc:107] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 23.15
I20260709 12:28:24.856132  5710 raft_consensus.cc:2889] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:24.855576 14549 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.851696 14561 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:28:24.858974 14561 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:28:24.859627 14561 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:24.852773  5559 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:24.861568 14552 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:24.862305  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:24.863112 13960 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:24.863795 13960 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:24.867856  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:24.781491 14342 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:24.864180  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:24.869580 14342 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.883684 14342 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:24.892141  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:24.896416  5553 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
W20260709 12:28:24.897393  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 192884us
W20260709 12:28:24.905131  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:24.866446 13960 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:24.908464  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 478162us
I20260709 12:28:24.915661  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:24.922233  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:24.902508 14552 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:24.923239  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:24.924836  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:24.925860 13861 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:24.932396 14208 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.009s
I20260709 12:28:24.894074 14512 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:28:24.935515 14439 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:24.936056 14439 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:24.936996 14439 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.964690  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.961810  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:24.966843  5631 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:28:24.968271 14208 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:24.968880 14208 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:24.969149 14552 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:24.969869 14208 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.970660 14552 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:24.970112 14511 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:24.972486  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:24.973176  6276 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:24.877417 14430 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:28:24.974027  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.968075 14550 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:24.982185  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:24.989813  5635 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:28:24.992719  5634 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
W20260709 12:28:24.979940 14536 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.627s	user 0.009s	sys 0.000s
I20260709 12:28:24.994899  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:24.976657  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 83546.3us
I20260709 12:28:25.005806  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:25.010459  5635 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:28:25.013818  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:25.015631  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:25.002446 14445 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.020040  5555 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:28:25.020308  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:25.023662  6276 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260709 12:28:25.024804  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:25.025864  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3133) took 2195 ms. Trace:
I20260709 12:28:25.025977  6274 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:25.026301  5627 rpcz_store.cc:276] 0709 12:28:22.830512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.900694 (+ 70182us) service_pool.cc:224] Handling call
0709 12:28:22.978391 (+ 77697us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:23.304037 (+325646us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:25.025641 (+1721604us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b540009e2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:25.025806 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":744384768}
I20260709 12:28:25.031908  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:25.034072  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:25.034624  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:25.034737  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:25.034814  6279 raft_consensus.cc:2368] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31.
I20260709 12:28:25.038981  6276 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 20: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:25.042340 14421 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:28:25.043613  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:25.044260  5710 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:25.046572  5702 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:28:25.033038 14445 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:25.047742 14341 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:28:25.022768 14512 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:25.049129 14293 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.011s
I20260709 12:28:25.057713 14445 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:25.061580 14511 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:25.065062 14415 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:25.067123  5630 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:25.065918  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:25.071246  5710 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:25.073799  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:25.074702  6279 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:25.074347  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:25.079517  5552 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:28:25.081653  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:25.097748 14578 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.098115  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:25.098995  6274 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:25.098577 14578 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:25.089284  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:25.080631  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:25.294404  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 147277us
I20260709 12:28:25.317691  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:25.326030  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:25.317904  5557 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:28:25.347905 14319 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:25.348984 14319 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:25.347908  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:25.327858  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:25.393488  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:25.394713  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:25.102891 14512 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:25.399843  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:25.401027  5552 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:25.402432  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:25.404731  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:25.407982  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:25.352126 14319 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:25.420656  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:25.438494  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 349315us
I20260709 12:28:25.440021  5627 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 19. Current term is 20. Ops: []
I20260709 12:28:25.439967  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:25.295027 14172 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:25.326653 14156 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.115s
I20260709 12:28:25.448221 14142 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:28:25.459726  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:25.454262  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 482477us
I20260709 12:28:25.408610 14511 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:25.464867  5554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3420) took 1971 ms (client timeout 1877 ms). Trace:
W20260709 12:28:25.465627  5554 rpcz_store.cc:269] 0709 12:28:23.493885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.494005 (+   120us) service_pool.cc:224] Handling call
0709 12:28:25.437570 (+1943565us) spinlock_profiling.cc:243] Waited 1.65 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754deee 000000000036eb39 00007f4fff5c11e0 00007f4fff68a0c4 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:28:25.464753 (+ 27183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3804416}
I20260709 12:28:25.439635 14525 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:25.463420 14535 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.026s	user 0.006s	sys 0.000s
I20260709 12:28:25.459302 14319 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.451067 14244 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:25.510219  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3598) took 3111 ms. Trace:
I20260709 12:28:25.510525 14244 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:25.515460  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:25.467243  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:25.492151  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:25.496336 14319 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:25.498342 14489 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 2.254s	user 0.005s	sys 0.017s
I20260709 12:28:25.500991 14578 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:25.503047 14415 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:25.461918 14550 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 12:28:25.460315 14172 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:25.507047 14534 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.022s	user 0.009s	sys 0.010s
I20260709 12:28:25.540763  6276 raft_consensus.cc:2410] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 26 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260709 12:28:25.475572 14511 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:25.541397 14172 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:25.543110 14511 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:25.541421  5555 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
W20260709 12:28:25.485061 14488 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 2.239s	user 0.007s	sys 0.013s
I20260709 12:28:25.463743 14142 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:25.510805  5703 rpcz_store.cc:276] 0709 12:28:22.398363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.135001 (+736638us) service_pool.cc:224] Handling call
0709 12:28:23.135415 (+   414us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:24.084652 (+949237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:24.084710 (+    58us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:24.097596 (+ 12886us) log.cc:844] Serialized 52 byte log entry
0709 12:28:24.273764 (+176168us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:24.407489 (+133725us) log.cc:844] Serialized 14 byte log entry
0709 12:28:24.408668 (+  1179us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:24.409023 (+   355us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:25.107365 (+698342us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:25.509833 (+402468us) raft_consensus.cc:1685] finished
0709 12:28:25.509957 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:25.510147 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":971785,"prepare.queue_time_us":232836,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":961315,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":7932,"spinlock_wait_cycles":342972672,"thread_start_us":235523,"threads_started":2,"wal-append.queue_time_us":263261}
W20260709 12:28:25.523360  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 56254.5us
I20260709 12:28:25.523409  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3601) took 1306 ms. Trace:
I20260709 12:28:25.563977  5704 rpcz_store.cc:276] 0709 12:28:24.217077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:24.412749 (+195672us) service_pool.cc:224] Handling call
0709 12:28:24.413841 (+  1092us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:24.486270 (+ 72429us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:25.523118 (+1036848us) spinlock_profiling.cc:243] Waited 51.6 ms on lock 0x7b54002d23d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:25.523352 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":118633728}
I20260709 12:28:25.471074 14430 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:25.570586  5703 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
W20260709 12:28:25.529167  5631 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.015s	sys 0.014s
W20260709 12:28:25.572309  5631 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.015s	sys 0.014s
I20260709 12:28:25.530191  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1752) took 2007 ms. Trace:
W20260709 12:28:25.576455  5629 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.027s	sys 0.033s
W20260709 12:28:25.577020  5629 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.027s	sys 0.033s
I20260709 12:28:25.576721  5561 rpcz_store.cc:276] 0709 12:28:23.522529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.522667 (+   138us) service_pool.cc:224] Handling call
0709 12:28:23.537573 (+ 14906us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:23.593590 (+ 56017us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:23.593622 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:23.605083 (+ 11461us) log.cc:844] Serialized 26 byte log entry
0709 12:28:23.851133 (+246050us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:24.199425 (+348292us) log.cc:844] Serialized 14 byte log entry
0709 12:28:24.200898 (+  1473us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:25.518788 (+1317890us) spinlock_profiling.cc:243] Waited 55.6 ms on lock 0x7b540023ffd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:25.518862 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:25.518924 (+    62us) raft_consensus.cc:1685] finished
0709 12:28:25.529904 (+ 10980us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:25.530116 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":195242,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":71,"raft.queue_time_us":695,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":4813,"spinlock_wait_cycles":127875456,"thread_start_us":487313,"threads_started":2,"wal-append.queue_time_us":466334,"wal-append.run_cpu_time_us":2911,"wal-append.run_wall_time_us":1212221}
W20260709 12:28:25.505137 14573 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:28:25.528658 14514 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:25.492668 14525 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:25.528180 14574 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:28:25.539322  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:25.562209 14142 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:25.584216  5554 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 20. (term mismatch)
I20260709 12:28:25.595614 14525 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:25.590147  5561 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260709 12:28:25.601199  5636 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:25.640152  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:25.568605  5706 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 12:28:25.641104  5706 pending_rounds.cc:85] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 14
I20260709 12:28:25.641412  5706 pending_rounds.cc:107] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 23.15
I20260709 12:28:25.641667  5706 raft_consensus.cc:2889] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:25.602281  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:25.580461 14552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.030s	sys 0.039s
I20260709 12:28:25.569736 14549 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:25.587203 14568 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:25.655579 14552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.030s	sys 0.043s
W20260709 12:28:25.623639 14531 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.267s	user 0.004s	sys 0.027s
I20260709 12:28:25.624990  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3160) took 1167 ms. Trace:
I20260709 12:28:25.626420  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3157) took 1172 ms. Trace:
I20260709 12:28:25.511768 14244 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:25.638876  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3586) took 3680 ms. Trace:
I20260709 12:28:25.645316  5703 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:25.653244  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:25.643882  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:25.693116  5708 rpcz_store.cc:276] 0709 12:28:21.958004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:22.486308 (+528304us) service_pool.cc:224] Handling call
0709 12:28:22.508272 (+ 21964us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:22.786312 (+278040us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:22.786432 (+   120us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:22.787961 (+  1529us) log.cc:844] Serialized 52 byte log entry
0709 12:28:23.032941 (+244980us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:23.032978 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:23.033043 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:25.401775 (+2368732us) spinlock_profiling.cc:243] Waited 9.99 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:25.637870 (+236095us) raft_consensus.cc:1685] finished
0709 12:28:25.637905 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:25.638800 (+   895us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7560,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":13765,"spinlock_wait_cycles":22974720,"thread_start_us":430849,"threads_started":2,"wal-append.queue_time_us":372343}
I20260709 12:28:25.698617  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:25.701576  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 57830.9us
I20260709 12:28:25.655676  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:25.656205  5554 pending_rounds.cc:85] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 19
I20260709 12:28:25.704340  5554 pending_rounds.cc:107] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 27.20
I20260709 12:28:25.704560  5707 raft_consensus.cc:2410] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:28:25.704758  5554 raft_consensus.cc:2889] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:28:25.621102  5632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3137) took 2669 ms (client timeout 2981 ms). Trace:
I20260709 12:28:25.582089 14430 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:25.705435  5632 rpcz_store.cc:269] 0709 12:28:22.951291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.202847 (+251556us) service_pool.cc:224] Handling call
0709 12:28:25.620703 (+2417856us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54001f2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:25.621005 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":986528640}
I20260709 12:28:25.705576  5560 rpcz_store.cc:276] 0709 12:28:24.453959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:24.454120 (+   161us) service_pool.cc:224] Handling call
0709 12:28:24.454435 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:24.508865 (+ 54430us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:25.626101 (+1117236us) spinlock_profiling.cc:243] Waited 31.8 ms on lock 0x7b540044eb54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:25.626350 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":73026304}
I20260709 12:28:25.706643  5553 rpcz_store.cc:276] 0709 12:28:24.457359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:24.486422 (+ 29063us) service_pool.cc:224] Handling call
0709 12:28:25.624794 (+1138372us) spinlock_profiling.cc:243] Waited 384 ms on lock 0x7b54003c4bd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:25.624926 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":883118336}
I20260709 12:28:25.657116 14525 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:25.741519  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 280470us
I20260709 12:28:25.743521  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:25.676942 14581 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.747094  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:25.751171  5560 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:28:25.715507  5708 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:28:25.734761 14341 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:25.734768 14558 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.729039 14415 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:25.661029 14568 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:25.658134 14549 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:25.763983 14341 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:25.805879  5703 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:28:25.744562 14525 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:25.815241  5526 leader_election.cc:336] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.609s (SENT)
I20260709 12:28:25.772907 14558 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:25.758469 14581 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:25.775027 14415 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:25.834746  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 267638us
I20260709 12:28:25.814680 14578 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:28:25.814378 14341 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:25.823685 14322 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:25.859495 14587 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:28:25.864704 14576 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:28:25.879747 14564 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:28:25.927102  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3621) took 1234 ms. Trace:
I20260709 12:28:25.927752  5705 rpcz_store.cc:276] 0709 12:28:24.692560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:24.906245 (+213685us) service_pool.cc:224] Handling call
0709 12:28:24.906722 (+   477us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:24.916907 (+ 10185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:24.916942 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:24.920084 (+  3142us) log.cc:844] Serialized 52 byte log entry
0709 12:28:25.069493 (+149409us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:25.069525 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:25.069574 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:25.926758 (+857184us) raft_consensus.cc:1685] finished
0709 12:28:25.926794 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:25.927010 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14529,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":8502,"thread_start_us":136081,"threads_started":1,"wal-append.queue_time_us":698198}
W20260709 12:28:25.880239 14583 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
W20260709 12:28:25.880887 14586 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:28:25.881954 14512 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:28:25.883373 14511 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:28:25.886776 14205 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:28:25.887436 14038 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:25.890117 13812 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:28:25.937000  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 190152us
I20260709 12:28:25.903482  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:25.957167  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:25.957248  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 142085us
W20260709 12:28:25.944110 14537 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.458s	user 0.000s	sys 0.011s
W20260709 12:28:25.919582  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:25.867611 14589 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.050s
I20260709 12:28:25.903764 14430 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:25.905095 14514 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:25.968482  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:25.969183  6274 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:25.944322 14558 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:25.937563 14578 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:25.937762 14322 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:25.937311 14512 consensus_queue.cc:1243] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 20.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:28:25.956271 13812 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:25.953051 14126 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:25.954700 14319 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:25.887933  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:25.920481 14568 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:25.964561  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:25.915067  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:25.972972  5675 connection.cc:256] RPC call timeout handler was delayed by -1.4258619870001894s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17600000000000002s for a response
I20260709 12:28:25.941489 14038 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:26.021353  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:26.023231 13812 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:25.965924 14337 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:26.024318  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:25.965674 14549 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:26.025429  6274 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:26.027014  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:26.028220  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:26.029563  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:26.030623  5675 connection.cc:256] RPC call timeout handler was delayed by -1.4203693720000956s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
W20260709 12:28:26.031014  5675 connection.cc:256] RPC call timeout handler was delayed by -1.4015619870001501s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1787s for a response
I20260709 12:28:26.032668  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:26.037698  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:26.040786  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:26.042670  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:26.044937  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3144) took 1914 ms. Trace:
I20260709 12:28:26.028915 14413 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.032522 14294 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:26.045887  5628 rpcz_store.cc:276] 0709 12:28:24.130099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:24.130389 (+   290us) service_pool.cc:224] Handling call
0709 12:28:24.130788 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:24.131054 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:28:24.153861 (+ 22807us) log.cc:844] Serialized 14 byte log entry
0709 12:28:25.893852 (+1739991us) spinlock_profiling.cc:243] Waited 2.72 ms on lock 0x7b180035b360. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:28:26.043657 (+149805us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:26.044497 (+   840us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:26.044569 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:26.044710 (+   141us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:26.044865 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7766,"prepare.queue_time_us":22449,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"spinlock_wait_cycles":6247168,"thread_start_us":141620,"threads_started":1}
I20260709 12:28:26.053604  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:26.039256 13861 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:26.042004 14358 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:28:26.063767 14205 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.075290  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:26.076305  5629 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:26.046332 14561 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:26.079903  6276 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:26.061566 14603 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.090309  6274 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:28:26.049935 14208 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
W20260709 12:28:26.086558  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45863.8us
I20260709 12:28:26.047837 14439 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.041s
I20260709 12:28:26.112659 14415 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:26.113523 14208 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:26.075966 14578 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.113987 14208 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:26.114919 14208 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.096963 14603 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.117175 14322 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:26.118578  5675 leader_election.cc:400] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31.
I20260709 12:28:26.119598  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:26.055042 14413 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:26.124085  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:26.096509 14358 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:26.127156  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:26.126063 14294 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.010s
W20260709 12:28:26.137131 14602 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
W20260709 12:28:26.138371 14608 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
W20260709 12:28:26.139662 14592 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
I20260709 12:28:26.140708  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:26.127621 13861 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:28:26.145021 14600 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:28:26.145581 14294 consensus_queue.cc:1243] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 25.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:28:26.146196 14294 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:26.146849 14294 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:26.147186 14341 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.148306  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:26.149199 13861 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:26.149633 13861 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:26.150019 13861 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:26.150324  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:26.150857 14341 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:26.152923 14591 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.182s	user 0.000s	sys 0.001s
I20260709 12:28:26.062242 14038 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.154042  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3174) took 1264 ms. Trace:
I20260709 12:28:26.154572 14038 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:26.154407  5552 rpcz_store.cc:276] 0709 12:28:24.889481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:25.522843 (+633362us) service_pool.cc:224] Handling call
0709 12:28:25.532694 (+  9851us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:25.550879 (+ 18185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:25.551126 (+   247us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:25.552082 (+   956us) log.cc:844] Serialized 26 byte log entry
0709 12:28:25.840204 (+288122us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:25.840261 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:25.897201 (+ 56940us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:26.153811 (+256610us) raft_consensus.cc:1685] finished
0709 12:28:26.153831 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:26.153976 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1977,"raft.queue_time_us":24359,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":8667,"thread_start_us":287732,"threads_started":1,"wal-append.queue_time_us":384679}
I20260709 12:28:26.150965  5628 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:26.156394  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:26.157096  5552 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:26.158357  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:26.159590  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:26.160586  5552 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 26: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:26.108062 14589 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:26.169809  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:26.171346  5552 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:28:26.084930 14205 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.104784 14587 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:26.118736 14525 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:26.187827  5552 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:28:26.182137  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:26.193728  5628 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
W20260709 12:28:26.194681  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:26.075793 14514 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:26.198068  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:26.184378 14601 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.220s	user 0.000s	sys 0.001s
I20260709 12:28:26.178926 14613 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:26.177160 14581 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:26.207856  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:26.209574  5703 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:28:26.186053 14358 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:26.212899  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3622) took 1213 ms. Trace:
I20260709 12:28:26.201851  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:26.183043 14599 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.179462 14445 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.034s
W20260709 12:28:26.209789 14422 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.709s	user 0.014s	sys 0.018s
I20260709 12:28:26.198069 14294 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:28:26.216684 14422 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.723s	user 0.014s	sys 0.018s
I20260709 12:28:26.216809 14294 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:26.217464 14294 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:26.213337  5710 rpcz_store.cc:276] 0709 12:28:24.999604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:25.075195 (+ 75591us) service_pool.cc:224] Handling call
0709 12:28:25.401371 (+326176us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:25.601278 (+199907us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:25.601327 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:25.609243 (+  7916us) log.cc:844] Serialized 52 byte log entry
0709 12:28:25.908589 (+299346us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:26.056525 (+147936us) log.cc:844] Serialized 14 byte log entry
0709 12:28:26.212603 (+156078us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:26.212683 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:26.212723 (+    40us) raft_consensus.cc:1685] finished
0709 12:28:26.212736 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:26.212846 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":199308,"prepare.queue_time_us":121608,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":257072,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":12264,"spinlock_wait_cycles":1084800,"thread_start_us":526279,"threads_started":3,"wal-append.queue_time_us":352049}
I20260709 12:28:26.189057 14434 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:26.218664 14548 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.628s	user 0.000s	sys 0.021s
W20260709 12:28:26.239097  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:26.231285  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:26.245812  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:26.244501 14616 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.248970 14616 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:26.249358 14205 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:26.250866  5708 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: [31.18-31.18]
I20260709 12:28:26.251261 14616 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:26.230749 14617 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.213166 14613 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:26.254070  5710 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 16. (term mismatch)
I20260709 12:28:26.254590  5710 pending_rounds.cc:85] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 15
I20260709 12:28:26.254886  5710 pending_rounds.cc:107] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 23.16
I20260709 12:28:26.255177  5710 raft_consensus.cc:2889] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:26.194993 14587 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.261322  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:26.262661  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 49440.5us
I20260709 12:28:26.257447 14617 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:26.263805 14603 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:26.265307 14617 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:26.257812 14422 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:26.265923 14403 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:26.266211 14604 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 12:28:26.267478  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3179) took 1261 ms. Trace:
I20260709 12:28:26.271360 14413 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:26.267832  5556 rpcz_store.cc:276] 0709 12:28:25.005965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:25.659720 (+653755us) service_pool.cc:224] Handling call
0709 12:28:25.735318 (+ 75598us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:25.735659 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:25.735688 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:25.741899 (+  6211us) log.cc:844] Serialized 26 byte log entry
0709 12:28:25.973812 (+231913us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:28:25.973867 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:25.973981 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:26.267260 (+293279us) raft_consensus.cc:1685] finished
0709 12:28:26.267285 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:26.267419 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4950,"raft.queue_time_us":2586,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":2649,"thread_start_us":193931,"threads_started":1,"wal-append.queue_time_us":343535}
W20260709 12:28:26.273762  5559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3164) took 1815 ms (client timeout 2331 ms). Trace:
W20260709 12:28:26.280426  5559 rpcz_store.cc:269] 0709 12:28:24.458472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:24.572211 (+113739us) service_pool.cc:224] Handling call
0709 12:28:26.273595 (+1701384us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b54003c46d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:26.273705 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":465099008}
I20260709 12:28:26.285562  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:26.287534  5710 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:26.290467  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3178) took 1284 ms. Trace:
I20260709 12:28:26.290841  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:26.290970  5561 rpcz_store.cc:276] 0709 12:28:25.005683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:25.649085 (+643402us) service_pool.cc:224] Handling call
0709 12:28:25.649457 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:25.649723 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:25.649761 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:25.655434 (+  5673us) log.cc:844] Serialized 26 byte log entry
0709 12:28:25.828974 (+173540us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:25.829038 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:25.829152 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:26.290199 (+461047us) raft_consensus.cc:1685] finished
0709 12:28:26.290223 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:26.290399 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":172724,"threads_started":1,"wal-append.queue_time_us":267003}
I20260709 12:28:26.292771  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:26.293597  5708 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:26.295296  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:26.296523  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:26.319643  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:26.322079  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 268515us
I20260709 12:28:26.324244  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:26.366747  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:26.325845  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:26.258423 14613 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:26.229197 14434 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.329272  5559 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:28:26.285662 14322 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.222765 14599 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:26.328163 14548 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:28:26.326162 14525 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.020s
I20260709 12:28:26.323764 14512 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:28:26.357769  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3136) took 3406 ms. Trace:
I20260709 12:28:26.402693  5633 rpcz_store.cc:276] 0709 12:28:22.950922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:23.801203 (+850281us) service_pool.cc:224] Handling call
0709 12:28:23.909890 (+108687us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:23.910507 (+   617us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:26.357448 (+2446941us) spinlock_profiling.cc:243] Waited 86.1 ms on lock 0x7b540008bbd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:26.357675 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":197993728}
I20260709 12:28:26.355497 14413 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.315325 14625 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.315672 14422 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:26.421636 14038 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.422284 14038 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:26.425174 14587 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:26.426141 14422 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:26.364418  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 180812us
I20260709 12:28:26.428948 14618 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.429626 14618 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.370716  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:26.431648  5556 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:26.431900 14618 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:26.372219  5561 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:26.434955 14618 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:28:26.435702  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:26.376490 14322 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:26.377893 14615 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:28:26.377961  5559 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:28:26.378254 14434 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:26.383481 13960 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:26.386978  5554 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:28:26.347649 14619 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.397377  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:26.507720  5628 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:26.509435  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:26.399577  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:26.371441  5710 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:26.493908  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:26.494328  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:26.511562  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 81264.7us
I20260709 12:28:26.370138 14622 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:28:26.493755  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 117976us
W20260709 12:28:26.517706  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 118262us
I20260709 12:28:26.518467  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:26.519572  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:26.520161  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:26.496512  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:26.438064 14512 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:26.493613  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55887.9us
I20260709 12:28:26.418128 14358 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:26.514729  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:26.515679  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:26.540750  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:26.543931  6276 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:26.545013  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:26.546872  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:26.547911  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:26.548591  5710 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:28:26.549012  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:26.549413  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:26.549820  5628 raft_consensus.cc:2410] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 30 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 17.
I20260709 12:28:26.549976  5710 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:26.529539 14618 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:26.514189 13960 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:26.555218  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:26.546969 14587 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.013s
I20260709 12:28:26.450507 14413 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.561911  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3155) took 2109 ms. Trace:
I20260709 12:28:26.557600  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:26.560310  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3428) took 1531 ms. Trace:
I20260709 12:28:26.450106 14625 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:26.571202  5705 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:26.575203  5627 rpcz_store.cc:276] 0709 12:28:25.028858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:25.029035 (+   177us) service_pool.cc:224] Handling call
0709 12:28:25.064762 (+ 35727us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:25.455920 (+391158us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:26.560067 (+1104147us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b54001f2a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:26.560250 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":827229440}
I20260709 12:28:26.579721  5709 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 12. Preceding OpId from leader: term: 29 index: 12. (term mismatch)
I20260709 12:28:26.580791  5709 pending_rounds.cc:85] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 11
I20260709 12:28:26.581219  5709 pending_rounds.cc:107] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 28.12
I20260709 12:28:26.581646  5709 raft_consensus.cc:2889] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:28:26.559460 14532 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 2.077s	user 0.012s	sys 0.001s
W20260709 12:28:26.556948 14620 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.223s	user 0.001s	sys 0.000s
I20260709 12:28:26.563619 14631 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:26.513413 13861 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.569257  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:26.559367 14038 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:26.553143 14358 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 12:28:26.580983 14244 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:26.530722 14578 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.056s
I20260709 12:28:26.582777 14624 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.591464 14631 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:26.593462 14322 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:26.594066 14625 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:26.531208 14619 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:26.595819 14439 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.598052 14512 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:26.599268 14607 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.463s	user 0.001s	sys 0.000s
I20260709 12:28:26.559571 14422 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:26.560927 14447 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:26.557096 14593 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:28:26.607905 14244 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:26.605245 14294 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:26.695572  5447 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.336s	user 0.007s	sys 0.005s
I20260709 12:28:23.850572  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:26.823508 14294 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:26.566977  5558 rpcz_store.cc:276] 0709 12:28:24.452892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:24.453042 (+   150us) service_pool.cc:224] Handling call
0709 12:28:24.453621 (+   579us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:24.459821 (+  6200us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:24.459857 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:24.460482 (+   625us) log.cc:844] Serialized 26 byte log entry
0709 12:28:26.561372 (+2100890us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b18004ab900. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1
0709 12:28:26.561502 (+   130us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:26.561526 (+    24us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:26.561576 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:26.561607 (+    31us) raft_consensus.cc:1685] finished
0709 12:28:26.561613 (+     6us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:26.561854 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5649,"raft.queue_time_us":10376,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":6999,"spinlock_wait_cycles":5128576,"thread_start_us":27962,"threads_started":1,"wal-append.queue_time_us":133234,"wal-append.run_cpu_time_us":1943,"wal-append.run_wall_time_us":1037675}
I20260709 12:28:26.725276  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3186) took 1211 ms. Trace:
I20260709 12:28:26.725636  5559 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:26.729996  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3171) took 1862 ms. Trace:
I20260709 12:28:26.589180  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:26.889259  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:26.889622 14038 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:26.890489  6280 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:26.890606 14038 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.891654  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:26.892174  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:26.892822  6273 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:26.894225 14322 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.860739  5560 rpcz_store.cc:276] 0709 12:28:25.514302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:25.814577 (+300275us) service_pool.cc:224] Handling call
0709 12:28:25.814971 (+   394us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:25.978703 (+163732us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:25.978742 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:26.021176 (+ 42434us) log.cc:844] Serialized 26 byte log entry
0709 12:28:26.097951 (+ 76775us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:28:26.098013 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:26.098113 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:26.724628 (+626515us) raft_consensus.cc:1685] finished
0709 12:28:26.724663 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:26.725070 (+   407us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8769,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":2691,"thread_start_us":234423,"threads_started":2,"wal-append.queue_time_us":200849}
I20260709 12:28:26.895155 14322 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.897681 14421 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:28:26.735833  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:26.901732 14434 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:28:26.901994 14578 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:28:26.743220  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3433) took 1228 ms. Trace:
W20260709 12:28:26.741433  5630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3425) took 1759 ms (client timeout 1591 ms). Trace:
I20260709 12:28:26.903050 14578 consensus_queue.cc:1243] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 26.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
W20260709 12:28:26.903172  5630 rpcz_store.cc:269] 0709 12:28:24.981579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:24.981861 (+   282us) service_pool.cc:224] Handling call
0709 12:28:26.740826 (+1758965us) spinlock_profiling.cc:243] Waited 84 ms on lock 0x7b540032cdd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:26.741089 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":193240832}
I20260709 12:28:26.743124  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:26.765360  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:26.768405  5701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1545) took 2491 ms (client timeout 1851 ms). Trace:
I20260709 12:28:26.769155  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3608) took 2549 ms. Trace:
W20260709 12:28:26.774273  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:26.775262  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:26.777359  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3628) took 1209 ms. Trace:
W20260709 12:28:26.779306  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:26.790047  5705 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: [24.18-26.19]
I20260709 12:28:26.588153  5708 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
W20260709 12:28:26.626724 14142 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.012s	user 0.014s	sys 0.002s
I20260709 12:28:26.793143  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3629) took 1222 ms. Trace:
I20260709 12:28:26.796864  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3609) took 2577 ms. Trace:
I20260709 12:28:26.796767  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:26.798455  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:26.800089  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3633) took 1198 ms. Trace:
I20260709 12:28:26.807545  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:26.807627 14405 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.658802 14613 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.945705 14142 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.811051 14244 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:26.641044 14413 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:26.626916 14439 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:26.650378 14466 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:26.982086 14409 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:26.982321 14466 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:26.661588 14538 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 2.157s	user 0.008s	sys 0.017s
W20260709 12:28:26.649327 14547 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 2.133s	user 0.005s	sys 0.008s
W20260709 12:28:26.631287 14561 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.546s	user 0.010s	sys 0.002s
I20260709 12:28:26.987890 14561 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:26.631697 14593 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:26.641463 14599 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:26.659420 14617 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:26.646456 14619 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:26.657076 14623 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:28:26.616837 14624 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:26.617163 14631 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:26.604796 14633 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.037s
I20260709 12:28:26.814600 14525 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:26.814489  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:26.815392 14618 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:26.722536  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3181) took 1690 ms. Trace:
I20260709 12:28:26.860886  5557 rpcz_store.cc:276] 0709 12:28:24.867628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:24.869287 (+  1659us) service_pool.cc:224] Handling call
0709 12:28:24.869613 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:25.342763 (+473150us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:26.729305 (+1386542us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x7b54002288d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:26.729769 (+   464us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":980057472}
I20260709 12:28:26.880616 13861 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:26.886101 14293 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:26.902391 14434 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:26.609212  5710 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 13. (term mismatch)
I20260709 12:28:26.893236  6272 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:28:26.905604  5702 rpcz_store.cc:276] 0709 12:28:24.219270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:24.622895 (+403625us) service_pool.cc:224] Handling call
0709 12:28:24.623160 (+   265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:25.046997 (+423837us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:26.768452 (+1721455us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x7b54000e01d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:26.768875 (+   423us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":972944256}
W20260709 12:28:26.905287  5701 rpcz_store.cc:269] 0709 12:28:24.126051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:24.126270 (+   219us) service_pool.cc:224] Handling call
0709 12:28:26.617005 (+2490735us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b5400075654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:26.617203 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":602945792}
W20260709 12:28:26.942315  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:26.931085  5709 rpcz_store.cc:276] 0709 12:28:24.219563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:24.631032 (+411469us) service_pool.cc:224] Handling call
0709 12:28:24.631680 (+   648us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:26.582593 (+1950913us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:26.796279 (+213686us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b54001982d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:26.796680 (+   401us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4478678784}
I20260709 12:28:26.930701  5706 rpcz_store.cc:276] 0709 12:28:25.570405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:25.642805 (+ 72400us) service_pool.cc:224] Handling call
0709 12:28:26.617454 (+974649us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:26.617705 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:26.617737 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:26.617791 (+    54us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:26.617828 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:26.617893 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:26.792699 (+174806us) spinlock_profiling.cc:243] Waited 974 ms on lock 0x7b5400075650. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:26.792957 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2240518016}
I20260709 12:28:26.960500  5632 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:26.953235  5635 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:26.903044  5636 rpcz_store.cc:276] 0709 12:28:25.514719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:25.708102 (+193383us) service_pool.cc:224] Handling call
0709 12:28:25.708460 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:25.708873 (+   413us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:25.708917 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:26.272129 (+563212us) spinlock_profiling.cc:243] Waited 35.4 ms on lock 0x7b2c00000de8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:28:26.272623 (+   494us) log.cc:844] Serialized 26 byte log entry
0709 12:28:26.402532 (+129909us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:26.402572 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:26.402642 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:26.742588 (+339946us) raft_consensus.cc:1685] finished
0709 12:28:26.742644 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:26.743020 (+   376us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":125,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":8634,"spinlock_wait_cycles":81353600,"thread_start_us":57956,"threads_started":1,"wal-append.queue_time_us":269255}
I20260709 12:28:26.932034  5707 rpcz_store.cc:276] 0709 12:28:25.601706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:25.742591 (+140885us) service_pool.cc:224] Handling call
0709 12:28:25.766334 (+ 23743us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:25.766544 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:28:25.977491 (+210947us) log.cc:844] Serialized 14 byte log entry
0709 12:28:25.978051 (+   560us) log.cc:844] Serialized 14 byte log entry
0709 12:28:26.798600 (+820549us) spinlock_profiling.cc:243] Waited 29.6 ms on lock 0x7b180048dea0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ffa228d70 00007f4ffa21c293 00007f4ffa211cea 00007f4ffa21223a 00007f4ffa67ca2a 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:28:26.799266 (+   666us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:26.799371 (+   105us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:26.799478 (+   107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:26.799661 (+   183us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:26.799916 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21251,"prepare.queue_time_us":289323,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":1133,"raft.queue_time_us":14739,"raft.run_cpu_time_us":486,"raft.run_wall_time_us":8417,"spinlock_wait_cycles":67971456,"thread_start_us":195224,"threads_started":1}
I20260709 12:28:26.948103 14405 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:26.663396 14577 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.167s	user 0.006s	sys 0.007s
I20260709 12:28:26.906358  5704 rpcz_store.cc:276] 0709 12:28:25.568192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:25.568477 (+   285us) service_pool.cc:224] Handling call
0709 12:28:25.568782 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:25.568977 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:28:25.850904 (+281927us) log.cc:844] Serialized 14 byte log entry
0709 12:28:26.775762 (+924858us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b180043b5e0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287
0709 12:28:26.776108 (+   346us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:26.776253 (+   145us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:26.776593 (+   340us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:26.776745 (+   152us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:26.777082 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21257,"prepare.queue_time_us":386449,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":37,"raft.queue_time_us":42531,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":8578,"spinlock_wait_cycles":46523648,"thread_start_us":509103,"threads_started":3,"wal-append.queue_time_us":344557}
W20260709 12:28:27.034951  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:26.984190 14439 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:27.002460 13960 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:27.008489  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 460463us
I20260709 12:28:27.059901 14617 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:27.060406  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 262464us
I20260709 12:28:27.041711 14208 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:27.066391  5705 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:27.067759 14293 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.071030 14525 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:27.071679 14581 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:27.071914 14293 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:27.032507  5553 rpcz_store.cc:276] 0709 12:28:25.031826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:25.746364 (+714538us) service_pool.cc:224] Handling call
0709 12:28:25.747969 (+  1605us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:25.800570 (+ 52601us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:25.800625 (+    55us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:25.816187 (+ 15562us) log.cc:844] Serialized 52 byte log entry
0709 12:28:26.033643 (+217456us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:26.033695 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:26.033781 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:26.598701 (+564920us) spinlock_profiling.cc:243] Waited 4.17 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:26.721269 (+122568us) raft_consensus.cc:1685] finished
0709 12:28:26.721349 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:26.722239 (+   890us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51895,"raft.queue_time_us":52175,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":4447,"spinlock_wait_cycles":9598208,"thread_start_us":197259,"threads_started":1,"wal-append.queue_time_us":251518}
W20260709 12:28:27.074347  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 212156us
I20260709 12:28:26.984092 14466 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.079128  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:27.059340 14142 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:27.080386 14617 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:27.081792 14617 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:27.084290 14405 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:27.085793 14038 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:27.085994  5707 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:28:27.078168 14640 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.108s
I20260709 12:28:27.088869 14640 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:27.089823 14640 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:27.090268 14640 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:27.095871  6272 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: [29.20-29.20]
I20260709 12:28:27.096791  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:27.104766 14466 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:27.105329  5710 pending_rounds.cc:85] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 12
I20260709 12:28:27.105687  5710 pending_rounds.cc:107] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 25.13
I20260709 12:28:27.106009  5710 raft_consensus.cc:2889] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:27.107578 14525 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:27.110150 13812 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:27.110212  5559 pending_rounds.cc:85] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 18
I20260709 12:28:27.113891  5559 pending_rounds.cc:107] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 26.19
I20260709 12:28:27.114265  5559 raft_consensus.cc:2889] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:27.115489 14647 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:27.077087 14613 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:27.111761 14593 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:27.112161  5709 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:28:27.130926  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:27.128506 14439 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:27.137760 14439 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:27.138981 14439 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:27.141119 14434 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:27.142551 14405 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:27.143532 14439 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:27.142787 14434 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:27.116261 14647 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:27.091593 14640 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.143699 14405 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:27.145115  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 513084us
I20260709 12:28:27.153766  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:27.152884  5705 pending_rounds.cc:85] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 11
I20260709 12:28:27.156641  5705 pending_rounds.cc:107] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 20.12
I20260709 12:28:27.129729 14525 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:27.131886 14293 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:27.156996  5705 raft_consensus.cc:2889] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:27.162606 14293 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:27.171245  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 345268us
I20260709 12:28:27.182116 14647 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:27.186611 14525 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:27.189495 14525 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:27.189771 14447 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:27.190418 14447 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:27.191877 14507 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:27.192847 14507 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:27.194269  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.194375 14624 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:27.195458 14507 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:27.197067 14640 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:27.197082  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:27.198855  6272 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:27.200779 14631 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:27.205164 14644 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:28:27.210386  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1812) took 1594 ms. Trace:
I20260709 12:28:27.210959  5632 rpcz_store.cc:276] 0709 12:28:25.616252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:25.742829 (+126577us) service_pool.cc:224] Handling call
0709 12:28:25.745475 (+  2646us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:27.049575 (+1304100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:27.210031 (+160456us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54000c1a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:27.210286 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2767103360}
I20260709 12:28:27.218601 14631 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:28:27.219702 14447 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:27.224888  5553 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:27.238637  6272 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:27.225437  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:27.177693 14322 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:27.253935  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:27.196174 14507 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:27.260774 14507 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:27.263447  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:27.256810  6274 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:27.265451  5554 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: []
W20260709 12:28:27.258312  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54352.5us
W20260709 12:28:27.265074  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 86893.7us
I20260709 12:28:27.276950  6272 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:28:27.282517 14593 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:27.284695  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 124470us
I20260709 12:28:27.284861  5629 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:27.285657  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:27.264165 14507 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:27.269166 14647 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:27.328907  5556 raft_consensus.cc:2410] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 24 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:28:27.329811 14409 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:28:27.331259  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:27.331728 14640 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:28:27.332998 14512 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.009s
I20260709 12:28:27.333505 14293 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:27.333766  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:27.324237 13861 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:27.337869 14593 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:27.340553  5633 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: []
I20260709 12:28:27.312434 14244 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.342882  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:27.266952 14525 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.077s
I20260709 12:28:27.296932  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:27.348304 14599 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:28:27.349884  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:27.361334 14405 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:27.361275  6274 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:27.362665  5450 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 12 to 19. New cstate: current_term: 19 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:27.364500  5703 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:28:27.445801  5554 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
W20260709 12:28:27.439671  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.452924 14647 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:27.459491  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.460893  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:27.461499  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:27.528337  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.461756  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.529310  5708 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:27.530539  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.531344  5708 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:27.462091  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.463776  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.534005  5707 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:27.476981 14156 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:27.535274 14156 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:27.525681 14208 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:27.536837 14405 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:27.318675 14322 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:28:27.482416  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:27.432040  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.497380  5553 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
W20260709 12:28:27.526567  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 209616us
I20260709 12:28:27.538837  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.537595 14405 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:27.537204 14208 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:27.543160  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 211988us
I20260709 12:28:27.540566  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:28:27.545730  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.546433  5632 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:27.545334 14405 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:27.548857  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:27.551748  5561 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:27.564811  5708 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 27: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:27.511809 13812 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.040s
I20260709 12:28:27.587718 13812 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:28:27.588156 13812 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:27.588786  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.589228 13812 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.590325  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.590607 13812 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:27.591027  5705 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:27.591965  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.565960  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:27.571097  5555 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: [29.20-29.20]
I20260709 12:28:27.572716 14117 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:27.573338 14554 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:27.577853 14647 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:27.578075  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 235302us
I20260709 12:28:27.578256  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:27.578923 14341 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:28:27.579700 14421 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:28:27.578923  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:27.582818  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.583225 14617 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:27.584511  5735 heartbeater.cc:646] Failed to heartbeat to 127.4.65.62:38921 (6 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.4.65.62:38921 timed out after 15.000s (SENT)
I20260709 12:28:27.566183 13861 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.586508  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.551290 14293 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:27.566901  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:27.556555 14322 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:27.563647 14038 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:27.615262  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:27.617965 14593 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:28:27.602175 14554 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:27.664562 14554 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:27.601711 14117 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:27.623976  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.622711  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.668573 14172 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:27.669044 14617 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:27.669519  5708 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 30 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 19.
I20260709 12:28:27.621632  5557 raft_consensus.cc:2410] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 30 because replica has last-logged OpId of term: 29 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 11.
I20260709 12:28:27.627708  5710 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:27.671236 14617 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:27.647989 14405 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.670550  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.627858 13861 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:27.638275 14421 consensus_queue.cc:1243] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 23.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:28:27.638160 14647 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:28:27.639523  5707 raft_consensus.cc:2368] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25.
I20260709 12:28:27.639222 14409 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:28:27.620673 14293 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:27.639626  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61463.9us
I20260709 12:28:27.641106 14205 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:28:27.641474 13960 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:27.641311 14322 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:27.641947  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51954us
I20260709 12:28:27.642477  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:27.647271  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 80445.5us
I20260709 12:28:27.647959 14578 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:28:27.648533  5705 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:27.649447 14581 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:27.564273 14613 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.055s
I20260709 12:28:27.657437 14208 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:27.661150  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:27.662797 14244 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:27.673839  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.608229 14156 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:27.676851  5708 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:27.672003  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3304) took 1357 ms. Trace:
I20260709 12:28:27.684065  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.668700 14117 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:27.701938  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.692411  5703 rpcz_store.cc:276] 0709 12:28:26.314230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:26.769677 (+455447us) service_pool.cc:224] Handling call
0709 12:28:26.770598 (+   921us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:27.522825 (+752227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:27.671693 (+148868us) spinlock_profiling.cc:243] Waited 593 ms on lock 0x7b5400075154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:27.671905 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1364685056}
I20260709 12:28:27.707195  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:27.686431 14439 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.542s	user 0.011s	sys 0.003s
I20260709 12:28:27.711081  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3646) took 1157 ms. Trace:
I20260709 12:28:27.675196 14647 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:27.677453 14293 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:27.717751  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 177249us
I20260709 12:28:27.669420 14172 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.687903 14581 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:27.715884  5709 rpcz_store.cc:276] 0709 12:28:26.553588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:27.128821 (+575233us) service_pool.cc:224] Handling call
0709 12:28:27.159511 (+ 30690us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:27.192698 (+ 33187us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:28:27.313733 (+121035us) log.cc:844] Serialized 14 byte log entry
0709 12:28:27.704139 (+390406us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:27.704317 (+   178us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:27.704380 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:27.704447 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:27.704560 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":112147,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":22860,"thread_start_us":452965,"threads_started":2}
I20260709 12:28:27.684693 13960 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:27.725286 13861 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:27.728838  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:27.641149  5445 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 26edf338d41e42f1a5ce0d4416ec67e1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260709 12:28:27.735937 14581 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:27.737875  5459 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35600 (request call id 280) took 31480 ms (client timeout 15000 ms). Trace:
I20260709 12:28:27.738933  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:27.739395  5445 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 98541.6us
I20260709 12:28:27.745586  5704 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:28:27.747852 14589 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:28:27.748302 14589 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:27.749122 14589 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.750394 14589 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:27.742406 14038 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:27.722349 14578 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:27.755897 14578 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.758280  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.758693  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.759351  5629 raft_consensus.cc:2410] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 25 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260709 12:28:27.726660 14617 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:27.760572  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.766214  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.766110  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.729331 13960 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:27.769704  5636 raft_consensus.cc:2410] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 12:28:27.771580  5627 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:27.751972 14117 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.777918 14117 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:27.781396 14647 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:27.783679 14647 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:28:27.784008  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.784490 14208 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:27.786211 14172 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:27.786525 14617 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:27.805734  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:27.738308  5459 rpcz_store.cc:269] 0709 12:27:56.256939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:06.589558 (+10332619us) service_pool.cc:224] Handling call
0709 12:28:27.737598 (+21148040us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:28:19.254612 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:28:19.255338 (+   726us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:28:19.255388 (+    50us) write_op.cc:435] Acquired schema lock
0709 12:28:19.255406 (+    18us) tablet.cc:662] Decoding operations
0709 12:28:19.257378 (+  1972us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:28:19.257490 (+   112us) write_op.cc:641] Partition lock acquired for write op
0709 12:28:19.257520 (+    30us) tablet.cc:685] Acquiring locks for 25 operations
0709 12:28:19.395011 (+137491us) tablet.cc:701] Row locks acquired
0709 12:28:19.395068 (+    57us) write_op.cc:260] PREPARE: finished
0709 12:28:19.395291 (+   223us) write_op.cc:270] Start()
0709 12:28:19.395684 (+   393us) write_op.cc:276] Timestamp: P: 1783600099395201 usec, L: 0
0709 12:28:19.395743 (+    59us) op_driver.cc:348] REPLICATION: starting
0709 12:28:19.397072 (+  1329us) log.cc:844] Serialized 9569 byte log entry
0709 12:28:20.020463 (+623391us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ff75680ac 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b
0709 12:28:20.426391 (+405928us) spinlock_profiling.cc:243] Waited 1.71 ms on lock 0x7b180001a0e0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:28:20.824294 (+397903us) op_driver.cc:464] REPLICATION: finished
0709 12:28:21.109652 (+285358us) write_op.cc:301] APPLY: starting
0709 12:28:21.127547 (+ 17895us) tablet.cc:1367] starting BulkCheckPresence
0709 12:28:22.385340 (+1257793us) tablet.cc:1370] finished BulkCheckPresence
0709 12:28:22.385395 (+    55us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:28:23.414145 (+1028750us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:28:23.414935 (+   790us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=50,key_file_lookups=50,delta_file_lookups=50,mrs_lookups=0
0709 12:28:23.415016 (+    81us) write_op.cc:312] APPLY: finished
0709 12:28:23.415673 (+   657us) log.cc:844] Serialized 220 byte log entry
0709 12:28:23.609399 (+193726us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:28:23.621379 (+ 11980us) write_op.cc:454] Released schema lock
0709 12:28:23.621497 (+   118us) write_op.cc:341] FINISH: Updating metrics
0709 12:28:23.640713 (+ 19216us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:28:23.640748 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":116,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":235725,"apply.run_cpu_time_us":66329,"apply.run_wall_time_us":2578296,"cfile_cache_hit":102,"cfile_cache_hit_bytes":108627,"num_ops":25,"prepare.queue_time_us":106237,"prepare.run_cpu_time_us":37681,"prepare.run_wall_time_us":1003010,"raft.queue_time_us":273123,"raft.run_cpu_time_us":13891,"raft.run_wall_time_us":249565,"replication_time_us":1389473,"spinlock_wait_cycles":12111360,"thread_start_us":1347591,"threads_started":5,"wal-append.queue_time_us":1018772,"wal-append.run_cpu_time_us":18969,"wal-append.run_wall_time_us":3025923}]]}
I20260709 12:28:27.789650  5632 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:28:27.790036  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:27.790534  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 165522us
I20260709 12:28:27.724802 14599 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.069s
I20260709 12:28:27.794032  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.793944 14293 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.797648 14142 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:28:27.801329  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.807739  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:27.804435  5702 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.036s	sys 0.059s
W20260709 12:28:27.808524  5702 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.036s	sys 0.059s
I20260709 12:28:27.810252 14507 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:27.817577 14507 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:28:27.818140  5459 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 277) took 16073 ms (client timeout 15000 ms). Trace:
W20260709 12:28:27.818611  5459 rpcz_store.cc:269] 0709 12:28:11.744883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:27.817868 (+16072985us) service_pool.cc:211] Skipping call since client already timed out
0709 12:28:27.818044 (+   176us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:28:27.818728 14507 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:27.820564  5707 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:28:27.821640  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.822547  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:27.822724  5707 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:27.823411  5704 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:28:27.824905  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 77013us
I20260709 12:28:27.825168  5709 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 18. (term mismatch)
I20260709 12:28:27.825666  5709 pending_rounds.cc:85] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 17
I20260709 12:28:27.825927  5709 pending_rounds.cc:107] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 24.18
I20260709 12:28:27.826202  5709 raft_consensus.cc:2889] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:27.829048 14142 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:27.829584 14142 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:27.829988 14421 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:28:27.830566 14142 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.831668  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:27.832590  5709 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:27.833145  5630 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:27.824116  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:27.837212  5707 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 30 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 19.
I20260709 12:28:27.810947  5710 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:27.836162  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.842788  6274 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:27.855859  5704 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
W20260709 12:28:27.858579  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.832034 14142 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:27.859947  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:27.881341  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3641) took 1344 ms. Trace:
I20260709 12:28:27.878433  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.882248  5553 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:27.883093  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.888144  5630 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 29 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:28:27.860937  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:27.896426  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.896616  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.904305  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.904778  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3639) took 1384 ms. Trace:
I20260709 12:28:27.881896  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:27.896051  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:27.908852  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.881965  5702 rpcz_store.cc:276] 0709 12:28:26.536975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:27.072410 (+535435us) service_pool.cc:224] Handling call
0709 12:28:27.073015 (+   605us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:27.128998 (+ 55983us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:27.129028 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:27.159543 (+ 30515us) log.cc:844] Serialized 52 byte log entry
0709 12:28:27.299835 (+140292us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:27.809252 (+509417us) log.cc:844] Serialized 14 byte log entry
0709 12:28:27.879784 (+ 70532us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:27.879884 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:27.880130 (+   246us) raft_consensus.cc:1685] finished
0709 12:28:27.880148 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:27.881265 (+  1117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":99712,"prepare.queue_time_us":32624,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":36279,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":5939,"thread_start_us":633923,"threads_started":2,"wal-append.queue_time_us":124574}
I20260709 12:28:27.910560  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:27.845381 14208 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:27.904327  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.913952  5707 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:27.915071  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.915771  5707 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:28:27.845270 14434 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:27.913363 14208 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.878749 14507 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:27.845790 14439 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:27.860520 14640 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:28:27.921247  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.905310  5701 rpcz_store.cc:276] 0709 12:28:26.519911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:27.069847 (+549936us) service_pool.cc:224] Handling call
0709 12:28:27.070435 (+   588us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:27.070829 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:28:27.294988 (+224159us) log.cc:844] Serialized 14 byte log entry
0709 12:28:27.904414 (+609426us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:27.904469 (+    55us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:27.904520 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:27.904585 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:27.904723 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41103,"prepare.queue_time_us":233697,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"thread_start_us":550905,"threads_started":2}
I20260709 12:28:27.924718  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.885350 14631 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:27.929260  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:27.930270  5702 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:28:27.918370  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:27.953500  5707 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:28:27.953969 14617 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.956455  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:27.945624  5710 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
W20260709 12:28:27.960496  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.965435  5627 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:27.924563 14434 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:27.970351 14293 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:27.971890 14434 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:27.960492 14631 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:27.974913  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.982573 14631 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:27.983623 14631 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:27.930763 14405 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:28.003952  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:27.935081 14640 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:28.009054  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:28.019174  5631 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:28.036670  5705 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:27.992224 14631 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:28.048606  5706 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.744s	user 0.015s	sys 0.032s
W20260709 12:28:28.056407  5706 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.016s	sys 0.033s
W20260709 12:28:28.064747  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:27.957288 14439 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:28.172168  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 344820us
I20260709 12:28:27.846083 14156 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.275s
I20260709 12:28:28.242985  5709 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:28.244961 14593 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:28.251405 14593 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:28.254730  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3450) took 1036 ms. Trace:
W20260709 12:28:28.255352  6234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.019s	sys 0.030s
W20260709 12:28:28.255785  6234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.797s	user 0.019s	sys 0.030s
I20260709 12:28:28.255245  5558 rpcz_store.cc:276] 0709 12:28:27.218352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:27.238694 (+ 20342us) service_pool.cc:224] Handling call
0709 12:28:27.269564 (+ 30870us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:27.296769 (+ 27205us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:27.296802 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:27.374540 (+ 77738us) log.cc:844] Serialized 26 byte log entry
0709 12:28:27.762543 (+388003us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:27.762594 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:27.762661 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:28.254153 (+491492us) raft_consensus.cc:1685] finished
0709 12:28:28.254194 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:28.254533 (+   339us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8382,"raft.queue_time_us":13754,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":9793,"thread_start_us":307471,"threads_started":1,"wal-append.queue_time_us":796381}
W20260709 12:28:28.263859  5445 catalog_manager.cc:5220] ChangeConfig:REMOVE_PEER RPC for tablet 26edf338d41e42f1a5ce0d4416ec67e1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 11
I20260709 12:28:28.283859  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:28.285794  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47174.7us
I20260709 12:28:28.286908  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:28.293335  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:28.296221  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:28.297230  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:28.306509  6276 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
W20260709 12:28:28.307780  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 442426us
I20260709 12:28:28.308755  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:28.288386 13812 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:28.316454  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.010s	sys 0.022s
W20260709 12:28:28.322809  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.010s	sys 0.022s
W20260709 12:28:28.323195  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 596639us
W20260709 12:28:28.325487  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 490093us
I20260709 12:28:28.346487  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:28.354065  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:28.352839 13861 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:28.363306  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:28.363016  5447 thread.cc:641] dir /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/master-0-root/data [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.023s	sys 0.051s
W20260709 12:28:28.385665  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61111.7us
W20260709 12:28:28.405606  5447 thread.cc:608] dir /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/master-0-root/data [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.023s	sys 0.051s
I20260709 12:28:28.406831 14262 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:28.407064  6279 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:28:28.379988 13861 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:28.411706  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:28.420117  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:28.322322 14434 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.086s
W20260709 12:28:28.421727  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:28.373937 14640 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:28.344158 13812 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:28.419776  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:28.473526  5628 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
W20260709 12:28:28.433511  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 503299us
I20260709 12:28:28.479789  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:28.488672  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 68666us
W20260709 12:28:28.494701  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 150625us
I20260709 12:28:28.499068  5635 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 24 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
W20260709 12:28:28.500337  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 84749.4us
I20260709 12:28:28.505950 13861 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:28.506752 13861 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:28.512889 13861 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:28.522111  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:28.523211  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:28.524169  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:28.516245  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:28.507637 14439 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:28.534449  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:28.508091 14631 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:28.546281  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47851.5us
I20260709 12:28:28.512845 14682 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:28.475569 14262 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:28.578269  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62079.8us
I20260709 12:28:28.580901  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:28.598664  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:28.520814 14447 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 12:28:28.551254 14578 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:28.577071 14439 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:28:28.628208  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47308.5us
W20260709 12:28:28.561023 14617 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.008s	sys 0.018s
I20260709 12:28:28.592032 14631 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:28.586513 14525 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:28.586755 14038 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:28:28.641108  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 143752us
I20260709 12:28:28.650813 13861 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:28.644573 14466 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.009s
W20260709 12:28:28.638844 14617 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.010s	sys 0.019s
I20260709 12:28:28.592283 14677 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:28.636543 14405 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:28:28.669546  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:28.639339 14439 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:28.587461 14421 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.075s
W20260709 12:28:28.697403  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 179919us
W20260709 12:28:28.661520 14554 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.993s	user 0.013s	sys 0.002s
I20260709 12:28:28.690037  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3647) took 2136 ms. Trace:
I20260709 12:28:28.643235 14525 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:28.591452 14663 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.216s
I20260709 12:28:28.658231 14687 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:28:28.678411 14405 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:28.683554 14439 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:28.650390 14631 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:28.702771 14466 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:28.730569  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:28.731029  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:28.649845 14578 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:28.712157 14525 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:28.707679 14617 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:28.737030  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:28.718863 14674 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:28.725630 14405 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:28.743012  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 115652us
W20260709 12:28:28.744887  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 110258us
I20260709 12:28:28.702584  5706 rpcz_store.cc:276] 0709 12:28:26.553831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:27.179248 (+625417us) service_pool.cc:224] Handling call
0709 12:28:27.195634 (+ 16386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:27.196143 (+   509us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:28:27.304481 (+108338us) log.cc:844] Serialized 14 byte log entry
0709 12:28:28.442673 (+1138192us) log.cc:844] Serialized 14 byte log entry
0709 12:28:28.689386 (+246713us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:28.689499 (+   113us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:28.689575 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:28.689719 (+   144us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:28.689942 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":153851,"prepare.queue_time_us":469697,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":106,"raft.queue_time_us":428106,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":4747,"thread_start_us":1277902,"threads_started":4,"wal-append.queue_time_us":992741}
I20260709 12:28:28.736745  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:28.750013  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:28.750427  6277 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:28.751003  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:28.709620 14554 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:28.744517 14660 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.347s	user 0.001s	sys 0.001s
W20260709 12:28:28.767218 14678 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.362s	user 0.001s	sys 0.000s
I20260709 12:28:28.746378 14617 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:28.768163 14666 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.304s	user 0.001s	sys 0.000s
I20260709 12:28:28.769793  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:28.770135 14554 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:28.770327 14692 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:28:28.759769 14142 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.872s	user 0.012s	sys 0.001s
W20260709 12:28:28.770817  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55970.7us
I20260709 12:28:28.772683 14117 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:28:28.814188  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44524.2us
I20260709 12:28:28.774061 14710 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:28:28.774721 14698 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:28:28.775718 14670 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.257s	user 0.000s	sys 0.001s
I20260709 12:28:28.739667 14466 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:28.818205  5636 raft_consensus.cc:2410] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 25 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:28:28.779676  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3649) took 1156 ms. Trace:
W20260709 12:28:28.784260 14686 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 12:28:28.739667 14680 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:28.801204  5633 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:28.765316 14677 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:28.820209  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:28.823613  6273 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:28.815625  6277 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:28.826893  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:28.830226  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:28.817895 14676 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.273s	user 0.000s	sys 0.001s
I20260709 12:28:28.819947  5703 rpcz_store.cc:276] 0709 12:28:27.623619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:27.719529 (+ 95910us) service_pool.cc:224] Handling call
0709 12:28:27.732760 (+ 13231us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:27.772706 (+ 39946us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:27.772754 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:27.785066 (+ 12312us) log.cc:844] Serialized 26 byte log entry
0709 12:28:28.279609 (+494543us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:28.279678 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:28.279798 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:28.779376 (+499578us) raft_consensus.cc:1685] finished
0709 12:28:28.779408 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:28.779596 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21971,"raft.queue_time_us":27851,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":12922,"thread_start_us":493988,"threads_started":1,"wal-append.queue_time_us":992023}
W20260709 12:28:28.836761 14695 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:28:28.788571 14262 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:28.824916 14677 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:28.842386 14677 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:28.845208 14672 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.276s	user 0.000s	sys 0.001s
I20260709 12:28:28.833600 14680 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:28.846942  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 156131us
I20260709 12:28:28.847817 14680 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:28.848737  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:28.848922  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47536us
I20260709 12:28:28.850234  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:28.850342  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:28.851095 14322 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:28.851819 14322 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:28.854238  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:28.855577 14593 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:28.757318 14674 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:28.861580 14581 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:28.862725 14466 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:28.863217 13812 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:28.866084 14647 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.082s	user 0.007s	sys 0.008s
I20260709 12:28:28.867338 14578 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:28.870918 14589 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.120s	user 0.016s	sys 0.000s
I20260709 12:28:28.819062 14117 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:28.819312 14421 raft_consensus.cc:2764] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:28:28.872320 14631 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:28.889158 14693 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:28.890372  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:28.890769 14617 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:28.882107 14613 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:28:28.883975 13960 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.093s	user 0.012s	sys 0.002s
I20260709 12:28:28.896602  5526 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:28.896255 14208 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:28.829562 14710 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:29.008797 14117 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:28.893779  5447 maintenance_manager.cc:643] P b3df7d767dc641b7bb01c3435d842dd5: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 4.314s	user 0.146s	sys 0.096s Metrics: {"cfile_cache_miss":9,"cfile_cache_miss_bytes":26121,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":1064002,"dirs.run_cpu_time_us":887,"dirs.run_wall_time_us":78353,"lbm_read_time_us":94556,"lbm_reads_1-10_ms":4,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":20,"lbm_write_time_us":104331,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":2,"lbm_writes_lt_1ms":8,"peak_mem_usage":276662,"reinsert_count":0,"spinlock_wait_cycles":110922368,"thread_start_us":1057933,"threads_started":3,"update_count":223,"wal-append.queue_time_us":236889,"wal-append.run_cpu_time_us":750,"wal-append.run_wall_time_us":1020544}
W20260709 12:28:28.906864 14713 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:28:28.914420 14640 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:28.936137  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 124447us
I20260709 12:28:28.936817 14262 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:28.930087 14525 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:28.904075 14038 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:29.009599 14706 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:28.945356  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3461) took 1110 ms. Trace:
W20260709 12:28:28.930673 14684 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.343s	user 0.003s	sys 0.000s
I20260709 12:28:28.918303  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:28.974109  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 222162us
W20260709 12:28:28.977643  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:28.981122  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:29.004381  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:29.011092 14613 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:29.009241  6280 raft_consensus.cc:2410] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 32 because replica has last-logged OpId of term: 31 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 21.
I20260709 12:28:29.017359 14647 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:28:29.014745  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 118260us
I20260709 12:28:28.921298 14405 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:29.017146 14507 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.083s	user 0.008s	sys 0.006s
I20260709 12:28:29.005508 14682 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:29.041178  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:28.998888 14341 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:29.050841 14712 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:29.040630  5635 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:29.051581 14712 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.036072 14262 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:29.052274  5448 maintenance_manager.cc:419] P b3df7d767dc641b7bb01c3435d842dd5: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260709 12:28:29.052605 14262 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:29.053331  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:29.054363  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.055217 14682 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:29.038439  5556 rpcz_store.cc:276] 0709 12:28:27.835402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:27.838133 (+  2731us) service_pool.cc:224] Handling call
0709 12:28:27.881655 (+ 43522us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:27.909800 (+ 28145us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:28:28.342728 (+432928us) log.cc:844] Serialized 14 byte log entry
0709 12:28:28.666651 (+323923us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:28.869238 (+202587us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b1800484fc0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a
0709 12:28:28.869748 (+   510us) log.cc:844] Serialized 52 byte log entry
0709 12:28:28.869910 (+   162us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:28.870134 (+   224us) log.cc:844] Serialized 14 byte log entry
0709 12:28:28.870318 (+   184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:28.870393 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:28.945034 (+ 74641us) raft_consensus.cc:1685] finished
0709 12:28:28.945062 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:28.945276 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":46805,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":96,"raft.queue_time_us":108490,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":7103,"spinlock_wait_cycles":23668480,"thread_start_us":657399,"threads_started":2,"wal-append.queue_time_us":364738}
I20260709 12:28:29.047972  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.035907 14687 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:29.032033 13960 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:29.042506 14421 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:29.050581  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.045002  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:29.059558  5447 maintenance_manager.cc:643] P b3df7d767dc641b7bb01c3435d842dd5: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {}
W20260709 12:28:29.062242  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:29.061734  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:29.064160  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 210259us
W20260709 12:28:29.065022  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 174762us
I20260709 12:28:29.065258 14172 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:29.066573  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:29.067059  6234 leader_election.cc:400] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25.
I20260709 12:28:29.067391  6234 leader_election.cc:403] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:28:29.053874 14712 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:29.068743 14172 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25.
I20260709 12:28:29.044787 14647 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:29.069526 14647 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:29.070411 14669 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:28:29.070369  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:29.070818  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:29.070993 14669 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:29.071412 14669 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:29.043416 14706 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:29.077174 14663 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:29.079252  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:29.061146 14439 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:29.052176 14674 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:29.084007 14613 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.085253 14682 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:29.085510 14613 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:29.058168  5559 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
W20260709 12:28:29.061386  5602 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.598s (SENT) [suppressed 25 similar messages]
I20260709 12:28:29.088568 14405 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:29.068635 14710 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.091701 14421 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.106171 14663 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:29.092608  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:29.102150  5555 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 24 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:28:29.115291 14663 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:29.098498  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:29.115818 14663 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:29.091861  5558 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:29.100412 14156 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 12:28:29.090579 14716 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:29.092146 13812 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.021s
I20260709 12:28:29.113407 14631 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:29.130632  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.113967 14208 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:29.114104 14421 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:29.113552  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.092123 14439 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:29.114581 14693 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:29.110916  6273 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:29.105662  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:29.104300  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:29.136010  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 43541.8us
W20260709 12:28:29.136996  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57875.7us
I20260709 12:28:29.097497 14682 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:29.101161 14525 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:29.137498  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 76198.5us
I20260709 12:28:29.103395 13960 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:29.104206 14593 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.011s
I20260709 12:28:29.106290 14117 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:29.143987  5556 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 33 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:28:29.144737  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:29.123132 14687 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:28:29.115386 14677 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.137320 14507 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:29.114920 14647 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.147382  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.148167  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:29.151458  5553 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 26: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:29.133260 14716 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:29.152900  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:29.154343  5561 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:29.157996  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:29.160234 14507 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:29.161597 14712 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:28:29.161413 14507 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.150962 14405 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:29.164034 14405 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:29.164799 14405 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:29.165870  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:29.168846 13960 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:29.169323 13960 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:29.169698 13960 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:29.170199 14117 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:29.143944 14693 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:29.175115  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.176337  5558 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:29.177237  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3464) took 1319 ms. Trace:
I20260709 12:28:29.177815  5555 rpcz_store.cc:276] 0709 12:28:27.857870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:27.881101 (+ 23231us) service_pool.cc:224] Handling call
0709 12:28:29.176915 (+1295814us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b540023fd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:29.177131 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2745339136}
I20260709 12:28:29.179278 14712 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:28:29.175179 14682 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:29.184487  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:29.186444  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 80868.3us
I20260709 12:28:29.146464 14208 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:29.189456  5553 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:29.153174 14439 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:29.197849  5675 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:29.199055  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.203263  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:29.204224  5556 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:29.196516 13812 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:29.209250 14525 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:29.209077 13812 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.209764 14525 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.211324 13812 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:29.212538 14525 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:29.213366 14525 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:29.213943 14525 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:29.214735 14677 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:29.145524 14631 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.216161 14525 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:29.216965 14631 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:29.217434 14525 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:29.217947 14262 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.218014 14525 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:29.220714 14262 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:29.195739 14156 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:29.222048 14156 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:29.222769 14208 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:29.223068 14156 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.220942  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:29.226999  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:29.196813 14674 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:28:29.229512  5601 leader_election.cc:336] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.864s (SENT)
I20260709 12:28:29.229650 14706 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:29.230711  5601 leader_election.cc:336] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.898s (SENT)
I20260709 12:28:29.231153  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:29.231587 14647 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:29.231927 14447 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:29.233373 14693 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:29.245836  5556 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:29.247045  6271 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:29.249334  6275 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:29.154901 14716 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:29.258312  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:29.259037  5709 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:29.248921  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.273579  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:29.280228  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.281383  5561 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:29.285921  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:29.282924  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:29.275681 14692 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:29.296734 14692 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:29.299326 14692 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:29.299898 14706 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:29.300681 14706 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:29.300787 14710 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:29.301182 14706 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:29.301595 14710 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:29.304786  5601 leader_election.cc:336] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.971s (SENT)
I20260709 12:28:29.305600  6277 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
W20260709 12:28:29.300941  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 104124us
I20260709 12:28:29.308841  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.265496 14687 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:28:29.314661  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 146356us
I20260709 12:28:29.264541 14434 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:29.318884  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 171094us
I20260709 12:28:29.273329 14439 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:29.320416  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:29.331979  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:29.332688  5636 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:29.333910  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:29.334654  5709 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:29.335072  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:29.338049  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.339854  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.341886  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:29.310103 14677 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:29.344182  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:29.349278  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.352913  6277 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:29.349278  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:29.355288  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1564) took 1091 ms. Trace:
I20260709 12:28:29.316740 14710 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:29.355706  6275 rpcz_store.cc:276] 0709 12:28:28.256973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:28.277766 (+ 20793us) service_pool.cc:224] Handling call
0709 12:28:29.347671 (+1069905us) spinlock_profiling.cc:243] Waited 906 ms on lock 0x7b54002922d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:29.347763 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2083481472}
I20260709 12:28:29.364697  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:29.361122  5631 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:29.341820 14208 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:29.372573  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:29.378659  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.379138  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:29.379714  5704 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:29.380342  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:29.385362 14617 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:29.386171 14617 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.390589 14205 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:29.391352 14205 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:29.392565  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:29.395360 14205 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:29.397055  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.397571 14617 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:29.398185  6275 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:29.400256  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:29.401485 14617 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:29.402103 14617 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:29.402698 14617 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:29.404814  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:29.407573  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:29.410377  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:29.412177  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:29.416088  5634 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:28:29.417685  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:29.419941  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:29.423139  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:29.347831 14677 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:29.345911 14692 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:29.359395 14674 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:29.457379  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 122390us
I20260709 12:28:29.434052  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:29.464161  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:29.363281 14693 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:29.427311 14581 raft_consensus.cc:2764] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election decision vote started in defunct term 25: won
I20260709 12:28:29.363179 14647 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:29.343134 14439 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:29.499398  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:29.497568  5628 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 15. (term mismatch)
I20260709 12:28:29.501060  5628 pending_rounds.cc:85] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 14
I20260709 12:28:29.501384  5628 pending_rounds.cc:107] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20260709 12:28:29.501670  5628 raft_consensus.cc:2889] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:29.504673  5630 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 24: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:29.504719  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.508615  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:29.427117 14687 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:29.511108  5561 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:29.464277 14677 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:29.368335 14262 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.135s
I20260709 12:28:29.521692  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:29.515409  5553 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:29.371696 14640 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.138s
W20260709 12:28:29.532843  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 168248us
I20260709 12:28:29.523362  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:29.472259 14205 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:29.371366 14631 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.135s
I20260709 12:28:29.522006  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:29.540055  5709 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:29.498245 14647 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:29.523856 14682 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:29.520735 14687 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:29.548828  5704 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
W20260709 12:28:29.571103  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 143413us
I20260709 12:28:29.568984  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:29.572681  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:29.503151 14674 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.584389  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:29.573498  6272 raft_consensus.cc:2376] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25. Re-sending same reply.
I20260709 12:28:29.586582  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.589151  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:29.592947  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 128864us
I20260709 12:28:29.515643 14692 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.588326  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:29.586918  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.585489  5706 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:29.596064  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.552520 14647 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:29.611342  5709 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:29.561415 14687 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:29.571225 14682 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.615818 14682 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:29.587342 14447 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:29.553788 14205 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.627730  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.628787  5627 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:29.599107 14507 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:29.633983  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.574258 14405 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:29.646147 14631 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:29.646580 14205 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:29.647518  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.650126  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:29.674690  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:29.651865  5636 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:29.656980  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.627424 14447 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:29.667900  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.668654  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:29.676504  6274 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:29.669348  5558 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:29.670447  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:29.673039  6280 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:29.685606  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:29.686427  6280 raft_consensus.cc:2410] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 32 because replica has last-logged OpId of term: 31 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 21.
I20260709 12:28:29.675251  6273 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:29.673892  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:29.687525  5701 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:29.585139  6277 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:29.694101  6277 raft_consensus.cc:606] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:29.694538  6277 raft_consensus.cc:614] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:29.639596 14507 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:29.698660  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:29.706197  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 88727.7us
I20260709 12:28:29.708323  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.708971  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.709079  5561 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:28:29.709980  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.710623  5557 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:29.711889  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.713078  5554 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:29.714519  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.715935  5558 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:29.726500  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:29.727236  6280 raft_consensus.cc:2376] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25. Re-sending same reply.
I20260709 12:28:29.727906  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.728677  6276 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 26: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:29.728838  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.731082  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.717417  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:29.737677  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.737677  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:29.731678  5555 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
W20260709 12:28:29.746026  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 92805.2us
I20260709 12:28:29.746317  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:29.731746  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:29.747180  6273 raft_consensus.cc:2376] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27. Re-sending same reply.
I20260709 12:28:29.753324  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:29.754299  5556 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:29.754567  5559 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:29.715229 14507 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:29.703699 14341 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:29.747834  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:29.759814  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:29.762861  6274 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:29.776510  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:29.794450  6278 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:29.758513 14613 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:29.800145  6274 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:28:29.785457 14038 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:29.776789 14341 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:29.789623 14712 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:28:29.841751  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 220615us
I20260709 12:28:29.842027  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:29.848232  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:29.858971  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:29.804236 14613 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:29.805461  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:29.859906  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 97074.3us
I20260709 12:28:29.818470 14439 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:29.852756 14716 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.568s	user 0.012s	sys 0.000s
W20260709 12:28:29.852303 13960 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.682s	user 0.013s	sys 0.000s
W20260709 12:28:29.908552  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 56797.6us
I20260709 12:28:29.837103 13812 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:28:29.850901 14244 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.002s	sys 0.014s
W20260709 12:28:29.903997 14723 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.268s	user 0.000s	sys 0.001s
I20260709 12:28:29.931229 14732 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:29.762326 14730 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.093s
W20260709 12:28:29.961676 14734 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:28:29.961984 14730 consensus_queue.cc:1243] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 22.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
W20260709 12:28:29.955902 14244 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.002s	sys 0.014s
W20260709 12:28:29.963135 14736 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.169s	user 0.000s	sys 0.000s
W20260709 12:28:29.933207  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:29.964246  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:29.964932 14727 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260709 12:28:29.966557  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:29.966768 14692 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:29.967842  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:29.968739 14669 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.897s	user 0.011s	sys 0.003s
I20260709 12:28:29.851662 14341 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:29.969292  5702 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: []
I20260709 12:28:29.969173 14669 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.971534  5702 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27. Ops: []
W20260709 12:28:29.972259 14737 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.140s	user 0.000s	sys 0.000s
I20260709 12:28:29.850719 14447 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.961154 14732 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.974090  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:29.974838  5706 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:28:29.975253 14732 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:29.976824 14729 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:28:29.859398 14617 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.979058 14617 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:29.979737 14680 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:29.980321 14680 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:29.980974 14680 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:29.928694 14738 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.075s
I20260709 12:28:29.984517 14674 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:29.986225 13960 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:29.935398  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:29.989055  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53823.7us
I20260709 12:28:29.989872  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:29.990955  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:29.505223 14663 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:30.003144 14663 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:29.951596  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 146212us
I20260709 12:28:29.837450 14421 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.075s
W20260709 12:28:30.005373 14731 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.383s	user 0.003s	sys 0.000s
I20260709 12:28:30.006261  5707 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:30.006093 14735 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:30.007143 14735 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:30.007537 14735 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:30.007985 14735 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:30.008149  6275 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29. Ops: []
W20260709 12:28:30.014645 14434 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.684s	user 0.007s	sys 0.003s
I20260709 12:28:30.017227  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:29.956421 14716 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:30.095003 14647 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:29.998735 14341 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:30.097069 14716 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:30.006075 14682 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:30.058241 14525 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:30.045830 14725 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.349s	user 0.001s	sys 0.000s
I20260709 12:28:30.060020  6273 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:30.071501 14735 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:30.042168 14687 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:30.015918 14434 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:30.020478  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:30.029212 14706 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.727s	user 0.008s	sys 0.004s
I20260709 12:28:30.072196  6271 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:29.950753 14613 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:30.040467 14156 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.006s	sys 0.038s
I20260709 12:28:30.072412  6277 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 32 index: 22. (term mismatch)
I20260709 12:28:30.073280  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:30.054948 14739 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:28:30.078174 14669 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:30.080286 14733 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.141s	user 0.001s	sys 0.001s
I20260709 12:28:30.030917 14677 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:29.938477 14117 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:30.040479 13960 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:29.945327  5602 leader_election.cc:336] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.331s (SENT)
I20260709 12:28:30.092610  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:30.093468 14405 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:30.098116 14341 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:30.135040 14405 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:28:30.098189  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 81064.9us
I20260709 12:28:30.135944 14405 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:30.136730  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:30.137058 14405 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:30.137492  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:30.137735 14244 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:30.098469 14716 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:28:30.099467  5451 catalog_manager.cc:5697] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 18 to 20, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 20 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:30.103281  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:30.104085  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:30.105669  5630 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:28:30.144682 14742 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:30.114718  5709 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:30.145602  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:30.105610 14507 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:30.146510 14554 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:28:30.114871 14156 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.809s	user 0.008s	sys 0.039s
I20260709 12:28:30.147039 14554 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:30.115002  6277 pending_rounds.cc:85] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 21
I20260709 12:28:30.115026  5702 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:30.115255  6274 raft_consensus.cc:2376] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24. Re-sending same reply.
I20260709 12:28:30.148025 14554 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:30.116128 14735 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:30.151436  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:30.152261  6271 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:30.114414 14687 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:30.152612 14710 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:30.153326 14710 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:30.097905 14647 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:30.147964  6277 pending_rounds.cc:107] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 31.22
I20260709 12:28:30.154486 14710 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:30.155267 14716 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:30.112231  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 92222us
I20260709 12:28:30.156245  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:30.110632  6273 raft_consensus.cc:606] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:28:30.125099 14677 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:30.104614  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:30.145159 14742 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:30.157719  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:30.155273  6277 raft_consensus.cc:2889] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:30.157733  5629 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:30.158988  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:30.160552 14735 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:30.160965 14735 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:30.161329 14735 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:30.161623 14208 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:30.157913 14742 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:30.162647 14687 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:30.113125  5553 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:30.157168  6273 raft_consensus.cc:614] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:30.112553 14706 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:30.125669 14613 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:30.165956 14706 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:30.166731  5710 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:30.166884  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:28:30.167717  5630 raft_consensus.cc:606] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:30.116923 14682 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:28:30.169003  6275 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:30.160118  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:30.171417  6275 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "786a49bad9be400ea9ce584030aad446"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:30.171984  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:30.171998  6275 raft_consensus.cc:606] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:30.172678  6275 raft_consensus.cc:614] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:30.123767 14117 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:30.174547 14706 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:30.174711  5634 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:30.175107 14706 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:30.176028 14706 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:30.176618  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:30.177687 14669 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:30.177876 14687 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:30.178148 14669 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:30.178398 14687 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:30.178655 14669 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:30.178890 14687 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:30.179023 14117 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:30.180462  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:30.182683 14599 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:28:30.183905 14117 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:30.184279 14117 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:30.185349 14117 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:30.186305 14117 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:30.187490 14669 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:30.156287 14716 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:30.188758  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:30.190379 14669 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:30.191704  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:30.194231  5634 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:30.176833 14613 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:30.196079 14693 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:28:30.197069 14693 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:30.197419 14693 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:30.197971 14693 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:30.198763  6275 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:30.199455  6275 raft_consensus.cc:606] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:30.199815  6275 raft_consensus.cc:614] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:28:30.124653  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 179384us
I20260709 12:28:30.187536 14735 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:30.198530 14748 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:30.205708 14748 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:30.208649  5629 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:30.192095 14208 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:30.205053  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:30.236145  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:28:30.237951  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 88308us
I20260709 12:28:30.242440 14750 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:30.243110 14750 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:30.245393 14750 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:30.245319  6275 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:30.246168  6275 raft_consensus.cc:606] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:30.247826  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
W20260709 12:28:30.256803  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 51850.7us
I20260709 12:28:30.240876  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:30.281818  5634 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:30.251641 14693 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:30.283960  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:30.257829  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:30.309719  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52004.1us
I20260709 12:28:30.311744  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:30.256191 14682 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:30.316141  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:30.318953 14172 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:30.240643 14613 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:30.322707  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:30.322304  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57239.2us
I20260709 12:28:30.256395 14735 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:30.331350  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:30.335943  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:30.319077 14208 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:30.338783  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:30.314479 14409 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:30.340870  5561 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:30.317292 14750 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:30.345640  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:30.347198  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:30.346210  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:30.346495  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:30.348168  5554 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:30.352844  5707 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:30.351291  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:30.353878  5555 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:30.354638  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:30.321722 14682 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:30.361954  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:30.355257  5559 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:30.356773  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:30.326678 14732 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:30.338111 14172 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:30.385393  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:30.375365  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:30.347100 14409 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:30.347807 14613 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:30.350943  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.002s	sys 0.032s
I20260709 12:28:30.402019  6273 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:28:30.346925 14208 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:30.323508 14742 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:30.351751 14750 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:30.338411 14745 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:30.356009 14205 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:30.368566 14682 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:30.404562  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:30.379320  5708 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 30 index: 20. (term mismatch)
W20260709 12:28:30.434098  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.003s	sys 0.032s
W20260709 12:28:30.453727  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 691586us
I20260709 12:28:30.339843 14687 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:30.455111  5704 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.804s	user 0.010s	sys 0.029s
I20260709 12:28:30.455575  5708 pending_rounds.cc:85] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 19
I20260709 12:28:30.400388 14674 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:28:30.467692  5708 pending_rounds.cc:107] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 29.20
I20260709 12:28:30.473783  5708 raft_consensus.cc:2889] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:28:30.467307  5704 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.816s	user 0.010s	sys 0.029s
I20260709 12:28:30.375931 14732 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:30.432554 14208 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:30.430456 14750 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:30.323359 14748 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:30.396844 14680 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:30.470485  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.775s	user 0.002s	sys 0.017s
W20260709 12:28:30.493340  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.003s	sys 0.018s
I20260709 12:28:30.473666 14674 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:30.470095  6272 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.000s	sys 0.019s
I20260709 12:28:30.481083  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:30.433997 14409 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:30.500406  6272 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.000s	sys 0.020s
I20260709 12:28:30.434568 14593 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.055s
I20260709 12:28:30.396235 14507 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:30.472764  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68303.3us
W20260709 12:28:30.494575  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 799705us
I20260709 12:28:30.513370  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:30.475713  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.006s	sys 0.013s
W20260709 12:28:30.516280  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.007s	sys 0.013s
W20260709 12:28:30.478226 14447 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.010s	sys 0.010s
I20260709 12:28:30.498111  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:30.482151 14205 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:28:30.490636 14038 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.009s	sys 0.009s
W20260709 12:28:30.488488  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.745s	user 0.001s	sys 0.018s
W20260709 12:28:30.543839  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.800s	user 0.002s	sys 0.018s
I20260709 12:28:30.543964  5553 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:30.480753 14732 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:30.527071  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:30.528774  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:30.532548 14172 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:30.534166  5558 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:30.503746 14674 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:28:30.559811  5554 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:30.514313  5634 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:30.476611 14405 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.029s
I20260709 12:28:30.475710 14554 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.032s
I20260709 12:28:30.585695  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:30.544368  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 800898us
I20260709 12:28:30.588637  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:30.541224  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:30.540553 14447 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.011s	sys 0.011s
I20260709 12:28:30.547205  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:30.550655  5630 pending_rounds.cc:85] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 16
I20260709 12:28:30.561113  5555 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
W20260709 12:28:30.599891  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 58673.8us
I20260709 12:28:30.565551  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:30.536645  6275 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: []
W20260709 12:28:30.543565 14038 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.009s	sys 0.009s
I20260709 12:28:30.604107  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:30.604959  6275 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:30.609617  6272 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 26: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:30.610585  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:30.611217  5556 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:30.599488  5557 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
W20260709 12:28:30.611351  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 271878us
I20260709 12:28:30.567276 14593 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:30.596730  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:30.619299  5554 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:28:30.600178  5630 pending_rounds.cc:107] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 22.17
I20260709 12:28:30.621340  5630 raft_consensus.cc:2889] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:30.601679  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:30.582923  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:30.551139 14631 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.004s	sys 0.019s
I20260709 12:28:30.568507  5561 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:30.614601  5559 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:30.615056  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:30.635192  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:30.635943  6277 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
W20260709 12:28:30.596876 14712 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.013s	sys 0.040s
I20260709 12:28:30.637671  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:30.646353  6271 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:30.652809  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 51277.2us
I20260709 12:28:30.655856  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:30.623319  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:30.649264  5561 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:28:30.608978 14692 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:30.612927 14038 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:30.611375 14693 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:30.509436 14613 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:30.636382 14631 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.005s	sys 0.019s
I20260709 12:28:30.662842  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:30.689879  6276 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 28: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:28:30.685026 14768 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260709 12:28:30.682446 14764 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:28:30.684645 14769 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:28:30.613526 14507 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:30.656708 14647 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:30.675856  6271 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:30.696274  5556 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:30.619026 14593 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:30.701995  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:30.645650 14617 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:30.692549  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:30.657199 14742 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:30.657474 14680 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:30.658335 14712 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:30.726476  6272 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 12:28:30.812549  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 204127us
I20260709 12:28:30.835379  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:30.720876 14743 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:28:30.848579  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:30.867192  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 174745us
W20260709 12:28:30.869014  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 203429us
I20260709 12:28:30.869982  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:30.870272  6275 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:30.889061  5554 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:30.717804 14755 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:30.868207  5526 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:30.913740  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:30.906940  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 294569us
W20260709 12:28:30.914573  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 483198us
I20260709 12:28:30.816766 14631 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:28:30.918066  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 294838us
W20260709 12:28:30.923806  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 53949us
I20260709 12:28:30.697834 14692 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:30.945683  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:30.946547  5561 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:28:30.947820  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:30.949415  5554 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:30.833233 14205 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.111s
I20260709 12:28:30.916626 14677 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:30.922983 14687 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:31.026752  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:30.923194 14617 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:30.906117 14765 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:31.018474  6273 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: [25.15-25.15]
W20260709 12:28:31.029247  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:31.058101  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:31.070382  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:30.922880 14647 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.105s
I20260709 12:28:31.087145  5561 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:28:31.126420  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 291144us
I20260709 12:28:31.127115  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:31.038837 14172 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:31.132687  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:31.134171  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:31.129586  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:31.037302 14631 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:31.064173 14764 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:31.147742  5554 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:31.068794 14677 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:31.163009  5627 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:31.163338  5636 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:31.012944 14038 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.014091 14763 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.176s
I20260709 12:28:31.177866  5556 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:28:31.133991 14693 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:30.953810 14593 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.139946 14761 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:31.213058  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:31.216286  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:31.170668 14790 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:31.189800 14693 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:31.163756 14716 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.050s
W20260709 12:28:31.215634 14762 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.491s	user 0.000s	sys 0.001s
I20260709 12:28:31.171473 14767 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.052s
I20260709 12:28:31.130187 14712 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.223951 14693 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:31.171046 14764 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.215272 14156 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.029s
W20260709 12:28:31.275108 14758 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:28:31.261044 14677 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:31.275635 14771 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:28:31.278725 14783 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:31.279384 14783 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.213908 14617 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.280956 14692 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:31.282349 14783 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:31.275156 14760 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
W20260709 12:28:31.280274 14750 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.786s	user 0.009s	sys 0.001s
W20260709 12:28:31.283128 14117 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.096s	user 0.015s	sys 0.000s
W20260709 12:28:31.285194 14759 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.649s	user 0.000s	sys 0.001s
I20260709 12:28:31.304716  5561 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: []
W20260709 12:28:31.314155 14773 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.412s	user 0.000s	sys 0.001s
I20260709 12:28:31.316735  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:31.247915 14631 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.317608  5560 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260709 12:28:31.318054  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:31.319067  5557 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:31.305757 14750 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.327687  5629 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:31.328127  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:31.228597 14790 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.329771  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:31.349924  5557 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:31.342372  5558 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:31.343140  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:31.228088 14761 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.364850  5552 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:31.361132  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:31.348529 14779 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:31.345387  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 407225us
W20260709 12:28:31.345660 14770 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.268s	user 0.000s	sys 0.001s
I20260709 12:28:31.346235 14755 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:31.346850 14647 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:31.348976  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 221251us
I20260709 12:28:31.377782  5553 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
W20260709 12:28:31.369284  6279 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.703s	user 0.003s	sys 0.020s
I20260709 12:28:31.376643  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:31.378629  6279 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.004s	sys 0.020s
I20260709 12:28:31.380985  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:31.384168  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:31.392030  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:31.394548  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:31.395731  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1589) took 1311 ms. Trace:
I20260709 12:28:31.395021  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:31.361055 14750 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:31.400583  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:31.401012  6271 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:28:31.409639  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:31.403946  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:31.412593  5632 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 24: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:31.415009  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:31.396576  6279 rpcz_store.cc:276] 0709 12:28:30.070027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:30.073883 (+  3856us) service_pool.cc:224] Handling call
0709 12:28:30.074295 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:30.074482 (+   187us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:28:30.363437 (+288955us) log.cc:844] Serialized 14 byte log entry
0709 12:28:31.380699 (+1017262us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:31.380772 (+    73us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:28:31.380836 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:31.380925 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:31.381082 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35949,"prepare.queue_time_us":474236,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":548828,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":12737,"thread_start_us":1275905,"threads_started":3,"wal-append.queue_time_us":957707}
I20260709 12:28:31.417480 14581 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:31.418576 14581 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.422187 14581 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:31.424608 14807 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:31.425899  5628 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:28:31.425432 14807 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.426350 14750 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:31.426904 14750 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.427500  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:31.429131 14750 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:31.432155 14750 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:31.432686  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:31.433461  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:31.374346 14797 raft_consensus.cc:2764] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:28:31.435401  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:31.437019  5632 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:28:31.414153 14738 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:31.412765 14663 raft_consensus.cc:2764] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:28:31.445179  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:31.439636  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:31.445822  6280 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.843s	user 0.011s	sys 0.004s
I20260709 12:28:31.432734 14750 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.441831  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:31.449111  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 101534us
W20260709 12:28:31.450217  6280 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.847s	user 0.012s	sys 0.004s
I20260709 12:28:31.439285 14738 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.451237 14799 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:31.452585 14764 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:31.453974 14693 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:31.456748  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:31.461767  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:31.459312 14788 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:31.460178 14732 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.884s	user 0.010s	sys 0.002s
I20260709 12:28:31.460834  5634 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:31.509174  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:31.461205  5676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.005s	sys 0.008s
W20260709 12:28:31.510602  5676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.006s	sys 0.008s
W20260709 12:28:31.448841 14405 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.831s	user 0.004s	sys 0.010s
I20260709 12:28:31.446895 14712 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:31.495748  5602 leader_election.cc:336] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.598s (SENT)
I20260709 12:28:31.458086 14790 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:31.524718  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 111627us
I20260709 12:28:31.513516 14631 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:31.373953 14779 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.461939  5558 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:31.496615  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1587) took 1505 ms. Trace:
I20260709 12:28:31.499229 14117 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:31.509621 14797 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:28:31.507421  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 65699.2us
I20260709 12:28:31.511804  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:31.395613 14647 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.498260 14716 raft_consensus.cc:993] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:28:31.449184 14792 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:31.388882 14755 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.514966  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:31.516742  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:31.553026  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:31.554591  5558 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:28:31.517880  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 148670us
I20260709 12:28:31.507841  5552 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:31.493711 14795 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.079s
I20260709 12:28:31.589654  5554 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:31.463719  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:31.519145 14732 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.443735 14706 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.070s
I20260709 12:28:31.513880 14793 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:31.531153  6280 rpcz_store.cc:276] 0709 12:28:29.991048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:29.991207 (+   159us) service_pool.cc:224] Handling call
0709 12:28:29.997838 (+  6631us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:29.998104 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:28:30.329052 (+330948us) log.cc:844] Serialized 14 byte log entry
0709 12:28:31.496099 (+1167047us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:31.496182 (+    83us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:31.496250 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:31.496345 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:31.496543 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":550353,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":521422,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":6422,"thread_start_us":1493356,"threads_started":3,"wal-append.queue_time_us":338782}
I20260709 12:28:31.538666  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:31.517489 14663 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:31.533339 14117 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:31.506832 14745 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.034s	user 0.014s	sys 0.000s
W20260709 12:28:31.514843 14405 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.006s	sys 0.011s
I20260709 12:28:31.590878  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:31.527398  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:31.583755 14803 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:31.536515 14796 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:31.527037 14712 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.509450 14799 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:31.608989  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 70439.4us
I20260709 12:28:31.720526  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:31.591364  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 83993.6us
W20260709 12:28:31.605270 14784 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.274s	user 0.000s	sys 0.001s
W20260709 12:28:31.693269 14682 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.226s	user 0.009s	sys 0.004s
I20260709 12:28:31.645394 14738 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:31.719435 14824 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:28:31.633335 14783 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:31.646889 14208 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.120s	user 0.013s	sys 0.001s
I20260709 12:28:31.651275  5557 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
W20260709 12:28:31.742254 14777 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.985s	user 0.003s	sys 0.001s
I20260709 12:28:31.627728 14754 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.131s
W20260709 12:28:31.628317 14780 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.295s	user 0.000s	sys 0.001s
I20260709 12:28:31.628753 14811 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:31.528863 14788 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:31.637396 14647 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:31.537220 14631 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.644107 14593 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:31.706202  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:31.747716 14796 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.609606 14663 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:31.640760  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 145071us
I20260709 12:28:31.642766 14755 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:31.628860 14805 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.078s
I20260709 12:28:31.634382  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3189) took 1118 ms. Trace:
W20260709 12:28:31.659466  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 66510us
I20260709 12:28:31.897802  5709 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:28:31.900115  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:31.901149  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:31.902022  5705 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:31.714406  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:31.611716 14745 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:31.605504 14742 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:31.555177 14797 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.140s
I20260709 12:28:31.704392 14439 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:31.924911 14439 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:31.740494  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:31.646559 14409 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.105s	user 0.012s	sys 0.000s
I20260709 12:28:31.749048  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:31.740043 14804 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:31.594089 14716 raft_consensus.cc:1081] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Signalling peer f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:28:31.718972 14761 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:31.762930  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:31.738909 14807 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:31.784322  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3190) took 1234 ms. Trace:
I20260709 12:28:31.790481 14799 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:31.791095 14765 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.666s	user 0.013s	sys 0.000s
I20260709 12:28:31.791672 14732 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:31.793632 14779 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:31.746948 14706 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:31.796026  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:31.748958 14682 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.761746 14117 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:31.806612 14776 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.679s	user 0.001s	sys 0.000s
I20260709 12:28:31.766232 14783 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.808117  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:31.808944 14764 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:31.613636 14793 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.809442 14669 raft_consensus.cc:993] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: : Instructing follower f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:28:31.970141 14669 raft_consensus.cc:1081] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Signalling peer f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:28:31.972587  5709 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:31.767143 14208 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.973956  5703 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:31.760944 14819 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:31.977417 14819 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:31.978703 14819 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.816538 14712 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:31.817772 14262 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:31.814033 14205 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:31.981781 14262 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:31.857724  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 246779us
W20260709 12:28:31.851843 14806 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.428s	user 0.002s	sys 0.000s
I20260709 12:28:31.982733 14792 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:31.985121  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:31.851253 14631 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:31.884735  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:31.856243 14693 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:31.886552 14680 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:31.988032 14680 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:31.988698  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:31.989060 14680 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.990491 14680 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:31.847886 14748 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:31.858047 14613 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:31.992674 14613 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:31.993059 14613 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:31.869009 14730 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:31.996835 14730 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.869546 14744 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:32.000928 14744 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.841236 14692 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:31.848826 14788 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:31.902997  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:32.004685 14692 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:31.845834 14674 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.257s	user 0.007s	sys 0.006s
I20260709 12:28:32.006726 14674 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:32.007400 14692 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:31.894510  5629 rpcz_store.cc:276] 0709 12:28:30.515831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:30.515987 (+   156us) service_pool.cc:224] Handling call
0709 12:28:31.634187 (+1118200us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b540043a154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:31.634312 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1796808448}
I20260709 12:28:31.914623  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:32.008647  5555 raft_consensus.cc:2376] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25. Re-sending same reply.
I20260709 12:28:32.009114  6275 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: [32.22-32.22]
I20260709 12:28:31.926133  5707 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:31.820719 14617 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:31.927238  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:32.015324  5706 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:31.862040 14796 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:31.846334 14750 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:31.932000  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:32.031590  5702 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:31.932042  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:31.939726  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:31.946936  5705 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:31.944545  5630 rpcz_store.cc:276] 0709 12:28:30.549987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:30.550144 (+   157us) service_pool.cc:224] Handling call
0709 12:28:30.550455 (+   311us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:30.929576 (+379121us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:30.929778 (+   202us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:30.932509 (+  2731us) log.cc:844] Serialized 52 byte log entry
0709 12:28:31.514415 (+581906us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:31.514452 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:31.514542 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:31.784037 (+269495us) raft_consensus.cc:1685] finished
0709 12:28:31.784058 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:31.784248 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3210,"raft.queue_time_us":375618,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":12502,"spinlock_wait_cycles":51456,"thread_start_us":752155,"threads_started":2,"wal-append.queue_time_us":765374}
W20260709 12:28:31.959890  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 151750us
I20260709 12:28:31.939798 14439 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:31.949787 14804 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.966468  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1579) took 1625 ms. Trace:
I20260709 12:28:31.952843 14765 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:31.968479 14682 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:31.933285 14640 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:31.858412 14811 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:32.052428  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:31.969440 14793 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:31.981911 14205 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:31.818652 14803 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:32.009838  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 284835us
I20260709 12:28:31.975826 14208 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:32.038581  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:31.960652 14764 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:32.003950 14744 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:32.046133  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:31.933215 14409 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:32.011034 14761 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:28:32.041227  5710 rpcz_store.cc:276] 0709 12:28:30.341553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:30.341708 (+   155us) service_pool.cc:224] Handling call
0709 12:28:30.342103 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:30.342582 (+   479us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:30.342609 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:30.353179 (+ 10570us) log.cc:844] Serialized 26 byte log entry
0709 12:28:30.724635 (+371456us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:28:31.145615 (+420980us) log.cc:844] Serialized 14 byte log entry
0709 12:28:31.446354 (+300739us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:31.446456 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:31.805916 (+359460us) spinlock_profiling.cc:243] Waited 5.12 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:31.966150 (+160234us) raft_consensus.cc:1685] finished
0709 12:28:31.966259 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:31.966404 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":89588,"prepare.queue_time_us":204297,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":9027,"raft.queue_time_us":260445,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":34580,"spinlock_wait_cycles":11784064,"thread_start_us":812784,"threads_started":3,"wal-append.queue_time_us":583331}
I20260709 12:28:32.013336 14262 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:32.031598 14706 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:32.055837  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.059212  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:32.013149 14742 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.041s
I20260709 12:28:32.072034  5704 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:32.093902  5710 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:28:32.079052  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:32.083182  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44736.7us
I20260709 12:28:32.085678  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.087380  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:32.088707  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:32.090006  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:32.047796 14640 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:32.035859 14617 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:32.104321 14640 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:32.105525 14617 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:32.102873  5710 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:32.107014  5561 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:32.105676 14640 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:32.118853  5710 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:28:32.123560  5630 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 26: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:28:32.123363  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 230627us
W20260709 12:28:32.126744  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 67635.4us
I20260709 12:28:32.147797  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:32.148598  6274 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:32.150910  5555 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:28:32.151568  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:32.127981  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:32.061420 14730 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:32.086432 14716 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:32.066082 14712 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.067s
I20260709 12:28:32.147408  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:32.153836  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.158128  5707 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:32.160211  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.170601  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:32.167904  5555 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:32.175417  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:32.164602  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.178431  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.178835  5554 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:32.179263  5553 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:28:32.181980  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:32.091216 14765 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.060s
I20260709 12:28:32.183183  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:32.185592  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.184434  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:32.184954  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57070.1us
I20260709 12:28:32.186471  5557 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:32.141714 14208 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:32.190785  5558 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:32.136776 14833 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:32.193360  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:32.244783  5560 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:32.246927  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.154410 14617 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:32.248824 14617 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:32.250897 14617 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:32.083459 14832 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.111s
I20260709 12:28:32.252149 14832 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:32.252530 14832 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:32.253294 14832 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:32.254551 14832 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:32.206977 14765 consensus_queue.cc:1243] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 24.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:28:32.223106  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:32.256742  5634 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:28:32.207305  5630 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:28:32.207233  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.207923 14409 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:32.261353  5553 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 24: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:32.208758  5557 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:28:32.204326 14796 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:32.192832 14447 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:32.271690  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.282029  5634 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:32.213892 14738 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:32.214221  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:32.189049 14593 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:32.219689  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:32.222157  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.223155 14835 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:32.224812  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.136300 14800 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.069s
I20260709 12:28:32.163544 14793 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.062s
I20260709 12:28:32.132107 14764 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:32.083698 14803 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:32.226888  5602 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.533s (SENT)
I20260709 12:28:32.261720 14779 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:32.231314  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:32.230919 14706 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:32.232281 14783 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:32.233701 14205 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:32.235142  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.305788  5702 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:32.204695 14208 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:32.237368 14804 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:32.239648 14761 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:32.280735  5633 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:32.266074  5706 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
W20260709 12:28:32.283465  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 290459us
I20260709 12:28:32.283157  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:32.285419  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:32.286722  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 67129.4us
W20260709 12:28:32.287285  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 73137.5us
I20260709 12:28:32.291213  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:32.296005 14788 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:32.295399  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:32.296041  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.297107  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.299072  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:32.331396  5560 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:32.332931  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.299866  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.227752  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:32.295125 14447 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:32.345621  5704 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
W20260709 12:28:32.346186  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 119322us
I20260709 12:28:32.303088  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:32.304522  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:32.261003 14833 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:32.292863  5628 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 26: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:32.320508  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.295008 14805 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:32.323200  6271 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 19.
I20260709 12:28:32.329595  5552 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:32.271091 14712 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.040s
I20260709 12:28:32.298386  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.345675  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.348273  5559 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:32.347078  5602 leader_election.cc:336] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.598s (SENT)
I20260709 12:28:32.373503  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:32.348177  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:32.359156  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.360581  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:32.363773  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.303399 14779 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:32.287976 14796 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:32.294819 14593 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:32.382124  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.387495  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.317282 14732 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:32.298496 14835 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:32.322223 14761 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:32.328882 14788 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:32.399713  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.399816  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.368077 14409 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:32.413975  5628 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 24: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:32.418509  5709 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:32.418871  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.420109  5634 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 27 because replica has last-logged OpId of term: 26 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
I20260709 12:28:32.362758  5706 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:38468
W20260709 12:28:32.406533  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59515us
W20260709 12:28:32.411356  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:32.421373  5602 leader_election.cc:336] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.608s (SENT)
I20260709 12:28:32.345008 14800 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:28:32.379626 14841 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:32.425621  5602 leader_election.cc:336] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.562s (SENT)
I20260709 12:28:32.425643  5557 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:28:32.427764 14447 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:32.446342  5556 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27. Ops: []
W20260709 12:28:32.405649 14763 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.188s	user 0.016s	sys 0.023s
I20260709 12:28:32.448607  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:32.450201  5705 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:32.401880 14805 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:32.412317 14788 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:32.456277  5559 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:32.484757  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.485131  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.459384  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:32.487195  5557 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:28:32.412437 14554 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:28:32.452534 14763 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.239s	user 0.017s	sys 0.024s
I20260709 12:28:32.494764  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:32.414796 14732 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:32.501693  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.412315 14613 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:32.529049  5602 leader_election.cc:336] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.183s (SENT)
I20260709 12:28:32.486584 14439 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:32.535151  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:32.546847  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:32.556972  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.486274 14793 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
W20260709 12:28:32.552243  5602 leader_election.cc:336] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.443s (SENT)
I20260709 12:28:32.451622 14841 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:32.555856  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:32.531105 14834 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
I20260709 12:28:32.495606 14692 raft_consensus.cc:2764] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:28:32.562582  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:32.575248  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.577593  5553 raft_consensus.cc:2376] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21. Re-sending same reply.
W20260709 12:28:32.504127 14680 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.513s	user 0.012s	sys 0.002s
I20260709 12:28:32.542212 14738 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:32.548696 14730 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:32.566577 14617 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:32.481920 14779 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:32.575690 14693 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:28:32.623495  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 82721.5us
I20260709 12:28:32.583398 14805 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:32.629045 14845 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
W20260709 12:28:32.630167 14850 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:28:32.642417  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:32.643177  5710 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:32.651216 14674 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:32.312698 14682 consensus_queue.cc:1175] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "38736cc545eb4389b220a3bd21d3b466" responder_term: 26 status { last_received { term: 26 index: 15 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: [25.15-25.15]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:28:32.661293 14581 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.040s
I20260709 12:28:32.669067 14803 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:32.671424 14208 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:32.492343 14761 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:32.675012 14847 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:28:32.564868 14804 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 12:28:32.564787 14677 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.046s
I20260709 12:28:32.681030 14693 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:32.581728 14738 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:32.708987  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:32.591853 14599 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:32.503151 14833 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:32.725255  5710 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:32.780812  5636 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
W20260709 12:28:32.715780  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 131364us
W20260709 12:28:32.724677  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 75693.2us
I20260709 12:28:32.721909  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:32.785622  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63796.5us
I20260709 12:28:32.786176  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:28:32.790563  5708 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.048s	sys 0.105s
I20260709 12:28:32.787447  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:32.791023  5708 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.049s	sys 0.106s
I20260709 12:28:32.719054 14617 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:32.794359  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:32.777055 14835 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:32.723326 14693 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:32.720880 14803 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:32.803427  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:32.763065 14796 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:32.802924  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:32.712815 14800 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.084s
I20260709 12:28:32.726032 14738 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:32.781770 14593 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:32.812381  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.831554  5556 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:32.814818  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.792862 14405 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:28:32.796484 14853 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:28:32.801666 14858 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:32.818823  5708 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:32.794278 14663 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:32.722424 14680 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.085s
W20260709 12:28:32.829653  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:32.832868  5555 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:32.835733  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.837563  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:32.838984  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:32.808542 14730 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:32.821398 14811 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:32.810038 14772 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:32.793228 14763 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.031s
I20260709 12:28:32.801796 14801 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
W20260709 12:28:32.816701 14852 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:28:32.820583 14835 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:32.847119 14858 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:32.870229  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:32.833803 14208 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:32.830724 14841 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:32.838706 14599 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:32.835636 14647 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:32.835395 14405 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:32.881340  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:32.813161  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:32.827751 14803 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:32.890163 14858 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:32.871367 14848 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:32.899392  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.874872 14554 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.025s
I20260709 12:28:32.884781 14835 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:32.885589 14772 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:32.906723  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.894763 14763 consensus_queue.cc:1243] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 24.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:28:32.918452 14848 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:32.921016 14848 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:32.911603  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:32.922339 14811 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:32.922863 14811 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:32.923424  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.925122 14811 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:32.933025 14730 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:28:32.935531 14832 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.681s	user 0.013s	sys 0.000s
I20260709 12:28:32.936223  5554 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:32.915483  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.907938 14599 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:32.937503  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:32.937397 14599 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:32.937942  5561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3207) took 1623 ms (client timeout 1562 ms). Trace:
W20260709 12:28:32.938587  5561 rpcz_store.cc:269] 0709 12:28:31.314650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:31.327755 (+ 13105us) service_pool.cc:224] Handling call
0709 12:28:32.937667 (+1609912us) spinlock_profiling.cc:243] Waited 727 ms on lock 0x7b54001c57d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:32.937829 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1672011648}
I20260709 12:28:32.905141 14405 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:32.941709 14117 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:32.904973 14647 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:32.945127 14647 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:32.945896 14593 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:32.946285 14593 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:32.946349  5557 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:28:32.947064 14593 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:32.916216  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:32.907012 14748 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.896s	user 0.012s	sys 0.004s
I20260709 12:28:32.888175 14617 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:32.949393  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:32.950676  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:32.911990  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 98933us
I20260709 12:28:32.951809  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:32.952557 14748 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:28:32.953025  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:32.953846 14790 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:32.954098 14748 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:32.954506 14790 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:32.918633  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.938081  5555 raft_consensus.cc:2376] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21. Re-sending same reply.
I20260709 12:28:32.958390  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.959806  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:32.916181  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:32.962167 14803 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:32.962513  5553 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:28:32.966951 14693 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:32.940035  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:32.964100  5555 raft_consensus.cc:2376] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26. Re-sending same reply.
I20260709 12:28:32.974565 14761 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:32.977260  5630 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:28:32.900666 14208 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:32.954680  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:32.982712 14687 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:28:32.980322 14779 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:32.981964 14208 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:32.986959  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:32.990582  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:32.948266 14593 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:32.991575  5707 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:32.991745  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.992105  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:32.990028 14687 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:32.993916  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1599) took 1614 ms. Trace:
I20260709 12:28:32.995482  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:32.994498  6275 rpcz_store.cc:276] 0709 12:28:31.379798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:31.395200 (+ 15402us) service_pool.cc:224] Handling call
0709 12:28:31.395524 (+   324us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:32.009520 (+613996us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:32.993543 (+984023us) spinlock_profiling.cc:243] Waited 613 ms on lock 0x7b5400591ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:32.993838 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1410543360}
I20260709 12:28:32.998191 14772 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:32.999209  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:32.917210 14835 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:32.976032 14788 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:32.988606 14804 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:33.014631  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:33.021742  5555 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
W20260709 12:28:33.021010  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:33.010035 14687 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:33.025367  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:33.025688  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:33.026057  5635 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:28:33.046483  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:33.047365  5706 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:33.047920  5706 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:33.048749  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:33.057600  6271 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:28:33.065547  5632 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:28:33.066851  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:33.067196  6275 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:28:33.069234  5627 raft_consensus.cc:2410] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:28:33.046911 14117 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.024s
I20260709 12:28:33.074469 14405 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.046398 14172 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:33.076543 14172 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.078681 14172 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:33.080272  6277 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:28:33.083146 14858 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.085053  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:33.086220 14763 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:33.086675  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:33.086760 14763 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.088341  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:33.088932  5636 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:33.090261 14748 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:33.050120 14706 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:33.092113  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:33.094748  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:33.095515 14785 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:28:33.095909 14785 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:33.096884 14674 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:33.099539 14790 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:33.109992 14785 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.111568 14785 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:28:33.079861 14738 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.095997  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:33.112897  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3715) took 1150 ms. Trace:
I20260709 12:28:33.120874  5704 rpcz_store.cc:276] 0709 12:28:31.962565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:32.170536 (+207971us) service_pool.cc:224] Handling call
0709 12:28:32.204405 (+ 33869us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:32.346163 (+141758us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:33.112592 (+766429us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b54004529d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:33.112821 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":323952000}
I20260709 12:28:33.119737  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3716) took 1100 ms. Trace:
I20260709 12:28:33.123289  5707 rpcz_store.cc:276] 0709 12:28:32.019424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:32.178708 (+159284us) service_pool.cc:224] Handling call
0709 12:28:32.208006 (+ 29298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:32.992028 (+784022us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:33.119427 (+127399us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b5400066654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:33.119662 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1801548672}
I20260709 12:28:33.032810 14828 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:33.143505  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47608.8us
I20260709 12:28:33.145478  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:33.156484  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:33.158216  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:33.159103  6272 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
W20260709 12:28:33.115239 14837 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.904s	user 0.007s	sys 0.000s
I20260709 12:28:33.114938 14693 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:33.178692 14738 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:33.117302 14405 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:33.182343 14405 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:28:33.183534 14405 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:28:33.184100 14832 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.184867 14832 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.189498 14525 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:33.190086 14525 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:33.112267 14613 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:33.142848 14172 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.176601  6279 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:28:33.190707  5451 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 17 to 19, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 19 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:28:33.195811  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50421.9us
I20260709 12:28:33.189575  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:33.168895 14811 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:33.281214  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3214) took 1056 ms. Trace:
W20260709 12:28:33.264504 14859 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:28:33.375713  5630 rpcz_store.cc:276] 0709 12:28:32.225153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:32.376068 (+150915us) service_pool.cc:224] Handling call
0709 12:28:32.376401 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:32.977669 (+601268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:33.270911 (+293242us) spinlock_profiling.cc:243] Waited 600 ms on lock 0x7b54001dd8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:33.281122 (+ 10211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1380577024}
I20260709 12:28:33.377121  5552 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:33.378904  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:33.381011  5627 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29. Ops: [28.18-28.18]
I20260709 12:28:33.382326  5635 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
W20260709 12:28:33.383320  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:33.171583 14828 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.390297 14804 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:33.393174 14828 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:33.343551  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:33.372417 14858 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:33.372686  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1457) took 1624 ms. Trace:
I20260709 12:28:33.394450  6280 rpcz_store.cc:276] 0709 12:28:31.748315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:31.748799 (+   484us) service_pool.cc:224] Handling call
0709 12:28:31.764764 (+ 15965us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:31.765154 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:31.765184 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:31.765689 (+   505us) log.cc:844] Serialized 26 byte log entry
0709 12:28:32.015170 (+249481us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:32.015212 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:32.015278 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:33.114605 (+1099327us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:33.372318 (+257713us) raft_consensus.cc:1685] finished
0709 12:28:33.372341 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:33.372608 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1056,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":33086,"spinlock_wait_cycles":67174656,"thread_start_us":249097,"threads_started":1,"wal-append.queue_time_us":365045}
I20260709 12:28:33.396324  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:33.396734  5707 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:33.396771  5703 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:42568
I20260709 12:28:33.397701  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:33.397923  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:33.398283  5705 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:33.372786 14835 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.398646  5707 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:33.399235  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:33.399791  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:33.400429  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:33.401091  5705 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:33.400753  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:33.401769 14835 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:33.198933 14674 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:33.375211 14803 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.269654 14613 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.029s
I20260709 12:28:33.385860  5632 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:33.386538  5635 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: []
W20260709 12:28:33.291013  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 409860us
I20260709 12:28:33.220988 14172 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.405781 14172 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:33.398142  5703 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:33.406147  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:33.400624  5708 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:28:33.398439 14828 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:28:33.402050  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:33.408679  5627 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:33.410024 14761 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:28:33.410691 14447 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20260709 12:28:33.411393  5633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3192) took 2045 ms (client timeout 2165 ms). Trace:
I20260709 12:28:33.411603 14761 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:28:33.402645 14803 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:33.411823  5633 rpcz_store.cc:269] 0709 12:28:31.366894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:31.379978 (+ 13084us) service_pool.cc:224] Handling call
0709 12:28:33.411243 (+2031265us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b54001cf2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:33.411338 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1286561024}
I20260709 12:28:33.413102 14828 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:33.413352  5558 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:33.413635 14828 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:33.413892  5558 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:28:33.375198  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:33.385604  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:33.414778  5558 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.414845 14828 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.406418  5703 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:33.415953  5558 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:33.414023 14803 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:33.418835 14805 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:33.419239 14805 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:33.419479 14677 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:33.420001 14677 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.420092 14805 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.421140 14805 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:33.422341 14677 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:33.423657 14677 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.424773 14801 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:28:33.427363 14409 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:28:33.426982  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:33.428987  5556 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:33.429450  5556 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:33.430222  5556 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.431744 14647 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:28:33.432614 14647 consensus_queue.cc:1243] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 24.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:28:33.430743  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:33.433259 14647 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:28:33.424104 14677 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.507658  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3204) took 1289 ms. Trace:
I20260709 12:28:33.507692 14828 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:33.449452 14172 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.497196 14525 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.434018 14835 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:33.526355 14172 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.523550  5629 rpcz_store.cc:276] 0709 12:28:32.218334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:32.224513 (+  6179us) service_pool.cc:224] Handling call
0709 12:28:33.507455 (+1282942us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540008bbd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:33.507594 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2374359296}
I20260709 12:28:33.482831  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3719) took 1143 ms. Trace:
I20260709 12:28:33.440109 14765 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:33.530457 14765 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:33.530298  5709 rpcz_store.cc:276] 0709 12:28:32.339389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:32.339508 (+   119us) service_pool.cc:224] Handling call
0709 12:28:32.339819 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:32.419108 (+ 79289us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:33.482411 (+1063303us) spinlock_profiling.cc:243] Waited 78.4 ms on lock 0x7b54001991d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:33.482729 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":180314368}
I20260709 12:28:33.435999 14788 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:33.486500 14763 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:33.435379  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:33.508847 14832 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:33.527475 14835 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:33.526953 14525 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:33.527727  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:33.528091  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:33.431996  5703 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:33.473876  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:33.463279 14787 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.066s
I20260709 12:28:33.490283 14732 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:33.531395 14172 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:33.557690 14805 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:33.558216 14805 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:33.559188 14805 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.437681  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:33.560216  5703 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:33.560954  5557 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:33.561203  5554 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:33.459117 14857 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.595s	user 0.002s	sys 0.005s
I20260709 12:28:33.437953  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:33.564548  5553 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:33.435242  5555 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:28:33.455458 14810 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.074s
W20260709 12:28:33.470239 14855 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.600s	user 0.003s	sys 0.001s
I20260709 12:28:33.565622  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:33.565855 14832 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.566370 14832 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.566957  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:33.434741  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3722) took 1094 ms. Trace:
I20260709 12:28:33.567812  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:33.434182 14647 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:33.568958 14647 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:33.568172  5701 rpcz_store.cc:276] 0709 12:28:32.340126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:32.725647 (+385521us) service_pool.cc:224] Handling call
0709 12:28:32.754340 (+ 28693us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:32.754712 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:32.754754 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:32.760850 (+  6096us) log.cc:844] Serialized 26 byte log entry
0709 12:28:33.031709 (+270859us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:28:33.031755 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:33.031831 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:33.434388 (+402557us) raft_consensus.cc:1685] finished
0709 12:28:33.434443 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:33.434663 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4777,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":11764,"thread_start_us":239091,"threads_started":1,"wal-append.queue_time_us":655908}
I20260709 12:28:33.569715 14647 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:33.567763  5553 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 29: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:33.571587  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:33.572906  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:33.453184 14409 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.081s
I20260709 12:28:33.442066 14554 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.093s
W20260709 12:28:33.536959  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 106324us
I20260709 12:28:33.577098 14810 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:33.577508 14810 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:33.577816  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:33.578363 14810 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.579321  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:33.579756 14810 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:33.580297 14732 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:33.581001 14732 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:33.581552 14732 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:33.582562  5557 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:33.583650  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:33.576735  5553 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:33.585234 14208 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:33.585649 14208 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:33.585996 14208 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:33.592113  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:33.568579  5555 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:33.596808  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:33.523070  5556 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:33.448753 14712 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:33.598809 14712 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:33.599714  5557 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:33.374688 14800 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:33.605242 14800 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:33.606304 14800 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.555577 14835 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:33.564404  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3480) took 1067 ms. Trace:
I20260709 12:28:33.612560 14788 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:33.613300 14788 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.612659  5628 rpcz_store.cc:276] 0709 12:28:32.497421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:32.497552 (+   131us) service_pool.cc:224] Handling call
0709 12:28:32.525389 (+ 27837us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:32.531962 (+  6573us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:32.531992 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:32.549223 (+ 17231us) log.cc:844] Serialized 26 byte log entry
0709 12:28:32.710915 (+161692us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:32.710954 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:32.711030 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:33.458344 (+747314us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:33.563985 (+105641us) raft_consensus.cc:1685] finished
0709 12:28:33.564020 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:33.564317 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10481,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":16729,"spinlock_wait_cycles":7269120,"thread_start_us":161371,"threads_started":1,"wal-append.queue_time_us":260520}
I20260709 12:28:33.568287 14716 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:33.614784 14716 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:33.615679 14716 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.616873 14716 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:33.619304 14832 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:33.619887 14785 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.508s	user 0.009s	sys 0.000s
I20260709 12:28:33.622090  5451 catalog_manager.cc:5697] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 17 to 21, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 21 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:28:33.387914  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:33.615686 14788 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:33.466499 14764 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:33.625614 14750 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:28:33.411396 14447 consensus_queue.cc:1243] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 25.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:28:33.502725  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:33.574753  5554 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:33.633569  5552 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:33.538141 14677 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:33.375435  5706 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:33.555004 14692 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.012s	sys 0.016s
W20260709 12:28:33.656608  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 64735.9us
I20260709 12:28:33.661111  5553 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
W20260709 12:28:33.628697 14617 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.028s	sys 0.063s
I20260709 12:28:33.681038  5556 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:28:33.681049  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:33.688901  5558 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 18. (term mismatch)
I20260709 12:28:33.710788  5558 pending_rounds.cc:85] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 17
W20260709 12:28:33.664234 14692 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.013s	sys 0.016s
W20260709 12:28:33.712369  5561 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.708s	user 0.000s	sys 0.017s
W20260709 12:28:33.679440 14772 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.000s	sys 0.015s
W20260709 12:28:33.716295 14772 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.000s	sys 0.015s
I20260709 12:28:33.711148  5558 pending_rounds.cc:107] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 28.18
I20260709 12:28:33.666666 14804 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:33.666059 14788 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:33.682668 14617 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.028s	sys 0.063s
I20260709 12:28:33.718827  5556 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:28:33.718400  5558 raft_consensus.cc:2889] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:33.723860  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1821) took 1279 ms. Trace:
W20260709 12:28:33.708968 14599 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.015s	sys 0.002s
I20260709 12:28:33.706131 14730 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.725399  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:33.709345 14677 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:33.709630 14848 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.713721 14172 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:28:33.718236  5561 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.000s	sys 0.018s
I20260709 12:28:33.683714 14800 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:33.688386 14833 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.795s	user 0.008s	sys 0.010s
I20260709 12:28:33.724452  5558 rpcz_store.cc:276] 0709 12:28:32.444978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:32.445176 (+   198us) service_pool.cc:224] Handling call
0709 12:28:32.484766 (+ 39590us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:33.413258 (+928492us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b54002344a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa6741c0 00007f4ffa675231 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c
0709 12:28:33.723467 (+310209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:33.723790 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":542672256}
I20260709 12:28:33.739542  5555 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:33.711891 14832 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:33.713409 14755 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260709 12:28:33.712138 14763 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:33.714699 14117 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.009s	sys 0.008s
I20260709 12:28:33.743487  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:33.750319  5557 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
W20260709 12:28:33.709805 14796 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.009s	sys 0.008s
W20260709 12:28:33.728412 14599 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.015s	sys 0.003s
W20260709 12:28:33.730057 14802 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 29 current term: 29 is higher.
I20260709 12:28:33.741338  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:33.797881  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54500.4us
W20260709 12:28:33.731660 14593 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.740s	user 0.007s	sys 0.004s
W20260709 12:28:33.751235 14833 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.858s	user 0.011s	sys 0.010s
I20260709 12:28:33.729871 14804 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:33.810638  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 69643.8us
W20260709 12:28:33.798429 14117 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.009s	sys 0.009s
I20260709 12:28:33.811556  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:33.738178 14172 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:33.802202 14796 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.816s	user 0.010s	sys 0.009s
I20260709 12:28:33.726944 14788 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.804363 14841 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:33.738515 14730 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:33.824591  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:33.766225 14811 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.051s
W20260709 12:28:33.818960  5675 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.022s	sys 0.050s
I20260709 12:28:33.834749  5706 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:33.738334 14677 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.822376  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:33.752017 14848 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:33.835978  5675 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.023s	sys 0.051s
W20260709 12:28:33.827069  5601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.016s	sys 0.020s
I20260709 12:28:33.840451  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:38468 (request call id 3718) took 1496 ms. Trace:
W20260709 12:28:33.841271  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 718534us
I20260709 12:28:33.841540  5706 rpcz_store.cc:276] 0709 12:28:32.339047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:32.339222 (+   175us) service_pool.cc:224] Handling call
0709 12:28:33.815287 (+1476065us) spinlock_profiling.cc:243] Waited 626 ms on lock 0x7b5400075654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa669a3d 00007f4fff639cd0 00007f4ff8bbe254 00007f4ff8bbe1e2 00007f4ff8bbe111 00007f4ff8bbe08c 00007f4ff8bbd342 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:33.835777 (+ 20490us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1440492288}
I20260709 12:28:33.826049 14692 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:33.819175 14172 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:33.844412  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:33.797654 14832 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.845355  5703 raft_consensus.cc:2410] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:28:33.766637 14765 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.808310 14763 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:33.841523  5601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.016s	sys 0.024s
I20260709 12:28:33.825187 14869 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:33.851481  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:33.846542  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:33.859432  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:33.860232  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:33.907609  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 784297us
I20260709 12:28:33.908187  5553 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:28:33.853401  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:33.838109 14841 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:33.914901  5705 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:33.855245  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 428306us
I20260709 12:28:33.916882  5701 raft_consensus.cc:1813] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Handling vote request from an unknown peer cea66aa15be3423880843974acdbf50f
I20260709 12:28:33.918419  5701 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 20 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260709 12:28:33.920030  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:33.859531  5552 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:28:33.922017  5707 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 30. Ops: []
I20260709 12:28:33.923367  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:33.878623  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 462564us
I20260709 12:28:33.924247  5706 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:28:33.908043 14680 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.021s	user 0.003s	sys 0.009s
I20260709 12:28:33.927063  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:33.845238 14750 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:33.934885  5709 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:33.928207  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:33.854242 14692 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:33.937052  5706 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
W20260709 12:28:33.938477  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:33.942670  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:33.945233 14868 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.949414 14208 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.950471 14848 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:33.950883  5705 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
W20260709 12:28:33.934341 14680 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.050s	user 0.004s	sys 0.009s
I20260709 12:28:33.958328 14447 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.961979 14447 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.963428 14872 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.964030 14872 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:33.960178  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:33.964112 14447 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:33.964126 14867 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260709 12:28:33.965268 14409 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.014s
I20260709 12:28:33.979856 14409 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:33.980113  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3523) took 1165 ms. Trace:
I20260709 12:28:33.981758 14677 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:33.980964 14772 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:33.982918 14772 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:33.983302 14714 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:28:33.983837 14772 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.986200 14716 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:33.986558 14716 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:33.987362 14716 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:33.981755  5559 rpcz_store.cc:276] 0709 12:28:32.815069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:32.827331 (+ 12262us) service_pool.cc:224] Handling call
0709 12:28:32.850211 (+ 22880us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:32.859896 (+  9685us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:32.860368 (+   472us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:32.860990 (+   622us) log.cc:844] Serialized 26 byte log entry
0709 12:28:32.983870 (+122880us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x7b480027e958. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:28:33.295594 (+311724us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:33.295648 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:33.295730 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:33.979817 (+684087us) raft_consensus.cc:1685] finished
0709 12:28:33.979838 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:33.980055 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4718,"raft.queue_time_us":8320,"raft.run_cpu_time_us":2566,"raft.run_wall_time_us":121175,"spinlock_wait_cycles":7031680,"thread_start_us":413884,"threads_started":1,"wal-append.queue_time_us":924281}
W20260709 12:28:33.989032 14879 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:28:33.970417  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:33.993691 14730 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:33.995321 14409 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:33.996186 14785 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:28:33.997035 14447 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:28:33.997911 14447 consensus_queue.cc:1243] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 25.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:28:33.998941 14785 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:28:33.999293 14878 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:28:33.999624 14785 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:28:33.981309 14872 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:33.949409 14868 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:34.004454 14868 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:34.005273 14868 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:28:34.006306 14785 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:34.006816 14785 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:34.007779 14785 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:34.000758  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:34.009262 14785 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:34.010421  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:34.010701 14732 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:34.003998 14409 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:34.016888  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:34.022060 14868 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:28:34.022595 14874 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:28:34.023438  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:33.943202 14765 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:34.027637  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.795s	user 0.022s	sys 0.046s
W20260709 12:28:34.033749  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.801s	user 0.022s	sys 0.047s
I20260709 12:28:33.853071 14804 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:28:34.034219  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 804736us
W20260709 12:28:34.028896  6234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.008s	sys 0.018s
W20260709 12:28:34.036970  6234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.008s	sys 0.018s
I20260709 12:28:33.955104 14593 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:33.965689 14870 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
W20260709 12:28:33.964615 14866 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
W20260709 12:28:33.966172 14871 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
W20260709 12:28:33.965183 14873 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
I20260709 12:28:34.028908  5702 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 15. (term mismatch)
I20260709 12:28:34.060103  5702 pending_rounds.cc:85] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 14
I20260709 12:28:34.039629  5704 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:28:34.060568  5706 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:33.840665 14864 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.118s
I20260709 12:28:33.960078 14750 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:34.052114  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:34.053447  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:34.053422  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:34.030220 14872 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:28:34.057080  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:34.060595  5702 pending_rounds.cc:107] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 25.15
I20260709 12:28:34.040194 14599 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:28:34.135819  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 79535.6us
W20260709 12:28:34.131686  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 69036.2us
I20260709 12:28:34.035808 14647 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:34.141227  5702 raft_consensus.cc:2889] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:34.160270  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:34.145745  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 92476.5us
I20260709 12:28:34.162765  5706 raft_consensus.cc:2368] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24.
I20260709 12:28:34.163264  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:34.165462  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:34.053665 14593 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:34.167197 14525 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:34.167773 14525 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:34.168241 14801 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:34.168694 14801 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:34.169533  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:34.167346  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:34.172303  6277 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:34.153805  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:34.173890  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:34.091722 14409 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:34.214254  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:34.214936  5706 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:34.214944  5555 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:34.158207 14803 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:34.100071 14811 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.155s
I20260709 12:28:34.170116 14439 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:34.042382 14841 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:34.170439 14732 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:34.232817  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3723) took 1862 ms. Trace:
I20260709 12:28:34.223762 14788 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:34.228578 14525 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:34.234123 14525 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:34.235592 14764 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:34.213299 14208 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:34.233392  5710 rpcz_store.cc:276] 0709 12:28:32.370061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:32.792764 (+422703us) service_pool.cc:224] Handling call
0709 12:28:32.793209 (+   445us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:32.795377 (+  2168us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:32.795425 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:32.834656 (+ 39231us) log.cc:844] Serialized 26 byte log entry
0709 12:28:33.265328 (+430672us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:33.265393 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:33.265513 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:34.232547 (+967034us) raft_consensus.cc:1685] finished
0709 12:28:34.232579 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:34.232745 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":388266,"threads_started":1,"wal-append.queue_time_us":902646}
I20260709 12:28:34.101063 14835 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:34.279542  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3522) took 1464 ms. Trace:
I20260709 12:28:34.280123  5560 rpcz_store.cc:276] 0709 12:28:32.814802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:32.815251 (+   449us) service_pool.cc:224] Handling call
0709 12:28:32.836686 (+ 21435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:32.846280 (+  9594us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:32.846301 (+    21us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:32.872807 (+ 26506us) log.cc:844] Serialized 26 byte log entry
0709 12:28:33.294276 (+421469us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:33.294338 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:33.294438 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:34.279087 (+984649us) raft_consensus.cc:1685] finished
0709 12:28:34.279127 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:34.279451 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11046,"raft.queue_time_us":13871,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":12053,"thread_start_us":398004,"threads_started":1,"wal-append.queue_time_us":852121}
I20260709 12:28:34.292266 14712 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:34.294001 14712 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:34.303357  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:34.303964  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:34.304898  5701 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 23. Ops: []
I20260709 12:28:34.210665 14117 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.087s
I20260709 12:28:34.135366 14692 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:34.232743 14599 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:34.236173 14764 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:34.312515  5704 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: [26.13-26.13]
I20260709 12:28:34.242093 14872 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.074s
I20260709 12:28:34.311265 14883 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:34.263887 14439 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:34.330279  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:34.318531 14779 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:34.316041 14599 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:34.336122 14786 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:28:34.350463  5561 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.010s	sys 0.022s
W20260709 12:28:34.371248  5561 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.010s	sys 0.022s
I20260709 12:28:34.357757  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:34.372310  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3254) took 1381 ms. Trace:
I20260709 12:28:34.328069 14593 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:34.336124 14764 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:34.325840 14832 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:34.373663  5561 rpcz_store.cc:276] 0709 12:28:32.990412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:33.003723 (+ 13311us) service_pool.cc:224] Handling call
0709 12:28:33.004057 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:33.004272 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:28:33.752285 (+748013us) log.cc:844] Serialized 14 byte log entry
0709 12:28:34.371858 (+619573us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:34.371936 (+    78us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:34.371990 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:34.372076 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:34.372234 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":834167,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":702,"thread_start_us":1290881,"threads_started":2}
W20260709 12:28:34.391561  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 222440us
W20260709 12:28:34.373538 14800 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.637s	user 0.010s	sys 0.002s
I20260709 12:28:34.393237  5701 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:34.371842 14599 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:34.318019 14750 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:34.406805  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:34.410984  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:34.373652 14869 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:34.381786 14786 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:34.414072  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:34.418581  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:34.418556  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:34.419317  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.564s (SENT) [suppressed 27 similar messages]
I20260709 12:28:34.388342 14763 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:34.424930  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:34.424250  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:34.432796  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:34.438498  5704 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:34.440426  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:34.441311  5704 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:34.453469  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 290310us
W20260709 12:28:34.421366 14796 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.025s	sys 0.062s
W20260709 12:28:34.457306  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 153109us
W20260709 12:28:34.458876  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:34.460662  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:34.460799  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:34.458678  6236 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:34.462942  6272 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:28:34.465400  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:34.466936  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:34.466099  5630 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:34.429533 14779 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:34.478837  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:34.486426  5704 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:34.487582  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:34.488526  5704 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 27 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:28:34.454459 14882 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:28:34.463622 14796 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.025s	sys 0.062s
I20260709 12:28:34.463409 14803 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:34.468649 14872 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:34.468972 14886 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:34.482023 14779 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:34.499295 14786 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:34.497782 14835 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:34.641566  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 222249us
W20260709 12:28:34.678810  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:34.683007  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:34.683552  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:34.684932  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:34.685699  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:34.686782  5704 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:28:34.687484  5555 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:34.498802 14882 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:34.630120 14785 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.620s	user 0.008s	sys 0.004s
I20260709 12:28:34.690394 14882 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:34.648897 14599 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:34.700296  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:34.701009  5556 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:34.701942  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:34.702929  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:34.704727 14903 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:28:34.706694 14903 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:34.707031 14903 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:34.708145 14903 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:34.709829  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:34.710943 14884 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:34.711318 14884 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:34.711654 14884 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:34.703436  5556 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:28:34.718351  5556 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:28:34.725133  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:34.726780 14785 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:34.728428 14785 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:34.734489 14906 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:34.658558 14803 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:28:34.685281 14765 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.649s	user 0.013s	sys 0.000s
I20260709 12:28:34.632758 14779 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:34.760368  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:34.683316 14886 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:34.762148  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:34.743422 14848 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:34.774173  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:34.775422  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:34.732617 14785 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:34.778316  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:34.772250 14904 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:28:34.770803  5559 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
W20260709 12:28:34.762965 14889 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.164s	user 0.000s	sys 0.000s
I20260709 12:28:34.785023  5701 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:34.608273 14872 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:34.757827 14786 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:34.754644 14803 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:34.790052  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:34.763164 14763 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:34.616274 14902 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.162s
I20260709 12:28:34.797482  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:34.797848  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:34.778308 14909 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:34.806756  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:34.546435 14869 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:34.779294 14835 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:34.779991 14804 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:34.800374  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:34.815378  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:34.816227  5701 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:28:34.817251  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:34.825132  5708 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:34.804385 14803 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:34.813144 14409 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.593s	user 0.009s	sys 0.003s
I20260709 12:28:34.830750  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:34.807735 14525 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:34.807428 14894 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:34.804762 14617 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:34.833217  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:34.819953 14172 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.966s	user 0.013s	sys 0.000s
I20260709 12:28:34.786360 14848 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:34.813232 14796 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:28:34.840713  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:34.842173  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1588) took 2620 ms. Trace:
I20260709 12:28:34.848742  5702 rpcz_store.cc:276] 0709 12:28:32.222150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:32.307549 (+ 85399us) service_pool.cc:224] Handling call
0709 12:28:32.307877 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:34.164619 (+1856742us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:34.841817 (+677198us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54004524d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:34.842090 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3955712256}
I20260709 12:28:34.852579  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:34.853938  5701 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 31 because replica has last-logged OpId of term: 30 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 19.
I20260709 12:28:34.855232  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:34.856034  5701 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:34.856637 14914 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:34.857162 14900 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.202s	user 0.000s	sys 0.000s
I20260709 12:28:34.857319 14914 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:34.847107 14896 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:34.813446 14909 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:34.809980 14763 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:34.868706  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3729) took 1012 ms. Trace:
I20260709 12:28:34.877789  5555 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:34.876859  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:34.884104  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 73392.5us
I20260709 12:28:34.884416  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:34.880741  5703 rpcz_store.cc:276] 0709 12:28:33.856429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:33.865241 (+  8812us) service_pool.cc:224] Handling call
0709 12:28:33.919048 (+ 53807us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:33.937017 (+ 17969us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:28:34.313223 (+376206us) log.cc:844] Serialized 14 byte log entry
0709 12:28:34.505651 (+192428us) log.cc:844] Serialized 14 byte log entry
0709 12:28:34.505906 (+   255us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:34.598216 (+ 92310us) log.cc:844] Serialized 26 byte log entry
0709 12:28:34.598541 (+   325us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:34.598611 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:34.598705 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:34.868438 (+269733us) raft_consensus.cc:1685] finished
0709 12:28:34.868472 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:34.868647 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31843,"prepare.queue_time_us":26485,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":59,"raft.queue_time_us":16666,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":51944,"spinlock_wait_cycles":298582400,"thread_start_us":469528,"threads_started":2,"wal-append.queue_time_us":462931}
I20260709 12:28:34.884886  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:34.885036  5701 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:34.821344 14599 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:34.895157  5630 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:28:34.900665  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:34.868340 14910 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:28:34.941123  5561 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: [23.14-23.14]
W20260709 12:28:34.941838  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:34.929975  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:34.908685  5630 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:28:34.938244  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:34.945013  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:34.946130  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:34.930356  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:34.949119  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:34.944274  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:34.876291 14763 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:34.909358  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:34.953261  5636 raft_consensus.cc:2410] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 22 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 12:28:34.911015  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:34.912729  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:34.913764  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:34.954185  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:34.955128  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:34.955482  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:34.956462  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:34.957070  5633 raft_consensus.cc:2376] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28. Re-sending same reply.
I20260709 12:28:34.957330  5706 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:28:34.915033  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:34.917419  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:34.959046  6273 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:34.863669 14913 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:34.928823  5557 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:28:34.860306 14914 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:34.924167  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:34.924628  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:34.963852  6278 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:28:34.964545  5707 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:28:34.965813  5707 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
W20260709 12:28:34.966061  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 505944us
I20260709 12:28:34.966828  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:34.967906  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:34.961755  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:34.926972  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:34.926023 14884 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:34.938169  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:34.939767 14872 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:34.940470 14894 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:28:34.867210 14886 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:34.940879  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3535) took 1352 ms. Trace:
I20260709 12:28:34.991334  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3735) took 1130 ms. Trace:
I20260709 12:28:34.991019  5553 rpcz_store.cc:276] 0709 12:28:33.588971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:33.802219 (+213248us) service_pool.cc:224] Handling call
0709 12:28:33.802662 (+   443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:33.914177 (+111515us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:34.940620 (+1026443us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b5400082354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:34.940821 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":237551488}
I20260709 12:28:34.991779  5709 rpcz_store.cc:276] 0709 12:28:33.861011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:33.939210 (+ 78199us) service_pool.cc:224] Handling call
0709 12:28:33.940907 (+  1697us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:33.941193 (+   286us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:33.941220 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:33.941265 (+    45us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:33.941294 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:33.941352 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:34.991145 (+1049793us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b54002f5b50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:34.991277 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2650496}
I20260709 12:28:34.993022  5704 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:34.993074  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:34.878682 14848 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:34.878809 14796 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:34.943411  5561 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:34.907037  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:34.950819  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:34.997792  5702 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:34.951807  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:34.952281  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:34.919405  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:34.864722 14525 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:34.872225 14909 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:34.886664 14617 raft_consensus.cc:2764] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:28:34.867348 14593 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.084s
I20260709 12:28:34.986155  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:34.986996  6278 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:28:34.987424  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:34.987906  6280 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:35.015938  5558 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:34.993994  5709 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:35.017354  5704 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:28:34.988502 14914 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:34.996059  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:34.990103 14763 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:35.023391  5557 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:35.002341  5703 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:35.027267  6273 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
W20260709 12:28:34.997400  5675 connection.cc:256] RPC call timeout handler was delayed by -1.3748468429998866s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2077s for a response
I20260709 12:28:34.864566 14172 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.029516  5702 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:35.019845  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:34.989500 14714 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:34.984558 14554 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:35.031854  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:35.000908 14796 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:35.041666  6279 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
I20260709 12:28:35.045295  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:35.047049  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:35.017678 14909 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:28:35.043823 14848 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:34.996557 14786 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.042s
I20260709 12:28:35.028888 14835 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:35.054322  6271 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 27: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:35.059496  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3538) took 1401 ms. Trace:
I20260709 12:28:35.059695  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:35.053766  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:35.084311  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 98251.9us
I20260709 12:28:35.078117  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:35.077455  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:35.044219 14716 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:35.085953  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:35.044730 14554 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:35.076596  5552 rpcz_store.cc:276] 0709 12:28:33.658462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:33.709232 (+ 50770us) service_pool.cc:224] Handling call
0709 12:28:35.059278 (+1350046us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54003a98d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:35.059425 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":297976576}
I20260709 12:28:35.083017  5558 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:28:35.054812 14800 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:35.029016 14765 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 12:28:35.060956 14617 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:35.049464 14872 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 12:28:35.111652  6273 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29. Ops: [28.18-28.18]
I20260709 12:28:35.034979 14914 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.109812 14712 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:35.102473 14800 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:35.086306 14868 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:35.051075 14810 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 29.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260709 12:28:35.087842 14750 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:35.114210  6273 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:28:35.099382 14554 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:35.084515 14755 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:35.120450  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:35.100067 14919 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:35.120903  5558 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:28:35.126667  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:35.073333 14804 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.043s
I20260709 12:28:35.136540  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:35.137383  6280 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:34.932277 14803 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.139523  6275 raft_consensus.cc:2368] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27.
W20260709 12:28:35.140151  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53394.8us
I20260709 12:28:35.086068 14835 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.102514 14864 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.019s
W20260709 12:28:35.124231 14915 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 12:28:35.145424  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:35.146602  6278 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:35.116984 14712 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:35.148449  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:35.125151 14800 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:35.155679  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:35.152734  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:35.158344  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:35.132164 14848 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:35.102586 14525 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.072476 14902 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:35.109467 14617 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.123534 14868 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:35.181079 14920 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260709 12:28:35.186239  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:35.189091  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:34.985298 14913 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:34.867722 14896 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.049934 14714 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.165244 14763 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:35.202225  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:35.202680  5631 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
W20260709 12:28:35.233987 14915 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:28:35.171408 14848 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.352442  5635 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:35.234983 14172 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:35.235759 14868 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.210410 14712 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.239090 14835 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:35.192222 14750 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.047s
I20260709 12:28:35.241235 14779 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.243434 14409 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:35.244038 14617 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:35.196453 14883 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.847s	user 0.010s	sys 0.004s
I20260709 12:28:35.246907 14801 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.254796  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1618) took 1101 ms. Trace:
I20260709 12:28:35.257048 14884 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:35.367116  6271 rpcz_store.cc:276] 0709 12:28:34.153302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:34.153429 (+   127us) service_pool.cc:224] Handling call
0709 12:28:35.254447 (+1101018us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b54000b54d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:35.254696 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2022414208}
I20260709 12:28:35.368383  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:35.369107  6271 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 27: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:28:35.258502 14922 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 12:28:35.203969 14803 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:35.267763 14911 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.456s	user 0.001s	sys 0.000s
I20260709 12:28:35.154908 14765 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:28:35.209046 14763 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:35.282157 14593 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:28:35.378669 14763 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:35.378719 14593 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:35.379401 14882 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.687s	user 0.008s	sys 0.005s
W20260709 12:28:35.285765  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:35.379927 14593 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 29.12, Last appended by leader: 12, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.381253 14872 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:35.381315 14593 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:35.381852 14872 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.288271  5555 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:28:35.291889  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:35.383423  5561 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:28:35.379851 14882 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.385525 14716 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:35.386025 14716 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.386816 14882 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:35.384176 14872 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:35.388010 14716 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:35.313179  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:35.388222 14939 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:28:35.319250  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:35.388968 14939 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:35.203297  6279 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:35.309108 14805 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:35.389634 14939 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:35.324414  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 182834us
I20260709 12:28:35.393121  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:35.394613 14617 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:35.316277 14439 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:35.395417 14617 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:35.396337 14617 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.329671 14913 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:35.330608 14896 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:35.399293  5561 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
W20260709 12:28:35.331508 14785 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.535s	user 0.015s	sys 0.000s
I20260709 12:28:35.347349  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:35.349699 14714 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:35.350438 14680 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:35.359730 14848 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:35.359462 14755 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:35.365556 14931 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:35.374203 14868 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:35.294973  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:35.411163  5631 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:35.405459 14785 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:35.401963  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:35.401338  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:35.416553 14848 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:35.416641 14835 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:35.417474 14848 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.418362 14835 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:35.372258 14883 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:35.372479 14409 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.420194  6274 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:35.400949 14712 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:35.371982 14554 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.419860 14835 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.408202 14801 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:35.408807 14680 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.433706 14811 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:35.434171 14811 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:35.435066 14811 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.436148 14811 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:35.470927  5634 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:28:35.474606  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:35.475226  5634 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:35.476132  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:35.479491  5631 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:35.480512  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1612) took 1333 ms. Trace:
I20260709 12:28:35.485589  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:35.486414  5702 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
W20260709 12:28:35.487963  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 298934us
I20260709 12:28:35.490990 14763 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.499604  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:35.501361  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:35.503105  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:35.517316  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:35.518234  5556 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:35.518345  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:35.470295 14714 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:35.481012  6276 rpcz_store.cc:276] 0709 12:28:34.147102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:34.147275 (+   173us) service_pool.cc:224] Handling call
0709 12:28:34.169446 (+ 22171us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:34.169665 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:28:34.474223 (+304558us) log.cc:844] Serialized 14 byte log entry
0709 12:28:35.480041 (+1005818us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b1800258220. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287
0709 12:28:35.480138 (+    97us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:35.480187 (+    49us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:35.480231 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:35.480294 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:35.480443 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6545,"prepare.queue_time_us":478412,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":120111,"raft.run_wall_time_us":12877,"spinlock_wait_cycles":32720000,"thread_start_us":551930,"threads_started":3,"wal-append.queue_time_us":305033}
I20260709 12:28:35.528111  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:35.523231  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:35.473512 14755 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:35.527607  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:35.532156  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:35.476094 14914 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:35.536901  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:35.537815  5556 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:28:35.488812 14785 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:35.543623  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:35.544792  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:35.545535  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:35.547987  5559 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:35.501441 14554 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:35.556696  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:35.557344  5556 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:35.550767  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:35.535168 14869 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:35.554205  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:35.524183 14680 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:35.534586 14804 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:35.592093  5556 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:28:35.540899 14864 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:35.543807 14931 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:35.527767 14439 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:35.619197  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 50073.2us
I20260709 12:28:35.532964 14896 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.081s
I20260709 12:28:35.623920  5556 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:28:35.575186 14593 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:35.577466 14765 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:35.637135  5553 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:35.637511  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:35.640650  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:35.540726 14714 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.546423 14755 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.646621  5554 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:28:35.649415  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:35.629714 14796 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.568s	user 0.013s	sys 0.002s
I20260709 12:28:35.650115  6278 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:28:35.549062 14944 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.095s
I20260709 12:28:35.653489  5557 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:35.655674  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:35.657215  6276 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 29: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:28:35.654960  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 123321us
I20260709 12:28:35.658999  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:35.660290  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:35.660759 14764 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:35.575627 14869 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.696511  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:35.652413 14554 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:28:35.655856 14785 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:35.637651 14409 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:35.701787 14764 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.634088 14864 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.717396 14764 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:35.707139 14800 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.538s	user 0.011s	sys 0.011s
I20260709 12:28:35.718811 14800 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.721676  5636 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:28:35.726323 14936 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:28:35.728511  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:35.728847 14755 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:35.642613 14593 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.730902 14593 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:35.730723 14800 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:35.731658  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:35.698640 14525 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:28:35.630491 14804 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.653409 14439 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.704707 14779 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:35.732434  5552 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:35.741376  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:35.764806  5554 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:35.670367 14848 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:35.739099 14804 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:35.747676  5702 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:35.741014 14763 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:35.732326 14942 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260709 12:28:35.770845  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:35.658509 14796 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:35.790925  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:35.747326 14439 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:35.704550 14785 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:35.732874 14525 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:35.796799  5710 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:28:35.793926  6276 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: []
W20260709 12:28:35.798262  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 139403us
W20260709 12:28:35.799919  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:35.771246 14714 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:35.795517 14940 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:28:35.811537 14714 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.007s
I20260709 12:28:35.818753 14714 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:35.819139 14714 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:35.820336 14714 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:35.822261 14714 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:35.824064 14869 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:35.824958 14447 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:35.825592 14864 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:35.825532 14447 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:35.841912  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:35.842849  5556 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 27: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:35.843931  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:35.845075  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:35.846268  5552 raft_consensus.cc:2410] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 29 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:28:35.846932  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:35.847959  5553 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:35.796643 14785 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:35.854130  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45214.9us
I20260709 12:28:35.856252  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1615) took 1704 ms. Trace:
I20260709 12:28:35.857676  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:35.858146  5554 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:35.856724  6272 rpcz_store.cc:276] 0709 12:28:34.152160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:34.152288 (+   128us) service_pool.cc:224] Handling call
0709 12:28:34.167834 (+ 15546us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:34.463431 (+295597us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:35.856010 (+1392579us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54002922d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:35.856193 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":678002944}
I20260709 12:28:35.861279  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:35.800969 14525 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:28:35.862129  5552 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:35.857553  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:35.867585 14802 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:35.700630  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:35.870623 14779 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:35.871346  5451 catalog_manager.cc:5697] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 17 to 20, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 20 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:35.877121  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:35.880887 14796 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:35.908726  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:35.917166  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59691.8us
I20260709 12:28:35.981297  6272 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:35.892508 14869 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.086s
I20260709 12:28:35.981732  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:35.918298  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:35.982656  6272 raft_consensus.cc:606] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:35.982811  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 64675.1us
I20260709 12:28:35.985477  5451 catalog_manager.cc:5697] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:35.991844  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:36.006304  5451 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:28:36.015800  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:35.996899 14868 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.586s	user 0.009s	sys 0.020s
I20260709 12:28:35.983271 14439 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:28:36.022758  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:35.995915 14882 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:36.026934  5704 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:28:36.028491  5451 catalog_manager.cc:5697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 20 to 25. New cstate: current_term: 25 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:36.033093  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3745) took 1151 ms. Trace:
I20260709 12:28:36.000998 14864 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:28:36.053988  5704 rpcz_store.cc:276] 0709 12:28:34.881313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:35.998358 (+1117045us) service_pool.cc:224] Handling call
0709 12:28:36.026428 (+ 28070us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.032667 (+  6239us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:36.033005 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:35.980513 14802 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:36.018536 14796 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:28:36.029282 14439 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:35.987174 14779 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:36.037156 14882 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:36.084007  5451 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 20 to 21, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 21 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:36.051559 14800 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:36.103449  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:36.149966  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 147014us
W20260709 12:28:36.087998 14801 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.663s	user 0.010s	sys 0.009s
I20260709 12:28:36.155572  5704 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 12:28:36.158413  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3746) took 1276 ms. Trace:
I20260709 12:28:36.159032  5704 rpcz_store.cc:276] 0709 12:28:34.881528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.084154 (+1202626us) service_pool.cc:224] Handling call
0709 12:28:36.155215 (+ 71061us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.158135 (+  2920us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:36.158356 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:36.092988 14864 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:36.083910 14796 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:36.160591  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3739) took 1280 ms. Trace:
I20260709 12:28:36.160171  5451 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 19 to 23, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 23 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:36.156051 14800 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:36.086588 14439 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:36.083914 14928 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.943s	user 0.013s	sys 0.000s
I20260709 12:28:36.157655 14894 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:36.171674  5707 rpcz_store.cc:276] 0709 12:28:34.879749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:34.966499 (+ 86750us) service_pool.cc:224] Handling call
0709 12:28:34.966821 (+   322us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:34.967200 (+   379us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:34.967233 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:34.967810 (+   577us) log.cc:844] Serialized 52 byte log entry
0709 12:28:35.873771 (+905961us) spinlock_profiling.cc:243] Waited 1.79 ms on lock 0x7b1800338980. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1
0709 12:28:35.873953 (+   182us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:36.149214 (+275261us) log.cc:844] Serialized 14 byte log entry
0709 12:28:36.150039 (+   825us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:36.150139 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:36.150185 (+    46us) raft_consensus.cc:1685] finished
0709 12:28:36.160223 (+ 10038us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:36.160509 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":290505,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":31217,"raft.run_cpu_time_us":249,"raft.run_wall_time_us":19573,"spinlock_wait_cycles":4109056,"thread_start_us":268911,"threads_started":2,"wal-append.queue_time_us":118961}
W20260709 12:28:36.150456 14948 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.363s	user 0.000s	sys 0.002s
I20260709 12:28:36.090404 14882 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:36.071612 14955 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.111s
W20260709 12:28:36.206342  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55597.6us
I20260709 12:28:36.207427  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:36.189914 14928 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.049s	user 0.013s	sys 0.003s
I20260709 12:28:36.171706 14796 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:28:36.213459  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:36.209882  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:36.192281 14936 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:28:36.157300 14880 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:36.224740 14800 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:36.229009  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:36.229816  5556 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
W20260709 12:28:36.230545 14956 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 12:28:36.230788  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:36.232578 14864 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 29.12, Last appended by leader: 12, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:36.245721  5561 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 12. (term mismatch)
I20260709 12:28:36.304214  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:36.241235  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 167175us
I20260709 12:28:36.382099  5561 pending_rounds.cc:85] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 11
I20260709 12:28:36.266077  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1621) took 1111 ms. Trace:
I20260709 12:28:36.385195  5561 pending_rounds.cc:107] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 23.12
I20260709 12:28:36.253607  5451 catalog_manager.cc:5697] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:28:36.257534  5635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3243) took 1302 ms (client timeout 1599 ms). Trace:
I20260709 12:28:36.255643  5554 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:28:36.291141  5559 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:28:36.281643  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:36.386914  5561 raft_consensus.cc:2889] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:36.261269  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3730) took 2404 ms. Trace:
W20260709 12:28:36.237957 14939 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.847s	user 0.013s	sys 0.001s
I20260709 12:28:36.258862  5627 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:36.392546 14647 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:36.393316  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:36.393172 14939 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:36.393982  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:36.386545 14896 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:36.395166 14896 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:36.392650  5627 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:36.396556  5627 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:36.390612  5708 rpcz_store.cc:276] 0709 12:28:33.857124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:33.858752 (+  1628us) service_pool.cc:224] Handling call
0709 12:28:33.859079 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:34.825787 (+966708us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:36.260859 (+1435072us) spinlock_profiling.cc:243] Waited 966 ms on lock 0x7b5400451fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:36.261162 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2221191552}
W20260709 12:28:36.399748  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:36.399703  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:36.400264  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:36.400799  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:36.295279 14917 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.239s	user 0.001s	sys 0.000s
I20260709 12:28:36.397645  5627 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:28:36.236330 14928 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:36.243975  6277 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:36.384866  6280 rpcz_store.cc:276] 0709 12:28:35.154906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:35.191970 (+ 37064us) service_pool.cc:224] Handling call
0709 12:28:35.192312 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:35.288784 (+ 96472us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:35.288815 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:35.289293 (+   478us) log.cc:844] Serialized 26 byte log entry
0709 12:28:35.404173 (+114880us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:35.404225 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:36.265571 (+861346us) spinlock_profiling.cc:243] Waited 96.1 ms on lock 0x7b54004c7a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:36.265635 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:36.265696 (+    61us) raft_consensus.cc:1685] finished
0709 12:28:36.265714 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:36.265990 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":251,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":2701,"spinlock_wait_cycles":220976384,"thread_start_us":63471,"threads_started":1,"wal-append.queue_time_us":241179}
W20260709 12:28:36.390417  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 108874us
I20260709 12:28:36.413969  5636 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30. Ops: [29.18-29.18]
W20260709 12:28:36.262984 14947 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.531s	user 0.005s	sys 0.004s
W20260709 12:28:36.391582  5635 rpcz_store.cc:269] 0709 12:28:34.954798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:34.955221 (+   423us) service_pool.cc:224] Handling call
0709 12:28:36.257301 (+1302080us) spinlock_profiling.cc:243] Waited 396 ms on lock 0x7b540018d654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:36.257455 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":911064320}
W20260709 12:28:36.381549  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:36.410662  5633 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:36.410825  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.236793 14804 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:36.407094  6277 raft_consensus.cc:606] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:36.417070  6277 raft_consensus.cc:614] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:28:36.261691 14899 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.648s	user 0.004s	sys 0.008s
I20260709 12:28:36.409448  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:36.397859 14896 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:36.417995  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:36.411846  5451 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 13 to 15, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 15 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:36.424641  5633 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:36.411322  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:36.412845  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:36.292214 14447 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:36.420874  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:36.261938 14802 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:36.420125  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:36.352267 14958 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:36.266077 14811 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.787s	user 0.012s	sys 0.000s
I20260709 12:28:36.423418 14647 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:36.423004  5632 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 27: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:36.410416  5634 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:28:36.292136 14779 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:36.421171  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:36.462190  6277 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:36.462244  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:36.285621  5450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 278) took 90231 ms (client timeout 15000 ms). Trace:
I20260709 12:28:36.474193  6273 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8c6474db7a70405694509f6976d850ce"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:36.475759  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1492) took 1137 ms. Trace:
I20260709 12:28:36.476074  6273 raft_consensus.cc:606] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:36.476989  6273 raft_consensus.cc:614] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:36.402900 14939 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:36.405037 14405 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:36.436021  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:36.423923 14804 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:36.443499  5627 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:28:36.490200  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3236) took 1567 ms. Trace:
I20260709 12:28:36.237048 14880 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:36.446631  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:36.446924  5633 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:36.503160  5558 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:36.503778  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.504616  5633 raft_consensus.cc:2368] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27.
I20260709 12:28:36.407608 14800 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:36.449585  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:36.422753  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:36.412421 14801 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:36.462934  6278 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:36.511797  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.511966  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:36.512722  5633 raft_consensus.cc:2410] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 27 because replica has last-logged OpId of term: 26 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:28:36.467353  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:36.470896 14882 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:36.471691 14647 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:36.482944  6277 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:36.463781  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.476285  6271 rpcz_store.cc:276] 0709 12:28:35.338407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:35.369650 (+ 31243us) service_pool.cc:224] Handling call
0709 12:28:35.398833 (+ 29183us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:35.399171 (+   338us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:28:35.399467 (+   296us) log.cc:844] Serialized 14 byte log entry
0709 12:28:35.575528 (+176061us) log.cc:844] Serialized 14 byte log entry
0709 12:28:35.616043 (+ 40515us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:35.616099 (+    56us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:28:35.616144 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:35.616219 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:36.226845 (+610626us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b180035e480. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:28:36.336228 (+109383us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b180035e480. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4
0709 12:28:36.475522 (+139294us) spinlock_profiling.cc:243] Waited 28.8 ms on lock 0x7b54002909d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:36.475685 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3033,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":1951,"raft.queue_time_us":1994,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":40567,"spinlock_wait_cycles":76652416,"thread_start_us":173321,"threads_started":1,"wal-append.queue_time_us":244336}
I20260709 12:28:36.507184  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:36.490688  5627 rpcz_store.cc:276] 0709 12:28:34.922517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:34.929757 (+  7240us) service_pool.cc:224] Handling call
0709 12:28:36.258487 (+1328730us) spinlock_profiling.cc:243] Waited 99.5 ms on lock 0x7b54003901a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa675e18 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:36.490117 (+231630us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":228840832}
I20260709 12:28:36.519743  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.519973  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3246) took 1048 ms. Trace:
I20260709 12:28:36.521307  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.510370  5628 raft_consensus.cc:2410] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 26 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:28:36.460270  6279 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:36.515847  5634 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:36.448112 14447 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:28:36.520500  5630 rpcz_store.cc:276] 0709 12:28:35.471331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:35.471465 (+   134us) service_pool.cc:224] Handling call
0709 12:28:35.473500 (+  2035us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:35.473733 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:35.473761 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:35.474253 (+   492us) log.cc:844] Serialized 26 byte log entry
0709 12:28:35.564096 (+ 89843us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:35.564139 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:35.564213 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:36.262341 (+698128us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:36.501538 (+239197us) raft_consensus.cc:1685] finished
0709 12:28:36.501561 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:36.519906 (+ 18345us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":42018560,"thread_start_us":89523,"threads_started":1,"wal-append.queue_time_us":257125}
I20260709 12:28:36.522553  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.578351  5632 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:36.522800  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.578677  5627 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:36.579874  5627 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:36.580212  5633 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:36.509932  6273 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 30. Ops: []
W20260709 12:28:36.527570  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 286589us
I20260709 12:28:36.581948  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:36.530224  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3749) took 1637 ms. Trace:
W20260709 12:28:36.574954  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50699.6us
I20260709 12:28:36.582810  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:36.582564  5707 rpcz_store.cc:276] 0709 12:28:34.892854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.189781 (+1296927us) service_pool.cc:224] Handling call
0709 12:28:36.212405 (+ 22624us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.212729 (+   324us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:28:36.222071 (+  9342us) log.cc:844] Serialized 14 byte log entry
0709 12:28:36.529775 (+307704us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:36.529853 (+    78us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:36.529931 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:36.530026 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:36.530154 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":72985,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":13402,"thread_start_us":293926,"threads_started":1}
I20260709 12:28:36.586486  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:36.575516  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.504317 14804 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:36.577098  5634 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 19. (term mismatch)
I20260709 12:28:36.588579  5634 pending_rounds.cc:85] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 18
I20260709 12:28:36.588894  5634 pending_rounds.cc:107] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 25.19
I20260709 12:28:36.589257  5634 raft_consensus.cc:2889] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:36.528784  5636 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:36.584072  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:36.600807  6277 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e69273530292453d87614627debcc13e"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:36.601459  6277 raft_consensus.cc:606] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:28:36.601991  6277 raft_consensus.cc:614] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:36.617326 14968 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:36.621109  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:36.622416  5552 raft_consensus.cc:2410] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 27 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 13.
I20260709 12:28:36.621282  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:36.623195  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:36.623885  5553 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:36.514773 14802 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:36.456394 14958 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:36.624179  5557 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:28:36.584978 14939 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:36.515589 14800 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:36.622283  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:36.588018 14750 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:36.584439 14805 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:36.602391 14949 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:36.645494  6278 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:36.619341 14894 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:36.525640 14801 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:36.662431  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3258) took 1976 ms. Trace:
I20260709 12:28:36.664124  5560 rpcz_store.cc:276] 0709 12:28:34.686516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:34.686637 (+   121us) service_pool.cc:224] Handling call
0709 12:28:35.059878 (+373241us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:35.060425 (+   547us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:35.060476 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:35.100002 (+ 39526us) log.cc:844] Serialized 26 byte log entry
0709 12:28:35.155733 (+ 55731us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:35.187001 (+ 31268us) log.cc:844] Serialized 14 byte log entry
0709 12:28:35.570827 (+383826us) spinlock_profiling.cc:243] Waited 2.5 ms on lock 0x7b18003a51c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:28:36.049855 (+479028us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:36.049967 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:36.050034 (+    67us) raft_consensus.cc:1685] finished
0709 12:28:36.082383 (+ 32349us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:36.662252 (+579869us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b54003a98d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:36.662370 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":28283,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":191713,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":22673,"spinlock_wait_cycles":709839488,"thread_start_us":505041,"threads_started":2,"wal-append.queue_time_us":152606,"wal-append.run_cpu_time_us":3423,"wal-append.run_wall_time_us":1010998}
I20260709 12:28:36.669335  5556 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:36.603602 14928 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.053s
I20260709 12:28:36.642591 14943 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:36.675230  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:36.672793  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:36.602579 14955 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.059s
W20260709 12:28:36.676144  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 227453us
I20260709 12:28:36.651947 14647 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:36.653199 14750 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:36.655730 14949 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:36.707036  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3323) took 2668 ms. Trace:
W20260709 12:28:36.681514  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 97534.8us
I20260709 12:28:36.704303  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:36.656121 14835 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:36.704773  6278 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f4f86de9f88949219896e4c020485a11"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:36.707881  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:36.711216  6278 raft_consensus.cc:606] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:36.711663  6278 raft_consensus.cc:614] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:36.715284  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.707520  5705 rpcz_store.cc:276] 0709 12:28:34.038536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:34.044820 (+  6284us) service_pool.cc:224] Handling call
0709 12:28:34.045596 (+   776us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:34.058299 (+ 12703us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:28:34.134737 (+ 76438us) log.cc:844] Serialized 14 byte log entry
0709 12:28:34.420922 (+286185us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:34.429917 (+  8995us) log.cc:844] Serialized 26 byte log entry
0709 12:28:34.430061 (+   144us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:34.455115 (+ 25054us) log.cc:844] Serialized 14 byte log entry
0709 12:28:34.464549 (+  9434us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:34.464640 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:36.260978 (+1796338us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:36.706781 (+445803us) raft_consensus.cc:1685] finished
0709 12:28:36.706812 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:36.706975 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38796,"prepare.queue_time_us":110470,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":85,"raft.queue_time_us":21221,"raft.run_cpu_time_us":579,"raft.run_wall_time_us":99914,"spinlock_wait_cycles":38754304,"thread_start_us":200394,"threads_started":1,"wal-append.queue_time_us":303606}
I20260709 12:28:36.718519  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:36.719691  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:36.717097  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:36.720976  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:36.722884  5705 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:36.723665  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:36.646010 14802 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:36.658197 14805 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:36.653508 14939 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:36.731935  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.732928  5634 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:36.725790  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:36.769114 14883 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:36.680919 14894 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:36.787529  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:36.771938  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1616) took 2581 ms. Trace:
I20260709 12:28:36.774560  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:36.706965 14750 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:36.714649 14949 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:36.721832 14647 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:36.789158  6274 rpcz_store.cc:276] 0709 12:28:34.152570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:34.152716 (+   146us) service_pool.cc:224] Handling call
0709 12:28:34.408963 (+256247us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:35.420755 (+1011792us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:36.733809 (+1313054us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400055854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:36.734022 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2325132672}
I20260709 12:28:36.794740  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.797780  5558 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 27 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260709 12:28:36.811025  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:36.812661  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:36.813234  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 307453us
W20260709 12:28:36.815550  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:36.817426  5707 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:28:36.821310  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:36.823701  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:36.679221 14943 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:36.822222  5634 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:36.835065  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.836270  5636 raft_consensus.cc:2410] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 23 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 12:28:36.840173  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:36.841702  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:36.795264 14974 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:36.796679 14978 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:36.854705  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.855085  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3752) took 1683 ms. Trace:
W20260709 12:28:36.831517 14439 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.621s	user 0.008s	sys 0.003s
I20260709 12:28:36.863653  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.863864  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:36.818645 14983 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:36.725951 14835 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:36.864565  5706 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:36.881948  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:36.883744  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3754) took 1526 ms. Trace:
I20260709 12:28:36.884146  5706 rpcz_store.cc:276] 0709 12:28:35.357107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.882959 (+1525852us) service_pool.cc:224] Handling call
0709 12:28:36.883250 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.883429 (+   179us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:36.883461 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:36.883507 (+    46us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:36.883543 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:36.883599 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:36.883690 (+    91us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:36.888677  5706 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 30. Ops: []
I20260709 12:28:36.858156  5708 rpcz_store.cc:276] 0709 12:28:35.171372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.400339 (+1228967us) service_pool.cc:224] Handling call
0709 12:28:36.401050 (+   711us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.401309 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:28:36.402583 (+  1274us) log.cc:844] Serialized 14 byte log entry
0709 12:28:36.854746 (+452163us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:36.854805 (+    59us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:36.854848 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:36.854913 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:36.855028 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4365,"prepare.queue_time_us":949,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":9990,"raft.queue_time_us":58277,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":11739,"spinlock_wait_cycles":17574656,"thread_start_us":447263,"threads_started":2,"wal-append.queue_time_us":395258}
W20260709 12:28:36.875274  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 157699us
I20260709 12:28:36.876154  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.792972 14883 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:36.891252  5636 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:36.890573  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3757) took 1418 ms. Trace:
I20260709 12:28:36.892190  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.854818 14974 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:36.856001 14978 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:36.898351  5630 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:36.900156  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:36.892207  5706 rpcz_store.cc:276] 0709 12:28:35.471950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.888005 (+1416055us) service_pool.cc:224] Handling call
0709 12:28:36.888440 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.890346 (+  1906us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:36.890519 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:36.907279  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:36.909034  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:36.911324  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.854419 14714 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 12:28:36.912081  5634 raft_consensus.cc:2368] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27.
I20260709 12:28:36.924703  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:36.894920 14882 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:36.914435  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:36.928815  5634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:28:36.931586  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:36.897871 14974 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:36.934089  5634 raft_consensus.cc:606] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:28:36.934675  5634 raft_consensus.cc:614] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:36.917522  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:36.869899 14439 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:36.916693 14864 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:28:37.031603  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:36.931731 14647 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:36.885475 14983 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:36.929368 14980 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:37.095265 14802 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:37.098714  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:37.051613  5701 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:37.023659 14958 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:37.037353 14801 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:36.909755 14894 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:37.109673 14864 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:28:37.285355 14796 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.063s	user 0.012s	sys 0.000s
I20260709 12:28:37.285832 14796 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:37.286442 14525 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.309s	user 0.008s	sys 0.004s
I20260709 12:28:37.286865 14525 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:37.288601 14939 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:37.288944 14525 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:37.289855 14835 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:37.290271 14439 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:37.290583 14835 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:37.256630 14928 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:37.291090 14439 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:37.291726 14928 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:37.292277 14928 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:37.293124 14647 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:37.195860 14800 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:37.249830 14805 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:37.106743 14978 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:37.291105 14835 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:37.298615 14835 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:37.306664 14983 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:37.242816 14980 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:37.288079 14796 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:37.299897 14883 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:37.333570 14785 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.437s	user 0.012s	sys 0.004s
W20260709 12:28:37.374212 14984 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.288s	user 0.000s	sys 0.000s
I20260709 12:28:37.260387 14864 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:37.291911 14439 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:37.411401 14973 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.524s	user 0.001s	sys 0.000s
W20260709 12:28:37.394373 14971 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.366s	user 0.001s	sys 0.000s
W20260709 12:28:37.394551 14977 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.452s	user 0.001s	sys 0.000s
I20260709 12:28:37.413972 14864 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:37.394538 14995 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:28:37.418838 14943 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:37.392302 14835 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:37.441385 14835 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:37.442799 14974 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:37.393332 14980 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:37.432406 14995 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:37.453349 14995 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:37.479228 14928 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:37.483032 14974 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:37.494431 14928 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:37.499905 14949 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.683s	user 0.011s	sys 0.001s
I20260709 12:28:37.500655 14949 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:37.720655 14903 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:37.722239 14880 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:36.918598  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:37.561335  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:37.723474  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:37.562894  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:37.567087  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:28:37.568809  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:37.586249  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:37.588788  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:37.593286  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3494) took 1075 ms. Trace:
W20260709 12:28:37.597321  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.005s	sys 0.001s
I20260709 12:28:37.598191  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1864) took 1186 ms. Trace:
I20260709 12:28:37.605490  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 1872) took 1159 ms. Trace:
I20260709 12:28:37.606160  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:37.611462  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3748) took 2722 ms. Trace:
I20260709 12:28:37.726904  5635 rpcz_store.cc:276] 0709 12:28:36.411188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.415268 (+  4080us) service_pool.cc:224] Handling call
0709 12:28:36.433249 (+ 17981us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.705628 (+272379us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:36.705661 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:36.705708 (+    47us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:36.705742 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:37.597461 (+891719us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b5400241154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:37.597558 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:37.597939 (+   381us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":625938176}
I20260709 12:28:37.727866  5704 rpcz_store.cc:276] 0709 12:28:34.888726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.172062 (+1283336us) service_pool.cc:224] Handling call
0709 12:28:36.207180 (+ 35118us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:36.207771 (+   591us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:36.207831 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:36.208544 (+   713us) log.cc:844] Serialized 26 byte log entry
0709 12:28:36.527107 (+318563us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:36.527142 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:36.527209 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:37.610760 (+1083551us) raft_consensus.cc:1685] finished
0709 12:28:37.610803 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:37.611163 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":282027,"threads_started":1,"wal-append.queue_time_us":430664}
I20260709 12:28:37.729377 14933 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:37.730357  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:37.726958  5632 rpcz_store.cc:276] 0709 12:28:36.445945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.463521 (+ 17576us) service_pool.cc:224] Handling call
0709 12:28:37.604967 (+1141446us) spinlock_profiling.cc:243] Waited 61.4 ms on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:37.605242 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":141290240}
W20260709 12:28:37.612823  5706 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.003s	sys 0.006s
I20260709 12:28:37.614084  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3744) took 2732 ms. Trace:
I20260709 12:28:37.732254  5710 rpcz_store.cc:276] 0709 12:28:34.881061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:35.279826 (+398765us) service_pool.cc:224] Handling call
0709 12:28:35.280213 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:35.797200 (+516987us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:37.613402 (+1816202us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b540006f754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:37.613839 (+   437us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1187599616}
W20260709 12:28:37.614727  6276 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.888s	user 0.000s	sys 0.010s
W20260709 12:28:37.625587  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3758) took 2147 ms (client timeout 2219 ms). Trace:
I20260709 12:28:37.734426  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:37.734717  5702 rpcz_store.cc:269] 0709 12:28:35.478081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:35.535698 (+ 57617us) service_pool.cc:224] Handling call
0709 12:28:37.624956 (+2089258us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54002d23d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:37.625306 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":421984000}
I20260709 12:28:37.737277  5702 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:37.731252  5704 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
W20260709 12:28:37.656474  5557 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.932s	user 0.002s	sys 0.011s
W20260709 12:28:37.423915 14868 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.000s	sys 0.010s
W20260709 12:28:37.739239  5557 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.025s	user 0.003s	sys 0.012s
I20260709 12:28:37.413494 14785 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:37.730391 14933 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:37.691339 14439 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:37.692070  5633 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.744s	user 0.002s	sys 0.007s
W20260709 12:28:37.740598  5633 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.822s	user 0.004s	sys 0.007s
I20260709 12:28:37.742022  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3501) took 1221 ms. Trace:
W20260709 12:28:37.423710 14680 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.772s	user 0.005s	sys 0.005s
W20260709 12:28:37.743194 14680 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.091s	user 0.006s	sys 0.006s
I20260709 12:28:37.696934  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:37.739295  5704 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: []
W20260709 12:28:37.698199  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.969s	user 0.004s	sys 0.020s
I20260709 12:28:37.700642 14943 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:37.702126  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3562) took 1048 ms. Trace:
W20260709 12:28:37.745541  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.034s	user 0.004s	sys 0.020s
W20260709 12:28:37.703142  5676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.767s	user 0.014s	sys 0.011s
W20260709 12:28:37.708774 14886 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 30 current term: 30 is higher.
W20260709 12:28:37.746281  5676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.810s	user 0.014s	sys 0.011s
I20260709 12:28:37.745976 14943 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:37.747102  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 829926us
W20260709 12:28:37.747557  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 1.03967e+06us
I20260709 12:28:37.699841  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:37.750540 14439 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:37.722512 14949 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:36.472612  5450 rpcz_store.cc:269] 0709 12:27:06.009490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:06.010327 (+   837us) service_pool.cc:224] Handling call
0709 12:28:36.239546 (+90229219us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:28:28.256841 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:28:28.257847 (+  1006us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:28:28.257911 (+    64us) write_op.cc:435] Acquired schema lock
0709 12:28:28.257943 (+    32us) tablet.cc:662] Decoding operations
0709 12:28:28.258532 (+   589us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:28:28.258655 (+   123us) write_op.cc:641] Partition lock acquired for write op
0709 12:28:28.258701 (+    46us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:28:28.259272 (+   571us) tablet.cc:701] Row locks acquired
0709 12:28:28.259479 (+   207us) write_op.cc:260] PREPARE: finished
0709 12:28:28.259810 (+   331us) write_op.cc:270] Start()
0709 12:28:28.260024 (+   214us) write_op.cc:276] Timestamp: P: 1783600108259685 usec, L: 0
0709 12:28:28.260087 (+    63us) op_driver.cc:348] REPLICATION: starting
0709 12:28:28.410350 (+150263us) log.cc:844] Serialized 527 byte log entry
0709 12:28:29.050576 (+640226us) op_driver.cc:464] REPLICATION: finished
0709 12:28:29.124547 (+ 73971us) write_op.cc:301] APPLY: starting
0709 12:28:29.124661 (+   114us) tablet.cc:1367] starting BulkCheckPresence
0709 12:28:29.127742 (+  3081us) tablet.cc:1370] finished BulkCheckPresence
0709 12:28:29.127924 (+   182us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:28:29.128956 (+  1032us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:28:29.129231 (+   275us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0709 12:28:29.129260 (+    29us) write_op.cc:312] APPLY: finished
0709 12:28:29.129493 (+   233us) log.cc:844] Serialized 25 byte log entry
0709 12:28:29.129698 (+   205us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:28:29.129782 (+    84us) write_op.cc:454] Released schema lock
0709 12:28:29.129838 (+    56us) write_op.cc:341] FINISH: Updating metrics
0709 12:28:29.146023 (+ 16185us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:28:29.149117 (+  3094us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1444314,"reported_tablets":49,"spinlock_wait_cycles":28544,"child_traces":[["op",{"apply.queue_time_us":73650,"apply.run_cpu_time_us":5872,"apply.run_wall_time_us":11557,"cfile_cache_hit":6,"cfile_cache_hit_bytes":8307,"cfile_init":1,"lbm_read_time_us":391,"lbm_reads_lt_1ms":4,"num_ops":1,"prepare.queue_time_us":610411,"prepare.run_cpu_time_us":59870,"prepare.run_wall_time_us":420026,"raft.queue_time_us":271324,"raft.run_cpu_time_us":18514,"raft.run_wall_time_us":82136,"replication_time_us":790286,"spinlock_wait_cycles":13696,"thread_start_us":792658,"threads_started":4,"wal-append.queue_time_us":328917,"wal-append.run_cpu_time_us":55661,"wal-append.run_wall_time_us":1370893}]]}
W20260709 12:28:37.739691 14868 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.969s	user 0.000s	sys 0.011s
I20260709 12:28:37.752671  5710 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
W20260709 12:28:37.753552  6276 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.029s	user 0.000s	sys 0.011s
I20260709 12:28:37.755625 14439 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:37.742447  5633 rpcz_store.cc:276] 0709 12:28:36.520137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.581247 (+ 61110us) service_pool.cc:224] Handling call
0709 12:28:36.581603 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.581840 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:28:36.917998 (+336158us) log.cc:844] Serialized 14 byte log entry
0709 12:28:37.741362 (+823364us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:37.741478 (+   116us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:37.741603 (+   125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:37.741727 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:37.741951 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":288386,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":3559,"raft.queue_time_us":1687,"raft.run_cpu_time_us":269,"raft.run_wall_time_us":271,"thread_start_us":1158551,"threads_started":2,"wal-append.queue_time_us":691133}
I20260709 12:28:37.742726 14785 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:37.756247 14439 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:37.749722 14943 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:37.723888  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 805173us
W20260709 12:28:37.709342  5553 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.057s	user 0.005s	sys 0.010s
I20260709 12:28:37.726610  5630 rpcz_store.cc:276] 0709 12:28:36.517507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.578557 (+ 61050us) service_pool.cc:224] Handling call
0709 12:28:36.587240 (+  8683us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.905973 (+318733us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:37.592407 (+686434us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b540032dcd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:37.592933 (+   526us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":715036544}
I20260709 12:28:37.742710  5634 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
W20260709 12:28:37.761719  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.006s	sys 0.001s
W20260709 12:28:37.763522  5560 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.034s	user 0.002s	sys 0.027s
W20260709 12:28:37.766964  5553 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.115s	user 0.006s	sys 0.011s
W20260709 12:28:37.769361  5560 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.040s	user 0.002s	sys 0.028s
I20260709 12:28:37.772735  5559 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:28:37.774650  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 860324us
W20260709 12:28:37.776564  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:37.779731 14886 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:37.769763 14880 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:37.780599 14785 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:37.781016 14886 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:37.781217 14880 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:37.781710 14785 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:37.776480 14983 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:37.784021 14886 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:37.788791  5556 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.773s	user 0.000s	sys 0.025s
W20260709 12:28:37.785900  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:37.776630  5702 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
W20260709 12:28:37.780006 14750 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.963s	user 0.015s	sys 0.005s
I20260709 12:28:37.797225  5675 leader_election.cc:400] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27.
I20260709 12:28:37.810137 14983 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:37.772781 14439 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:37.745695  5552 rpcz_store.cc:276] 0709 12:28:36.653665 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.653787 (+   122us) service_pool.cc:224] Handling call
0709 12:28:36.654148 (+   361us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:36.760037 (+105889us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:36.760070 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:36.792251 (+ 32181us) log.cc:844] Serialized 52 byte log entry
0709 12:28:37.626208 (+833957us) spinlock_profiling.cc:243] Waited 5.49 ms on lock 0x7b18000899e0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:28:37.700878 (+ 74670us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:37.701114 (+   236us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:37.701294 (+   180us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:37.701387 (+    93us) raft_consensus.cc:1685] finished
0709 12:28:37.701402 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:37.701854 (+   452us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":204340,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":25357,"spinlock_wait_cycles":12619008,"thread_start_us":156537,"threads_started":2,"wal-append.queue_time_us":825343}
W20260709 12:28:37.788623  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.846s	user 0.031s	sys 0.023s
I20260709 12:28:37.793975  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3561) took 1147 ms. Trace:
I20260709 12:28:37.784057 14880 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:37.819849  5556 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.909s	user 0.000s	sys 0.029s
I20260709 12:28:37.784106 14785 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:37.793332 14647 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:28:37.834484  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.917s	user 0.034s	sys 0.025s
I20260709 12:28:37.839726  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3564) took 1168 ms. Trace:
I20260709 12:28:37.801513 14933 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:37.789491  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:37.789034 14943 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:37.812127 14939 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:37.819258  5557 rpcz_store.cc:276] 0709 12:28:36.646468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.646669 (+   201us) service_pool.cc:224] Handling call
0709 12:28:36.646945 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.647236 (+   291us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:28:36.713764 (+ 66528us) log.cc:844] Serialized 14 byte log entry
0709 12:28:37.767762 (+1053998us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:37.771583 (+  3821us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:37.793583 (+ 22000us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:37.793717 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:37.793898 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29291,"prepare.queue_time_us":74158,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":19476,"raft.queue_time_us":31549,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":219,"thread_start_us":1078032,"threads_started":2,"wal-append.queue_time_us":942909}
I20260709 12:28:37.862048 14943 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:37.865525 14802 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:37.879907 14952 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.799s	user 0.015s	sys 0.010s
I20260709 12:28:37.878821  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1850) took 2001 ms. Trace:
I20260709 12:28:37.887457 14983 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:37.889178  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:37.889827 15016 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:37.883834  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:37.785614 15017 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:37.892318  5554 rpcz_store.cc:276] 0709 12:28:35.877089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:35.879237 (+  2148us) service_pool.cc:224] Handling call
0709 12:28:36.255158 (+375921us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.403145 (+147987us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:37.877869 (+1474724us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b54003a89d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:37.878298 (+   429us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":863826176}
I20260709 12:28:37.857873 14647 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:37.908442  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:37.911640  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3753) took 2586 ms. Trace:
I20260709 12:28:37.911804  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:37.912256 14929 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:37.912635 14802 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:37.913025 14929 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:37.915153 14802 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:37.841094 14750 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:37.916870  5707 rpcz_store.cc:276] 0709 12:28:35.325635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.586118 (+1260483us) service_pool.cc:224] Handling call
0709 12:28:36.612989 (+ 26871us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.818009 (+205020us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:37.911014 (+1093005us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b54001982d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:37.911431 (+   417us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":452256000}
I20260709 12:28:37.920661 14439 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:37.927685 14943 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:37.928613 15017 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:37.931118 15017 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:37.927794  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:37.925144  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:37.933990  5552 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:28:37.916206 14880 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:28:37.939626  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3547) took 2423 ms. Trace:
I20260709 12:28:37.940310  5555 rpcz_store.cc:276] 0709 12:28:35.516422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:35.521592 (+  5170us) service_pool.cc:224] Handling call
0709 12:28:35.521956 (+   364us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:35.542959 (+ 21003us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:35.542995 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:35.568583 (+ 25588us) log.cc:844] Serialized 52 byte log entry
0709 12:28:35.797892 (+229309us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:35.797998 (+   106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:35.798092 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:37.873166 (+2075074us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:37.932426 (+ 59260us) raft_consensus.cc:1685] finished
0709 12:28:37.932482 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:37.939367 (+  6885us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11538,"raft.queue_time_us":18577,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":17623,"spinlock_wait_cycles":294676352,"thread_start_us":157864,"threads_started":1,"wal-append.queue_time_us":357054}
I20260709 12:28:37.942468  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:37.943316  5555 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:37.945052  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61578us
I20260709 12:28:37.946218  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:37.946761  5559 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:37.915242 14929 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:37.956957  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 1862) took 1127 ms. Trace:
I20260709 12:28:37.957545  5559 rpcz_store.cc:276] 0709 12:28:36.820206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:37.567989 (+747783us) service_pool.cc:224] Handling call
0709 12:28:37.947417 (+379428us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:37.960543 14943 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:37.963667 14943 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:37.965209 14943 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:37.965283  5552 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:37.947556 14802 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:28:37.983465  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 186619us
I20260709 12:28:37.985432  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:37.987996  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:37.986491  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 210350us
I20260709 12:28:37.993072  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:38.003820  5708 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: []
I20260709 12:28:38.005262 14647 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.012303  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3560) took 1370 ms. Trace:
I20260709 12:28:38.013259  5553 rpcz_store.cc:276] 0709 12:28:36.641997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.643541 (+  1544us) service_pool.cc:224] Handling call
0709 12:28:36.643831 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.644019 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:28:36.652033 (+  8014us) log.cc:844] Serialized 14 byte log entry
0709 12:28:37.987373 (+1335340us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:38.002531 (+ 15158us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:38.011744 (+  9213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:38.011913 (+   169us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:38.012190 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":22496,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":8105,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":154,"thread_start_us":1308316,"threads_started":2,"wal-append.queue_time_us":1022619}
W20260709 12:28:38.015987 14804 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.390s	user 0.019s	sys 0.002s
W20260709 12:28:38.012537 14951 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.817s	user 0.002s	sys 0.019s
W20260709 12:28:38.016953  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 70973us
I20260709 12:28:38.016793 14804 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.018646  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:37.956794 14939 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.025725  5554 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:28:38.031469  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3498) took 1512 ms. Trace:
I20260709 12:28:38.034675 14939 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:38.036520  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:37.840286  5556 rpcz_store.cc:276] 0709 12:28:36.671100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.671221 (+   121us) service_pool.cc:224] Handling call
0709 12:28:36.675134 (+  3913us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.675326 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:28:36.910325 (+234999us) log.cc:844] Serialized 14 byte log entry
0709 12:28:37.839148 (+928823us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:37.839258 (+   110us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:37.839343 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:37.839454 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:37.839655 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":122072,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":3264,"spinlock_wait_cycles":91193600,"thread_start_us":1109605,"threads_started":2,"wal-append.queue_time_us":749755}
I20260709 12:28:38.037578  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.038602  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1863) took 1126 ms. Trace:
I20260709 12:28:38.038981  5556 rpcz_store.cc:276] 0709 12:28:36.912162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:38.037720 (+1125558us) service_pool.cc:224] Handling call
0709 12:28:38.038076 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:38.038266 (+   190us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:38.038293 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:38.038333 (+    40us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:38.038359 (+    26us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:38.038407 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:38.038506 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:38.040392  5552 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:38.042387  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3751) took 3057 ms. Trace:
I20260709 12:28:38.042800  5701 rpcz_store.cc:276] 0709 12:28:34.984983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.398344 (+1413361us) service_pool.cc:224] Handling call
0709 12:28:36.398700 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:37.148753 (+750053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:38.041875 (+893122us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b54002f5154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:38.042194 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1500823424}
I20260709 12:28:38.044795  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.046605  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.047509  5630 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:38.048573 14802 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:38.048599  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3565) took 1361 ms. Trace:
I20260709 12:28:38.049232 14802 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:38.049726 14802 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:38.049367  5560 rpcz_store.cc:276] 0709 12:28:36.687247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.687475 (+   228us) service_pool.cc:224] Handling call
0709 12:28:36.728809 (+ 41334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:37.859572 (+1130763us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:37.862574 (+  3002us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:37.863226 (+   652us) log.cc:844] Serialized 26 byte log entry
0709 12:28:37.979812 (+116586us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:37.980796 (+   984us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:37.984931 (+  4135us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:38.048294 (+ 63363us) raft_consensus.cc:1685] finished
0709 12:28:38.048317 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:38.048526 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":188855,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":8870,"thread_start_us":1156601,"threads_started":2,"wal-append.queue_time_us":182807}
I20260709 12:28:38.052148  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.053139  5707 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:38.053023  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:38.053689  5701 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:38.054463 14918 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:38.055244 14918 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.049237  5559 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:28:38.062543  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1480) took 1352 ms. Trace:
I20260709 12:28:37.992547 15016 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.067318  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3567) took 1174 ms. Trace:
I20260709 12:28:38.068859  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.067761  5561 rpcz_store.cc:276] 0709 12:28:36.892886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:37.609108 (+716222us) service_pool.cc:224] Handling call
0709 12:28:37.610122 (+  1014us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:37.627273 (+ 17151us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:28:37.696908 (+ 69635us) log.cc:844] Serialized 14 byte log entry
0709 12:28:37.891744 (+194836us) log.cc:844] Serialized 14 byte log entry
0709 12:28:37.891933 (+   189us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:37.959409 (+ 67476us) log.cc:844] Serialized 26 byte log entry
0709 12:28:37.959554 (+   145us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:28:37.959586 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:37.959652 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:38.066967 (+107315us) raft_consensus.cc:1685] finished
0709 12:28:38.066990 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:38.067127 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8815,"prepare.queue_time_us":19137,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":103,"raft.queue_time_us":63610,"raft.run_cpu_time_us":628,"raft.run_wall_time_us":12111,"thread_start_us":162881,"threads_started":1,"wal-append.queue_time_us":260597}
I20260709 12:28:38.070619  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.064862  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1611) took 1451 ms. Trace:
I20260709 12:28:38.072484 14841 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:38.073329  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:38.074082  5702 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:28:38.074637 14883 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:28:38.075070 14883 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:38.075294  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:38.076849  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:38.077536  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:38.072284  5706 rpcz_store.cc:276] 0709 12:28:36.613941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.913176 (+299235us) service_pool.cc:224] Handling call
0709 12:28:36.913496 (+   320us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:37.732206 (+818710us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:37.732251 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:37.733220 (+   969us) log.cc:844] Serialized 26 byte log entry
0709 12:28:37.896267 (+163047us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:37.896304 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:37.903870 (+  7566us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:38.064568 (+160698us) raft_consensus.cc:1685] finished
0709 12:28:38.064590 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:38.064798 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":189039,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":125798,"thread_start_us":980569,"threads_started":2,"wal-append.queue_time_us":294747}
I20260709 12:28:38.078636 14117 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:38.079173 14117 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.079439 14903 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:38.079630 14117 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:38.079901 14903 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.080382 14903 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:38.080619  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.069244 14617 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:38.037608  5553 raft_consensus.cc:2410] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 28 because replica has last-logged OpId of term: 26 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
I20260709 12:28:38.076344 14883 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.032066  5627 rpcz_store.cc:276] 0709 12:28:36.519005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.580552 (+ 61547us) service_pool.cc:224] Handling call
0709 12:28:37.605245 (+1024693us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:37.605860 (+   615us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:37.605900 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:37.605960 (+    60us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:37.606021 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:37.606118 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:38.031019 (+424901us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54001dddd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:38.031239 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2355645312}
I20260709 12:28:38.048629  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.085196  5710 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:38.085811  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.086099  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.086763  5709 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:38.072901  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.087647  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.088944  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.077566  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.093178 14983 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:38.095321  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:38.095773 14779 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:38.086110  5560 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 12:28:38.096306 14983 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:28:38.097548 14864 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.667s	user 0.010s	sys 0.003s
I20260709 12:28:38.098212  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.098526 14864 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.101267 14864 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.102911  5702 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:28:38.103026 14750 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.104861 14804 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:38.105069  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1599) took 1515 ms. Trace:
I20260709 12:28:38.105844  5706 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:38.067739 15016 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.107590 14918 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.079183  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:38.107831 14617 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.111881  5707 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:38.113204 14617 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:38.109241 14803 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:38.114424 14803 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.109891 14896 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.116671 14896 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.117865  5555 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
W20260709 12:28:38.118516  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 242582us
I20260709 12:28:38.125239  5559 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:38.110687 14882 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:38.124970 14943 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.128509  5629 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:38.132908 14882 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.133494 14599 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.134009 14599 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:28:38.129323  5524 leader_election.cc:336] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.564s (SENT)
I20260709 12:28:38.132079 14896 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:38.105538  5708 rpcz_store.cc:276] 0709 12:28:36.589216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.895659 (+306443us) service_pool.cc:224] Handling call
0709 12:28:36.895999 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:38.004517 (+1108518us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:38.104751 (+100234us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400197dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:38.105002 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2419788032}
I20260709 12:28:38.137502  5552 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:38.062999  6277 rpcz_store.cc:276] 0709 12:28:36.710105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.710238 (+   133us) service_pool.cc:224] Handling call
0709 12:28:36.710566 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:36.710795 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:28:36.837697 (+126902us) log.cc:844] Serialized 14 byte log entry
0709 12:28:36.899103 (+ 61406us) log.cc:844] Serialized 14 byte log entry
0709 12:28:38.061457 (+1162354us) spinlock_profiling.cc:243] Waited 8.32 ms on lock 0x7b1800391000. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:28:38.061773 (+   316us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:38.061885 (+   112us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:38.062001 (+   116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:38.062134 (+   133us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:38.062312 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4747,"prepare.queue_time_us":319469,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":106,"raft.queue_time_us":5423,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":8307,"spinlock_wait_cycles":19134848,"thread_start_us":168577,"threads_started":2,"wal-append.queue_time_us":358526}
I20260709 12:28:38.110037 14117 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.134644 14943 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.138710 14882 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:38.143867  5556 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 12. Preceding OpId from leader: term: 30 index: 13. (index mismatch)
I20260709 12:28:38.140537  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.154337  5554 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:28:38.140544 14983 raft_consensus.cc:2764] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:28:38.132910 14786 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:38.160885 14939 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:28:38.161135 14975 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:28:38.155447 14779 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.206286 14117 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:38.208860  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 110853us
I20260709 12:28:38.162009 14786 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.162750 14802 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.187901 14803 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:38.213078 14786 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:28:38.225843  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:38.226557  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:38.213627 14943 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:28:38.227614  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 148569us
I20260709 12:28:38.238878  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:38.244484  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 115221us
I20260709 12:28:38.249583  5701 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:38.245697 14980 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.266577  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:28:38.263386  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 192848us
I20260709 12:28:38.271090 14117 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:38.271514 14117 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.272653 14117 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:38.272867 14779 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:38.271466  5524 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.577s (SENT)
I20260709 12:28:38.252557 14943 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:38.282979 14117 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:38.282521  6279 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1631) took 2468 ms (client timeout 2413 ms). Trace:
W20260709 12:28:38.289400  6279 rpcz_store.cc:269] 0709 12:28:35.770336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:35.770522 (+   186us) service_pool.cc:224] Handling call
0709 12:28:38.221565 (+2451043us) spinlock_profiling.cc:243] Waited 674 ms on lock 0x7b54002a8d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:38.238894 (+ 17329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1549417344}
I20260709 12:28:38.292433 14835 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:38.293205 14835 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.295641 14835 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:38.296517 14787 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:38.299238  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:38.300640 14117 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.307999  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:38.308496  5524 leader_election.cc:336] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.614s (SENT)
I20260709 12:28:38.309017  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:38.310849  5524 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.695s (SENT)
I20260709 12:28:38.311358  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.297173 14787 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.317559 14787 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:38.317835  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:38.321524  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 97415.5us
W20260709 12:28:38.304637  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 1.33468e+06us
W20260709 12:28:38.330834 15014 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.650s	user 0.000s	sys 0.004s
I20260709 12:28:38.331619 14965 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:38.332310 14965 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:38.332590 15036 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:38.333559 14965 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:38.334349  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 83148.6us
I20260709 12:28:38.335584 14117 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:28:38.338322  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3556) took 1699 ms. Trace:
I20260709 12:28:38.338832  5558 rpcz_store.cc:276] 0709 12:28:36.619884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.621948 (+  2064us) service_pool.cc:224] Handling call
0709 12:28:38.283932 (+1661984us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b54000ea6d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:38.319383 (+ 35451us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":340769024}
I20260709 12:28:38.318344 15031 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:28:38.339279 14903 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.345743  5526 leader_election.cc:400] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27.
I20260709 12:28:38.346095  5526 leader_election.cc:403] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:28:38.347380 14903 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:38.291038 14980 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.348382 14883 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:38.349967  6236 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.350308 14980 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.306689 14802 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:38.335788  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:38.354507  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:38.352439  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1474) took 1946 ms. Trace:
I20260709 12:28:38.350853 15036 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:28:38.360445 14979 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.334s	user 0.008s	sys 0.034s
I20260709 12:28:38.370705 14982 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:38.371178 14887 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:38.368739 14802 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27.
I20260709 12:28:38.364794 15032 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.011s
W20260709 12:28:38.385118  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3225) took 3493 ms (client timeout 1793 ms). Trace:
I20260709 12:28:38.438412 15035 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:38.438871  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:38.438453  5629 rpcz_store.cc:269] 0709 12:28:34.891847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:34.891981 (+   134us) service_pool.cc:224] Handling call
0709 12:28:38.384813 (+3492832us) spinlock_profiling.cc:243] Waited 3.11 s on lock 0x7b540009e2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:38.384994 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7143132928}
I20260709 12:28:38.438992 15035 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:38.444504 14117 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:38.445310  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:38.445428 14117 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.385757  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:38.446377 15035 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:38.447147  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61490us
I20260709 12:28:38.447368 14617 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:38.448096 14617 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:38.448992 14117 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.449292 14617 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.448030  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:38.361658 15031 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:38.457070 15031 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:38.457273  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 96208.9us
I20260709 12:28:38.394974  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:38.458942  6279 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 27 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260709 12:28:38.395874 15036 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:38.396064  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:38.396796 14841 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.397298  6274 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:38.398135  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:38.400501 14980 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:38.461916  6275 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:28:38.404700  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54833.6us
I20260709 12:28:38.405212  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:38.406014  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:38.408281  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.403808 14903 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.406021 14978 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:38.411823 14779 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:38.416203 14786 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:38.420838  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 110018us
I20260709 12:28:38.426393 14976 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:38.432695  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:38.437911  6278 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:38.386945 15020 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:28:38.389700  5631 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
W20260709 12:28:38.392879  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47308.2us
W20260709 12:28:38.456493 14979 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.539s	user 0.008s	sys 0.035s
I20260709 12:28:38.458813  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.403784 14599 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:28:38.458312 14117 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.467780  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:38.474602  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:38.458155 15031 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.356144  6276 rpcz_store.cc:276] 0709 12:28:36.405479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:36.405619 (+   140us) service_pool.cc:224] Handling call
0709 12:28:36.407446 (+  1827us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:36.407938 (+   492us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:36.407980 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:36.413629 (+  5649us) log.cc:844] Serialized 26 byte log entry
0709 12:28:36.669109 (+255480us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:37.754192 (+1085083us) log.cc:844] Serialized 14 byte log entry
0709 12:28:38.290810 (+536618us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:38.302158 (+ 11348us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:38.323231 (+ 21073us) raft_consensus.cc:1685] finished
0709 12:28:38.323247 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:38.352210 (+ 28963us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":327449,"prepare.queue_time_us":188563,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":6589,"raft.queue_time_us":356003,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":12682,"thread_start_us":1433827,"threads_started":3,"wal-append.queue_time_us":283566}
I20260709 12:28:38.495813 14117 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.496421 14949 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:38.496537  6277 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:38.465415 14980 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.496628 14903 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.500944  5558 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:38.505097 14916 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:38.505290  6274 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:38.505658  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:38.504997 14949 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:38.505779  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:38.515533  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:38.510661 15020 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:38.516121 14949 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.516893 15020 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:38.517242  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.517918 15020 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.519174  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:38.516139 15032 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:38.505968 14841 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.523286 14841 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:38.506773 14976 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.524542 14982 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.525166 14982 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.525983 14976 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:38.508028  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 146391us
I20260709 12:28:38.472038 14779 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:38.527338  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68663.5us
I20260709 12:28:38.527750 14982 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:38.528337 14779 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.508466  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:38.529819  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 171276us
I20260709 12:28:38.529983 14117 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:38.506946  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:38.515137  5556 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:28:38.508675 14980 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:28:38.531311  5524 leader_election.cc:336] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.765s (SENT)
I20260709 12:28:38.528810  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:38.530100  6278 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:38.536511 14117 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 12:28:38.538825  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:38.541093  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.544114  5554 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:38.545526  5560 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:38.546864 15032 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:38.547153 14865 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:28:38.548087 15032 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.548905 14787 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:38.551900  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:38.547520 15035 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:38.566815  6275 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 20.
W20260709 12:28:38.580677  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:38.582147  6279 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:28:38.583225  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:38.547237 15034 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:28:38.556751 14882 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:38.497957 15036 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.593580  5558 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:38.597009  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.575948 14787 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.605762 14787 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:38.607218  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 103368us
I20260709 12:28:38.614144 14787 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.615540  5633 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: [26.18-26.18]
W20260709 12:28:38.618571  5675 connection.cc:256] RPC call timeout handler was delayed by -1.8210009990002618s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15990000000000001s for a response
W20260709 12:28:38.619036  5675 connection.cc:256] RPC call timeout handler was delayed by -1.6464009990004342s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17930000000000001s for a response
W20260709 12:28:38.619486  5675 connection.cc:256] RPC call timeout handler was delayed by -1.5501009990002785s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19s for a response
W20260709 12:28:38.619779  5675 connection.cc:256] RPC call timeout handler was delayed by -1.3107009990003462s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21660000000000001s for a response
W20260709 12:28:38.620071  5675 connection.cc:256] RPC call timeout handler was delayed by -1.2030133580001348s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1705s for a response
W20260709 12:28:38.620656  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 101640us
I20260709 12:28:38.621459  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.622344 14804 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:38.622931 14804 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.624519 14918 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.625062 14918 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.626982 14918 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:38.627738  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:38.628742  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.629390  5706 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:38.630384  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.634168 14692 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:28:38.634749 14692 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:38.645082 14982 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.645741 14982 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.648205 14982 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.657779 14903 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.582532 14887 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:38.582284 14803 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:38.591437 14937 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.591220 14802 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:38.675088  6272 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:38.668900  5707 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 27 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:28:38.681649  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:38.703934  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 179655us
I20260709 12:28:38.699913  5632 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 24: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:28:38.675252  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63479.8us
I20260709 12:28:38.701190  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.713630  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.664069 14995 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:38.746014  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:38.746191 14995 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.747578  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.720043 14811 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:38.720608 14937 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.721585  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:38.750797  5636 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:38.749315  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:38.722216  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.682473 14787 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.750350 14937 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:38.750362  5701 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:38.755722  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.727340  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:38.727543  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:38.728783  5524 leader_election.cc:336] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.853s (SENT)
I20260709 12:28:38.730257  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:38.730352  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.731788  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:38.732800  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.733865  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.734719 14779 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:38.772857  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44179us
I20260709 12:28:38.735982  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.635982 14692 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:38.776196  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.776785  6277 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:38.682711 14965 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
I20260709 12:28:38.742856  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.778762  5708 raft_consensus.cc:2410] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:28:38.737277  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.780978  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.781328  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:38.782140  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.782683  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.783205  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.783768  5709 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:38.745568 14786 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.784518  5633 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:38.777761  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:38.785446  5705 raft_consensus.cc:2376] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24. Re-sending same reply.
I20260709 12:28:38.786365  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:38.787747  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.779994  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.789232  5708 raft_consensus.cc:2368] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
I20260709 12:28:38.790410  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.791285  5708 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:38.724576  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:38.759562 15036 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:38.724460  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3251) took 3224 ms. Trace:
I20260709 12:28:38.793115  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:38.794057  5709 raft_consensus.cc:2368] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28.
W20260709 12:28:38.760617  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 202917us
I20260709 12:28:38.794235 14804 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.794909  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:38.795085  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.761662 14995 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.797302  5710 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:38.772605  6275 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 33: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:38.774094  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.798189  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.798659  5702 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:38.798841  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:38.799497  5636 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:38.799623  5561 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:28:38.772954  5703 raft_consensus.cc:2368] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24.
I20260709 12:28:38.711506  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.800331  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:38.801348  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:38.801523  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:38.802173  5627 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:38.802949  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:38.752341 14787 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.748334  5632 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:38.784298  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.793313  5631 rpcz_store.cc:276] 0709 12:28:35.486700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:35.486818 (+   118us) service_pool.cc:224] Handling call
0709 12:28:38.389097 (+2902279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:38.477232 (+ 88135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:38.690527 (+213295us) spinlock_profiling.cc:243] Waited 2.9 s on lock 0x7b540009e2d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:38.710618 (+ 20091us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6673790848}
I20260709 12:28:38.804774  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.805128  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:38.805541 14787 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:38.718096 14882 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:38.793478  6277 raft_consensus.cc:2410] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 26 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:28:38.806478 14882 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:38.806854  5633 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:38.807567 14882 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.811337 15036 raft_consensus.cc:2764] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:28:38.750721  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.799767  5702 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:38.812212  5702 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:38.812559  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:28:38.812798  5524 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.869s (SENT)
I20260709 12:28:38.792166  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.813367  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.813190  5702 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.803041  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:38.744104  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:38.814322  5702 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:38.814509  5704 raft_consensus.cc:2368] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
W20260709 12:28:38.814740  5524 leader_election.cc:336] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.890s (SENT)
I20260709 12:28:38.814949  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.793684  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.815817  5707 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:38.815915  5628 raft_consensus.cc:2368] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34.
W20260709 12:28:38.735216  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 113855us
I20260709 12:28:38.816010  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.816620 14887 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:28:38.804045  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.817397  5630 raft_consensus.cc:2410] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 28 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:28:38.798874  5634 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:38.817274  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.818009  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:38.797406  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:38.818538  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:38.818992  5705 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:38.819248  5554 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:38.796293  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.820259 14811 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:28:38.818614  5627 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:38.819772 14883 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:28:38.821151  5636 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:38.819866  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:38.813656  5708 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:38.831496  5635 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:38.831646  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.678298 14802 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:38.821337 14803 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:38.881500  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.796324  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.818688  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:38.825206  5556 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:38.788900 14980 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:38.886545  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 124749us
W20260709 12:28:38.827344  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:38.882323  5701 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:38.871202  5560 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:28:38.813218  5633 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:38.883280  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.888880  5555 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: []
I20260709 12:28:38.884793  5634 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:38.893293  5709 raft_consensus.cc:1813] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Handling vote request from an unknown peer cea66aa15be3423880843974acdbf50f
W20260709 12:28:38.899933  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:38.901027  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:38.903398  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 107212us
I20260709 12:28:38.905423  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:38.906409  5556 raft_consensus.cc:2410] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 23 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:28:38.900918  5709 raft_consensus.cc:2368] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21.
I20260709 12:28:38.908890  5556 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:28:38.881067 14802 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:28:38.909842  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 90064.3us
W20260709 12:28:38.889621 14935 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.034s	sys 0.079s
W20260709 12:28:38.910753  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 96046.6us
I20260709 12:28:38.913057  5631 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:38.883970 14803 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.886319 14943 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:38.922464  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:38.932140  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:38.932112  5702 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:28:38.935138  6272 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
W20260709 12:28:38.916522 14935 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.034s	sys 0.079s
I20260709 12:28:38.943992  5556 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 25 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:28:38.918948 14803 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:38.922964 15035 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:38.947430  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:38.952742  5709 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:28:38.953801  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:38.954648  5702 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 22.
I20260709 12:28:38.959265  5556 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:38.961501  5634 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:38.959266 15032 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.003s
I20260709 12:28:38.948297  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:38.975553  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.977631 14811 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:28:38.978121 14811 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:38.979127 14811 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:38.977950  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.981196  5631 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 30: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:38.988754 14811 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:38.949432 15035 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:38.989823 15035 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:38.994889  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.996085  5630 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 26: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:38.997226  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:38.998415  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:38.999610  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:38.999801 14804 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:39.000568 14980 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:39.000885 14804 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:38.998664  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:39.005813 14439 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.021s	sys 0.037s
I20260709 12:28:39.000514 14965 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:28:39.012473  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:39.014549  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:38.944944  5459 catalog_manager.cc:5697] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 17 to 19, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 19 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:39.037786  5635 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:39.039409  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:39.001955 14949 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:28:39.022151 14439 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.021s	sys 0.037s
I20260709 12:28:39.046777  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:39.048105  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 100786us
I20260709 12:28:39.011307 15032 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:39.021498 14617 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.010s	sys 0.009s
I20260709 12:28:39.052779 14894 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.032s
W20260709 12:28:39.053498 15046 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260709 12:28:39.035683 14883 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:39.080154  5559 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:28:39.070007  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:39.070623  5635 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27. Ops: [26.19-26.19]
W20260709 12:28:39.071689 14617 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.010s	sys 0.010s
I20260709 12:28:39.044667 14949 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:39.037629 14864 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.042s
I20260709 12:28:39.069016 14965 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:39.143582  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 58631.7us
I20260709 12:28:39.069346 14995 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:39.072818 14882 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:28:39.123304  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.024s	sys 0.019s
W20260709 12:28:39.148025  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.025s	sys 0.020s
W20260709 12:28:39.148723  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 618793us
I20260709 12:28:39.161657  5709 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:28:39.123888 14949 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:39.148196 14779 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:39.127641 15031 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.044s
W20260709 12:28:39.187559  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:39.188262  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:39.192934 15031 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:39.193441 15031 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:39.194453 15031 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:39.195770 15031 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:39.200572  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.173606  5459 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:39.146826 14883 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:39.207643  5702 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:39.127157 14804 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:39.212476  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.214756  5559 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:39.220227  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.167837 14692 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.043s
I20260709 12:28:39.228475  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.176190 14995 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:39.229136  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:39.217536 14803 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:39.245872 14883 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:39.246570 14803 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:39.246989  6272 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:39.251204  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:39.255923 15051 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:28:39.256775  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.257586  5557 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:39.258363  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.259665  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:39.263019  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.263479  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.263924  5557 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:39.267086  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:39.265120  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.233419 15035 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:39.267192 15057 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:28:39.268757 14865 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:39.270460 14865 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:39.270895 14865 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:39.270979 15035 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:39.271803 14787 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:28:39.272884  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:39.273305 14980 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:28:39.273499  5559 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:39.273811 14980 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:39.275004 14980 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:39.276511  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52560.9us
I20260709 12:28:39.276996 15057 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:28:39.277400 15057 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:39.278038  5558 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:39.279146  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:39.272096 14995 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:39.285035  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:39.296949  5633 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:39.296558  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:39.301139  6272 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:39.305025  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.306267  5560 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:39.312464  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.332413  5559 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:39.336201  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:39.289407 14835 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:39.339843  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.342674  6278 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:28:39.343333  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:39.280647 14802 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:39.278187 15057 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:39.346357  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60777.2us
I20260709 12:28:39.347007  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:39.348801  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:39.352382  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:39.357137  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 79820.2us
I20260709 12:28:39.361546 15074 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:39.372002  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:39.252398 14965 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:39.381814  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46881.9us
I20260709 12:28:39.382665  5557 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:39.387981  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:39.391469  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.393751  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:39.394420  6272 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:39.373565 14811 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:39.396063  5558 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:28:39.398336  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.400343  5559 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:28:39.400750  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:39.403801  5524 leader_election.cc:336] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.921s (SENT)
I20260709 12:28:39.404580  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:39.408775  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:39.418404  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:39.419080  5633 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 24
W20260709 12:28:39.402035 14978 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.929s	user 0.014s	sys 0.000s
I20260709 12:28:39.424929  5559 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: []
I20260709 12:28:39.426175  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:39.428505  5557 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:28:39.432659  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:39.433471  5561 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:28:39.426824  5635 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:28:39.435511  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:39.447181  5628 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:28:39.447626  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:39.422294 14811 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:39.458285  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 67176.8us
I20260709 12:28:39.458935  6279 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 19. Current term is 20. Ops: []
I20260709 12:28:39.459977  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:39.460896 14976 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:28:39.461230 14779 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:28:39.447122 15057 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:28:39.465729  5561 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:28:39.470397  6272 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:39.465353 14865 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:39.473593  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 74583.5us
I20260709 12:28:39.473948 14865 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:39.474731 14865 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:28:39.475589 14979 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:28:39.488143  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:39.477964 15065 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:28:39.432313 14939 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:28:39.484980  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 96422.5us
I20260709 12:28:39.490403  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:39.445662 14787 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:28:39.500387  5633 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:39.484087 14835 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:39.517198  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:39.449401 15035 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:39.448813 14903 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:39.524683  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:39.527000  5524 leader_election.cc:336] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.948s (SENT)
I20260709 12:28:39.527504  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:39.541908 15070 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.208s	user 0.002s	sys 0.000s
I20260709 12:28:39.548524 15063 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:39.550895 15063 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:39.551302 15063 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:39.518632 14935 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:39.552910 14935 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:39.553210  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:39.531157 15068 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:28:39.528270 14903 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:39.524339 15035 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:39.443938 14949 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:39.562856  5557 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:28:39.571986  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:39.572736  6272 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
W20260709 12:28:39.579926  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55866.9us
W20260709 12:28:39.581079  5524 leader_election.cc:336] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.965s (SENT)
I20260709 12:28:39.581460  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:39.581836  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:39.552111 15063 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:39.583509  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:39.584864  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:39.586122  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:39.587038  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:39.587852  5702 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 20: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:39.572204 14803 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:39.589273 14803 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:39.590288 15077 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260709 12:28:39.592135 15080 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260709 12:28:39.592902  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44633us
I20260709 12:28:39.593807  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:39.579236  6278 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:28:39.583559 15063 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:39.581596  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:39.594655  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:39.533756 14979 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:39.554670 14787 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:39.555091 14978 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:28:39.567981 15032 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.507s	user 0.008s	sys 0.003s
I20260709 12:28:39.609926 15068 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:39.647441 15032 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:39.612154 14803 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:39.568596 15035 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:39.612943  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:39.616652 14864 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:39.648653  6277 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:39.601444 14979 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:39.649335 14979 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:39.650995 14978 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:39.630371 15081 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 12:28:39.594453 14995 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
W20260709 12:28:39.638180  5601 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.858s (SENT) [suppressed 36 similar messages]
I20260709 12:28:39.640718 14802 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:39.606849 14811 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:39.611616  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:39.612949 14935 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:39.698472 15031 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.502s	user 0.014s	sys 0.001s
I20260709 12:28:39.699379  6275 raft_consensus.cc:2410] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 27 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:28:39.693555  5634 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:39.694722  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:39.695391  5633 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:28:39.695566  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:39.709074  5702 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:28:39.710014  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:39.710847  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 193740us
I20260709 12:28:39.714498  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:39.714494  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:39.715701  5701 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 21: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:39.718119  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:39.718202  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:39.718932  6277 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:39.719013  6278 raft_consensus.cc:1813] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Handling vote request from an unknown peer cea66aa15be3423880843974acdbf50f
I20260709 12:28:39.719380  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:39.720031  6276 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:39.720913  6278 raft_consensus.cc:2368] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21.
I20260709 12:28:39.718467  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:39.706974 15036 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:39.726078  5599 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:39.728999  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:39.728382  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:39.727783  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:39.737353  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 142804us
I20260709 12:28:39.737671  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:28:39.739161  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 158225us
I20260709 12:28:39.779857  5702 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:28:39.652283 14978 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:39.780485 15052 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.542s	user 0.004s	sys 0.001s
I20260709 12:28:39.725895 14803 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:39.780886 15036 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:39.781345 14979 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:39.783529  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:39.784451  5701 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:28:39.784418  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 46830.8us
I20260709 12:28:39.792785  5701 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:39.738777 14787 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:28:39.791450  6279 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:39.796641  5701 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:39.797880 15085 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:39.798121  5701 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:39.798719 14787 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:39.799917 14787 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:39.800762 15058 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.560s	user 0.002s	sys 0.001s
I20260709 12:28:39.803684 14865 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:28:39.804271 15085 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:39.804782 15085 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:39.807286 15076 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.280s	user 0.000s	sys 0.001s
I20260709 12:28:39.740617 14976 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:39.809901 14903 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:39.811383  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 173249us
W20260709 12:28:39.812431  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:39.812623  5701 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 17. (term mismatch)
W20260709 12:28:39.812856  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:39.818399  5628 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:28:39.816195  5459 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:39.813309  5701 pending_rounds.cc:85] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 16
I20260709 12:28:39.826889  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:39.804178 14865 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:39.795045 14882 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.642s	user 0.011s	sys 0.000s
I20260709 12:28:39.828667  6275 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:28:39.828972  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:39.790989 14949 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:39.831316  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:39.830857  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:39.792644 14803 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:39.831940  5709 raft_consensus.cc:2376] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34. Re-sending same reply.
W20260709 12:28:39.831852  5524 leader_election.cc:336] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.110s (SENT)
I20260709 12:28:39.831629  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:39.832772  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:39.833155  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:39.833931  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:39.824010  5701 pending_rounds.cc:107] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 26.17
I20260709 12:28:39.838291  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:39.837874  6278 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:28:39.837146  5701 raft_consensus.cc:2889] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:39.837075  6279 raft_consensus.cc:2410] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 27 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:28:39.845362  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:39.846956  5706 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:39.849119 15091 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:28:39.864761 15056 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.637s	user 0.000s	sys 0.004s
I20260709 12:28:39.850008  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:39.870141  5702 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
W20260709 12:28:39.867614  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44712.8us
I20260709 12:28:39.849254  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:39.875058  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:39.881194  5706 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:39.876022  6279 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: []
I20260709 12:28:39.839085 14978 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:39.836953 14882 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:39.836400 14803 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:39.864894 14979 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:39.866742 14982 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:28:39.877539 15036 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:39.877715 15091 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:39.897799 15091 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:39.892984 14865 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:39.899214 14865 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:28:39.878069  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:39.900696 14939 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:39.901285 14939 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:39.902355  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:39.903441 14939 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:39.904265 15089 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:39.905222 15057 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:39.903700  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:39.869735  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:39.882892  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:39.888763  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60681.5us
I20260709 12:28:39.889459  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:39.907542  5601 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:39.908149  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:39.909185 14980 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:39.835107 15034 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:28:39.932471 14976 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:39.933154 14976 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:39.933622 14976 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:39.896025  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:39.849921  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:39.835862 14864 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.056s
I20260709 12:28:39.913188  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:39.944054  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:39.914665  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45041.3us
I20260709 12:28:39.945770  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:39.947081  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:39.952957  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:39.957674  5526 leader_election.cc:400] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21.
I20260709 12:28:39.957969  5526 leader_election.cc:403] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:28:39.917064 15031 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:39.959389  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:39.914736 15036 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:28:39.915496 14939 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:39.967092  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:39.904241 15035 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:39.979719  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:39.952401 14882 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:39.955926 15057 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:39.955734 14779 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:28:39.894286 14617 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.066s
I20260709 12:28:39.964036 14982 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:39.960167 15071 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:39.977031 15031 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:39.985703 15068 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:28:40.000470  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:40.003485  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:40.023661  6273 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:28:40.025524  6274 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:39.980285 14903 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:39.990478 14802 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 19 FOLLOWER]: Advancing to term 21
W20260709 12:28:39.996493  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 77055.2us
I20260709 12:28:40.029753  5704 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:40.028934 15031 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:40.003453 15102 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:28:39.994446 14887 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:39.976529 15032 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.049s
W20260709 12:28:39.994058 15097 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:28:40.045024  5675 leader_election.cc:400] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34.
I20260709 12:28:40.052856  5675 leader_election.cc:403] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:28:40.028550 15071 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:40.005179 14982 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:40.055083  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:40.036265 14903 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:40.065086  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 103285us
I20260709 12:28:40.082886 14965 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:40.083376 14965 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:40.083775 14965 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:40.084630 14439 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:28:40.085829 14439 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:40.086186 14439 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:40.087057 14439 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.048875 14887 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:40.088635 14887 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:40.050166 14918 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.042s
I20260709 12:28:40.092839 14882 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:40.093357  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:40.092849 14864 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:40.096520 14864 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:40.097366 15035 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:40.062139 14982 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:40.098227 14692 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:28:40.098920 14692 consensus_queue.cc:1243] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 21.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:28:40.099397 14692 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:28:40.100186 14995 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:40.100621 14995 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:40.101526 14995 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:40.102795 14995 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:40.059736 15071 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:40.104384 14903 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:40.097404 14864 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.110189 14864 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:40.109104  5634 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:28:40.112380 14935 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:40.113361 14802 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21.
I20260709 12:28:40.088256 14439 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:40.156852  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:40.152269  5601 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:40.167698  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:40.168735  5634 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:28:40.169041  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:40.059454 15106 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 33. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34.
I20260709 12:28:40.173339  5704 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:40.071936 15091 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:40.180730  6273 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:28:40.183470  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:40.184504  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:40.188884  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:40.193435  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:40.193608  6275 raft_consensus.cc:2376] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26. Re-sending same reply.
W20260709 12:28:40.194012  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:40.195421  5630 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:28:40.196069  5559 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:28:40.196050  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:40.197764  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:40.199157  5634 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:28:40.191875  6279 raft_consensus.cc:2376] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34. Re-sending same reply.
I20260709 12:28:40.201814  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:40.202451  5629 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:28:40.201844  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:40.200390  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:40.208267  5703 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:40.211483  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:40.210170  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:40.217082  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:40.181850 14978 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:28:40.219983  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 310162us
I20260709 12:28:40.224810  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:40.231467  5706 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:40.228760  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:40.233942  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:40.234009  5704 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:40.183053 15057 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:40.196400 14882 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:40.240103  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:40.237883  5632 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:40.238581  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:40.244603  5630 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:28:40.212033 14779 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:28:39.963757  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:40.254114 15068 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.011s
W20260709 12:28:40.269130  5524 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.175s (SENT)
I20260709 12:28:40.269562  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:40.268527 15104 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:28:40.272781  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48295.8us
I20260709 12:28:40.276450  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:40.253082 14787 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:28:40.282306  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 75966.8us
I20260709 12:28:40.251190 14939 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:40.244686 15036 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.034s
I20260709 12:28:40.299352  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:40.240170 14979 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 12:28:40.275265 15104 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:40.313460  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 92799.3us
I20260709 12:28:40.319964  5631 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
W20260709 12:28:40.327808  5451 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 276) took 53271 ms (client timeout 15000 ms). Trace:
I20260709 12:28:40.247809 15102 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.044s
I20260709 12:28:40.332080  5459 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 21 to 24, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 24 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:40.251271 14882 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.265345 15057 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:40.303638 15089 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:28:40.397089  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 49519.6us
I20260709 12:28:40.398574  5631 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
W20260709 12:28:40.408538  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:40.335517 14802 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:40.373615 15104 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:28:40.300452 14982 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.417641  5635 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: []
W20260709 12:28:40.408629 15080 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:28:40.333530 14976 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:40.426769  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 161454us
I20260709 12:28:40.434911  5459 catalog_manager.cc:5697] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 20 to 23, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 23 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:40.334975 14939 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.418097 14980 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:40.443405  5459 catalog_manager.cc:5697] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 19 to 21, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 21 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:40.395085 14865 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.043s
I20260709 12:28:40.422125 15102 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:28:40.453403  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 156016us
I20260709 12:28:40.447737 14978 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:40.434588 14779 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:40.458231  5524 leader_election.cc:336] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.201s (SENT)
I20260709 12:28:40.465260  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:40.414116 14811 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.031s
W20260709 12:28:40.454317 15096 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.285s	user 0.000s	sys 0.001s
I20260709 12:28:40.469925  5459 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 21 to 25, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 25 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:40.489567  5459 catalog_manager.cc:5697] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 19 to 21, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 21 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:40.477183 14779 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.420876 14802 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.455688 14980 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:40.509565  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 210361us
W20260709 12:28:40.510324  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46457.9us
I20260709 12:28:40.510607  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:40.523028  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:40.460405 15117 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:40.532253  5459 catalog_manager.cc:5697] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:28:40.347894  5451 rpcz_store.cc:269] 0709 12:27:47.056149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:52.532730 (+5476581us) service_pool.cc:224] Handling call
0709 12:27:53.774455 (+1241725us) spinlock_profiling.cc:243] Waited 4.05 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5dc090 00007f4ffa67af3a 00007f5000430e9c 00007f500052f04e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc 00007f4ffe9ef392 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:40.327574 (+46553119us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:28:37.195319 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:28:37.279641 (+ 84322us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:28:37.279704 (+    63us) write_op.cc:435] Acquired schema lock
0709 12:28:37.279732 (+    28us) tablet.cc:662] Decoding operations
0709 12:28:37.280867 (+  1135us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:28:37.280948 (+    81us) write_op.cc:641] Partition lock acquired for write op
0709 12:28:37.280974 (+    26us) tablet.cc:685] Acquiring locks for 11 operations
0709 12:28:37.282150 (+  1176us) tablet.cc:701] Row locks acquired
0709 12:28:37.282179 (+    29us) write_op.cc:260] PREPARE: finished
0709 12:28:37.282387 (+   208us) write_op.cc:270] Start()
0709 12:28:37.282533 (+   146us) write_op.cc:276] Timestamp: P: 1783600117282317 usec, L: 0
0709 12:28:37.282553 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 12:28:37.283410 (+   857us) log.cc:844] Serialized 4294 byte log entry
0709 12:28:37.520013 (+236603us) op_driver.cc:464] REPLICATION: finished
0709 12:28:37.529114 (+  9101us) write_op.cc:301] APPLY: starting
0709 12:28:37.529184 (+    70us) tablet.cc:1367] starting BulkCheckPresence
0709 12:28:38.061258 (+532074us) tablet.cc:1370] finished BulkCheckPresence
0709 12:28:38.061408 (+   150us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:28:38.157976 (+ 96568us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:28:38.233190 (+ 75214us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=22,key_file_lookups=22,delta_file_lookups=0,mrs_lookups=0
0709 12:28:38.241449 (+  8259us) write_op.cc:312] APPLY: finished
0709 12:28:38.285684 (+ 44235us) log.cc:844] Serialized 105 byte log entry
0709 12:28:38.285959 (+   275us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:28:38.286314 (+   355us) write_op.cc:454] Released schema lock
0709 12:28:38.286390 (+    76us) write_op.cc:341] FINISH: Updating metrics
0709 12:28:38.287017 (+   627us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:28:38.287048 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5700625,"reported_tablets":50,"spinlock_wait_cycles":9308544,"child_traces":[["op",{"apply.queue_time_us":8813,"apply.run_cpu_time_us":21750,"apply.run_wall_time_us":757848,"cfile_cache_hit":46,"cfile_cache_hit_bytes":50107,"num_ops":11,"prepare.queue_time_us":347706,"prepare.run_cpu_time_us":11190,"prepare.run_wall_time_us":383852,"raft.queue_time_us":48665,"raft.run_cpu_time_us":1270,"raft.run_wall_time_us":9302,"replication_time_us":237204,"thread_start_us":385355,"threads_started":4,"wal-append.queue_time_us":184840,"wal-append.run_cpu_time_us":5959,"wal-append.run_wall_time_us":1828696}]]}
I20260709 12:28:40.542989  5559 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:28:40.566099  5555 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:28:40.515771 14980 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:40.529469 15111 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:28:40.530228 15089 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:40.540227  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:40.535300 15117 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:40.542656 15102 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:40.570147 14887 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.570520 15074 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:40.580057 15111 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:40.582700 15089 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:40.589898 15117 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:28:40.592908  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 82369.7us
W20260709 12:28:40.642611  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 105899us
I20260709 12:28:40.655504  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:40.659590  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:40.664072  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:40.604053 15085 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.799s	user 0.009s	sys 0.003s
I20260709 12:28:40.667456  5635 raft_consensus.cc:2368] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
W20260709 12:28:40.604815 15110 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
W20260709 12:28:40.620642 15100 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.307s	user 0.000s	sys 0.001s
I20260709 12:28:40.605993 15125 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:28:40.677752 15085 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:40.681380 15085 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:40.682220  5633 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:40.673485  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:40.674515  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:40.608497 14978 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.676247  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:40.686581 15104 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:40.687870  5553 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:28:40.620654 15120 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:28:40.599218 15071 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.675191  6274 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:40.675479  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:40.594340 14976 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:40.691334 15071 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:40.644587 15091 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:40.691852 15032 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:40.692476 15032 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.595206 14939 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:40.595978 14882 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:40.695364 15104 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:40.677580  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 231810us
W20260709 12:28:40.697021  5524 leader_election.cc:336] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.213s (SENT)
I20260709 12:28:40.697407  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:40.698367 14980 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.699604  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:40.700423 14939 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:40.595564 15057 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:40.613647 15089 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:40.702736 15089 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:40.704917 14995 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.602s	user 0.013s	sys 0.000s
I20260709 12:28:40.603091 14949 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.103s
I20260709 12:28:40.604156 15102 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:40.623939 15031 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.564s	user 0.006s	sys 0.002s
W20260709 12:28:40.677876 15063 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.073s	user 0.008s	sys 0.005s
W20260709 12:28:40.624222 14935 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.511s	user 0.004s	sys 0.004s
I20260709 12:28:40.709096 15031 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:40.709883 14864 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.599s	user 0.009s	sys 0.002s
I20260709 12:28:40.710026  5705 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:28:40.710242  6279 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 21. Current term is 22. Ops: []
W20260709 12:28:40.712170 14439 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.595s	user 0.006s	sys 0.005s
I20260709 12:28:40.713580  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:40.635641  5555 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260709 12:28:40.716228 14903 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.611s	user 0.009s	sys 0.003s
I20260709 12:28:40.716709 14903 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.718803 14903 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:40.719283  5633 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:28:40.641538 14804 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.078s
I20260709 12:28:40.721050 15102 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:40.721437 14882 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260709 12:28:40.722839 14803 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.812s	user 0.012s	sys 0.000s
I20260709 12:28:40.723637 14803 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.728674  6272 raft_consensus.cc:2368] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:28:40.730265  5630 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:40.642797  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:40.593487 14982 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:40.643476  5703 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:28:40.598430 14887 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:40.731751  5524 connection.cc:256] RPC call timeout handler was delayed by -1.3907651150002494s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1766s for a response
W20260709 12:28:40.605839 14965 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.522s	user 0.007s	sys 0.004s
I20260709 12:28:40.606736 15111 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.737823 14965 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:40.614714 14802 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:40.716243 14918 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:40.640595  5706 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:40.740641 14965 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:40.741299 15103 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:28:40.742632 14803 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:40.643277 14894 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:40.768409  6280 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
W20260709 12:28:40.748509  5524 connection.cc:256] RPC call timeout handler was delayed by -1.233728935000272s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
I20260709 12:28:40.712924 15031 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:40.753829  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:40.640103  5459 catalog_manager.cc:5697] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:40.641664  5632 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:28:40.755906 15128 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:40.757092  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 114355us
I20260709 12:28:40.757166  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:40.775004  5553 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:28:40.757768 15085 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:28:40.777168  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:40.759661 15117 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:40.591912 14787 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:40.761644  5633 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:28:40.762857  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:40.763923  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:40.812083  5629 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:40.731843  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:40.694801 15032 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:40.766785  5526 leader_election.cc:400] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
I20260709 12:28:40.975485  5526 leader_election.cc:403] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
W20260709 12:28:40.766853  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 120821us
I20260709 12:28:40.746922 15111 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:28:40.770330  6235 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.007s	sys 0.003s
W20260709 12:28:40.977221  6235 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.007s	sys 0.003s
W20260709 12:28:40.779172  5524 leader_election.cc:336] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.376s (SENT)
I20260709 12:28:40.978394  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:40.759687  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:40.777411  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:40.779865  5632 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:40.767445 15137 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:40.781255  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:40.987221  5628 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:28:40.784065  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:40.772884 14980 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:40.834976  5459 catalog_manager.cc:5697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 22 to 24, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 24 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:40.969687  5631 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: []
W20260709 12:28:40.993336  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 226864us
I20260709 12:28:40.873714  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:40.871460  5702 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.001s	sys 0.022s
W20260709 12:28:40.995376  5702 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.002s	sys 0.025s
W20260709 12:28:40.835894 14750 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.013s	sys 0.007s
W20260709 12:28:40.871209  5561 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.016s	sys 0.007s
W20260709 12:28:40.972651  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 240889us
W20260709 12:28:40.996133  5561 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.018s	sys 0.008s
I20260709 12:28:40.996129  5633 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
W20260709 12:28:40.893355  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:40.776623 14903 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:40.966005  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 391452us
W20260709 12:28:40.979411  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 202088us
I20260709 12:28:41.004336  6272 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:28:41.004724  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:41.005429  6277 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:40.980937  5629 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:28:40.782382  6276 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: []
W20260709 12:28:40.990669  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 206725us
I20260709 12:28:40.991321  5630 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:41.007702  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:41.008559  5561 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:40.932154 15032 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.055s
I20260709 12:28:40.976917 14882 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:40.783267 15128 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:41.019752  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:41.024065  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:40.998314 14750 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.016s	sys 0.011s
I20260709 12:28:41.031522  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:41.037192  5701 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:40.989884 15125 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:40.987293 15068 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:41.009689 14802 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:41.024538  5459 catalog_manager.cc:5697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 19 to 20, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 20 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:41.053329  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:41.013794 14439 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.035651  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:41.027115  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:41.030988  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:41.324461  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 288828us
I20260709 12:28:41.027555  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:40.978042 15103 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:41.008772 15031 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:40.997510 15126 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:41.003037 14978 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:41.049146  6277 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
W20260709 12:28:41.054963  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47389.1us
I20260709 12:28:41.015257  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:41.334947  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 319771us
I20260709 12:28:41.336061  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:41.034971 14935 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.297298 15140 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:41.111069 15131 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
W20260709 12:28:41.167243  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 116303us
I20260709 12:28:41.021409 14887 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.092s
I20260709 12:28:41.015458 14903 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:41.027186 14982 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.122s
I20260709 12:28:41.169389  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:41.027671 15102 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.123s
I20260709 12:28:40.993764 15137 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:41.056237 14835 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:28:41.121753  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 68078us
I20260709 12:28:41.385460  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:41.029511 14779 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.036245 14882 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:41.057794  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 278653us
W20260709 12:28:41.392935  5524 connection.cc:256] RPC call timeout handler was delayed by -1.1495651150003141s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2034s for a response
I20260709 12:28:41.392875  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:41.393712  5524 connection.cc:256] RPC call timeout handler was delayed by -1.0276289350003935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
W20260709 12:28:41.394173  5524 connection.cc:256] RPC call timeout handler was delayed by -1.0265000040003542s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20800000000000002s for a response
W20260709 12:28:41.362700 15141 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
I20260709 12:28:41.164880 15032 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:41.196399  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:41.402134  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 207655us
I20260709 12:28:41.207659 15121 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.181195 15138 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.028s
I20260709 12:28:41.058871 14980 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.127s
I20260709 12:28:41.283404  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:41.433429  6272 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:41.165186 14439 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:41.307879  5553 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.532s	user 0.007s	sys 0.002s
I20260709 12:28:41.325577  6279 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:41.437883  5553 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:28:41.285902 15068 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.329175  5709 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:41.330948  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:41.286509 15117 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:41.332760  5706 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:28:41.444267  5701 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:41.337153  6278 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:41.145586  5459 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 15 to 21. New cstate: current_term: 21 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:41.445626  5459 catalog_manager.cc:5697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 18 to 22. New cstate: current_term: 22 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:41.319932 14864 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.348866  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:41.448614 14864 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:41.449829  5703 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:41.451743 14864 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:41.353531  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:41.346894  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:41.395165  5524 connection.cc:256] RPC call timeout handler was delayed by -1.0217651150001075s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21760000000000002s for a response
I20260709 12:28:41.454008  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:41.365664 15136 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.078s
I20260709 12:28:41.343973 15104 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:41.378844 14918 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:41.458853 15146 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.263s	user 0.001s	sys 0.000s
I20260709 12:28:41.458600 14918 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:41.459440 15138 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:41.364280 15140 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:41.460959 14835 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:28:41.435060 14692 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:28:41.379935 14978 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:41.344601 15103 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:41.427857  5704 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:41.401149 14779 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:41.401352 15063 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:41.348739 14935 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:41.162415 14787 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:41.021062 14803 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:41.383594 15089 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.680s	user 0.008s	sys 0.005s
I20260709 12:28:41.461633 14918 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:41.462419  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1625) took 1265 ms. Trace:
I20260709 12:28:41.419695  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:41.462809  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3291) took 1267 ms. Trace:
I20260709 12:28:41.401239 15032 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:41.436902 14894 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:41.490363  5702 rpcz_store.cc:276] 0709 12:28:40.197206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:40.197332 (+   126us) service_pool.cc:224] Handling call
0709 12:28:40.219179 (+ 21847us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:40.232978 (+ 13799us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:40.233008 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:40.233523 (+   515us) log.cc:844] Serialized 26 byte log entry
0709 12:28:40.315557 (+ 82034us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x7b48001701d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:28:40.995736 (+680179us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:40.995781 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:40.995854 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:41.457290 (+461436us) raft_consensus.cc:1685] finished
0709 12:28:41.457344 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:41.462301 (+  4957us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":156149,"raft.queue_time_us":13594,"raft.run_cpu_time_us":2150,"raft.run_wall_time_us":80274,"spinlock_wait_cycles":9448704,"thread_start_us":740907,"threads_started":1,"wal-append.queue_time_us":925677}
I20260709 12:28:41.503218  6272 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:41.467694 14835 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:41.521857  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 236840us
I20260709 12:28:41.464778 15135 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.113s
I20260709 12:28:41.525877  5704 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
W20260709 12:28:41.493911  5628 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.502s	user 0.012s	sys 0.000s
I20260709 12:28:41.463981 15104 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:41.463574 15068 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:41.489801 15063 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:41.490728  5627 rpcz_store.cc:276] 0709 12:28:40.197963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:40.311385 (+113422us) service_pool.cc:224] Handling call
0709 12:28:40.311761 (+   376us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:40.511156 (+199395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:40.511188 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:40.543469 (+ 32281us) log.cc:844] Serialized 26 byte log entry
0709 12:28:40.963701 (+420232us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:40.994821 (+ 31120us) log.cc:844] Serialized 14 byte log entry
0709 12:28:40.995018 (+   197us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:40.995085 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:41.462546 (+467461us) raft_consensus.cc:1685] finished
0709 12:28:41.462571 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:41.462744 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35952,"prepare.queue_time_us":31285,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":5655,"raft.queue_time_us":136445,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":7782,"spinlock_wait_cycles":26391808,"thread_start_us":519062,"threads_started":2,"wal-append.queue_time_us":462081}
I20260709 12:28:41.490876 15117 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:41.534801  5628 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:28:41.536303  5631 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:28:41.490604 15128 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:41.541669 15139 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.341s	user 0.001s	sys 0.001s
I20260709 12:28:41.493685 15149 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.504244 15157 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:28:41.505225 15032 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:41.435359 15121 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:41.500871 14803 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:41.559167  5629 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:28:41.522558 14978 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:41.542534 15104 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:41.545501 15063 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:41.546150 15140 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:41.532053 15089 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.035s
W20260709 12:28:41.602522  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 176062us
I20260709 12:28:41.524631 14894 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:41.572918 14779 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:41.617424  6279 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:41.641703  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:28:41.681864  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 77507.9us
I20260709 12:28:41.563331 15149 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:41.711308  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 364527us
I20260709 12:28:41.716121  5526 leader_election.cc:400] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
W20260709 12:28:41.675721 15154 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.183s	user 0.000s	sys 0.000s
I20260709 12:28:41.723812  5526 leader_election.cc:403] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:28:41.786091  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:41.787164  5629 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
W20260709 12:28:41.789944  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:41.798414  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 82431.2us
I20260709 12:28:41.811793  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:41.810521  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:41.817528 15157 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:41.817528  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:41.818562  5627 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:41.818476 15157 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:41.821439 15151 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.275s	user 0.001s	sys 0.001s
I20260709 12:28:41.823742 15162 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:28:41.829872  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 375939us
I20260709 12:28:41.611443 15155 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.109s
I20260709 12:28:41.831629  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:41.803906 15110 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 12:28:41.608129 14835 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:41.803435 15137 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:41.833655 14439 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:41.868841  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 338938us
I20260709 12:28:41.870515  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:41.850696 14882 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:41.854280 14692 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:41.891037  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:41.856161 15103 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:41.899348  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 479255us
I20260709 12:28:41.865023 15137 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:41.862819 14935 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:41.944605  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:41.941845 15111 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.950s	user 0.006s	sys 0.009s
I20260709 12:28:41.946853 15161 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.093s
I20260709 12:28:41.948247 14439 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:41.948856 15068 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:41.840248 14978 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:41.950839  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:41.953672 14935 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:28:41.954373 14935 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:41.955539 14935 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:41.931661 14835 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:28:41.940567 14949 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.036s	sys 0.048s
I20260709 12:28:41.864497 15036 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.080s
I20260709 12:28:41.938052 14882 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:41.941713 15102 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.017s	sys 0.026s
W20260709 12:28:41.975184 15102 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.018s	sys 0.026s
I20260709 12:28:41.963143 15116 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:28:41.978854  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:41.972565  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 141043us
W20260709 12:28:41.970047 14949 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.036s	sys 0.048s
I20260709 12:28:41.966462 15168 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:41.966701 15128 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:41.963986 15171 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:28:41.982439  6235 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.004s	sys 0.029s
W20260709 12:28:41.987011  6235 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.004s	sys 0.029s
I20260709 12:28:41.939622 15165 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
I20260709 12:28:41.975565 15174 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:41.995013 15174 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:41.998347 15174 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:41.999727 15150 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.491s	user 0.000s	sys 0.001s
I20260709 12:28:42.000437  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.001900  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:42.002923  5702 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:42.004637 15140 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:41.983358 15164 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:28:42.005277 15140 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.006223 15121 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:42.007750 15149 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:42.007910 15140 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:41.975469 15157 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:41.999451  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.003959  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:41.984081 15111 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:28:42.057204  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 75980.5us
I20260709 12:28:41.984889 15117 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.985831 15068 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:42.062475 14811 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:42.058843  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:42.004654  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.058827  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:42.013933 15116 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.059536 15176 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:42.068150 15164 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:42.060544 15157 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.074678  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:42.075490  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.012882 14894 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:42.079672  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:42.060011 14692 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.084985  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:42.053350 15121 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.994061 15137 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:28:42.002924 15156 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.440s	user 0.006s	sys 0.000s
I20260709 12:28:42.092255  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:41.978922 14882 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.010133 14949 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:41.993889 15177 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:42.013461 14803 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.090919 14811 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:42.073823 15031 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:42.056658  6279 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:42.066219 15102 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:41.986915 15128 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:41.988230 15168 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.085878 15117 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.067359 15140 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:42.094746  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:42.116482  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57793.1us
I20260709 12:28:42.117063  5702 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:42.118553  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:42.127175 14802 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.066s	user 0.014s	sys 0.001s
I20260709 12:28:42.120012  5553 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:42.121722  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:42.175017  5556 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:42.109560  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:42.095765  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1492) took 1426 ms. Trace:
I20260709 12:28:42.122340  5555 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:42.136225  6279 raft_consensus.cc:606] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:28:42.179191  6279 raft_consensus.cc:614] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:28:42.136833  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 245916us
I20260709 12:28:42.138283  6272 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:28:42.118425 15171 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:42.119396  5559 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:28:42.143674  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:42.187526  5706 raft_consensus.cc:2376] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27. Re-sending same reply.
I20260709 12:28:42.139279  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:42.114069 15167 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:42.176481  6275 rpcz_store.cc:276] 0709 12:28:40.669768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:40.670772 (+  1004us) service_pool.cc:224] Handling call
0709 12:28:40.671101 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:40.684998 (+ 13897us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:40.685032 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:40.685575 (+   543us) log.cc:844] Serialized 26 byte log entry
0709 12:28:40.994317 (+308742us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:40.994365 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:40.994742 (+   377us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:41.999016 (+1004274us) spinlock_profiling.cc:243] Waited 34.1 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:42.091452 (+ 92436us) raft_consensus.cc:1685] finished
0709 12:28:42.091478 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:42.095660 (+  4182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1314,"raft.queue_time_us":1471,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"spinlock_wait_cycles":78403328,"thread_start_us":308085,"threads_started":1,"wal-append.queue_time_us":748991}
W20260709 12:28:42.183606  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 88898.9us
I20260709 12:28:42.118207 15036 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.113910 15158 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:42.140215 15031 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:42.189162  6277 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 28 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260709 12:28:42.201678  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:42.094303 15111 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:42.190337  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:42.134605 15188 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:42.116039 15169 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:42.196201  5706 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:42.230078  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1511) took 2077 ms. Trace:
I20260709 12:28:42.240965  6276 rpcz_store.cc:276] 0709 12:28:40.153172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:40.153468 (+   296us) service_pool.cc:224] Handling call
0709 12:28:40.332619 (+179151us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:41.009975 (+677356us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:41.784718 (+774743us) spinlock_profiling.cc:243] Waited 449 ms on lock 0x7b54004f9cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:42.229872 (+445154us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54004f9cd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:42.230013 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1363170304}
I20260709 12:28:42.203521  6274 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:42.196463  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:42.242766  5702 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:42.243048  5559 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:42.182483 14802 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25.
I20260709 12:28:42.198839  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:42.200904  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:42.094447 15068 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:42.245945  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45124us
W20260709 12:28:42.246117  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 260300us
I20260709 12:28:42.186933 15171 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:42.122203 15157 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.205287  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:42.201057  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:42.250837  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 49841.7us
I20260709 12:28:42.210326  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:42.210564 15117 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:42.107995 15116 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.127156 15121 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:42.191064 15153 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:28:42.202687  5554 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:42.180373 15140 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:42.224572 14882 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.235059  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1881) took 1795 ms. Trace:
I20260709 12:28:42.213897 14811 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.196723 15167 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:42.237349 15144 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.028s	user 0.010s	sys 0.010s
I20260709 12:28:42.284731  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.204681 15036 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.285593  5634 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:42.286775  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.237753 14935 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:42.288046  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.185612 14894 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:42.247231  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:42.290020 15157 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:42.251520 15171 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:42.290132 14882 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:42.290899 14882 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.282740  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:42.293084  5630 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:42.293450 14882 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.287389 15036 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.294169 15168 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:42.295105 15168 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:42.295598 15168 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:42.093313  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.190537  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.125447 15164 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:42.115262 15176 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.224625 15129 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:42.255700 15169 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:42.172508 15102 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.263085  5631 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 12. Preceding OpId from leader: term: 30 index: 13. (index mismatch)
W20260709 12:28:42.264950  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 66185.8us
I20260709 12:28:42.327927  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:42.271481  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 66278.2us
I20260709 12:28:42.272992  5629 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: []
I20260709 12:28:42.249514 15126 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:42.330955  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:42.274560 14965 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:42.280325 15187 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:42.280656  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.252223 14979 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:42.266512  5558 rpcz_store.cc:276] 0709 12:28:40.440111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:40.440773 (+   662us) service_pool.cc:224] Handling call
0709 12:28:40.458421 (+ 17648us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:40.969436 (+511015us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:40.969471 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:41.002955 (+ 33484us) log.cc:844] Serialized 26 byte log entry
0709 12:28:41.317353 (+314398us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:28:41.317410 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:41.317502 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:42.002096 (+684594us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:42.234761 (+232665us) raft_consensus.cc:1685] finished
0709 12:28:42.234784 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:42.234998 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28370,"raft.queue_time_us":122599,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":2639,"spinlock_wait_cycles":64857984,"thread_start_us":574206,"threads_started":2,"wal-append.queue_time_us":440833}
I20260709 12:28:42.282740  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.283679  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.268898 15165 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.287374  5634 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:42.262907 14750 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.220492 14980 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.043s
I20260709 12:28:42.142072 15162 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:42.262277 14887 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:42.269361 15117 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:28:42.310079 15129 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:42.315173 15063 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.714s	user 0.012s	sys 0.000s
I20260709 12:28:42.245030 15188 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:42.317356 15164 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.290242 15068 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.317975 15140 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.320600 15173 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:28:42.324678 15032 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.744s	user 0.013s	sys 0.001s
I20260709 12:28:42.326244 15121 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:42.334143  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:42.465725  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:42.466248  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 132206us
I20260709 12:28:42.334710 14894 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:42.459915 15063 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.336319 15148 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:42.338366 15176 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:42.339711  5524 leader_election.cc:400] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 20 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 20. Current term is 21.
I20260709 12:28:42.340209 14802 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:42.278864  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:42.453131 14750 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.354287 15168 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.452986 15165 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.343503 15102 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:42.348327 14864 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:42.350013 15149 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.350395 15111 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:42.205003 15158 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:42.368643 15126 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:42.483472  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 132814us
I20260709 12:28:42.480654  5702 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:42.439718  5632 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:42.440315  5636 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
W20260709 12:28:42.449687  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 248786us
I20260709 12:28:42.455482  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:42.457628  6278 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
W20260709 12:28:42.485031  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 206421us
I20260709 12:28:42.457562  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:42.487957  6277 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 20.
I20260709 12:28:42.347823 15031 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:42.439729 15136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.949s	user 0.051s	sys 0.055s
W20260709 12:28:42.452960 15104 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.862s	user 0.008s	sys 0.004s
I20260709 12:28:42.462970  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:42.464931  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:42.449854 14887 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:42.453068 14979 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:42.469290 15162 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.514828  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.472752 14918 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:28:42.470161 15191 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.057s	user 0.003s	sys 0.000s
I20260709 12:28:42.472167 15063 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:42.538245  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.472677 15185 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.067s
I20260709 12:28:42.454754 15187 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.474476 15188 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.482506 15173 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:42.542096 15173 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.543592 15158 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.493024 14864 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:42.545679 15165 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.545503 14864 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.547032 15126 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.515195 14787 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.505884 15117 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:42.505712 15149 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.552022 15149 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:42.507094  6273 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:42.581847 14439 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.582435 14439 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:42.475287 15116 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.507377  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:42.584411 15202 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:42.584894 15202 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:42.584604  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77281.6us
I20260709 12:28:42.585371 15202 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:42.585528 15148 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.586112 15148 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.586918  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:42.588502 15164 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:42.589063 14894 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:42.589649 14894 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.590075  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:42.591019 15174 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:42.487797  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:42.514456  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:42.474256 14835 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.503s	user 0.006s	sys 0.004s
I20260709 12:28:42.593988  5707 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 34 because replica has last-logged OpId of term: 33 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 19.
I20260709 12:28:42.594707 14439 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:42.514629  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:42.595734  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 81239.7us
I20260709 12:28:42.597525 14835 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:42.598052 14835 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:42.598471 14835 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:42.601868  6277 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:42.605315 14439 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:28:42.605868  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:42.508608 15136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.020s	user 0.051s	sys 0.056s
I20260709 12:28:42.522308 15167 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:42.607504 15167 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:28:42.476827 15129 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.474180 15032 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.610128 15032 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.524102 15089 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.610812 15089 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:42.613502 15089 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.529767 14979 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.530182 14750 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:42.532827 14903 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.479010 15142 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:42.614842 15142 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:42.537159  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.615862  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:42.537743  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:42.616593  6275 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:28:42.616823  6278 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:28:42.512591 15176 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:28:42.506124 14692 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.047s
W20260709 12:28:42.619009  5704 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3804) took 1283 ms (client timeout 1663 ms). Trace:
I20260709 12:28:42.512800 15104 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:42.512262 15102 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.042s
I20260709 12:28:42.554731 14787 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.623663 14787 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:42.557334  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1663) took 1560 ms. Trace:
I20260709 12:28:42.624536  6280 rpcz_store.cc:276] 0709 12:28:41.003192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:41.003355 (+   163us) service_pool.cc:224] Handling call
0709 12:28:41.003814 (+   459us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:41.604966 (+601152us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:28:41.605355 (+   389us) log.cc:844] Serialized 14 byte log entry
0709 12:28:42.067282 (+461927us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:42.084733 (+ 17451us) log.cc:844] Serialized 26 byte log entry
0709 12:28:42.084960 (+   227us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:42.085023 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:42.556828 (+471805us) spinlock_profiling.cc:243] Waited 598 ms on lock 0x7b54000b54d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:42.556887 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:42.556945 (+    58us) raft_consensus.cc:1685] finished
0709 12:28:42.557053 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:42.557262 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31128,"prepare.queue_time_us":223,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":67,"raft.queue_time_us":126213,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":12290,"spinlock_wait_cycles":1375000576,"thread_start_us":458272,"threads_started":1,"wal-append.queue_time_us":471448}
I20260709 12:28:42.472601 14802 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:42.627177 14802 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:42.561311 15198 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:28:42.513530 14887 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:42.333257 15169 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.565888  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:28:42.639446  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 73734.7us
I20260709 12:28:42.570070  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.640574  5636 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:42.641619  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.642218  5636 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:42.643060  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.643576  5636 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:28:42.571522  6274 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:42.644618  6274 raft_consensus.cc:606] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:28:42.644964  6274 raft_consensus.cc:614] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:42.571552  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.646445 15128 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.647531  5629 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:42.648705  5629 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:42.648985 15169 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:42.571561 14982 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:42.649698 14982 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:42.650487 15184 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:42.574308 15188 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:42.575803 15171 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:42.650893 15184 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:42.651393 15184 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:42.651322 15188 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:42.651803  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:42.576462  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:42.577099  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.579577  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.580312  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.653707  5635 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:42.652475  5557 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:42.669857  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.476223 15201 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.606707  5634 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:42.678439 15201 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.608069 15116 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:42.679991 15116 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.681865 15129 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:42.683082 15116 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:42.619587  5704 rpcz_store.cc:269] 0709 12:28:41.335659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:41.335818 (+   159us) service_pool.cc:224] Handling call
0709 12:28:42.618790 (+1282972us) spinlock_profiling.cc:243] Waited 91.2 ms on lock 0x7b540033fc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:42.618935 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":209676160}
I20260709 12:28:42.572255  5631 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:42.685104 15102 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:42.580312  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.655215  6274 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "786a49bad9be400ea9ce584030aad446"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:42.685940 15102 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:42.664032 15184 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.688797 15184 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:42.689771  6274 raft_consensus.cc:606] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:42.690073 15203 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.198s	user 0.002s	sys 0.000s
I20260709 12:28:42.693440  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3806) took 1284 ms. Trace:
I20260709 12:28:42.693898  5708 rpcz_store.cc:276] 0709 12:28:41.409273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:41.415898 (+  6625us) service_pool.cc:224] Handling call
0709 12:28:42.693160 (+1277262us) spinlock_profiling.cc:243] Waited 821 ms on lock 0x7b540006f754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:42.693358 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1887875328}
I20260709 12:28:42.665238  6277 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:28:42.592612  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.696089  5706 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 30 because replica has last-logged OpId of term: 28 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
I20260709 12:28:42.696727 15117 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:42.697495 15117 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:42.671732 14811 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:42.661586 15140 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:42.654522  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.701110  5636 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:28:42.690248  6274 raft_consensus.cc:614] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:42.650816 14982 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:42.701720  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:42.702723 14982 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:42.706496  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 56248.2us
I20260709 12:28:42.710829 15103 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:42.710865  6274 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bf758b480ecf41619471236195bf1838"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:42.711370 15103 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:42.711558  6274 raft_consensus.cc:606] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:42.712050  6274 raft_consensus.cc:614] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:42.712314 15103 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.714448 15103 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:42.716439 15068 consensus_queue.cc:1175] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f613ba929b1046d3b5aed909414a95bf" responder_term: 26 status { last_received { term: 24 index: 18 } last_committed_idx: 18 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: [25.19-25.19]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:28:42.588287 15148 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:42.719981  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.720301  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.720680  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.721016  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.721302  5631 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 21.
I20260709 12:28:42.721504  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:42.721659  5635 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 20.
I20260709 12:28:42.720693  5632 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:28:42.728229  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:42.722076  5629 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:42.737540  5705 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:28:42.738302  5705 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:42.735996  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.739207  5706 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:42.740276  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.741770  5636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:28:42.742640  5636 raft_consensus.cc:606] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:42.743072  5636 raft_consensus.cc:614] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:42.744762  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:42.739490  5705 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:42.747411  5705 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:42.754598  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:42.760071  5634 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:42.761552  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:42.755463  6277 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:42.773324  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.765856  6236 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:42.786053  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:42.768206  5553 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:42.601897 15031 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:42.788542  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 185037us
I20260709 12:28:42.775426  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:42.793702  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.794508  5702 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:42.778043 15190 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:42.779021  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:42.783463  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:42.763572  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 275977us
I20260709 12:28:42.768947  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.796022  6278 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:28:42.799127  5706 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:42.797487  6237 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:42.809128  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.809978  5702 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:42.812700  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:42.788638  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:42.811543  5629 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:42.826588  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:42.829780  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:42.825969  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:42.832870  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:42.844261  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:42.844789  5702 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:28:42.839994  6275 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
W20260709 12:28:42.799881 15115 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.585s	user 0.023s	sys 0.027s
W20260709 12:28:42.859725  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 255532us
I20260709 12:28:42.870947  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:42.882622  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:42.591717 14894 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:42.799396 15184 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:42.815696 15116 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:42.805038 15190 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:42.898166  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 96968.1us
W20260709 12:28:42.901398  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:42.845925 15183 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:42.906919  6277 raft_consensus.cc:606] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:28:42.832645 15201 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:42.909152  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:42.872910 15115 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.666s	user 0.023s	sys 0.027s
I20260709 12:28:42.939396  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:42.957195 15068 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:28:42.957690  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49020.5us
I20260709 12:28:42.957931 15068 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:42.958635  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:42.916137 14903 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.045s
I20260709 12:28:42.860037 14803 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:28:42.941527 15125 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.744s	user 0.029s	sys 0.026s
I20260709 12:28:42.997629  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:42.918269  6277 raft_consensus.cc:614] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:28:42.920933  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45968.2us
I20260709 12:28:42.890125 15031 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:42.920027 15190 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:42.929824  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:43.007505  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 77828.9us
I20260709 12:28:42.919252 15116 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:42.937983  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:42.949612  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:42.937142 15183 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:42.994776  5557 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:43.000463  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:43.002503  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:42.931488 15202 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:43.037397  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:43.038268  5560 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
W20260709 12:28:43.006011 15192 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.060s	sys 0.129s
W20260709 12:28:43.012176 15125 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.819s	user 0.029s	sys 0.027s
W20260709 12:28:42.937634 15204 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.338s	user 0.000s	sys 0.001s
I20260709 12:28:42.958524 15068 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:43.008666  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:42.907608 15165 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:28:42.909040 15184 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:42.917990  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:43.016268 15116 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:43.021726  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:43.014546  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 65071.6us
W20260709 12:28:43.053709  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45131.6us
I20260709 12:28:43.025287  5561 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:42.980314  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:43.030563  5705 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 20. (term mismatch)
I20260709 12:28:42.913838  5634 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:28:43.035542  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:42.898041 15166 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:43.036593 14982 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:42.990015 14803 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:43.049647  5602 leader_election.cc:336] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.574s (SENT)
I20260709 12:28:43.067677  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:43.014989 14949 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:43.012212 15031 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:43.055742  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:43.069252  5702 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:43.055821  5706 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:43.057237  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:43.055482  5705 pending_rounds.cc:85] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 19
I20260709 12:28:43.072254  5705 pending_rounds.cc:107] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 30.20
I20260709 12:28:43.035809 15183 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:43.078019  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:28:43.079522 15202 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:43.079967  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:28:43.080695  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 100504us
I20260709 12:28:43.072981 14949 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:43.081418  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:43.081634 14949 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:43.084451 15158 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:43.073213 15031 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:43.084964 15158 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:43.085395 15158 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:43.039870 15157 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.028s
I20260709 12:28:43.087790 15157 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:28:43.043254 15192 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.061s	sys 0.130s
I20260709 12:28:43.088212 15157 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:43.088652 15157 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:42.916949 15148 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.128s
I20260709 12:28:43.054330  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:43.090273  5602 leader_election.cc:336] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.698s (SENT)
I20260709 12:28:43.090763  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:43.091820  5602 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.770s (SENT)
I20260709 12:28:43.091847  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:43.094049 15188 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:43.094658 15188 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:43.094980 15188 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:43.095295 15188 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:43.096715 15202 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:43.097563 15085 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:43.097744 15202 consensus_queue.cc:1243] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 27.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:28:43.054610  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:43.099354 15032 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:28:43.099467  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:43.099901  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45321.2us
I20260709 12:28:43.102274 15149 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:43.102784 15149 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:43.102963  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 81379.4us
I20260709 12:28:43.103536 15149 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:43.104296  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:43.106163 15201 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:28:43.106243 15168 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:43.106643 15201 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:43.106825 15168 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:43.107116 15201 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:43.107331 15168 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:43.063325 14982 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:43.012365 15126 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:28:43.075349  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:43.043960 15125 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:43.076900  5705 raft_consensus.cc:2889] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:43.077677  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:43.050706 15165 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:43.098431 15085 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:28:43.109313  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:43.092295  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:43.110461  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:43.098382  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:43.065328 15138 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:28:43.074581 14803 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:43.062546 15184 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:43.098496 15202 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:43.144399  5706 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:43.162911 15126 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:43.120842  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:43.118165 14982 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:43.199498 15169 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:43.208372  5702 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:43.064055 14887 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:43.213434  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 115135us
I20260709 12:28:43.126866 15148 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:43.100009 15032 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:43.230343  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 138422us
I20260709 12:28:43.215447 15185 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:43.288424  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:43.252163 14978 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:43.253221  5601 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:43.254689  6271 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:28:43.263324 15104 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:43.264712 14864 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:43.127570 15219 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.111s
I20260709 12:28:43.217334 15138 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:43.267829  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:43.222945 15202 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:43.219010 15165 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:43.285948  5702 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:43.286295  5629 pending_rounds.cc:85] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 14
I20260709 12:28:43.305302  5629 pending_rounds.cc:107] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20260709 12:28:43.287604  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:43.306321  5632 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 28: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:43.307400  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:43.307821 14811 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.609s	user 0.012s	sys 0.000s
I20260709 12:28:43.308595  6274 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:43.305667  5629 raft_consensus.cc:2889] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:43.295336  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:43.281106 15213 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.051s	user 0.000s	sys 0.003s
W20260709 12:28:43.268427  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.006s	sys 0.009s
W20260709 12:28:43.314451  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.006s	sys 0.009s
W20260709 12:28:43.314815  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 723277us
I20260709 12:28:43.315004  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:43.380795 14887 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:43.300827 15138 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:28:43.301553 15202 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:28:43.400110  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 345333us
I20260709 12:28:43.305670 15089 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:43.406112  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:43.367681 14903 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:28:43.292694 15148 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:43.368316 15104 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:43.299697 14978 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:43.410032 15089 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:43.462009  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 370198us
W20260709 12:28:43.471976  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 218821us
I20260709 12:28:43.432166 15155 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:43.476374  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:43.312250 15125 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:43.483215  5708 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:28:43.450704 14903 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:43.457969 15032 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:43.445104 15104 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:43.497712  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 181931us
W20260709 12:28:43.488266  5602 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.971s (SENT)
I20260709 12:28:43.484184 15155 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:43.525691  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:43.381449 15126 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:43.530670  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:43.526136  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:43.538244  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 50021.5us
W20260709 12:28:43.539397  5602 leader_election.cc:336] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.019s (SENT)
I20260709 12:28:43.541127  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:43.526110 15221 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.310s	user 0.000s	sys 0.001s
W20260709 12:28:43.549027  5602 leader_election.cc:336] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.039s (SENT)
I20260709 12:28:43.550362  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:43.559478  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 274969us
I20260709 12:28:43.522847 15111 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:43.496493 15227 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:28:43.559386  5602 leader_election.cc:336] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.078s (SENT)
I20260709 12:28:43.562067  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:43.529204 15155 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:43.548179 14887 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:43.572007  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:43.554731 15174 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:28:43.576857  5602 leader_election.cc:336] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.087s (SENT)
I20260709 12:28:43.513928 14903 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:43.553540 15171 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:43.545714 15230 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:43.546128 15166 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:43.581461  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:43.570885 14802 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.941s	user 0.017s	sys 0.008s
W20260709 12:28:43.575095  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 307321us
I20260709 12:28:43.590446  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:43.593815  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:43.569829 15227 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:43.576437 15115 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:28:43.599385 15140 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.892s	user 0.039s	sys 0.018s
I20260709 12:28:43.525733 15142 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:43.569332 15111 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:43.548415 15089 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:43.603786 15171 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:43.570691 15183 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:43.606273 15137 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:28:43.668908 15212 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.279s	user 0.001s	sys 0.001s
I20260709 12:28:43.599858 15227 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:43.593632 14802 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:43.689190 14803 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:43.683796 15129 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.001s	user 0.018s	sys 0.008s
W20260709 12:28:43.696798 15231 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
W20260709 12:28:43.697606 15233 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:28:43.710872  6280 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:28:43.695168 14835 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.096s	user 0.024s	sys 0.016s
I20260709 12:28:43.683250 15230 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:43.728973  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:43.708148 15174 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:43.737620  6277 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:43.703634 15089 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:28:43.734392  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 173942us
W20260709 12:28:43.716157 15168 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.592s	user 0.010s	sys 0.000s
W20260709 12:28:43.716727 15234 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
W20260709 12:28:43.755810  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 162076us
W20260709 12:28:43.695708 15201 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.588s	user 0.020s	sys 0.000s
I20260709 12:28:43.741451  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:43.711885 14949 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.630s	user 0.008s	sys 0.005s
I20260709 12:28:43.742794  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:43.744733  5558 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:43.702487 14980 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:28:43.703321 15148 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:43.699575 15235 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:43.715363 15237 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:43.758070  5631 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:43.728632 15142 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:43.689972 15171 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:43.761440 15226 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.243s	user 0.001s	sys 0.000s
I20260709 12:28:43.744549 15174 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:43.790036  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:28:43.804172  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 242204us
I20260709 12:28:43.733319 14835 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:43.762310 14978 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:28:43.771088 15235 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:43.772110 15209 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:28:43.759712 14894 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:43.788302 15148 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:28:43.809981  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:43.831323  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 357266us
W20260709 12:28:43.842373 15116 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.768s	user 0.007s	sys 0.004s
I20260709 12:28:43.845654 15116 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:43.844211  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:43.847496 15175 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:43.850195  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3656) took 1146 ms. Trace:
I20260709 12:28:43.849169 15116 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:43.852564  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 111004us
W20260709 12:28:43.852633  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 96077.4us
I20260709 12:28:43.866345 14887 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:43.869241 15102 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:43.882841  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:43.709364 15166 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:43.888006  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:43.890058  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3293) took 1639 ms. Trace:
I20260709 12:28:43.765198 15201 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:43.850715  5558 rpcz_store.cc:276] 0709 12:28:42.703950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:42.761171 (+ 57221us) service_pool.cc:224] Handling call
0709 12:28:43.849910 (+1088739us) spinlock_profiling.cc:243] Waited 966 ms on lock 0x7b54000271d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:43.850116 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2220939136}
I20260709 12:28:43.759243 15168 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:43.897363  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1905) took 1323 ms. Trace:
I20260709 12:28:43.765633 14949 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:43.928879  5559 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:28:43.827306 15248 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:43.892149  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:43.739009 15230 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:43.961959  5701 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:43.827539 15235 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:43.920926  5628 rpcz_store.cc:276] 0709 12:28:42.573895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:42.576572 (+  2677us) service_pool.cc:224] Handling call
0709 12:28:42.576870 (+   298us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:42.577197 (+   327us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:42.577226 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:42.577718 (+   492us) log.cc:844] Serialized 26 byte log entry
0709 12:28:42.911424 (+333706us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:43.050177 (+138753us) log.cc:844] Serialized 14 byte log entry
0709 12:28:43.155987 (+105810us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:43.156526 (+   539us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:43.894953 (+738427us) raft_consensus.cc:1685] finished
0709 12:28:43.894975 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:43.897290 (+  2315us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":114389,"prepare.queue_time_us":25270,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":109750,"raft.run_cpu_time_us":4481,"raft.run_wall_time_us":63278,"thread_start_us":455799,"threads_started":2,"wal-append.queue_time_us":718477}
I20260709 12:28:43.831300 14894 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:43.896181  5633 rpcz_store.cc:276] 0709 12:28:42.250232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:42.250928 (+   696us) service_pool.cc:224] Handling call
0709 12:28:42.251307 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:42.251720 (+   413us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:42.251759 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:42.260703 (+  8944us) log.cc:844] Serialized 26 byte log entry
0709 12:28:42.602085 (+341382us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:42.882246 (+280161us) log.cc:844] Serialized 14 byte log entry
0709 12:28:42.937850 (+ 55604us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:42.937924 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:43.507030 (+569106us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b18004b60a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1
0709 12:28:43.872322 (+365292us) spinlock_profiling.cc:243] Waited 6.63 ms on lock 0x7b18004b60a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:28:43.889726 (+ 17404us) raft_consensus.cc:1685] finished
0709 12:28:43.889759 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:43.889967 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74542,"prepare.queue_time_us":295028,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":65150,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":26960,"spinlock_wait_cycles":46728448,"thread_start_us":294757,"threads_started":1,"wal-append.queue_time_us":645740}
I20260709 12:28:43.963923  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:43.898016 14978 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:43.965005  5633 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 21: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:43.827304 15104 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:43.939219  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 181608us
I20260709 12:28:43.792371 15237 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:43.918833 15103 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.204s	user 0.010s	sys 0.005s
I20260709 12:28:43.895318 15251 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:43.977845  5632 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:43.978051 15251 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:43.978514 15251 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:43.980959 15239 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260709 12:28:43.981235 15174 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:43.960389 15200 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:43.965243  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 155329us
W20260709 12:28:43.982347 15238 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 12:28:43.982187 15200 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:43.848192 15175 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:43.830682 15173 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.121s
I20260709 12:28:43.992813  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:43.968217  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1678) took 1092 ms. Trace:
W20260709 12:28:43.954521 15117 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.253s	user 0.016s	sys 0.000s
I20260709 12:28:43.975435  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:43.975982 15171 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:43.977387 15237 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:43.901506 15187 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:43.985576  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:43.985848  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.005729  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:43.992877  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.006462  5552 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:43.986687  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:43.993806  5560 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:43.996920  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 51694.6us
I20260709 12:28:43.993793  6273 rpcz_store.cc:276] 0709 12:28:42.876208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:42.877071 (+   863us) service_pool.cc:224] Handling call
0709 12:28:42.905462 (+ 28391us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:42.936687 (+ 31225us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:42.936721 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:42.949112 (+ 12391us) log.cc:844] Serialized 52 byte log entry
0709 12:28:43.281468 (+332356us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:28:43.281524 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:43.281624 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:43.967878 (+686254us) raft_consensus.cc:1685] finished
0709 12:28:43.967910 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:43.968137 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":33790,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":62572,"thread_start_us":234823,"threads_started":1,"wal-append.queue_time_us":430920}
I20260709 12:28:43.944291 15140 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.042s
I20260709 12:28:43.999979  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.011658  5558 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:43.966670 14979 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:44.002796  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:44.002240 15202 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.565s	user 0.004s	sys 0.007s
I20260709 12:28:43.968302 14894 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:44.004828  5553 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:44.008185  5555 raft_consensus.cc:2410] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:28:44.008931  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:43.983848 15190 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.944s	user 0.005s	sys 0.008s
I20260709 12:28:43.974817 15057 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:43.983541 15111 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.002962  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:44.004154 15157 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.915s	user 0.012s	sys 0.001s
I20260709 12:28:43.984071 14978 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.022243  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.004427 15183 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:44.023280  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.031569  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.031715  6276 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:44.032256  5553 raft_consensus.cc:2410] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 27 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:28:44.008642 14803 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:44.033161  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:44.028434  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:44.028802 15242 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.219s	user 0.001s	sys 0.000s
I20260709 12:28:44.045598 15175 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.046236  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:44.046826  5629 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:44.049057  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:28:44.049610 14982 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.781s	user 0.014s	sys 0.002s
W20260709 12:28:44.050171 15188 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.954s	user 0.016s	sys 0.005s
I20260709 12:28:44.050060 14982 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.050616 15188 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.005842 14802 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.026350 15200 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:44.053154 15168 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.022246 15057 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.029917  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.056704  5556 raft_consensus.cc:2410] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 31 because replica has last-logged OpId of term: 30 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 12.
I20260709 12:28:44.057591  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:44.041460  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:44.037297 15236 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:28:44.020073 15031 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.935s	user 0.010s	sys 0.004s
I20260709 12:28:44.041487 15201 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.050321  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:44.042656 15256 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:28:44.061304  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.063306  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:44.066491  5552 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:44.068310  5561 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:44.000007 15117 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.074699  5629 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:28:44.077958  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.080582  5634 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:44.081110  5552 raft_consensus.cc:2376] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28. Re-sending same reply.
I20260709 12:28:44.078883  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.085708  5560 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:28:44.090126  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.092557  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.093349  5560 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:44.093523  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.094755  5561 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:44.096042  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.096930  5557 raft_consensus.cc:2410] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:28:44.098945  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:44.079902 15158 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.994s	user 0.012s	sys 0.014s
I20260709 12:28:44.104807  5561 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260709 12:28:44.105786  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.070132 15148 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:44.108309  5552 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:28:44.111001 15117 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:44.111266 14864 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.799s	user 0.004s	sys 0.011s
W20260709 12:28:44.107025 15068 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.041s	user 0.024s	sys 0.015s
I20260709 12:28:44.112396 15057 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.115458 15251 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.116761 15257 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:44.181174  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:44.073429 14982 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.190179  5703 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:44.120800 15230 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.021247 15190 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:44.123754 15165 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.763s	user 0.008s	sys 0.002s
I20260709 12:28:44.090018 15254 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.034s
I20260709 12:28:44.080103 15031 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:44.127015 15149 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.023s	user 0.013s	sys 0.012s
W20260709 12:28:44.128875 15184 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.893s	user 0.002s	sys 0.011s
I20260709 12:28:44.020215 15157 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.018790 15202 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:44.134260 15138 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.712s	user 0.010s	sys 0.001s
I20260709 12:28:44.132386  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.111541 15188 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.148625  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:44.148576  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:44.229522  5709 raft_consensus.cc:2368] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28.
I20260709 12:28:44.230096  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:44.149475  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:44.151116  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:44.159647 15158 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.206291 14864 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:44.177340  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.258697  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:44.182469  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:44.189181  5629 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:28:44.260881  5632 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
W20260709 12:28:44.194738  5602 leader_election.cc:336] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 2.206s (SENT)
I20260709 12:28:44.262015  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:44.262640  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1677) took 1387 ms. Trace:
I20260709 12:28:44.262971  6280 rpcz_store.cc:276] 0709 12:28:42.875632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:42.891031 (+ 15399us) service_pool.cc:224] Handling call
0709 12:28:44.262491 (+1371460us) spinlock_profiling.cc:243] Waited 697 ms on lock 0x7b54005ee554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:44.262596 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1602372992}
W20260709 12:28:44.263170  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 68476.7us
I20260709 12:28:44.219367  5634 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:44.223416  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1517) took 1475 ms. Trace:
I20260709 12:28:44.231571  5710 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:44.231410  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:44.253410  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.172897  5558 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:44.258211  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.178903  5555 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:44.266849  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.264663  6274 rpcz_store.cc:276] 0709 12:28:42.727708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:42.727903 (+   195us) service_pool.cc:224] Handling call
0709 12:28:44.202476 (+1474573us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b54006121d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:44.202631 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1332775808}
I20260709 12:28:44.267439  5555 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:44.268769  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:44.269984  5705 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260709 12:28:44.191205 15068 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.190135 15148 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.272693  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:44.276286  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.254237 15254 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:44.280725 15254 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.283716 15254 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.285550  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.259953 14802 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:44.287175  5631 raft_consensus.cc:2368] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
W20260709 12:28:44.293507 15155 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.707s	user 0.006s	sys 0.011s
I20260709 12:28:44.286496  5556 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:28:44.296713  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.297955 14978 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.298707 15220 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:44.225168 15149 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.296160  5555 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:44.272734 15117 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:28:44.323012  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.324136  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.324741  5555 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:44.221480 15165 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.227301 15184 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.300427  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:44.320845  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.328271  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.328645  5560 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 31: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:44.327077  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.327507  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:44.329003  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:44.329351  5554 raft_consensus.cc:2410] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:28:44.278472 15246 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:44.331314  5559 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 28: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:44.329286  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.333482  5556 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:44.334573  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.334731  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.335810  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.335983  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.336407  5555 raft_consensus.cc:2376] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25. Re-sending same reply.
I20260709 12:28:44.337180  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.337321  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.338330  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.338905  5552 raft_consensus.cc:2376] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25. Re-sending same reply.
I20260709 12:28:44.339749  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.341792 15068 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.315338 14802 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.345822 15184 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.346251 14802 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.346545 15201 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:44.347007 15201 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:44.347361 15201 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:44.348066 15158 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.267212 14864 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.349810 15165 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.350575 14864 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.351008  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:44.352155 15142 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.562s	user 0.009s	sys 0.000s
I20260709 12:28:44.353101 15168 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:44.353578  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:44.354725  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:44.354992 14787 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:44.355743 14787 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:44.356606  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:44.357326  5629 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 22: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:44.358029 15148 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:44.358778 15222 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.006s	user 0.005s	sys 0.000s
I20260709 12:28:44.359601 15190 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:44.360863  6272 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: [31.21-31.21]
I20260709 12:28:44.361076  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:44.362623  5556 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: []
W20260709 12:28:44.363788 15225 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.878s	user 0.000s	sys 0.002s
I20260709 12:28:44.365617 15031 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:44.366290  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1904) took 1793 ms. Trace:
I20260709 12:28:44.367183 15251 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.367686  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3819) took 1568 ms. Trace:
I20260709 12:28:44.368111  5704 rpcz_store.cc:276] 0709 12:28:42.798933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:42.806279 (+  7346us) service_pool.cc:224] Handling call
0709 12:28:42.806752 (+   473us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:42.807262 (+   510us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:28:42.807304 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:42.807888 (+   584us) log.cc:844] Serialized 26 byte log entry
0709 12:28:43.041830 (+233942us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:28:43.041872 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:43.041946 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:44.358153 (+1316207us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:44.367393 (+  9240us) raft_consensus.cc:1685] finished
0709 12:28:44.367419 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:44.367625 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":110,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":26116,"spinlock_wait_cycles":41392896,"thread_start_us":142521,"threads_started":1,"wal-append.queue_time_us":378689}
I20260709 12:28:44.230818 15138 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.371380 15138 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.362617  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1526) took 1248 ms. Trace:
I20260709 12:28:44.372309  6272 rpcz_store.cc:276] 0709 12:28:43.114064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:43.114228 (+   164us) service_pool.cc:224] Handling call
0709 12:28:43.114546 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:44.361311 (+1246765us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:44.362360 (+  1049us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54002e3954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:44.362557 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2865651456}
I20260709 12:28:44.372991 15126 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:44.373740  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.373894 15209 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:28:44.375227  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.375722 15256 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:44.376294 15256 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:44.376478 15089 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.618s	user 0.010s	sys 0.002s
I20260709 12:28:44.377213 15246 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:44.377591 15256 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.378088 15246 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.378625  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1649) took 1280 ms. Trace:
I20260709 12:28:44.378971  5701 rpcz_store.cc:276] 0709 12:28:43.097715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:43.099213 (+  1498us) service_pool.cc:224] Handling call
0709 12:28:44.378404 (+1279191us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b54001987d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:44.378556 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1981490432}
I20260709 12:28:44.380959 15149 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.381807  5707 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:28:44.382370 15202 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.315411 15155 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.384253  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3823) took 1273 ms. Trace:
I20260709 12:28:44.352114  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.379319 15246 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:44.387456  5707 rpcz_store.cc:276] 0709 12:28:43.110531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:43.110695 (+   164us) service_pool.cc:224] Handling call
0709 12:28:43.127572 (+ 16877us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:44.382245 (+1254673us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:44.383777 (+  1532us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b5400066154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:44.384145 (+   368us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2883853568}
I20260709 12:28:44.392067  5560 raft_consensus.cc:2410] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 32 because replica has last-logged OpId of term: 29 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:28:44.378990 15256 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:44.397791 15157 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.375876  5558 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:44.398164 15235 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.398609 15273 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:44.399250  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.399514 14978 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:28:44.369437 15227 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.652s	user 0.011s	sys 0.000s
I20260709 12:28:44.399731  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:44.408046  5554 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:44.400866  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:44.401196  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.410365  5556 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:28:44.401400 15155 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.366734  5627 rpcz_store.cc:276] 0709 12:28:42.573072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:42.573188 (+   116us) service_pool.cc:224] Handling call
0709 12:28:42.573492 (+   304us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:42.882828 (+309336us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:42.882875 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:42.938765 (+ 55890us) log.cc:844] Serialized 26 byte log entry
0709 12:28:43.117693 (+178928us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:43.119502 (+  1809us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:43.863755 (+744253us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b540009e7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:43.863820 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:44.363158 (+499338us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:44.366047 (+  2889us) raft_consensus.cc:1685] finished
0709 12:28:44.366073 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:44.366227 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":202627,"raft.queue_time_us":210361,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":7236,"spinlock_wait_cycles":298913536,"thread_start_us":93498,"threads_started":1,"wal-append.queue_time_us":403073}
W20260709 12:28:44.405154  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 77749.4us
I20260709 12:28:44.412931  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:44.340298  5552 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:44.406718  5555 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:44.409612 14978 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:44.416517  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 249912us
I20260709 12:28:44.421208  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.421551  5558 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:44.424072  5558 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:28:44.406618 14864 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:44.429921  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:44.431280  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.435420  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.437839  5560 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:44.411947 15057 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:44.443112  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:44.441355 15157 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:44.460275  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:44.461845  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:44.462473  5635 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:44.463888  5631 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:44.464834  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:44.465936  5629 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:44.437084 15272 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:44.411767 15115 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.032s
I20260709 12:28:44.412130 15227 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.473582  5554 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 32 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:28:44.474256  5552 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
I20260709 12:28:44.447515 15220 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:44.477939  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.447522 15185 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:44.482861  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.412144 15273 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.485548  5554 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:44.485707  5629 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:44.469774 15157 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:44.446724 14894 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.453284 15138 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:28:44.493391  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:44.472822 15202 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:44.453621 14976 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
W20260709 12:28:44.495637  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:44.453971 15089 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.034s
I20260709 12:28:44.469856 15057 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.019s
I20260709 12:28:44.482117 15167 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:44.502496  5554 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:44.503813  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:44.504601  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.506049  5554 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:44.506554  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3559) took 1018 ms. Trace:
I20260709 12:28:44.506914  5634 rpcz_store.cc:276] 0709 12:28:43.488571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:43.884154 (+395583us) service_pool.cc:224] Handling call
0709 12:28:44.506381 (+622227us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b540018d654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:44.506497 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":286598016}
I20260709 12:28:44.507809  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.509202  5554 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:44.510039  5633 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260709 12:28:44.511119 14903 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.515388 15166 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:44.515972 15166 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.516383 15235 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:44.518244 15166 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:44.518515  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1681) took 1389 ms. Trace:
I20260709 12:28:44.519605  5561 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
W20260709 12:28:44.521219  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:44.523187 15269 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.299s	user 0.001s	sys 0.002s
I20260709 12:28:44.523789 15265 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:28:44.476241 15272 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:44.524320  6276 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1668) took 2074 ms (client timeout 1663 ms). Trace:
I20260709 12:28:44.448508 14978 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:44.552917  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49199.4us
W20260709 12:28:44.604547  6276 rpcz_store.cc:269] 0709 12:28:42.449945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:42.455137 (+  5192us) service_pool.cc:224] Handling call
0709 12:28:44.524032 (+2068895us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54006383d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:44.524212 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2732624640}
I20260709 12:28:44.563386  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:44.565213  6277 rpcz_store.cc:276] 0709 12:28:43.129237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:43.231656 (+102419us) service_pool.cc:224] Handling call
0709 12:28:43.236171 (+  4515us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:43.738067 (+501896us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:44.518283 (+780216us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b54004c7a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:44.518457 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1152816000}
I20260709 12:28:44.601893  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:44.606500  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:44.558174 15249 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:28:44.559916 15220 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:44.607817  5554 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:44.609556 15166 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:44.611385 15275 log.cc:927] Time spent T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20260709 12:28:44.502442 15202 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.601191 14903 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:44.612747  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:44.613340 15257 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:44.613941 15174 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:44.562461 14887 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:44.692050  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:44.690428  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:44.567993 15209 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.047s
I20260709 12:28:44.615953 15190 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:44.694861 15227 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.694983 15190 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.695472 14887 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.697635 15190 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:44.616533  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53207.1us
I20260709 12:28:44.608929  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:44.617439  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.699671  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:44.617782 15273 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.618103 14978 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:44.560428 14949 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.623123  5558 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
W20260709 12:28:44.634783  5703 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1637) took 1895 ms (client timeout 1528 ms). Trace:
I20260709 12:28:44.630534 15155 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:44.604138 15265 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 21 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:44.525233 15167 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.561561 15148 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
I20260709 12:28:44.667582  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:44.667912  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:44.552912 15201 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.685220  5635 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
W20260709 12:28:44.715397  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 106548us
I20260709 12:28:44.715745 14979 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:28:44.704126 15252 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:44.619686  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:44.687966  5708 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 35 because replica has last-logged OpId of term: 33 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 19.
I20260709 12:28:44.689292  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:44.679360  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 51146.1us
I20260709 12:28:44.719070  5704 raft_consensus.cc:2368] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26.
I20260709 12:28:44.564620 15157 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.699153  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:44.703816  5709 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
W20260709 12:28:44.707217  5703 rpcz_store.cc:269] 0709 12:28:42.733421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:42.739807 (+  6386us) service_pool.cc:224] Handling call
0709 12:28:44.628727 (+1888920us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b5400324454. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:44.628851 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3332056960}
I20260709 12:28:44.697863 14887 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:44.711921  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44438us
I20260709 12:28:44.694219 15174 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:44.712970  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:44.726275  5705 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:44.713233  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:44.679131 15166 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:44.723222  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 103664us
I20260709 12:28:44.710975 15155 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:44.724664  5526 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.098s (SENT) [suppressed 22 similar messages]
I20260709 12:28:44.724329  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:44.728863  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:44.732611  5627 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:44.690172 14903 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.734354  5599 leader_election.cc:400] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26.
I20260709 12:28:44.734805  5599 leader_election.cc:403] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:28:44.717043 15227 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:44.735790  5703 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
I20260709 12:28:44.736295  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:44.708809 15115 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.021s
I20260709 12:28:44.744799  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:44.721330 15220 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:44.726739 15165 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:44.746932  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:44.733120 15155 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:44.749183  5634 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 31: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:44.757860  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:44.734243 14949 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:28:44.729797 15190 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:44.760538  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:44.740576 15168 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:44.763710  5705 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:44.768611  5709 raft_consensus.cc:2410] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 29 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:28:44.724336 15252 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.741039 15227 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:44.750279 15115 consensus_queue.cc:1243] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 25.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:28:44.808105  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:44.813082  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:44.813905  5629 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:28:44.755035 15272 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.771008 15273 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:44.811518  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:44.822654  5557 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:28:44.769476 14949 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 20 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:44.820447  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:44.739686 15174 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.740290 14979 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.069s
I20260709 12:28:44.813048 15227 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:44.824819  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:44.772640 15190 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:44.773078 15165 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:44.826932  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 102293us
I20260709 12:28:44.827535  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:44.837353  5707 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:44.828019  5702 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:44.832912  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 105102us
I20260709 12:28:44.810461 15260 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:44.778644  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.810853 15184 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:44.745488 15265 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:44.825495 14787 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:44.806962 15117 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 34.23, Next index: 24, Last known committed idx: 21, Time since last communication: 0.028s
I20260709 12:28:44.864802  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:44.877331  5634 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:44.849920  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:44.876216  5705 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: [26.19-26.19]
W20260709 12:28:44.878991  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 118563us
I20260709 12:28:44.881070  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:44.814347 15241 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.000s	sys 0.022s
I20260709 12:28:44.882083  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:44.897354  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:44.851056 14903 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:44.901847  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:44.899314 15241 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.000s	sys 0.023s
I20260709 12:28:44.840459 15190 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:44.844908 15174 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:44.843709 15136 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:44.909523  5702 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
W20260709 12:28:44.904181  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 175380us
I20260709 12:28:44.850515 15260 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:44.916908  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:44.933290  5635 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:44.881184 15167 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.919530  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:44.934311  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:44.932196  5634 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
W20260709 12:28:44.900821 15279 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.193s	user 0.000s	sys 0.002s
I20260709 12:28:44.883203 15202 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.902940 15274 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:44.938905 15136 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:44.913460 15256 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.507s	user 0.013s	sys 0.000s
I20260709 12:28:44.900746 15257 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:28:44.949601 15281 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:28:44.916148 15265 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:44.959426 15185 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:44.843861 15220 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:44.960258  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 116860us
I20260709 12:28:44.964022  5557 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:44.970122  5553 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:28:44.974900  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:44.897743 14787 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:44.968605 15201 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:44.977926  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.978602  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:44.979565  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:44.993465  5557 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 29: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:44.957994 15280 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:45.004711 15280 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:45.004951  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:45.005259 15280 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:44.979656  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:44.969348 15241 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:44.972424 15185 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:44.913319 14949 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.002471 15166 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:45.028642 14835 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:45.008805 15241 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:44.962025 15165 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.045274  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:45.017465 15274 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:45.046067  5636 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
W20260709 12:28:45.026508 15246 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.620s	user 0.014s	sys 0.000s
I20260709 12:28:45.050742 15284 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:45.051368 15284 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:45.052273 15284 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:45.054528  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:45.063447  5559 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:45.065876  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3315) took 1702 ms. Trace:
I20260709 12:28:45.066414  5633 rpcz_store.cc:276] 0709 12:28:43.363462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:43.984710 (+621248us) service_pool.cc:224] Handling call
0709 12:28:44.509808 (+525098us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:44.510496 (+   688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:45.065628 (+555132us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b540018d650. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:45.065794 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1206898048}
I20260709 12:28:45.068024 15287 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:45.068924 15287 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.072120 15287 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:45.082101  5701 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:28:45.036077 14949 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:28:45.088178  5526 leader_election.cc:400] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
I20260709 12:28:45.088636  5526 leader_election.cc:403] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:28:45.090190  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:45.040702 15289 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:45.085690  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:45.017681 15288 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.063s
I20260709 12:28:45.044441 15241 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:45.090063  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:45.093618  6277 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 21. Current term is 22. Ops: []
I20260709 12:28:45.045639 15252 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:45.098157  6276 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: []
I20260709 12:28:45.113526  5633 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:28:45.113657  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:45.114534  6277 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:45.114635  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:45.100498  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:45.109552  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:45.109912  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 130467us
I20260709 12:28:45.115674  5636 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:45.044664 15166 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.055s
I20260709 12:28:45.018355 15260 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.083645 15274 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:45.165833  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:45.166241  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3673) took 1020 ms. Trace:
I20260709 12:28:45.119614  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:45.166477  6276 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
W20260709 12:28:45.167461  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47949.8us
I20260709 12:28:45.117347  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:45.157028  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 56766.6us
W20260709 12:28:45.170392  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 53151.1us
I20260709 12:28:45.159920  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:45.094185 15202 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
I20260709 12:28:45.101518 15293 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:28:45.173930  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:45.171577  6272 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:45.180500  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:45.172623  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:45.182957  6274 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 22: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:45.166692  5552 rpcz_store.cc:276] 0709 12:28:44.146227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:44.437147 (+290920us) service_pool.cc:224] Handling call
0709 12:28:44.474073 (+ 36926us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:44.474622 (+   549us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:45.166012 (+691390us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b540041a0d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:45.166176 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":52989696}
I20260709 12:28:45.115442 15136 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:45.097075 15167 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.074s
I20260709 12:28:45.188140  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:45.195176  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:45.191613  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:45.082497 15185 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.170886 15287 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:28:45.203976  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:45.168479 15292 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:45.194350  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:45.198525  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:45.201903  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:45.209048  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 193375us
I20260709 12:28:45.197861  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:45.209844  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:45.206588  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:45.210227  6277 raft_consensus.cc:2410] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 31 because replica has last-logged OpId of term: 30 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 21.
I20260709 12:28:45.205689  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:45.187711 14982 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:45.179183 15257 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:45.211205  6274 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:45.212138  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:45.217810  5556 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:28:45.220031 15185 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:45.221357  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:45.229274  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:45.230075  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55057.3us
I20260709 12:28:45.231345  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:45.231352  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 141360us
I20260709 12:28:45.201686 15125 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:45.232343  5560 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:45.235172  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:45.224084 15257 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:45.239763 15257 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:45.212378 15136 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:45.240855 15136 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:28:45.241194 15283 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.196s	user 0.000s	sys 0.001s
I20260709 12:28:45.241819 15136 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:45.242889  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:45.236558  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:45.237010  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:45.198485 15295 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:45.244836  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:45.246049  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:45.269760  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:45.270972  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:45.271574  6276 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:45.266381  6272 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:45.267952  5703 raft_consensus.cc:2368] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
I20260709 12:28:45.187491 14835 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.041s
I20260709 12:28:45.222126 14864 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:45.279188  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:45.280580  6272 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:45.287020 15304 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:28:45.278704  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:45.290033  6236 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:45.282697  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:45.285086  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:45.292007  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:45.293474  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 107044us
I20260709 12:28:45.292239  5708 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:28:45.225831 15287 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:45.301543  6276 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:45.298396  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:45.233780 14982 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:45.305298  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:45.305775  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:45.242916 15136 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:45.305603  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:45.221369 15236 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.062s
I20260709 12:28:45.278316 15252 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:45.225502 15291 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.073s
I20260709 12:28:45.272480 15295 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:45.223080 15292 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.317988  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:45.319204  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:45.321435  6273 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:28:45.320521  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:45.322808  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:45.307498 15166 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:45.326488  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:45.328966  6273 raft_consensus.cc:2376] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28. Re-sending same reply.
I20260709 12:28:45.316290 14787 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:45.321169 15290 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:45.277417 15125 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.325917 14887 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:45.277619 15274 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.322589 15303 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:28:45.385825  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:45.386548  6276 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:45.334117 15305 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:45.321242 15252 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.337255 15290 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:45.346479 14982 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.321512 15202 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:45.323189 15167 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:45.342777  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:45.395239  6273 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 21: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:45.343119  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:45.396211  6272 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 22: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:45.316676 15299 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:28:45.315349 15295 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:45.346066 15057 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:28:45.388672  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:45.392390 15292 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:45.382813 14887 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:45.435179  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:45.429351  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:45.441087 15296 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
I20260709 12:28:45.394637 15202 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:45.394052 15290 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:45.446341  6273 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
W20260709 12:28:45.433806  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 143884us
I20260709 12:28:45.341295 15287 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.398942 15299 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:45.397416 15057 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:45.387275 15305 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.439567 14887 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:45.456144 15202 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:45.473517  6273 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:28:45.459302 15299 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:28:45.482000 15155 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.677s	user 0.013s	sys 0.017s
I20260709 12:28:45.484108 15155 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.485140 15310 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:28:45.486392 15155 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:45.487197  6276 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:28:45.488018  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:45.469034 14835 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:45.492637  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:45.493741  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:45.494763  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:45.502631  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:45.466514 15237 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 19, Time since last communication: 0.020s
I20260709 12:28:45.515920  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:45.481473 15274 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:28:45.506937 15168 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.691s	user 0.009s	sys 0.015s
I20260709 12:28:45.478096 14864 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:45.521230  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:45.517132  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 87820us
I20260709 12:28:45.514288 15309 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:45.531946  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:45.529290  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:45.543018  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:45.546196  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:45.554150  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:45.519721 15318 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:45.539494 15167 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:45.518298 14835 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.526883 15117 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:45.525275 15274 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:45.528946 15168 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26.
I20260709 12:28:45.539860 15304 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:28:45.564914  5709 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 20. (term mismatch)
I20260709 12:28:45.539351 14787 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:28:45.516114 15157 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.046s
I20260709 12:28:45.523308 15292 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.043s
I20260709 12:28:45.516269 15220 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.049s
I20260709 12:28:45.556092 15320 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:45.559960 15201 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:28:45.573037  5709 pending_rounds.cc:85] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 19
I20260709 12:28:45.579895  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:45.570660 15190 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.630s	user 0.017s	sys 0.001s
I20260709 12:28:45.583034  5709 pending_rounds.cc:107] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 28.20
I20260709 12:28:45.567958 15117 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:45.588748  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:45.601718  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 99211.1us
I20260709 12:28:45.514838 15057 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.564774 15312 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:45.566720 15318 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.582768 15320 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:45.585060 14864 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.617901 15330 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:28:45.612325 15168 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:28:45.577940 15158 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:28:45.621037 15319 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:28:45.622617 15252 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:45.622731 14835 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:45.581096 15309 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.038s
W20260709 12:28:45.627393  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:45.583478  5556 raft_consensus.cc:2410] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 27 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:28:45.631141  5709 raft_consensus.cc:2889] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:28:45.585407 14978 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.879s	user 0.010s	sys 0.007s
I20260709 12:28:45.624909 15190 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:45.639035  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 109678us
W20260709 12:28:45.640332  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 97493.3us
I20260709 12:28:45.583877 15321 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:45.582544 15125 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:45.649905 15274 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.613816 15167 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.636080 15117 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:45.649672 15057 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:28:45.639158 15305 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:45.580718 15315 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.082s
I20260709 12:28:45.670660 15157 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:45.654265 15158 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:45.726254  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:45.773207  5560 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:45.773640  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:45.774473  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:45.721997  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:45.729851  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:45.722954  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3839) took 1023 ms. Trace:
I20260709 12:28:45.750896  5524 leader_election.cc:400] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28.
W20260709 12:28:45.700098 15265 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.724s	user 0.013s	sys 0.000s
I20260709 12:28:45.761240  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:45.637125  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:45.720445 15327 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.079s	user 0.001s	sys 0.002s
W20260709 12:28:45.704378 15184 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.809s	user 0.010s	sys 0.003s
W20260709 12:28:45.707758 15227 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.859s	user 0.005s	sys 0.008s
I20260709 12:28:45.773774  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:45.775904  5524 leader_election.cc:403] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:28:45.757589  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:45.775498  5701 rpcz_store.cc:276] 0709 12:28:44.699271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:44.699418 (+   147us) service_pool.cc:224] Handling call
0709 12:28:45.722771 (+1023353us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b5400074c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:45.722897 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":876948864}
W20260709 12:28:45.758358 15280 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.753s	user 0.014s	sys 0.000s
I20260709 12:28:45.778729  5704 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:28:45.788002  6272 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:45.793429  5703 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
W20260709 12:28:45.762396 15257 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.522s	user 0.012s	sys 0.001s
I20260709 12:28:45.714120 15318 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:45.778316  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:45.731186 15190 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:45.703413 14894 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:45.724776 15321 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:45.701010 15334 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:45.843394  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:45.853449  5704 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:28:45.722335 15168 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.073s
I20260709 12:28:45.850871 15184 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:45.786090 14976 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:45.858435  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 107625us
W20260709 12:28:45.857595  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 83202us
I20260709 12:28:45.854423  5556 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:45.721508 14978 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.077s
I20260709 12:28:45.784325 14787 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:28:45.867033  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 372371us
I20260709 12:28:45.763039 15209 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:45.860162 15284 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.798s	user 0.024s	sys 0.027s
W20260709 12:28:45.859256 14949 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.765s	user 0.003s	sys 0.019s
I20260709 12:28:45.857746  6275 raft_consensus.cc:2410] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:28:45.871076  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:45.872488  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:45.799597  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 162696us
I20260709 12:28:45.865759 15325 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:28:45.786897 14887 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.854285 15335 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:28:45.875145  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:45.858572 15280 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.714079 15320 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:45.864779 15185 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.632s	user 0.012s	sys 0.009s
I20260709 12:28:45.868096 14864 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:45.921592 14864 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:45.921988 14864 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:45.925037  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:45.925328 15348 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:45.925962 15348 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.928296 15280 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:45.886968 15332 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:45.908569  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:45.932298  6275 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: [31.21-31.21]
W20260709 12:28:45.934099  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63117.1us
I20260709 12:28:45.886783  5708 raft_consensus.cc:2410] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
W20260709 12:28:45.882028 15284 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.820s	user 0.024s	sys 0.028s
I20260709 12:28:45.777740 15157 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.873694 14787 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:45.942050  5561 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:45.929157 15305 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:45.886289 15325 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:45.918462 15282 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.918s	user 0.001s	sys 0.001s
I20260709 12:28:45.926355  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:45.949998  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:45.868996 15292 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:45.860595 15257 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.938660  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:45.860620 15227 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.883415 15209 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.881464 14976 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.945920  6277 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:45.946388 15351 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28.
I20260709 12:28:45.870415 15334 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.959959 15334 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:45.960556  5558 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:45.944591 15321 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.963594  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3320) took 1510 ms. Trace:
I20260709 12:28:45.954316 15292 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.964783  5709 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:45.963995  5627 rpcz_store.cc:276] 0709 12:28:44.452926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:44.453112 (+   186us) service_pool.cc:224] Handling call
0709 12:28:44.455549 (+  2437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:44.734403 (+278854us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:45.963354 (+1228951us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b5400242054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:45.963536 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":636638592}
I20260709 12:28:45.945012  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:45.933781 15332 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.957052 15325 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.960536 15334 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:45.966835  5561 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:45.904862 15315 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:45.931576 15290 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.886133 15158 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:45.974774 15339 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:28:45.960458 14976 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:45.974838 15292 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:45.983134 15334 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:45.975666 15273 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:45.984197 15273 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:45.985342  5555 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:28:45.985219 15273 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:45.980600 14787 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:45.986845 15273 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:45.991315 15165 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:28:45.998816 15202 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.516s	user 0.009s	sys 0.000s
I20260709 12:28:45.999293 15202 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.000377  5557 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:28:46.002019 15348 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:46.005483 15136 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.693s	user 0.014s	sys 0.000s
I20260709 12:28:46.006435  5559 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:28:46.006261 15315 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:46.007476 15241 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.846s	user 0.019s	sys 0.000s
I20260709 12:28:46.007866 15241 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.010161 15241 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:46.011853  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:46.013696  5559 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
W20260709 12:28:46.014537 15346 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.094s	user 0.000s	sys 0.002s
I20260709 12:28:46.015262  5559 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:28:46.016299 15236 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Stepping down as leader of term 31
W20260709 12:28:46.016604 15340 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:28:46.016952 15236 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:46.018100 15236 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.019033 15342 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:28:46.019382 15236 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:46.019418 15290 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:45.953902 15337 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:46.021579 15315 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:46.023389  6273 raft_consensus.cc:2393] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 29: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:46.024327  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3354) took 1887 ms. Trace:
I20260709 12:28:46.024724  5553 rpcz_store.cc:276] 0709 12:28:44.137113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:44.423283 (+286170us) service_pool.cc:224] Handling call
0709 12:28:44.960306 (+537023us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:44.973313 (+ 13007us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:46.024127 (+1050814us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b54002279d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:46.024271 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1233994496}
I20260709 12:28:46.026345  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:46.027139 15332 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:46.027412 15337 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.029577 15257 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:46.029700 15337 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:46.032740  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:46.003726 15157 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:46.041719  5701 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:28:45.946612 15260 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.022s
I20260709 12:28:46.034417  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:46.048846  6277 raft_consensus.cc:2368] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29.
I20260709 12:28:46.044538  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:46.033430  6273 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:46.043057  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.051774  5675 leader_election.cc:400] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29.
I20260709 12:28:46.052208  5675 leader_election.cc:403] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:28:46.069742  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:46.070587  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.071321  5561 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:46.072499  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.073565  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.073763  5561 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:46.076159  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:46.078207  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:46.069204  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:46.079187  5552 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:46.070426  5553 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
W20260709 12:28:46.048982 15299 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.532s	user 0.008s	sys 0.003s
I20260709 12:28:46.081496  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:46.090148  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.002s	sys 0.013s
W20260709 12:28:46.090654  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.003s	sys 0.013s
W20260709 12:28:46.091270  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 537185us
I20260709 12:28:46.083904 15259 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:28:46.097599 15259 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:28:46.098270  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.068986 15174 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:46.103422 15174 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:28:46.104713  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:46.109694 15202 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:46.110384 15241 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:46.048098 15325 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:46.112389  5556 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:46.122440  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:46.051070 15209 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:46.114739  5552 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:46.084640  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:46.126317  5705 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
I20260709 12:28:46.116109  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.117151  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:46.100129  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:46.080506 15295 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.662s	user 0.010s	sys 0.003s
I20260709 12:28:46.119369 15335 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:46.124591  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:46.129251  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:46.084590 15315 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:46.130434  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49042.7us
I20260709 12:28:46.131516  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.131527  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:46.128387 15254 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.124715 15169 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.144663 15273 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:46.146567 15167 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:46.147358 15259 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:46.124590 14982 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.132032  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:46.153950  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:46.155189  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.085363 15342 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.067s
I20260709 12:28:46.158824  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.160008  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:46.161769 15241 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:46.179827 15241 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:46.180828 15241 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.162626 15031 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:46.163267 15117 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.184430 15031 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:46.185390 15031 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.186101 15117 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:46.163362 14982 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.155979 15321 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.008s
I20260709 12:28:46.189136 15249 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:46.157791 15284 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.032s
I20260709 12:28:46.162174 15136 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:46.191299 15136 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:46.192221 15136 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.129961 15166 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:46.163271 15273 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:46.159446 15169 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.167208  5706 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:46.162166  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:46.167610  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.168179 15259 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:46.198171  5705 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:46.160872 15174 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:28:46.197875 15169 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:46.169423  5707 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:28:46.147451 15355 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:46.157485 15254 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.223354  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:46.224213  5634 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:46.171053  5630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3550) took 3592 ms (client timeout 2098 ms). Trace:
I20260709 12:28:46.170166 15167 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:46.177484 15341 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.253s	user 0.001s	sys 0.000s
I20260709 12:28:46.189592 15249 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:46.225472  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:46.198817  5627 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:46.200574  5524 leader_election.cc:400] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
W20260709 12:28:46.232138  5630 rpcz_store.cc:269] 0709 12:28:42.578440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:42.579279 (+   839us) service_pool.cc:224] Handling call
0709 12:28:46.170814 (+3591535us) spinlock_profiling.cc:243] Waited 2.53 s on lock 0x7b5400438354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:46.170969 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5822293248}
W20260709 12:28:46.227872  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:46.206643  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 96647.7us
I20260709 12:28:46.214305  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:46.215270  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:46.235221  5636 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:46.194865 15201 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:46.192317 14982 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:46.170441 15293 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:28:46.121035 15299 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:46.228951  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62625us
W20260709 12:28:46.166968  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3845) took 1443 ms (client timeout 1601 ms). Trace:
W20260709 12:28:46.437615  5702 rpcz_store.cc:269] 0709 12:28:44.723874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:44.724036 (+   162us) service_pool.cc:224] Handling call
0709 12:28:46.166784 (+1442748us) spinlock_profiling.cc:243] Waited 96.4 ms on lock 0x7b54002f5654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:46.166909 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":221711232}
I20260709 12:28:46.438197  5560 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: []
I20260709 12:28:46.438445 15299 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:46.438719  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:46.234154  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.210181  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:46.440716  6273 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 29: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:46.441187  6277 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 28: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:28:46.441134  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 207047us
I20260709 12:28:46.169437 14887 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:46.442051 15297 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.128s	user 0.003s	sys 0.000s
I20260709 12:28:46.442553 14887 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:28:46.442631  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:46.152602 15295 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.443838  6276 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:46.238560 15201 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:46.444473  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.445255 15201 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.446524 15201 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:46.446870  5702 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:28:46.447919  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:46.448536 15329 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:46.448877  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.448536  5702 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:28:46.237643 15117 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.446921 15227 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:46.450304 15227 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:46.449961 15117 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.451416 15227 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:46.453588 15259 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:28:46.453915 15128 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:46.224169 14864 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.454491 15128 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.455375  5710 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:46.456686 14864 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:46.458637  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.461062  6276 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:46.217322 15334 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:46.429939 15355 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.478541 15057 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.478713 15355 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.479151 15057 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.481642 15355 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:46.482137 15057 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:46.487319  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:46.488134  5630 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 30: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:46.487977 15057 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:46.423890  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:46.434202 15315 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29.
I20260709 12:28:46.434904 15305 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:46.495929 15320 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:46.435801 15364 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.198s	user 0.004s	sys 0.001s
I20260709 12:28:46.233642 15332 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:46.216734 15348 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:28:46.448982 15329 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:46.457126 15220 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:46.458775 15089 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:46.459333 15125 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:28:46.460333  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:46.461344 15292 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:46.462428  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:46.463084 15128 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:46.464064 15295 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:46.195242 15273 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.467772 15117 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:46.516170 15329 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:46.470086  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:46.482733 15334 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.472813  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:46.436762 15249 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.477741 15209 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:46.482014 15301 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:28:46.427111  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:46.501895  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:46.550356  5559 raft_consensus.cc:2368] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28.
I20260709 12:28:46.503607  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.424947 15254 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:46.479157  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:46.430574  5636 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:28:46.505337  5704 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:46.516615  6274 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:28:46.525962  5634 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:46.526858  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:46.527308  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:46.530589  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.430112  5627 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:46.533335  6279 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
W20260709 12:28:46.468393  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:46.538813  6275 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:28:46.472920  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:46.561292  6237 leader_election.cc:400] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28.
I20260709 12:28:46.575755  6237 leader_election.cc:403] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260709 12:28:46.577183  5559 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: [26.15-28.16]
W20260709 12:28:46.576643  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:46.569703  6272 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:46.477790 15167 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:46.597532  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:46.601323  5560 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 29: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:46.549703 15149 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.050s
I20260709 12:28:46.602725 15149 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:46.603094 15149 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:46.599690  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:46.604557 15031 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:28:46.605063 15031 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:46.606009 15031 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.607277 15031 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:46.603869 15149 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:46.612648  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52765.5us
I20260709 12:28:46.615473  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3418) took 1439 ms. Trace:
I20260709 12:28:46.617940  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:46.613420 15342 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28.
I20260709 12:28:46.616030  5706 rpcz_store.cc:276] 0709 12:28:45.175608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:45.315095 (+139487us) service_pool.cc:224] Handling call
0709 12:28:46.166942 (+851847us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:46.200945 (+ 34003us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:46.615139 (+414194us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b54002f5650. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:46.615382 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1958007552}
I20260709 12:28:46.623464 15274 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:46.622439 14787 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:46.545856 15117 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.626566 15117 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.528962 15220 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.629959 15220 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:46.630568 15220 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:46.630950 15220 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:46.631428 15220 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:46.530090 15089 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.634063 15155 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:46.634625 15155 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.635042 15332 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:46.635389 15089 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:46.635809 15332 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:46.562219 15209 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.638089 15209 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:46.638541 15155 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:46.624166  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:46.641959  6272 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:46.642480  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:46.643257  5560 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 24: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:46.643114  6272 raft_consensus.cc:606] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:28:46.644326  6272 raft_consensus.cc:614] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:46.644721  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1506) took 1475 ms. Trace:
I20260709 12:28:46.645368  6275 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:46.645258  6280 rpcz_store.cc:276] 0709 12:28:45.169701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:45.171250 (+  1549us) service_pool.cc:224] Handling call
0709 12:28:45.193531 (+ 22281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:46.633671 (+1440140us) spinlock_profiling.cc:243] Waited 6.42 ms on lock 0x7b180035bcc0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffa66a060 00007f4ffa66abfb 00007f4ffa67564b
0709 12:28:46.633800 (+   129us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:46.633830 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:46.633917 (+    87us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:46.633949 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:46.644555 (+ 10606us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:46.644671 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13940,"raft.queue_time_us":75533,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":14061,"spinlock_wait_cycles":14765312,"thread_start_us":88254,"threads_started":1}
W20260709 12:28:46.650842  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:46.650884  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:46.610982  5629 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:28:46.665940  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:46.669816  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:46.660974  5555 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:28:46.671561  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:46.674461  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:46.654512  5635 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:46.658881  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:46.668756  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:46.667024  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:46.677920  5553 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:46.676301  5706 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:46.677546  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.679726  5629 raft_consensus.cc:2368] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23.
I20260709 12:28:46.681131  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:46.686309  5630 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
W20260709 12:28:46.687820  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 225393us
I20260709 12:28:46.689456  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:46.690789  5560 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 26: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:46.674750  5555 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 12:28:46.612295 15292 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:46.702152  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:46.704090  5561 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:46.649979 15209 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.711022  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:46.715641  5555 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
W20260709 12:28:46.728415  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:46.729523  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:46.694796 15254 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.706816 15155 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:46.748950  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:46.765324  6272 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:46.765231  5627 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 12:28:46.766631  5555 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:46.766688  6272 raft_consensus.cc:606] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:46.767864  6272 raft_consensus.cc:614] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:46.707214 15274 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:46.759133  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:46.737613  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:46.769117  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:46.780432  5702 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:28:46.780601  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:46.781934  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:46.778962  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:46.784067  5556 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:46.784766  5553 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:46.783041 15265 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.006s
I20260709 12:28:46.783169 15310 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.007s
I20260709 12:28:46.732991 15209 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.794560 15209 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:46.796001 15274 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.796795  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.798224  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.798981 15274 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:46.799311  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.800429  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:46.800679  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:46.801587  5635 raft_consensus.cc:2368] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29.
I20260709 12:28:46.802162  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:46.802894  5636 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:46.804563 15310 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:46.805012 15310 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:46.805299  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:46.806083 15310 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.809083  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:46.810216  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:46.810714  5702 raft_consensus.cc:2376] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24. Re-sending same reply.
I20260709 12:28:46.811407  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:46.812129  5559 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:28:46.815269 15201 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:46.815796 15201 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:46.816509  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:46.817152  5526 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.015s	sys 0.014s
I20260709 12:28:46.735409 15292 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:28:46.817633  5526 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.016s	sys 0.014s
W20260709 12:28:46.818038  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 602010us
I20260709 12:28:46.818202 15321 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:46.818830 15321 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:46.819015 14835 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:28:46.819882 15321 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:46.821575  5709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1659) took 1285 ms (client timeout 1636 ms). Trace:
I20260709 12:28:46.822230  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:46.822077  5709 rpcz_store.cc:269] 0709 12:28:45.536500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:45.677311 (+140811us) service_pool.cc:224] Handling call
0709 12:28:46.821351 (+1144040us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b5400278654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:46.821499 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":253046144}
I20260709 12:28:46.823472  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:46.825832  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 57205.7us
I20260709 12:28:46.852497  5636 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:46.853068  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:46.718312 15305 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.826385  5702 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:46.783499 15254 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.828938 15136 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.830899  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1651) took 1619 ms. Trace:
I20260709 12:28:46.863431  5703 rpcz_store.cc:276] 0709 12:28:45.211874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:45.309613 (+ 97739us) service_pool.cc:224] Handling call
0709 12:28:46.830675 (+1521062us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b54002f5b54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:46.830839 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1080031104}
I20260709 12:28:46.801702  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.832026 15299 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.834120  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:46.732219 15336 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.734757 15089 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:46.802809  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.871256  5636 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:28:46.872017  5552 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 28: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:46.857910  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:46.872911  5702 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:28:46.874171  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:46.874317  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:46.875795  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:46.873004  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:46.773672 15155 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:46.725524 15117 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:46.786944 15359 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:28:46.867257  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:46.840605 15374 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.800781  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.889312  5627 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 24: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:28:46.881892  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:46.859233 15252 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:46.734902 15235 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.134s
I20260709 12:28:46.876012  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.890192  5636 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:28:46.897751  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:46.897992  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.898651  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:46.899103  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:46.894148  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:46.824859 15201 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.877306 15273 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:46.922315 15273 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:46.922853 15273 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:46.896853 15252 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:46.925384 15252 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.885022 15155 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:46.934204 15254 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:46.927735 15220 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.936656  5630 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:46.938211  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.938933  5630 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:46.939965  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:46.907971 15259 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:28:46.908165  6280 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:46.909305  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.973279  5630 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:46.909873  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1663) took 1008 ms. Trace:
I20260709 12:28:46.974205  5704 rpcz_store.cc:276] 0709 12:28:45.901927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:45.908240 (+  6313us) service_pool.cc:224] Handling call
0709 12:28:46.909700 (+1001460us) spinlock_profiling.cc:243] Waited 563 ms on lock 0x7b54002e81d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:46.909814 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1294722816}
I20260709 12:28:46.911054 15227 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:46.879586 15089 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.877628 15330 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:46.897007 15374 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.917819  5633 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
W20260709 12:28:46.918803 15373 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
W20260709 12:28:46.918737  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 264217us
W20260709 12:28:47.000633  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:47.001189  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:46.920542 15148 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:47.004943 15148 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:47.007443 15148 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:46.921218 15308 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:46.941217  5705 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:47.008311 15308 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.010357  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:46.943285 15342 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:47.011439 15342 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.013778 15342 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:46.944018  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:47.014721  6272 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:47.017144 15252 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:46.944051 15201 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:46.944903 15220 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:46.879457 15336 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.894888 15288 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:47.026413 15288 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.028168  5709 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:47.027962  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:47.029036  6279 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:28:47.030464  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:47.030761  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:47.031388 15274 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:28:47.031766 15274 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:47.031924  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:47.044064  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1650) took 1832 ms. Trace:
I20260709 12:28:47.044168  5709 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 21.
I20260709 12:28:47.045686  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3681) took 2270 ms. Trace:
I20260709 12:28:47.046160  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:47.046209  5558 rpcz_store.cc:276] 0709 12:28:44.775282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:44.877960 (+102678us) service_pool.cc:224] Handling call
0709 12:28:44.878349 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:45.961224 (+1082875us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:47.045351 (+1084127us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540044f554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:47.045604 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2488382336}
I20260709 12:28:46.946903  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:47.049494  5701 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 22.
I20260709 12:28:46.949000 15158 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:46.870779 15136 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.876405 15299 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:46.868755 15305 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:47.057526 15305 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:46.958748 14887 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:47.057961 15310 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:47.058526 14887 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:47.059895 14887 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:47.064245  5704 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 34 index: 20. (term mismatch)
I20260709 12:28:47.064826  5704 pending_rounds.cc:85] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 19
I20260709 12:28:47.065091  5704 pending_rounds.cc:107] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 33.20
I20260709 12:28:47.065366  5704 raft_consensus.cc:2889] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:47.065725 15249 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:46.959360  5628 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:28:46.959705 15334 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:47.068668  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.960911 15292 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:47.070384  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:47.071450  5628 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 30: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:47.070708 15292 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:46.907049  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:47.072759  5702 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
W20260709 12:28:46.964655  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 58353.4us
I20260709 12:28:47.075403  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:46.946913  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:47.076766  5634 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:46.966889 15138 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:47.078684 15138 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.079466  5704 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:28:46.979005  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:46.992553  6274 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:47.081079  5707 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 23
W20260709 12:28:46.995865  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:47.082312  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:47.032123 15274 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:47.032773  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:47.084744 15351 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:47.085273 15351 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:47.085764 15351 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:46.891150 15057 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:47.087476 15057 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:47.044718  5708 rpcz_store.cc:276] 0709 12:28:45.211496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:45.297944 (+ 86448us) service_pool.cc:224] Handling call
0709 12:28:47.043782 (+1745838us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x7b540006f754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:47.043967 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1341689216}
I20260709 12:28:47.010870 15308 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:47.124019 15299 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:47.125468 15336 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:47.126684 15057 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
W20260709 12:28:47.128033  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44567.5us
I20260709 12:28:47.127893  6272 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:28:47.130240 15288 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:47.130842  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 98160.8us
I20260709 12:28:47.132777 15348 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:28:47.134454 15348 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:28:47.137420 15138 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:47.138562  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:47.139047 15136 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:47.139451  5628 raft_consensus.cc:2376] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35. Re-sending same reply.
W20260709 12:28:47.140013  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 115184us
I20260709 12:28:47.142951 15220 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:28:47.132973  5633 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:28:47.145326  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:47.148255  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:47.148814  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:47.148991  6272 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:47.149046  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:47.149616  6274 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
W20260709 12:28:47.150741  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 73689.2us
I20260709 12:28:47.152606  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:47.154043  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:47.157756  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:47.161202  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:47.167212  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:47.169163  5634 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:28:47.170282  5636 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:46.991487 15330 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:47.172909  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:46.998798 15259 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:28:47.264665  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 371645us
I20260709 12:28:47.228154  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:47.223583  6236 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:47.281121  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:47.281304  5636 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:47.235356 14787 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:47.165849  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:47.285765  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 62278.5us
I20260709 12:28:47.282256  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:47.283218  5559 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:28:47.255430  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1541) took 1244 ms. Trace:
I20260709 12:28:47.288736  6276 rpcz_store.cc:276] 0709 12:28:46.011397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:46.011571 (+   174us) service_pool.cc:224] Handling call
0709 12:28:47.255215 (+1243644us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b54005eef54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:47.255365 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":991548160}
I20260709 12:28:47.287469  5636 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 35: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:47.177209  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:47.293088  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 127315us
W20260709 12:28:47.295262  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:46.999193 15227 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:47.295928  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3691) took 1386 ms. Trace:
I20260709 12:28:47.298898  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:47.302361  5636 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:47.302091  5557 rpcz_store.cc:276] 0709 12:28:45.909877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:45.910988 (+  1111us) service_pool.cc:224] Handling call
0709 12:28:45.911304 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:46.000802 (+ 89498us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:47.295364 (+1294562us) spinlock_profiling.cc:243] Waited 88.9 ms on lock 0x7b54000ec4d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:47.295781 (+   417us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":204425600}
I20260709 12:28:47.305435  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:47.308929  5599 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:47.310248  5707 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:47.170946 15136 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:28:47.177896 15372 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.176s	user 0.000s	sys 0.000s
W20260709 12:28:47.255257 15373 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 12:28:47.165498 15299 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:47.324039 15227 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:47.240608 15291 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.085s
I20260709 12:28:47.326381 15291 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:47.326875 15291 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:47.319631 15337 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:47.327322 15284 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:47.327826 15337 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:47.328413 15136 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:47.328920 15136 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:47.329111 15337 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:47.327960 15284 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.331971 15284 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:47.287556 15330 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:47.334322 15284 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:47.334816 15284 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:47.336452 15332 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.700s	user 0.010s	sys 0.003s
I20260709 12:28:47.337018 15284 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:47.336895 15332 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.327744 15291 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.339107 15332 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:47.342331  5636 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 12:28:47.343391  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:47.343927  5636 raft_consensus.cc:2376] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26. Re-sending same reply.
I20260709 12:28:47.317654  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:47.174060 15220 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:47.347908 15220 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:47.348388 15220 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:47.350016  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:47.350992  5556 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 23: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:47.172698 15288 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:47.315423 15315 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:47.352821 15315 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.354712  5557 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:47.355536 15315 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:47.348049  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:47.323261 15382 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:28:47.322975 15380 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:47.357620 15380 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:47.358089 15380 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:47.358995 15125 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:47.359441 15125 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:47.359292 15380 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:47.223017 15303 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:47.356663  5552 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:47.359891 15125 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:47.323556 15126 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:28:47.360237  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:47.363797  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:47.371480  5552 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:47.307551 15175 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:47.365837  5707 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 18. (term mismatch)
I20260709 12:28:47.366283  5558 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:28:47.378013  5707 pending_rounds.cc:85] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 17
I20260709 12:28:47.378402  5707 pending_rounds.cc:107] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 26.18
I20260709 12:28:47.363095  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:47.378688  5707 raft_consensus.cc:2889] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:47.255488 15342 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.113s
I20260709 12:28:47.364869  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:47.385025  5636 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:47.386083  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:47.165252 15057 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.156s
W20260709 12:28:47.373833  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:47.387892  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:47.388293  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:47.358119 15288 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:47.383080  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:47.395367  5636 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:28:47.282306 14787 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.367338  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:47.361515 15299 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:47.404676  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:47.416002  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:47.415169  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:47.426065  5627 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 32 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 19.
I20260709 12:28:47.428120  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:47.461089  5707 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:47.380862 15175 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:47.423463  5451 catalog_manager.cc:5697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 21 to 24. New cstate: current_term: 24 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:28:47.465586  5636 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:28:47.400570 15288 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:28:47.471556  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56462.1us
W20260709 12:28:47.471931  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 394255us
I20260709 12:28:47.395869 15380 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:47.483084  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:47.488953  6276 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:28:47.460660 15128 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:47.499426  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:47.499996  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:47.510815  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:47.466419 15167 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:47.461302 15117 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:47.517343  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:47.472191 15175 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:47.489534 15138 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:47.412866 15308 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.077s
I20260709 12:28:47.470919 15202 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:47.383572 15303 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.420720 15325 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.075s
I20260709 12:28:47.534145  5557 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:28:47.412472 15321 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.092s
I20260709 12:28:47.425866 15299 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:47.585860  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:47.586413 15391 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:28:47.501443  6274 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 26: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:47.545521 14894 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:47.588570 14894 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:47.588994 14894 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:47.508028 15128 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:47.592638 15128 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:47.592971  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:47.553937  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:47.565913 15305 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:47.577342 15325 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:28:47.577575 14864 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:28:47.578884 15321 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:47.579609 15303 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:47.588915 15334 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:28:47.524742 15117 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:47.603309  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:47.620250  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:47.601408 15031 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.994s	user 0.016s	sys 0.000s
W20260709 12:28:47.607344  5710 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.026s	sys 0.017s
I20260709 12:28:47.539767 15138 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:47.570294  5630 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.596s	user 0.014s	sys 0.000s
I20260709 12:28:47.466924 15259 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:47.537683 15290 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.071s
I20260709 12:28:47.631990  6280 raft_consensus.cc:606] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:28:47.632565  6280 raft_consensus.cc:614] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:47.545042 15237 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.070s
W20260709 12:28:47.625380  5710 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.027s	sys 0.017s
I20260709 12:28:47.594864 15089 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:47.630017  5630 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
W20260709 12:28:47.637068  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44202.1us
I20260709 12:28:47.638895  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:47.648414  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:47.649188  5627 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:47.650189  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:47.613232 14864 consensus_queue.cc:1243] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 25.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:28:47.601253 15325 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:47.667627  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:47.651383  5631 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: [29.18-31.19]
I20260709 12:28:47.550395 15167 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:47.612612 15374 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:47.621932 15329 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.070s	user 0.011s	sys 0.002s
W20260709 12:28:47.624913 15383 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.369s	user 0.001s	sys 0.000s
I20260709 12:28:47.551973 15202 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.610780 15321 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:47.566313 15057 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.062s
I20260709 12:28:47.663153 15342 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:28:47.631906 15138 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:28:47.675107 15252 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.657s	user 0.009s	sys 0.002s
I20260709 12:28:47.678755 15165 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:28:47.679726 15299 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:47.623631 15117 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:47.682590  6275 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:28:47.613440 15305 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.732776  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:47.734858  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:47.736670  6274 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:28:47.669922  5630 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 21.
I20260709 12:28:47.666707 15089 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:47.737802 14787 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:47.660084 15330 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:47.756090  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:47.770210  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:47.764696  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1668) took 1565 ms. Trace:
I20260709 12:28:47.751863 15057 consensus_queue.cc:1243] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 29.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:28:47.699390 15329 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.773087  5706 rpcz_store.cc:276] 0709 12:28:46.199396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:46.694998 (+495602us) service_pool.cc:224] Handling call
0709 12:28:46.695433 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:46.706029 (+ 10596us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:28:46.706056 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:46.706601 (+   545us) log.cc:844] Serialized 26 byte log entry
0709 12:28:46.881261 (+174660us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:46.881428 (+   167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:46.882555 (+  1127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:47.764435 (+881880us) raft_consensus.cc:1685] finished
0709 12:28:47.764463 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:47.764623 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10118,"raft.queue_time_us":19098,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":14185,"thread_start_us":174272,"threads_started":1,"wal-append.queue_time_us":293973}
W20260709 12:28:47.768150  5675 leader_election.cc:336] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.525s (SENT)
I20260709 12:28:47.811029  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:47.813504  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45396us
W20260709 12:28:47.686595 14887 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.008s	sys 0.017s
I20260709 12:28:47.767199 15308 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:47.823071  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:47.828091  6274 raft_consensus.cc:2376] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28. Re-sending same reply.
I20260709 12:28:47.832852  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:47.835421  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:47.838032  5554 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 28. Current term is 29. Ops: []
W20260709 12:28:47.825342 14887 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.009s	sys 0.018s
I20260709 12:28:47.812184 14982 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:47.821386 15257 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:28:47.765970 15089 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.848001  6279 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31. Ops: []
I20260709 12:28:47.850704  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:47.835471  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 372270us
W20260709 12:28:47.853873  5704 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.012s	sys 0.029s
I20260709 12:28:47.824210 15308 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:47.894956  5704 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.012s	sys 0.029s
W20260709 12:28:47.899852  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:47.900622  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:47.901386  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:47.845932 15249 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.779s	user 0.015s	sys 0.000s
I20260709 12:28:47.910102 15089 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:47.891378 15155 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.959s	user 0.013s	sys 0.000s
W20260709 12:28:47.979629  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 458059us
I20260709 12:28:47.825588 15252 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.067s
I20260709 12:28:47.984467  6272 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bf758b480ecf41619471236195bf1838"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
W20260709 12:28:47.929309 15158 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.876s	user 0.012s	sys 0.003s
W20260709 12:28:47.938432 15389 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:28:47.940609 15308 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:28:47.958120 15201 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.937s	user 0.011s	sys 0.000s
W20260709 12:28:47.999709  5635 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.013s	sys 0.050s
I20260709 12:28:47.977519  5701 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
W20260709 12:28:47.893138 15220 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.544s	user 0.007s	sys 0.006s
I20260709 12:28:47.915720 14894 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:47.846549 15392 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.304s	user 0.000s	sys 0.001s
I20260709 12:28:47.985301  6272 raft_consensus.cc:606] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:47.932053 15149 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:47.811074 15321 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:47.989218  5602 connection.cc:256] RPC call timeout handler was delayed by -1.0248226099997737s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2194s for a response
I20260709 12:28:47.994659 14787 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:47.994338  5632 raft_consensus.cc:2410] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 27 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
W20260709 12:28:47.977923 15273 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.054s	user 0.010s	sys 0.003s
I20260709 12:28:47.960008 15202 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:47.967643 15400 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
W20260709 12:28:48.004810  5635 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.014s	sys 0.051s
I20260709 12:28:48.006717  5628 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:48.011701  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:47.992614  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:48.075484  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 82984.5us
W20260709 12:28:48.012809  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 113294us
I20260709 12:28:47.994549 15374 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.004024 15220 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:47.998680 15249 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.073s
I20260709 12:28:47.988453 15155 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:48.086827  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 75224.9us
I20260709 12:28:48.079362  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:48.097882  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:48.087685  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3347) took 1292 ms. Trace:
I20260709 12:28:48.152730  5635 rpcz_store.cc:276] 0709 12:28:46.795524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:46.802202 (+  6678us) service_pool.cc:224] Handling call
0709 12:28:46.866340 (+ 64138us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:46.866657 (+   317us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:28:47.375924 (+509267us) log.cc:844] Serialized 14 byte log entry
0709 12:28:48.087183 (+711259us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:48.087267 (+    84us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:48.087332 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.087428 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:48.087605 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12506,"prepare.queue_time_us":678675,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"thread_start_us":1194200,"threads_started":2}
I20260709 12:28:48.155424  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:48.102638  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:48.124539 14894 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:48.077152 15374 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:48.127379 15385 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.751s	user 0.001s	sys 0.002s
I20260709 12:28:48.127517 14864 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:48.007478 14887 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:48.153501  5706 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 35: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:28:48.174018  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.040s	sys 0.094s
W20260709 12:28:48.150996  5674 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.010s	sys 0.022s
I20260709 12:28:48.168016  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:48.168049  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:48.175485  5635 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:48.173207  6272 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e20e319f46504838a20c558a66692da6"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:48.176136  6272 raft_consensus.cc:606] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:48.176563  6272 raft_consensus.cc:614] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:48.150483 15209 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:28:48.173431  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 70870us
W20260709 12:28:48.174602  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.041s	sys 0.094s
W20260709 12:28:48.168277  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 70485.3us
I20260709 12:28:48.075742 15273 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:48.077700 15389 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
W20260709 12:28:48.179080  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 669857us
I20260709 12:28:48.179224  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:48.181061  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:48.066691 15292 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:48.168430 15291 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.826s	user 0.005s	sys 0.030s
I20260709 12:28:48.201838 15284 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:48.205353 15284 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.070539 14787 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:48.157485 15338 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.145s	user 0.005s	sys 0.033s
W20260709 12:28:48.206871 15338 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.194s	user 0.006s	sys 0.034s
I20260709 12:28:48.182085  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:48.184441  5674 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.012s	sys 0.022s
W20260709 12:28:48.185353  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3349) took 1389 ms (client timeout 1525 ms). Trace:
W20260709 12:28:48.208004  5629 rpcz_store.cc:269] 0709 12:28:46.796080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:46.797939 (+  1859us) service_pool.cc:224] Handling call
0709 12:28:48.185119 (+1387180us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400134f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:48.185284 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2357330688}
I20260709 12:28:48.191164  6276 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:28:48.169416 15167 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:48.189373 14887 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.191745  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:48.198323  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:28:48.188400 15315 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:48.103061 15321 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:48.200771  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:48.110423 15237 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.079s
I20260709 12:28:48.236433 14894 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:48.249181 15395 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:28:48.202898  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1547) took 1347 ms. Trace:
I20260709 12:28:48.249253 14894 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.249936  6273 rpcz_store.cc:276] 0709 12:28:46.856104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:46.856422 (+   318us) service_pool.cc:224] Handling call
0709 12:28:46.876401 (+ 19979us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:46.885039 (+  8638us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:46.885088 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:46.885745 (+   657us) log.cc:844] Serialized 26 byte log entry
0709 12:28:47.170231 (+284486us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:28:47.170276 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:47.170344 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:48.202698 (+1032354us) raft_consensus.cc:1685] finished
0709 12:28:48.202718 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:48.202846 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8136,"raft.queue_time_us":8347,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":8428,"thread_start_us":229404,"threads_started":1,"wal-append.queue_time_us":563282}
W20260709 12:28:48.203253  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:48.253331 15290 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.253531 14894 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.253886 15290 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:48.256812  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56117.5us
I20260709 12:28:48.258090  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:48.097565 15257 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:48.200712  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:48.189504  5705 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 29: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:48.208652  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3380) took 1486 ms. Trace:
I20260709 12:28:48.261583  5561 rpcz_store.cc:276] 0709 12:28:46.722635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:46.722930 (+   295us) service_pool.cc:224] Handling call
0709 12:28:46.730304 (+  7374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:46.733720 (+  3416us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:28:46.812334 (+ 78614us) log.cc:844] Serialized 14 byte log entry
0709 12:28:46.926857 (+114523us) spinlock_profiling.cc:243] Waited 33.6 ms on lock 0x7b1800357d60. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:28:47.324663 (+397806us) spinlock_profiling.cc:243] Waited 30 ms on lock 0x7b1800357d60. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:28:47.393012 (+ 68349us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:47.412280 (+ 19268us) log.cc:844] Serialized 26 byte log entry
0709 12:28:47.412540 (+   260us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:28:47.412592 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:47.412690 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:48.208300 (+795610us) raft_consensus.cc:1685] finished
0709 12:28:48.208320 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:48.208592 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2926,"prepare.queue_time_us":78231,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":3134,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":52333,"spinlock_wait_cycles":146361600,"thread_start_us":454750,"threads_started":1,"wal-append.queue_time_us":932557}
W20260709 12:28:48.261651  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61015.5us
I20260709 12:28:48.261525 15257 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:48.212831  5635 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:28:48.214391  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3443) took 1340 ms. Trace:
I20260709 12:28:48.265597  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:48.265084  5701 rpcz_store.cc:276] 0709 12:28:46.873526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:47.050437 (+176911us) service_pool.cc:224] Handling call
0709 12:28:47.050821 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:48.002870 (+952049us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.214072 (+211202us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b5400067054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:48.214312 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2130414464}
I20260709 12:28:48.266832  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:48.267067  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:48.267449  5701 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:28:48.267948 14894 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:28:48.268704 15237 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:48.268877 14894 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:48.270080  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:48.198464  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:28:48.270918  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 72520.6us
I20260709 12:28:48.174356 15305 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.199090  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:48.238766  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1520) took 1363 ms. Trace:
I20260709 12:28:48.272984 15304 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:48.272923  6275 rpcz_store.cc:276] 0709 12:28:46.875224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:46.875567 (+   343us) service_pool.cc:224] Handling call
0709 12:28:48.238578 (+1363011us) spinlock_profiling.cc:243] Waited 757 ms on lock 0x7b54005f5d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:48.238702 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1740198656}
I20260709 12:28:48.273646 15295 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.240561  5629 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: [30.20-30.20]
W20260709 12:28:48.227082 15259 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.018s	sys 0.023s
W20260709 12:28:48.225780 15291 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.871s	user 0.006s	sys 0.031s
I20260709 12:28:48.274174 15295 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:48.275063 15259 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.018s	sys 0.024s
W20260709 12:28:48.194509 15125 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.810s	user 0.007s	sys 0.006s
I20260709 12:28:48.276710 15125 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.277271  5629 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:48.278740  5629 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:48.218712  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1945) took 1441 ms. Trace:
I20260709 12:28:48.271116 14894 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.23, Last appended by leader: 23, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.225549 15252 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.282159 15252 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.283586 15303 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:28:48.285501 15252 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.280275 15125 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.280295  5554 rpcz_store.cc:276] 0709 12:28:46.777176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:46.777393 (+   217us) service_pool.cc:224] Handling call
0709 12:28:46.777765 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:47.838580 (+1060815us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.218378 (+379798us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54001be254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:48.218626 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2437821312}
I20260709 12:28:48.286564 15295 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.271253 15321 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.016s
I20260709 12:28:48.286952 15125 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:48.287510 15321 consensus_queue.cc:1243] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 21.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:28:48.287753 15125 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:48.273669 15304 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:48.289367 15125 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.291051 15290 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.290055 15304 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:48.201638 15202 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.091s
W20260709 12:28:48.294833 15136 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.965s	user 0.011s	sys 0.000s
I20260709 12:28:48.295432 15136 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:48.297573 15310 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.239s	user 0.011s	sys 0.004s
I20260709 12:28:48.290645 15257 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:48.298318 15257 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:48.299209 15201 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:48.300503 15201 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:28:48.269189 15237 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:48.301874 15237 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:28:48.287753 15175 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.737s	user 0.000s	sys 0.008s
I20260709 12:28:48.305335 15175 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.306731 15089 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:48.307628 15089 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:28:48.308440 15089 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:48.208069 14787 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.313024  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:48.314834 15374 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.315656 15374 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:28:48.212124 15292 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.321157  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3714) took 1488 ms. Trace:
I20260709 12:28:48.321753  5555 rpcz_store.cc:276] 0709 12:28:46.833114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:46.833868 (+   754us) service_pool.cc:224] Handling call
0709 12:28:46.834247 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:46.834973 (+   726us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:46.835008 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:47.299712 (+464704us) spinlock_profiling.cc:243] Waited 7.97 ms on lock 0x7b2c000fb608. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:28:47.300299 (+   587us) log.cc:844] Serialized 26 byte log entry
0709 12:28:47.774590 (+474291us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:28:47.774633 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:47.774700 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:48.320497 (+545797us) raft_consensus.cc:1685] finished
0709 12:28:48.320536 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:48.321021 (+   485us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2642,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"spinlock_wait_cycles":18356992,"thread_start_us":452328,"threads_started":1,"wal-append.queue_time_us":980394}
W20260709 12:28:48.076583 15274 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.993s	user 0.017s	sys 0.000s
I20260709 12:28:48.335464  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1935) took 1440 ms. Trace:
W20260709 12:28:48.337033  5632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3330) took 1810 ms (client timeout 2103 ms). Trace:
I20260709 12:28:48.337357  5630 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 19. (term mismatch)
I20260709 12:28:48.338217  5630 pending_rounds.cc:85] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 18
W20260709 12:28:48.337682  5632 rpcz_store.cc:269] 0709 12:28:46.526918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:46.527071 (+   153us) service_pool.cc:224] Handling call
0709 12:28:48.336829 (+1809758us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b540018b854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:48.336979 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3271184768}
I20260709 12:28:48.338507  5630 pending_rounds.cc:107] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 26.19
I20260709 12:28:48.338932  5630 raft_consensus.cc:2889] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:48.339486 15259 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.340029 15259 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.342293 15259 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.312849 14787 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Stepping down as leader of term 30
W20260709 12:28:48.234275 15351 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.148s	user 0.008s	sys 0.009s
I20260709 12:28:48.290028  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:48.335888  5631 rpcz_store.cc:276] 0709 12:28:46.895529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:47.333477 (+437948us) service_pool.cc:224] Handling call
0709 12:28:47.651030 (+317553us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:47.669158 (+ 18128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.335258 (+666100us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b54001f2550. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:48.335402 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":719163136}
I20260709 12:28:48.203761 15249 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:28:48.354811  5710 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.026s	sys 0.025s
I20260709 12:28:48.298288 15336 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.204273 15329 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:48.352739  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:48.352182  5701 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:48.232005 15330 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:48.354961  5708 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.009s	sys 0.041s
W20260709 12:28:48.352164  5673 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.047s	sys 0.073s
I20260709 12:28:48.316100 15374 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:48.244603 15397 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.333441 15167 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:48.364480  5710 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.027s	sys 0.025s
W20260709 12:28:48.353190 15227 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.015s	sys 0.035s
I20260709 12:28:48.243247 15209 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.123s
I20260709 12:28:48.373824  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:48.373687  5703 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.844s	user 0.018s	sys 0.035s
W20260709 12:28:48.375437  5708 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.831s	user 0.010s	sys 0.041s
W20260709 12:28:48.395337  5703 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.880s	user 0.018s	sys 0.035s
W20260709 12:28:48.375873  5673 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.047s	sys 0.074s
I20260709 12:28:48.347034 14787 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:48.362365 15128 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.769s	user 0.014s	sys 0.000s
I20260709 12:28:48.399739 15128 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.378732 15374 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:28:48.401158 15138 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.647s	user 0.008s	sys 0.003s
I20260709 12:28:48.401579 15138 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.402024 15128 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.393906 15167 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:48.402848 15398 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.424s	user 0.000s	sys 0.001s
I20260709 12:28:48.404481 15167 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:48.377632 15302 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.697s	user 0.039s	sys 0.045s
I20260709 12:28:48.345953 15274 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.388926 15329 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:48.351455 15259 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:28:48.397173  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:48.410300 15387 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.397836  5701 raft_consensus.cc:2368] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29.
I20260709 12:28:48.398882 15209 consensus_queue.cc:1243] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 25.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:28:48.374462 15336 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.358057 15351 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:48.393620 15227 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.792s	user 0.016s	sys 0.035s
I20260709 12:28:48.411944 15175 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.433135  5706 raft_consensus.cc:2368] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22.
I20260709 12:28:48.435493  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:48.410840 15302 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.039s	sys 0.045s
I20260709 12:28:48.391675  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1669) took 1827 ms. Trace:
W20260709 12:28:48.427713 15380 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.936s	user 0.010s	sys 0.004s
I20260709 12:28:48.467555  5710 rpcz_store.cc:276] 0709 12:28:46.564311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:46.962298 (+397987us) service_pool.cc:224] Handling call
0709 12:28:46.962797 (+   499us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:46.963301 (+   504us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:46.963342 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:46.964151 (+   809us) log.cc:844] Serialized 52 byte log entry
0709 12:28:47.634066 (+669915us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:48.388169 (+754103us) log.cc:844] Serialized 14 byte log entry
0709 12:28:48.388743 (+   574us) log.cc:844] Serialized 14 byte log entry
0709 12:28:48.391222 (+  2479us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.391332 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:48.391388 (+    56us) raft_consensus.cc:1685] finished
0709 12:28:48.391405 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:48.391587 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2111,"raft.queue_time_us":8216,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":9721,"spinlock_wait_cycles":28734336,"thread_start_us":1385374,"threads_started":2,"wal-append.queue_time_us":857494}
I20260709 12:28:48.431439 15329 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.391434 15397 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:48.473767  5709 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.843s	user 0.009s	sys 0.038s
I20260709 12:28:48.428560 15138 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:48.467914  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:48.461615  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:48.477429  5709 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.847s	user 0.009s	sys 0.038s
I20260709 12:28:48.488818  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3440) took 1799 ms. Trace:
I20260709 12:28:48.449846 15259 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:48.489581  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:48.495366  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:48.489218  5709 rpcz_store.cc:276] 0709 12:28:46.689205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:47.047348 (+358143us) service_pool.cc:224] Handling call
0709 12:28:47.047692 (+   344us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:47.172976 (+125284us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:47.173014 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:47.230372 (+ 57358us) log.cc:844] Serialized 52 byte log entry
0709 12:28:47.595230 (+364858us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:48.487895 (+892665us) log.cc:844] Serialized 14 byte log entry
0709 12:28:48.488147 (+   252us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.488237 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:48.488282 (+    45us) raft_consensus.cc:1685] finished
0709 12:28:48.488292 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:48.488752 (+   460us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3967,"raft.queue_time_us":4258,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":7521,"thread_start_us":1117406,"threads_started":2,"wal-append.queue_time_us":479887}
I20260709 12:28:48.497871  5707 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:48.404502 14787 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.473232 14887 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.474460 15417 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:48.491806 15117 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.726s	user 0.011s	sys 0.002s
I20260709 12:28:48.450315 15387 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.489395 15310 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.489902 15220 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.484155 15295 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:48.497648 15291 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:28:48.503979 15419 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.504869 15303 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:48.519021 15155 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.535138 15136 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.543138 15303 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:48.540920 15295 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.521914 15117 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.601989 15117 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:48.604816  5705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3873) took 1600 ms (client timeout 1993 ms). Trace:
I20260709 12:28:48.605243 14982 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:48.605597  5705 rpcz_store.cc:269] 0709 12:28:47.004324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:47.010007 (+  5683us) service_pool.cc:224] Handling call
0709 12:28:48.604222 (+1594215us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54004515d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:48.604710 (+   488us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2695159040}
W20260709 12:28:48.582340 15408 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260709 12:28:48.583150 15416 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:28:48.583535 15295 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.545816 15419 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.589900 15273 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.531711 15310 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.580101 15303 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:48.591456 15329 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:48.591697 15308 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.586s	user 0.003s	sys 0.007s
W20260709 12:28:48.593063 15399 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.164s	user 0.002s	sys 0.000s
I20260709 12:28:48.595279 15237 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.634624 15237 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:48.635351  5706 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:28:48.599547 15338 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:48.609206  5705 raft_consensus.cc:1217] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Deduplicated request from leader. Original: 26.15->[28.16-28.16]   Dedup: 28.16->[]
I20260709 12:28:48.520560 15259 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.526973 15417 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.603574 15397 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.605794 14982 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:48.626937 15274 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.640015 15338 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:48.680459  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:48.637063 15308 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:48.668737 15404 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
I20260709 12:28:48.671765 15351 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.650485 15257 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:48.692157 15149 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.677s	user 0.012s	sys 0.001s
W20260709 12:28:48.693173 15414 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.186s	user 0.002s	sys 0.000s
I20260709 12:28:48.660748 15374 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.699297 15387 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.709244 15302 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:28:48.718164 15302 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:48.719153 15302 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.724397 15308 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.725221 15310 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:48.726145 15417 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:48.705448 15338 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.751155 15291 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:28:48.751933 15338 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:48.751207 15303 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.754537 15303 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:48.715037 15423 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.146s	user 0.001s	sys 0.000s
I20260709 12:28:48.728729 15419 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.722018 15325 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.755996 15325 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:48.731968 15304 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.728142 15336 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:48.746476 15374 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.724397 15292 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.763052 15304 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:48.813133  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3730) took 1286 ms. Trace:
I20260709 12:28:48.814157  5558 rpcz_store.cc:276] 0709 12:28:47.526196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:47.527423 (+  1227us) service_pool.cc:224] Handling call
0709 12:28:47.527797 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:47.746171 (+218374us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:47.746205 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:47.746258 (+    53us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:47.746292 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.812239 (+1065947us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b54003c41d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:48.812395 (+   156us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:48.812843 (+   448us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":501722496}
I20260709 12:28:48.820245  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:48.824182  5451 catalog_manager.cc:5697] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 14 to 24, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 24 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:48.833137  5627 raft_consensus.cc:2368] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24.
I20260709 12:28:48.835995  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3367) took 1260 ms. Trace:
I20260709 12:28:48.836688  5627 rpcz_store.cc:276] 0709 12:28:47.575667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:47.732344 (+156677us) service_pool.cc:224] Handling call
0709 12:28:48.835653 (+1103309us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540008bbd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:48.835853 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2476436736}
W20260709 12:28:48.839550  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:48.842878  5599 leader_election.cc:400] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29.
I20260709 12:28:48.844893  5629 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:28:48.856294  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3876) took 1428 ms. Trace:
W20260709 12:28:48.845552  5633 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.114s	user 0.007s	sys 0.006s
I20260709 12:28:48.857537  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3448) took 1701 ms. Trace:
W20260709 12:28:48.847267  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.006s	sys 0.008s
W20260709 12:28:48.858889  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.007s	sys 0.009s
W20260709 12:28:48.859566  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 589318us
I20260709 12:28:48.858244  5704 rpcz_store.cc:276] 0709 12:28:47.156457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:47.156592 (+   135us) service_pool.cc:224] Handling call
0709 12:28:47.156919 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:47.157265 (+   346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:47.157292 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:47.157779 (+   487us) log.cc:844] Serialized 26 byte log entry
0709 12:28:47.899794 (+742015us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:47.899832 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:47.899899 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:48.592377 (+692478us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:48.856764 (+264387us) raft_consensus.cc:1685] finished
0709 12:28:48.856812 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:48.857293 (+   481us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46624,"raft.queue_time_us":336,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":9022,"spinlock_wait_cycles":8774528,"thread_start_us":741701,"threads_started":1,"wal-append.queue_time_us":1233208}
I20260709 12:28:48.860970  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:48.858660  5599 leader_election.cc:403] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:28:48.849231  5631 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:28:48.647390  5705 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: [28.16-28.16]
W20260709 12:28:48.869336  5628 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3590) took 1749 ms (client timeout 1720 ms). Trace:
W20260709 12:28:48.853626  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:48.870251  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:48.857760  5633 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.156s	user 0.008s	sys 0.007s
W20260709 12:28:48.870023  5628 rpcz_store.cc:269] 0709 12:28:47.120309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:47.140668 (+ 20359us) service_pool.cc:224] Handling call
0709 12:28:48.868750 (+1728082us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b54001de2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:48.869046 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1978958080}
I20260709 12:28:48.873984  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3586) took 1977 ms. Trace:
I20260709 12:28:48.874879  5628 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: [30.20-30.20]
I20260709 12:28:48.877496  5631 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: []
W20260709 12:28:48.848631  5636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.005s	user 0.005s	sys 0.008s
W20260709 12:28:48.878300  5636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.034s	user 0.005s	sys 0.009s
I20260709 12:28:48.879576  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:48.880792  5705 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:28:48.846902  5634 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 17. (term mismatch)
I20260709 12:28:48.864464  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3877) took 1433 ms. Trace:
I20260709 12:28:48.884771  5634 pending_rounds.cc:85] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 16
I20260709 12:28:48.885457  5634 pending_rounds.cc:107] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20260709 12:28:48.886332  5634 raft_consensus.cc:2889] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:48.884887  5708 rpcz_store.cc:276] 0709 12:28:47.430545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:47.461666 (+ 31121us) service_pool.cc:224] Handling call
0709 12:28:47.462045 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:47.472667 (+ 10622us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:47.472714 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:47.486312 (+ 13598us) log.cc:844] Serialized 26 byte log entry
0709 12:28:48.395915 (+909603us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:48.395957 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.396068 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:48.863751 (+467683us) raft_consensus.cc:1685] finished
0709 12:28:48.863801 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:48.864170 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":175,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":7357,"thread_start_us":851213,"threads_started":1,"wal-append.queue_time_us":950750}
I20260709 12:28:48.865288  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3447) took 1716 ms. Trace:
I20260709 12:28:48.889009  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1937) took 1703 ms. Trace:
I20260709 12:28:48.868031  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3361) took 1589 ms. Trace:
I20260709 12:28:48.890210  5630 rpcz_store.cc:276] 0709 12:28:47.278345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:47.771290 (+492945us) service_pool.cc:224] Handling call
0709 12:28:48.336973 (+565683us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:48.339468 (+  2495us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.867448 (+527980us) spinlock_profiling.cc:243] Waited 565 ms on lock 0x7b540018b850. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:48.867808 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1300076800}
I20260709 12:28:48.874523  5633 rpcz_store.cc:276] 0709 12:28:46.896270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:47.337603 (+441333us) service_pool.cc:224] Handling call
0709 12:28:47.337945 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:47.338314 (+   369us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:28:47.701500 (+363186us) log.cc:844] Serialized 14 byte log entry
0709 12:28:48.737808 (+1036308us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:48.873288 (+135480us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:48.873416 (+   128us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:48.873547 (+   131us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.873664 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:48.873924 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":242976,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":93,"spinlock_wait_cycles":27156352,"thread_start_us":1533626,"threads_started":2,"wal-append.queue_time_us":1007052}
I20260709 12:28:48.918286  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3375) took 1063 ms. Trace:
I20260709 12:28:48.901553  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:48.935525  5634 rpcz_store.cc:276] 0709 12:28:47.855242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:48.337172 (+481930us) service_pool.cc:224] Handling call
0709 12:28:48.354787 (+ 17615us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:48.887472 (+532685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.917744 (+ 30272us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b540032bed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:48.918187 (+   443us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1129605376}
I20260709 12:28:48.905946  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3373) took 1085 ms. Trace:
I20260709 12:28:48.961464  5629 rpcz_store.cc:276] 0709 12:28:47.820781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:48.279681 (+458900us) service_pool.cc:224] Handling call
0709 12:28:48.279987 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:48.857060 (+577073us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.905535 (+ 48475us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b5400438854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:48.905859 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1297107840}
I20260709 12:28:48.906440  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:48.963467  5706 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:48.965013 14887 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:48.966094 14887 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.969643 14887 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.970281 14887 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.970992 14887 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.641564  5710 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:28:48.973816 14887 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.974165  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1673) took 1385 ms. Trace:
I20260709 12:28:48.974973 15136 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.976055 15136 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.977209 15057 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.976723 14982 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:48.978011 15057 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.978717 15136 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.979297 15136 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:48.979684 14982 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.726660 15089 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:48.974601  5710 rpcz_store.cc:276] 0709 12:28:47.588447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:48.505383 (+916936us) service_pool.cc:224] Handling call
0709 12:28:48.505787 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:48.972927 (+467140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.973723 (+   796us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b54002e81d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:48.973968 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":300445568}
I20260709 12:28:48.981734  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:48.980057 15136 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.983681 15126 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:28:48.985498 15136 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.986102 15136 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.986866 15136 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.889972  5636 rpcz_store.cc:276] 0709 12:28:47.185487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:47.469910 (+284423us) service_pool.cc:224] Handling call
0709 12:28:47.470284 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:47.470525 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:28:47.843583 (+373058us) log.cc:844] Serialized 14 byte log entry
0709 12:28:48.879738 (+1036155us) log.cc:844] Serialized 14 byte log entry
0709 12:28:48.880071 (+   333us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:48.880138 (+    67us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:48.880231 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.888641 (+  8410us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:48.888925 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":111100,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"thread_start_us":1389284,"threads_started":2,"wal-append.queue_time_us":859427}
I20260709 12:28:48.988386 15057 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.733388 15257 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:48.991590 15257 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:48.730198 15284 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:48.992846 15288 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.498s	user 0.008s	sys 0.007s
I20260709 12:28:48.993228 15252 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:28:48.993758 15288 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.995777 15284 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:48.996896  5701 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 23: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:48.998087 15284 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:28:48.937412  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:48.902974  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:48.940104  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:48.999522 15288 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.899753  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:48.945031  5554 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:28:48.857117  5703 rpcz_store.cc:276] 0709 12:28:47.427983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:47.429069 (+  1086us) service_pool.cc:224] Handling call
0709 12:28:47.484661 (+ 55592us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:47.484930 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:28:48.396858 (+911928us) log.cc:844] Serialized 14 byte log entry
0709 12:28:48.854558 (+457700us) log.cc:844] Serialized 14 byte log entry
0709 12:28:48.855126 (+   568us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:48.855251 (+   125us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:48.855348 (+    97us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.855523 (+   175us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:48.855909 (+   386us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1456,"prepare.queue_time_us":1032667,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"thread_start_us":1336844,"threads_started":2,"wal-append.queue_time_us":314318}
I20260709 12:28:48.953845  5561 pending_rounds.cc:85] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 14
I20260709 12:28:49.006421  5561 pending_rounds.cc:107] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 25.15
I20260709 12:28:48.982930 15089 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:48.989993 15149 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:48.992404 15257 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.999521 15284 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:48.945894  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:49.010020 15284 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:49.010571  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 65061.7us
I20260709 12:28:49.011811 15257 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:48.843199  5451 catalog_manager.cc:5697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 25 to 32, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 32 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:48.888365  5702 rpcz_store.cc:276] 0709 12:28:47.148391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:47.148513 (+   122us) service_pool.cc:224] Handling call
0709 12:28:47.175431 (+ 26918us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:47.175927 (+   496us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:47.175970 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:47.523125 (+347155us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b180039bb00. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a
0709 12:28:48.462646 (+939521us) spinlock_profiling.cc:243] Waited 47.6 ms on lock 0x7b2c0008bd48. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:28:48.476152 (+ 13506us) log.cc:844] Serialized 26 byte log entry
0709 12:28:48.747936 (+271784us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:48.854807 (+106871us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:48.855682 (+   875us) log.cc:844] Serialized 14 byte log entry
0709 12:28:48.856342 (+   660us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:48.864609 (+  8267us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:48.864708 (+    99us) raft_consensus.cc:1685] finished
0709 12:28:48.864726 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:48.865072 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":47286,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":22757,"spinlock_wait_cycles":157151616,"thread_start_us":377780,"threads_started":1,"wal-append.queue_time_us":240482}
I20260709 12:28:49.011780 15284 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:48.925729  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 62118.4us
I20260709 12:28:49.014223 15284 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:49.016932 15031 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:49.014235  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:49.017865 15031 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:49.018627 15257 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:28:49.022401 15031 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:49.004343  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:49.025447 15290 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:49.026216 15290 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:49.026784 15290 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:49.026823 15165 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:28:49.028741 15165 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:49.029522 15165 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:49.030665  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:49.031769  6279 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:28:49.033003 15155 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:49.033594 15155 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:49.034039 15155 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:48.768739 15302 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:49.036799 15291 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:28:49.037266 15291 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:49.038757 15291 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:48.763300 15293 raft_consensus.cc:993] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: : Instructing follower f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:28:49.041170 15293 raft_consensus.cc:1081] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Signalling peer f613ba929b1046d3b5aed909414a95bf to start an election
I20260709 12:28:49.041311  6234 leader_election.cc:400] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22.
I20260709 12:28:49.041785  6234 leader_election.cc:403] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:28:49.042783 15301 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:28:48.984190 15117 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.007s
I20260709 12:28:49.008529 15149 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:49.006807  5561 raft_consensus.cc:2889] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:49.049584 15149 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:49.005672  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:49.053021  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:49.053161 15149 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:49.054064 15291 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:28:49.054833  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49499.5us
W20260709 12:28:49.056028  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:49.032188 15165 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:48.914942  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:28:49.078397  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.819s	user 0.012s	sys 0.021s
W20260709 12:28:49.080819  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.822s	user 0.013s	sys 0.021s
W20260709 12:28:49.080025  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.843s	user 0.003s	sys 0.011s
W20260709 12:28:49.079312  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.867s	user 0.007s	sys 0.023s
W20260709 12:28:49.082408  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.846s	user 0.004s	sys 0.011s
W20260709 12:28:49.082496  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.871s	user 0.007s	sys 0.024s
W20260709 12:28:49.082984  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 891296us
I20260709 12:28:49.094063  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:49.072576 14887 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:49.097062  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:49.106531  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:49.113459  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.114488  5554 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:28:49.124996  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:49.097057  5451 catalog_manager.cc:5697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 27 to 32. New cstate: current_term: 32 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:49.135234  5702 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:49.136682  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:49.148156  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.148528  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:49.149530  5556 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:49.150928  5451 catalog_manager.cc:5697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 17 to 28, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 28 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:49.153052  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:49.152992  5451 catalog_manager.cc:5697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 22 to 26. New cstate: current_term: 26 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:49.153973  5557 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 28: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:28:49.154870  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49698.9us
I20260709 12:28:49.108759 14887 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:49.155997  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:49.163105  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:49.165522  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:49.167230  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62932.2us
I20260709 12:28:49.169588  5705 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:49.174563  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.175869  5554 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:49.178362  5703 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 33 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:28:49.178268  5708 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:28:49.179714  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:49.180380  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:49.181346 15290 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:49.185416  5558 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:28:49.194887  5634 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:49.209893  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:49.196836  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:49.167349 14887 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:49.218742  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:49.193621  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.222819  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:49.189000  5459 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 281) took 35978 ms (client timeout 15000 ms). Trace:
I20260709 12:28:49.201793  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:49.208245  5707 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:49.241047  5706 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:28:49.192935  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:49.197005  5709 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.006s	sys 0.008s
I20260709 12:28:49.261302  5557 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:28:49.188833  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:49.220541  5703 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:28:49.185963  5451 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:49.199059  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.219882  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:49.298300  6272 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:49.197254  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:49.217893  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:49.218289 15305 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:49.200974  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:49.301976  5709 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.011s	sys 0.011s
I20260709 12:28:49.190025 15430 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:49.305187 15430 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:49.297144  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:49.306532  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 1.04855e+06us
I20260709 12:28:49.307255  6276 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:49.307572  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:49.308389  5557 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:49.306583  5451 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 24 to 25, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 25 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:28:49.307610  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.292126 15128 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:49.316891  5601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.905s	user 0.005s	sys 0.009s
I20260709 12:28:49.317745  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.318640  5553 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:49.321998  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:49.322976  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:49.323706  5708 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bf758b480ecf41619471236195bf1838"
dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:42568
I20260709 12:28:49.325464  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:49.326427  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:49.327334  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:49.269762 15155 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:49.427153  5601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.239s	user 0.006s	sys 0.009s
I20260709 12:28:49.328533  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:49.329499  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:49.330945 15290 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:49.331235  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:49.195252 15301 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 21. Current term is 22.
I20260709 12:28:49.339629  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:49.316165  5705 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:49.224068 15136 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:49.346331  5559 raft_consensus.cc:2410] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 28 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:28:49.347198  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.359474  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:49.357749  6272 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:49.364864  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:49.359965 15291 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.008s
I20260709 12:28:49.369349 15089 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29.
I20260709 12:28:49.358064  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:49.370075  5555 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:49.370667  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:49.372987  5634 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:49.373157 15305 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:49.453768  5451 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 15 to 24, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 24 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:49.456540  5706 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:49.457314  5558 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:49.457733  5451 catalog_manager.cc:5697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 23 to 26, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 26 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:49.405763  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:49.460281  5633 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:49.461519  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:49.461737 15128 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:49.413892 15419 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:49.464778 15419 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:49.467108 15419 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:49.415789  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:49.418301 14887 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:49.468016  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 52353.2us
I20260709 12:28:49.468091 14887 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:49.470501 14887 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:49.422181  5556 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:49.470660  6274 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:49.471695  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:49.262466  5459 rpcz_store.cc:269] 0709 12:28:13.141758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:27.819599 (+14677841us) service_pool.cc:224] Handling call
0709 12:28:28.924296 (+1104697us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5dc090 00007f4ffa67af3a 00007f5000430e9c 00007f500052f04e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc 00007f4ffe9ef392 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:49.120100 (+20195804us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:28:42.314789 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:28:42.315707 (+   918us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:28:42.315780 (+    73us) write_op.cc:435] Acquired schema lock
0709 12:28:42.315810 (+    30us) tablet.cc:662] Decoding operations
0709 12:28:42.317771 (+  1961us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:28:42.317929 (+   158us) write_op.cc:641] Partition lock acquired for write op
0709 12:28:42.317970 (+    41us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:28:42.320028 (+  2058us) tablet.cc:701] Row locks acquired
0709 12:28:42.320114 (+    86us) write_op.cc:260] PREPARE: finished
0709 12:28:42.354487 (+ 34373us) write_op.cc:270] Start()
0709 12:28:42.453559 (+ 99072us) write_op.cc:276] Timestamp: P: 1783600122320309 usec, L: 0
0709 12:28:42.456637 (+  3078us) op_driver.cc:348] REPLICATION: starting
0709 12:28:42.477416 (+ 20779us) log.cc:844] Serialized 5415 byte log entry
0709 12:28:42.691512 (+214096us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ff75680ac 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b
0709 12:28:43.925580 (+1234068us) op_driver.cc:464] REPLICATION: finished
0709 12:28:44.309944 (+384364us) write_op.cc:301] APPLY: starting
0709 12:28:44.323225 (+ 13281us) tablet.cc:1367] starting BulkCheckPresence
0709 12:28:44.617535 (+294310us) tablet.cc:1370] finished BulkCheckPresence
0709 12:28:44.617571 (+    36us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:28:45.057587 (+440016us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:28:45.057898 (+   311us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=0,mrs_lookups=0
0709 12:28:45.057958 (+    60us) write_op.cc:312] APPLY: finished
0709 12:28:45.058327 (+   369us) log.cc:844] Serialized 129 byte log entry
0709 12:28:45.343794 (+285467us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ff75680ac 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b
0709 12:28:45.433162 (+ 89368us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:28:45.433350 (+   188us) write_op.cc:454] Released schema lock
0709 12:28:45.433422 (+    72us) write_op.cc:341] FINISH: Updating metrics
0709 12:28:45.476906 (+ 43484us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:28:45.479850 (+  2944us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1194222,"reported_tablets":49,"spinlock_wait_cycles":868965760,"child_traces":[["op",{"apply.queue_time_us":278751,"apply.run_cpu_time_us":52912,"apply.run_wall_time_us":1191196,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62647,"num_ops":14,"prepare.queue_time_us":408170,"prepare.run_cpu_time_us":198169,"prepare.run_wall_time_us":778881,"raft.queue_time_us":413889,"raft.run_cpu_time_us":29730,"raft.run_wall_time_us":206895,"replication_time_us":1384840,"spinlock_wait_cycles":108276864,"thread_start_us":1607052,"threads_started":5,"wal-append.queue_time_us":1197166,"wal-append.run_cpu_time_us":22364,"wal-append.run_wall_time_us":2559675}]]}
I20260709 12:28:49.429670  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.472344  5556 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:49.472595  5703 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:49.473479  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.473613  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.474139  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:49.474737  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.474941  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.423743  5708 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:49.475749  5556 raft_consensus.cc:2410] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 32 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 19.
W20260709 12:28:49.433961  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:49.389127  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.476475  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:49.476939  5561 raft_consensus.cc:2410] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 29 because replica has last-logged OpId of term: 28 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
W20260709 12:28:49.437111  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:49.445695  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:49.478206  5560 raft_consensus.cc:2410] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 27 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:28:49.478487  5629 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:49.439468 15351 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:49.480650 15431 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:49.480755 15351 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:49.475971  5708 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:49.481418 15431 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:49.483453  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:49.483978 15351 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:49.484205 15432 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:49.449419  5631 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:28:49.475669  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:49.483985 15431 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:49.484977 15432 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:49.485854  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:49.486631 15336 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:28:49.487020 15336 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:49.487138  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:49.487306 15432 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:49.487869  5631 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:49.488986  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:49.451241  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:49.490854  6277 raft_consensus.cc:2368] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29.
I20260709 12:28:49.452078  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.447480  5554 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:49.491048  5557 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:49.453321  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:49.377204  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1945) took 1398 ms. Trace:
I20260709 12:28:49.455808  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:49.397974  5707 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:28:49.462239  5633 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
W20260709 12:28:49.437638  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 1.24977e+06us
I20260709 12:28:49.448726  5552 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:49.449656  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:49.446808  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.451076  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:49.485972  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:49.489560  6271 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:28:49.449994  5628 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:28:49.490207 15290 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:49.478045  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:49.454255  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:49.487396 15336 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:28:49.497107  6275 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 30: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:28:49.494746  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 163600us
I20260709 12:28:49.497644  5635 raft_consensus.cc:2368] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29.
I20260709 12:28:49.497080 15149 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:49.493602  5451 catalog_manager.cc:5697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 17 to 21, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 21 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:49.471997  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:49.493093  5559 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:49.499531 15149 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:49.504137  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48475.4us
I20260709 12:28:49.536386  5701 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:49.538798  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:49.534966  5526 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:49.544525  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 219148us
I20260709 12:28:49.494586 15290 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:49.548640  6278 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:49.467360 14787 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:49.491853  5632 rpcz_store.cc:276] 0709 12:28:47.978320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:48.347972 (+369652us) service_pool.cc:224] Handling call
0709 12:28:48.348372 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:48.348811 (+   439us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:48.348852 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:48.357812 (+  8960us) log.cc:844] Serialized 26 byte log entry
0709 12:28:49.224901 (+867089us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:49.225765 (+   864us) log.cc:844] Serialized 14 byte log entry
0709 12:28:49.230078 (+  4313us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:49.235418 (+  5340us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:49.376213 (+140795us) spinlock_profiling.cc:243] Waited 46.1 ms on lock 0x7b5400070420. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e7dbe 00007f4ffa5f2930 00007f4ffa5f28d8 00007f4ffa5f2858 00007f4ffa5f2813 00007f4ffa5f1a9c 00007f50004dd2f6 00007f50004d7aa8 00007f4ffa62e7cb 00007f4ffa6312c0
0709 12:28:49.376652 (+   439us) raft_consensus.cc:1685] finished
0709 12:28:49.376793 (+   141us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:49.377014 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3905,"prepare.queue_time_us":191,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":11259,"raft.run_cpu_time_us":366,"raft.run_wall_time_us":21442,"spinlock_wait_cycles":105957760,"thread_start_us":242818,"threads_started":1,"wal-append.queue_time_us":344211}
I20260709 12:28:49.534967 15301 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:49.543524  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:49.571280  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58003.9us
I20260709 12:28:49.576056  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:49.574487  5675 leader_election.cc:336] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.602s (SENT)
I20260709 12:28:49.583789  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:49.585048  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:49.585919  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:49.578027  5702 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:49.581033  5708 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:49.585695  5703 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:49.558902 15290 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:49.600450 15429 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:28:49.604079  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 106242us
I20260709 12:28:49.625101  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:49.623631 15429 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:49.634203 15429 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:49.641144 15430 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:49.642854  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:49.643029  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:49.644555 15031 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:49.648130 15031 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:49.614502 15432 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:49.648681 15031 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:49.652478 15412 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:28:49.653121 15155 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:49.653223 15315 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:49.653579  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:49.654628  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1948) took 1381 ms. Trace:
I20260709 12:28:49.655038  5627 rpcz_store.cc:276] 0709 12:28:48.273194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:48.840659 (+567465us) service_pool.cc:224] Handling call
0709 12:28:48.841347 (+   688us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:48.842087 (+   740us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:48.842116 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:48.843262 (+  1146us) log.cc:844] Serialized 52 byte log entry
0709 12:28:49.081983 (+238721us) spinlock_profiling.cc:243] Waited 1.97 ms on lock 0x7b54001d8e20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5dc090 00007f4ffa5e96b8 00007f4ffa67292b 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c
0709 12:28:49.082071 (+    88us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:49.549961 (+467890us) log.cc:844] Serialized 14 byte log entry
0709 12:28:49.590023 (+ 40062us) log.cc:844] Serialized 14 byte log entry
0709 12:28:49.642130 (+ 52107us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b18003d5940. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:28:49.654288 (+ 12158us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:49.654373 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:49.654413 (+    40us) raft_consensus.cc:1685] finished
0709 12:28:49.654426 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:49.654572 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":117999,"prepare.queue_time_us":31864,"prepare.run_cpu_time_us":125,"prepare.run_wall_time_us":88,"raft.queue_time_us":164899,"raft.run_cpu_time_us":390,"raft.run_wall_time_us":14012,"spinlock_wait_cycles":32303744,"thread_start_us":679771,"threads_started":2,"wal-append.queue_time_us":230382}
I20260709 12:28:49.657533  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:49.658298  6271 raft_consensus.cc:2368] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30.
I20260709 12:28:49.658794  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:49.659082  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:49.659614  6275 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 31 because replica has last-logged OpId of term: 30 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
I20260709 12:28:49.660617  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:49.603940  5451 catalog_manager.cc:5697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:28:49.663681  6271 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:28:49.664788  6274 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 26: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:28:49.668488  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 232657us
I20260709 12:28:49.670583 15284 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:49.671218 15284 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:49.673492 15305 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:49.674317 15284 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:49.668824  5708 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:49.677037  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:49.677029 15351 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:28:49.679352  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 120412us
I20260709 12:28:49.644587 15431 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:49.680503  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:49.680328 15431 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:49.681321  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 240010us
I20260709 12:28:49.682476  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:49.676285 15334 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:28:49.682631 15431 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:49.589697 15301 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:49.690670  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:49.693446  6278 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 23.
I20260709 12:28:49.582531 15128 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:49.698348  5629 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
W20260709 12:28:49.699581  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:49.699891  6273 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
W20260709 12:28:49.700681  6279 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.038s	sys 0.074s
W20260709 12:28:49.711179  6279 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.039s	sys 0.074s
I20260709 12:28:49.704416  5451 catalog_manager.cc:5697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 20 to 29, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 29 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:49.653932 15031 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:49.721627  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:49.748301  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:49.750038  6278 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
W20260709 12:28:49.752213  5636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.006s	sys 0.036s
W20260709 12:28:49.752707  5636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.007s	sys 0.036s
I20260709 12:28:49.753446  5708 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: [27.18-27.18]
I20260709 12:28:49.752369  6274 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:28:49.759297  5705 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
W20260709 12:28:49.761162  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 78835.9us
I20260709 12:28:49.770241  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:49.771178  5707 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:49.771512  5701 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:49.713367 15430 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:49.779575  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.782040  5708 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:49.720098 15412 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:49.775775  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:49.776227  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:49.760429  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 155353us
I20260709 12:28:49.753109 15431 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:28:49.785130  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:49.773664 15089 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:49.789561 15089 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:49.790184 15089 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:49.795785  5558 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:28:49.798422  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1677) took 2117 ms. Trace:
I20260709 12:28:49.756639 15128 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:49.770532  5451 catalog_manager.cc:5697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 18 to 21. New cstate: current_term: 21 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:28:49.813151  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:49.835851  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 193136us
I20260709 12:28:49.845911 15201 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:49.846016  5559 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:49.848212  5706 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:49.847457  5630 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
W20260709 12:28:49.862836  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:49.863766  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:49.807585  5709 rpcz_store.cc:276] 0709 12:28:47.681403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:48.516171 (+834768us) service_pool.cc:224] Handling call
0709 12:28:48.520913 (+  4742us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:48.521130 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:28:49.356481 (+835351us) log.cc:844] Serialized 14 byte log entry
0709 12:28:49.702182 (+345701us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:49.713758 (+ 11576us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:49.748721 (+ 34963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:49.768769 (+ 20048us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:49.798347 (+ 29578us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15989,"prepare.queue_time_us":39557,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":2174,"raft.queue_time_us":78462,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":12419,"thread_start_us":999667,"threads_started":2}
I20260709 12:28:49.909029  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:49.837209 15334 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:28:49.939369  6237 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.983s (SENT) [suppressed 20 similar messages]
I20260709 12:28:49.954782  6280 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:28:49.949991  5451 catalog_manager.cc:5697] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:49.938339 15288 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:49.961023 15288 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:49.963784 15288 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:49.869656 15301 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:49.965523  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46659us
I20260709 12:28:49.966645  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:49.927520 15128 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:49.983423  5451 catalog_manager.cc:5697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 19 to 26, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 26 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:28:49.953498 15391 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:49.993691  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:49.928020 15334 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:50.007206  5558 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 33 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:28:49.842032 15430 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:49.971590 15290 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:50.016402  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 206209us
W20260709 12:28:50.029960  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 90642.5us
I20260709 12:28:50.037709  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:50.038420  5558 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:50.060798  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:50.006379 15391 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:50.072299  6271 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: []
W20260709 12:28:50.073324  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:50.089527 15128 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:50.090545 15128 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:28:50.092620  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:50.093293 15149 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:50.102351  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:50.103098  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:50.103129  5559 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:50.110376  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:50.096117 15334 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:50.111785 15431 consensus_queue.cc:1175] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "38736cc545eb4389b220a3bd21d3b466" responder_term: 27 status { last_received { term: 27 index: 19 } last_committed_idx: 19 last_received_current_leader { term: 27 index: 19 } }
I20260709 12:28:50.112859 15334 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:50.115155  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:50.115804 15155 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:50.116251  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:50.117017  5556 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:28:50.117878  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:50.120086 15031 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:28:50.121714  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:50.122398  5558 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:28:50.123092  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44326.7us
I20260709 12:28:50.116302 15155 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:50.136758 15302 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:50.137085  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:50.137104 15440 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:50.140161  5556 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:50.142005  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:50.120612 15031 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:50.159361 15430 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:50.159072 15031 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:50.159225 15089 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:50.160209 15430 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:28:50.165930  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 371752us
W20260709 12:28:50.186033  5631 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.698s	user 0.011s	sys 0.000s
I20260709 12:28:50.189911  5631 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:28:50.138810 15155 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:28:50.193023  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54805.6us
I20260709 12:28:50.149212 15291 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:50.195659  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:50.201416  5555 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.750s	user 0.012s	sys 0.000s
I20260709 12:28:50.217403  5555 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:50.174224  5451 catalog_manager.cc:5697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 24 to 28. New cstate: current_term: 28 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:28:50.176505 14887 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.007s	sys 0.038s
I20260709 12:28:50.211213  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:50.220300  5552 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
W20260709 12:28:50.211810  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48461.5us
I20260709 12:28:50.232048  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:50.245877  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3740) took 1072 ms. Trace:
I20260709 12:28:50.247898  5708 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:50.251355  5558 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:28:50.255903  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:50.259935  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:50.261137 15466 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20260709 12:28:50.261422  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:50.262228  5552 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:50.263229  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:50.138110 15440 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:50.222071 14887 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.007s	sys 0.038s
I20260709 12:28:50.277490  5705 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:50.277648  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:50.232776 15136 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.028s	sys 0.029s
I20260709 12:28:50.260187  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:50.286420  5706 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:28:50.246503  5555 rpcz_store.cc:276] 0709 12:28:49.146311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:49.186125 (+ 39814us) service_pool.cc:224] Handling call
0709 12:28:50.218855 (+1032730us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:49.950953 15412 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:50.212266 15291 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:50.281962  5451 catalog_manager.cc:5697] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 23 to 24, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 24 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:28:50.283876 15136 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.029s	sys 0.030s
W20260709 12:28:50.311322  5675 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.002s	sys 0.014s
W20260709 12:28:50.311452  5633 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.003s	sys 0.012s
W20260709 12:28:50.313313  5633 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.004s	sys 0.012s
W20260709 12:28:50.313016  5675 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.003s	sys 0.014s
W20260709 12:28:50.322562  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 642327us
I20260709 12:28:50.278959 15351 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.046s
I20260709 12:28:50.325130 15475 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:50.320883  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:50.334816  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1693) took 1002 ms. Trace:
I20260709 12:28:50.337393  5560 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:50.335830  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:50.330921  5451 catalog_manager.cc:5697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 22 to 29, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 29 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:50.337437  5710 rpcz_store.cc:276] 0709 12:28:49.323743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:49.340108 (+ 16365us) service_pool.cc:224] Handling call
0709 12:28:49.387834 (+ 47726us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:49.399880 (+ 12046us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:28:49.399937 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:49.401039 (+  1102us) log.cc:844] Serialized 26 byte log entry
0709 12:28:49.694116 (+293077us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:49.694356 (+   240us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:49.694425 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:50.325729 (+631304us) raft_consensus.cc:1685] finished
0709 12:28:50.325761 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:50.325986 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":291319,"threads_started":1,"wal-append.queue_time_us":754757}
I20260709 12:28:50.349905  5451 catalog_manager.cc:5697] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 20 to 25, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 25 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:50.364120  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:50.358145  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46131.7us
I20260709 12:28:50.349563 15089 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:50.352423 15336 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.816s	user 0.003s	sys 0.009s
W20260709 12:28:50.367975 15433 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.007s	sys 0.009s
I20260709 12:28:50.400534  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:50.402311  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:50.403347 15472 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:50.408696 15089 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:50.409258 15089 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:50.409991 15453 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.305s	user 0.001s	sys 0.000s
I20260709 12:28:50.411515 15089 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:50.412063 15089 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:50.413041  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1560) took 1195 ms. Trace:
I20260709 12:28:50.417786  6273 rpcz_store.cc:276] 0709 12:28:49.217167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:49.217413 (+   246us) service_pool.cc:224] Handling call
0709 12:28:50.412799 (+1195386us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b540020c454. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:50.412948 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":458065792}
W20260709 12:28:50.418349 15465 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:28:50.414036  6274 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:50.411369  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1556) took 1250 ms. Trace:
W20260709 12:28:50.420789 15461 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
W20260709 12:28:50.401783 15433 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.007s	sys 0.009s
W20260709 12:28:50.421916 14787 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.838s	user 0.012s	sys 0.000s
I20260709 12:28:50.422890  5554 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:50.424446  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:50.424618  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:50.425354  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:50.425941  5710 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:28:50.426230  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:50.426780  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:50.427472  5706 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:50.427907  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:50.428313 15249 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:50.429036 15249 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:50.430734  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:50.431382  5703 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:50.431519  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:50.432166  5710 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: [29.19-29.19]
I20260709 12:28:50.432735  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:50.433579  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:50.435216 15249 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:50.435499 15089 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:50.434168  5451 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 18 to 23. New cstate: current_term: 23 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:50.441180 15249 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:50.371887 15290 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:50.443308 15089 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:50.364127 15432 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.051s
I20260709 12:28:50.420018  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3385) took 1111 ms. Trace:
W20260709 12:28:50.449946  6275 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.007s	sys 0.034s
W20260709 12:28:50.456138  6275 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.007s	sys 0.035s
W20260709 12:28:50.451967  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:50.455207  5702 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.012s	sys 0.007s
W20260709 12:28:50.458154  5702 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.012s	sys 0.007s
I20260709 12:28:50.449381  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:50.419647  6279 rpcz_store.cc:276] 0709 12:28:49.161190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:49.162371 (+  1181us) service_pool.cc:224] Handling call
0709 12:28:49.163098 (+   727us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:49.163701 (+   603us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:49.163734 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:49.164374 (+   640us) log.cc:844] Serialized 26 byte log entry
0709 12:28:49.711724 (+547350us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:49.711784 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:49.711886 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:50.411140 (+699254us) raft_consensus.cc:1685] finished
0709 12:28:50.411160 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:50.411290 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19673,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":117,"spinlock_wait_cycles":86912,"thread_start_us":546893,"threads_started":1,"wal-append.queue_time_us":789965}
I20260709 12:28:50.463928  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1955) took 1284 ms. Trace:
I20260709 12:28:50.466079  5636 rpcz_store.cc:276] 0709 12:28:49.179545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:49.179708 (+   163us) service_pool.cc:224] Handling call
0709 12:28:49.180178 (+   470us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:49.180738 (+   560us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:49.180771 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:49.185925 (+  5154us) log.cc:844] Serialized 52 byte log entry
0709 12:28:49.754806 (+568881us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:49.754848 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:49.754929 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:50.463683 (+708754us) raft_consensus.cc:1685] finished
0709 12:28:50.463705 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:50.463864 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":442,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"spinlock_wait_cycles":97664,"thread_start_us":568424,"threads_started":1,"wal-append.queue_time_us":1214138}
I20260709 12:28:50.455917  5561 rpcz_store.cc:276] 0709 12:28:49.308750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:49.478674 (+169924us) service_pool.cc:224] Handling call
0709 12:28:49.478999 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:49.479396 (+   397us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:49.479420 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:49.479845 (+   425us) log.cc:844] Serialized 26 byte log entry
0709 12:28:49.703027 (+223182us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:49.703306 (+   279us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:49.703581 (+   275us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:50.419810 (+716229us) raft_consensus.cc:1685] finished
0709 12:28:50.419832 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:50.419963 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2672,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"thread_start_us":222368,"threads_started":1,"wal-append.queue_time_us":768583}
I20260709 12:28:50.480930  5451 catalog_manager.cc:5697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:28:50.459662 15136 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:50.488575  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 393302us
I20260709 12:28:50.459715 15249 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:50.487798  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:50.401947 15336 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:50.400573 15291 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:50.396875 15475 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:50.530143  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.014s	sys 0.040s
I20260709 12:28:50.506953  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:50.569912  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.014s	sys 0.040s
I20260709 12:28:50.502087  5451 catalog_manager.cc:5697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 14 to 21. New cstate: current_term: 21 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:28:50.552529  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:50.607533 15478 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:50.517149 15351 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:50.608609  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:50.606163  5451 catalog_manager.cc:5697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 19 to 27, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 27 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:50.593451 15302 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:28:50.608335 15478 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:50.621181 15468 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:50.609578  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:50.555015 15440 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:50.606681 15291 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:50.633665  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 667137us
W20260709 12:28:50.634567  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 139223us
I20260709 12:28:50.635246 15334 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:50.635892  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3896) took 1315 ms. Trace:
I20260709 12:28:50.516782 15325 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.166s
I20260709 12:28:50.642452 15089 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:50.642942  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:50.661700  5451 catalog_manager.cc:5697] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 18 to 27, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 27 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:28:50.521723 15136 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:50.660425  5704 rpcz_store.cc:276] 0709 12:28:49.320578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:49.340647 (+ 20069us) service_pool.cc:224] Handling call
0709 12:28:49.358795 (+ 18148us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:49.377796 (+ 19001us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:49.377844 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:49.378661 (+   817us) log.cc:844] Serialized 26 byte log entry
0709 12:28:49.623250 (+244589us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:49.623307 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:49.623399 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:50.635634 (+1012235us) raft_consensus.cc:1685] finished
0709 12:28:50.635658 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:50.635827 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3650,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":17362,"thread_start_us":243470,"threads_started":1,"wal-append.queue_time_us":1229758}
I20260709 12:28:50.640551  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:50.624244 15302 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:50.672865 15430 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:28:50.673600 15433 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:50.674243  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:50.675169 15430 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:28:50.676910 15480 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:28:50.676790  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:50.678297  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3601) took 1349 ms. Trace:
I20260709 12:28:50.678766  5633 rpcz_store.cc:276] 0709 12:28:49.329007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:49.542007 (+213000us) service_pool.cc:224] Handling call
0709 12:28:49.570045 (+ 28038us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:49.574873 (+  4828us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:49.574919 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:49.602523 (+ 27604us) log.cc:844] Serialized 26 byte log entry
0709 12:28:50.328205 (+725682us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:50.328645 (+   440us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:50.328759 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:50.678002 (+349243us) raft_consensus.cc:1685] finished
0709 12:28:50.678035 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:50.678222 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56732,"raft.queue_time_us":2803,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":5401,"thread_start_us":594049,"threads_started":1,"wal-append.queue_time_us":787409}
W20260709 12:28:50.680867  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49630.8us
I20260709 12:28:50.683293 15431 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:50.683517  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:50.684005 15431 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:50.684602 15431 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:50.685730 15468 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:50.662819 15334 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:50.686782 15334 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:50.686923 15468 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:50.687325 15468 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:50.676092  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:50.701087  5559 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.573s	user 0.009s	sys 0.004s
I20260709 12:28:50.701822  5559 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:50.699321  5451 catalog_manager.cc:5697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index 3 to 11, term changed from 15 to 19, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4), VOTER cea66aa15be3423880843974acdbf50f (127.4.65.1) evicted. New cstate: current_term: 19 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:50.627149 15351 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:50.708209  5601 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:50.671008 15302 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:28:50.711736  5526 leader_election.cc:400] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29.
I20260709 12:28:50.745672  5526 leader_election.cc:403] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:28:50.704773 15128 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:28:50.631641 15249 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:50.767952  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 56356.3us
I20260709 12:28:50.769614  5526 leader_election.cc:400] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30.
I20260709 12:28:50.758611  5451 catalog_manager.cc:5697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 21 to 25, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 25 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:50.688241 15468 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:50.770112  5526 leader_election.cc:403] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Cancelling election due to peer responding with higher term
I20260709 12:28:50.708894 15433 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.053s
W20260709 12:28:50.776798  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 68732.1us
I20260709 12:28:50.777422 15468 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:28:50.778630 15470 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.208s	user 0.002s	sys 0.002s
I20260709 12:28:50.779078 15471 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:50.779811 14787 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:50.780206 15471 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:50.780328 14787 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:50.779397 15489 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:50.757185 15128 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:50.760900 15430 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:28:50.788432  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 342217us
I20260709 12:28:50.812673 15469 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:50.813611 15469 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:28:50.814560 15483 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:28:50.773273 15432 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29.
I20260709 12:28:50.783129 15325 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:28:50.821091 15481 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:50.798135 15128 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:28:50.804312 15484 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election lost for term 29. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30.
W20260709 12:28:50.830669  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 116845us
I20260709 12:28:50.831198 15478 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:50.782474 15336 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:50.828182  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3604) took 1498 ms. Trace:
I20260709 12:28:50.780817 15471 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:50.798892 15472 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:50.842067  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:50.846637  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 77214.3us
W20260709 12:28:50.838351 15288 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.006s	sys 0.023s
I20260709 12:28:50.817335 15475 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:50.841912  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.022s	sys 0.038s
I20260709 12:28:50.804747  5451 catalog_manager.cc:5697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 18 to 22, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 22 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:50.854647  5632 rpcz_store.cc:276] 0709 12:28:49.329844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:49.626141 (+296297us) service_pool.cc:224] Handling call
0709 12:28:49.708391 (+ 82250us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:49.779506 (+ 71115us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:49.779535 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:49.819306 (+ 39771us) log.cc:844] Serialized 26 byte log entry
0709 12:28:50.404992 (+585686us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:50.405032 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:50.405106 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:50.827724 (+422618us) raft_consensus.cc:1685] finished
0709 12:28:50.827764 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:50.828069 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2999,"raft.queue_time_us":322297,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"thread_start_us":497473,"threads_started":1,"wal-append.queue_time_us":854137}
W20260709 12:28:50.887046  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.022s	sys 0.038s
I20260709 12:28:50.832584 15430 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:50.850756 15284 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.000s	sys 0.029s
W20260709 12:28:50.902563 15284 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.000s	sys 0.029s
I20260709 12:28:50.902765 15315 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:50.904501 15472 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:50.905706 15472 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:50.908950  5451 catalog_manager.cc:5697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 22 to 27. New cstate: current_term: 27 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:50.910473  5451 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 20 to 26. New cstate: current_term: 26 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:28:50.918594 15471 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:50.919589  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:50.838666 15481 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:50.874416 15336 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:28:50.922240 15336 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:50.922640 15334 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:50.816015 15489 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:50.923219 15336 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:50.874910 15136 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:50.878314 15155 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.601s	user 0.006s	sys 0.005s
I20260709 12:28:50.879803 15351 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:50.880566 15432 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:50.881639  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:28:50.873575 15288 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.007s	sys 0.023s
W20260709 12:28:50.882134 15486 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.388s	user 0.000s	sys 0.002s
I20260709 12:28:50.882004  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:50.884402 14787 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:50.885985 15391 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:50.866394 15469 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:50.887387  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 632489us
I20260709 12:28:50.901695 15430 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:50.922554 15481 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:50.878895 15089 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:50.911811  5451 catalog_manager.cc:5697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 25 to 28. New cstate: current_term: 28 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:50.939965  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:28:50.931140  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.014s	sys 0.019s
W20260709 12:28:50.945541  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.014s	sys 0.019s
W20260709 12:28:50.946132  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 610391us
I20260709 12:28:50.953418  5451 catalog_manager.cc:5697] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:28:50.956465  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:50.958204  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 76734.8us
I20260709 12:28:50.941320  6280 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:50.930802 15155 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:50.960884  6279 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 22: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:50.947113 14787 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:50.961617  5451 catalog_manager.cc:5697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 20 to 25. New cstate: current_term: 25 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:50.965258  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1711) took 1318 ms. Trace:
I20260709 12:28:50.955205 15391 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:50.960777  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1694) took 1354 ms. Trace:
I20260709 12:28:50.952945  6236 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:50.944298 15431 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:50.943493 15432 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:50.978765  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:50.979319  6274 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: [29.21-29.21]
I20260709 12:28:50.969199  5702 rpcz_store.cc:276] 0709 12:28:49.606072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:49.696571 (+ 90499us) service_pool.cc:224] Handling call
0709 12:28:49.743395 (+ 46824us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:49.785534 (+ 42139us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:49.785579 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:49.811263 (+ 25684us) log.cc:844] Serialized 52 byte log entry
0709 12:28:50.464634 (+653371us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:50.959735 (+495101us) log.cc:844] Serialized 14 byte log entry
0709 12:28:50.960083 (+   348us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:50.960164 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:50.960508 (+   344us) raft_consensus.cc:1685] finished
0709 12:28:50.960530 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:50.960705 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":364801,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":10081,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":31247,"thread_start_us":946747,"threads_started":2,"wal-append.queue_time_us":943694}
W20260709 12:28:50.972487  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 495503us
I20260709 12:28:50.971645  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:50.987838  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:50.984131  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:50.967864 14787 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:50.978909 15031 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.015s	sys 0.026s
W20260709 12:28:50.977581 15201 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.131s	user 0.010s	sys 0.010s
W20260709 12:28:50.999610  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:50.967546  6275 rpcz_store.cc:276] 0709 12:28:49.647170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:49.660172 (+ 13002us) service_pool.cc:224] Handling call
0709 12:28:49.698387 (+ 38215us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:49.774378 (+ 75991us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:28:50.460594 (+686216us) log.cc:844] Serialized 14 byte log entry
0709 12:28:50.964781 (+504187us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:50.964860 (+    79us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:28:50.964936 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:50.965028 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:50.965186 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":593990,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":84,"raft.queue_time_us":5481,"thread_start_us":1015160,"threads_started":2}
W20260709 12:28:51.001286 15031 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.016s	sys 0.026s
I20260709 12:28:50.969136 15391 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:51.002723  5451 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 15 to 19, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 19 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:28:51.007215  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 125315us
I20260709 12:28:51.006832  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:50.998644  6276 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:51.005986  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:51.014307  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:50.976388 15481 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:51.002367 14787 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:51.011879 15495 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:51.015758 15488 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:51.010772 15136 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:51.030212  5636 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:28:51.020949  5632 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:28:51.001758 15475 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:28:51.059881  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 81182.7us
W20260709 12:28:51.020499 15201 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.174s	user 0.010s	sys 0.010s
I20260709 12:28:51.025130 15391 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:51.086792  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61864.8us
I20260709 12:28:51.001111 15315 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.059s
I20260709 12:28:51.055378 14887 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:51.093178  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:51.033147 15495 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:51.108479  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 195638us
I20260709 12:28:51.109364  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:28:51.066658 15429 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.418s	user 0.001s	sys 0.031s
I20260709 12:28:51.032246 15489 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:51.066471 15201 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:51.022483 15433 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.064s
I20260709 12:28:51.107898  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:51.092870 15249 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:51.093551 14887 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:51.120577  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:51.101608 15488 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:51.133117  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:51.111518 15495 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:51.114372 15429 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.465s	user 0.001s	sys 0.032s
I20260709 12:28:51.139288  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:51.139645  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:51.142429  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3755) took 1008 ms. Trace:
W20260709 12:28:51.146864  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 140813us
W20260709 12:28:51.147506  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 146433us
I20260709 12:28:51.118270 15249 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:51.146895  5556 rpcz_store.cc:276] 0709 12:28:50.134290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:50.311518 (+177228us) service_pool.cc:224] Handling call
0709 12:28:50.338866 (+ 27348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:50.392576 (+ 53710us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:28:50.943341 (+550765us) log.cc:844] Serialized 14 byte log entry
0709 12:28:51.141784 (+198443us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:51.141902 (+   118us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:28:51.141978 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:51.142103 (+   125us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:51.142341 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1936,"prepare.queue_time_us":692895,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"thread_start_us":631758,"threads_started":2}
I20260709 12:28:51.155797  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:51.155632 15201 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:51.157999 15488 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:51.134323 15489 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:51.163228 14887 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:51.193042  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:51.153949  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:51.131709 15334 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:51.188949  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 125089us
I20260709 12:28:51.113497 15412 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:51.196200  6276 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
W20260709 12:28:51.203243  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:51.204609  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 88256.3us
I20260709 12:28:51.205519  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:51.148801 15432 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:51.251623  5705 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
W20260709 12:28:51.277565  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 108349us
I20260709 12:28:51.279549  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:51.282897  6274 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:28:51.289670  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 133950us
I20260709 12:28:51.290098  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:51.152995 15325 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:51.296142  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 90759.4us
I20260709 12:28:51.297891  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:51.295253  5524 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:51.210412 15429 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:51.207021 15155 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:51.183832 15249 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:51.309412  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:51.253191 15288 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:51.253633 14887 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:51.306649  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:51.313855  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:51.241444 15431 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:51.190016 15489 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:51.312467 15302 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.551s	user 0.011s	sys 0.002s
I20260709 12:28:51.318781  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:51.206913 15136 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:28:51.326804  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62832.5us
W20260709 12:28:51.327904  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 173154us
I20260709 12:28:51.334441  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:51.341537  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:51.346624  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:51.349102  5710 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:51.349575  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:51.310685 15336 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:51.352766 15336 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:51.323848 15351 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:51.323486 15432 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:51.368526  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:51.363583  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:51.329906 15489 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:51.363580  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:51.369201  5703 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:28:51.365322  5710 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:51.332587 15498 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:51.400784  5526 leader_election.cc:400] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29.
I20260709 12:28:51.326771 15334 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.048s
I20260709 12:28:51.333099 15136 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:51.315116 15429 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:51.417373  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:51.366081 15351 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:51.370271 15431 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:51.400656 15472 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:51.425177  6274 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:28:51.428170  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:51.429309  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:51.407766 15128 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.541s	user 0.012s	sys 0.000s
I20260709 12:28:51.368885 15432 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:51.341037 15302 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:51.445070  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:51.417212 15136 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:51.434530  6237 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:51.448263  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:51.444651 15128 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:51.449577  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:51.450769 15515 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 12:28:51.450986 15503 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:28:51.456889 15494 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.250s	user 0.000s	sys 0.000s
I20260709 12:28:51.458134 15485 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:51.458881 15511 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:51.459477 15511 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:51.460294 15478 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:28:51.460846 15478 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:51.461838 15478 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:51.462996 15478 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:51.460407 15507 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:51.466797  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:51.469167  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 157750us
I20260709 12:28:51.431309 15351 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:51.479413  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 115935us
I20260709 12:28:51.450842 15510 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:51.446893 15432 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:51.486639  6274 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: [29.21-29.21]
I20260709 12:28:51.407279 15498 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:51.482582 15291 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:51.487471 15510 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:51.450798 15509 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:51.450769 15128 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:51.481737 15484 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:51.470484 15507 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:51.492697  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:51.489352  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 166368us
I20260709 12:28:51.504644  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:51.510274  5636 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:28:51.510361  5633 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:28:51.510686  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:51.514437  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:51.491762  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:51.494573 15508 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20260709 12:28:51.529611  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 80106.5us
I20260709 12:28:51.571229  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:51.529680  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:51.530189  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:51.530405  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:51.572606  5634 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:51.493017 15491 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:28:51.533236 15525 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:28:51.535727 15302 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:51.537648  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:51.538527  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:51.538836 15391 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:51.539225  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:51.495796 15504 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:28:51.493461 15490 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:28:51.512177 15468 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:28:51.681268 15468 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:51.681334  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:51.681819  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:51.556814  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:51.682557  6278 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:28:51.684100  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:51.685096  5627 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:51.685292  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:51.434111 15325 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:51.495143 15521 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:28:51.686684  6280 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: [29.19-29.19]
I20260709 12:28:51.568055  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:51.682235 15468 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:51.687343  5628 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:28:51.687742 15468 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:28:51.526767 15513 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.148s	user 0.001s	sys 0.001s
I20260709 12:28:51.511070 15429 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:51.688040  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:51.569649 15336 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:51.493867 15463 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:28:51.572780  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58478.4us
I20260709 12:28:51.584775  5635 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 28: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:51.513700 15511 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:51.570822 15128 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:51.658061  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 91023.4us
I20260709 12:28:51.659930  5632 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:51.682674 15433 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:51.670249  5629 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 25: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:51.690922 15412 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:51.565089 15507 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:51.526245 15510 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:51.522832 15291 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:51.563036  5703 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
W20260709 12:28:51.493268 15499 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.163s	user 0.000s	sys 0.000s
W20260709 12:28:51.653194 14787 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.598s	user 0.011s	sys 0.000s
I20260709 12:28:51.693521  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:51.666285 15031 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:28:51.729817 15288 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:51.582177 15518 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:28:51.697830  5673 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.000s	sys 0.013s
W20260709 12:28:51.791545  5673 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.000s	sys 0.014s
I20260709 12:28:51.698540  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:51.716873  6273 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:28:51.664315 15155 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:51.582245 15484 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:51.719424  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:51.728925  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 157772us
I20260709 12:28:51.730324  6279 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: []
I20260709 12:28:51.670966  5630 raft_consensus.cc:2368] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
W20260709 12:28:51.732477  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 236579us
I20260709 12:28:51.669126  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:51.762135  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:51.557292 15509 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:51.786435  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 93004.5us
I20260709 12:28:51.731192 15301 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:51.792661  5636 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:51.796207  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 266124us
I20260709 12:28:51.756331 15291 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:51.724936 15128 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:51.787843 15334 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:51.791494 15518 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:51.811435  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:51.807325  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 103396us
I20260709 12:28:51.828984  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:51.787038 15031 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:51.790787 15485 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:28:51.907683  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:51.910854  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 81994.3us
W20260709 12:28:51.813959  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 309447us
I20260709 12:28:51.914662  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:51.807600  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:51.733711 15249 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:51.828230  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:51.835956 15493 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.122s
W20260709 12:28:51.916672  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:51.788708 14787 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:51.837635 15517 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:51.837884 15315 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:51.813948 15301 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:51.840461 15530 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 12:28:51.841463 15288 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:51.819553 15538 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:51.751260 15433 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:51.845973 15128 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:51.846764 15495 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.664s	user 0.007s	sys 0.005s
I20260709 12:28:51.825911 15484 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:51.852555  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:51.853430  5628 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:28:51.855441 15431 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:28:51.875049  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:51.846632  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:51.881855 15498 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:51.883101  5636 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:51.883601 15509 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:51.883687 15291 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:51.903442 15136 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:51.836459 15391 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:51.836977  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:51.928030  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:51.918262  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 90146.8us
I20260709 12:28:51.940963  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:51.930204  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:52.020264  5632 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:28:51.906378 15031 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:51.953133  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:51.958842  5704 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 29. Ops: []
W20260709 12:28:51.924521 15430 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.059s	sys 0.139s
I20260709 12:28:51.904855 15485 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:51.962837  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48335.5us
I20260709 12:28:51.834328 15518 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:51.926687 15336 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:28:51.967109  5675 leader_election.cc:336] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.790s (SENT)
I20260709 12:28:51.968065  5634 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:51.929461 15201 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:28:51.993256  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:51.997445  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:51.949434 15472 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:51.949501 15315 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:52.020179  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 183261us
W20260709 12:28:51.990162 15475 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.924s	user 0.013s	sys 0.000s
I20260709 12:28:52.025527  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:52.030329  6276 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:52.034969  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:51.927978 15334 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:52.024571 15351 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.530s	user 0.011s	sys 0.000s
I20260709 12:28:51.909138 15539 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.186s
I20260709 12:28:52.036914  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:52.036712  5633 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:52.000257 15495 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:52.030381 15469 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.006s	sys 0.006s
I20260709 12:28:52.150005 15249 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:52.117027  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 270516us
W20260709 12:28:52.183538 15489 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.731s	user 0.009s	sys 0.003s
W20260709 12:28:52.123649  5525 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.012s	sys 0.012s
I20260709 12:28:52.228417  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3474) took 1024 ms. Trace:
W20260709 12:28:52.229465 15468 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.541s	user 0.010s	sys 0.000s
I20260709 12:28:52.223496 15507 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:52.217113 15469 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.007s	sys 0.007s
W20260709 12:28:52.031805 15430 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.059s	sys 0.144s
I20260709 12:28:52.036849 15538 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:52.232218 15507 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:52.224555 15201 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:52.234087 15201 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:28:52.235332 15478 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.772s	user 0.006s	sys 0.005s
I20260709 12:28:52.236812 15291 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:52.237185  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 211762us
I20260709 12:28:52.150884 14787 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:52.227272  5525 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.036s	user 0.012s	sys 0.020s
W20260709 12:28:52.239580  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 204694us
I20260709 12:28:52.030922 15284 raft_consensus.cc:2764] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:28:52.223543 15489 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:52.044553  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:52.230183  5632 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:28:52.232892 15538 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:52.229039  5704 rpcz_store.cc:276] 0709 12:28:51.204312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:51.207442 (+  3130us) service_pool.cc:224] Handling call
0709 12:28:51.278672 (+ 71230us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:52.027541 (+748869us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:52.228153 (+200612us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b5400323054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:52.228335 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1563596160}
I20260709 12:28:52.028538 15128 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:28:52.240079  5630 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:52.249109  6279 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 26: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:52.047716  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:52.152302  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 203354us
I20260709 12:28:52.222728  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:52.258452  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:52.035681 15536 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:52.203547 15481 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.173s	user 0.013s	sys 0.001s
I20260709 12:28:52.037200 15540 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:52.270890  5552 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:28:52.270999 15540 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:52.272646  5627 raft_consensus.cc:2368] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27.
I20260709 12:28:52.272744 15489 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:52.273590  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:52.149000 15535 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:28:52.257033  5630 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:52.037060 15485 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:52.251740  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:52.250566 15475 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:28:52.131521 15432 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.632s	user 0.011s	sys 0.000s
I20260709 12:28:52.247334 15315 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:28:52.270154  5675 leader_election.cc:336] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.791s (SENT)
I20260709 12:28:52.255970  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:52.259447  5559 raft_consensus.cc:2376] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29. Re-sending same reply.
I20260709 12:28:52.238346 15136 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:52.193440 15433 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:52.238265 15495 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:52.275888  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 234745us
W20260709 12:28:52.321312  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 65423.3us
I20260709 12:28:52.149266 15544 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:52.278505  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:52.324167  6276 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:28:52.325292  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 102665us
I20260709 12:28:52.061233  6271 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:52.328680  6271 raft_consensus.cc:606] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:52.329048  6271 raft_consensus.cc:614] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:52.323351  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:52.330036  5559 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:52.149204 15301 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:52.065088 15336 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:52.331027  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:52.284775  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:52.250967 15351 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:52.285656  5628 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:28:52.285998  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:52.297014 15249 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:28:52.289443 14887 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.962s	user 0.012s	sys 0.000s
I20260709 12:28:52.253266 15325 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:52.292622  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:28:52.133591 15516 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:28:52.257339 15468 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:52.290231 15334 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:52.355168  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 62630.7us
I20260709 12:28:52.321792  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:52.404644  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 82939us
I20260709 12:28:52.275838  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:52.280577 15430 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:52.322854  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:52.283799 15540 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:52.333982  5558 raft_consensus.cc:2410] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 27 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:28:52.290566 15489 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:28:52.283816  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:52.337452  6276 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:52.297931 15284 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.041s
I20260709 12:28:52.338515 15336 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:52.333168 15431 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:52.341503 15351 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:52.344079  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:52.423204  6276 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:46870
W20260709 12:28:52.423861  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 88949.4us
I20260709 12:28:52.347886  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:52.424825  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:52.348625 15543 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:28:52.356668  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:52.403446 15490 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:52.294646  5630 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:52.405225  5559 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:52.406843  5557 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
W20260709 12:28:52.407407  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:52.407284  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53185.1us
I20260709 12:28:52.413437 15433 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:52.437976  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:52.438618  5552 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:52.404196  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:52.413130  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:52.351774 15249 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:52.416826  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 94089.5us
I20260709 12:28:52.316596 15432 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:52.352682 15325 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:52.458400  5557 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:28:52.421022 15534 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:52.344879 15544 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:52.435627 15490 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:52.262594 15509 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.059s
I20260709 12:28:52.428879  5706 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:28:52.470805  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:52.333966 15493 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.080s
I20260709 12:28:52.448276 15433 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:52.430011  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:52.429912  5554 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:28:52.282094 15536 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:52.434793  5710 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:28:52.410030  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 149469us
I20260709 12:28:52.452661 15301 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:52.441020  5559 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: [28.21-28.21]
I20260709 12:28:52.446621  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:52.448328  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:52.449272  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:52.414039 15468 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:52.423614 15334 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:52.518901  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:52.518725  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 114623us
I20260709 12:28:52.522058 15547 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:28:52.453805  5630 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:52.459167  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:52.423945  6276 raft_consensus.cc:606] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:52.454396 15249 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:52.465636 15325 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:52.465931 15490 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:28:52.466741  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 122801us
I20260709 12:28:52.467689 15500 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:52.545507 15500 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:52.547923 15500 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:52.458371  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:52.548641  5708 raft_consensus.cc:2376] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30. Re-sending same reply.
I20260709 12:28:52.549286  6276 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "786a49bad9be400ea9ce584030aad446"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:52.549574  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:52.551069  5708 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:52.468321 15031 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:52.544934 15201 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:52.554217 15031 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:52.449887 15128 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:52.555338 15128 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:52.554571  5554 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:28:52.439817 15542 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:28:52.388027  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:52.556571 15542 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:52.475672  5560 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:52.478274  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:52.557696 15542 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:52.503400  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:52.560827 15542 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:52.458361 15491 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:28:52.561790 15491 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:52.562666  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:52.562830 15491 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:52.555830 15128 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:52.514524  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 66311.4us
I20260709 12:28:52.474854 15534 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:52.564586  6276 raft_consensus.cc:606] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:52.565053  6276 raft_consensus.cc:614] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:28:52.565099  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:52.566414 15534 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:52.511538 15510 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.725s	user 0.011s	sys 0.000s
I20260709 12:28:52.567476 15510 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:52.569710 15510 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:52.543016 15468 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:52.571720 15201 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:52.544661  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:52.458453 15351 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.058s
I20260709 12:28:52.580204  5552 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:52.573261  5705 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31. Ops: []
I20260709 12:28:52.478207 15433 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:52.578177  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:52.583550  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:52.596212  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:52.597244  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:52.597782  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:52.598733  5706 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:52.599126  5552 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 31: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:52.474749 15471 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.099s
I20260709 12:28:52.517674 15544 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:52.518113 15334 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:52.611478  5706 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:28:52.476435 15495 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.110s
I20260709 12:28:52.542326 15431 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:28:52.612659  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:52.612866  5558 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
W20260709 12:28:52.613909  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 161729us
I20260709 12:28:52.617789  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:52.591617  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:52.630930  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:52.632865  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:52.634881  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:52.644289  5526 leader_election.cc:400] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
I20260709 12:28:52.645080  5526 leader_election.cc:403] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:28:52.517493 15149 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.089s
I20260709 12:28:52.657415  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:52.651885  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:52.605032 15540 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:52.578883 15284 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.059s
I20260709 12:28:52.618103 15334 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:52.618394 15031 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:52.665306 15518 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.600s	user 0.009s	sys 0.002s
I20260709 12:28:52.695748  5558 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260709 12:28:52.696411  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 271665us
I20260709 12:28:52.698318  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:52.654577 15288 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:52.716104  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:52.727664  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:52.743364  6271 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:28:52.691824 15548 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:28:52.744482  6279 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:52.699303 15493 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:28:52.745276  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:52.727662 15431 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:52.689378 15554 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
I20260709 12:28:52.762269  6273 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:28:52.725515 15288 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:52.764501  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:52.764245  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:52.808301  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:52.808686  6275 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:52.808894  6279 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 29: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:52.763223  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:52.737262 15547 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:52.806288  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:52.809257  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 152508us
I20260709 12:28:52.695197 15509 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.061s
I20260709 12:28:52.714826 15031 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:52.813243  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 50090.7us
I20260709 12:28:52.759100 15429 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:52.756578 15484 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:52.840689 15500 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:52.810050  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:52.850080  5675 leader_election.cc:336] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 3.124s (SENT)
I20260709 12:28:52.850469 15554 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:52.850584 15128 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:52.806576 15431 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:52.732407 15518 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:52.853227  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:52.857966  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3625) took 1033 ms. Trace:
I20260709 12:28:52.764683 15430 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:52.875342  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:28:52.864771  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:52.865413  5561 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:28:52.766791 15493 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:52.869791  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45152.3us
I20260709 12:28:52.807740 15351 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:52.812721 15301 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:52.807933 15336 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:52.872803  5555 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:52.853514 15548 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:52.743212 15569 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.135s
I20260709 12:28:52.854163 15432 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:52.872045  5630 rpcz_store.cc:276] 0709 12:28:51.824775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:51.885829 (+ 61054us) service_pool.cc:224] Handling call
0709 12:28:51.886156 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:52.239971 (+353815us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b54001da720. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa6741c0 00007f4ffa675231 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c
0709 12:28:52.600870 (+360899us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:28:52.746174 (+145304us) log.cc:844] Serialized 14 byte log entry
0709 12:28:52.857411 (+111237us) log.cc:844] Serialized 14 byte log entry
0709 12:28:52.857617 (+   206us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:52.857671 (+    54us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:52.857719 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:52.857790 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:52.857906 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":988,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":4525,"spinlock_wait_cycles":810853632,"thread_start_us":157900,"threads_started":2}
I20260709 12:28:52.866364 14887 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:52.760262 15315 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:52.899427  5707 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:28:52.890086  5703 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:52.866670 15539 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 25 current term: 25 is higher.
I20260709 12:28:52.894297  5709 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:52.898365 15468 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:52.866480 15429 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:52.893946 15149 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:28:52.998553  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:52.996907  5561 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:28:52.894790 15301 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:52.866952 15547 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:52.805670 15544 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:52.869524 15500 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:52.880545 15471 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.111s
I20260709 12:28:52.885308 15351 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:52.888029 15249 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:53.002939 15468 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:53.008642 15509 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:53.040972  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:53.028724 15567 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.204s	user 0.001s	sys 0.000s
W20260709 12:28:53.042320  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:52.889165 15548 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.043452  5702 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:52.911767 15336 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:53.012113 15536 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:53.038357 15563 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.396s	user 0.002s	sys 0.001s
W20260709 12:28:53.086668  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 276687us
W20260709 12:28:53.090371  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 197818us
W20260709 12:28:53.047535  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:53.048275 15576 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:53.045397 15509 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:53.049741 15249 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:53.050889 15128 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:28:53.051990 15583 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.166s
I20260709 12:28:53.100117 15301 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.21, Last appended by leader: 21, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.106956  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:53.101330 15500 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:53.108439 15431 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:53.052829 15570 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.228s	user 0.000s	sys 0.001s
I20260709 12:28:53.108695  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:53.053151  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 203130us
I20260709 12:28:53.109731 15291 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:53.110132 15291 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:53.110172 15351 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:53.054116 15575 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.167s	user 0.000s	sys 0.001s
I20260709 12:28:53.054697 15031 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:53.110805 15576 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:53.055058 15475 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:53.114032 15475 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.114775 15547 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:53.118347 15542 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.557s	user 0.007s	sys 0.002s
I20260709 12:28:53.098007 15429 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:53.119863  5561 raft_consensus.cc:2368] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29.
I20260709 12:28:53.111989  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1976) took 1072 ms. Trace:
I20260709 12:28:53.112262 15592 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:53.120637 15509 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:53.121008 15592 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.123299 15592 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:53.112829 15500 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:53.113219 15564 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.414s	user 0.003s	sys 0.000s
I20260709 12:28:53.123937  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:53.113260 15576 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:53.124681  6274 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
W20260709 12:28:53.118438 15485 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.802s	user 0.003s	sys 0.006s
I20260709 12:28:53.119761 15429 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:53.052327 15315 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:53.095497 15518 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:53.128552  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:53.129925  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:53.131489  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44032.3us
I20260709 12:28:53.144104  5553 raft_consensus.cc:2368] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31.
I20260709 12:28:53.144212  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:53.120607  5635 rpcz_store.cc:276] 0709 12:28:52.039322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:52.117122 (+ 77800us) service_pool.cc:224] Handling call
0709 12:28:52.147536 (+ 30414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:52.293367 (+145831us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:52.293403 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:52.293463 (+    60us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:52.293498 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:53.111764 (+818266us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:53.111809 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:53.111934 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":334610688}
I20260709 12:28:53.138412  6275 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:53.141003  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:53.143989  5557 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:53.116184 15475 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:53.144686  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:53.164410 15596 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:53.134087 15548 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:53.141194 15485 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.123980 15500 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.165899  5556 raft_consensus.cc:2368] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32.
I20260709 12:28:53.167750 15485 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:53.111153 15291 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.171241 15291 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:53.045913 15493 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.170516  5526 leader_election.cc:400] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27.
I20260709 12:28:53.146055 15429 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:53.162911 15325 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.626s	user 0.010s	sys 0.002s
I20260709 12:28:53.181546  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:53.191464  6275 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:28:53.176697  5526 leader_election.cc:403] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260709 12:28:53.193456  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:53.201037  5703 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:28:53.201300  5560 raft_consensus.cc:1217] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Deduplicated request from leader. Original: 24.14->[26.15-28.16]   Dedup: 28.16->[]
I20260709 12:28:53.202034  5560 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: [26.15-28.16]
I20260709 12:28:53.204294  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:53.213601  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:53.214282  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:53.214340  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:53.215219  6271 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:28:53.200008 15540 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.504s	user 0.005s	sys 0.006s
I20260709 12:28:53.180258 15429 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:53.192896 15475 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:53.216931  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.227101  6275 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:28:53.227511  5630 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:53.216929  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:53.172323 15554 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:53.206355 15596 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:53.232744  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.236255  5627 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:53.233978  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:53.237308  5631 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:53.222343 15351 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:53.239430  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.244729  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:53.199347 15334 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:53.198074 15325 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:53.226015 15540 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:53.246536  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.247638  5632 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: [29.18-32.20]
I20260709 12:28:53.223863 15431 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:53.215705 15583 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.035s
I20260709 12:28:53.208467 15542 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.036s
W20260709 12:28:53.307333  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:53.189740 15468 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:53.240417 15432 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:53.241988 15591 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:53.321165  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.323228  5630 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:28:53.245898 15336 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:53.240731 15430 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:53.227329 15475 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:53.344292  5706 raft_consensus.cc:2368] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
I20260709 12:28:53.344929  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:53.243935 15602 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:53.371268  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:53.373665  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.429824  5630 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:53.377559  5628 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:53.378131  5632 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:53.378309  5552 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:53.379787 15475 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:53.381120 15430 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.383927 15325 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:53.461638  6272 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:28:53.390142  5557 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:28:53.464016  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:53.465214  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:53.465924  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:53.466399  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:53.404799 15590 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:28:53.467190 15590 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:53.467536  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:53.468215 15590 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.469580 15590 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:28:53.406365 15601 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:28:53.406723  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:53.474664  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:53.475088  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:53.407727 15334 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:53.409678 15523 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:53.476940 15523 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:53.411805 15429 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:53.369582 15432 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:53.413300  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:53.416848 15554 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:53.370667 15301 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:53.419768 15433 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.802s	user 0.010s	sys 0.002s
I20260709 12:28:53.423156  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:53.508275  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:53.509356  6273 raft_consensus.cc:2368] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
I20260709 12:28:53.433331  5633 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35. Ops: []
I20260709 12:28:53.429704  5635 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:53.320735 15431 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.444963 15602 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:53.344056 15490 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.712s	user 0.006s	sys 0.006s
I20260709 12:28:53.453776  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:53.454820  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:53.455236  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.527655  5632 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:53.485652  5630 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:28:53.500692  6271 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:28:53.530032  6274 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 29: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:53.307135 15351 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:53.509140  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 86087us
I20260709 12:28:53.533396  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:53.243772 15589 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:53.427954  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:53.446777  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:53.519012  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:53.534155  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:53.537715  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:53.542820  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 115038us
W20260709 12:28:53.589377  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53565.4us
I20260709 12:28:53.517554 15509 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:28:53.624812  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 64533.4us
I20260709 12:28:53.506130 15485 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.034s
I20260709 12:28:53.629865  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.630831  6273 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:53.631649  6273 raft_consensus.cc:606] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:53.526616 15602 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:53.379135 15591 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.540609 15583 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:28:53.511623 15429 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.512785 15554 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.517967 15433 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:53.608147 15472 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:28:53.624063  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:53.683836  5632 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:28:53.624022 15509 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:53.670776  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:53.631429 15548 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:28:53.680703  5553 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
W20260709 12:28:53.696849 15604 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 12:28:53.704581  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.706250  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.707078  5631 raft_consensus.cc:2368] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30.
I20260709 12:28:53.707898  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:53.711781  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.712244  5634 raft_consensus.cc:2368] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
I20260709 12:28:53.713092  5635 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:53.681160  5630 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:53.517735 15301 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:53.684742  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60772.7us
I20260709 12:28:53.532595 15490 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.689841  5557 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:28:53.687346 15290 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:53.691124 15509 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:53.607962 15589 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:53.726555  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:53.750113  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:53.751070  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:53.751518 15485 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:53.751981 15485 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:53.733656  6273 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:28:53.752372 15485 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:53.752702  6273 raft_consensus.cc:606] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Received request to transfer leadership to TS 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:53.754124 15469 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:28:53.756117 15540 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:53.690582 15468 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:53.757115 15468 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:53.757504 15468 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:53.759328 15589 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:53.762432  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:53.763212  5633 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 36: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:53.765913  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.732615 15290 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:53.769456 15290 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.770594 15290 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:53.772096 15591 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:53.773101 15558 log.cc:927] Time spent T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.435s	user 0.007s	sys 0.015s
I20260709 12:28:53.770515  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:53.774510  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:53.739353 15588 raft_consensus.cc:993] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
W20260709 12:28:53.790092  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 156505us
I20260709 12:28:53.779834  5632 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:53.799618  5631 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:53.801159  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:53.802165  5631 raft_consensus.cc:2368] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
I20260709 12:28:53.805130  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.805857  5632 raft_consensus.cc:2376] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29. Re-sending same reply.
I20260709 12:28:53.807914  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:53.804472  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:53.812247  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:53.814582  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.829660  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:53.830945 15596 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.588s	user 0.011s	sys 0.000s
I20260709 12:28:53.790758 15591 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:53.810427 15588 raft_consensus.cc:1081] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:28:53.850278 15429 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:53.850955 15431 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:53.855278  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1704) took 1419 ms. Trace:
W20260709 12:28:53.855238  5627 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.618s	user 0.007s	sys 0.003s
I20260709 12:28:53.863588  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:53.864049 15500 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:53.864897  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:53.862517 15596 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27.
I20260709 12:28:53.866048  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:53.862417  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3914) took 2375 ms. Trace:
I20260709 12:28:53.866875  5561 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:53.867043  5705 rpcz_store.cc:276] 0709 12:28:51.479036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:51.480583 (+  1547us) service_pool.cc:224] Handling call
0709 12:28:51.489789 (+  9206us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:52.585062 (+1095273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:53.854460 (+1269398us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54002d0ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:53.854626 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2491225856}
I20260709 12:28:53.871315 15602 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.872313 15596 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:53.861925  5708 rpcz_store.cc:276] 0709 12:28:52.436507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:52.551746 (+115239us) service_pool.cc:224] Handling call
0709 12:28:52.552006 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:52.573107 (+ 21101us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:52.573138 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:52.573178 (+    40us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:52.573216 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:53.855066 (+1281850us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b54002f6054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:53.855116 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:53.855228 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7844,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":14018,"spinlock_wait_cycles":47820544}
I20260709 12:28:53.861146  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:53.862327  5627 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:53.733376 15583 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 34.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.086s
I20260709 12:28:53.860690  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:53.878422  5633 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 22.
I20260709 12:28:53.882474  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.898499  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.899593  5632 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:53.904510  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:53.921717  5708 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:28:53.922039  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.930986  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:53.929847  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:53.950023  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:53.985858  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.987907  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:53.988160  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:53.994809  5629 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:54.005092  5632 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 28 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:28:53.914790 15432 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:54.006706  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:54.008337  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:53.880733 15591 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.966207 15469 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:53.986982 15602 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:54.019786  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:54.022061  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:54.022264  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:54.023393  5633 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:53.903890 15596 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:53.922324 15485 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:54.030876  5554 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:54.001796 15620 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:28:54.003284 15351 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:54.035923  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:28:54.025036 15590 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.551s	user 0.007s	sys 0.005s
I20260709 12:28:54.016741 15031 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:54.048897  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:54.049453  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:54.050155  5627 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 35: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:54.055097  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:54.056147  5633 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:54.027707 15583 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:54.058697  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:54.061033  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:54.063745  5559 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:54.068632  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:54.069644  5627 raft_consensus.cc:2368] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
I20260709 12:28:54.026013 15432 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:54.073328  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:54.075763 15615 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:28:54.080343  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:54.081640 15485 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:54.059921 15031 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:54.085526  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:54.044961 15540 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:54.048657 15433 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:54.088941  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:54.091190  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:54.095391 15540 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:54.097174  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:54.081640  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:54.122100  5636 raft_consensus.cc:2410] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 30 because replica has last-logged OpId of term: 29 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:28:54.122154  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:54.084851 15490 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:54.099889  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:54.124063  5630 raft_consensus.cc:2368] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30.
W20260709 12:28:54.101179  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 861660us
I20260709 12:28:54.034695 15509 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:54.101558 15542 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:28:54.102128  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:54.102397  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1712) took 1408 ms. Trace:
I20260709 12:28:54.129410  5632 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:54.130314  5629 pending_rounds.cc:85] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 10
I20260709 12:28:54.102916 15471 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:54.105369  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:54.130726  5629 pending_rounds.cc:107] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 21.11
I20260709 12:28:54.106362  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3495) took 1392 ms. Trace:
W20260709 12:28:54.084180 15475 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.584s	user 0.008s	sys 0.001s
I20260709 12:28:54.085217 15031 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:54.113345  5676 connection.cc:256] RPC call timeout handler was delayed by -1.0087392460000046s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:28:54.117372  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:54.118184 15469 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:54.129333  5710 rpcz_store.cc:276] 0709 12:28:52.694047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:53.088405 (+394358us) service_pool.cc:224] Handling call
0709 12:28:53.088731 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:53.088975 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:28:53.089275 (+   300us) log.cc:844] Serialized 14 byte log entry
0709 12:28:53.323964 (+234689us) log.cc:844] Serialized 14 byte log entry
0709 12:28:54.098631 (+774667us) spinlock_profiling.cc:243] Waited 6.9 ms on lock 0x7b18004b5fe0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ffe54658f 00007f4ff7585571 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd
0709 12:28:54.102005 (+  3374us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:54.102063 (+    58us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:28:54.102120 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:54.102196 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:54.102333 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":789995,"prepare.queue_time_us":18593,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":784747,"raft.run_cpu_time_us":401,"raft.run_wall_time_us":14543,"spinlock_wait_cycles":15878016,"thread_start_us":229027,"threads_started":2,"wal-append.queue_time_us":150816}
I20260709 12:28:54.134618  5627 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: [26.18-28.19]
I20260709 12:28:54.132772  5554 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
W20260709 12:28:54.138384  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:54.133291  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:54.053912 15620 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:54.126587  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:54.131094  5629 raft_consensus.cc:2889] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:28:54.131292  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:54.131428  5703 rpcz_store.cc:276] 0709 12:28:52.714267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:53.475693 (+761426us) service_pool.cc:224] Handling call
0709 12:28:53.475977 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:53.476182 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:28:53.506456 (+ 30274us) log.cc:844] Serialized 14 byte log entry
0709 12:28:54.106031 (+599575us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:54.106088 (+    57us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:28:54.106140 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:54.106208 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:54.106310 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":389966,"prepare.queue_time_us":30174,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":5197,"raft.queue_time_us":390726,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":71,"thread_start_us":209217,"threads_started":1,"wal-append.queue_time_us":396188}
I20260709 12:28:54.170006  5632 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:54.171018  5632 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:28:54.163278  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:54.173182  5627 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:54.176059  5630 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: [26.16-26.16]
W20260709 12:28:54.190233  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 63737.9us
I20260709 12:28:54.193506  5628 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36. Ops: []
I20260709 12:28:54.195868  6273 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:28:54.199719 14887 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:54.200291 14887 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:54.203682 15596 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:54.203684  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3406) took 1019 ms. Trace:
I20260709 12:28:54.205632  5561 rpcz_store.cc:276] 0709 12:28:53.184020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:53.467193 (+283173us) service_pool.cc:224] Handling call
0709 12:28:54.203528 (+736335us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x7b540023ffd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:54.203627 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":916034304}
I20260709 12:28:54.205185  6280 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: []
W20260709 12:28:54.208642  5675 leader_election.cc:336] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.115s (SENT)
I20260709 12:28:54.209399  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:54.209702  5675 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.219s (SENT)
I20260709 12:28:54.208804  6279 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:28:54.211371 15433 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:54.211887 15433 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:54.214080 15433 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:54.214576  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:54.215862  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:54.216807 15431 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:54.217352 15431 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:54.221271 15599 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.132s	user 0.003s	sys 0.008s
I20260709 12:28:54.224534  5560 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:54.226125  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:54.227730  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:54.231374 15598 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.143s	user 0.004s	sys 0.003s
I20260709 12:28:54.232789  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:54.233436  5561 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:28:54.143142 15468 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:54.234474 15579 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.132s	user 0.005s	sys 0.002s
I20260709 12:28:54.234856  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:54.235556  5561 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:54.173997 15301 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:54.161039 15471 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:54.237995  5636 pending_rounds.cc:85] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 17
I20260709 12:28:54.238306  5636 pending_rounds.cc:107] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 24.18
I20260709 12:28:54.204545 15336 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.034s
I20260709 12:28:54.238680  5636 raft_consensus.cc:2889] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:28:54.240991  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53242.7us
I20260709 12:28:54.242442 15495 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:28:54.242980  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3923) took 1818 ms. Trace:
I20260709 12:28:54.243363  5709 rpcz_store.cc:276] 0709 12:28:52.424252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:52.424781 (+   529us) service_pool.cc:224] Handling call
0709 12:28:52.425114 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:52.984221 (+559107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:54.242774 (+1258553us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b54004529d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:54.242926 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1069163776}
I20260709 12:28:54.243769 15249 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:54.087038 15583 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:54.247771 15440 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:54.248338 15440 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:54.205961 15623 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:54.249411 15623 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:28:54.249547 15594 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.009s	user 0.007s	sys 0.000s
I20260709 12:28:54.250648 15440 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:54.250383 15623 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:54.251137  5556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 1975) took 1994 ms (client timeout 1560 ms). Trace:
W20260709 12:28:54.251596  5556 rpcz_store.cc:269] 0709 12:28:52.256941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:52.257073 (+   132us) service_pool.cc:224] Handling call
0709 12:28:54.250889 (+1993816us) spinlock_profiling.cc:243] Waited 829 ms on lock 0x7b54002283d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:54.251032 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1907141888}
I20260709 12:28:54.088861 15432 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:54.255314 15554 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:54.255446 15620 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:54.174224 15534 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.083s
I20260709 12:28:54.257589  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3404) took 1112 ms. Trace:
I20260709 12:28:54.257972  5553 rpcz_store.cc:276] 0709 12:28:53.145463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:53.193088 (+ 47625us) service_pool.cc:224] Handling call
0709 12:28:54.257408 (+1064320us) spinlock_profiling.cc:243] Waited 51.4 ms on lock 0x7b54003a8ed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:54.257520 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":118137728}
I20260709 12:28:54.259191  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:54.259732  5553 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:54.260061  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1990) took 1427 ms. Trace:
I20260709 12:28:54.242458  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:54.261766  5559 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:54.263690  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:54.269389  5633 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:54.271760  5630 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:28:54.164834 15546 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:54.260735  5558 rpcz_store.cc:276] 0709 12:28:52.833111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:52.834562 (+  1451us) service_pool.cc:224] Handling call
0709 12:28:52.834942 (+   380us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:52.855119 (+ 20177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:52.855154 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:52.870997 (+ 15843us) log.cc:844] Serialized 26 byte log entry
0709 12:28:53.038533 (+167536us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:53.038574 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:54.241336 (+1202762us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b54001c57d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:54.241386 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:54.248787 (+  7401us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:54.259774 (+ 10987us) raft_consensus.cc:1685] finished
0709 12:28:54.259803 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:54.259975 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":29532,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":3700,"spinlock_wait_cycles":49129216,"thread_start_us":131029,"threads_started":1,"wal-append.queue_time_us":277100}
I20260709 12:28:54.280284  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:54.281328  6276 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:54.283624  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:54.285113  5558 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 28 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:28:54.263096  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:54.179235 15315 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:54.316231 15315 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:54.317338 15315 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:54.244210 15249 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:54.172135 15475 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:54.312139  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49130.7us
I20260709 12:28:54.313620  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:54.314860  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:54.293138 15598 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.054s	user 0.003s	sys 0.001s
I20260709 12:28:54.321400  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:54.356151  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:54.324476  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:54.329519  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3405) took 1145 ms. Trace:
I20260709 12:28:54.329923  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:54.362702  5560 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:28:54.291384  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:54.364333  6276 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:28:54.364106  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3411) took 1163 ms. Trace:
I20260709 12:28:54.335675  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 1982) took 2069 ms. Trace:
I20260709 12:28:54.358199  5561 raft_consensus.cc:2410] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 31 because replica has last-logged OpId of term: 29 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 18.
I20260709 12:28:54.362628  5558 rpcz_store.cc:276] 0709 12:28:53.183742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:54.283288 (+1099546us) service_pool.cc:224] Handling call
0709 12:28:54.329445 (+ 46157us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:54.365242  5555 rpcz_store.cc:276] 0709 12:28:52.266163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:52.269880 (+  3717us) service_pool.cc:224] Handling call
0709 12:28:52.270153 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:52.882569 (+612416us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:54.335324 (+1452755us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b54003a93d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:54.335595 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1385551616}
I20260709 12:28:54.366456  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:54.290268 15336 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:54.362993  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:54.365023  5560 rpcz_store.cc:276] 0709 12:28:53.200542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:54.277355 (+1076813us) service_pool.cc:224] Handling call
0709 12:28:54.364025 (+ 86670us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:54.382040  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3409) took 1197 ms. Trace:
I20260709 12:28:54.385973  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:54.328984 15500 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:54.386984  5709 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:54.278651 15554 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.091s
I20260709 12:28:54.388094  5631 raft_consensus.cc:2368] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24.
I20260709 12:28:54.388240  5553 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:54.392175  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:54.403730  6271 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 29: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:54.384289  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:54.404815  5633 raft_consensus.cc:2368] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
I20260709 12:28:54.407794  5559 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:54.409322  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:54.409937  6276 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:54.414665 15539 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:54.415470 15539 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:54.417657  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:54.418131 15433 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:54.418566 15539 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:54.418633 15433 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:54.420200 15539 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:54.421018 15539 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:54.421934  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:54.379944 15533 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:54.289577 15546 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:54.425722  5556 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:28:54.432097  5560 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31. Ops: []
I20260709 12:28:54.377619 15391 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:54.438138  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:54.402779 15534 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:54.453702  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:54.454519  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 244847us
I20260709 12:28:54.456732  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:54.457551  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:54.411082 15500 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:54.460177  5555 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:28:54.463208  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:54.465641  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:54.465879  5559 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: [28.21-28.21]
I20260709 12:28:54.468757  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:54.457580  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:54.445827 15584 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.355s	user 0.010s	sys 0.010s
I20260709 12:28:54.472469 15430 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:54.432720 15533 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:54.478842  6271 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:28:54.434587 15591 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:54.459223 15391 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:54.474012 15543 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:28:54.462488 15592 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:54.482228  5555 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:28:54.456857 15481 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:54.473460 15510 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:54.483467 15336 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:28:54.484078 15433 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:54.470291 15602 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.015s
W20260709 12:28:54.485373  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3483) took 1905 ms (client timeout 1549 ms). Trace:
W20260709 12:28:54.660386  5702 rpcz_store.cc:269] 0709 12:28:52.580227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:52.583216 (+  2989us) service_pool.cc:224] Handling call
0709 12:28:54.485134 (+1901918us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b5400066654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:54.485266 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1034989312}
I20260709 12:28:54.485411  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3926) took 1723 ms. Trace:
W20260709 12:28:54.499097  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45512.7us
I20260709 12:28:54.501817 15475 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:54.503273 15484 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:54.472905 15249 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:54.467947 15511 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:54.472132  6276 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:28:54.469717 15538 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.159s
I20260709 12:28:54.523758  5709 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:54.706739  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:54.459558 15534 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:54.467461 15544 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.353s	user 0.000s	sys 0.029s
W20260709 12:28:54.520854 15584 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.416s	user 0.010s	sys 0.010s
I20260709 12:28:54.471748 15500 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:54.622720  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:54.719223  6273 raft_consensus.cc:2410] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 37 because replica has last-logged OpId of term: 35 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
W20260709 12:28:54.708267 15611 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:28:54.664575  5704 rpcz_store.cc:276] 0709 12:28:52.762040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:53.475944 (+713904us) service_pool.cc:224] Handling call
0709 12:28:53.476238 (+   294us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:54.103671 (+627433us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:54.103704 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:54.104201 (+   497us) log.cc:844] Serialized 26 byte log entry
0709 12:28:54.259610 (+155409us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:54.259653 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:54.259721 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:54.485212 (+225491us) raft_consensus.cc:1685] finished
0709 12:28:54.485230 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:54.485361 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":626943,"raft.queue_time_us":627440,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":76,"thread_start_us":154982,"threads_started":1,"wal-append.queue_time_us":366449}
I20260709 12:28:54.589253 15533 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:54.688835  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:54.715404 15544 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.601s	user 0.000s	sys 0.029s
W20260709 12:28:54.705241  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:54.766160  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:54.525755  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:54.707422  6271 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:28:54.702693 15484 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:54.709277 15534 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:28:54.793026  5629 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.022s	sys 0.013s
I20260709 12:28:54.715987  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:54.716382  5555 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:54.794828  5709 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:54.699394 15471 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:54.703540 15510 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:54.698431 15554 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:54.627988  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:54.740530 15507 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:28:54.703092 15543 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:54.709203 15636 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:28:54.698438  5602 leader_election.cc:400] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31.
I20260709 12:28:54.624917 15481 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:54.763002  5710 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:28:54.768785 15643 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:28:54.627496 15592 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:54.779127  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:54.717883  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:54.796904  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:54.910187  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 442179us
I20260709 12:28:54.944032  5675 leader_election.cc:400] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
I20260709 12:28:54.944527  5675 leader_election.cc:403] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20260709 12:28:54.883412  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:54.945291  6271 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:54.847295 15540 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.011s	sys 0.009s
I20260709 12:28:54.860574  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:28:54.949636  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 89189.4us
I20260709 12:28:54.950295  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3648) took 1167 ms. Trace:
I20260709 12:28:54.950889  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:54.796177 15577 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:54.841351  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:54.855778 15562 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 2.425s	user 0.016s	sys 0.022s
I20260709 12:28:54.910778  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3501) took 1703 ms. Trace:
I20260709 12:28:54.925223  5633 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:28:54.904126 15620 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.010s	sys 0.023s
I20260709 12:28:54.882513 15602 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.040s
W20260709 12:28:54.965340 15640 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
W20260709 12:28:54.947832  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 282863us
I20260709 12:28:54.943615  5704 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:28:54.744948 15249 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:54.950676  5627 rpcz_store.cc:276] 0709 12:28:53.783448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:54.189992 (+406544us) service_pool.cc:224] Handling call
0709 12:28:54.190288 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:54.190498 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:28:54.562893 (+372395us) log.cc:844] Serialized 14 byte log entry
0709 12:28:54.949945 (+387052us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:54.950003 (+    58us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:54.950055 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:54.950124 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:54.950239 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":144046,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":127848,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":4948,"thread_start_us":680541,"threads_started":2}
I20260709 12:28:54.928730 15546 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:54.927326 15507 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:54.949097 15510 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:54.954948  5708 rpcz_store.cc:276] 0709 12:28:53.207590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:53.922688 (+715098us) service_pool.cc:224] Handling call
0709 12:28:53.981220 (+ 58532us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:28:54.113555 (+132335us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:54.113611 (+    56us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:28:54.114371 (+   760us) log.cc:844] Serialized 78 byte log entry
0709 12:28:54.414128 (+299757us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:54.613905 (+199777us) log.cc:844] Serialized 14 byte log entry
0709 12:28:54.910070 (+296165us) log.cc:844] Serialized 14 byte log entry
0709 12:28:54.910371 (+   301us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:54.910454 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:54.910503 (+    49us) raft_consensus.cc:1685] finished
0709 12:28:54.910516 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:54.910702 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":138654,"prepare.queue_time_us":300730,"prepare.run_cpu_time_us":151,"prepare.run_wall_time_us":3189,"raft.queue_time_us":383836,"raft.run_cpu_time_us":493,"raft.run_wall_time_us":27652,"thread_start_us":759665,"threads_started":3,"wal-append.queue_time_us":299740}
I20260709 12:28:54.966087  5703 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 25. Current term is 26. Ops: []
W20260709 12:28:54.946967  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 489386us
I20260709 12:28:54.989971  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:54.978881 15645 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:28:54.931870 15554 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:54.986263 15651 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:54.971547 15471 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:54.955194 15433 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:28:54.985117  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:54.985620  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:28:54.977538 15620 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.010s	sys 0.024s
I20260709 12:28:54.987496  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:54.979354 15315 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.021s	sys 0.086s
W20260709 12:28:54.953023 15540 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.011s	sys 0.011s
I20260709 12:28:54.969079 15391 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:54.992740  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:54.992952  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 1984) took 1542 ms. Trace:
I20260709 12:28:54.993031  5632 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:54.996662  6271 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
W20260709 12:28:55.000934  5557 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3419) took 1535 ms (client timeout 1645 ms). Trace:
I20260709 12:28:55.030140  5635 rpcz_store.cc:276] 0709 12:28:53.450718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:53.518431 (+ 67713us) service_pool.cc:224] Handling call
0709 12:28:54.992766 (+1474335us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b5400439254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:54.992898 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":403204992}
I20260709 12:28:54.926256 15288 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:54.988456 15511 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.003s
I20260709 12:28:55.011438  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:55.002216 15291 raft_consensus.cc:993] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: : Instructing follower 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:28:55.007581 15471 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:55.014027 14887 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.813s	user 0.011s	sys 0.000s
I20260709 12:28:55.013592 15507 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:55.014518  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:55.014760  5704 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:28:54.999328 15571 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:28:55.018579 15637 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.226s	user 0.001s	sys 0.000s
I20260709 12:28:55.021157 15478 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:55.022410 15431 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:55.023456 15484 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:55.071486 15581 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.977s	user 0.007s	sys 0.013s
I20260709 12:28:55.071844 15484 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:55.024739 15469 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:55.073225 15469 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:55.006572 15543 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:55.025126 15510 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:55.024746  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:54.997327 15642 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260709 12:28:55.027417  5553 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 24. Current term is 25. Ops: []
W20260709 12:28:55.001441 15638 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
W20260709 12:28:55.033109  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 47559.4us
W20260709 12:28:55.030548  5557 rpcz_store.cc:269] 0709 12:28:53.465887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:53.466188 (+   301us) service_pool.cc:224] Handling call
0709 12:28:55.000679 (+1534491us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b540044dc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:55.000850 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":512178048}
I20260709 12:28:54.970126 15577 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:55.029448 15646 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:55.034307 15288 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:55.034685  5560 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:55.034976  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:55.028638 15315 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.021s	sys 0.086s
I20260709 12:28:54.382608  5558 rpcz_store.cc:276] 0709 12:28:53.184850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:54.380836 (+1195986us) service_pool.cc:224] Handling call
0709 12:28:54.381157 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:54.381591 (+   434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:54.381635 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:54.381695 (+    60us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:54.381749 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:54.381828 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:54.381968 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":139}
I20260709 12:28:55.034332  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:55.041216  5632 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:28:55.044226 15620 raft_consensus.cc:2764] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election decision vote started in defunct term 29: won
I20260709 12:28:55.037029  5709 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:28:55.035498  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:55.086251  5557 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
W20260709 12:28:55.086495  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51084.4us
I20260709 12:28:54.997874 15592 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:55.053013  5633 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:28:54.992082 15546 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.054s
W20260709 12:28:55.061480  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47037.5us
I20260709 12:28:55.030140 15433 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:55.059608 15544 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:28:55.059060 15538 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.009s
I20260709 12:28:55.069026 15602 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:55.056537 15471 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:55.089541  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:55.090011  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:55.090171  5556 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:55.092039  5602 leader_election.cc:400] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29.
I20260709 12:28:55.092212  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:55.077672 15469 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:55.094012  5524 connection.cc:256] RPC call timeout handler was delayed by -1.121233403000133s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17980000000000002s for a response
I20260709 12:28:55.056149 15647 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:28:55.099311  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:55.100633  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:55.102351  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:55.095588  5627 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:55.103775  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:55.061998 14887 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:55.106175  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:55.107545 15481 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:55.107563 15128 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:55.107160 15478 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:28:55.105614 15315 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.009s
I20260709 12:28:55.094590 15620 raft_consensus.cc:1081] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Signalling peer 4a1c5e6dd8e249bf82d38f90fa37b43d to start an election
I20260709 12:28:55.115154  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3382) took 1423 ms. Trace:
I20260709 12:28:55.116482  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:55.132850  5629 rpcz_store.cc:276] 0709 12:28:53.691932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:54.129641 (+437709us) service_pool.cc:224] Handling call
0709 12:28:54.130063 (+   422us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:54.174749 (+ 44686us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:54.174782 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:54.189266 (+ 14484us) log.cc:844] Serialized 52 byte log entry
0709 12:28:54.793818 (+604552us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:28:54.926671 (+132853us) log.cc:844] Serialized 14 byte log entry
0709 12:28:54.927085 (+   414us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:54.927165 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:55.114891 (+187726us) raft_consensus.cc:1685] finished
0709 12:28:55.114911 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:55.115080 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14136,"prepare.queue_time_us":133384,"raft.queue_time_us":323,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":19863,"spinlock_wait_cycles":7945472,"thread_start_us":501186,"threads_started":1,"wal-append.queue_time_us":564759}
I20260709 12:28:55.118497  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:55.109824  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:55.155337  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45596.6us
I20260709 12:28:55.156256  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:55.160434  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 73017.5us
I20260709 12:28:55.161343  5627 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
W20260709 12:28:55.115777 15648 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:28:55.104102 15623 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:55.166388  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:55.165061  5627 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:28:55.107630 15602 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:28:55.167212  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:55.167604  5629 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:28:55.106201 15469 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:55.116623 15431 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.107393 15656 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:55.163470  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:55.130944 15652 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:55.196007  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3651) took 1193 ms. Trace:
I20260709 12:28:55.196650  5627 rpcz_store.cc:276] 0709 12:28:54.002447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:55.164397 (+1161950us) service_pool.cc:224] Handling call
0709 12:28:55.164712 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:55.195495 (+ 30783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:55.195905 (+   410us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:55.187860  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:55.160027 15249 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:55.202292 15249 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:55.093021 15646 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:55.142722 15538 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260709 12:28:55.158600 15511 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:28:55.158600 15391 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:28:55.161188 15478 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:55.166877 15128 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:28:55.205071  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 122971us
I20260709 12:28:55.183962  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 1988) took 1198 ms. Trace:
I20260709 12:28:55.206909  5633 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:28:55.211514  5629 rpcz_store.cc:276] 0709 12:28:53.985348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:55.158895 (+1173547us) service_pool.cc:224] Handling call
0709 12:28:55.159209 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:55.183619 (+ 24410us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:55.183893 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:55.195698 15656 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:55.213762  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:55.199329 15602 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:55.187624 15623 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:55.211480  5627 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35. Ops: []
W20260709 12:28:55.217883  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:55.261441  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:55.262256 15634 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:28:55.263445  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:55.194327 15469 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:55.259473  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3656) took 1043 ms. Trace:
I20260709 12:28:55.198733 15431 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:55.207403 15540 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.049s
I20260709 12:28:55.278947  5627 rpcz_store.cc:276] 0709 12:28:54.216052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:55.210417 (+994365us) service_pool.cc:224] Handling call
0709 12:28:55.210941 (+   524us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:55.259107 (+ 48166us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:55.259394 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:55.298202  5633 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
W20260709 12:28:55.299206  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:55.299131  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3431) took 1080 ms. Trace:
I20260709 12:28:55.301260  5633 rpcz_store.cc:276] 0709 12:28:54.218555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:55.252878 (+1034323us) service_pool.cc:224] Handling call
0709 12:28:55.297758 (+ 44880us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:55.298775 (+  1017us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:55.299059 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:55.183698 15433 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:55.215307 15481 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:55.221122 15652 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:55.234334 15651 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
I20260709 12:28:55.198861 15655 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.063s
I20260709 12:28:55.208673 15511 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:55.305447  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:55.297181 15539 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:55.308830 14887 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:55.311769 15533 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.532s	user 0.008s	sys 0.000s
I20260709 12:28:55.312224 15577 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:55.315851 15652 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:55.321195 15658 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:28:55.264094 15478 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.070s
W20260709 12:28:55.327196  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 137219us
I20260709 12:28:55.260792 15602 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:55.331907  5675 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.532s (SENT)
I20260709 12:28:55.257225 15656 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:28:55.341425  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 113481us
W20260709 12:28:55.343164  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 255940us
W20260709 12:28:55.324196  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:55.305233 15571 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:28:55.344638  6279 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: [21.15-23.16]
I20260709 12:28:55.356176  5633 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:40390
I20260709 12:28:55.356844  5633 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:55.357167  5633 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:55.358072  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:55.358940  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:55.344228  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:55.347308  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:55.343323 15500 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.601s	user 0.010s	sys 0.004s
I20260709 12:28:55.360409  6280 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:28:55.360379 15500 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:55.361260 15620 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:55.361646 15620 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:55.362017  5627 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:55.362210 15620 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:55.362916 15500 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:55.365151 15663 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:55.367981 15432 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:55.369498 15469 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:55.373502 15431 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:55.374303 15534 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.522s	user 0.008s	sys 0.003s
I20260709 12:28:55.343487 15652 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:55.375329  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:55.374770 15534 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:55.377485 15534 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:55.377568  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3419) took 1506 ms. Trace:
I20260709 12:28:55.378051  5632 rpcz_store.cc:276] 0709 12:28:53.870887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:54.192451 (+321564us) service_pool.cc:224] Handling call
0709 12:28:54.992908 (+800457us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:55.084567 (+ 91659us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:55.377343 (+292776us) spinlock_profiling.cc:243] Waited 800 ms on lock 0x7b5400439250. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:55.377501 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1840173824}
W20260709 12:28:55.381757  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:55.352110  5602 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.645s (SENT) [suppressed 24 similar messages]
W20260709 12:28:55.354096  5675 leader_election.cc:336] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.594s (SENT)
I20260709 12:28:55.389329  5627 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:55.335377 15533 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:55.393121  6275 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36. Ops: []
W20260709 12:28:55.381454 15649 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.283s	user 0.001s	sys 0.000s
I20260709 12:28:55.400928  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:55.410606  5627 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
W20260709 12:28:55.405210 15597 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 2.318s	user 0.011s	sys 0.012s
I20260709 12:28:55.268105 15249 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:55.413877  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:55.415779  5554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3421) took 1949 ms (client timeout 2105 ms). Trace:
W20260709 12:28:55.417972  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:55.418430  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:55.419648  5627 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
 from {username='slave'} at 127.0.0.1:40390
I20260709 12:28:55.420457  5627 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:55.420894  5627 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:28:55.416205  5554 rpcz_store.cc:269] 0709 12:28:53.466452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:53.815585 (+349133us) service_pool.cc:224] Handling call
0709 12:28:55.415600 (+1600015us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54000eb0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:55.415718 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":358258560}
I20260709 12:28:55.392410 14887 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:55.355505 15481 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.063s
W20260709 12:28:55.408562 15662 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:28:55.427400  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:55.432940 15602 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:55.437224  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:55.438045 15523 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:55.426623  5710 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: [30.17-30.17]
I20260709 12:28:55.439851  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:55.451527  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:55.456146  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 67057.3us
I20260709 12:28:55.454073  5627 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:55.461958  5554 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:55.469420  5704 raft_consensus.cc:2410] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
W20260709 12:28:55.446105 15654 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.259s	user 0.001s	sys 0.000s
I20260709 12:28:55.425784 15571 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:55.474541 15571 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:55.439880 14887 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:55.504207  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 152128us
I20260709 12:28:55.459077 15249 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:55.508462  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 124658us
I20260709 12:28:55.506994  5627 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:55.471844  5633 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:55.512612  5630 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 28. Current term is 29. Ops: []
I20260709 12:28:55.512801  5709 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:28:55.477152 15655 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:55.490012 15667 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:28:55.518564  5633 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:55.514194  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:55.514981  5554 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:55.519693  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:55.495728 15646 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:55.511585  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:55.496035 14887 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:55.520433  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:55.534545  5554 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 29: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:55.535755  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:55.536583  5554 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:55.537529  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:55.537774 15533 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:55.538589  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:55.540189  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1702) took 3104 ms (client timeout 1718 ms). Trace:
I20260709 12:28:55.518358 15667 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:55.543854 15667 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:55.515007 15571 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:55.545275 15667 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:55.547263  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3645) took 2012 ms. Trace:
I20260709 12:28:55.547672  5636 rpcz_store.cc:276] 0709 12:28:53.534991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:54.122783 (+587792us) service_pool.cc:224] Handling call
0709 12:28:54.123161 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:54.239237 (+116076us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:54.239273 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:54.239701 (+   428us) log.cc:844] Serialized 26 byte log entry
0709 12:28:54.505586 (+265885us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:54.505636 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:55.546843 (+1041207us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b540032d2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:55.546892 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:55.546944 (+    52us) raft_consensus.cc:1685] finished
0709 12:28:55.546957 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:55.547203 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":263640448,"thread_start_us":265577,"threads_started":1,"wal-append.queue_time_us":534468}
I20260709 12:28:55.517001  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:55.549700  5636 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:28:55.549888  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:55.550989  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:55.551520  5704 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:28:55.539113  5554 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:55.519704  5709 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:55.517726  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:55.421402 15511 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:55.516028 15655 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:55.553944  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3775) took 2977 ms. Trace:
I20260709 12:28:55.567658  5552 rpcz_store.cc:276] 0709 12:28:52.576170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:52.648906 (+ 72736us) service_pool.cc:224] Handling call
0709 12:28:52.667649 (+ 18743us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:53.433236 (+765587us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:55.553289 (+2120053us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x7b54002288d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:55.553838 (+   549us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1633927040}
I20260709 12:28:55.552327  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:55.573985  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56327.9us
W20260709 12:28:55.559893  5706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3494) took 2881 ms (client timeout 1798 ms). Trace:
W20260709 12:28:55.577883  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:55.520481  5633 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:55.569418  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:28:55.575470  5706 rpcz_store.cc:269] 0709 12:28:52.678786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:52.697977 (+ 19191us) service_pool.cc:224] Handling call
0709 12:28:55.559632 (+2861655us) spinlock_profiling.cc:243] Waited 604 ms on lock 0x7b5400210f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:55.559809 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1389488000}
I20260709 12:28:55.584224  5552 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:55.584671  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:55.540613  5707 rpcz_store.cc:269] 0709 12:28:52.435711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:52.435830 (+   119us) service_pool.cc:224] Handling call
0709 12:28:55.539978 (+3104148us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b54001987d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:55.540096 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":857687936}
W20260709 12:28:55.614282  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 102754us
I20260709 12:28:55.615175  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:55.579562  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:55.627213  5554 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
W20260709 12:28:55.562330 15669 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:28:55.620329  5709 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:28:55.562594 15623 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:55.559463 15543 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:55.566636 15655 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:55.633381  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:55.565762 15583 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:55.642294  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:55.656710  5554 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
W20260709 12:28:55.647461  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:55.657357  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:55.596213 15481 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:55.652989  5552 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
W20260709 12:28:55.653002  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 198773us
I20260709 12:28:55.655144  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:55.614493 15646 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:28:55.659241  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:55.680138  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:55.680899  5552 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:28:55.681228  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:55.681993  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:55.682787  5552 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:55.624482 15539 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.030s
I20260709 12:28:55.688308  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:55.688723  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:55.689237  5554 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:55.641423 15500 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:55.641563 15543 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 24 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:28:55.681620 15471 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.579s	user 0.008s	sys 0.008s
I20260709 12:28:55.697669  5675 leader_election.cc:400] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
W20260709 12:28:55.897945  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 203985us
W20260709 12:28:55.770742  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:55.845449  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:55.887684 15663 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:55.658670 15288 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:28:55.900862  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:55.902179  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:55.696044 15544 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:28:55.690225 15657 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:28:55.990335  5707 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:28:55.904255  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: true dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:55.991430  5558 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:55.946988  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47257.5us
I20260709 12:28:55.947983  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:56.007486  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 391929us
I20260709 12:28:56.007503  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:56.008615  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:55.658391 15534 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:55.957301  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:55.981289  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:56.029196  5554 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:56.030422  5704 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:56.065593  5552 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:55.906683  5632 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:28:56.094470  5632 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: []
W20260709 12:28:56.101280  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:55.902864  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 323262us
I20260709 12:28:55.992726  5675 leader_election.cc:400] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30.
I20260709 12:28:56.113379  5675 leader_election.cc:403] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Cancelling election due to peer responding with higher term
I20260709 12:28:55.968108 15433 raft_consensus.cc:2764] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:28:56.118877  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3430) took 1009 ms. Trace:
I20260709 12:28:56.128906  5554 rpcz_store.cc:276] 0709 12:28:55.117609 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:55.972235 (+854626us) service_pool.cc:224] Handling call
0709 12:28:56.118810 (+146575us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:56.108814  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:56.139912 15678 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260709 12:28:56.141824  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3424) took 1033 ms. Trace:
I20260709 12:28:56.142293  5553 rpcz_store.cc:276] 0709 12:28:55.108180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:55.108332 (+   152us) service_pool.cc:224] Handling call
0709 12:28:55.116875 (+  8543us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:55.327707 (+210832us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:28:55.447094 (+119387us) log.cc:844] Serialized 14 byte log entry
0709 12:28:55.630322 (+183228us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:55.630375 (+    53us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:55.630420 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:56.141518 (+511098us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b540023f354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:56.141576 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:56.141758 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6008,"prepare.queue_time_us":28122,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":51,"raft.queue_time_us":1553,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":17846,"spinlock_wait_cycles":484276992,"thread_start_us":283186,"threads_started":2,"wal-append.queue_time_us":232815}
I20260709 12:28:56.149372  5554 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:56.150125  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3427) took 1041 ms. Trace:
I20260709 12:28:56.150442  5554 rpcz_store.cc:276] 0709 12:28:55.109065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:56.148845 (+1039780us) service_pool.cc:224] Handling call
0709 12:28:56.149190 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:56.149853 (+   663us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:56.150068 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:28:56.152104  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 159530us
I20260709 12:28:56.153187  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:55.945186 15471 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:55.982307 15663 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:56.188759  5553 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:28:56.190860  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3431) took 1072 ms. Trace:
I20260709 12:28:56.191231  5553 rpcz_store.cc:276] 0709 12:28:55.118046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:56.151821 (+1033775us) service_pool.cc:224] Handling call
0709 12:28:56.188328 (+ 36507us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:56.190570 (+  2242us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:56.190795 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:56.192685  5553 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:28:56.193305  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:56.196277  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3429) took 1086 ms. Trace:
I20260709 12:28:56.196676  5552 rpcz_store.cc:276] 0709 12:28:55.109505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:56.118246 (+1008741us) service_pool.cc:224] Handling call
0709 12:28:56.196216 (+ 77970us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:28:56.201083  5558 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:28:56.201790  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1724) took 1188 ms. Trace:
I20260709 12:28:56.202178  5710 rpcz_store.cc:276] 0709 12:28:55.013131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:55.013504 (+   373us) service_pool.cc:224] Handling call
0709 12:28:55.013887 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:55.461061 (+447174us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:56.201490 (+740429us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b54002f6054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:56.201725 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":948615552}
I20260709 12:28:56.206588 15471 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:56.151789 15657 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:56.212159  5554 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
W20260709 12:28:56.218443  5631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3395) took 2461 ms (client timeout 1988 ms). Trace:
W20260709 12:28:56.236267  5634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3633) took 3006 ms (client timeout 1798 ms). Trace:
W20260709 12:28:56.237318 15679 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 12:28:56.241227 15509 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:56.242666  6237 leader_election.cc:336] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.560s (SENT)
I20260709 12:28:56.243249  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:56.245535 15617 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:28:56.246869 15647 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:56.247371 15647 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:56.248672 15647 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:56.250051 15647 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:56.151630 15543 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:56.253309 15543 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:56.254254 15433 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:56.254956 15433 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:56.258690 15608 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:56.259393 15608 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:56.260480 15608 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:28:56.263582  5600 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.007s	sys 0.062s
I20260709 12:28:56.169448 15651 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:28:56.264106  5600 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.007s	sys 0.062s
W20260709 12:28:56.236985  5634 rpcz_store.cc:269] 0709 12:28:53.229507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:53.229641 (+   134us) service_pool.cc:224] Handling call
0709 12:28:56.235943 (+3006302us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b5400242054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:56.236163 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":934840192}
I20260709 12:28:56.119346 15534 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:56.212325 15656 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.830s	user 0.015s	sys 0.000s
I20260709 12:28:56.239962 15523 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:56.278987 15652 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.890s	user 0.012s	sys 0.001s
W20260709 12:28:56.278532 15620 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.916s	user 0.010s	sys 0.003s
I20260709 12:28:56.263511 15595 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:28:56.309351  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:56.319015  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3781) took 2209 ms. Trace:
W20260709 12:28:56.335394  5701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3919) took 3923 ms (client timeout 2837 ms). Trace:
W20260709 12:28:56.336074  5701 rpcz_store.cc:269] 0709 12:28:52.411466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:52.411615 (+   149us) service_pool.cc:224] Handling call
0709 12:28:56.315000 (+3903385us) spinlock_profiling.cc:243] Waited 2.42 s on lock 0x7b5400451fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:56.335272 (+ 20272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5559992192}
W20260709 12:28:56.219017  5631 rpcz_store.cc:269] 0709 12:28:53.747948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:53.985509 (+237561us) service_pool.cc:224] Handling call
0709 12:28:56.209356 (+2223847us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b540018d154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:56.209514 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":922329344}
I20260709 12:28:56.299633 15656 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:56.339893 15595 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:56.309530 15620 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:56.407474 15684 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:28:56.415282 15595 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:56.460945  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 218341us
W20260709 12:28:56.461917  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 271655us
I20260709 12:28:56.319861  5556 rpcz_store.cc:276] 0709 12:28:54.109154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:54.433810 (+324656us) service_pool.cc:224] Handling call
0709 12:28:54.456394 (+ 22584us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:55.095580 (+639186us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:56.318551 (+1222971us) spinlock_profiling.cc:243] Waited 634 ms on lock 0x7b540023f854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:56.318902 (+   351us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1457131904}
W20260709 12:28:56.468434  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 459847us
W20260709 12:28:56.476073  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:56.469693  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:56.477455  5631 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:28:56.492980  5634 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:28:56.492491  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:56.500262  5627 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:28:56.506665  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44341.2us
W20260709 12:28:56.486152 15683 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:28:56.493906 15334 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:56.494570 15484 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:56.518594 15653 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.414s	user 0.010s	sys 0.002s
I20260709 12:28:56.519061 15509 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:56.519506 15538 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:28:56.526844  5627 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
W20260709 12:28:56.542750  5633 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.922s	user 0.011s	sys 0.002s
W20260709 12:28:56.543181  5633 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.929s	user 0.012s	sys 0.002s
I20260709 12:28:56.507941 15577 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:56.545766  5674 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.008s	sys 0.006s
W20260709 12:28:56.546195  5674 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.008s	sys 0.006s
I20260709 12:28:56.505242 15534 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:56.485231 15681 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.052s
W20260709 12:28:56.631019  6235 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.013s	sys 0.042s
W20260709 12:28:56.644987  6235 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.014s	sys 0.042s
I20260709 12:28:56.551159  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:56.646003  5631 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 23.
I20260709 12:28:56.648774  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:56.494513 15500 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.049s
W20260709 12:28:56.658761  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:56.660714  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3439) took 1154 ms. Trace:
I20260709 12:28:56.661337  5636 rpcz_store.cc:276] 0709 12:28:55.506619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:55.550437 (+ 43818us) service_pool.cc:224] Handling call
0709 12:28:55.550706 (+   269us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:55.551060 (+   354us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:55.551089 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:55.551523 (+   434us) log.cc:844] Serialized 26 byte log entry
0709 12:28:56.007216 (+455693us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:56.137710 (+130494us) log.cc:844] Serialized 14 byte log entry
0709 12:28:56.253938 (+116228us) spinlock_profiling.cc:243] Waited 6.1 ms on lock 0x7b18003ac840. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:28:56.648514 (+394576us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:56.648587 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:56.648628 (+    41us) raft_consensus.cc:1685] finished
0709 12:28:56.660416 (+ 11788us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:56.660615 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22450,"prepare.queue_time_us":143360,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":125424,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":10674,"spinlock_wait_cycles":14040832,"thread_start_us":836959,"threads_started":2,"wal-append.queue_time_us":666496}
I20260709 12:28:56.476413 15652 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:56.538921 15688 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:56.684144  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:56.686125  5630 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:28:56.535539  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 66068.1us
I20260709 12:28:56.694262  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:56.695194  5631 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:28:56.696978  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 156256us
W20260709 12:28:56.680766 15481 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.780s	user 0.127s	sys 0.196s
W20260709 12:28:56.704371  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:56.704864  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:56.536492 15334 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:56.699676  5636 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:28:56.700855  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:56.529901 15620 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:56.655270 15249 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.145s	user 0.016s	sys 0.000s
W20260709 12:28:56.539479  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 430708us
I20260709 12:28:56.722148  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:56.541308  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:56.702913  5601 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.021s	sys 0.008s
W20260709 12:28:56.723560  5601 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.021s	sys 0.008s
I20260709 12:28:56.724231  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:56.540454  5675 leader_election.cc:400] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
I20260709 12:28:56.531672 15433 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:56.709544  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:56.539476 15484 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:56.718550  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:56.726943  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 187393us
I20260709 12:28:56.719385  5634 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 31: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:28:56.539532 15429 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.970s	user 0.000s	sys 0.014s
I20260709 12:28:56.735442  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:56.728736  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:56.739022  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:56.650568 15577 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:56.703917 15481 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.001s	user 0.128s	sys 0.196s
I20260709 12:28:56.542395 15690 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:56.680874 15509 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } attrs { replace: true } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:56.745654 15507 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:28:56.716480 15663 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:56.747062 15617 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:28:56.747370  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:56.542297 15651 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.218s
I20260709 12:28:56.748029 15689 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:56.748539  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3507) took 2291 ms. Trace:
I20260709 12:28:56.748749 15685 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:56.750265 15692 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:56.741854  5632 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:28:56.769075  5553 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.042s	sys 0.016s
W20260709 12:28:56.832280  5553 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.042s	sys 0.016s
I20260709 12:28:56.770340  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:56.742909  5675 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:56.772979  5631 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:56.834371  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:56.835515  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:56.731371 15429 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.162s	user 0.000s	sys 0.014s
I20260709 12:28:56.747303 15468 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:56.814106  5630 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
W20260709 12:28:56.703181 15602 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.012s	sys 0.013s
I20260709 12:28:56.837047  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:56.837720  5632 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:28:56.733690 15325 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:56.823870  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 76582.2us
I20260709 12:28:56.716019 15682 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.053s
I20260709 12:28:56.748569 15433 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:56.828225  5702 rpcz_store.cc:276] 0709 12:28:54.456953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:54.952033 (+495080us) service_pool.cc:224] Handling call
0709 12:28:54.952467 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:54.980810 (+ 28343us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:54.980861 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:54.981356 (+   495us) log.cc:844] Serialized 26 byte log entry
0709 12:28:55.029645 (+ 48289us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:55.029689 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:55.029772 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:56.517916 (+1488144us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:56.743805 (+225889us) raft_consensus.cc:1685] finished
0709 12:28:56.743840 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:56.748472 (+  4632us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36542,"raft.queue_time_us":29168,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":8298,"spinlock_wait_cycles":6653696,"thread_start_us":19036,"threads_started":1,"wal-append.queue_time_us":65875}
I20260709 12:28:56.755102 15518 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:56.694686 15688 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:56.839464  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 69213us
I20260709 12:28:56.760020 15481 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:56.776785 15315 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:28:56.841012  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:56.818136 15661 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:28:56.848369  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 105556us
W20260709 12:28:56.839787 15602 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.013s	sys 0.014s
I20260709 12:28:56.846110  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:56.832046 15663 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:56.845351 15433 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:56.846901 14887 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.313s	user 0.018s	sys 0.002s
I20260709 12:28:57.008080  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:56.849987 15429 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:56.838195 15580 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.045s
I20260709 12:28:57.011101  5628 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
W20260709 12:28:57.011507  5552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.025s	sys 0.021s
W20260709 12:28:57.032693  5552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.026s	sys 0.021s
I20260709 12:28:56.755033 15478 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:28:56.873312 15534 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:28:57.015067  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:57.035837  5526 connection.cc:256] RPC call timeout handler was delayed by -1.5796485939999911s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17980000000000002s for a response
I20260709 12:28:57.036535  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:57.036963  5526 connection.cc:256] RPC call timeout handler was delayed by -1.028611429999728s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1542s for a response
I20260709 12:28:57.015202  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:57.039000  5631 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
W20260709 12:28:57.017773  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 170330us
I20260709 12:28:56.845876 15325 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:57.019261  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 178314us
I20260709 12:28:57.037436  5702 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:28:56.909684 15433 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:57.045996  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.046880  5702 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
W20260709 12:28:57.047137  5557 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.001s	sys 0.009s
W20260709 12:28:57.047670  5557 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.001s	sys 0.009s
I20260709 12:28:57.009549  5636 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:28:57.042824  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:56.873239 15518 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:56.836196 15689 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.043915  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:57.016300 15617 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:56.840194 15685 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:57.029520 15488 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.001s	sys 0.008s
I20260709 12:28:57.059556  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.025008 15692 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:56.873252 15698 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.196s
I20260709 12:28:56.910181 15663 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.064883  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:57.065657  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.025029 15694 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:28:57.029654 15315 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:57.009536 14887 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.075030  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:57.075363  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 430094us
I20260709 12:28:57.042050 15534 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:57.079648  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.059156 15699 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:57.039672 15510 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.038s
W20260709 12:28:57.070348 15488 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.001s	sys 0.009s
W20260709 12:28:57.090723  5635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3428) took 2873 ms (client timeout 1810 ms). Trace:
I20260709 12:28:57.048988  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3800) took 1658 ms. Trace:
I20260709 12:28:57.134678  5557 rpcz_store.cc:276] 0709 12:28:55.390151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:56.149104 (+758953us) service_pool.cc:224] Handling call
0709 12:28:56.149451 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:56.149704 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:28:56.544735 (+395031us) log.cc:844] Serialized 14 byte log entry
0709 12:28:57.048248 (+503513us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:57.048481 (+   233us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:57.048552 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:57.048761 (+   209us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:57.048919 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":682779,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":2651,"raft.queue_time_us":6813,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":7875,"thread_start_us":891303,"threads_started":2}
W20260709 12:28:57.134608  5635 rpcz_store.cc:269] 0709 12:28:54.217606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:55.034735 (+817129us) service_pool.cc:224] Handling call
0709 12:28:57.090485 (+2055750us) spinlock_profiling.cc:243] Waited 599 ms on lock 0x7b54001ce8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:57.090636 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1377435008}
I20260709 12:28:57.093688  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.039965 15429 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:57.094489  5558 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 33 index: 20. (term mismatch)
I20260709 12:28:57.140517  5558 pending_rounds.cc:85] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 19
I20260709 12:28:57.095602  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3783) took 2986 ms. Trace:
I20260709 12:28:57.140952  5560 rpcz_store.cc:276] 0709 12:28:54.109627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:54.450185 (+340558us) service_pool.cc:224] Handling call
0709 12:28:54.474955 (+ 24770us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:55.085554 (+610599us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:57.095315 (+2009761us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b54001bdd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:57.095529 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1286828288}
I20260709 12:28:57.104974  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:57.106029  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:57.143115  5554 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 34 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 20.
I20260709 12:28:57.108028  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3942) took 1530 ms. Trace:
I20260709 12:28:57.143527  5707 rpcz_store.cc:276] 0709 12:28:55.577692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:55.633837 (+ 56145us) service_pool.cc:224] Handling call
0709 12:28:57.107798 (+1473961us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b5400066654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:57.107959 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":763974400}
I20260709 12:28:57.088004 15468 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:28:57.113298  5701 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:28:57.089440  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3780) took 2980 ms. Trace:
W20260709 12:28:57.084604 15472 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.007s	sys 0.003s
I20260709 12:28:57.071198 15682 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:28:57.041370 15681 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.039s
I20260709 12:28:57.125782  5702 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:57.076668 15511 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:57.128611  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.129144  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:57.069667 15702 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:28:57.046052 15693 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.090s
I20260709 12:28:57.174700 15693 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:57.175344 15693 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.176402 15429 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.176848  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.177795  5705 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 33 because replica has last-logged OpId of term: 30 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
I20260709 12:28:57.178311 14887 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:57.136899  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:57.181591 15325 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:57.182965  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3440) took 1572 ms. Trace:
I20260709 12:28:57.081771 15652 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.184203  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.148237  5558 pending_rounds.cc:107] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 32.20
I20260709 12:28:57.184957  5558 raft_consensus.cc:2889] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:57.121949 15608 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:57.140065 15602 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:57.123363 15661 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:57.146397  5559 rpcz_store.cc:276] 0709 12:28:54.108921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:54.411637 (+302716us) service_pool.cc:224] Handling call
0709 12:28:54.465660 (+ 54023us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:54.466357 (+   697us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:57.089226 (+2622869us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b54001c5cd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:57.089377 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33126144}
I20260709 12:28:57.154055  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.192746  5710 raft_consensus.cc:2376] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30. Re-sending same reply.
I20260709 12:28:57.126430 15534 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:57.120656 15691 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:57.160001 15554 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:57.071926 15617 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:57.085254 15612 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.746s	user 0.019s	sys 0.024s
I20260709 12:28:57.166455  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:57.167173  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:57.161558  5554 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:57.171567 15509 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:57.214008 15577 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.135649 15699 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:57.196605 15701 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.061s
W20260709 12:28:57.273168 15720 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260709 12:28:57.273713 15667 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.728s	user 0.010s	sys 0.014s
I20260709 12:28:57.275128 15471 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:57.275547 15471 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:57.276851 15705 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.208s	user 0.001s	sys 0.000s
I20260709 12:28:57.276456 15471 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:57.277720 15471 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:57.173076  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:57.181514  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:57.139451  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:57.280550  5524 leader_election.cc:336] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.542s (SENT)
I20260709 12:28:57.281214  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:57.169687  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:57.088752 15694 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:57.183395  5552 rpcz_store.cc:276] 0709 12:28:55.610149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:56.197392 (+587243us) service_pool.cc:224] Handling call
0709 12:28:56.197717 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:56.197922 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:56.197947 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:56.198437 (+   490us) log.cc:844] Serialized 26 byte log entry
0709 12:28:57.033167 (+834730us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:57.033209 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:57.033282 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:57.182631 (+149349us) raft_consensus.cc:1685] finished
0709 12:28:57.182656 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:57.182889 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":834369,"threads_started":1,"wal-append.queue_time_us":956876}
I20260709 12:28:57.066483 15481 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.197984  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:57.286509  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 116925us
I20260709 12:28:57.198908  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:57.199429  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 141700us
W20260709 12:28:57.159596  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:57.204304  5631 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:57.160792 15468 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:57.213369  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:57.214594  5561 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: []
I20260709 12:28:57.214883  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:57.215161  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:57.215706  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:57.213938  6278 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: [29.20-29.20]
I20260709 12:28:57.224229  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.295022  5710 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:28:57.295976  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.203728  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:57.249450  5560 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:57.250105  5556 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
W20260709 12:28:57.251300  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:57.190557 15692 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:57.200985 15334 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:57.212375 14887 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:57.250624 15612 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.886s	user 0.019s	sys 0.024s
I20260709 12:28:57.261245  5559 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:57.136420 15708 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.066s
I20260709 12:28:57.172812  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:57.213807 15484 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:57.317503 15692 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:57.265084  5554 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: [30.17-30.17]
I20260709 12:28:57.201799 15661 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:57.200084 15608 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:57.279623  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:57.281544  5524 leader_election.cc:336] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.604s (SENT)
I20260709 12:28:57.291556 15694 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:28:57.303257  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 104441us
I20260709 12:28:57.201938 15602 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:57.283354 15703 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.270s	user 0.000s	sys 0.001s
I20260709 12:28:57.309535  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:57.338128  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:57.310055  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.310598  5526 leader_election.cc:400] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
I20260709 12:28:57.340093  5526 leader_election.cc:403] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:28:57.259127 15688 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.213409 15691 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.273182 15701 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:57.317766  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.279465 15693 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.379065  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:57.327905  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 125490us
I20260709 12:28:57.331785  5554 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
W20260709 12:28:57.333247  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51766.8us
I20260709 12:28:57.298795  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:57.335395  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:57.327391 15711 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:57.342486 15507 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.527s	user 0.009s	sys 0.002s
I20260709 12:28:57.405391  5554 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:28:57.345544  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:57.327234 15681 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:57.346714 15692 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.297394 15716 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:57.347936 15608 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:57.323796 15488 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:28:57.352769  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:57.373118 15325 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:28:57.354606  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:57.355463  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:57.355651 15663 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.356212 15714 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:28:57.357213 15710 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:28:57.359696 15661 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:57.360642  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 64658.3us
W20260709 12:28:57.414268 15647 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.164s	user 0.008s	sys 0.012s
I20260709 12:28:57.362200 15469 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
W20260709 12:28:57.365108 15713 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:28:57.371986  5710 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:57.391413  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:57.392541  5559 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:28:57.401340  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:28:57.402052  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 62068.2us
I20260709 12:28:57.344924 15689 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:57.301548 15509 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.071s
I20260709 12:28:57.413621 15715 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:28:57.409179  5524 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.690s (SENT)
I20260709 12:28:57.410614  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:28:57.479727  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70572.1us
I20260709 12:28:57.319583 15510 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.080s
I20260709 12:28:57.479925  6274 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:28:57.353749 15728 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:57.417174  6277 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:57.481616  6274 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:57.417815 15726 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:57.418669 15543 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.165s	user 0.008s	sys 0.011s
I20260709 12:28:57.418563  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:57.378542 15701 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:57.394098 15336 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:57.415133  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:57.421722  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:57.488306  5636 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
I20260709 12:28:57.431142  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:57.327020 14887 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:28:57.361212 15523 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.054s	user 0.010s	sys 0.007s
I20260709 12:28:57.450285  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:57.452116  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44196.6us
I20260709 12:28:57.453361 15691 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:57.419839 15721 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
W20260709 12:28:57.455008  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 413239us
W20260709 12:28:57.466964  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:57.420066 15488 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:57.524987 15488 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:57.525092 15661 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:57.525624 15661 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.494926 15726 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:57.470904  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 69697.7us
I20260709 12:28:57.504823 15580 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:28:57.528788 15580 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:57.528883  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:57.529507  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:57.529659 15580 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.530755  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:57.530961 15580 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:57.466457  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:57.269099 15682 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:57.531611 15746 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:28:57.478446 15692 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:57.532429  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:57.445175 15608 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:57.490222  6276 raft_consensus.cc:2410] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:28:57.489627  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:57.533475  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:28:57.500156  5524 leader_election.cc:336] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.742s (SENT)
W20260709 12:28:57.533566  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:57.539512  5701 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:28:57.539747  5524 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.748s (SENT)
I20260709 12:28:57.496065  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:57.540629  5524 leader_election.cc:336] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.798s (SENT)
W20260709 12:28:57.520550 15697 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.296s	user 0.001s	sys 0.000s
I20260709 12:28:57.542294  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:57.543681  6277 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:28:57.503259 15716 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:57.447788 15468 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:57.413239 15681 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.506760 15715 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:57.508723 15543 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:28:57.499074 15433 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.509961 14887 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:57.510550 15694 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.510720 15523 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:28:57.511126 15656 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.586391  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:57.587041  5559 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:57.587603 15694 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:57.587777 15681 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.588104  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:57.588881  5559 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
W20260709 12:28:57.511415 15722 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.142s	user 0.001s	sys 0.001s
I20260709 12:28:57.589516 15715 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:57.511932  5708 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:28:57.592288  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1721) took 2581 ms. Trace:
I20260709 12:28:57.592733  5708 rpcz_store.cc:276] 0709 12:28:55.010524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:55.010678 (+   154us) service_pool.cc:224] Handling call
0709 12:28:55.017666 (+  6988us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:57.591285 (+2573619us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:57.592057 (+   772us) spinlock_profiling.cc:243] Waited 2.49 s on lock 0x7b5400324454. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:57.592232 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5735547776}
I20260709 12:28:57.593242 15539 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:28:57.593757 15539 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:57.594240  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.595501 15685 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.521495  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:57.596974 15602 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.555156 15728 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:28:57.597803 15728 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:57.514341  5555 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:28:57.532655 15682 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.548048 15608 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:57.603062 15608 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.522940  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:57.603931 15518 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:57.584710 15717 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.326s	user 0.001s	sys 0.000s
W20260709 12:28:57.523147 15657 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.266s	user 0.006s	sys 0.012s
I20260709 12:28:57.605033 15657 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election lost for term 29. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30.
I20260709 12:28:57.605930 15651 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:57.606158  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3440) took 2099 ms. Trace:
I20260709 12:28:57.606371 15651 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:57.606761 15651 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:57.523918 15701 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:57.531719 15689 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:28:57.606544  5633 rpcz_store.cc:276] 0709 12:28:55.506910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:55.592236 (+ 85326us) service_pool.cc:224] Handling call
0709 12:28:55.614184 (+ 21948us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:56.625002 (+1010818us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:56.625049 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:56.648159 (+ 23110us) log.cc:844] Serialized 26 byte log entry
0709 12:28:57.075084 (+426925us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:28:57.252940 (+177856us) log.cc:844] Serialized 14 byte log entry
0709 12:28:57.253870 (+   930us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:57.254166 (+   296us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:57.605933 (+351767us) raft_consensus.cc:1685] finished
0709 12:28:57.605954 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:57.606098 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18284,"prepare.queue_time_us":63733,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":83744,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":209671,"thread_start_us":1426037,"threads_started":3,"wal-append.queue_time_us":522786}
I20260709 12:28:57.307431 15498 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.069s
I20260709 12:28:57.539019  5710 raft_consensus.cc:2410] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 29 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:28:57.604178 15523 consensus_queue.cc:1243] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 26.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:28:57.609241 15498 consensus_queue.cc:1243] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 23.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:28:57.609506  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.419870 15617 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.544773  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:57.540477  5705 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:28:57.610875  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.611366  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.611631  6277 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:57.501061 15325 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:57.613078  5705 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:28:57.613235 15325 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.614729  5705 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 25. Current term is 26. Ops: []
I20260709 12:28:57.547017  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:57.501550  6274 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.410550 15711 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:57.547528  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58014us
I20260709 12:28:57.532922 15692 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:57.554837  5704 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:57.557909  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:57.559887  5602 connection.cc:256] RPC call timeout handler was delayed by -1.6749296900002264s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1787s for a response
I20260709 12:28:57.632414  6274 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:57.528227 15726 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.532684 15746 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:57.562990  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.560983 15667 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:57.567198 15433 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:57.567884  5631 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30. Ops: []
I20260709 12:28:57.532557 15688 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:28:57.570425  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:57.553797 15469 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:57.652959  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:57.572783 15661 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:57.494448 15336 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:57.578970 15695 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:57.579977 15716 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.561071 15702 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:28:57.579783  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:57.450891 15714 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.598187 15728 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:28:57.605083  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 82230.1us
I20260709 12:28:57.595840  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:57.514463 15655 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.853s	user 0.022s	sys 0.009s
I20260709 12:28:57.618897  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:57.621834  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:57.616690  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 69735.8us
I20260709 12:28:57.670547  5559 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:57.616851  5702 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:57.622123  5629 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: []
I20260709 12:28:57.613091  6277 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:28:57.561749  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:57.632608 15692 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:57.635222  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:57.670583  5636 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 36.
I20260709 12:28:57.675523  5554 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:28:57.647421  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:57.651407  5701 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:28:57.617524 15249 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:57.582223 15488 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.632731 15608 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:28:57.633710 15315 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:57.677551  5705 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: [30.19-30.19]
I20260709 12:28:57.681403  6274 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:28:57.644789 15746 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:57.688606  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:57.616060 15534 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:57.684588  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.691180  5560 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:28:57.687495  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:57.658663 15336 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:57.674034 15602 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:57.729825  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:57.655822 15617 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.032s
I20260709 12:28:57.682224  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:57.673626 15726 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:57.734532  6272 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:57.739601  5561 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:28:57.735555  5552 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: [29.15-29.15]
I20260709 12:28:57.724403  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.658768 15667 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:57.742527  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:28:57.729633 15490 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 29 current term: 29 is higher.
W20260709 12:28:57.744688  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:57.685225 15702 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:57.747521  5636 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:57.738435 15663 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.007s
I20260709 12:28:57.729188 15523 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:57.733150 15725 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:28:57.733963 15469 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:57.678326 15655 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.686570 15249 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.774230 15748 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 35.24, Next index: 25, Last known committed idx: 21, Time since last communication: 0.092s
I20260709 12:28:57.744369 15608 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:57.693025 15315 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:57.762084 15511 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.591s	user 0.010s	sys 0.001s
I20260709 12:28:57.793483  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:57.737362 15602 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.803787  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.778851 15469 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:57.825671  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.825902  6275 raft_consensus.cc:2368] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
I20260709 12:28:57.745738 15726 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.835367  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:57.837499  5708 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:28:57.823758 15500 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:57.759049 15325 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.044s
I20260709 12:28:57.840965 15702 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.842948  5702 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:28:57.766762 15523 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:57.826602 15748 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:57.838420 15500 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:57.881961  6273 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:28:57.842548 15655 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:57.886631  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3437) took 2378 ms. Trace:
I20260709 12:28:57.884569  5703 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:57.879242 15748 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:57.891657  5709 raft_consensus.cc:2410] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 30 because replica has last-logged OpId of term: 29 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:28:57.887157  5553 rpcz_store.cc:276] 0709 12:28:55.508481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:56.193508 (+685027us) service_pool.cc:224] Handling call
0709 12:28:56.193861 (+   353us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:56.832765 (+638904us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:56.832805 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:57.476613 (+643808us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b2c000fb608. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:28:57.489435 (+ 12822us) log.cc:844] Serialized 52 byte log entry
0709 12:28:57.565487 (+ 76052us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:28:57.878186 (+312699us) log.cc:844] Serialized 14 byte log entry
0709 12:28:57.878933 (+   747us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:57.879024 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:57.879056 (+    32us) raft_consensus.cc:1685] finished
0709 12:28:57.885512 (+  6456us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:57.886540 (+  1028us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":88739,"prepare.run_cpu_time_us":172,"prepare.run_wall_time_us":168,"raft.queue_time_us":10869,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":2237,"spinlock_wait_cycles":9185664,"thread_start_us":896209,"threads_started":3,"wal-append.queue_time_us":135729}
I20260709 12:28:57.895785 15748 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:57.892725 15711 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:57.911379 15523 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:57.897043 15488 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.915339 15534 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:57.929586 15735 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:57.930261 15735 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:57.931313 15735 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:57.932879  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:57.933255  5705 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:28:57.933828  5705 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:57.934505 15500 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:57.934720  5705 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:57.936851  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:57.937578  5632 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:57.938675  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:57.941407 15534 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:28:57.941896  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:57.942375 15534 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:28:57.943760 15534 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:28:57.944065  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:57.944706  5636 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:57.945713  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:57.946254 15714 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:57.946786  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:57.947371  5636 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:57.947431 15469 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.947902 15646 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:28:57.948493 15646 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:57.949355 15646 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:57.950448  5634 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3658) took 1408 ms (client timeout 1640 ms). Trace:
I20260709 12:28:57.950812 15646 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:57.951463 15699 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:57.952219 15495 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:28:57.952666 15495 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:57.952991 15726 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:57.953454 15495 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.24, Last appended by leader: 24, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.890065 15608 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:57.954497 15495 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:28:57.958834 15647 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:57.959334 15647 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:57.961949  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:57.962832  5635 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 35: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:28:57.963677  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:57.965312  6279 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:28:57.965691  6279 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:57.950989  5634 rpcz_store.cc:269] 0709 12:28:56.541587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:56.541962 (+   375us) service_pool.cc:224] Handling call
0709 12:28:57.950207 (+1408245us) spinlock_profiling.cc:243] Waited 58.7 ms on lock 0x7b54000c1054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:57.950377 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":135124736}
I20260709 12:28:57.973773  6277 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:28:57.974295  6279 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:57.981596  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:57.984968  5634 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36. Ops: [35.24-35.24]
I20260709 12:28:57.985349  5707 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:57.986308  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:57.986824  6278 raft_consensus.cc:2376] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29. Re-sending same reply.
I20260709 12:28:57.987349  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:57.987952  5558 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:28:57.989305  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:57.989835  5636 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:28:57.990005  5558 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:28:57.975409  6279 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:28:57.993865  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:57.994479  5707 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:57.996229  5704 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:28:57.997511  5704 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:28:57.975339 15336 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.004642  5705 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:58.006922  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:58.006527  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:58.011531  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:58.012111  5707 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:57.963811 15692 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.015347  5676 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:58.031708  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:58.031554  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:58.034185  5706 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:58.034966  5561 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:28:58.036509  5560 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:58.035636  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:58.038834  5634 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: [29.17-29.17]
I20260709 12:28:58.016496 15694 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.024s
I20260709 12:28:58.040645 15714 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:28:58.041792 15595 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:28:58.041465 15728 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.042274 15714 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:28:58.042676 15714 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:57.960260 15647 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.043794 15728 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:58.044229 15647 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:58.031071 15735 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:28:58.047397 15580 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.516s	user 0.004s	sys 0.008s
I20260709 12:28:58.047933 15725 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:58.034185 15655 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:58.047525  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:58.049193 15655 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.049991  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:58.051581 15655 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:58.043517 15714 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.054247 15291 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.009s
I20260709 12:28:58.052202  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:58.064848 15291 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:28:58.065733 15291 consensus_queue.cc:1243] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 28.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:28:58.065850 15663 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:58.066355 15291 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:58.066520 15663 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:58.066896 15291 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:58.067082 15663 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:28:58.066991  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:58.074034  5705 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260709 12:28:58.074752  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:58.074990  5707 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:28:58.075596  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:58.077489  6279 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 23.
I20260709 12:28:58.078428  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:58.081404  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:58.089596  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:58.089825  6276 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:28:58.090636  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:58.086983  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:58.052124 15696 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:28:58.094080 15617 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:58.052608 15728 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.058s
I20260709 12:28:58.109820 15617 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.057507 15580 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.054s
I20260709 12:28:58.091642 15730 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:58.052211 15655 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:28:58.096861 15688 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:58.113075 15730 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.098352 15481 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:28:58.113934 15481 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:58.114338 15481 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:28:58.115401 15730 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:58.094280 15571 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.027s
I20260709 12:28:58.116446 15571 consensus_queue.cc:1243] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 29.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:28:58.117502  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3945) took 1020 ms. Trace:
I20260709 12:28:58.117934  5702 rpcz_store.cc:276] 0709 12:28:57.096896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:57.172497 (+ 75601us) service_pool.cc:224] Handling call
0709 12:28:58.117324 (+944827us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x7b54004515d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:58.117442 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":669353856}
I20260709 12:28:58.059195 15651 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.120519  5635 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: []
I20260709 12:28:58.122097 15651 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:58.067916 15291 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.123940 15291 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:28:58.124054 15656 consensus_queue.cc:1175] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cea66aa15be3423880843974acdbf50f" responder_term: 28 status { last_received { term: 27 index: 20 } last_committed_idx: 20 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:28:58.100118  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:58.126264  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:58.127406  5634 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:28:58.128532 15690 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:58.129050 15690 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:58.129462 15690 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:58.132086  6279 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:58.133066  5707 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 27: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:28:58.096201 15763 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:28:58.134183  6279 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: [21.15-23.16]
W20260709 12:28:58.102480 15498 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 23 current term: 23 is higher.
I20260709 12:28:58.135181 15315 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:58.100188 15488 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:28:58.136552 15488 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:58.104015 15692 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:58.137321 15538 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260709 12:28:58.137802 15692 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:28:58.137446 15488 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.138170 15692 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:58.138527 15692 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:58.062948 15534 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:28:58.139338 15534 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:58.139760 15534 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:58.113425 15688 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:58.127678  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:58.097811 15336 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:58.100935 15589 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:28:58.102489 15646 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:58.048499 15725 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.144698  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:28:58.144865  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57971.5us
I20260709 12:28:58.145349  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:58.145879  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:58.149410  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:58.169339  5558 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 29 because replica has last-logged OpId of term: 28 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
I20260709 12:28:58.151206 15469 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:58.169059  5634 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:28:58.179678  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:58.151292  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:58.181859  6274 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:58.155885  6279 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36. Ops: []
I20260709 12:28:58.157534 15538 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.003s
I20260709 12:28:58.159608  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:58.154904 15688 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:58.180473  5634 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:58.194721  6237 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:58.159976 15539 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.157930  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:58.196331  5552 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:28:58.196725  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:58.167333  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:58.172042  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:58.177651  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:58.163017 15748 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.178481 15469 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:58.153584 15617 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:58.207758  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:58.205885  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:58.208043  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:58.211544  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:28:58.218413  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:58.214771 15510 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:58.232705 15510 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:58.233529 15510 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:58.210310 15469 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:58.181491 15646 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.155241 15694 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.055s
I20260709 12:28:58.199707 15580 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:58.203269 15655 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:58.241628  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:58.244621  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:58.227559  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:58.206370 15730 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:28:58.244781  5552 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:58.245889  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:58.242147  6274 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:58.255982 15646 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:58.227968 15539 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:28:58.176676 15651 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.064s
W20260709 12:28:58.258653  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:58.199628 15495 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 12:28:58.200237 15695 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.044s
I20260709 12:28:58.239279 15540 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:28:58.269928  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:58.263047  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:58.263885  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:58.265900  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:58.272089  5554 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:58.268572  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3421) took 4397 ms. Trace:
I20260709 12:28:58.226845 15617 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:58.272787  5628 rpcz_store.cc:276] 0709 12:28:53.871444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:54.194375 (+322931us) service_pool.cc:224] Handling call
0709 12:28:56.209501 (+2015126us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:57.040500 (+830999us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:57.188852 (+148352us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b540018d154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:58.268286 (+1079434us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b540018d150. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:58.268482 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6246073344}
W20260709 12:28:58.273733  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 67969.6us
I20260709 12:28:58.274734 15747 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:58.215322 15523 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.046s
I20260709 12:28:58.274603  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:58.273017 15540 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:58.283147  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 832723us
I20260709 12:28:58.258487 15589 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:28:58.250896 15509 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:58.287693  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:58.290386  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:58.250669 15336 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:58.261615 15538 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:58.242991 15596 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
I20260709 12:28:58.301061  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:58.276546 15543 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:28:58.278653 15646 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:58.288529 15707 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:28:58.275902 15617 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:58.300004 15538 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:58.296211 15468 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.012s
I20260709 12:28:58.287789 15540 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:28:58.271082 15432 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:28:58.305188 15336 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.292341 15507 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:28:58.364167 15715 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.002s	sys 0.036s
I20260709 12:28:58.363049 15646 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:58.315964 15651 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.002s
I20260709 12:28:58.298223 15509 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.396332  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:58.383874 15715 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.002s	sys 0.037s
W20260709 12:28:58.403825  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:58.327764  5710 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:28:58.276763 15730 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:28:58.341226  5555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3426) took 3232 ms (client timeout 1787 ms). Trace:
W20260709 12:28:58.408774  5555 rpcz_store.cc:269] 0709 12:28:55.108878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:55.259548 (+150670us) service_pool.cc:224] Handling call
0709 12:28:58.340970 (+3081422us) spinlock_profiling.cc:243] Waited 2.09 s on lock 0x7b54003a98d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:58.341105 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4801578368}
I20260709 12:28:58.382206 15507 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:28:58.342569  5627 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3657) took 1801 ms (client timeout 1635 ms). Trace:
I20260709 12:28:58.418468  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:58.345587  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1593) took 1016 ms. Trace:
I20260709 12:28:58.419493  6273 rpcz_store.cc:276] 0709 12:28:57.329366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:57.329538 (+   172us) service_pool.cc:224] Handling call
0709 12:28:57.372929 (+ 43391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:57.885208 (+512279us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:58.345319 (+460111us) spinlock_profiling.cc:243] Waited 507 ms on lock 0x7b54005f8f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:58.345519 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1166175616}
I20260709 12:28:58.346968  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:58.389884 15646 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:58.423200  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:58.349255  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:58.425091  5558 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:58.349802 15748 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:58.344249  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:58.340039  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:58.438993  5556 raft_consensus.cc:2376] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35. Re-sending same reply.
I20260709 12:28:58.353027  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1727) took 1227 ms. Trace:
I20260709 12:28:58.400240 15602 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:58.446286  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:58.370929  5554 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:28:58.448096  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:58.449580  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:58.375819  5552 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:28:58.314467 15495 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:28:58.315451 15701 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:58.457834  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:58.364104 15617 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:58.365345 15538 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:58.407145  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:58.464772  5710 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
W20260709 12:28:58.464926  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:58.352272 15534 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.298584 15661 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:58.358551 15690 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.348011  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3512) took 1307 ms. Trace:
I20260709 12:28:58.470152  5555 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: []
I20260709 12:28:58.472836  5709 rpcz_store.cc:276] 0709 12:28:57.040875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:57.042487 (+  1612us) service_pool.cc:224] Handling call
0709 12:28:58.347797 (+1305310us) spinlock_profiling.cc:243] Waited 843 ms on lock 0x7b5400451fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:58.347947 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1937883008}
I20260709 12:28:58.473927  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:58.474548  5709 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:28:58.467438 15432 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:28:58.423213  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:58.348281 15725 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:58.354084 15692 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:58.436761  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:58.437284  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:58.440110  5706 rpcz_store.cc:276] 0709 12:28:57.129651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:57.678811 (+549160us) service_pool.cc:224] Handling call
0709 12:28:57.679115 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:58.037066 (+357951us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:58.352800 (+315734us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b54002e81d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:58.352970 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":813495040}
I20260709 12:28:58.359727  5633 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:28:58.421629 15507 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:58.465911  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:58.467342  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:58.469554  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:58.504679  5701 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:58.475320  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:58.506062  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:28:58.419063  5627 rpcz_store.cc:269] 0709 12:28:56.540925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:56.541052 (+   127us) service_pool.cc:224] Handling call
0709 12:28:58.342324 (+1801272us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b54001cf7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:58.342482 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1936153856}
I20260709 12:28:58.504256  5710 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:28:58.483894  5558 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
W20260709 12:28:58.508694  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:58.508430  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 205800us
I20260709 12:28:58.478353  6280 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
W20260709 12:28:58.513312  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:58.494902  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:58.495415  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:28:58.498369  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:58.500787  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:58.515528  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:58.516175  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:58.517493  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:28:58.519392  5636 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 29. Ops: []
I20260709 12:28:58.473459 15661 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:58.566530  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:28:58.565302  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 71870.1us
I20260709 12:28:58.568147  6273 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
W20260709 12:28:58.569564  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 68912.7us
I20260709 12:28:58.570704 15580 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:28:58.495896 15647 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:58.492770 15589 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:28:58.584784  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 67393.2us
I20260709 12:28:58.585264  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:58.589692  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:58.575121 15539 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:28:58.592646 15539 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:58.593990 15539 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.514679 15290 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.077s
I20260709 12:28:58.572539 15695 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:28:58.564920 15602 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:58.601112  5675 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:28:58.584100 15657 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:28:58.572608 15596 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:58.610610  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:58.575518 15707 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:58.604465  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:58.628206  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:58.628665  5701 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:28:58.629220  5631 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:28:58.572779 15735 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:28:58.573786 15661 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:58.599517 15691 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:28:58.608386  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 318158us
I20260709 12:28:58.576185 15291 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:28:58.607416  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:58.624804  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:58.605150 15518 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:28:58.627251  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 56802.5us
I20260709 12:28:58.631956  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:58.564734 15748 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.047s
I20260709 12:28:58.647538  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:58.667192  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:58.669111  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:28:58.588781 15315 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 30.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:28:58.505128 15730 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.683306  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:58.590098 15647 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.599717 15718 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.051s
W20260709 12:28:58.694689  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:58.697161  5601 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:58.700320  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:58.704820 15719 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:58.721343  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1752) took 1369 ms. Trace:
I20260709 12:28:58.751896  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:28:58.660393 15518 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:58.723258  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:58.754130  5552 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:58.731463  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:58.756868  5629 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
W20260709 12:28:58.737882 15773 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.351s	user 0.002s	sys 0.003s
I20260709 12:28:58.641695 15735 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:58.611253 15725 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.061s
I20260709 12:28:58.765918  5701 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:58.749557  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:58.652160 15291 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:58.743098 15510 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.509s	user 0.014s	sys 0.000s
I20260709 12:28:58.627832 15596 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.680156 15608 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:58.678814 15691 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:58.752238  6271 rpcz_store.cc:276] 0709 12:28:57.352281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:57.597337 (+245056us) service_pool.cc:224] Handling call
0709 12:28:57.598055 (+   718us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:57.680185 (+ 82130us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:28:57.692021 (+ 11836us) log.cc:844] Serialized 14 byte log entry
0709 12:28:57.921726 (+229705us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b18004428a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:28:58.068552 (+146826us) log.cc:844] Serialized 14 byte log entry
0709 12:28:58.069887 (+  1335us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:58.069943 (+    56us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:58.069994 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:58.719810 (+649816us) spinlock_profiling.cc:243] Waited 6.95 ms on lock 0x7b18004428a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419
0709 12:28:58.720974 (+  1164us) spinlock_profiling.cc:243] Waited 81.9 ms on lock 0x7b54002e2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:58.721033 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:58.721251 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50723,"prepare.queue_time_us":11964,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":3977,"raft.queue_time_us":58334,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":21815,"spinlock_wait_cycles":254981760,"thread_start_us":95405,"threads_started":1,"wal-append.queue_time_us":280452}
I20260709 12:28:58.783156  5552 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:28:58.686952 15702 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:58.786151  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 89135.9us
I20260709 12:28:58.760128 15518 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:58.787217  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:58.754441 15663 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.687s	user 0.013s	sys 0.001s
I20260709 12:28:58.643321 15469 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.633641 15707 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.745468 15336 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:58.764844 15735 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:28:58.752017 15719 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:28:58.813395  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:58.840137 15518 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.840646 15719 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.760695 15777 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:58.842240 15719 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:58.799075 15737 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:28:58.785446 15509 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:58.791489 15716 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:28:58.842864 15737 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:58.828960  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:58.843398 15737 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:28:58.845504 15775 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 12:28:58.847548 15771 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 12:28:58.847700 15518 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:58.849066  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1739) took 1011 ms. Trace:
I20260709 12:28:58.849548  5704 rpcz_store.cc:276] 0709 12:28:57.837693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:57.998759 (+161066us) service_pool.cc:224] Handling call
0709 12:28:57.998943 (+   184us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:58.005969 (+  7026us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:58.005989 (+    20us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:58.010200 (+  4211us) log.cc:844] Serialized 26 byte log entry
0709 12:28:58.386388 (+376188us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:58.386433 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:58.386514 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:58.848749 (+462235us) raft_consensus.cc:1685] finished
0709 12:28:58.848781 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:58.848967 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6598,"raft.queue_time_us":7148,"raft.run_cpu_time_us":62,"raft.run_wall_time_us":62,"thread_start_us":375834,"threads_started":1,"wal-append.queue_time_us":566605}
I20260709 12:28:58.857633  5701 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:28:58.685868 15589 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.806409 15781 raft_consensus.cc:2764] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election decision vote started in defunct term 29: won
I20260709 12:28:58.815176 15776 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:28:58.886371  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:58.816509 15543 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:28:58.812563 15432 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:28:58.823266 15695 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:58.924950 15481 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.810s	user 0.013s	sys 0.000s
I20260709 12:28:58.800426 15510 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.851671 15534 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:58.952284  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:58.961218  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 301984us
I20260709 12:28:58.816761 15602 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.133s
W20260709 12:28:58.970903  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:58.821866 15663 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.816608 15702 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:58.806094 15608 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.966125 15657 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:58.953845 15776 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:58.951894 15690 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:58.959542 15692 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:59.011366  6277 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
W20260709 12:28:59.004007  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 334954us
I20260709 12:28:58.982949  5557 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:28:58.954900 15432 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:59.029517 15663 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:58.999377 15776 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:59.044510  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:59.048738  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:58.973454 15481 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:58.971144 15695 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:59.060688  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2007) took 1025 ms. Trace:
W20260709 12:28:59.056183  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 448809us
I20260709 12:28:59.035987 15432 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:59.057231  5560 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
W20260709 12:28:59.045607 15688 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.838s	user 0.012s	sys 0.002s
I20260709 12:28:59.073561  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:58.970886 15291 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.044977 15510 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:59.081079  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:59.067530  5554 rpcz_store.cc:276] 0709 12:28:58.035120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:58.450584 (+415464us) service_pool.cc:224] Handling call
0709 12:28:58.451029 (+   445us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:58.451308 (+   279us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:28:58.477516 (+ 26208us) log.cc:844] Serialized 14 byte log entry
0709 12:28:59.060271 (+582755us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:59.060337 (+    66us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:28:59.060405 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:59.060478 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:59.060609 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4412,"prepare.queue_time_us":136981,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":125956,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":41625,"thread_start_us":581769,"threads_started":2,"wal-append.queue_time_us":507291}
I20260709 12:28:59.056910 15663 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:28:59.050930 15711 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:59.083935  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:59.105806  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:59.076436 15488 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.838s	user 0.011s	sys 0.039s
W20260709 12:28:59.153227  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 69397.3us
I20260709 12:28:59.071484 15647 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:59.161149  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 55452us
I20260709 12:28:59.167078  5526 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:59.157068 15488 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.918s	user 0.011s	sys 0.039s
I20260709 12:28:59.051206 15778 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.047s
I20260709 12:28:58.999888 15691 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.069875 15780 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:28:59.074823 15583 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:28:59.181699  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:59.187759  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:59.098307 15699 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:59.157433 15663 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:59.150506 15794 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:28:59.160882 15510 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:28:59.197139  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 355624us
I20260709 12:28:59.078037 15748 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.109s
I20260709 12:28:59.177881 15509 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:28:59.207221  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 352330us
W20260709 12:28:59.244028  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:28:59.245245  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:59.077627 15688 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.187915 15291 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:28:59.216977  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 241902us
I20260709 12:28:59.207882 15663 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:59.194341 15583 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:59.076697 15657 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.260795 15583 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:28:59.243521 15509 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:28:59.181279 15647 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.035s
W20260709 12:28:59.263118  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:59.263772  6278 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:28:59.206835 15789 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:59.214376 15595 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:59.211516 15596 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:59.268314  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:59.271589  6271 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:28:59.214066 15730 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:28:59.273665  6278 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23. Ops: []
I20260709 12:28:59.261986 15707 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:59.285748 15507 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.720s	user 0.012s	sys 0.001s
I20260709 12:28:59.266687 15748 consensus_queue.cc:1243] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 24.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:28:59.276872 15510 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:59.303237  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:59.265198 15803 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:28:59.268129 15509 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:59.277146 15802 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:59.277024 15795 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:28:59.295998 15647 consensus_queue.cc:1243] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 30.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:28:59.296155 15792 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:59.303679 15608 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:59.288535 15595 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:59.286911 15737 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.315266 15702 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:59.285729 15789 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.336783 15695 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:59.341480 15793 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.135s	user 0.001s	sys 0.001s
I20260709 12:28:59.314289 15809 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.046s
I20260709 12:28:59.347575 15715 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Stepping down as leader of term 29
W20260709 12:28:59.347836 15617 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.875s	user 0.011s	sys 0.000s
W20260709 12:28:59.375286 15701 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.908s	user 0.011s	sys 0.000s
I20260709 12:28:59.338534 15792 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:59.380090 15646 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.936s	user 0.008s	sys 0.006s
I20260709 12:28:59.373049 15617 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:59.381711 15719 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.539s	user 0.015s	sys 0.000s
I20260709 12:28:59.383657 15617 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:59.380837 15646 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.365422 15715 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:59.328462 15802 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:59.386447 15715 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:59.383710 15735 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.549s	user 0.010s	sys 0.000s
I20260709 12:28:59.387809 15802 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:59.388832 15688 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:59.388545 15735 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.420670 15792 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:59.418356 15789 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:28:59.460436  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 272459us
I20260709 12:28:59.463068  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3463) took 1190 ms. Trace:
I20260709 12:28:59.464835  5526 leader_election.cc:400] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
I20260709 12:28:59.465762  5526 leader_election.cc:403] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:28:59.466492  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:59.469048  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 235744us
I20260709 12:28:59.471132  5703 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
W20260709 12:28:59.472908  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 229228us
I20260709 12:28:59.463796  5553 rpcz_store.cc:276] 0709 12:28:58.272135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:59.005659 (+733524us) service_pool.cc:224] Handling call
0709 12:28:59.011431 (+  5772us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:59.011872 (+   441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:59.011913 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:59.032710 (+ 20797us) log.cc:844] Serialized 26 byte log entry
0709 12:28:59.266502 (+233792us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:59.266594 (+    92us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:59.266725 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:59.462216 (+195491us) raft_consensus.cc:1685] finished
0709 12:28:59.462288 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:59.462748 (+   460us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":226791,"threads_started":1,"wal-append.queue_time_us":272630}
W20260709 12:28:59.478780  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 207463us
I20260709 12:28:59.493526  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:28:59.479771  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:59.314577  6278 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 30: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:59.297003  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3546) took 1150 ms. Trace:
I20260709 12:28:59.488340  6271 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: [21.15-23.16]
I20260709 12:28:59.344023 15510 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.511240 15595 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:59.514613 15735 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:59.515295 15481 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:59.311154 15507 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:59.533829 15791 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:59.509439 15803 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:28:59.422720 15776 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:59.497594  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:59.522085  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:28:59.531334  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:59.305672 15290 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.207s
W20260709 12:28:59.539958  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 46706.5us
I20260709 12:28:59.511899 15646 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:59.540699 15818 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:28:59.563732 15818 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:59.564517 15818 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:28:59.565973  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:59.566694  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:59.539436  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:28:59.570361  5633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3444) took 1923 ms (client timeout 2249 ms). Trace:
W20260709 12:28:59.571044  5633 rpcz_store.cc:269] 0709 12:28:57.646303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:57.646498 (+   195us) service_pool.cc:224] Handling call
0709 12:28:59.569764 (+1923266us) spinlock_profiling.cc:243] Waited 683 ms on lock 0x7b540032cdd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:59.570051 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1570517760}
I20260709 12:28:59.573400  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:28:59.573705  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1747) took 2222 ms. Trace:
I20260709 12:28:59.574110  6272 rpcz_store.cc:276] 0709 12:28:57.350969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:57.392144 (+ 41175us) service_pool.cc:224] Handling call
0709 12:28:57.411357 (+ 19213us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:57.739306 (+327949us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:59.573067 (+1833761us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b54006bdad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:59.573461 (+   394us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":741646080}
I20260709 12:28:59.504968  5707 rpcz_store.cc:276] 0709 12:28:58.146788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:58.750747 (+603959us) service_pool.cc:224] Handling call
0709 12:28:58.751054 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:58.751251 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:28:59.151835 (+400584us) log.cc:844] Serialized 14 byte log entry
0709 12:28:59.296505 (+144670us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:59.296583 (+    78us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:59.296647 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:59.296749 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:59.296924 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28829,"prepare.queue_time_us":450927,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":3294,"raft.queue_time_us":18944,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":6159,"thread_start_us":490672,"threads_started":2,"wal-append.queue_time_us":188400}
I20260709 12:28:59.511250 15699 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26.
W20260709 12:28:59.590204  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:59.599216  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:59.593089  5559 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:28:59.606379 15657 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:28:59.607308  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 111024us
I20260709 12:28:59.607558 15657 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:59.608845  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:28:59.576054 15789 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.034s
I20260709 12:28:59.612794 15803 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:28:59.613302 15803 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:59.613709 15803 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:28:59.587898 15735 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:59.616726 15802 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:28:59.618242 15490 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:28:59.587131  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1768) took 1183 ms. Trace:
I20260709 12:28:59.620785  6272 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 17. (term mismatch)
I20260709 12:28:59.621368  6272 pending_rounds.cc:85] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 16
I20260709 12:28:59.620747 15735 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.621651  6272 pending_rounds.cc:107] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 30.17
I20260709 12:28:59.621982  6272 raft_consensus.cc:2889] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:59.620090 15490 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:59.624375 15490 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.625615  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3457) took 1365 ms. Trace:
I20260709 12:28:59.626116  5552 rpcz_store.cc:276] 0709 12:28:58.259750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:58.805227 (+545477us) service_pool.cc:224] Handling call
0709 12:28:58.815147 (+  9920us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:58.852913 (+ 37766us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:28:58.952497 (+ 99584us) log.cc:844] Serialized 14 byte log entry
0709 12:28:59.271626 (+319129us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:59.460197 (+188571us) log.cc:844] Serialized 26 byte log entry
0709 12:28:59.460579 (+   382us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:28:59.461289 (+   710us) log.cc:844] Serialized 14 byte log entry
0709 12:28:59.461643 (+   354us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:59.461814 (+   171us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:59.625212 (+163398us) raft_consensus.cc:1685] finished
0709 12:28:59.625235 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:59.625385 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56609,"prepare.queue_time_us":136966,"prepare.run_cpu_time_us":170,"prepare.run_wall_time_us":4968,"raft.queue_time_us":146193,"raft.run_cpu_time_us":657,"raft.run_wall_time_us":31207,"spinlock_wait_cycles":6400,"thread_start_us":278057,"threads_started":1,"wal-append.queue_time_us":347930}
I20260709 12:28:59.628335 15490 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:59.575462 15776 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:59.620586 15817 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:28:59.587388  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:28:59.629086 15699 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:59.630035 15737 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:59.640905 15802 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:59.638325 15817 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:59.645821 15802 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.646876 15817 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:59.639796 15290 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:28:59.649030  5627 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:28:59.649524  5627 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:28:59.630975  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 69261.7us
I20260709 12:28:59.632483 15481 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:28:59.633095 15468 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:28:59.639300  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52357.9us
I20260709 12:28:59.639703  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:28:59.642203  6277 rpcz_store.cc:276] 0709 12:28:58.403262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:58.404215 (+   953us) service_pool.cc:224] Handling call
0709 12:28:58.404689 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:59.014695 (+610006us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:59.582735 (+568040us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b5400772754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:59.586865 (+  4130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1394627968}
I20260709 12:28:59.631567 15595 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:28:59.649600  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:59.649061 15290 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:59.654912  6278 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:59.655390  5636 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:59.655185  5627 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.655692 15699 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:59.656841  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:59.658764 15789 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.002s
I20260709 12:28:59.662292 15802 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:59.659056 15468 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:59.649047  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:28:59.668324  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 101891us
I20260709 12:28:59.635438  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:59.667665 15290 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:28:59.672993  5627 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:28:59.673395 15699 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:59.631011 15657 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:59.673560 15481 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.674327 15714 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:59.675323  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:59.696280  5631 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:28:59.675735  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:59.654791 15823 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:28:59.592419 15507 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:59.689793 15726 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:59.689505 15468 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:28:59.690588  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55501.7us
I20260709 12:28:59.676674 15490 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:59.721426  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3830) took 1073 ms. Trace:
I20260709 12:28:59.733875  5630 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:28:59.737046  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:59.586452 15701 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.152s
I20260709 12:28:59.738538  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:59.738925 15701 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:28:59.739358  5628 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:28:59.739583  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:59.739876 15701 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:59.740988  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:59.742462  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:59.743665  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:59.743928 15735 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:59.747226  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:59.758220  6278 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:28:59.750202 15823 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:59.745205 15776 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:28:59.768949 15661 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.071s	user 0.019s	sys 0.005s
I20260709 12:28:59.769142  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:28:59.770231  5707 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: []
I20260709 12:28:59.773372  5631 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 37.
I20260709 12:28:59.774888  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:59.728976  5557 rpcz_store.cc:276] 0709 12:28:58.512395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:59.464711 (+952316us) service_pool.cc:224] Handling call
0709 12:28:59.465569 (+   858us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:59.466154 (+   585us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:28:59.466584 (+   430us) log.cc:844] Serialized 14 byte log entry
0709 12:28:59.562898 (+ 96314us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:28:59.579487 (+ 16589us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:28:59.584148 (+  4661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:59.584202 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:59.585749 (+  1547us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2342,"prepare.queue_time_us":38239,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":79,"raft.queue_time_us":5694,"spinlock_wait_cycles":88608640,"thread_start_us":90148,"threads_started":1,"wal-append.queue_time_us":225128}
I20260709 12:28:59.744777 15735 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.022s
I20260709 12:28:59.744313 15290 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.786496  5633 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:28:59.787298 15583 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.526s	user 0.000s	sys 0.020s
I20260709 12:28:59.788435 15777 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:28:59.789013 15777 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:28:59.790362 15777 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 30.13, Last appended by leader: 13, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:59.791894 15777 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:28:59.793601 15735 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:28:59.794081 15735 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:59.797216  6280 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 37: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:28:59.750572 15490 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.774587 15776 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:59.803346 15667 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:28:59.805462  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47449.9us
I20260709 12:28:59.798079 15661 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:59.809026 15701 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:59.786553 15772 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.120s	user 0.004s	sys 0.000s
W20260709 12:28:59.817993 15540 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.436s	user 0.008s	sys 0.016s
I20260709 12:28:59.818435  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3457) took 2135 ms. Trace:
I20260709 12:28:59.818573 15540 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:59.806932 15707 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:28:59.795338 15735 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.787863 15583 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:59.831866  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:59.818995  5635 rpcz_store.cc:276] 0709 12:28:57.683254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:58.121274 (+438020us) service_pool.cc:224] Handling call
0709 12:28:58.121586 (+   312us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:58.121945 (+   359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:58.121976 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:58.122550 (+   574us) log.cc:844] Serialized 52 byte log entry
0709 12:28:58.420429 (+297879us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:28:58.420484 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:58.420573 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:59.785173 (+1364600us) spinlock_profiling.cc:243] Waited 54.8 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:28:59.817893 (+ 32720us) raft_consensus.cc:1685] finished
0709 12:28:59.817918 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:59.818226 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":773,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":84,"spinlock_wait_cycles":126115072,"thread_start_us":297434,"threads_started":1,"wal-append.queue_time_us":462415}
I20260709 12:28:59.815745  5630 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: [32.23-32.23]
W20260709 12:28:59.830518  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48202.3us
I20260709 12:28:59.839432  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:28:59.840694  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:59.834676  5526 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:59.841620  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:59.809165 15823 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.857313  5631 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
I20260709 12:28:59.857604 15595 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:28:59.858275 15583 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:59.859637 15656 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:28:59.846593 15789 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:28:59.866740  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:59.865617 15681 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:59.872133  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:28:59.864972 15779 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:59.873759  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:59.835990 15818 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:59.880438  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:28:59.886045  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:28:59.852125 15481 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:28:59.892578  5630 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:28:59.893821  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 59305.1us
I20260709 12:28:59.882843 15656 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 37 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:28:59.880687 15779 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:28:59.911466  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1757) took 2212 ms. Trace:
I20260709 12:28:59.895423 15735 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:28:59.913488  6276 rpcz_store.cc:276] 0709 12:28:57.684119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:57.742737 (+ 58618us) service_pool.cc:224] Handling call
0709 12:28:58.078448 (+335711us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:58.090522 (+ 12074us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:59.895892 (+1805370us) spinlock_profiling.cc:243] Waited 335 ms on lock 0x7b5400534dd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:28:59.896140 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":771444096}
I20260709 12:28:59.925973  5627 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:28:59.928789  5628 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:28:59.930439  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:28:59.933470 15711 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.781s	user 0.011s	sys 0.012s
I20260709 12:28:59.933646  5558 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:28:59.934270 15711 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:28:59.935453  5560 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: []
I20260709 12:28:59.936550 15789 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:59.938475 15538 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.466s	user 0.009s	sys 0.015s
W20260709 12:28:59.938697 15509 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.609s	user 0.011s	sys 0.003s
W20260709 12:28:59.938762  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:28:59.939284 15509 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:28:59.939962 15291 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.686s	user 0.013s	sys 0.008s
I20260709 12:28:59.940531  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:59.941427  5552 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:28:59.941867  5627 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:28:59.945683 15776 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:28:59.949745 15818 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:59.961179  5635 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 33. Current term is 34. Ops: []
I20260709 12:28:59.961753  5630 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:28:59.874310  5557 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:38170
I20260709 12:28:59.949786 15290 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:28:59.972059  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 158760us
I20260709 12:28:59.973852  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:28:59.973995 15818 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:28:59.975239 15818 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:59.975687 15818 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:28:59.976661  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1743) took 1905 ms. Trace:
I20260709 12:28:59.977209  5705 rpcz_store.cc:276] 0709 12:28:58.070362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:58.080721 (+ 10359us) service_pool.cc:224] Handling call
0709 12:28:58.081120 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:58.081612 (+   492us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:58.081655 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:58.082615 (+   960us) log.cc:844] Serialized 26 byte log entry
0709 12:28:59.628204 (+1545589us) spinlock_profiling.cc:243] Waited 9.45 ms on lock 0x7b1800292600. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:28:59.822488 (+194284us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:59.830842 (+  8354us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:59.841249 (+ 10407us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:59.975819 (+134570us) raft_consensus.cc:1685] finished
0709 12:28:59.975856 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:59.976177 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":102,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":34631,"spinlock_wait_cycles":21737856,"thread_start_us":120477,"threads_started":1,"wal-append.queue_time_us":1772045}
W20260709 12:28:59.979911 15663 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.708s	user 0.002s	sys 0.018s
I20260709 12:28:59.979411 15595 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:28:59.980433 15663 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.980949  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:59.981722 15595 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:28:59.982482 15737 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:28:59.984879 15747 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:28:59.985644 15747 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.986253  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3550) took 1838 ms. Trace:
I20260709 12:28:59.986972  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3822) took 1938 ms. Trace:
I20260709 12:28:59.980187  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:59.986676  5707 rpcz_store.cc:276] 0709 12:28:58.147668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:59.714651 (+1566983us) service_pool.cc:224] Handling call
0709 12:28:59.760000 (+ 45349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:59.777218 (+ 17218us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:59.985781 (+208563us) spinlock_profiling.cc:243] Waited 5.77 ms on lock 0x7b5400210554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:28:59.986154 (+   373us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13259648}
W20260709 12:28:59.977635 15844 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:28:59.965057 15540 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:28:59.931789 15681 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.992170  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:28:59.912725 15803 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:28:59.930773 15699 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:28:59.982250 15797 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.684s	user 0.002s	sys 0.000s
I20260709 12:28:59.994707  5557 raft_consensus.cc:606] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:28:59.982389 15595 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:28:59.999262  5707 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:28:59.998812 15656 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.24, Last appended by leader: 24, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.001314  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:00.003681 15617 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:00.021128  5557 raft_consensus.cc:614] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:00.005700  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:00.007165  6276 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: []
I20260709 12:29:00.007305  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:00.008052  6280 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:00.008214  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.008666  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 22 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.009879  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:00.023092  6277 raft_consensus.cc:1813] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Handling vote request from an unknown peer cea66aa15be3423880843974acdbf50f
W20260709 12:29:00.009925  5556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3826) took 1816 ms (client timeout 2212 ms). Trace:
I20260709 12:29:00.013717 15747 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:00.015479  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45695.1us
W20260709 12:29:00.029390  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 175419us
I20260709 12:29:00.030309  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:00.031327  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.031756 15249 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:00.032069  6275 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:00.033133  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.032979 15595 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.033898  6275 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 28: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:00.034418 15595 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:00.034617  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.035888  5558 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:29:00.024339 15540 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:00.037132 15617 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:28:59.987493  5561 rpcz_store.cc:276] 0709 12:28:58.014693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:58.038531 (+ 23838us) service_pool.cc:224] Handling call
0709 12:28:58.096379 (+ 57848us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:58.151961 (+ 55582us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:58.151998 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:58.152669 (+   671us) log.cc:844] Serialized 26 byte log entry
0709 12:28:59.529053 (+1376384us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b18001fe7c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:28:59.559174 (+ 30121us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:28:59.559242 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:59.582056 (+ 22814us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:59.912320 (+330264us) raft_consensus.cc:1685] finished
0709 12:28:59.912344 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:28:59.953248 (+ 40904us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":85965,"raft.queue_time_us":55780,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":3271,"spinlock_wait_cycles":4657920,"thread_start_us":29692,"threads_started":1,"wal-append.queue_time_us":1514430}
I20260709 12:29:00.039702  5552 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:29:00.018481  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:00.026420  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.026692  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.027346  6277 raft_consensus.cc:2368] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 24.
W20260709 12:29:00.024585  5556 rpcz_store.cc:269] 0709 12:28:58.193858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:58.439796 (+245938us) service_pool.cc:224] Handling call
0709 12:29:00.004343 (+1564547us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400026cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:00.009677 (+  5334us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2492172032}
I20260709 12:28:59.963868 15538 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:00.046180  6272 raft_consensus.cc:2410] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 31 because replica has last-logged OpId of term: 30 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 19.
I20260709 12:29:00.049054  6280 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:29:00.050454 15681 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:00.050328  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:00.056845 15837 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.169s	user 0.002s	sys 0.000s
I20260709 12:29:00.057603  5705 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:00.058245  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:00.050760  6276 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23. Ops: []
I20260709 12:29:00.059659  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:00.051504 15540 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:00.059437  5707 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:29:00.042519 15663 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:00.063249  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:00.064147 15774 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.488s	user 0.002s	sys 0.000s
I20260709 12:29:00.064688  5710 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:00.060608  5705 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:00.067809  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:00.068487  5705 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:00.068743  5704 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:29:00.068925  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:00.069758  5561 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:00.070778  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3547) took 1923 ms. Trace:
I20260709 12:29:00.071219  5704 rpcz_store.cc:276] 0709 12:28:58.147008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:58.851823 (+704815us) service_pool.cc:224] Handling call
0709 12:28:58.852264 (+   441us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:00.069296 (+1217032us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:00.070465 (+  1169us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400451ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:00.070712 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2796582784}
W20260709 12:29:00.071835  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:00.073232 15841 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260709 12:29:00.073745 15803 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:00.075970  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:00.077855  5709 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:00.078763  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:00.078990  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:00.080235  5554 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:00.081286 15509 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:00.081694  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:00.082703  5708 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:00.083547  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:00.083931  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.086062 15779 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:00.086287 15432 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.001s	user 0.015s	sys 0.007s
I20260709 12:29:00.087134 15661 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:00.086905 15432 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:00.083575  5557 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:00.088464 15818 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:00.064679 15538 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:00.090279 15818 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:00.092646  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:00.093021 15249 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:29:00.093279  5599 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:00.093688 15249 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:00.094439 15805 raft_consensus.cc:2764] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:29:00.094775  5705 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 37.
I20260709 12:29:00.095577  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:00.094985 15249 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.081874  5553 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:29:00.095934  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:00.096489  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:00.098894  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2014) took 1933 ms. Trace:
I20260709 12:29:00.084345  5554 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 27
W20260709 12:29:00.105955  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.107283  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.108450  5704 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:29:00.109148  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3956) took 1907 ms. Trace:
I20260709 12:29:00.109514 15540 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:00.109438  5704 rpcz_store.cc:276] 0709 12:28:58.202046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.107587 (+1905541us) service_pool.cc:224] Handling call
0709 12:29:00.108008 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:00.108854 (+   846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:00.109067 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:00.110420 15509 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:29:00.111342 15509 consensus_queue.cc:1243] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 27.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:29:00.105072  5555 rpcz_store.cc:276] 0709 12:28:58.165353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:58.473959 (+308606us) service_pool.cc:224] Handling call
0709 12:28:58.474318 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:58.474711 (+   393us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:28:58.739202 (+264491us) log.cc:844] Serialized 14 byte log entry
0709 12:28:59.008211 (+269009us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:59.532098 (+523887us) spinlock_profiling.cc:243] Waited 1.97 ms on lock 0x7b2c000fb608. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:28:59.542950 (+ 10852us) log.cc:844] Serialized 26 byte log entry
0709 12:28:59.543432 (+   482us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:59.544052 (+   620us) log.cc:844] Serialized 14 byte log entry
0709 12:28:59.547583 (+  3531us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:59.553701 (+  6118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:28:59.981244 (+427543us) spinlock_profiling.cc:243] Waited 3.08 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:00.098427 (+117183us) raft_consensus.cc:1685] finished
0709 12:29:00.098455 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:00.098680 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":122044,"prepare.queue_time_us":130653,"prepare.run_cpu_time_us":156,"prepare.run_wall_time_us":171,"raft.queue_time_us":109174,"raft.run_cpu_time_us":619,"raft.run_wall_time_us":70490,"spinlock_wait_cycles":12542848,"thread_start_us":419016,"threads_started":2,"wal-append.queue_time_us":475726}
I20260709 12:29:00.113052  6274 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: [29.20-29.20]
I20260709 12:29:00.112023 15776 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:00.114642 15780 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:00.115175 15776 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.115547  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:00.116765  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:00.117542 15776 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:00.117733 15803 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:29:00.118299 15803 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:00.115198 15780 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:00.116829  5705 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:29:00.121096  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.121402  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3963) took 1899 ms. Trace:
I20260709 12:29:00.121799  5705 rpcz_store.cc:276] 0709 12:28:58.221618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.116141 (+1894523us) service_pool.cc:224] Handling call
0709 12:29:00.116569 (+   428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:00.121117 (+  4548us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:00.121323 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:00.122387  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.108335 15432 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:00.124526 15805 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:00.114683  6272 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
I20260709 12:29:00.125195 15805 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.127502 15805 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:00.124758  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.117556  6273 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 33: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:00.133265  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.129683  5561 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:29:00.112154 15490 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:00.140698  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44985.9us
I20260709 12:29:00.141947  5708 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
I20260709 12:29:00.140059  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:00.110023 15538 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:00.119592 15803 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.137441  5710 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:00.146221  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:00.152947  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.135540 15779 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:00.154702  6276 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:29:00.157212  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3954) took 1953 ms. Trace:
I20260709 12:29:00.162899  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:00.161510  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3965) took 1782 ms. Trace:
I20260709 12:29:00.164779  5705 rpcz_store.cc:276] 0709 12:28:58.372294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.127968 (+1755674us) service_pool.cc:224] Handling call
0709 12:29:00.151071 (+ 23103us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:00.151465 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:00.151491 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:00.154635 (+  3144us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:29:00.154679 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:00.154749 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:00.154854 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:00.161510  5708 rpcz_store.cc:276] 0709 12:28:58.198276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.084063 (+1885787us) service_pool.cc:224] Handling call
0709 12:29:00.134790 (+ 50727us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:00.151572 (+ 16782us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:00.151840 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:00.175083  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:00.175668  5710 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:29:00.176499  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.167456 15490 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:00.170120 15805 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:00.191237  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:00.192025  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:00.193943  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:00.193931  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:00.194305  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:00.176216 15538 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:00.212798  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 117336us
I20260709 12:29:00.214798  6276 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 28: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:00.219069  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:00.188107 15690 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.024s
I20260709 12:29:00.239651  5705 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:00.244288  5554 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:00.190460 15490 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.225893 15803 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:00.246866  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3951) took 2059 ms. Trace:
I20260709 12:29:00.258493 15478 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:29:00.254916  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:00.269802 15853 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
W20260709 12:29:00.269802 15851 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:29:00.264943  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:29:00.260398 15478 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:00.276659 15478 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:00.278163 15478 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:00.288632  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 50772.2us
I20260709 12:29:00.290320  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:00.291266 15481 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:29:00.293030 15481 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:29:00.294837 15840 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.407s	user 0.001s	sys 0.000s
W20260709 12:29:00.305462  5703 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3955) took 2103 ms (client timeout 2100 ms). Trace:
W20260709 12:29:00.306188  5703 rpcz_store.cc:269] 0709 12:28:58.201686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:58.498124 (+296438us) service_pool.cc:224] Handling call
0709 12:29:00.305134 (+1807010us) spinlock_profiling.cc:243] Waited 956 ms on lock 0x7b540026e654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:00.305332 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2199925504}
I20260709 12:29:00.307876  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:00.345996  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:00.347201  5703 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
W20260709 12:29:00.314133  5701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3964) took 2086 ms (client timeout 2666 ms). Trace:
I20260709 12:29:00.266678 15509 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:00.315227 15539 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:29:00.316095  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 76301.3us
I20260709 12:29:00.320281 15468 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:00.321435 15538 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:00.280007 15478 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:00.370280  5524 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:00.322904  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:00.371670  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.371990 15657 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:00.325340 15780 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.329562  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:00.329761 15701 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:00.331054  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.332125  6280 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:29:00.334556 15617 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:00.339448 15469 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:00.248389  5709 rpcz_store.cc:276] 0709 12:28:58.183595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.078861 (+1895266us) service_pool.cc:224] Handling call
0709 12:29:00.079249 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:00.079478 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:29:00.079811 (+   333us) log.cc:844] Serialized 14 byte log entry
0709 12:29:00.241478 (+161667us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:00.241668 (+   190us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:00.241759 (+    91us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:00.241926 (+   167us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:00.242504 (+   578us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2318,"prepare.queue_time_us":156,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":2569,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":3748,"thread_start_us":157572,"threads_started":1}
I20260709 12:29:00.372596 15509 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:00.351267  5558 raft_consensus.cc:2410] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:29:00.351543 15730 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:00.353468 15746 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.375772  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:00.312945  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:00.381980  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:00.360931  5710 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
W20260709 12:29:00.357424  5701 rpcz_store.cc:269] 0709 12:28:58.227295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:58.678928 (+451633us) service_pool.cc:224] Handling call
0709 12:29:00.313668 (+1634740us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b5400066654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:00.313911 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30222080}
I20260709 12:29:00.365739  6274 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:29:00.372032 15580 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:00.375603  5708 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:00.365660 15539 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:00.390353  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:00.393957  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.370339 15432 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.021s
I20260709 12:29:00.382864 15509 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:00.398459  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:00.399998  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 87208us
I20260709 12:29:00.373088 15777 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.024s
I20260709 12:29:00.382587 15730 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:00.407147  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:00.407801  5709 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:00.372656 15657 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:00.392287 15580 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:00.405326  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:00.411649  5708 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:00.405227  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:00.406224 15432 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:00.392198 15539 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:00.410095  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:00.397286 15509 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:00.409175 15746 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:29:00.401466 15803 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:00.414968  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:00.412858 15818 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:00.412878 15595 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:00.413749 15432 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:00.443315 15730 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.409649 15469 raft_consensus.cc:2764] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:29:00.429832  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:00.437479 15580 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.450862  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:00.409240 15538 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.016s
I20260709 12:29:00.419190 15803 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:00.444125 15789 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:00.457309  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.427009 15791 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:00.445935 15657 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:00.463600 15791 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:00.469321  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52094.9us
I20260709 12:29:00.461068 15595 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:00.462738 15803 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:00.466389 15468 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:00.463217 15481 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:29:00.491431  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.497359  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3821) took 2482 ms. Trace:
I20260709 12:29:00.497822  5559 rpcz_store.cc:276] 0709 12:28:58.014490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:59.525654 (+1511164us) service_pool.cc:224] Handling call
0709 12:28:59.526438 (+   784us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:59.637573 (+111135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:00.497092 (+859519us) spinlock_profiling.cc:243] Waited 20.5 ms on lock 0x7b540044eb54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:00.497288 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47172992}
I20260709 12:29:00.476311 15580 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:29:00.503369 15842 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.624s	user 0.000s	sys 0.004s
I20260709 12:29:00.504614 15730 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:00.513813 15805 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:00.516952 15780 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:00.527031  5709 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:00.532187  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:00.533056  5706 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:29:00.533907  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:00.534157  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3967) took 2023 ms. Trace:
I20260709 12:29:00.535018  5556 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:29:00.521504 15805 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:00.538815  5559 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:29:00.534683  5706 rpcz_store.cc:276] 0709 12:28:58.510234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.531916 (+2021682us) service_pool.cc:224] Handling call
0709 12:29:00.532494 (+   578us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:00.533773 (+  1279us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:00.534050 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:29:00.541949  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.544588  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:00.545518  5554 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 29: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:00.545387  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:00.544942  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:00.563488  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 133827us
I20260709 12:29:00.568634  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:00.572902 15856 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.297s	user 0.004s	sys 0.000s
I20260709 12:29:00.575582  5552 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 32: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:00.592562 15690 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:00.593278 15690 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:00.594136 15725 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 33.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:00.595618 15690 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:00.597028 15725 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:00.597532 15725 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:00.598824 15725 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.600476 15725 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:00.609750  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:00.612569  5526 leader_election.cc:400] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 22 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 22. Current term is 24.
W20260709 12:29:00.613498  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 82321.1us
I20260709 12:29:00.617961  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:00.618253  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:00.618964  5630 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 33: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:00.626072 15509 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:00.629006  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:00.630396  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:00.631587  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:00.632539  5631 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:00.632717  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:00.633337  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1747) took 1822 ms. Trace:
I20260709 12:29:00.633742  5706 rpcz_store.cc:276] 0709 12:28:58.810904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.568310 (+1757406us) service_pool.cc:224] Handling call
0709 12:29:00.632701 (+ 64391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:00.633001 (+   300us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:00.633032 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:00.633084 (+    52us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:00.633111 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:00.633172 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:00.633286 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:00.634922  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:00.635761  5635 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:29:00.635883 15490 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:00.636870 15726 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:00.637403 15726 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.639349  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:00.637425  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:00.536089 15432 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:00.584724 15478 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:29:00.644789  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:00.644785  5632 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:00.642710  5559 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:00.647799  5703 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:00.648646 15692 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:00.649724  5703 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: [27.17-27.17]
W20260709 12:29:00.651705  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.654655  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3551) took 1579 ms. Trace:
I20260709 12:29:00.655668  5703 rpcz_store.cc:276] 0709 12:28:59.075776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.648793 (+1573017us) service_pool.cc:224] Handling call
0709 12:29:00.649333 (+   540us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:00.654292 (+  4959us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:00.654583 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:00.631672 15539 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.662458  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:00.665735  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:00.661247  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1748) took 1850 ms. Trace:
I20260709 12:29:00.666817  5561 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:00.670279  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:00.640478 15690 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:00.640234 15726 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:00.673719  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:00.664813 15792 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.851s	user 0.004s	sys 0.033s
I20260709 12:29:00.576321 15791 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:00.616149 15595 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:00.672418  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:00.676836  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:00.689814  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:00.685182  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:00.731411  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.685395  5636 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:29:00.666870  5706 rpcz_store.cc:276] 0709 12:28:58.811224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.634694 (+1823470us) service_pool.cc:224] Handling call
0709 12:29:00.635145 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:00.654302 (+ 19157us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:00.654348 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:00.660882 (+  6534us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:00.660938 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:00.661022 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:00.661181 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11385,"raft.queue_time_us":11502,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100}
W20260709 12:29:00.735150  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 122818us
W20260709 12:29:00.669323  5675 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.544s (SENT) [suppressed 24 similar messages]
I20260709 12:29:00.649727 15511 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:29:00.713286  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54633.4us
I20260709 12:29:00.691200  5631 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:00.790917  5636 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:00.783669  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:00.684713  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:00.825163  5557 raft_consensus.cc:2410] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:29:00.824982  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:00.832772  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 160436us
W20260709 12:29:00.762300 15864 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.199s	user 0.001s	sys 0.001s
I20260709 12:29:00.834424  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:00.715197  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
W20260709 12:29:00.730888 15863 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.205s	user 0.001s	sys 0.001s
W20260709 12:29:00.796507 15792 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.172s	user 0.007s	sys 0.035s
I20260709 12:29:00.735945 15432 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:00.684890 15509 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:00.684731 15726 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:29:00.693527 15468 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:00.857373  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.859911  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3554) took 1676 ms. Trace:
I20260709 12:29:00.820526 15511 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 22 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:00.731288 15803 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:00.663265 15780 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.173s
W20260709 12:29:00.864419  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 225276us
I20260709 12:29:00.860569  5703 rpcz_store.cc:276] 0709 12:28:59.183365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.851364 (+1667999us) service_pool.cc:224] Handling call
0709 12:29:00.859038 (+  7674us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:00.859395 (+   357us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:00.859433 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:00.859516 (+    83us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:00.859561 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:00.859627 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:00.859779 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:00.689244 15690 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:00.879935  6271 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 25. Current term is 26. Ops: []
I20260709 12:29:00.880800  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.875200  5636 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:00.881438  6280 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:00.882371  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:00.885454  5557 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:00.883080  6275 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:00.883889  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:00.887343  5703 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:00.887939  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:00.869443 15580 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:29:00.889678  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 54532.6us
I20260709 12:29:00.870824 15726 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:00.898911  5636 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:00.899750  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:00.874258 15468 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:00.888442 15701 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:00.904178  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:00.912410  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.907914  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:00.909003  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:00.902408  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 233114us
I20260709 12:29:00.913458  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:00.882018 15478 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:29:00.915241  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.914534  5708 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:00.919355  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:00.920023  5554 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:00.924809  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:00.902277 15539 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:00.915654 15865 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.252s	user 0.000s	sys 0.002s
I20260709 12:29:00.928279  5553 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:00.912497 15490 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:29:00.917196 15580 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:00.954413  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:00.936982  5555 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 31 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 19.
W20260709 12:29:00.989424  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:01.009745  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:01.030556  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.948604  5561 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
W20260709 12:29:00.943756  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:00.961041  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:00.949399  5710 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: []
I20260709 12:29:00.949055  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:00.961648  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3827) took 2452 ms. Trace:
I20260709 12:29:00.977404  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3966) took 2590 ms. Trace:
W20260709 12:29:00.953819 15746 consensus_meta.cc:287] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.532s	user 0.012s	sys 0.000s
W20260709 12:29:00.930971 15869 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
W20260709 12:29:01.028610  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 83310.8us
I20260709 12:29:00.926422 15509 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:00.950093 15794 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:29:01.034099  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 79772.8us
I20260709 12:29:01.036387  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3556) took 1848 ms. Trace:
I20260709 12:29:00.923354 15692 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:00.921583 15871 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.026943  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:01.038075  5560 rpcz_store.cc:276] 0709 12:28:58.509431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:59.073916 (+564485us) service_pool.cc:224] Handling call
0709 12:28:59.151857 (+ 77941us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:28:59.935929 (+784072us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:00.961400 (+1025471us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b540044e154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:00.961592 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1801183488}
I20260709 12:29:01.039850  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:00.931061 15778 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:00.951134 15539 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.035781  5701 rpcz_store.cc:276] 0709 12:28:58.387414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.471758 (+2084344us) service_pool.cc:224] Handling call
0709 12:29:00.535987 (+ 64229us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:00.606996 (+ 71009us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:00.607043 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:00.659233 (+ 52190us) log.cc:844] Serialized 26 byte log entry
0709 12:29:00.868786 (+209553us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:00.868828 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:00.868892 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:00.976823 (+107931us) raft_consensus.cc:1685] finished
0709 12:29:00.976854 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:00.977244 (+   390us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1877,"raft.queue_time_us":20958,"spinlock_wait_cycles":11205504,"thread_start_us":187478,"threads_started":1,"wal-append.queue_time_us":276439}
I20260709 12:29:00.937080 15596 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:00.933583 15715 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:00.938546 15803 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:00.943665 15818 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:29:01.047574  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:01.046315  5710 rpcz_store.cc:276] 0709 12:28:59.187577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.948752 (+1761175us) service_pool.cc:224] Handling call
0709 12:29:00.949120 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:01.036083 (+ 86963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.036297 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:01.066479  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:01.068836  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:01.034904 15580 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:00.936906 15868 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:29:00.936854 15835 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 1.106s	user 0.000s	sys 0.003s
I20260709 12:29:00.918766 15701 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.056885  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:00.896939 15511 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.066262 15534 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.087710  5629 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:01.060513  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:01.038754 15735 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:00.961061 15468 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:01.075482  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 182116us
I20260709 12:29:01.060254  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3555) took 1876 ms. Trace:
I20260709 12:29:01.089661  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:01.049192 15794 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:01.089345  5708 rpcz_store.cc:276] 0709 12:28:59.183722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.927259 (+1743537us) service_pool.cc:224] Handling call
0709 12:29:01.048525 (+121266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:01.059848 (+ 11323us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:01.059888 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:01.059945 (+    57us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:01.059988 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.060087 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:01.060202 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:01.090437  5560 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 20. (term mismatch)
I20260709 12:29:01.085412  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:01.094415  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3976) took 1076 ms. Trace:
I20260709 12:29:01.094816  5705 rpcz_store.cc:276] 0709 12:29:00.017699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:01.069005 (+1051306us) service_pool.cc:224] Handling call
0709 12:29:01.088330 (+ 19325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:01.094084 (+  5754us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:01.094111 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:01.094158 (+    47us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:29:01.094188 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.094240 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:01.094363 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:01.098248  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:01.091763  5560 pending_rounds.cc:85] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 19
I20260709 12:29:01.099881  5560 pending_rounds.cc:107] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 26.20
I20260709 12:29:01.100155  5560 raft_consensus.cc:2889] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:01.063061 15596 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:01.076529 15580 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:01.102867  5710 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: []
I20260709 12:29:01.076164 15778 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:00.949889 15726 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.105007  5554 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:29:01.106015  5559 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:29:01.083002 15538 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:29:01.111217  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44865.4us
I20260709 12:29:01.112835  5561 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:29:01.089756 15490 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:01.093945 15794 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:01.091231 15735 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:01.097201 15714 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:29:01.122702  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:01.077178 15539 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.123574  5708 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:01.066395 15478 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.087702 15871 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:01.130249  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:01.077736 15692 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.133363  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:01.136916  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 47350.4us
W20260709 12:29:01.119184 15805 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.552s	user 0.004s	sys 0.008s
W20260709 12:29:01.219111  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 81212.2us
I20260709 12:29:01.120431 15509 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:29:01.220451  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 85395.5us
I20260709 12:29:01.119117 15468 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:01.220749  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:01.218992  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1749) took 1221 ms. Trace:
I20260709 12:29:01.118839 15735 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:01.269572  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1744) took 2974 ms. Trace:
W20260709 12:29:01.278280  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 144501us
I20260709 12:29:01.286012  5631 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:29:01.286295  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:01.099506 15534 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.287070  5705 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:29:01.303229  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:01.278926 15865 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
W20260709 12:29:01.349807  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52876.6us
I20260709 12:29:01.278131 15730 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:29:01.358086  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 249215us
I20260709 12:29:01.355360  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:01.359262  6274 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:01.356055  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:01.277046  5702 rpcz_store.cc:276] 0709 12:28:58.134271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:58.141947 (+  7676us) service_pool.cc:224] Handling call
0709 12:28:58.142295 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:28:58.251208 (+108913us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:58.251239 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:28:58.261936 (+ 10697us) log.cc:844] Serialized 26 byte log entry
0709 12:29:00.297834 (+2035898us) spinlock_profiling.cc:243] Waited 9.41 ms on lock 0x7b180006da20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:29:00.536311 (+238477us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:00.576694 (+ 40383us) log.cc:844] Serialized 14 byte log entry
0709 12:29:00.588321 (+ 11627us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.078547 (+490226us) spinlock_profiling.cc:243] Waited 96.6 ms on lock 0x7b5400066154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:01.078605 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:01.102266 (+ 23661us) raft_consensus.cc:1685] finished
0709 12:29:01.102308 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:01.108567 (+  6259us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15090,"prepare.queue_time_us":20159,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":49684,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":11014,"spinlock_wait_cycles":243924224,"thread_start_us":171011,"threads_started":1,"wal-append.queue_time_us":2282525}
W20260709 12:29:01.362133  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 143110us
I20260709 12:29:01.364140  5561 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:29:01.365554  5705 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:29:01.366427  5704 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 12:29:01.366487  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3981) took 1274 ms. Trace:
I20260709 12:29:01.367048  5705 rpcz_store.cc:276] 0709 12:29:00.091713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:01.310878 (+1219165us) service_pool.cc:224] Handling call
0709 12:29:01.365113 (+ 54235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:01.366153 (+  1040us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.366413 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:01.366792  5704 pending_rounds.cc:85] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 14
I20260709 12:29:01.369233  5704 pending_rounds.cc:107] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 21.15
I20260709 12:29:01.369601  5704 raft_consensus.cc:2889] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:01.370003  5704 pending_rounds.cc:107] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 23.16
I20260709 12:29:01.370285  5704 raft_consensus.cc:2889] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:01.237983 15646 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.111s
W20260709 12:29:01.374256 15878 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.290s	user 0.001s	sys 0.000s
I20260709 12:29:01.374467 15872 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:01.375016  5702 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:01.375739  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3562) took 1185 ms. Trace:
I20260709 12:29:01.376060  5702 rpcz_store.cc:276] 0709 12:29:00.190227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:01.361889 (+1171662us) service_pool.cc:224] Handling call
0709 12:29:01.374803 (+ 12914us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:01.375457 (+   654us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.375681 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:01.297736 15690 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:01.233889 15805 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.220018 15818 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.141s
W20260709 12:29:01.378849  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:01.379041  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:01.238332 15803 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.124s
I20260709 12:29:01.378667  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:01.386132  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3552) took 2301 ms. Trace:
I20260709 12:29:01.390087  5633 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:29:01.479554  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:01.481189  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:01.487392  5708 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:01.390048  5706 rpcz_store.cc:276] 0709 12:28:59.076103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.791634 (+1715531us) service_pool.cc:224] Handling call
0709 12:29:00.867514 (+ 75880us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:00.885794 (+ 18280us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:00.885828 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:00.901018 (+ 15190us) log.cc:844] Serialized 26 byte log entry
0709 12:29:01.093130 (+192112us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:29:01.093185 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.093276 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:01.376804 (+283528us) raft_consensus.cc:1685] finished
0709 12:29:01.376825 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:01.376969 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13005,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":6307,"thread_start_us":138093,"threads_started":1,"wal-append.queue_time_us":362739}
I20260709 12:29:01.490288  5601 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:01.497866  5702 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:01.383713 15730 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.382424 15468 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:01.507936  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3572) took 1115 ms. Trace:
I20260709 12:29:01.505600  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3571) took 1115 ms. Trace:
I20260709 12:29:01.394210 15803 consensus_queue.cc:1243] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 30.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:29:01.509096  5708 rpcz_store.cc:276] 0709 12:29:00.392848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:01.488824 (+1095976us) service_pool.cc:224] Handling call
0709 12:29:01.490142 (+  1318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:01.506908 (+ 16766us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:01.506973 (+    65us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:01.507085 (+   112us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:01.507155 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.507380 (+   225us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:01.507742 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:01.509810  5702 rpcz_store.cc:276] 0709 12:29:00.389851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:01.384734 (+994883us) service_pool.cc:224] Handling call
0709 12:29:01.385131 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:01.505272 (+120141us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.505523 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:29:01.512868  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 160620us
I20260709 12:29:01.513761  5705 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:29:01.522321  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3570) took 1251 ms. Trace:
I20260709 12:29:01.522830  5705 rpcz_store.cc:276] 0709 12:29:00.271364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:01.368078 (+1096714us) service_pool.cc:224] Handling call
0709 12:29:01.478673 (+110595us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:01.521946 (+ 43273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.522249 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:29:01.524025  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:01.515139  5706 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: []
I20260709 12:29:01.376902 15714 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.121s
I20260709 12:29:01.531858 15888 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.007s
I20260709 12:29:01.534757 15888 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.535429 15888 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.537163  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3573) took 1142 ms. Trace:
I20260709 12:29:01.537906  5706 rpcz_store.cc:276] 0709 12:29:00.394789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:01.492241 (+1097452us) service_pool.cc:224] Handling call
0709 12:29:01.514897 (+ 22656us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:01.536853 (+ 21956us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.537087 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:01.297001 15596 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.23, Last appended by leader: 23, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.393784 15746 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.133s
W20260709 12:29:01.548338  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 179125us
I20260709 12:29:01.548625  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:01.492231 15509 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:01.375056 15872 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.549795  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:01.556150  5702 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:29:01.573302  5708 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: [29.17-29.17]
W20260709 12:29:01.578985  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:01.357487 15778 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.586513  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3574) took 1184 ms. Trace:
I20260709 12:29:01.588841  5702 rpcz_store.cc:276] 0709 12:29:00.402592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:01.521331 (+1118739us) service_pool.cc:224] Handling call
0709 12:29:01.555671 (+ 34340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:01.575035 (+ 19364us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.586427 (+ 11392us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:01.597386 15890 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:01.546252 15701 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:01.614221 15888 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:01.616920 15805 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:01.616945  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1772) took 2134 ms. Trace:
I20260709 12:29:01.571744 15791 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.070s
I20260709 12:29:01.621381 15791 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.622103 15791 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.617751 15894 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:29:01.528090 15730 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.235176  5701 rpcz_store.cc:276] 0709 12:28:59.923289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:01.067829 (+1144540us) service_pool.cc:224] Handling call
0709 12:29:01.068149 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:01.068396 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:29:01.068674 (+   278us) log.cc:844] Serialized 14 byte log entry
0709 12:29:01.132916 (+ 64242us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:01.132972 (+    56us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:29:01.133029 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.133110 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:01.133238 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2990,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":4975,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":21598,"thread_start_us":58816,"threads_started":1}
I20260709 12:29:01.526353 15468 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:01.651746  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 161548us
I20260709 12:29:01.547151 15580 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.624791 15692 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:01.639236 15803 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:29:01.643158 15871 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:01.644390 15890 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.618656  6279 rpcz_store.cc:276] 0709 12:28:59.251166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:59.251314 (+   148us) service_pool.cc:224] Handling call
0709 12:28:59.251779 (+   465us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:28:59.261968 (+ 10189us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:28:59.262002 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:28:59.276955 (+ 14953us) log.cc:844] Serialized 52 byte log entry
0709 12:28:59.693916 (+416961us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:28:59.694017 (+   101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:28:59.694128 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:00.902200 (+1208072us) spinlock_profiling.cc:243] Waited 1.35 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:01.377414 (+475214us) raft_consensus.cc:1685] finished
0709 12:29:01.377435 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:01.385744 (+  8309us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22333,"raft.queue_time_us":10444,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":6886,"spinlock_wait_cycles":3114880,"thread_start_us":344777,"threads_started":1,"wal-append.queue_time_us":429168}
I20260709 12:29:01.646672 15872 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:01.647769  5703 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.755s	user 0.006s	sys 0.006s
I20260709 12:29:01.657789  5703 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:01.543628 15818 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.064s
W20260709 12:29:01.653656  5636 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.754s	user 0.010s	sys 0.002s
I20260709 12:29:01.661976  5636 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 18. (term mismatch)
I20260709 12:29:01.662681  5636 pending_rounds.cc:85] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 17
I20260709 12:29:01.663015  5636 pending_rounds.cc:107] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 25.18
I20260709 12:29:01.663301  5636 raft_consensus.cc:2889] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:01.652717 15646 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:01.659960  6274 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 22. Current term is 23. Ops: []
I20260709 12:29:01.671059  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2040) took 1005 ms. Trace:
I20260709 12:29:01.672602  5636 rpcz_store.cc:276] 0709 12:29:00.668699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.668855 (+   156us) service_pool.cc:224] Handling call
0709 12:29:00.669241 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:01.670723 (+1001482us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.670981 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:01.633502 15894 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:01.639988 15892 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.017s
I20260709 12:29:01.686120  6277 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:01.652063 15888 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:01.657684 15539 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:01.692342 15888 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.695070 15888 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:01.657222 15803 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:01.695730 15803 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:01.691706 15818 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.700870 15818 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.702594 15580 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:01.703683 15818 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:01.706406 15730 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:01.667560 15871 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:01.670341 15646 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:01.719173 15803 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.723816 15871 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:01.728036 15871 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:01.730253 15432 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.881s	user 0.007s	sys 0.007s
I20260709 12:29:01.731132  5558 raft_consensus.cc:2410] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:29:01.731824 15735 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.734118 15735 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:01.735594 15646 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:01.737031  6272 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:01.761298  5633 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:29:01.736681 15509 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:29:01.736222 15538 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.621s	user 0.007s	sys 0.003s
I20260709 12:29:01.767905 15894 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.768756 15538 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:01.769519 15538 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.770200 15538 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.775446 15871 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.769579 15894 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:01.736830 15791 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:01.736979 15889 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:01.798105  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:01.760664 15803 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:01.803313  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 263334us
W20260709 12:29:01.781646  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 205042us
W20260709 12:29:01.813009  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 304998us
I20260709 12:29:01.788405 15538 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:01.839566  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 215373us
I20260709 12:29:01.803987 15889 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.844095  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:01.846794 15715 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.774s	user 0.010s	sys 0.002s
I20260709 12:29:01.847924  5707 raft_consensus.cc:2410] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 31 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 16.
W20260709 12:29:01.805819 15794 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.674s	user 0.006s	sys 0.005s
W20260709 12:29:01.854862  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:01.861974  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:01.849305 15789 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:01.868826  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:01.870543 15478 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:29:01.872906 15890 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:01.873569 15890 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.874058 15890 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:01.875384 15881 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.810s	user 0.009s	sys 0.008s
I20260709 12:29:01.876695 15902 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:29:01.877005 15789 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.877550  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3474) took 1206 ms. Trace:
I20260709 12:29:01.877909  5557 rpcz_store.cc:276] 0709 12:29:00.670800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.825963 (+155163us) service_pool.cc:224] Handling call
0709 12:29:00.826430 (+   467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:00.891406 (+ 64976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.877314 (+985908us) spinlock_profiling.cc:243] Waited 56.4 ms on lock 0x7b54003a98d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:01.877490 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":129797376}
I20260709 12:29:01.879459  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1784) took 1010 ms. Trace:
I20260709 12:29:01.867825 15778 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.880115 15778 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.855437 15794 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.870209  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:01.878330 15871 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:01.879909  6276 rpcz_store.cc:276] 0709 12:29:00.868680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.868849 (+   169us) service_pool.cc:224] Handling call
0709 12:29:00.877721 (+  8872us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:00.882696 (+  4975us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:00.882725 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:00.897567 (+ 14842us) log.cc:844] Serialized 26 byte log entry
0709 12:29:01.065789 (+168222us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:29:01.065843 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.065930 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:01.874807 (+808877us) spinlock_profiling.cc:243] Waited 7.79 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:01.879153 (+  4346us) raft_consensus.cc:1685] finished
0709 12:29:01.879176 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:01.879401 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":202,"raft.queue_time_us":429,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":5033,"spinlock_wait_cycles":17906944,"thread_start_us":116942,"threads_started":1,"wal-append.queue_time_us":156861}
I20260709 12:29:01.894084 15794 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:01.894244 15490 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:01.894711 15490 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:01.895044 15902 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:29:01.895493 15902 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:29:01.887903  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:01.889483  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:01.890830  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3557) took 2703 ms. Trace:
I20260709 12:29:01.897792  6274 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:01.897737  5710 rpcz_store.cc:276] 0709 12:28:59.187867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:01.065909 (+1878042us) service_pool.cc:224] Handling call
0709 12:29:01.101040 (+ 35131us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:01.105970 (+  4930us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:01.890531 (+784561us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b5400323a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:01.890764 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3559168}
I20260709 12:29:01.891698 15900 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:29:01.901950 15789 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:01.902374 15779 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:29:01.903090 15779 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:01.904179 15779 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.883085 15778 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:01.912118  6274 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:01.911720 15646 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.923240 15646 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:01.926983  5676 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:01.900074 15490 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.931370 15871 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:29:01.931490 15778 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.933652 15490 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:01.896576 15902 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.934072 15778 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.935240 15902 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:01.912164 15890 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:01.924204 15468 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:01.936334 15778 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:01.940011 15794 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.940730 15794 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.943944 15794 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:01.938232 15900 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:29:01.945878 15900 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:01.946798 15900 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.952185 15746 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:29:01.953403 15657 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:29:01.953989 15657 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 23 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:01.955368 15657 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:01.956835 15657 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:29:01.960191 15823 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:01.960482 15890 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.960809 15823 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.961112 15890 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.963120 15823 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:01.957280  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3851) took 1010 ms. Trace:
I20260709 12:29:01.964504 15468 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.965111 15468 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:01.967339 15468 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:01.963799  5558 rpcz_store.cc:276] 0709 12:29:00.946692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.960774 (+ 14082us) service_pool.cc:224] Handling call
0709 12:29:01.957087 (+996313us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b540044e654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:01.957222 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1601579776}
I20260709 12:29:01.926959 15646 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.011s
I20260709 12:29:01.974005 15901 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:29:01.974715 15867 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.150s	user 0.001s	sys 0.012s
I20260709 12:29:01.975708  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:01.964123  6278 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
W20260709 12:29:01.982059  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 111970us
I20260709 12:29:01.976045  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:01.985280  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48082.2us
I20260709 12:29:01.989705  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:01.988639  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:01.987776  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:01.990382  5559 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:01.963753 15890 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:01.997236  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:01.998530  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:01.998775  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:01.999275  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:01.999302  5556 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:01.999439  5561 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:29:02.001917  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:02.002247  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:02.006814  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:02.012403  5554 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:02.009414 15478 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:02.017824 15478 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.019973 15478 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:02.009686  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:02.021262  5560 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:29:02.021186  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:02.023276  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:01.990924 15899 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:02.004235 15509 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:01.998359 15646 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:02.027382  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:02.006263 15903 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:29:02.020394 15791 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:02.006381 15707 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:02.023910 15880 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.005s	sys 0.015s
I20260709 12:29:02.027343 15890 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:29:02.066135  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:02.057894  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:02.000387 15778 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.031s
I20260709 12:29:02.059072  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:02.073271  6274 raft_consensus.cc:2368] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:02.009647 15871 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:02.050184 15899 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:02.070151  5561 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:29:02.050261 15794 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:02.071074  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:02.075135  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:02.061509  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56848.9us
I20260709 12:29:02.071631  6278 pending_rounds.cc:85] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 18
I20260709 12:29:02.079878  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.085688  6278 pending_rounds.cc:107] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 30.19
I20260709 12:29:02.086105  6278 raft_consensus.cc:2889] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:02.075822  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:02.082556  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.082715  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:02.065495 15880 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.005s	sys 0.016s
I20260709 12:29:02.084241  5702 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:29:02.087397  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:02.088809  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:02.065749 15714 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:02.059774 15791 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:02.036661 15894 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:29:02.094553  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:02.062110 15735 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:02.109925  6279 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:02.059980 15779 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:02.095180  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:02.110492  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:02.094017  6273 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:29:02.076761 15903 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:02.108038  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:02.115182  5701 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:02.112875  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.113536  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:02.115764  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:02.116518  5703 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:02.060139 15818 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 12:29:02.059988 15715 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.038s
I20260709 12:29:02.060067 15538 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.038s
I20260709 12:29:02.092118 15794 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:02.062282 15792 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.042s
I20260709 12:29:02.121688  5703 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:02.070974 15730 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.044s
I20260709 12:29:02.087018 15899 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:02.059139 15898 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.061s
I20260709 12:29:02.101624 15896 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:02.138621  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.109545 15791 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:02.097922 15714 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:02.145835  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58590.4us
I20260709 12:29:02.159682  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:02.146646  5708 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 31 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 19.
I20260709 12:29:02.084156 15871 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.121515 15903 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.167644 15903 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:02.168104  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:02.169025  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.169937  5710 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:02.172176  5559 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:29:02.136834 15902 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:02.079010 15707 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.141503 15794 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:02.174000  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:02.205137  5560 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: [25.13-25.13]
W20260709 12:29:02.205406  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:02.159842 15509 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:02.120306 15735 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.226217  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:02.220270  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:02.222921  5561 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: [28.21-28.21]
I20260709 12:29:02.223685  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:02.228883  5559 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:29:02.168608 15714 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:02.185380 15896 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:02.233958  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:02.236480  5560 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:02.236580  5553 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:02.217420 15509 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.198355 15823 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:02.206732 15880 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:29:02.232517 15910 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.243s	user 0.001s	sys 0.001s
I20260709 12:29:02.251520  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:02.253290  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:02.244681 15913 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.109s	user 0.001s	sys 0.001s
I20260709 12:29:02.242004 15595 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:02.233369 15890 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:02.250312 15617 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:02.245777 15919 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:29:02.264312  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:02.250665 15894 consensus_queue.cc:1243] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 29.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:29:02.269731 15735 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:02.263795 15919 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:02.284729 15735 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:02.267277 15792 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.018s
I20260709 12:29:02.285187 15735 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:02.285244 15919 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.266006 15922 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 12:29:02.286741 15919 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:02.287335 15922 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:29:02.288126  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:02.276641 15897 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:02.277704 15890 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:02.282179 15918 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 12:29:02.283267 15789 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:02.285645 15792 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:02.285704 15735 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:02.250139 15823 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:02.295359  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 43927.5us
I20260709 12:29:02.287783 15922 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:02.270929 15617 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:02.265352 15880 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:02.338891  5701 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
W20260709 12:29:02.303025 15911 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
I20260709 12:29:02.304271 15897 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:02.332230  5556 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:02.362558  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:02.366115  6278 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:02.306818 15789 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:02.364259  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 245228us
I20260709 12:29:02.367479  5560 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: [31.17-31.17]
I20260709 12:29:02.370963  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:02.321281 15871 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:02.319046 15707 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:02.353253 15901 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:02.338467 15880 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:02.377944 15880 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:29:02.375623  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 201745us
I20260709 12:29:02.364646 15897 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:02.357091 15792 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:02.408159  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:02.408298  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:02.404985 15871 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:02.307617 15896 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.21, Last appended by leader: 21, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.413146 15914 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 29.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 12:29:02.396814 15778 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:02.391896 15657 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.025s
W20260709 12:29:02.428004  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:02.374945 15900 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.043s
I20260709 12:29:02.430150 15935 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:29:02.431238 15926 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
W20260709 12:29:02.437278  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:02.447235 15914 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:02.426436 15789 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.420445 15595 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:29:02.394017 15901 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.467058  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:02.468034  5627 raft_consensus.cc:2410] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 29 because replica has last-logged OpId of term: 28 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
I20260709 12:29:02.449435 15890 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:02.419863 15617 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.423946 15778 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.502888 15823 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:02.504779 15920 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:02.505421 15920 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.509335  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:02.510025  6280 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
W20260709 12:29:02.510653 15938 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:29:02.510793 15920 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:02.503032 15596 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:02.469233 15914 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.045s
W20260709 12:29:02.512984  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:02.513852  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:02.514755  5560 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:29:02.515162  5555 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:02.465291 15657 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:02.516783 15657 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:02.520339  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 3970) took 2782 ms (client timeout 3081 ms). Trace:
W20260709 12:29:02.520829  5707 rpcz_store.cc:269] 0709 12:28:59.738348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.062831 (+324483us) service_pool.cc:224] Handling call
0709 12:29:02.520083 (+2457252us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54004524d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:02.520253 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4101862272}
I20260709 12:29:02.521839 15880 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.478188 15890 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:02.523451  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:02.525590  5709 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 17. (term mismatch)
I20260709 12:29:02.526049  5709 pending_rounds.cc:85] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 16
I20260709 12:29:02.526388  5709 pending_rounds.cc:107] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 29.17
I20260709 12:29:02.526716  5709 raft_consensus.cc:2889] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:02.526510 15890 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:02.529068  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:02.432916 15922 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.525939  5627 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:29:02.521127 15794 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.528102  5701 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 35 index: 24. Preceding OpId from leader: term: 37 index: 25. (index mismatch)
I20260709 12:29:02.512243 15914 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:02.523787  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:02.523948 15880 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:02.528332  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1745) took 3953 ms. Trace:
I20260709 12:29:02.546657  5709 rpcz_store.cc:276] 0709 12:28:58.575246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.536796 (+1961550us) service_pool.cc:224] Handling call
0709 12:29:02.525198 (+1988402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:02.527254 (+  2056us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:02.528136 (+   882us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b54004524d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:02.528279 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4532607360}
I20260709 12:29:02.528709  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.548010  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.548840  5701 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:02.503971 15927 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.104s
I20260709 12:29:02.529390  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:02.535903  5526 leader_election.cc:400] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:02.551276  5526 leader_election.cc:403] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:29:02.533365  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.528709  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:02.552233  5633 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
W20260709 12:29:02.540619  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 159558us
I20260709 12:29:02.544293  5628 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:29:02.494961 15789 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:02.555285  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:02.556031  5636 raft_consensus.cc:2368] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:02.557945  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.552843  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:02.559023  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.566984  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:02.569655  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:02.570580  5709 raft_consensus.cc:2376] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31. Re-sending same reply.
I20260709 12:29:02.567291  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:02.573637  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.573997  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.574148  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:02.543978 15914 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:02.582335  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:02.578447  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:02.588480  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.585305  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:02.589876  5633 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:02.591878 15933 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:02.591616 15897 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.592415 15933 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:02.592859 15933 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:02.593096  5702 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:02.594172 15897 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:02.594771 15897 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:02.595214 15897 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:02.595798 15931 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.086s	user 0.002s	sys 0.000s
W20260709 12:29:02.600827 15885 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.239s	user 0.005s	sys 0.000s
I20260709 12:29:02.583372 15920 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:02.603325 15432 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:02.603679 15920 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:02.603798 15432 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:02.604218 15920 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:02.606323  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.607434  5703 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 31 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 19.
I20260709 12:29:02.582087 15880 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:02.611485  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.611670  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:02.578130 15922 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:02.612569 15701 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:02.612933 15701 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:02.613269 15701 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:02.595399 15791 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.622162  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:02.623427 15432 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.624763 15872 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:29:02.625100  5628 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:29:02.579036 15780 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:29:02.626085  5554 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: [28.21-28.21]
I20260709 12:29:02.627573 15901 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:02.631973 15707 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:02.636857 15903 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:02.636942 15657 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:02.640856 15899 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.676154  5710 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:29:02.643347  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:02.626030  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:02.645686  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:02.649060  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:02.636171 15886 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.129s	user 0.001s	sys 0.001s
I20260709 12:29:02.681109  5706 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
W20260709 12:29:02.681638  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:02.678484  5629 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:02.665601  5560 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: []
I20260709 12:29:02.610598 15880 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.662921  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:02.672842  5709 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31. Ops: [30.15-30.15]
I20260709 12:29:02.674728  5636 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:29:02.641724 15794 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:02.680006  5628 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:02.691476  5628 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:02.692541  5634 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: []
I20260709 12:29:02.665110  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:02.652637 15902 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:02.697258  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:02.682165  5675 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.249s (SENT)
I20260709 12:29:02.693748  5553 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:02.673676 15432 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:02.661363 15707 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:02.709492  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:02.710515  5636 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:02.695180  5710 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: [29.20-29.20]
I20260709 12:29:02.669921 15903 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:02.692340  5628 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:02.705852 15887 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.037s
I20260709 12:29:02.707880 15511 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:29:02.724416  5628 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:02.725662 15481 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:29:02.726562 15707 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.749929 15778 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:02.752818  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:02.708813 15794 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:02.709678 15725 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:02.801102  5633 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
W20260709 12:29:02.802417  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 120263us
W20260709 12:29:02.802512  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:02.806072  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:02.705753 15799 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.126s
I20260709 12:29:02.809674 15596 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:02.833716 15596 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.703250 15902 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.728421 15595 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.081s
I20260709 12:29:02.611529 15914 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.816509 15934 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:02.736171 15481 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:02.822090 15880 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:02.823693 15899 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:02.759699 15801 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.072s
I20260709 12:29:02.826292 15791 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:02.828878 15707 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:02.832963 15897 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:02.811347  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:02.833202 15889 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:02.870898  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:02.850754  5553 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:02.901327  5628 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:29:02.901768  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:02.906615  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 244122us
W20260709 12:29:02.910252  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 244183us
I20260709 12:29:02.911466  5601 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:02.916260  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:02.917060  5628 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:02.918464  5599 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:02.850476 15823 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:02.925096 15481 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:02.920006 15902 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:02.931965 15481 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:02.933240 15823 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.933885 15902 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:02.934556 15902 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:02.935608 15823 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:02.862274 15794 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:02.937047 15794 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:02.941319  5675 leader_election.cc:336] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.510s (SENT)
I20260709 12:29:02.942055  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:02.960618  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:02.925622 15735 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.569s	user 0.008s	sys 0.004s
I20260709 12:29:02.915186 15596 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:02.968405  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 56777.1us
W20260709 12:29:02.973266  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:02.977320  5675 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.544s (SENT)
I20260709 12:29:02.977823  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:02.935582 15902 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.927412 15791 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:02.978864  5634 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: []
W20260709 12:29:02.979435  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:02.995926 15725 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:02.996526 15682 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:02.996569 15725 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.997187 15682 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.970680 15646 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:02.979069 15891 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:02.979905  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:03.001518  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:02.984452  5628 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:29:02.979146  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:03.011618 15780 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:03.014160  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:02.978297 15914 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:29:03.025957  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 107571us
I20260709 12:29:03.026185  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.027307  5627 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 22.
W20260709 12:29:03.028157  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60122.3us
I20260709 12:29:03.016464  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.015321 15746 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:03.006331 15791 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.022074 15890 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:03.029300  5675 leader_election.cc:336] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.620s (SENT)
I20260709 12:29:03.043761  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:03.011696 15920 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:02.998873 15725 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:03.034179  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.052471  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.053175  5706 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:03.057788  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.058981  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.062284  5702 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 26 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 12.
I20260709 12:29:03.062479  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:02.978411 15735 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.062677  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:02.940618 15903 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.15, Last appended by leader: 15, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:03.069685  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:03.074527  6236 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:03.001435 15896 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.058s
I20260709 12:29:03.082563  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:03.057960 15714 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.818s	user 0.012s	sys 0.002s
I20260709 12:29:03.086777  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:03.087512  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58243.4us
I20260709 12:29:03.088188  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.088253 15701 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:03.089628  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.090229  5629 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:03.090750 15701 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:03.091435 15714 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.092469 15746 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:29:03.086915  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.093914  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.128906  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.130977  5706 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:03.095062  5710 raft_consensus.cc:2368] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29.
I20260709 12:29:03.085040  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:03.129858  6271 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:03.130756  5702 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:03.133287  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:03.132686  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47730.7us
I20260709 12:29:03.094316  6277 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 29: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:03.064663 15891 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:03.078696 15901 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:03.097129 15933 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.504s	user 0.009s	sys 0.004s
I20260709 12:29:03.058269 15682 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:03.087198 15897 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:03.095358 15890 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:03.149248  5675 leader_election.cc:336] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.803s (SENT)
I20260709 12:29:03.151126  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.093335 15903 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:03.093590 15714 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:03.144436  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 165488us
I20260709 12:29:03.171643  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:03.173314  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:03.178103  5552 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:03.172763  5554 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:29:03.183435  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.184587  5633 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:03.131319 15887 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:03.189419 15930 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:29:03.189623  6280 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:03.190027 15930 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:03.190479 15930 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:29:03.183446 15928 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:03.191350 15928 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.193702 15928 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:03.169966 15945 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:03.194741 15735 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:03.194561 15945 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:03.197197 15964 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 12:29:03.197343 15944 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:29:03.199622 15920 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:03.143163 15952 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:03.201159  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51958.3us
W20260709 12:29:03.202232  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 141336us
I20260709 12:29:03.200937 15952 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:03.197198 15945 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:03.202342  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:03.185045  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:03.205794  5703 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:03.200838  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.209697  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.169735 15957 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:03.212791  5634 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 37: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:03.215974 15534 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:03.217168  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:03.170266 15682 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:03.219457  5675 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.898s (SENT)
I20260709 12:29:03.224025  5552 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:29:03.227638 15682 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.230163 15682 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:03.225180  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:03.233297  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:03.235209  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.173386 15958 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:03.155192 15901 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:03.211580 15896 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:03.164878 15933 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.239509  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.241043  5706 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:29:03.210934 15953 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.078s	user 0.001s	sys 0.001s
I20260709 12:29:03.254936  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:03.255575  6273 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:03.175019 15897 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:03.286324 15933 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:03.287513 15897 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:03.289467  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:03.290512  5705 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
W20260709 12:29:03.292657  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:03.293929 15901 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:03.295994  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:03.296276 15511 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.552s	user 0.008s	sys 0.004s
I20260709 12:29:03.256002 15896 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.296846  5635 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 37: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:03.253422  5709 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:29:03.297708  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.250783  5703 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:03.257422  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:03.304903  6279 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:03.258669  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.242028 15958 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:03.305805  5628 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:03.266218  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.255120  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.306594  5629 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:29:03.268447  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.274466 15952 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:03.255841 15891 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:03.318293 15789 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.763s	user 0.011s	sys 0.000s
I20260709 12:29:03.318589  5701 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:03.323143  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:03.279114  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.281126 15792 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.282547  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:03.285008 15791 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:03.298625  5707 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:29:03.306167 15958 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:03.340867  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.341625  5703 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:29:03.308145  6275 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:03.308647  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.341800  5632 raft_consensus.cc:2376] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26. Re-sending same reply.
W20260709 12:29:03.310484 15942 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.805s	user 0.003s	sys 0.000s
I20260709 12:29:03.347328  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:03.311687  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.315241  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.327129  5702 raft_consensus.cc:2410] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 38 because replica has last-logged OpId of term: 35 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
I20260709 12:29:03.327632  5628 raft_consensus.cc:2410] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:29:03.351454  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.353677  5705 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:29:03.355073  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.329280  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.329280  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.337385  6237 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:03.344544  6273 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:03.350075  5633 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:03.356161  5632 raft_consensus.cc:2410] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:29:03.360752  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.366278  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.340826 15976 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:03.357206  5634 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
W20260709 12:29:03.379518  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:03.339069 15707 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:03.340224 15929 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:03.391417  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54169.2us
I20260709 12:29:03.392021  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.394279  5633 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:03.396288  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.397215  5629 raft_consensus.cc:2376] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30. Re-sending same reply.
W20260709 12:29:03.399986  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:03.405040 15896 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:03.411614  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.350077 15792 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:29:03.417007  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 134524us
W20260709 12:29:03.421530  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 182638us
I20260709 12:29:03.421614  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.422686  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.423017  5552 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 23.
I20260709 12:29:03.423303  5632 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:03.355911 15969 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:03.424211  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.424571  5599 leader_election.cc:400] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29.
I20260709 12:29:03.429659  5599 leader_election.cc:403] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:29:03.426124  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.428615  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:03.425597  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:03.431636  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:03.433122  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:03.427078  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:03.427328  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.436431  5631 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:03.428033  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.428586  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.426512  5560 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:29:03.441018  5552 raft_consensus.cc:2410] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 34 because replica has last-logged OpId of term: 32 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 22.
I20260709 12:29:03.415937 15823 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:29:03.443426  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:03.447839  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:03.421748 15976 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:03.406754 15707 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:03.457854 15707 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:03.446177 15478 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:03.462495 15880 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:03.462965 15478 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:03.465126 15690 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:03.465502 15478 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:03.465595 15690 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:03.466665  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.466928 15902 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:03.467144 15891 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:03.467418  5705 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:29:03.467499 15902 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.467741 15891 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:03.466701 15690 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.469519 15690 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:03.469797 15902 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:03.470839 15891 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:03.472833  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.447570 15823 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:03.474009 15823 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:03.479133  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 3958) took 5273 ms. Trace:
I20260709 12:29:03.479580  5704 rpcz_store.cc:276] 0709 12:28:58.205316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:00.110435 (+1905119us) service_pool.cc:224] Handling call
0709 12:29:00.110807 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:01.387558 (+1276751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:03.478847 (+2091289us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54002f5b54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:03.479070 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2881141120}
I20260709 12:29:03.480284 15932 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:03.480723 15932 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:03.481412  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:03.481595  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1574) took 1423 ms. Trace:
I20260709 12:29:03.481542 15932 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.482097  6277 rpcz_store.cc:276] 0709 12:29:02.057907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:02.059843 (+  1936us) service_pool.cc:224] Handling call
0709 12:29:03.481372 (+1421529us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54001e87d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:03.481515 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2319494528}
I20260709 12:29:03.483023 15932 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:03.484947  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.485071 15936 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:29:03.485926 15934 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:03.486073  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3868) took 1191 ms. Trace:
I20260709 12:29:03.486438 15934 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:03.485945  6278 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:03.488373  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1641) took 1076 ms. Trace:
I20260709 12:29:03.489148  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:03.486459  5556 rpcz_store.cc:276] 0709 12:29:02.294795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:02.370584 (+ 75789us) service_pool.cc:224] Handling call
0709 12:29:02.371000 (+   416us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:02.396644 (+ 25644us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:02.396684 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:02.407163 (+ 10479us) log.cc:844] Serialized 52 byte log entry
0709 12:29:02.419970 (+ 12807us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:29:02.420598 (+   628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:02.420782 (+   184us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:03.310193 (+889411us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x7b240013f5c8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff740b9c4 00007f4ff7409acb 00007f4ff75c4dc9 00007f4ff75c4a55 00007f4ff75c409d 00007f4ffa1f610e 00007f4ffa20d88c 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221
0709 12:29:03.485775 (+175582us) raft_consensus.cc:1685] finished
0709 12:29:03.485803 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:03.486019 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4386,"raft.queue_time_us":19951,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":13929,"spinlock_wait_cycles":64889344,"thread_start_us":9175,"threads_started":1,"wal-append.queue_time_us":94895}
I20260709 12:29:03.494395  5634 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:03.496848  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.497808  5629 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:03.488725  6278 rpcz_store.cc:276] 0709 12:29:02.412157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:02.412330 (+   173us) service_pool.cc:224] Handling call
0709 12:29:03.485744 (+1073414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:03.487095 (+  1351us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:03.488160 (+  1065us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54001e87d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:03.488292 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2467773184}
I20260709 12:29:03.505015  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:03.510529 15925 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:29:03.426052 15896 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.029s
I20260709 12:29:03.437772 15933 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:03.525784  5631 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:03.533448  5552 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
W20260709 12:29:03.533931  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:03.534502  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:03.527558  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.407312 15929 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.535980  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:03.533505  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:03.537703  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:03.537822  6278 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:03.451120 15956 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:03.450848 15920 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:03.511986 15925 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:29:03.452955 15974 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:03.567091  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:03.600836  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.512243 15880 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29.
I20260709 12:29:03.516541 15914 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.112s
I20260709 12:29:03.640565 15914 consensus_queue.cc:1243] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 30.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:29:03.640704  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.641122 15914 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:03.641474 15914 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:03.642081  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.642314 15914 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.570459  5633 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:29:03.643316  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.643519 15914 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:03.606626  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.643905  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.432657 15969 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:03.644768  5633 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:03.645901  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.646315 15969 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:03.517889 15937 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:03.646677  5633 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:29:03.647024 15929 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:03.647192 15937 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:03.647761 15872 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:29:03.500730 15580 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:03.648687  5632 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:03.649315 15682 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:03.625589  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:03.650025  5553 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:03.649849 15682 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.652016 15682 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:03.634027  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.530923 15617 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.106s
I20260709 12:29:03.639222  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.643945  5555 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:03.659435  5552 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 28: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:03.642858  5554 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 26: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:03.648600 15580 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:03.654085  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 111966us
I20260709 12:29:03.647877  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.621960  5629 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:03.624185  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:03.664124  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:03.540513 15799 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:29:03.666322  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.666853  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.667019  5633 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:03.667464  5627 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:03.667927  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.648186 15937 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.614845 15956 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:03.666157  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.668172  5553 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:03.668318  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.693044 15876 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:03.623063 15892 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:03.676625 15580 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:03.695003 15892 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:03.695557 15892 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:03.695896  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:03.698328 15956 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:03.618693 15974 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:03.696542 15969 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:03.668737  5633 raft_consensus.cc:2410] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 25 because replica has last-logged OpId of term: 24 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:29:03.697127  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.654561 15930 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.700510  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.671555  5635 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:03.701655 15930 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:03.660961 15890 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:03.685652  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 61574.5us
I20260709 12:29:03.702790 15890 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:03.703363  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.699070 15967 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:03.619803 15947 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:03.704663 15967 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:03.705065 15967 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:03.704849 15947 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:03.699311 15969 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:03.706770  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 166120us
I20260709 12:29:03.708307  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:03.709975 15929 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:03.539381 15933 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.711177  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.659672 15880 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:03.713318  5633 raft_consensus.cc:2410] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 29 because replica has last-logged OpId of term: 28 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
I20260709 12:29:03.615247 15735 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:03.680640  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.674882  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.719156  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.719931  5631 raft_consensus.cc:2376] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26. Re-sending same reply.
I20260709 12:29:03.703719  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:03.720117  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:03.673974 15799 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:03.699522 15976 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.726294  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.726998  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:03.727960  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:03.729696  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:03.719195  5635 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:29:03.752771  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:03.754102  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:03.756646 15952 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:29:03.757584 15952 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:03.758267 15952 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:03.758730 15952 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:03.760855  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:03.765089  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:03.766835  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.767123 15478 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:03.767550  5706 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
W20260709 12:29:03.770722 15903 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.584s	user 0.009s	sys 0.000s
I20260709 12:29:03.699059 15974 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:03.748135  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.782090  5636 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:03.756057  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:03.776705  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:03.783478  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:03.784140  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56258.3us
I20260709 12:29:03.784231  5705 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:29:03.747452 15707 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:29:03.665318 15920 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:03.787140  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:03.751223 15799 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:29:03.750051 15880 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.047s
I20260709 12:29:03.800386  6280 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:29:03.801713 15947 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:03.802105 15902 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:03.802753 15933 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:03.804850 15890 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:03.807278 15823 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.808545  6279 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
I20260709 12:29:03.808423 15891 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:29:03.809231  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.810026 15538 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:03.810319  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.810750 15538 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:03.811633 15538 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 30.13, Last appended by leader: 13, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:03.789551  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:03.779099  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:03.783720 15478 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:29:03.786338 15707 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:03.813074  5705 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:29:03.788810  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.810072  5633 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:03.814697  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:03.818316  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1757) took 1293 ms (client timeout 1682 ms). Trace:
W20260709 12:29:03.835517  5707 rpcz_store.cc:269] 0709 12:29:02.524522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:02.533130 (+  8608us) service_pool.cc:224] Handling call
0709 12:29:03.818035 (+1284905us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b540006f754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:03.818183 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1716998912}
I20260709 12:29:03.819316  5706 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:29:03.842255  5708 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:29:03.850994  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.853859  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.854316  5636 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:03.855880  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.807772 15799 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.867189  5634 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:29:03.869485 15799 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:03.868057  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.852105  5629 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
W20260709 12:29:03.878881  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 64249.1us
I20260709 12:29:03.881245  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:03.815079 15792 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:03.889451  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:03.813941 15920 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:03.818567 15876 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:03.844030 15478 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:03.827729 15929 raft_consensus.cc:2764] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:29:03.892186  5705 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
W20260709 12:29:03.895603  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 361349us
I20260709 12:29:03.843946 15935 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:03.843770 15707 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:03.898489  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.902660  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.889415 15932 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:29:03.827489 15902 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.072s
I20260709 12:29:03.908135  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.908135  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:03.909170  5552 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:29:03.909340  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:03.904644  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.904634  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.910907  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:03.911330  5558 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 29: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:03.912806  5630 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:03.914543  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.915684  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.917688  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.920070 15901 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:03.903430 15478 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:03.902721 15947 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.020s
I20260709 12:29:03.903616 15876 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:03.886305 15715 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.020s
I20260709 12:29:03.923987 15715 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:03.929091 15715 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:03.927623  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:03.938246  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 404788us
I20260709 12:29:03.906513 15935 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.925101 15967 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.938299  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:03.909821  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:03.952963  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:03.954429  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.955251  5558 raft_consensus.cc:2410] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 26 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 12.
I20260709 12:29:03.927119  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:03.956133  5627 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:03.924250 15956 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:03.879052 15934 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:03.916092 15969 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:03.940310  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:03.985765  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.940500  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:03.986583  5552 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
W20260709 12:29:03.947235  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:03.956408  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.908041 15932 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:03.963470  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 57500.4us
I20260709 12:29:03.990692  5526 leader_election.cc:400] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:03.991616  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:03.968217  5635 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:03.992563  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:03.983506  5708 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1760) took 1458 ms (client timeout 1671 ms). Trace:
I20260709 12:29:03.988806  6237 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:03.984452  5706 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:29:03.994654  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.995035  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.995625  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:03.996421  5556 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 30 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:29:03.998049  5635 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:29:04.000877  5557 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:04.001405  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:04.002022  5629 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:29:04.002874  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:04.004390  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:04.005221  5708 rpcz_store.cc:269] 0709 12:29:02.525516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:02.528788 (+  3272us) service_pool.cc:224] Handling call
0709 12:29:03.982651 (+1453863us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54001a8954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:03.983387 (+   736us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2844991616}
I20260709 12:29:04.003501  5709 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
I20260709 12:29:04.008028  5703 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:29:04.008641  5628 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:04.007776  5630 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 37 index: 25. (index mismatch)
I20260709 12:29:04.009255  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:04.009601  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2052) took 1009 ms. Trace:
I20260709 12:29:04.010502  5635 raft_consensus.cc:1217] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Deduplicated request from leader. Original: 20.14->[21.15-23.16]   Dedup: 21.15->[23.16-23.16]
I20260709 12:29:04.011034  5635 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 23. Current term is 24. Ops: [21.15-23.16]
I20260709 12:29:04.011284  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:04.010078  5630 rpcz_store.cc:276] 0709 12:29:03.000699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:03.974665 (+973966us) service_pool.cc:224] Handling call
0709 12:29:04.007457 (+ 32792us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:04.009290 (+  1833us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:04.009535 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:03.964955 15956 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:03.906534 15920 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.090s
I20260709 12:29:03.994400 15932 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:04.025488  5557 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:04.026644  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:04.027280  5557 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:04.028980 15823 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:03.986886 15933 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:29:04.032814 15933 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:04.034060 15933 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:04.041639 15934 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:04.043493 15967 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:04.003234 15977 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:29:03.993255 15889 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:03.986264 15730 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.023852  5633 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:29:03.996527 15891 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:04.044924  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:04.024462  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:04.045953  5636 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 32. Ops: []
I20260709 12:29:04.047154  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:04.053274  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 51461.3us
I20260709 12:29:04.063994  5635 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
W20260709 12:29:04.053957  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 115771us
I20260709 12:29:04.012612  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:04.016534 15791 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:29:04.064095  5557 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:04.067909  5556 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 26: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:29:04.068272  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55808.7us
I20260709 12:29:04.069222  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:04.074617  6275 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:04.021970 15876 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:04.056604  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 2057) took 1052 ms. Trace:
W20260709 12:29:04.070363  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 46103us
I20260709 12:29:04.070459  5634 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24.
I20260709 12:29:04.073391  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:04.022038 15956 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:04.080037  5630 raft_consensus.cc:1217] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Deduplicated request from leader. Original: 28.16->[29.17-29.17]   Dedup: 29.17->[]
I20260709 12:29:04.080861  5630 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: [29.17-29.17]
I20260709 12:29:04.081835  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:04.076628  5552 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:04.082680  5630 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:29:04.082808  5708 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:04.069469  5635 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:04.078387  5628 rpcz_store.cc:276] 0709 12:29:03.004464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:03.799985 (+795521us) service_pool.cc:224] Handling call
0709 12:29:04.056261 (+256276us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b54001cedd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:04.056523 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":454059776}
I20260709 12:29:04.086851  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:04.061594 15965 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:04.052979 15889 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:04.062340 15921 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:04.086036  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:04.094156  5709 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:29:04.062947 15891 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:04.063972 15977 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:29:04.078774 15958 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.726s	user 0.008s	sys 0.005s
I20260709 12:29:04.097411  5634 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:29:04.098752  5559 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
I20260709 12:29:04.084770 15799 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:04.105078  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:04.091502 15902 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:04.105652  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:04.106755  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:04.108263  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:04.117280  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:04.085891 15925 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:29:04.089429 15876 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.113006  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:04.066087 15730 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:04.116483  5556 raft_consensus.cc:2368] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29.
I20260709 12:29:04.099247 15889 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:04.121308  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:04.122170  5560 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.029s	sys 0.045s
I20260709 12:29:04.066671 15715 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:04.128813 15876 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:04.088460 15823 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.026s
I20260709 12:29:04.095659 15965 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:04.134838  5560 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.029s	sys 0.045s
I20260709 12:29:04.112288 15896 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:29:04.139426  5627 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:04.144793  5557 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:29:04.115367 15799 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:04.134305 15924 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.017s
I20260709 12:29:04.120612 15872 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:04.179008  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:04.149255  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:04.183537  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:04.093849 15735 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.027s
I20260709 12:29:04.154759 15876 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:04.096087 15934 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.038s
I20260709 12:29:04.136677  5552 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: [32.21-32.21]
I20260709 12:29:04.201659  6278 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
W20260709 12:29:04.205539  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:04.163861  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:04.118785 15902 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:04.165324  6237 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:04.210004  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44812.7us
I20260709 12:29:04.120499 15903 raft_consensus.cc:2764] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:29:04.115340 15714 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.044s
I20260709 12:29:04.219072 15902 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:04.120939 15972 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.077s
I20260709 12:29:04.157048 15965 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:04.160739 15936 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:29:04.241916 15730 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:04.197880 15872 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:04.252815 15872 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:29:04.254263 15892 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.558s	user 0.009s	sys 0.004s
I20260709 12:29:04.254734 15892 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:04.255898  5558 raft_consensus.cc:2368] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29.
I20260709 12:29:04.257155 15925 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.196482 15988 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.091s
I20260709 12:29:04.257659 15925 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:04.168015 15929 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:04.262815 15929 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:04.255959  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:04.266855  5559 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:29:04.227599 15928 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:29:04.232252  5552 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:04.269138 15928 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:04.270085 15928 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:04.233093 15989 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:04.271085 15989 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:04.271554 15989 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:04.225642 15903 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.005s
I20260709 12:29:04.293393  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:04.184906 15896 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:04.270464  5552 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:29:04.186091 15799 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:04.116340 15958 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:04.280743  5558 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
W20260709 12:29:04.280850  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:04.288338  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:04.328344  6275 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:04.268462  5559 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:04.235842  5557 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
W20260709 12:29:04.335632  6274 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.032s	sys 0.067s
W20260709 12:29:04.336169  6274 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.032s	sys 0.067s
I20260709 12:29:04.344238  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:04.338352 15580 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:04.198681 15891 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:04.396909  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:04.401111  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:04.414719  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:04.416612  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:04.294694 15977 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:04.397203 15892 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:04.427304  6277 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:04.429266  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:04.433161  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:04.433897  5556 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:04.422303 15914 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:29:04.407987 15891 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:04.411343 15876 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.409070 15929 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:04.411759 15925 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:04.446008  5676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.013s	sys 0.029s
W20260709 12:29:04.454015  5676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.013s	sys 0.029s
W20260709 12:29:04.454325  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 559586us
I20260709 12:29:04.426403 15967 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:29:04.458004  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:04.435114 15791 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:04.441874 15690 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:29:04.461758  5554 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.013s	sys 0.006s
I20260709 12:29:04.435952 15935 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:04.465130  5554 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.014s	sys 0.006s
W20260709 12:29:04.471776  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 280058us
W20260709 12:29:04.473461  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 294528us
I20260709 12:29:04.474519  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:04.465354 15952 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.706s	user 0.013s	sys 0.004s
W20260709 12:29:04.465548 15974 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.669s	user 0.011s	sys 0.001s
W20260709 12:29:04.464233 15794 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.028s	sys 0.028s
I20260709 12:29:04.476784  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:04.477465  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:04.452334 15876 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:04.476089  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:04.487460  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:04.487346 15478 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.566s	user 0.007s	sys 0.008s
W20260709 12:29:04.469568 15657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.008s	sys 0.012s
I20260709 12:29:04.430125 15896 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:04.538743  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:04.480664  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2047) took 1002 ms. Trace:
I20260709 12:29:04.492987 15930 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:04.510404  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.018s	sys 0.015s
I20260709 12:29:04.516579  6274 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:29:04.486934 15780 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:04.486065 15935 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:04.523409  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:04.524291  5559 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: []
I20260709 12:29:04.525125  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:04.516811 15789 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.006s	sys 0.022s
I20260709 12:29:04.506048  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:04.490108 15952 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:04.493497 15974 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:04.671705  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:04.584316  5633 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:29:04.690918  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.779s	user 0.030s	sys 0.021s
I20260709 12:29:04.544273  5560 rpcz_store.cc:276] 0709 12:29:03.478491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:03.507553 (+ 29062us) service_pool.cc:224] Handling call
0709 12:29:03.507846 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:03.508032 (+   186us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:29:04.146476 (+638444us) log.cc:844] Serialized 14 byte log entry
0709 12:29:04.476820 (+330344us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:04.476872 (+    52us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:04.476918 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:04.480423 (+  3505us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:04.480591 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":831314,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":3527,"raft.queue_time_us":554,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":3186,"thread_start_us":909693,"threads_started":2}
I20260709 12:29:04.512856 15538 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:04.675498  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:04.600394  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:04.691672  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 780803us
W20260709 12:29:04.696170  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 190244us
I20260709 12:29:04.702054  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:04.703194  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 102898us
W20260709 12:29:04.685055 15995 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.243s	user 0.001s	sys 0.000s
I20260709 12:29:04.708930  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1799) took 1173 ms. Trace:
I20260709 12:29:04.709498  6277 rpcz_store.cc:276] 0709 12:29:03.535962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:03.536105 (+   143us) service_pool.cc:224] Handling call
0709 12:29:03.607879 (+ 71774us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:04.435308 (+827429us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:04.708492 (+273184us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b54005ee554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:04.708852 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1884047104}
I20260709 12:29:04.705284  5526 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:04.700956  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:04.524825 15880 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:04.519519 15903 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:29:04.676225 16000 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.200s	user 0.000s	sys 0.000s
W20260709 12:29:04.544586 15657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.008s	sys 0.012s
I20260709 12:29:04.717061  5554 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:29:04.702855 15976 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:04.692169 15935 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:04.545061 15929 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:04.690342 15789 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.012s	sys 0.027s
I20260709 12:29:04.612699 15924 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:04.542974 15958 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:04.740525  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:04.747987 15996 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.313s	user 0.000s	sys 0.001s
I20260709 12:29:04.735877 15937 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:04.722354  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:04.522110 15920 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.221s
I20260709 12:29:04.523264 15478 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:04.716054 15988 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:04.771680  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:04.743278 15934 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:04.747987  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:04.787397  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 358212us
I20260709 12:29:04.763157 15947 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:04.613389 15965 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:04.716276 15936 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 18, Time since last communication: 0.074s
I20260709 12:29:04.736660 15538 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:05.064855 15990 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.916s	user 0.007s	sys 0.000s
W20260709 12:29:05.065092 15889 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.904s	user 0.009s	sys 0.013s
I20260709 12:29:05.064546 15538 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.065491 15889 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.066815  5552 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:04.779752 15935 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:05.067494 15989 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.795s	user 0.001s	sys 0.013s
I20260709 12:29:05.068770 15478 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:05.069473 15889 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:05.069613 15714 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:29:05.071069 15799 consensus_meta.cc:287] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.732s	user 0.010s	sys 0.003s
I20260709 12:29:05.071509 15799 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:04.768590 15880 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:05.072180  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1769) took 1416 ms. Trace:
W20260709 12:29:05.072799 15902 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.779s	user 0.015s	sys 0.001s
I20260709 12:29:05.073141 15880 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:05.078387 15988 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:05.082399  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 334563us
I20260709 12:29:04.774093 15903 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:05.083206  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 283731us
W20260709 12:29:04.798396 15707 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.892s	user 0.016s	sys 0.000s
I20260709 12:29:04.780699 15929 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:04.906046 16006 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
W20260709 12:29:04.920781 15999 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.255s	user 0.001s	sys 0.001s
W20260709 12:29:04.906960 16004 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
W20260709 12:29:04.921864 16011 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:29:04.882709 15682 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.021s
W20260709 12:29:04.830231  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:05.120050  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:04.771018 15901 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.082s
I20260709 12:29:04.698197 15930 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:04.871381  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:05.124115  5633 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
W20260709 12:29:04.810149 16007 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.160s	user 0.001s	sys 0.001s
I20260709 12:29:04.792697 15924 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:04.849480  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:05.044912  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:05.048120 15876 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
W20260709 12:29:05.059805  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:05.055488  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:05.073345 15902 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:05.082476  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 342305us
I20260709 12:29:05.142890  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:05.143762  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 88502.7us
W20260709 12:29:05.048398 15891 consensus_meta.cc:287] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.596s	user 0.012s	sys 0.003s
I20260709 12:29:05.072742  5710 rpcz_store.cc:276] 0709 12:29:03.656156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:03.656384 (+   228us) service_pool.cc:224] Handling call
0709 12:29:03.687986 (+ 31602us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:03.688265 (+   279us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:03.688301 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:03.721488 (+ 33187us) log.cc:844] Serialized 26 byte log entry
0709 12:29:04.157114 (+435626us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:04.218852 (+ 61738us) log.cc:844] Serialized 14 byte log entry
0709 12:29:04.219087 (+   235us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:04.219166 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:04.533068 (+313902us) spinlock_profiling.cc:243] Waited 40.5 ms on lock 0x7b1800147940. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:29:05.071805 (+538737us) raft_consensus.cc:1685] finished
0709 12:29:05.071938 (+   133us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:05.072108 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":68686,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":15071,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":107,"spinlock_wait_cycles":93105408,"thread_start_us":388389,"threads_started":1,"wal-append.queue_time_us":623838}
I20260709 12:29:05.144593  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:04.936225  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:05.093819  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2058) took 1871 ms. Trace:
I20260709 12:29:05.143215  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:05.055572 15932 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.030s	user 0.014s	sys 0.002s
I20260709 12:29:05.097456 15730 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:29:05.098968 16009 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:05.099299 16010 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.327s	user 0.000s	sys 0.001s
I20260709 12:29:05.067951 15989 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:05.082595 16002 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.272s	user 0.000s	sys 0.005s
W20260709 12:29:05.104507  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 329899us
I20260709 12:29:05.102144 15707 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:05.095889  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:05.109287  6234 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:05.084144 15715 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:29:05.099291 15690 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.628s	user 0.013s	sys 0.000s
I20260709 12:29:05.090685 15657 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:05.131179  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:05.144152  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:05.048946  5634 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: []
I20260709 12:29:05.146791  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:05.145799  5629 rpcz_store.cc:276] 0709 12:29:03.222778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:04.023187 (+800409us) service_pool.cc:224] Handling call
0709 12:29:04.060590 (+ 37403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:04.072641 (+ 12051us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:04.072689 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:04.073300 (+   611us) log.cc:844] Serialized 26 byte log entry
0709 12:29:04.439154 (+365854us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:04.439202 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:04.440021 (+   819us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:05.093563 (+653542us) raft_consensus.cc:1685] finished
0709 12:29:05.093589 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:05.093744 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38519,"raft.queue_time_us":12609,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":9564,"spinlock_wait_cycles":2304,"thread_start_us":320364,"threads_started":1,"wal-append.queue_time_us":675870}
I20260709 12:29:05.149613  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:05.149792  6275 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:05.151036  5630 raft_consensus.cc:2410] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 28 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260709 12:29:05.143185 15478 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:05.152419  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:05.166173  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 56989.4us
I20260709 12:29:05.079664 15988 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:05.147356 15998 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:05.207840  5633 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:29:05.136452 15929 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:05.150406 15876 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:29:05.217450  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:05.223810  5701 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:05.230962  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:05.231518  5629 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:05.224792  5636 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:05.225528  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:05.143203 15914 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:05.228003  5631 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:05.229679  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:05.149309 15730 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:05.224236 16004 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:29:05.151669 15932 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.235355 15902 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:05.235682 16020 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
W20260709 12:29:05.236554 16018 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 12:29:05.236286 15935 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.236469 16027 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:05.237377 16023 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
W20260709 12:29:05.238471 16024 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:29:05.239311 15965 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:05.233891 15998 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.240825 15930 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:05.232398 15478 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.290681 15998 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:05.291275 15998 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:05.292407 15478 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:05.262636 15876 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:05.292850 15799 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:05.294710 15932 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:05.295336 15932 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:05.263971 15657 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:05.297008 15657 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.299700 15818 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:05.300223 15818 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:05.300688 15657 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:05.301103 15818 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:05.263921 15914 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.302346 15818 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:05.302521 15935 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:05.303851 15989 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:05.301115 16027 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:05.305058 15947 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:29:05.305919 15876 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:29:05.306731 16027 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:05.291652 15998 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:05.304189 15914 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:05.240679  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4008) took 1451 ms. Trace:
I20260709 12:29:05.310529  5707 rpcz_store.cc:276] 0709 12:29:03.789994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:03.836630 (+ 46636us) service_pool.cc:224] Handling call
0709 12:29:03.837025 (+   395us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:03.884456 (+ 47431us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:03.884496 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:03.893621 (+  9125us) log.cc:844] Serialized 52 byte log entry
0709 12:29:04.371861 (+478240us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:04.371911 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:04.372006 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:05.240432 (+868426us) raft_consensus.cc:1685] finished
0709 12:29:05.240453 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:05.240607 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46929,"raft.queue_time_us":50141,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":3138,"thread_start_us":457425,"threads_started":1,"wal-append.queue_time_us":637042}
W20260709 12:29:05.241181 15956 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.145s	user 0.010s	sys 0.010s
I20260709 12:29:05.268256  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:05.242408  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4009) took 1425 ms. Trace:
I20260709 12:29:05.242926  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1805) took 1430 ms. Trace:
I20260709 12:29:05.243187 15924 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:05.243413  5555 raft_consensus.cc:1217] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Deduplicated request from leader. Original: 28.20->[29.21-29.21]   Dedup: 29.21->[]
I20260709 12:29:05.321074  5555 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: [29.21-29.21]
I20260709 12:29:05.234336  5708 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:05.171041 15901 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:29:05.322086 15901 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:05.172346 15690 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:05.192304 15977 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:05.168915 16033 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:05.169353 15952 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:05.332378 15947 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:29:05.333443 15947 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:05.333863 15947 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:05.334677 15947 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.335808 15947 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:05.337507  5627 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:05.337514  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:05.280334 15934 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:05.151623 16021 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:05.172729 15974 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:05.163997 16009 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.325229 15924 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:05.341681 15780 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:05.342401 15780 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:05.342811 16009 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:05.245657 15891 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:05.343533 15891 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.240825 15707 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:05.241791  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2050) took 1346 ms. Trace:
I20260709 12:29:05.316313  5702 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:29:05.264775 15872 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.011s	user 0.020s	sys 0.001s
I20260709 12:29:05.314605  5629 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:05.344642  5553 rpcz_store.cc:276] 0709 12:29:03.895832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:03.900456 (+  4624us) service_pool.cc:224] Handling call
0709 12:29:03.900794 (+   338us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:03.901148 (+   354us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:03.901176 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:03.906133 (+  4957us) log.cc:844] Serialized 52 byte log entry
0709 12:29:04.013566 (+107433us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b1800418540. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:29:04.433574 (+420008us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:04.433631 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:04.433758 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:05.241456 (+807698us) raft_consensus.cc:1685] finished
0709 12:29:05.241478 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:05.241730 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1992,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":3474,"spinlock_wait_cycles":38813056,"thread_start_us":418405,"threads_started":1,"wal-append.queue_time_us":735848}
I20260709 12:29:05.345773  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:05.315749  5705 rpcz_store.cc:276] 0709 12:29:03.817382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:03.920757 (+103375us) service_pool.cc:224] Handling call
0709 12:29:03.921088 (+   331us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:04.006860 (+ 85772us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:04.006905 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:04.017964 (+ 11059us) log.cc:844] Serialized 26 byte log entry
0709 12:29:04.442898 (+424934us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:29:04.442944 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:04.443026 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:05.242186 (+799160us) raft_consensus.cc:1685] finished
0709 12:29:05.242206 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:05.242352 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9577,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":12680,"thread_start_us":361287,"threads_started":1,"wal-append.queue_time_us":686269}
I20260709 12:29:05.346513  5629 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:05.295828 15932 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:05.169183 15903 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.325728  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:05.326327 16033 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:05.329815 15993 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.944s	user 0.000s	sys 0.003s
I20260709 12:29:05.265964 16026 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:05.262418  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:05.338781  5557 raft_consensus.cc:2410] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 32 because replica has last-logged OpId of term: 30 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 15.
I20260709 12:29:05.341578 15924 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.315990  5636 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:05.367524  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:05.350885  5554 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:05.350975  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:05.365214  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:05.316191  6279 rpcz_store.cc:276] 0709 12:29:03.812225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:03.844676 (+ 32451us) service_pool.cc:224] Handling call
0709 12:29:03.889870 (+ 45194us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:03.890308 (+   438us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:03.890344 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:03.899299 (+  8955us) log.cc:844] Serialized 26 byte log entry
0709 12:29:04.434006 (+534707us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:04.434059 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:04.434142 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:05.242724 (+808582us) raft_consensus.cc:1685] finished
0709 12:29:05.242746 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:05.242873 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8358,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":4470,"thread_start_us":482408,"threads_started":1,"wal-append.queue_time_us":781146}
W20260709 12:29:05.370550  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 221402us
I20260709 12:29:05.320444 15956 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:05.373217  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 111036us
I20260709 12:29:05.379916  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:05.382400  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:05.382727  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:05.373780  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:05.374377  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 231571us
I20260709 12:29:05.363847 15920 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:05.403169  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2069) took 1243 ms. Trace:
I20260709 12:29:05.400101  5602 leader_election.cc:400] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29.
I20260709 12:29:05.402048  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:05.403983  5628 rpcz_store.cc:276] 0709 12:29:04.159627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:04.160060 (+   433us) service_pool.cc:224] Handling call
0709 12:29:04.201038 (+ 40978us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:04.230159 (+ 29121us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:04.230196 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:04.230902 (+   706us) log.cc:844] Serialized 52 byte log entry
0709 12:29:04.661104 (+430202us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:29:04.661371 (+   267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:04.661573 (+   202us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:05.402813 (+741240us) raft_consensus.cc:1685] finished
0709 12:29:05.402851 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:05.403076 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28675,"raft.queue_time_us":30341,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":91,"thread_start_us":405244,"threads_started":1,"wal-append.queue_time_us":818639}
I20260709 12:29:05.407156 15707 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:05.408480  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1770) took 1751 ms. Trace:
I20260709 12:29:05.372195 16026 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:05.408911  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:05.409636  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:05.414742  5560 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:05.337869 15730 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.381211 15977 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:29:05.349149 15872 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.417124  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:05.422521  5553 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:29:05.384444 15892 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:29:05.399899 15974 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:05.426321 15730 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:05.408500 15928 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:05.429457 15932 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:05.413957 15799 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.023s
I20260709 12:29:05.347350 15901 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.352489 15998 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:05.409054  5704 rpcz_store.cc:276] 0709 12:29:03.656614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:03.657108 (+   494us) service_pool.cc:224] Handling call
0709 12:29:03.669148 (+ 12040us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:03.669636 (+   488us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:03.669678 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:03.670309 (+   631us) log.cc:844] Serialized 26 byte log entry
0709 12:29:03.999062 (+328753us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:04.057484 (+ 58422us) log.cc:844] Serialized 14 byte log entry
0709 12:29:04.058834 (+  1350us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:04.058965 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:04.261707 (+202742us) spinlock_profiling.cc:243] Waited 4.3 ms on lock 0x7b1800448180. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:29:05.329152 (+1067445us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:05.408250 (+ 79098us) raft_consensus.cc:1685] finished
0709 12:29:05.408273 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:05.408425 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":55549,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":13856,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":18299,"spinlock_wait_cycles":17730560,"thread_start_us":328254,"threads_started":1,"wal-append.queue_time_us":652288}
I20260709 12:29:05.440517  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:05.432185 15780 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:05.415158 16035 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:05.409670 15920 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:05.497593 15920 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:05.500164 15780 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:05.500545 15932 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:05.475293  5635 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37. Ops: []
I20260709 12:29:05.469643  5705 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:29:05.501127 15932 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:05.503913 15932 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:05.505532 15930 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:05.506135 15930 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:05.485944  5560 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
W20260709 12:29:05.494421  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 85624.6us
W20260709 12:29:05.495669  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55214.5us
I20260709 12:29:05.426326 15707 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:05.508296  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:05.416121 16026 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:05.487192  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:05.518348  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:05.519941  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:05.520712  5708 raft_consensus.cc:2376] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29. Re-sending same reply.
I20260709 12:29:05.516849  5707 raft_consensus.cc:2376] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 24. Re-sending same reply.
I20260709 12:29:05.432253 15891 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:05.435303 16033 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:05.527702  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:05.466238 16034 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:05.524315  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:05.437565 15956 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:05.517988  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:05.531090  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:05.531183  5630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 2068) took 1425 ms (client timeout 1813 ms). Trace:
I20260709 12:29:05.482962 15928 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:05.534175  5630 rpcz_store.cc:269] 0709 12:29:04.105629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:04.112678 (+  7049us) service_pool.cc:224] Handling call
0709 12:29:05.530933 (+1418255us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400240754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:05.531112 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2367675648}
I20260709 12:29:05.544812  6278 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:05.549365  5602 leader_election.cc:400] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29.
W20260709 12:29:05.549597  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:05.549737  5602 leader_election.cc:403] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:29:05.541424  6274 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
I20260709 12:29:05.515797 15707 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:05.434213 15924 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:05.455766 15714 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.062s
I20260709 12:29:05.534955  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:05.550267 15690 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.003s
W20260709 12:29:05.553193 16029 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.213s	user 0.001s	sys 0.000s
I20260709 12:29:05.517589 15735 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:05.466667 15967 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.067s
I20260709 12:29:05.546341 16033 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:05.530005 15799 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:05.558444  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:05.557368  5630 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:05.507131 15969 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.059s
W20260709 12:29:05.535434 16042 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:29:05.562764  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:05.562713  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:05.591153  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:05.548180 15928 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:05.593930  6271 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:05.568888  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:05.589076  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:05.560613 15974 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29.
I20260709 12:29:05.601065  6277 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:05.515105 16035 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.572667  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:05.619916  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47363.5us
I20260709 12:29:05.570011 16034 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:29:05.634919  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 72236.6us
I20260709 12:29:05.601368 15969 consensus_queue.cc:1243] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 33.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:29:05.655313  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:05.656078 15902 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:05.657202 15902 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:05.661129 16043 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:29:05.661551  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3889) took 1154 ms. Trace:
I20260709 12:29:05.662235  5558 rpcz_store.cc:276] 0709 12:29:04.507494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:04.507661 (+   167us) service_pool.cc:224] Handling call
0709 12:29:04.508090 (+   429us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:04.517310 (+  9220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:04.517346 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:04.526243 (+  8897us) log.cc:844] Serialized 26 byte log entry
0709 12:29:05.458542 (+932299us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b18004c6360. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:29:05.559152 (+100610us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:05.559207 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:05.559305 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:05.661031 (+101726us) raft_consensus.cc:1685] finished
0709 12:29:05.661066 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:05.661443 (+   377us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8713,"raft.queue_time_us":14303,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":72255,"spinlock_wait_cycles":31939840,"thread_start_us":100267,"threads_started":1,"wal-append.queue_time_us":1096296}
I20260709 12:29:05.665940 15690 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:05.666913 15690 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:05.668108 15958 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:05.569527 15972 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.067s
I20260709 12:29:05.673736 15972 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:29:05.674538 15972 consensus_queue.cc:1243] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 24.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:29:05.675176 16037 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:29:05.675969  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3732) took 1142 ms. Trace:
I20260709 12:29:05.676373  5635 rpcz_store.cc:276] 0709 12:29:04.533920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:04.540073 (+  6153us) service_pool.cc:224] Handling call
0709 12:29:04.540514 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:05.502674 (+962160us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:05.675734 (+173060us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b540032d7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:05.675911 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2137382144}
I20260709 12:29:05.679059 15690 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:05.680416 15965 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:05.681464 15965 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:05.684103 15965 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:05.684901  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:05.686182  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:05.686468 15902 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:05.687414 15872 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:05.687805  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3488) took 1729 ms. Trace:
I20260709 12:29:05.688225  5555 rpcz_store.cc:276] 0709 12:29:03.958135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:03.994385 (+ 36250us) service_pool.cc:224] Handling call
0709 12:29:05.243273 (+1248888us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:05.321448 (+ 78175us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:05.687603 (+366155us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54003a89d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:05.687746 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2871132288}
I20260709 12:29:05.689531 15892 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:05.690013 15892 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:05.690351 15892 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:05.692818  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 1809) took 1206 ms. Trace:
I20260709 12:29:05.693253  6275 rpcz_store.cc:276] 0709 12:29:04.486859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:04.487121 (+   262us) service_pool.cc:224] Handling call
0709 12:29:05.692616 (+1205495us) spinlock_profiling.cc:243] Waited 637 ms on lock 0x7b54005f5d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:05.692750 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1466236672}
I20260709 12:29:05.695209  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:05.696581  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:05.592489 15914 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.062s
I20260709 12:29:05.564625 15735 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:05.705506  6279 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:05.706936 15735 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:05.590101 15799 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:05.708927 16036 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.294s	user 0.000s	sys 0.001s
I20260709 12:29:05.709476  6278 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:05.619704 15924 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:05.592792 16021 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.118s
I20260709 12:29:05.711226 16021 consensus_queue.cc:1243] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 27.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:29:05.711701 16021 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:05.711992 16021 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:05.633234 15478 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:05.713124 15478 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:05.712819 16021 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.713505 15478 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:05.714098 16021 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:05.716723  6279 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:05.717527 16014 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:05.717958 16014 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:05.718740 16014 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:05.719733 16014 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:29:05.720628 16042 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 12:29:05.722541  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 67314.6us
I20260709 12:29:05.723080 15780 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:05.723502  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:05.724488  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:05.725236  5554 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:05.725273 15956 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:29:05.725914 15956 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:05.726487 15956 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:05.723582 15780 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:05.728528 15617 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:05.724560  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:05.729048  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:05.751158 16034 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:05.751739 16034 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:05.731639  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:05.734028  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:05.752318  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:05.754614  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:05.766675  5552 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
W20260709 12:29:05.768292  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:05.768399  5554 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
W20260709 12:29:05.771093  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 84994.3us
I20260709 12:29:05.772188  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:05.776072  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:05.776770  5633 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:05.778645  5628 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:05.779981  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:05.753643  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:05.780884  5628 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:05.783133  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:05.783222  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:05.784096  5628 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
W20260709 12:29:05.784137  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46029.2us
I20260709 12:29:05.784641  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:05.785662  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:05.786063  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:05.785858  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:05.786620  5553 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:05.768329 15617 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:05.787724 16012 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.072s	user 0.003s	sys 0.003s
I20260709 12:29:05.785034  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:05.799829  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:05.808945  6278 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:05.823841  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:05.825785  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:05.840240  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:05.842468  5553 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:29:05.843197  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:05.769016 15921 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:05.845909  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:05.846798  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:05.849558  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:05.788081 15967 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:05.788540 15792 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:05.790468 15934 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:29:05.820106 15880 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.040s	sys 0.096s
I20260709 12:29:05.787762 15789 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:05.886152  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58391.1us
I20260709 12:29:05.888553  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:05.905059  5560 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:29:05.890285  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:05.849108  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:05.883667 15902 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:05.893527  5636 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:05.848865 15988 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.063s
I20260709 12:29:05.896862  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:05.782109 16009 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.104s
W20260709 12:29:05.902441  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.808s (SENT) [suppressed 31 similar messages]
I20260709 12:29:05.849633 15690 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:05.906004  5702 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:05.819720 15947 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:05.905141  5633 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:05.930255  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:05.884609 15921 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:05.819551 15872 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:05.894049  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47331.9us
I20260709 12:29:05.979514  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:05.991844  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 170193us
W20260709 12:29:05.915241  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:05.900527 15880 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.040s	sys 0.102s
W20260709 12:29:05.922816  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:05.898561 15965 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:29:05.910706  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:05.909354 15934 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:05.752642 16034 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:05.908576 15902 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:05.941043  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:05.991910  5636 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:05.920302 15930 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:05.827242 15791 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.096s
I20260709 12:29:06.005729  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:05.897689 15967 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:05.997217  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:05.973747 16012 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 12:29:05.997298 16051 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
I20260709 12:29:05.934396 15690 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:06.000572  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:05.973764 15921 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:05.911693 15789 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:06.002998  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:06.007159  5701 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:29:06.001857 15920 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.504s	user 0.008s	sys 0.001s
I20260709 12:29:06.002601 15932 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:06.001947 15965 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:06.032615  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:06.012084  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4011) took 2082 ms. Trace:
I20260709 12:29:06.003706 15902 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:06.064070  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 161711us
I20260709 12:29:06.016992 15617 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:06.066452  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:06.067135 15791 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:06.067638  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:06.068101 15791 consensus_queue.cc:1243] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 35.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:29:06.034804  6272 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:06.069024  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:06.070359  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:06.071606  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:05.978749 15538 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.038s
I20260709 12:29:06.076997  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:06.053854 16055 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:29:06.011044 16009 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:06.054466  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:06.052814 15929 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:06.054522  5706 rpcz_store.cc:276] 0709 12:29:03.936796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:04.010602 (+ 73806us) service_pool.cc:224] Handling call
0709 12:29:04.010995 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:04.011646 (+   651us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:04.011688 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:04.016023 (+  4335us) log.cc:844] Serialized 26 byte log entry
0709 12:29:04.443229 (+427206us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:29:04.459005 (+ 15776us) log.cc:844] Serialized 14 byte log entry
0709 12:29:04.472833 (+ 13828us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:04.472938 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:05.427700 (+954762us) spinlock_profiling.cc:243] Waited 4.7 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:05.429281 (+  1581us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b180041d7c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:29:05.787268 (+357987us) spinlock_profiling.cc:243] Waited 9.88 ms on lock 0x7b240025b6c8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff740b9c4 00007f4ff7409acb 00007f4ff75c4dc9 00007f4ff75c4a55 00007f4ff75c409d 00007f4ffa1f610e 00007f4ffa20d88c 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221
0709 12:29:06.011864 (+224596us) raft_consensus.cc:1685] finished
0709 12:29:06.011889 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:06.012012 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4824,"prepare.queue_time_us":11856,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":61,"raft.queue_time_us":13888,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":7309,"spinlock_wait_cycles":36701184,"thread_start_us":351408,"threads_started":1,"wal-append.queue_time_us":601297}
I20260709 12:29:06.007658 15872 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:06.082185  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:06.000209 16032 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:06.010116  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:06.086705  5633 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:29:06.017359  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.016184 15789 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:06.110472  5702 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:06.104404  5704 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:06.111446  5635 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:06.090469 15932 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:06.062757 15920 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:06.096951 16050 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:29:06.085707 15780 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:06.099618 16009 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:06.101586 16045 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.467s	user 0.003s	sys 0.000s
I20260709 12:29:06.101760 15967 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:06.102756  5560 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: [30.18-32.19]
I20260709 12:29:06.095870 15478 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:06.104962  5636 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 30: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:06.016438 16048 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:06.086604 16051 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:29:06.135504 15780 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:29:06.137133  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:06.150863  5635 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 24: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:06.151782  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:06.137787  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:06.153040  5633 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:29:06.133980  5555 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
W20260709 12:29:06.144548  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:06.102654 15988 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:29:06.157454  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:06.148784  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 140442us
I20260709 12:29:06.111171 15998 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:29:06.142261  5675 connection.cc:256] RPC call timeout handler was delayed by -1.1588273649995244s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
I20260709 12:29:06.155483  5559 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:29:06.161149  5675 connection.cc:256] RPC call timeout handler was delayed by -1.0949273649998759s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1757s for a response
I20260709 12:29:06.183072  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:06.183660  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:06.155345  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:06.158932  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:06.012068 15690 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:06.131052 15914 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:06.189994  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:06.185148  5636 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 37: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:06.186268  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.110708 15929 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:06.155186 15780 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:06.201998  5704 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:06.160552 16037 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:06.159925 15892 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:06.215057  5709 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:06.088214 15934 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:06.223520  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3743) took 1482 ms. Trace:
I20260709 12:29:06.185397  5628 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 35: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:06.210359  5636 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 19. (term mismatch)
I20260709 12:29:06.225684  5636 pending_rounds.cc:85] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 18
I20260709 12:29:06.155893 15988 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:06.212239  5708 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 24. Current term is 25. Ops: []
I20260709 12:29:06.212174  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:06.225984  5636 pending_rounds.cc:107] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 27.19
I20260709 12:29:06.230985  5636 raft_consensus.cc:2889] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:29:06.208329 16045 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:29:06.206377 15914 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:06.228025  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:06.223989  5634 rpcz_store.cc:276] 0709 12:29:04.740908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:05.474481 (+733573us) service_pool.cc:224] Handling call
0709 12:29:05.508746 (+ 34265us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:05.541129 (+ 32383us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:05.541174 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:05.550921 (+  9747us) log.cc:844] Serialized 26 byte log entry
0709 12:29:05.796159 (+245238us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:05.796200 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:05.796271 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:06.223258 (+426987us) raft_consensus.cc:1685] finished
0709 12:29:06.223301 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:06.223453 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21701,"raft.queue_time_us":22390,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":10867,"thread_start_us":226290,"threads_started":1,"wal-append.queue_time_us":369213}
I20260709 12:29:06.232002  5706 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:06.208966  5635 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 34: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:06.300173  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:06.285056  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45652.3us
I20260709 12:29:06.286973  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:06.290052 16051 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:29:06.162822 16048 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:06.223069 15989 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:06.309247 16060 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:29:06.311205 15730 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:06.311239  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:06.311924  5631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3518) took 2839 ms (client timeout 2231 ms). Trace:
I20260709 12:29:06.312800  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.312731  5632 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
W20260709 12:29:06.322664  5631 rpcz_store.cc:269] 0709 12:29:03.472153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:03.867687 (+395534us) service_pool.cc:224] Handling call
0709 12:29:06.311697 (+2444010us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400134f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:06.311834 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3120404224}
I20260709 12:29:06.313906  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.324519  5708 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:29:06.318025  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:06.162110 15965 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:06.301788 16032 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:06.292249 16068 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:06.331378  5556 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:06.332391  5701 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:06.303915 15914 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:06.321331 15730 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:06.335158  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.335453 15730 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:06.336742 15730 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:06.341179  5710 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:29:06.342633  5710 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:29:06.342540  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:06.240219 15780 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:06.345108 15780 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:29:06.354951  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 195881us
I20260709 12:29:06.325115  5631 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:29:06.357828  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:06.380676  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3739) took 1844 ms. Trace:
W20260709 12:29:06.383332  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:06.387446  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:06.391832  5710 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:29:06.394214  6274 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: [32.21-32.21]
W20260709 12:29:06.398614  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:06.399448  5706 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:06.349923 16065 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:06.381192  5629 rpcz_store.cc:276] 0709 12:29:04.536278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:05.349507 (+813229us) service_pool.cc:224] Handling call
0709 12:29:05.350309 (+   802us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:05.350717 (+   408us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:29:05.543028 (+192311us) log.cc:844] Serialized 14 byte log entry
0709 12:29:05.623375 (+ 80347us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:05.636801 (+ 13426us) log.cc:844] Serialized 26 byte log entry
0709 12:29:05.636929 (+   128us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:29:05.637111 (+   182us) log.cc:844] Serialized 14 byte log entry
0709 12:29:05.653270 (+ 16159us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:05.653359 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:06.313037 (+659678us) spinlock_profiling.cc:243] Waited 44 ms on lock 0x7b18004f6f60. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:29:06.380432 (+ 67395us) raft_consensus.cc:1685] finished
0709 12:29:06.380458 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:06.380615 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46554,"prepare.queue_time_us":939084,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":91,"raft.queue_time_us":18004,"raft.run_cpu_time_us":364,"raft.run_wall_time_us":13981,"spinlock_wait_cycles":101294464,"thread_start_us":158740,"threads_started":2,"wal-append.queue_time_us":108487}
W20260709 12:29:06.385533 16026 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.829s	user 0.000s	sys 0.011s
I20260709 12:29:06.324172 15989 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:06.412940  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:06.413025  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:06.414908  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:06.425750  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:06.426622  6274 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:06.417292  5629 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 13. Preceding OpId from leader: term: 32 index: 14. (index mismatch)
I20260709 12:29:06.430961  5629 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
W20260709 12:29:06.425879 15928 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.772s	user 0.007s	sys 0.004s
I20260709 12:29:06.432780  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:06.349692 16032 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:06.410843 16065 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:06.416013  5634 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:29:06.440951  5634 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:06.439021  5561 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 28: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:06.436342  5556 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:29:06.456374 15924 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:06.457650 15924 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:06.457738  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2053) took 3456 ms. Trace:
I20260709 12:29:06.458068 15924 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:06.458511 15924 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:06.459632  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:06.417048 16026 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:06.460970 16035 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:06.354637 16068 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:06.461966 16035 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:06.462452 16035 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:06.463505 16026 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:06.464249 16068 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:06.437275 15872 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:06.465078  5554 raft_consensus.cc:2393] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 34: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:06.427973 16009 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:06.428366 15478 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:06.458292  5632 rpcz_store.cc:276] 0709 12:29:03.000989 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:03.963849 (+962860us) service_pool.cc:224] Handling call
0709 12:29:06.312486 (+2348637us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:06.325763 (+ 13277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:06.457478 (+131715us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b5400134f50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:06.457642 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5400447232}
I20260709 12:29:06.463677 16035 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:06.467132 16021 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.753s	user 0.011s	sys 0.000s
I20260709 12:29:06.467722 16035 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:06.469254  5554 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: [32.21-32.21]
I20260709 12:29:06.469964  6279 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34. Ops: [33.22-33.22]
I20260709 12:29:06.434469 15920 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:06.472069  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:06.324533 15988 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:06.472796 16032 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:06.478180  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:06.481238  5701 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:06.483103 15914 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:06.484658  5560 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:06.486285 16048 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:06.487918 15989 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:06.488332  5701 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:06.488615  5627 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:29:06.489318  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.479991 15988 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:06.492488  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:06.442142  5634 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:06.429112 15799 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:06.490736 16014 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.736s	user 0.006s	sys 0.004s
I20260709 12:29:06.512133  6274 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:06.446658 16065 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:06.514879  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:06.495373 16021 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:06.522347  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.432240 15928 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:06.523430  5701 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:06.460145 15921 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:06.534070  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 146690us
I20260709 12:29:06.536592  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:06.518465 15929 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:06.470607 15924 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:06.550868  5701 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:06.522507 15892 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:06.554430  6280 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
W20260709 12:29:06.636456  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:06.664331 15478 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:06.665467 15478 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:06.671633  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:06.673755 15478 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:06.684376  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:06.691037  5702 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
W20260709 12:29:06.695652  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 159157us
I20260709 12:29:06.695830  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:06.697649  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:06.700709 16014 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:06.701185 16014 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:06.701545 16014 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:06.706261  5702 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:06.708335  5702 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: []
I20260709 12:29:06.542169 16021 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:06.707980  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:06.708938 15902 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.628s	user 0.006s	sys 0.005s
I20260709 12:29:06.744524  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.742735  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.751703 16021 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:06.753414 15707 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:06.756881  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48980.5us
I20260709 12:29:06.757647 15934 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:06.761155  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.816617 15934 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:06.817588 15934 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.23, Last appended by leader: 23, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:06.823308  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:06.779366  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:06.736817 16073 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:06.824643  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:06.824839 16073 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:06.825330  6276 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:29:06.825778 16073 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:06.829677 16009 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:29:06.829810 15998 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.647s	user 0.008s	sys 0.001s
I20260709 12:29:06.741585 15914 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:06.830566  5704 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:29:06.831035 16048 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:06.831493 16048 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:06.793210 16078 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260709 12:29:06.833063 15956 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.106s	user 0.006s	sys 0.007s
I20260709 12:29:06.833636 15956 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:06.834455  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 55220.4us
I20260709 12:29:06.797787 16067 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.017s
I20260709 12:29:06.759537 15902 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:06.836826  5633 raft_consensus.cc:2410] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 32 because replica has last-logged OpId of term: 30 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 15.
W20260709 12:29:06.802389 16079 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:29:06.805516  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.839145  5701 raft_consensus.cc:2410] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 33 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 16.
I20260709 12:29:06.841259 15902 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:06.842481  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:06.807273 16077 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:29:06.813369 15924 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:06.814406 15921 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:06.815431 15928 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:06.823556  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:06.834692  5707 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:06.854959 15707 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:06.806488 16084 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:29:06.856904  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:06.858043  6274 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:29:06.858529 15921 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:29:06.845683 15730 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.509s	user 0.009s	sys 0.001s
I20260709 12:29:06.860327  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 2078) took 1061 ms. Trace:
I20260709 12:29:06.887197  5628 rpcz_store.cc:276] 0709 12:29:05.799276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:05.823535 (+ 24259us) service_pool.cc:224] Handling call
0709 12:29:06.860042 (+1036507us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b54001f2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:06.860241 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":781707520}
I20260709 12:29:06.862385  5707 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:29:06.864578  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:06.889020  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:06.865624  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:06.889941  5633 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:06.850571 16048 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:06.774494  5702 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:06.890074  5629 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:29:06.787199  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:06.895015  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:06.898654  5675 leader_election.cc:336] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.231s (SENT)
I20260709 12:29:06.899322  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:06.842823 15956 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:06.888633  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:06.904217  5628 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:06.904918  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:06.905812  5633 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:29:06.908814  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:06.913004 16064 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.006s	sys 0.051s
I20260709 12:29:06.935371  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:06.939515  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:06.940651  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:06.920015  5629 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:29:06.892762 16067 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:06.896586 15928 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:06.923142  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:06.913228 15924 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:06.937019  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:06.911950 15903 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:06.929505  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:06.935544 16064 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.007s	sys 0.051s
I20260709 12:29:06.921238 15818 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:06.928033 15956 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:06.927754 16074 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:06.967240  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:06.959961  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:06.933406 16068 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:06.939276 16082 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:06.977545  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:06.958523  5628 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:29:06.978173  5633 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:06.978518  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:06.946005 15989 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:06.941180 15730 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:06.959394 16035 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:29:06.943908 16058 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:29:06.959187 15735 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:07.012009  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:07.013072  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:07.013913  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:07.006162  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.927145 15682 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.054s
I20260709 12:29:07.016942  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.981003 16009 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:07.017647  5705 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:29:06.971011 15921 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:06.959235 15928 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:06.972726 15924 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.023667  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:06.984061 15989 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:06.984143 15901 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:29:07.030609  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 93660.6us
I20260709 12:29:07.026415  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:07.025657  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.972910 15903 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.031903  5701 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:07.029031  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:07.006608 15799 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:07.039812  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:06.980427 15818 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.008253 16035 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:06.985106 16074 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.064548  5701 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 37.
W20260709 12:29:07.063869 16065 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.542s	user 0.010s	sys 0.000s
I20260709 12:29:07.016692 15478 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:07.023494 15928 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:07.070338  5633 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:07.060426 16027 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:07.064574  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 117620us
I20260709 12:29:07.031841 15989 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:07.067703  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:07.007994 15730 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.009913 16068 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.073383  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:07.076761  5704 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:07.079273  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:07.069042 16035 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:07.079739  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:07.054477 16073 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:07.082201  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:07.112166  6279 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 31: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:07.111780 16073 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:07.082283  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 65069.5us
I20260709 12:29:07.022390 15735 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.113689  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:07.114974 15735 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:07.085397 16019 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.916s	user 0.003s	sys 0.002s
I20260709 12:29:07.086458 16074 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:07.117728  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4015) took 2792 ms. Trace:
I20260709 12:29:07.118192  5703 rpcz_store.cc:276] 0709 12:29:04.325726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:04.326174 (+   448us) service_pool.cc:224] Handling call
0709 12:29:04.347508 (+ 21334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:04.347962 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:29:04.398176 (+ 50214us) log.cc:844] Serialized 14 byte log entry
0709 12:29:04.784742 (+386566us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:04.786014 (+  1272us) log.cc:844] Serialized 26 byte log entry
0709 12:29:04.786207 (+   193us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:04.786256 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:04.786370 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:07.046355 (+2259985us) spinlock_profiling.cc:243] Waited 37 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:07.117394 (+ 71039us) raft_consensus.cc:1685] finished
0709 12:29:07.117500 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:07.117659 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13719,"prepare.queue_time_us":24587,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":10849,"raft.queue_time_us":29502,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":17150,"spinlock_wait_cycles":85088640,"thread_start_us":365729,"threads_started":1,"wal-append.queue_time_us":725700}
I20260709 12:29:07.075940 16027 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:07.121080  5559 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3898) took 1768 ms (client timeout 2064 ms). Trace:
W20260709 12:29:07.121598  5559 rpcz_store.cc:269] 0709 12:29:05.352258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:05.352403 (+   145us) service_pool.cc:224] Handling call
0709 12:29:07.120664 (+1768261us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b540016ba54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:07.120975 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1766513408}
I20260709 12:29:07.123081  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:07.123219 16027 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:07.087365 15934 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:07.088696  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3524) took 1285 ms. Trace:
I20260709 12:29:07.093348 16014 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.097680 15924 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:07.097966  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2061) took 1027 ms. Trace:
I20260709 12:29:07.100337 15792 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:07.144825  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:07.101190  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:07.072388 15478 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:07.103592 16085 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.346s	user 0.000s	sys 0.001s
I20260709 12:29:07.070778 16065 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.104736  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2063) took 1033 ms. Trace:
I20260709 12:29:07.043573 15930 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
W20260709 12:29:07.105402  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:07.077392 15690 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:07.137821 15921 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:07.166533  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:07.167387  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:07.106704  6278 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1590) took 1548 ms (client timeout 1640 ms). Trace:
I20260709 12:29:07.106293  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:07.109434 15818 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:07.110831 15730 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:07.111191 15682 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:07.174804 15682 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.150568 16067 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:29:07.177681 15682 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:07.177488 15818 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:29:07.178821 15924 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:07.179282 15924 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:07.180096 15924 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.172611 15921 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.181975 15924 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:07.182585 15921 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:07.182448 15928 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.191493  5630 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 33 index: 21. (term mismatch)
I20260709 12:29:07.191985  5630 pending_rounds.cc:85] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 20
I20260709 12:29:07.192222  5630 pending_rounds.cc:107] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 32.21
I20260709 12:29:07.192514  5630 raft_consensus.cc:2889] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:07.192530  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:07.123910  5559 raft_consensus.cc:2368] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27.
W20260709 12:29:07.128177  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48538.5us
I20260709 12:29:07.125339  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:07.125695  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:07.198374  5552 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:07.198561 15902 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:29:07.142114  5561 rpcz_store.cc:276] 0709 12:29:06.070464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:06.513011 (+442547us) service_pool.cc:224] Handling call
0709 12:29:06.517083 (+  4072us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:06.553082 (+ 35999us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:06.553121 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:06.565557 (+ 12436us) log.cc:844] Serialized 52 byte log entry
0709 12:29:06.893645 (+328088us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:29:06.893688 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:06.893758 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:07.097734 (+203976us) raft_consensus.cc:1685] finished
0709 12:29:07.097759 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:07.097907 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74581,"raft.queue_time_us":34576,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":139176,"thread_start_us":152568,"threads_started":1,"wal-append.queue_time_us":318282}
I20260709 12:29:07.136636  5555 rpcz_store.cc:276] 0709 12:29:05.803266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:06.468167 (+664901us) service_pool.cc:224] Handling call
0709 12:29:06.468542 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:06.468862 (+   320us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:06.468891 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:06.469369 (+   478us) log.cc:844] Serialized 26 byte log entry
0709 12:29:06.873401 (+404032us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:29:06.893047 (+ 19646us) log.cc:844] Serialized 14 byte log entry
0709 12:29:06.893262 (+   215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:06.893340 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:07.088437 (+195097us) raft_consensus.cc:1685] finished
0709 12:29:07.088464 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:07.088625 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4905,"prepare.queue_time_us":19427,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":4440,"raft.queue_time_us":4665,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":7099,"thread_start_us":403590,"threads_started":1,"wal-append.queue_time_us":540345}
I20260709 12:29:07.200448  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:07.140745 15988 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.632s	user 0.012s	sys 0.000s
W20260709 12:29:07.147892  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 46814.5us
I20260709 12:29:07.201289  5555 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:07.201884  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:07.202723 15921 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:29:07.203182  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:07.144860 15792 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:07.204272 16014 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:07.083837 16066 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:29:07.106060 16081 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.400s	user 0.001s	sys 0.000s
I20260709 12:29:07.143033 16074 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:29:07.169126  6274 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:29:07.157187  5556 rpcz_store.cc:276] 0709 12:29:06.071037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:06.853908 (+782871us) service_pool.cc:224] Handling call
0709 12:29:06.854275 (+   367us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:06.854475 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:06.854500 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:06.854983 (+   483us) log.cc:844] Serialized 26 byte log entry
0709 12:29:07.009212 (+154229us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:07.009256 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:07.009326 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:07.104521 (+ 95195us) raft_consensus.cc:1685] finished
0709 12:29:07.104541 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:07.104677 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":153831,"threads_started":1,"wal-append.queue_time_us":247698}
W20260709 12:29:07.168694  6278 rpcz_store.cc:269] 0709 12:29:05.558045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:05.558169 (+   124us) service_pool.cc:224] Handling call
0709 12:29:07.106532 (+1548363us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b5400343ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:07.106647 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":329172864}
I20260709 12:29:07.115002 16073 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:07.149374 15965 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:29:07.173899  5454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 178) took 127631 ms (client timeout 15000 ms). Trace:
I20260709 12:29:07.199451 15902 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:07.200615 15478 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:07.148479  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:07.237579  5553 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:29:07.205945  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:07.187626 15989 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:07.206545  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 163323us
I20260709 12:29:07.206951  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:07.124112 15934 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.205794  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:07.212136  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:07.255049  6236 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:07.213047  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68319.2us
W20260709 12:29:07.257098  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51385us
I20260709 12:29:07.257730  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:07.259008  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:07.259668  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:07.197788 15707 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:07.228236  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:07.262861  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:07.262764  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:07.264310  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:07.264815  5633 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:07.230962  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:07.232146  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2059) took 1162 ms. Trace:
I20260709 12:29:07.196466 15914 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:07.223385 16050 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:29:07.239639  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:07.265412  5554 rpcz_store.cc:276] 0709 12:29:06.069813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:06.469978 (+400165us) service_pool.cc:224] Handling call
0709 12:29:06.470226 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:06.472300 (+  2074us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:29:06.484852 (+ 12552us) log.cc:844] Serialized 14 byte log entry
0709 12:29:06.705788 (+220936us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:06.705846 (+    58us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:06.705894 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:07.231803 (+525909us) spinlock_profiling.cc:243] Waited 1.76 ms on lock 0x7b54000ebfd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:07.231878 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:07.232068 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":175315,"prepare.queue_time_us":16741,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":153503,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":13641,"spinlock_wait_cycles":4044672,"thread_start_us":68617,"threads_started":1,"wal-append.queue_time_us":295560}
I20260709 12:29:07.269932  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:07.271384  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:07.272054  5554 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:07.247143  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:07.211618 15921 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:07.253657  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 2060) took 1183 ms. Trace:
I20260709 12:29:07.276098  5557 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:07.209291 15730 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:07.258039  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:07.230283  5454 rpcz_store.cc:269] 0709 12:26:59.604317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:59.617705 (+ 13388us) service_pool.cc:224] Handling call
0709 12:27:00.966042 (+1348337us) spinlock_profiling.cc:243] Waited 6.41 ms on lock 0x7b5400002490. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754e0b2 00007f50004c7759 00007f500045fa76 00007f5000430d58 00007f500052f04e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc
0709 12:29:07.173665 (+126207623us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3365106,"reported_tablets":32,"spinlock_wait_cycles":14740864}
I20260709 12:29:07.211953  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:07.221046  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1586) took 1697 ms. Trace:
I20260709 12:29:07.255010  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:07.276527  5560 rpcz_store.cc:276] 0709 12:29:06.070163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:06.164760 (+ 94597us) service_pool.cc:224] Handling call
0709 12:29:07.253451 (+1088691us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b54001c48d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:07.253586 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":584202496}
I20260709 12:29:07.266465  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:07.281092  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 74205.4us
I20260709 12:29:07.228281 16066 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:07.247552  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:07.253764  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:07.291595  5556 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:07.277500  5632 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:07.293500  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:07.278426  6280 rpcz_store.cc:276] 0709 12:29:05.523374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:05.523888 (+   514us) service_pool.cc:224] Handling call
0709 12:29:07.220647 (+1696759us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400773154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:07.220949 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2347898752}
I20260709 12:29:07.284641  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:07.285583  5553 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 29: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:07.234385 16074 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:07.295656  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:07.295812  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48365.2us
I20260709 12:29:07.297691  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:07.262274 16067 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:07.261822 15928 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:07.337864  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:07.267149 16014 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:07.205382 15792 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:07.339083  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 72694.9us
I20260709 12:29:07.349556  5560 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27. Ops: []
W20260709 12:29:07.350116  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 109549us
I20260709 12:29:07.350560  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:07.276981 16099 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:07.343078  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:07.339180  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45776.6us
I20260709 12:29:07.352754  5632 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:29:07.354018  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:07.341681  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:07.354797  5632 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:07.354828  5554 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:07.354437  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:07.345033  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:07.267460 16101 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:07.337904 15932 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:07.356711 15901 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:29:07.357769 15976 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:07.348932  5559 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:07.359323 16107 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:29:07.348963 16110 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.023s
I20260709 12:29:07.360672 15682 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:07.350203  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2064) took 1278 ms. Trace:
I20260709 12:29:07.361303 16110 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:07.263222 15965 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:07.361970 16110 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:07.362404 16105 raft_consensus.cc:2764] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:29:07.362280 15965 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.363855 16068 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:07.363517 16066 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.242571 15902 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.364410 16048 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.012s
I20260709 12:29:07.366039 15902 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:07.301177 15730 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:07.367208 15730 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:07.361300 15682 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.349345 15921 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:07.297443 16058 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:29:07.367288 16070 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.944s	user 0.003s	sys 0.000s
W20260709 12:29:07.361215  5454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 278) took 27751 ms (client timeout 15000 ms). Trace:
I20260709 12:29:07.370065  5554 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:07.361590  5560 rpcz_store.cc:276] 0709 12:29:06.071334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:07.292639 (+1221305us) service_pool.cc:224] Handling call
0709 12:29:07.349361 (+ 56722us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:07.349948 (+   587us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:07.350146 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:07.373082  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:07.375358  5632 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:29:07.338367 16067 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:07.402064  6280 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: []
I20260709 12:29:07.396641  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:07.407019  5454 rpcz_store.cc:269] 0709 12:28:39.609450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:07.360990 (+27751540us) service_pool.cc:211] Skipping call since client already timed out
0709 12:29:07.361128 (+   138us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:29:07.375232  5633 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:07.277381 15707 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.413870  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:07.288256 15914 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.415235  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:07.416186  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:07.417208  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:07.418478  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:07.421914  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:29:07.414538  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 132257us
I20260709 12:29:07.439674  6275 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:07.425295  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:07.430564  5556 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:07.430704  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:07.443382  5556 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:29:07.431226  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:07.448068  6280 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:29:07.448211  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:07.362450 16110 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:07.357700 16014 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:07.454736  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:07.372074 15735 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:07.352749 15903 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:07.352012 15930 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:07.453023  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:07.372378 15902 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:07.406363 15921 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:07.454602  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:07.440392  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:07.448565 15901 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:07.456653 16037 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:07.403681 16048 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:07.410589 15714 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.056s
I20260709 12:29:07.462646  5556 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
W20260709 12:29:07.462874  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4022) took 1638 ms (client timeout 2105 ms). Trace:
I20260709 12:29:07.351368 15780 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.101s
I20260709 12:29:07.500880 15707 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:07.468289 15965 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:07.502033 16014 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.15, Last appended by leader: 15, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.504527 16048 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.506898 16048 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:07.468307  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:07.473441  5701 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:29:07.509246  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:07.474239  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 219275us
I20260709 12:29:07.376192 16064 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.083s
I20260709 12:29:07.455799  5554 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:29:07.455269 15998 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:07.451705 15934 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:07.420884 16067 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:07.549220  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:07.550501  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:07.493425  5707 rpcz_store.cc:269] 0709 12:29:05.824250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:05.849251 (+ 25001us) service_pool.cc:224] Handling call
0709 12:29:07.462712 (+1613461us) spinlock_profiling.cc:243] Waited 948 ms on lock 0x7b54002f4254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:07.462815 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2180598656}
I20260709 12:29:07.551635  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:07.402338 16105 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.080s
I20260709 12:29:07.494138 15901 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.511952  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:07.488924 16037 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:07.518282 15989 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:07.566196  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:07.527992 15682 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:07.489732 16106 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:07.529693  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:07.534071  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 93782.9us
I20260709 12:29:07.536782  5560 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 13. Preceding OpId from leader: term: 32 index: 14. (index mismatch)
I20260709 12:29:07.481947 15538 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:07.552191  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:07.356717 16101 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.487661 15956 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.074s
I20260709 12:29:07.541527 16065 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:07.550915 15789 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:07.573292  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:07.556975 15965 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:07.592600  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:07.484611 16074 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.484571 16108 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.117s
I20260709 12:29:07.559063 16048 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:07.614302  5631 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:07.621942  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:07.619354  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 46129.1us
I20260709 12:29:07.626820 15901 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:07.627542 15901 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:29:07.628021 15901 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:07.630746  6279 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:29:07.635893  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:07.650543  5706 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:07.651683  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:07.652390  5706 raft_consensus.cc:2376] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31. Re-sending same reply.
I20260709 12:29:07.653512  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:07.654222  5706 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:07.642935 15730 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.643736  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:07.645364  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:07.646140  5676 leader_election.cc:400] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27.
I20260709 12:29:07.658043  5676 leader_election.cc:403] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
W20260709 12:29:07.646549  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:07.591339 16066 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:07.558563 16009 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.056s
I20260709 12:29:07.648141  5560 pending_rounds.cc:85] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 18
I20260709 12:29:07.661365  5560 pending_rounds.cc:107] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 28.19
I20260709 12:29:07.661705  5560 raft_consensus.cc:2889] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:07.649113  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:07.642421 16027 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:29:07.654313  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:07.542858 16109 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:29:07.624419  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:07.664252  5635 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:29:07.689543  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:07.690523  5709 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:29:07.693449  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 69181.9us
I20260709 12:29:07.572012 15538 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:07.622232 16048 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:07.707240  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61237.2us
I20260709 12:29:07.709491  5632 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:07.711194  5632 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:07.647059 16065 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:07.707792 16106 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:07.712828  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:29:07.666285 16035 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.525s	user 0.008s	sys 0.004s
I20260709 12:29:07.592200 15902 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:07.658345 15965 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:29:07.748735  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 196281us
I20260709 12:29:07.666849 16027 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:07.680876 15682 raft_consensus.cc:2764] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election decision vote started in defunct term 34: won
I20260709 12:29:07.726754  5632 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.758709  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:07.759682  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:07.750500 16014 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:07.753911 16021 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:07.750909 16105 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:07.768975 15902 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.758587  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:07.716966  5706 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:07.697538 16068 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:07.779106  5709 raft_consensus.cc:2410] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 29 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:29:07.697364 16119 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:29:07.779729  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 187246us
I20260709 12:29:07.618218 15789 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.781421  5632 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:07.710568 15538 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:07.710096 15690 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:07.714586 15924 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:07.716284 16065 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:07.717268 16048 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:07.866801 16101 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:07.850783  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2069) took 1519 ms. Trace:
I20260709 12:29:07.857671 16027 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.860785 16106 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.864084  5634 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:29:07.714751 16059 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.135s
I20260709 12:29:07.865103 16129 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:29:07.856330 16115 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.239s	user 0.001s	sys 0.000s
I20260709 12:29:07.855305 15965 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.867525  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:07.931502  5632 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:29:07.797243 16021 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.916195 16137 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:07.897543 15791 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:07.897768  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 184717us
I20260709 12:29:07.945894 16137 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.946908  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:07.843356 15901 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.948098  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:07.948319 16137 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:07.948954 16137 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:07.949304 16137 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:07.949641 16137 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:29:07.897943  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 119732us
I20260709 12:29:07.900195  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:07.901513  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3534) took 1908 ms. Trace:
W20260709 12:29:07.901885  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 143455us
I20260709 12:29:07.954164  5634 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:29:07.904247  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2074) took 2763 ms. Trace:
I20260709 12:29:07.905896  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:07.751250 16035 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.917130  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:07.869656 15921 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27.
I20260709 12:29:07.920892  5557 rpcz_store.cc:276] 0709 12:29:06.331537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:07.289093 (+957556us) service_pool.cc:224] Handling call
0709 12:29:07.289637 (+   544us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:07.289949 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:07.289990 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:07.342330 (+ 52340us) log.cc:844] Serialized 26 byte log entry
0709 12:29:07.483030 (+140700us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:07.483144 (+   114us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:07.483217 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:07.850465 (+367248us) raft_consensus.cc:1685] finished
0709 12:29:07.850494 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:07.850702 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":79733,"threads_started":1,"wal-append.queue_time_us":232966}
I20260709 12:29:07.960388  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:07.799203 16014 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.909530 16073 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:07.933945 15789 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:07.935386 15902 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:07.680102 15930 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.102s
I20260709 12:29:07.842646 16105 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.936655  5558 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:07.750741 16108 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:07.914134 16065 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:07.942067 16027 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:07.942287  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:07.944429 16021 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:07.916792 15924 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:07.951922  5627 rpcz_store.cc:276] 0709 12:29:05.992679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:06.402226 (+409547us) service_pool.cc:224] Handling call
0709 12:29:06.402746 (+   520us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:06.488989 (+ 86243us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:07.900320 (+1411331us) spinlock_profiling.cc:243] Waited 85.7 ms on lock 0x7b5400242054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:07.901434 (+  1114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":197055360}
I20260709 12:29:07.902303  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:07.991917  5706 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:07.992944  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:07.993521  5706 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:29:07.994174  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:07.994714  5559 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: []
W20260709 12:29:07.953703  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:07.951745  5552 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
W20260709 12:29:07.955533  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49733.8us
I20260709 12:29:07.998278  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:07.998947  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:07.954927  5630 rpcz_store.cc:276] 0709 12:29:05.140856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:05.592213 (+451357us) service_pool.cc:224] Handling call
0709 12:29:05.592551 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:07.193293 (+1600742us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:07.904022 (+710729us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:07.904196 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3631254272}
I20260709 12:29:07.956995  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2085) took 2051 ms. Trace:
I20260709 12:29:07.999603  5634 rpcz_store.cc:276] 0709 12:29:05.905423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:06.385344 (+479921us) service_pool.cc:224] Handling call
0709 12:29:06.415848 (+ 30504us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:07.863585 (+1447737us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b58009536d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5ce0aa 00007f4ffa650447 00007f4ffa668fe7 00007f4ffa66a7de 00007f4ffa674b49 00007f4ffa675416 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4
0709 12:29:07.954858 (+ 91273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:07.956899 (+  2041us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3128826240}
I20260709 12:29:07.957319  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:07.957441  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:07.928999 16129 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:07.911141 15690 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:07.952483 16116 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.255s	user 0.001s	sys 0.000s
I20260709 12:29:07.967365  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:07.980589  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:07.983085 15789 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:07.983261 16014 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:07.983623 15921 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:07.984576  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:07.977548 16105 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:07.978302 16026 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:29:07.975081 16108 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:07.916847 16067 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:07.945967 15791 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.001701  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:07.913070 16110 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.004842  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:08.006880  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49533.9us
I20260709 12:29:08.007297  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:08.026952  5552 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:29:08.027371  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:07.986245 16125 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:29:08.011327  5557 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:29:08.011955  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63932.8us
I20260709 12:29:08.012908  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:07.970036 16073 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:07.962169 16138 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:29:07.945685 16037 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.027454  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:08.027890  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:07.964423 16038 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260709 12:29:08.028211  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:08.030452 15735 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:08.031013 16105 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:08.031250  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:08.031488 16105 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.031401 15735 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.031950 16105 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:08.032409  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:08.009800 16068 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:08.010093 16021 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:08.033491  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:08.006532 15902 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:29:08.036095  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:08.036726  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:08.041244  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:08.043452  5627 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: [26.13-26.13]
I20260709 12:29:08.045104  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:08.042450  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:08.045953  5701 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:08.023602 16026 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:08.030709  5709 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:08.013924 16143 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:08.056402 16143 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.056928 16143 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:08.058400 15924 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.050186  5634 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:08.055187 16059 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:29:08.060734 16067 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:08.049526  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:08.066704  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:08.035801 16073 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:08.069635  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:08.054555  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:08.037588 16119 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:08.036177 16138 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.077162  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:08.077594  5557 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:08.038342 15735 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:08.039949 16106 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:08.083034 16101 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.113443  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:08.048950 16021 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.085719 16126 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:29:08.115960  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 67368.6us
I20260709 12:29:08.047657 16124 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:08.080727 16059 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:08.082981 15902 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:29:08.047824 15921 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.071s
I20260709 12:29:08.122465  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:08.128672  5634 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:08.051250 15789 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.065s
I20260709 12:29:08.081278 16138 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:08.083181 16134 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:08.048826 16068 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.120991 16021 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:08.123463 16059 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.132884 16126 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.135965 16128 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.131098 16026 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.122516 16101 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.086005 16108 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.048234 16129 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.135156 16134 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.141458 16059 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:29:08.233201  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:08.172684  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:08.181641  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:08.243142  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 122839us
I20260709 12:29:08.183647  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:08.139180 16124 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:08.245970  5705 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
W20260709 12:29:08.241076  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 127708us
W20260709 12:29:08.177016  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:08.223472  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:08.248433  5599 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:08.250380  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:08.251096  5633 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:08.140426 15902 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:08.249953  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:08.155329 16123 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
W20260709 12:29:08.175951 15998 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.622s	user 0.008s	sys 0.001s
I20260709 12:29:08.176417 15789 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:08.147303 16035 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:08.258317  5705 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:29:08.268404  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:08.266276  5633 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:08.179831 15914 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.165524 16139 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:08.248437  5709 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
W20260709 12:29:08.252396  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:08.269328  5705 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:08.329674  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:08.164685 16067 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.101s
I20260709 12:29:08.331008  5709 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:08.159691 15538 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:08.331976  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 74836.4us
I20260709 12:29:08.258665 15707 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:08.335916  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:08.338382  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:08.266157 16026 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:08.338382  5710 raft_consensus.cc:2368] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29.
I20260709 12:29:08.267311 16037 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.155874 16143 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.342216  5709 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:08.342564  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:08.328858 15930 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:08.220054 16065 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:08.364671 16152 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 12:29:08.365350 15538 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.168448 16128 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.328512 16139 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:08.265503 15998 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:08.265256 15789 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:08.328291 16033 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.242s	user 0.015s	sys 0.037s
W20260709 12:29:08.360689 16144 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.239s	user 0.001s	sys 0.000s
I20260709 12:29:08.160395 16126 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.376622  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:08.387164  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 59509.4us
I20260709 12:29:08.349500  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:08.334726 16110 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:08.392874  5708 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:29:08.395757  5707 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 29: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:08.401074  5706 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
W20260709 12:29:08.401355  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61496.2us
I20260709 12:29:08.342134 15707 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:08.386478 15914 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:08.405508  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:08.403556  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:08.384846 15538 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:08.395751 16033 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.310s	user 0.016s	sys 0.037s
I20260709 12:29:08.409402 15538 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:08.409943 16135 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
W20260709 12:29:08.411166 16100 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.058s	user 0.008s	sys 0.006s
I20260709 12:29:08.411667 15538 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:08.255779 15929 consensus_queue.cc:1175] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" responder_term: 27 status { last_received { term: 24 index: 12 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: [26.13-26.13]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:29:08.413633 16058 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:08.414059 16058 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:08.414934 16058 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.415985 16058 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:08.417200  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:08.395949 15789 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:29:08.402386 16147 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.258s	user 0.001s	sys 0.000s
I20260709 12:29:08.359885 16154 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.060s
I20260709 12:29:08.368997 15901 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
I20260709 12:29:08.408594 15707 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:08.443872  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:08.444885  5560 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:08.426887 16139 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.445856 16139 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:08.419821 15914 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.428869 15538 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:08.408445 16068 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.418049 16127 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:08.454725 16127 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:08.455252 16127 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:08.466109 16134 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:08.469189 16101 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:08.468181 15998 consensus_queue.cc:1175] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" responder_term: 29 status { last_received { term: 29 index: 15 } last_committed_idx: 14 last_received_current_leader { term: 29 index: 15 } }
I20260709 12:29:08.386802 15930 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.388499 16158 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.053s
I20260709 12:29:08.433889 16153 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:08.490021 15730 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.368372 15902 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.21, Last appended by leader: 21, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:08.506093  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:08.500751 16153 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:08.518450 16137 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.564s	user 0.007s	sys 0.006s
I20260709 12:29:08.518333 16153 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.518929 16137 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.507702 16134 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.522451 16129 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:08.522742 16134 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.524663 16161 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:08.525367 16161 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.528330 16161 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.529881 16134 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:08.531543 16151 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:29:08.532002 16151 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:08.533280 16151 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.534212 15730 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.535017 16151 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:08.534950 15730 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.537748 15730 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:08.538115 15930 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:08.540926 15735 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.549396  5560 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
W20260709 12:29:08.550222  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 146722us
I20260709 12:29:08.550843  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:08.551455  5560 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:08.551815  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:08.552610  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:08.558341  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:08.563133  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:08.564072  5633 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:08.553273  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:08.565132  6275 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:08.516043 16159 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:08.569274  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:08.549938 15914 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:08.589637  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:08.599066  6275 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:08.579200  5628 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:29:08.604987  5628 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:29:08.605497  5560 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:29:08.574139  5633 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:08.606990  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:08.608047  5633 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:29:08.601219  6271 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:08.607115  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:08.554579 15690 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.541s	user 0.006s	sys 0.006s
I20260709 12:29:08.609830  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:08.613016  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:08.576495 15956 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.959s	user 0.011s	sys 0.029s
I20260709 12:29:08.564462 16134 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:08.566290 16101 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:08.618999  5634 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:29:08.628192  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:08.619438  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:08.544173 15735 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:08.633971  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:08.634485  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 2072) took 1518 ms. Trace:
I20260709 12:29:08.635164  5553 rpcz_store.cc:276] 0709 12:29:07.115633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:07.238287 (+122654us) service_pool.cc:224] Handling call
0709 12:29:08.634204 (+1395917us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b54001bd854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:08.634405 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47975680}
I20260709 12:29:08.576879 16161 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:08.576738 15947 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.639076  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:08.631922  5633 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:29:08.623098 15690 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:29:08.623656 15956 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.009s	user 0.012s	sys 0.030s
I20260709 12:29:08.661466 16107 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:08.663208  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:08.666181  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:08.669860  5634 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:29:08.658222 16158 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:08.643045 16027 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:08.662191 16107 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.668886 16161 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.609131 15998 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:08.629568 16048 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.696s	user 0.015s	sys 0.000s
I20260709 12:29:08.692698  5627 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:08.693070  5636 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:08.692180  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:08.695181  5555 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:08.684509 16118 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:08.658502 16033 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.658645 15682 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.698746  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:08.708201  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:08.658872 16134 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:08.674587 16158 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:08.711576  5556 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: []
I20260709 12:29:08.680696 16143 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:08.662429 15914 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.052s
I20260709 12:29:08.724967 15791 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.728266 16126 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:08.729229  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:08.730017  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:08.730755  5553 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:08.731580  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:08.732882 16106 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.612s	user 0.012s	sys 0.000s
I20260709 12:29:08.733366 16106 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:08.733978  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:08.734941 15976 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:08.735363 15976 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:08.735733 15976 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:08.735872 16137 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.737285  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:08.738205 16137 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:29:08.741106  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3907) took 1813 ms. Trace:
I20260709 12:29:08.740909 16134 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.741499  5559 rpcz_store.cc:276] 0709 12:29:06.927867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:07.359545 (+431678us) service_pool.cc:224] Handling call
0709 12:29:07.642969 (+283424us) spinlock_profiling.cc:243] Waited 9.94 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:29:07.643179 (+   210us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:07.994988 (+351809us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:08.720296 (+725308us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b540044f054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:08.741047 (+ 20751us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":831017216}
W20260709 12:29:08.743176 16105 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.696s	user 0.012s	sys 0.001s
I20260709 12:29:08.743624 16105 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.745774 16105 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:08.674952 16027 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:08.747089  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:08.747828  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3553) took 1512 ms. Trace:
I20260709 12:29:08.748184  5632 rpcz_store.cc:276] 0709 12:29:07.235412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:07.406815 (+171403us) service_pool.cc:224] Handling call
0709 12:29:07.407903 (+  1088us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:07.937354 (+529451us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:08.747605 (+810251us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b54001dd8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:08.747770 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":693214592}
I20260709 12:29:08.714259  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:08.674234 15947 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.751540  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:08.753031 15947 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:08.719182 16158 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:08.730578  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:08.754521 15989 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.754616  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:08.755281  5702 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:08.749625  5555 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:08.749799  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:08.757220  6272 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:08.710031  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:08.610472 16159 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.750676  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:08.753362 15730 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:08.767177  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:08.708082 16065 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.660593 16101 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:08.755168 16073 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.668s	user 0.011s	sys 0.000s
I20260709 12:29:08.768932  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:08.770408  5631 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3766) took 1495 ms (client timeout 1599 ms). Trace:
I20260709 12:29:08.761932  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:08.768837  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:08.758184  5704 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:08.773537  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:08.783215  6274 raft_consensus.cc:2368] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29.
I20260709 12:29:08.774912  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:08.775225  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:08.777134  6280 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
W20260709 12:29:08.784420  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:08.782105  5631 rpcz_store.cc:269] 0709 12:29:07.275390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:07.284263 (+  8873us) service_pool.cc:224] Handling call
0709 12:29:08.770197 (+1485934us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b54000c1a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:08.770335 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":734250752}
I20260709 12:29:08.786005  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:08.789688  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:08.793583  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:08.793792  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:08.794349  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:08.794360  6280 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:08.791960  5702 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:08.755082 15989 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.695477 16048 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.710595 16128 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.798108  5553 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:29:08.695302 15998 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.869563  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:08.872115  5627 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:08.872637  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:08.765522 15818 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:08.770470 16163 log.cc:927] Time spent T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:29:08.759287 16064 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.870260  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:08.873749  6278 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34. Ops: [33.22-33.22]
I20260709 12:29:08.882195  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:08.802616  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:08.790359  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:08.807310  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3535) took 1094 ms. Trace:
I20260709 12:29:08.767858 15965 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:08.875550  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:08.875901  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1827) took 1457 ms. Trace:
I20260709 12:29:08.883107  5553 rpcz_store.cc:276] 0709 12:29:07.713309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.769391 (+1056082us) service_pool.cc:224] Handling call
0709 12:29:08.797715 (+ 28324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:08.806978 (+  9263us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:08.807236 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:08.875936  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:08.877893  5704 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:08.870079 16107 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:08.800599 16108 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:08.716269 15682 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.905308  6276 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 28: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:08.905966  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:08.906289  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:08.879726  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:08.808331  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:08.907450  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 99239.9us
I20260709 12:29:08.778600 16035 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:08.904883  5708 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:08.713472 16033 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.797559 15947 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:08.916956 15947 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.919200 15947 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.922281 15989 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:08.923105 15989 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:08.923542 15989 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.880637 15818 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.926687 15682 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:08.929100 16048 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:08.930326 16026 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.539s	user 0.009s	sys 0.000s
I20260709 12:29:08.931133 16159 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.931836 16159 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:08.933092 16058 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.509s	user 0.010s	sys 0.000s
I20260709 12:29:08.802448 15932 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:08.807001 16143 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:08.909643  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:08.799539 15730 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.791446 16073 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.789217 16074 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:08.906636  6278 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
W20260709 12:29:08.952975  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 162683us
I20260709 12:29:08.953433  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:08.956992  6278 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:29:08.881664 16172 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:08.913429  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:08.882992  6280 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:08.914497 15998 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.883646  6279 rpcz_store.cc:276] 0709 12:29:07.418383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:07.418534 (+   151us) service_pool.cc:224] Handling call
0709 12:29:07.418866 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:07.650046 (+231180us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:08.875662 (+1225616us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b54002e2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:08.875848 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":486345600}
I20260709 12:29:08.960748  5553 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:08.916388 16033 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:08.909904  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:08.914345 15930 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:08.978413 16009 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:08.979035 16009 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:08.908754  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:08.981107  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72427.1us
I20260709 12:29:08.962167  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:08.871308 16153 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:08.941169 15932 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:08.924041 15989 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:08.933166 15789 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.945806  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:08.799918 15791 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:29:08.950392  6272 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:08.955106  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:08.999612  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44561.8us
I20260709 12:29:08.951347  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:08.760816 16067 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.124s
W20260709 12:29:08.976992 16165 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.206s	user 0.001s	sys 0.000s
W20260709 12:29:08.989893 16059 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.727s	user 0.015s	sys 0.000s
I20260709 12:29:08.747156 16027 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.983229  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:09.015249  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3643) took 1537 ms. Trace:
I20260709 12:29:08.942580 16159 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.951170 16143 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.907552  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:08.948786 15682 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:09.018903 15538 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:29:09.041905 16143 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:09.042205 15538 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:09.042836 15538 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:08.976475 16065 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.041s
W20260709 12:29:09.043269 16151 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.507s	user 0.011s	sys 0.004s
I20260709 12:29:08.994952 16153 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:09.046501 16153 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:09.047857  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 140369us
I20260709 12:29:08.985404 16009 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:09.049549 16065 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:29:09.049975 16065 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:09.050315 16065 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:09.051685 16158 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:08.990880  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:08.994338  5552 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:08.949962 15818 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:09.088745 15989 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:08.912719 16035 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:08.997452  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51787.4us
I20260709 12:29:09.091297 16035 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:08.999667  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:08.916258 15976 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:09.093266  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 93713.1us
I20260709 12:29:09.095314 15976 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:09.095527 16143 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:09.000754  6276 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:08.905042 16064 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:09.096681  6271 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:09.097338  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3923) took 1994 ms. Trace:
I20260709 12:29:08.976449 16058 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.015s
I20260709 12:29:09.097736  5556 rpcz_store.cc:276] 0709 12:29:07.102627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:07.493254 (+390627us) service_pool.cc:224] Handling call
0709 12:29:07.513568 (+ 20314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:08.716327 (+1202759us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:09.097054 (+380727us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400026cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:09.097282 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2754571136}
I20260709 12:29:08.912246 15965 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.099502  6277 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:08.940474 16134 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:09.014768  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:09.013274 16021 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.872s	user 0.012s	sys 0.001s
I20260709 12:29:09.016819 16059 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.022377 15791 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:29:09.027673 16073 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:09.081149 15707 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.643s	user 0.012s	sys 0.000s
I20260709 12:29:09.081698 16118 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:29:09.028497 16101 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:08.954712 15730 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.106146 15989 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:09.115542  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:09.116039 16127 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.660s	user 0.013s	sys 0.000s
I20260709 12:29:09.115144 15707 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.119508 15707 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:09.001483 15930 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:29:09.120643  5635 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:09.101944 16134 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:09.011399 16113 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:09.122151 16113 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.123275 16134 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:09.077872 16074 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.047s
I20260709 12:29:09.124122 16021 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.126670 15965 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:09.127310 16021 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:09.127801  5630 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 30: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:29:09.029325 16138 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.893s	user 0.011s	sys 0.002s
W20260709 12:29:09.032845 16129 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.510s	user 0.011s	sys 0.000s
I20260709 12:29:09.034444 15789 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:09.034857 16159 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:09.017858 16160 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:29:09.133263 16160 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:09.133227 16138 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:09.134658 16160 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.135787 16138 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:09.053723  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:09.136685  5636 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:29:09.137053 15690 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:29:09.097949 16064 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:09.138724  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:09.055755  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2070) took 1941 ms. Trace:
I20260709 12:29:09.140551 16161 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:08.990749  5553 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:29:08.987524 16172 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.075129 16176 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:09.096457 16143 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:09.099421  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:09.150209  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 149479us
I20260709 12:29:08.992069 15932 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:09.015740  5706 rpcz_store.cc:276] 0709 12:29:07.477556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.012635 (+535079us) service_pool.cc:224] Handling call
0709 12:29:09.015017 (+1002382us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b5400067054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:09.015174 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":853884288}
I20260709 12:29:09.153769  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:09.153862  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2088) took 1929 ms. Trace:
I20260709 12:29:09.102160  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:09.076687  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:09.104508  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:09.080812  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:09.108223  6271 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:09.110333  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:09.117295 15730 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:09.001492 16137 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:09.002862 16106 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:09.128458 16127 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:09.131578  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:09.188915  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 57761.9us
W20260709 12:29:09.158572  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 56552.6us
I20260709 12:29:09.185912  6271 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:09.081693 16158 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:09.139936 16059 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:09.142503  5560 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 28: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:09.143628  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:09.151044 16026 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:09.140323  5554 rpcz_store.cc:276] 0709 12:29:07.114900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:07.530854 (+415954us) service_pool.cc:224] Handling call
0709 12:29:07.531156 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:07.531368 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:29:07.712213 (+180845us) log.cc:844] Serialized 14 byte log entry
0709 12:29:09.055008 (+1342795us) spinlock_profiling.cc:243] Waited 5.3 ms on lock 0x7b1800444f40. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:29:09.055231 (+   223us) log.cc:844] Serialized 14 byte log entry
0709 12:29:09.055418 (+   187us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:09.055469 (+    51us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:29:09.055512 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:09.055588 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:09.055696 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1151675,"prepare.queue_time_us":1532831,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":93,"raft.queue_time_us":33874,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":8291,"spinlock_wait_cycles":12183424,"thread_start_us":366322,"threads_started":2,"wal-append.queue_time_us":336632}
I20260709 12:29:09.196594  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2079) took 1527 ms. Trace:
I20260709 12:29:09.197232  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:09.101281  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:09.199323  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2078) took 1530 ms. Trace:
I20260709 12:29:09.177676  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:09.199719  5561 rpcz_store.cc:276] 0709 12:29:07.669138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.055460 (+386322us) service_pool.cc:224] Handling call
0709 12:29:08.055850 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:08.056100 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:29:09.056169 (+1000069us) log.cc:844] Serialized 14 byte log entry
0709 12:29:09.198806 (+142637us) log.cc:844] Serialized 14 byte log entry
0709 12:29:09.199028 (+   222us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:09.199073 (+    45us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:09.199121 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:09.199175 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:09.199270 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1000211,"prepare.queue_time_us":1000387,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"thread_start_us":142028,"threads_started":1}
I20260709 12:29:09.163506  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:09.185513  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.187858  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.150967 16160 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:09.151364  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:09.200268  5553 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:09.182469  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.203600  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:09.209825  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:09.184554 15930 raft_consensus.cc:2764] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election decision vote started in defunct term 29: won
I20260709 12:29:09.131121 16151 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.060s
I20260709 12:29:09.212450  5632 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:09.155782 16133 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:09.176959 16176 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:09.204928  6271 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:09.228885  6271 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:09.155791 15976 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.056s
I20260709 12:29:09.205683  5629 rpcz_store.cc:276] 0709 12:29:07.224376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:07.224605 (+   229us) service_pool.cc:224] Handling call
0709 12:29:07.225144 (+   539us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:07.225534 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:29:07.291015 (+ 65481us) log.cc:844] Serialized 14 byte log entry
0709 12:29:08.876328 (+1585313us) spinlock_profiling.cc:243] Waited 5.03 ms on lock 0x7b18004f7380. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287
0709 12:29:08.998227 (+121899us) log.cc:844] Serialized 14 byte log entry
0709 12:29:09.153441 (+155214us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:09.153498 (+    57us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:29:09.153546 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:09.153628 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:09.153785 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87852,"prepare.queue_time_us":167268,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":106,"raft.queue_time_us":104235,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":13877,"spinlock_wait_cycles":11625856,"thread_start_us":561398,"threads_started":4,"wal-append.queue_time_us":271343,"wal-append.run_cpu_time_us":4247,"wal-append.run_wall_time_us":1368042}
I20260709 12:29:09.231801  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.225066 15965 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:09.229877 15707 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:09.233011 15965 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.233182 15707 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.220329  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:09.234778  6279 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 34: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:09.235298 15965 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:09.210893  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:09.235530 15707 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:09.193907 16129 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:09.228840  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:09.236579 16129 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:09.236842  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.237071 16129 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:09.237495  5629 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:09.238217 15892 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:09.238375  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.239176  5629 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 25: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:09.239991  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.240542  5629 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:09.241374  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:09.197182  5557 rpcz_store.cc:276] 0709 12:29:07.669431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.108988 (+439557us) service_pool.cc:224] Handling call
0709 12:29:08.118403 (+  9415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:08.118633 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:29:09.057386 (+938753us) log.cc:844] Serialized 14 byte log entry
0709 12:29:09.195157 (+137771us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:09.195801 (+   644us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:09.195889 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:09.196009 (+   120us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:09.196425 (+   416us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":938235,"prepare.queue_time_us":938827,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":1551,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"thread_start_us":135994,"threads_started":1}
I20260709 12:29:09.241902  5629 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:09.241899 16172 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:09.195919 16106 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:09.255084  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:09.236189  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 84919.5us
I20260709 12:29:09.206015 16064 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:09.260110  5629 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:29:09.262455  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.265367  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:09.265712  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:09.267529  5629 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:09.230715 16113 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:09.230942 16026 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:09.205017 16137 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.280133  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.280328  5629 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:09.280871  5634 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:09.281105  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:09.281417  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.281749  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:09.260728 15930 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:29:09.282876  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 72125.7us
I20260709 12:29:09.231251 16105 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.049s
I20260709 12:29:09.263248  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.288790  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.272079 16021 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:09.289894  5561 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:09.274116 16172 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:29:09.293453  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:09.293819  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:09.277729  5553 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:09.342954  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.343879  6279 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:09.331418  5526 leader_election.cc:400] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29.
I20260709 12:29:09.345544  5526 leader_election.cc:403] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:29:09.304960  5704 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:09.344317  5553 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:09.297868  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:09.350674  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52890.2us
I20260709 12:29:09.352114  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:09.352933  6276 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:29:09.302940  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.271677 16059 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 12:29:09.270408 16064 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:09.355899  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:09.356599  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 213082us
I20260709 12:29:09.290489 16182 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.075s
I20260709 12:29:09.360910 15989 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:29:09.358954 16153 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:09.354954  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:09.416878  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 62011us
I20260709 12:29:09.417846  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:09.365134 16137 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:09.285981 15892 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.068s
I20260709 12:29:09.374857  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:09.377679 16181 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:29:09.378700 16021 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:09.343204 16027 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.052s
W20260709 12:29:09.426826  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 52078.5us
I20260709 12:29:09.403124 16159 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:09.405572  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 207918us
I20260709 12:29:09.409586  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:09.452056  5524 leader_election.cc:336] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.640s (SENT)
I20260709 12:29:09.405656 15903 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29.
I20260709 12:29:09.458587  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:29:09.410272  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:09.408670  5553 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:09.295612 16161 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:09.410643  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:09.254191 16143 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:09.421365  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:09.467101  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45827.9us
I20260709 12:29:09.288457 16178 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:09.347656 16176 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.340543 15707 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.080s
I20260709 12:29:09.468143  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:09.480638  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:09.445811 16134 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:29:09.419404 16153 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:09.423532 15789 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:09.448243  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:09.446717 16182 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:29:09.484269  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 233590us
I20260709 12:29:09.542363  5553 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:09.427091 15965 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:09.557520  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:09.555936  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.556949  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.456889 16059 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:09.558832  5561 raft_consensus.cc:2410] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 31 because replica has last-logged OpId of term: 29 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:29:09.354249 16106 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:09.560201  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:09.427143 16172 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:09.486364  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.470587 16178 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:09.478615 16137 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:09.554584 16151 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:09.568849  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:09.488061 16153 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:09.583698  5632 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:09.569414  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.571237  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.590924  5627 raft_consensus.cc:2410] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 31 because replica has last-logged OpId of term: 29 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:29:09.591850  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.593030  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.593611  5627 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:29:09.565989 16059 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.595326  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.595969 16064 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:09.594398  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:09.477650 16161 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.453058 16021 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.609175  6280 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:29:09.475054 16014 raft_consensus.cc:2764] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election decision vote started in defunct term 32: won
I20260709 12:29:09.554047 16048 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:09.579135  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:09.599236  5627 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:09.599207  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:09.566334 16177 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:29:09.600554  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:09.602636  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:09.604275  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.604383  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:09.625502  5557 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:09.554786 15789 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:09.468327 16159 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:09.561707  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 113558us
I20260709 12:29:09.579003 16178 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:09.607594  6271 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:29:09.576315  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:09.565565 16138 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:09.567090 15903 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:09.605665 16059 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:09.626452 16035 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.523s	user 0.005s	sys 0.009s
W20260709 12:29:09.618444  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:09.619457  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:09.624871  5632 raft_consensus.cc:2410] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 30 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:29:09.567174 16172 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:09.486430 16127 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:09.629642  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:09.631361  5553 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 33: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:09.605808  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.632838  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:09.667093  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:09.582664 16151 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:09.616174  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:09.673142  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.674067  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.675129  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:09.642171  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 65929us
I20260709 12:29:09.618207  5702 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:09.565177 15965 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.654655  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:09.659716  5710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3633) took 2187 ms (client timeout 1640 ms). Trace:
W20260709 12:29:09.662956  5701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3632) took 2191 ms (client timeout 1553 ms). Trace:
I20260709 12:29:09.634474 15789 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:29:09.669791  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:09.653048 15921 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:09.614583 16137 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:09.632438 16048 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.648550 15903 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:09.712452  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:09.713104  5701 rpcz_store.cc:269] 0709 12:29:07.471789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:07.509002 (+ 37213us) service_pool.cc:224] Handling call
0709 12:29:09.662741 (+2153739us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b54002f5654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:09.662889 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1232113536}
W20260709 12:29:09.713799  5710 rpcz_store.cc:269] 0709 12:29:07.472006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:07.645116 (+173110us) service_pool.cc:224] Handling call
0709 12:29:09.659473 (+2014357us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b5400452ed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:09.659637 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1564701568}
I20260709 12:29:09.669800  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:09.768067  5632 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 31: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:09.732189  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:09.715020  5634 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:09.676909  5601 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:09.764523  5627 raft_consensus.cc:2410] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 33 because replica has last-logged OpId of term: 32 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 13.
I20260709 12:29:09.769418  5702 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:09.770529  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.769423  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.771844  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.773545  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.775230  5632 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
W20260709 12:29:09.676184 15538 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.633s	user 0.011s	sys 0.006s
I20260709 12:29:09.679382 15892 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:09.780184  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:09.667991 16183 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
I20260709 12:29:09.783715  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:09.783881  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:09.671288 16033 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:09.676676 16151 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:09.671205 16113 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:09.651738 16138 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:09.794895  5710 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:09.711110 16127 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:09.668778 16059 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:09.799273 15792 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:09.664511 16035 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:09.802196 15932 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.618s	user 0.010s	sys 0.001s
W20260709 12:29:09.803170 16065 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.752s	user 0.014s	sys 0.001s
I20260709 12:29:09.778574 15903 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:09.790918 16134 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:09.718217 16129 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:09.804519 15929 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:09.804514 16033 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.817281 16065 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.815310 16059 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:09.788797 16172 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:09.848938 16159 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:09.852511 16172 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:09.852476 16134 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:09.856742 16129 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:09.858170 16161 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:09.870204 16160 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.655s	user 0.013s	sys 0.000s
I20260709 12:29:09.874030 16178 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:09.909010  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:09.910878  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.914041  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.678129  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.771880  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:09.793236 15538 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:09.915902  5553 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 38 because replica has last-logged OpId of term: 36 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 20.
I20260709 12:29:09.969576  5627 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:29:09.936189  5631 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 15. Preceding OpId from leader: term: 32 index: 16. (index mismatch)
W20260709 12:29:09.947002  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:09.970502  5636 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29. Ops: [27.17-28.18]
I20260709 12:29:09.937672  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:09.921622  5558 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
W20260709 12:29:09.934175  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4040) took 1847 ms (client timeout 1643 ms). Trace:
I20260709 12:29:10.000029 16058 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:09.827157 16137 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:10.000783  5707 rpcz_store.cc:269] 0709 12:29:08.086832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.181159 (+ 94327us) service_pool.cc:224] Handling call
0709 12:29:09.933632 (+1752473us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54004529d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:09.933914 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":345359616}
I20260709 12:29:10.001592  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:10.001327 16058 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:10.002529  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:10.002774  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:09.963513  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:09.979327 15947 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:09.971621 15538 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.004523 15947 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:10.005367 15538 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:10.005769  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:10.005970 15538 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:10.006458 16065 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:10.006650 15538 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:09.951479  5557 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:10.007664 16160 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:09.816740 15932 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.008960 16058 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:09.829694 15929 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.011329 16033 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:09.932446  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:09.950424  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:09.972123  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:09.976097  5560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3926) took 1570 ms (client timeout 2048 ms). Trace:
I20260709 12:29:09.962080  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:10.012861  5556 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 23. Current term is 24. Ops: []
W20260709 12:29:09.931440  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 151241us
W20260709 12:29:09.924614  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 247394us
I20260709 12:29:10.000154 16158 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:09.841672 15965 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:10.036928 16158 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.038110  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:10.038571  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4046) took 1653 ms. Trace:
I20260709 12:29:10.038967  5708 rpcz_store.cc:276] 0709 12:29:08.385708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.405206 (+ 19498us) service_pool.cc:224] Handling call
0709 12:29:10.038167 (+1632961us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x7b540026fa54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:10.038400 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1084901248}
I20260709 12:29:10.039507  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:10.039865 16158 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:10.040035 16059 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:10.031589  5560 rpcz_store.cc:269] 0709 12:29:08.405403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.606708 (+201305us) service_pool.cc:224] Handling call
0709 12:29:09.975686 (+1368978us) spinlock_profiling.cc:243] Waited 534 ms on lock 0x7b54001bdd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:09.975912 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1228893440}
I20260709 12:29:10.040705 16176 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:29:10.041157  5630 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3560) took 1999 ms (client timeout 2393 ms). Trace:
I20260709 12:29:10.041322 16176 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:10.041855 16176 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:10.042291  5560 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:10.043028  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3932) took 1541 ms. Trace:
I20260709 12:29:10.043392  5560 rpcz_store.cc:276] 0709 12:29:08.501517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.041483 (+1539966us) service_pool.cc:224] Handling call
0709 12:29:10.041994 (+   511us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.042772 (+   778us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.042952 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.044397  5601 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:10.018230 16160 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:09.918018  5629 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 37.
I20260709 12:29:10.045457 16160 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:10.045961 16127 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:29:09.932375  5632 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:10.048745 15974 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:10.049986  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3777) took 1474 ms. Trace:
I20260709 12:29:09.967603  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:09.915861  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:10.033340  5561 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:10.034432 16058 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:10.037415  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3775) took 1464 ms. Trace:
I20260709 12:29:09.876564 16178 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.057042 16027 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:29:10.041648  5630 rpcz_store.cc:269] 0709 12:29:08.041813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.041940 (+   127us) service_pool.cc:224] Handling call
0709 12:29:10.040721 (+1998781us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540018bd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:10.040960 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2477711104}
W20260709 12:29:10.068761  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 153420us
I20260709 12:29:10.069490 16058 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.069998 16178 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:10.070537 16178 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:10.070340  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:10.005908 15947 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.052037  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3930) took 1552 ms. Trace:
I20260709 12:29:10.026515  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:10.056941  5703 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:10.066602  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:10.040895 16059 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:29:10.052699  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 91042.1us
I20260709 12:29:10.074519  5558 rpcz_store.cc:276] 0709 12:29:08.500045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.033073 (+1533028us) service_pool.cc:224] Handling call
0709 12:29:10.033616 (+   543us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.051609 (+ 17993us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:10.051646 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:10.051716 (+    70us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:10.051757 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.051818 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:10.051950 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.082690 15947 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.000630 16021 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:10.068668  5635 rpcz_store.cc:276] 0709 12:29:08.572983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.574832 (+  1849us) service_pool.cc:224] Handling call
0709 12:29:08.575210 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:09.121117 (+545907us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.036815 (+915698us) spinlock_profiling.cc:243] Waited 545 ms on lock 0x7b540018d654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:10.037191 (+   376us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1253890816}
I20260709 12:29:10.070940 16178 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:10.050493 15929 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:10.022241 16048 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:10.052131 16107 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:10.028527 16138 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:10.109165  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:10.076956 16059 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:10.108911  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:10.117959 15791 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:10.093220 16027 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:10.116710 16059 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:10.132931 16133 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:10.135385  5557 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:29:10.136843 16035 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.137624 16009 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:10.138440  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:10.150095  5707 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:10.050423  5628 rpcz_store.cc:276] 0709 12:29:08.575422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.606112 (+ 30690us) service_pool.cc:224] Handling call
0709 12:29:08.606496 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:08.606709 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:29:08.756850 (+150141us) log.cc:844] Serialized 14 byte log entry
0709 12:29:09.854328 (+1097478us) spinlock_profiling.cc:243] Waited 4.79 ms on lock 0x7b18004f7800. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:29:10.048755 (+194427us) spinlock_profiling.cc:243] Waited 9.2 ms on lock 0x7b18004f7800. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a
0709 12:29:10.049165 (+   410us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:10.049250 (+    85us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:10.049347 (+    97us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.049456 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:10.049811 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":202093,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":3359,"raft.queue_time_us":868,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":159,"spinlock_wait_cycles":32190720,"thread_start_us":239388,"threads_started":2,"wal-append.queue_time_us":112226}
I20260709 12:29:10.141578 16058 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.142514  5710 raft_consensus.cc:2410] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 25 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
W20260709 12:29:10.144755  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 127162us
I20260709 12:29:10.097834 16127 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.034s
I20260709 12:29:10.144538  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:10.151346  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2076) took 2373 ms. Trace:
I20260709 12:29:10.152175  5560 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: [32.14-32.14]
I20260709 12:29:10.152756  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3941) took 1212 ms. Trace:
I20260709 12:29:10.156026  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4058) took 1008 ms. Trace:
I20260709 12:29:10.155481  5557 rpcz_store.cc:276] 0709 12:29:08.935489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.116526 (+1181037us) service_pool.cc:224] Handling call
0709 12:29:10.134917 (+ 18391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.147769 (+ 12852us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.148027 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.156610  5707 rpcz_store.cc:276] 0709 12:29:09.147933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.097434 (+949501us) service_pool.cc:224] Handling call
0709 12:29:10.155943 (+ 58509us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.154493  5552 rpcz_store.cc:276] 0709 12:29:07.662881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.048006 (+385125us) service_pool.cc:224] Handling call
0709 12:29:08.048291 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:09.088574 (+1040283us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.035874 (+947300us) spinlock_profiling.cc:243] Waited 941 ms on lock 0x7b54001c5cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:10.036267 (+   393us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2165068672}
I20260709 12:29:10.144495 15932 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:10.132514 15791 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:10.163990  5635 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:29:10.162899  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3933) took 1656 ms. Trace:
I20260709 12:29:10.166641  5560 rpcz_store.cc:276] 0709 12:29:08.501921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.095063 (+1593142us) service_pool.cc:224] Handling call
0709 12:29:10.146550 (+ 51487us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:10.156208 (+  9658us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.158049 (+  1841us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.168342  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:10.169222  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2087) took 1407 ms. Trace:
I20260709 12:29:10.169698  5561 rpcz_store.cc:276] 0709 12:29:08.762088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.105203 (+1343115us) service_pool.cc:224] Handling call
0709 12:29:10.168438 (+ 63235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.168828 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:10.168852 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:10.168908 (+    56us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:29:10.168983 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.169048 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:10.169161 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.164641  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:10.176913 16129 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:10.165251 15791 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:10.177757 16129 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.180895 16134 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:10.181567 16134 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:10.182842 16134 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.193707  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:10.194797  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:10.196707  5705 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:10.197810 16162 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:10.198598  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 387302us
I20260709 12:29:10.198448 16162 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.200321  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:10.202317 16048 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:10.203485  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4059) took 1051 ms. Trace:
I20260709 12:29:10.203954  5710 rpcz_store.cc:276] 0709 12:29:09.151839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:09.783543 (+631704us) service_pool.cc:224] Handling call
0709 12:29:10.163723 (+380180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.205154  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:10.206116  5710 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:10.207250  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:10.209285  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:10.210386  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:10.211776 16048 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:10.212630 16048 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.214622 15789 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.214955 16048 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.217084  6272 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:29:10.217167 16027 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:10.217941 16026 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.005s	sys 0.017s
W20260709 12:29:10.218812 16026 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.923s	user 0.005s	sys 0.018s
I20260709 12:29:10.220038 16026 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:10.196815  5707 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:10.234355 16035 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:29:10.236666  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 76665.4us
I20260709 12:29:10.243763  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:10.249277  5559 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: [30.18-32.19]
I20260709 12:29:10.236166 16035 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:10.250214  5628 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:10.250617 16035 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:10.253402  5635 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:29:10.254995  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:10.256835  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3537) took 2260 ms. Trace:
I20260709 12:29:10.257251  5556 rpcz_store.cc:276] 0709 12:29:07.996147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:09.927627 (+1931480us) service_pool.cc:224] Handling call
0709 12:29:09.934952 (+  7325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.044871 (+109919us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.256597 (+211726us) spinlock_profiling.cc:243] Waited 77.4 ms on lock 0x7b54000ebfd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:10.256757 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":178019584}
I20260709 12:29:10.264739 16021 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:10.266004 16021 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.268708 16113 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:10.269723 16113 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.270480 16118 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:10.271198 16118 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.273759 15707 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:29:10.275233 16106 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:10.275689 16106 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:10.276269 16118 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.276932 16106 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.279266 16106 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:10.283901  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:10.284698  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:10.241704  6271 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
W20260709 12:29:10.271622  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:10.250216  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3536) took 2535 ms. Trace:
I20260709 12:29:10.287357  5559 rpcz_store.cc:276] 0709 12:29:07.714713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.248171 (+2533458us) service_pool.cc:224] Handling call
0709 12:29:10.248773 (+   602us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:10.249878 (+  1105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.250138 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.308450  5710 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
W20260709 12:29:10.309703  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:10.294114 16127 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:10.317557  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:10.321367  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:10.286413 15929 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.027s
I20260709 12:29:10.339497 16127 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.349551 15932 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:10.350003 15932 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:10.351099 15932 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.321470 16113 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.333923  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:10.333163  5559 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
W20260709 12:29:10.337550 15903 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.514s	user 0.007s	sys 0.010s
I20260709 12:29:10.338840 16021 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:10.311512 15789 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:10.356644  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:10.363564  6279 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 25.
W20260709 12:29:10.367866  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54640.5us
I20260709 12:29:10.356644  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:10.359349  5559 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
W20260709 12:29:10.354346  5704 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3660) took 1608 ms (client timeout 2095 ms). Trace:
W20260709 12:29:10.370694  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54577.7us
W20260709 12:29:10.404780  5704 rpcz_store.cc:269] 0709 12:29:08.746110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.985993 (+239883us) service_pool.cc:224] Handling call
0709 12:29:10.353920 (+1367927us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b54001991d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:10.354127 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":528009088}
I20260709 12:29:10.406840  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2081) took 2185 ms. Trace:
W20260709 12:29:10.412482  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:10.352638 15932 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:10.413621  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:10.414446  5710 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:10.352298 16127 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.366616 16133 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:10.421586  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4052) took 1512 ms. Trace:
I20260709 12:29:10.419478  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:10.422001  5710 rpcz_store.cc:276] 0709 12:29:08.909452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.323900 (+1414448us) service_pool.cc:224] Handling call
0709 12:29:10.421524 (+ 97624us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.424042  5559 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 37 index: 22. (index mismatch)
I20260709 12:29:10.425271  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:10.425973  5559 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:10.427044  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:10.427769  5559 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:10.433117  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4050) took 1882 ms. Trace:
W20260709 12:29:10.433351  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:10.433539  5703 rpcz_store.cc:276] 0709 12:29:08.550680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.001092 (+1450412us) service_pool.cc:224] Handling call
0709 12:29:10.432982 (+431890us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b5400451ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:10.433072 (+    90us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26833280}
I20260709 12:29:10.422683  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:10.442257  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:10.427680 15538 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.307994 16026 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.447525  5703 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:10.446792  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:10.448889  5710 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:10.434063 16133 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:10.410175  5555 rpcz_store.cc:276] 0709 12:29:07.979669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:09.971660 (+1991991us) service_pool.cc:224] Handling call
0709 12:29:09.972256 (+   596us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:09.972863 (+   607us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:29:10.030580 (+ 57717us) log.cc:844] Serialized 14 byte log entry
0709 12:29:10.158804 (+128224us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:10.158859 (+    55us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:29:10.161373 (+  2514us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.161483 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:10.164773 (+  3290us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":132453,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":1132,"raft.queue_time_us":4765,"raft.run_cpu_time_us":359,"raft.run_wall_time_us":81152,"thread_start_us":65143,"threads_started":2,"wal-append.queue_time_us":159437}
I20260709 12:29:10.445678  6275 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:10.453385  5709 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29. Ops: []
W20260709 12:29:10.453779 16204 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 12:29:10.404960 15903 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.457261  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1606) took 1667 ms. Trace:
I20260709 12:29:10.450352  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:10.457839  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 143818us
I20260709 12:29:10.458662 16129 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:05.050621  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:10.468159  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:10.451349 16176 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.480150  5707 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
W20260709 12:29:10.488821  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:10.470745 16210 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 12:29:10.432611 15791 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.484851  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3664) took 1373 ms. Trace:
W20260709 12:29:10.468628 16216 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.094s	user 0.002s	sys 0.003s
I20260709 12:29:10.488850  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:10.417578 16048 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.062s
I20260709 12:29:10.495814  5561 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:10.493469  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:10.496937  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:10.496047  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:10.495260  5707 rpcz_store.cc:276] 0709 12:29:09.111079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.193881 (+1082802us) service_pool.cc:224] Handling call
0709 12:29:10.484780 (+290899us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.497290  5556 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:10.475252  6273 rpcz_store.cc:276] 0709 12:29:08.789586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.792568 (+  2982us) service_pool.cc:224] Handling call
0709 12:29:08.870461 (+ 77893us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:08.934393 (+ 63932us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:08.934424 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:08.934966 (+   542us) log.cc:844] Serialized 52 byte log entry
0709 12:29:09.198910 (+263944us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:09.449731 (+250821us) log.cc:844] Serialized 14 byte log entry
0709 12:29:09.453142 (+  3411us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.430843 (+977701us) spinlock_profiling.cc:243] Waited 63.5 ms on lock 0x7b54006379d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:10.439387 (+  8544us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:10.450263 (+ 10876us) raft_consensus.cc:1685] finished
0709 12:29:10.456725 (+  6462us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:10.456996 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3041,"prepare.queue_time_us":364213,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":4387,"raft.queue_time_us":9387,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":19921,"spinlock_wait_cycles":146106752,"thread_start_us":438853,"threads_started":2,"wal-append.queue_time_us":325650}
I20260709 12:29:10.500288  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:10.501134  5704 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:10.468847 16127 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:10.504846  5627 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:10.505242 16127 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.507506 16127 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:10.507232 16035 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.494228  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:10.510975  5555 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 30: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:29:10.511616 16153 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.897s	user 0.002s	sys 0.017s
I20260709 12:29:10.511992  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:10.512248 16153 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.513707  5561 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:10.514504 16153 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.515604  5561 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
W20260709 12:29:10.516826 16151 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.705s	user 0.013s	sys 0.007s
I20260709 12:29:10.517627 16151 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.521075  5701 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:10.522250  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4060) took 1292 ms. Trace:
I20260709 12:29:10.522591  5701 rpcz_store.cc:276] 0709 12:29:09.230328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.254530 (+1024202us) service_pool.cc:224] Handling call
0709 12:29:10.522053 (+267523us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b54002d23d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:10.522190 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":609779072}
I20260709 12:29:10.527132 16176 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:10.527418  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:10.474375 16160 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.528599 16107 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:10.529172 16107 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:10.529491 16202 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:10.530128 16202 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.531720  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:10.532591 16127 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:29:10.532964  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3585) took 1179 ms. Trace:
I20260709 12:29:10.533059  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:10.533567  5631 rpcz_store.cc:276] 0709 12:29:09.353746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.071362 (+717616us) service_pool.cc:224] Handling call
0709 12:29:10.128435 (+ 57073us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:10.143475 (+ 15040us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:10.143514 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:10.144180 (+   666us) log.cc:844] Serialized 26 byte log entry
0709 12:29:10.320338 (+176158us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:10.320437 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.320543 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:10.532729 (+212186us) raft_consensus.cc:1685] finished
0709 12:29:10.532751 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:10.532883 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1475,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":8011,"thread_start_us":175605,"threads_started":1,"wal-append.queue_time_us":180171}
I20260709 12:29:10.537420 16058 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:29:10.538599 16160 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.532975 16127 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:10.541023 15921 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.778s	user 0.011s	sys 0.006s
I20260709 12:29:10.541194 16127 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:10.541658  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3655) took 2292 ms. Trace:
I20260709 12:29:10.541749 15921 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:10.534514 16021 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:29:10.543183  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:10.543591 16021 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:10.544013 16021 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:10.546299 16200 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.410s	user 0.004s	sys 0.000s
I20260709 12:29:10.548882  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2093) took 1906 ms. Trace:
W20260709 12:29:10.540675 16203 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.405s	user 0.003s	sys 0.000s
I20260709 12:29:10.549321  5633 rpcz_store.cc:276] 0709 12:29:08.642677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.660659 (+ 17982us) service_pool.cc:224] Handling call
0709 12:29:08.663666 (+  3007us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:08.664267 (+   601us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:08.664313 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:08.666353 (+  2040us) log.cc:844] Serialized 26 byte log entry
0709 12:29:09.946804 (+1280451us) spinlock_profiling.cc:243] Waited 5.75 ms on lock 0x7b18004f7aa0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:29:10.050997 (+104193us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:10.051098 (+   101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.051230 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:10.545530 (+494300us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:10.548420 (+  2890us) raft_consensus.cc:1685] finished
0709 12:29:10.548442 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:10.548699 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":115,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":18946,"spinlock_wait_cycles":36448384,"thread_start_us":38227,"threads_started":1,"wal-append.queue_time_us":1469107}
I20260709 12:29:10.551306  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3792) took 1231 ms. Trace:
I20260709 12:29:10.551800  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:10.552680  5708 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:29:10.495668 15682 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:10.553313  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4057) took 1421 ms. Trace:
I20260709 12:29:10.564230  5708 rpcz_store.cc:276] 0709 12:29:09.131719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.551467 (+1419748us) service_pool.cc:224] Handling call
0709 12:29:10.553258 (+  1791us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.566149  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:10.569305  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 103235us
I20260709 12:29:10.523904  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:10.570030 15903 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.470382  5452 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 23 to 26, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 26 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:10.570626  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3666) took 1016 ms. Trace:
I20260709 12:29:10.571040  5701 rpcz_store.cc:276] 0709 12:29:09.554607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.523538 (+968931us) service_pool.cc:224] Handling call
0709 12:29:10.570528 (+ 46990us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.518404  5560 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:29:10.573693  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:10.554455  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3629) took 3083 ms. Trace:
I20260709 12:29:10.574939 16118 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:29:10.580039  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 81217.2us
W20260709 12:29:10.575265  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 84807.3us
I20260709 12:29:10.542095  5705 rpcz_store.cc:276] 0709 12:29:08.248800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.342285 (+ 93485us) service_pool.cc:224] Handling call
0709 12:29:10.541442 (+2199157us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b54002e81d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:10.541588 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4117626240}
I20260709 12:29:10.572300  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:10.560843 15682 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.621454 16151 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:10.622231 15682 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:10.551671  5636 rpcz_store.cc:276] 0709 12:29:09.319674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.008658 (+688984us) service_pool.cc:224] Handling call
0709 12:29:10.009292 (+   634us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:10.009984 (+   692us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:10.010016 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:10.064702 (+ 54686us) log.cc:844] Serialized 52 byte log entry
0709 12:29:10.108653 (+ 43951us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:10.108704 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.108783 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:10.539508 (+430725us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:10.550984 (+ 11476us) raft_consensus.cc:1685] finished
0709 12:29:10.551003 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:10.551224 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":817,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":254,"spinlock_wait_cycles":23081344,"thread_start_us":43350,"threads_started":1,"wal-append.queue_time_us":69817}
I20260709 12:29:10.623443 15791 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.624250 16162 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.624923 16202 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:10.610003 16129 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:10.625751  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:10.625716 16129 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.628093 16129 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:10.628866 15791 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:10.629069 16129 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:10.629580 15791 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.630342 16035 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:10.628084 15735 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:29:10.632711 15735 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:10.632048 15791 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:10.633705 15947 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:10.574627 16153 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:10.575397  5709 rpcz_store.cc:276] 0709 12:29:07.471070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:08.369274 (+898204us) service_pool.cc:224] Handling call
0709 12:29:09.934478 (+1565204us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.487091 (+552613us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.553297 (+ 66206us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x7b54004529d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:10.554174 (+   877us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54004529d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:10.554338 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4683987584}
I20260709 12:29:10.634770 15947 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.636238  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:10.636781  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:10.637001  5709 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 27.
I20260709 12:29:10.638139  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:10.638963  5709 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 32 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 19.
I20260709 12:29:10.639334  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 1818) took 1406 ms. Trace:
W20260709 12:29:10.639565  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:10.510033 16059 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.591552  5452 catalog_manager.cc:5697] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 19 to 22. New cstate: current_term: 22 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:10.632745  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4061) took 1377 ms. Trace:
I20260709 12:29:10.641803  5701 rpcz_store.cc:276] 0709 12:29:09.255803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.571875 (+1316072us) service_pool.cc:224] Handling call
0709 12:29:10.632683 (+ 60808us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.642303 16059 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:10.643612  5701 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:10.644724 15538 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:10.645210  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3648) took 2882 ms. Trace:
I20260709 12:29:10.625196  5710 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
W20260709 12:29:10.593259  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 100097us
I20260709 12:29:10.645745  5701 rpcz_store.cc:276] 0709 12:29:07.762439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.642832 (+2880393us) service_pool.cc:224] Handling call
0709 12:29:10.643338 (+   506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.644845 (+  1507us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.645117 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:29:10.640153  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:10.639801  5709 rpcz_store.cc:276] 0709 12:29:09.233154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.637805 (+1404651us) service_pool.cc:224] Handling call
0709 12:29:10.639283 (+  1478us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.646036  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2102) took 1083 ms. Trace:
I20260709 12:29:10.646726  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3934) took 2114 ms. Trace:
I20260709 12:29:10.649318  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:10.652629  5553 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 31: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:10.653388  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3546) took 1186 ms. Trace:
I20260709 12:29:10.653961  5556 rpcz_store.cc:276] 0709 12:29:09.466603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.258210 (+791607us) service_pool.cc:224] Handling call
0709 12:29:10.258612 (+   402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.499994 (+241382us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.653106 (+153112us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b54000eb5d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:10.653329 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":548488576}
W20260709 12:29:10.659184  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 46887.6us
I20260709 12:29:10.665768  5706 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:10.672703  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:10.673457  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:10.688822  5559 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 31: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:10.650051  5558 rpcz_store.cc:276] 0709 12:29:08.532042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.262627 (+1730585us) service_pool.cc:224] Handling call
0709 12:29:10.263113 (+   486us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:29:10.263710 (+   597us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:10.263740 (+    30us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:29:10.286067 (+ 22327us) log.cc:844] Serialized 78 byte log entry
0709 12:29:10.488128 (+202061us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:10.488190 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.490539 (+  2349us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:10.646516 (+155977us) raft_consensus.cc:1685] finished
0709 12:29:10.646533 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:10.646675 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":951,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":140,"thread_start_us":192218,"threads_started":1,"wal-append.queue_time_us":289463}
I20260709 12:29:10.665364  5708 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:10.665747  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:10.672823  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:10.692564  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:10.674044  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:10.694523  5704 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:10.695190  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4056) took 1570 ms. Trace:
I20260709 12:29:10.694777  5627 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:29:10.695580  5704 rpcz_store.cc:276] 0709 12:29:09.124981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.495857 (+1370876us) service_pool.cc:224] Handling call
0709 12:29:10.695128 (+199271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.700222  5703 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 26.
I20260709 12:29:10.700485  5709 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:10.703902  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:10.696592  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:10.649612  5630 rpcz_store.cc:276] 0709 12:29:09.562740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.068992 (+506252us) service_pool.cc:224] Handling call
0709 12:29:10.069445 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:10.069944 (+   499us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:10.069974 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:10.070474 (+   500us) log.cc:844] Serialized 26 byte log entry
0709 12:29:10.146474 (+ 76000us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:10.366606 (+220132us) log.cc:844] Serialized 14 byte log entry
0709 12:29:10.367609 (+  1003us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.367686 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:10.645844 (+278158us) raft_consensus.cc:1685] finished
0709 12:29:10.645865 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:10.645984 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":611,"prepare.queue_time_us":228065,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":1974,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":63345,"thread_start_us":275604,"threads_started":2,"wal-append.queue_time_us":228736}
W20260709 12:29:10.694123 16178 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.576s	user 0.010s	sys 0.004s
I20260709 12:29:10.633862 15735 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.721284  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:10.725385  5701 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:10.709968  5708 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:10.673681 15947 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:10.719480  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:10.728467  5710 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:29:10.736164  5556 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 30: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:10.736927  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3653) took 2680 ms. Trace:
I20260709 12:29:10.737414  5710 rpcz_store.cc:276] 0709 12:29:08.048977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.653288 (+2604311us) service_pool.cc:224] Handling call
0709 12:29:10.709742 (+ 56454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.729023 (+ 19281us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.729320 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38656}
I20260709 12:29:10.709002 16106 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:29:10.741822  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3652) took 2693 ms. Trace:
I20260709 12:29:10.742388  5709 rpcz_store.cc:276] 0709 12:29:08.048683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.663676 (+2614993us) service_pool.cc:224] Handling call
0709 12:29:10.695163 (+ 31487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.708620 (+ 13457us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.741601 (+ 32981us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b5400074c50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:10.741751 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2917632}
W20260709 12:29:10.745210  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:10.739389  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3650) took 2968 ms. Trace:
I20260709 12:29:10.739866  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4039) took 2641 ms. Trace:
I20260709 12:29:10.684440 16162 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.045s
I20260709 12:29:10.748239  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:10.750401  5703 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:10.723148 16059 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:29:10.754070  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50283.8us
I20260709 12:29:10.752260  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:10.760507  5707 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:10.752764  5708 rpcz_store.cc:276] 0709 12:29:08.085789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.692591 (+2606802us) service_pool.cc:224] Handling call
0709 12:29:10.709727 (+ 17136us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.726487 (+ 16760us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.726783 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33152}
I20260709 12:29:10.755182  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:10.756999  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4047) took 2291 ms. Trace:
I20260709 12:29:10.758158  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:10.758462  5704 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:29:10.762365  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:10.762100  5703 rpcz_store.cc:276] 0709 12:29:08.466044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.710907 (+2244863us) service_pool.cc:224] Handling call
0709 12:29:10.733514 (+ 22607us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.756594 (+ 23080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.756909 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.762982  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4041) took 2675 ms. Trace:
I20260709 12:29:10.763334  5556 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:29:10.763374  5704 rpcz_store.cc:276] 0709 12:29:08.087473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.703588 (+2616115us) service_pool.cc:224] Handling call
0709 12:29:10.750952 (+ 47364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.762707 (+ 11755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.762930 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.752969  5701 rpcz_store.cc:276] 0709 12:29:07.763042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.665108 (+2902066us) service_pool.cc:224] Handling call
0709 12:29:10.716129 (+ 51021us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.730659 (+ 14530us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.730946 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.757622 15792 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:10.740437 16113 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:10.750527 15682 consensus_queue.cc:1175] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" responder_term: 25 status { last_received { term: 22 index: 15 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: [24.16-24.16]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:29:10.777024  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:10.778234  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:10.779284  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:10.780717 16134 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:10.781246 16134 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:10.781612 16134 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:10.784636 15921 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:10.785223 15921 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:10.785650 15921 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:10.786886 15998 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:10.786378  5452 catalog_manager.cc:5697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 19 to 24. New cstate: current_term: 24 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:10.787410 15998 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:10.787874 15998 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:10.788590 15932 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:10.789520  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4048) took 2323 ms. Trace:
I20260709 12:29:10.790606  5702 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:10.790477 16107 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.770385 15947 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 30.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.022s
I20260709 12:29:10.775447 16221 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 29.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:29:10.793362  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 91396.2us
I20260709 12:29:10.793586 16221 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:10.793992 16221 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:10.795077 16221 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 29.22, Last appended by leader: 22, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.796373  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:10.797353  5599 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:10.797087 15998 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.797673 15682 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:10.798276 15682 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.798705  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:10.800010 15792 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.802879  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1811) took 2165 ms. Trace:
I20260709 12:29:10.803251  5709 rpcz_store.cc:276] 0709 12:29:08.637058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.750471 (+2113413us) service_pool.cc:224] Handling call
0709 12:29:10.802358 (+ 51887us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.802566 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:10.802592 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:10.802649 (+    57us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:29:10.802681 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.802731 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:10.802826 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.756829 15735 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:10.806269 16129 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:29:10.808590  5556 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:29:10.809836 16048 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:29:10.811184 16048 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:10.761865 16151 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.052s
I20260709 12:29:10.812255 16048 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.816480 15538 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:29:10.817945  5451 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35600 (request call id 281) took 41296 ms (client timeout 15000 ms). Trace:
I20260709 12:29:10.818288 16021 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.723743 16178 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.820901  5707 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:29:10.821566  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3656) took 2076 ms. Trace:
I20260709 12:29:10.822005  5707 rpcz_store.cc:276] 0709 12:29:08.745039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.819895 (+2074856us) service_pool.cc:224] Handling call
0709 12:29:10.820275 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.821291 (+  1016us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.821474 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.823396  5557 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:10.789911  5710 rpcz_store.cc:276] 0709 12:29:08.466426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.748837 (+2282411us) service_pool.cc:224] Handling call
0709 12:29:10.788834 (+ 39997us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.789100 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:10.789124 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:10.789204 (+    80us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:29:10.789235 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.789294 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:10.789450 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.815485 16133 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.822921 16178 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:10.840301  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2083) took 2311 ms. Trace:
I20260709 12:29:10.840590 15690 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:10.822016 16221 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:10.829618 16106 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:10.830307 16138 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:10.854943  5560 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:29:10.850479  5708 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:10.826654 15682 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:10.885128  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3658) took 2128 ms. Trace:
I20260709 12:29:10.886116  5708 rpcz_store.cc:276] 0709 12:29:08.745567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.849907 (+2104340us) service_pool.cc:224] Handling call
0709 12:29:10.850298 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.873251 (+ 22953us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.873489 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:29:10.889101  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:10.885103  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:10.852246 16118 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:10.890280  5452 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 15 to 21, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 21 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:29:10.904464  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:10.897058  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61985.3us
I20260709 12:29:10.882869 16106 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:10.897904  5706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3661) took 2149 ms (client timeout 2130 ms). Trace:
W20260709 12:29:10.898365  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 99762.1us
I20260709 12:29:10.833230 16113 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.899276  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:10.900576  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3541) took 1846 ms. Trace:
I20260709 12:29:10.900998  5710 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:10.901777  5704 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:10.880889 16138 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:10.921396  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4055) took 1824 ms. Trace:
I20260709 12:29:10.906277  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:10.898651  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4064) took 1480 ms (client timeout 1764 ms). Trace:
I20260709 12:29:10.922861  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:10.923100  5702 rpcz_store.cc:269] 0709 12:29:09.418219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:09.783392 (+365173us) service_pool.cc:224] Handling call
0709 12:29:10.898424 (+1115032us) spinlock_profiling.cc:243] Waited 748 ms on lock 0x7b5400066154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:10.898592 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1721503104}
I20260709 12:29:10.920323  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:10.921698  5635 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:10.909363  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:10.848011  5554 rpcz_store.cc:276] 0709 12:29:08.447598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:09.981585 (+1533987us) service_pool.cc:224] Handling call
0709 12:29:10.260456 (+278871us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.260755 (+   299us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:29:10.261137 (+   382us) log.cc:844] Serialized 14 byte log entry
0709 12:29:10.451948 (+190811us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:10.452000 (+    52us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:10.452050 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.452144 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:10.532822 (+ 80678us) spinlock_profiling.cc:243] Waited 3.74 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:10.697249 (+164427us) spinlock_profiling.cc:243] Waited 9.9 ms on lock 0x7b1800437200. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4
0709 12:29:10.755518 (+ 58269us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54000ebfd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:10.759103 (+  3585us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1189,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":7984,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":7994,"spinlock_wait_cycles":611044992,"thread_start_us":182377,"threads_started":1,"wal-append.queue_time_us":224033}
W20260709 12:29:10.907168  5706 rpcz_store.cc:269] 0709 12:29:08.746361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:09.153451 (+407090us) service_pool.cc:224] Handling call
0709 12:29:10.880672 (+1727221us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400210554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:10.895778 (+ 15106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2929041280}
W20260709 12:29:10.910051  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 100621us
I20260709 12:29:10.910692  5557 rpcz_store.cc:276] 0709 12:29:09.053917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.222079 (+1168162us) service_pool.cc:224] Handling call
0709 12:29:10.256673 (+ 34594us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.824087 (+567414us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.900277 (+ 76190us) spinlock_profiling.cc:243] Waited 566 ms on lock 0x7b540023ffd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:10.900509 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1302761728}
I20260709 12:29:10.934456  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:10.912560  5452 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 18 to 21, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 21 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:10.938769  5702 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:10.943930  5627 raft_consensus.cc:2410] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 33 because replica has last-logged OpId of term: 30 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 15.
I20260709 12:29:10.912688  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4053) took 1862 ms. Trace:
I20260709 12:29:10.944433  5710 rpcz_store.cc:276] 0709 12:29:09.050362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.900409 (+1850047us) service_pool.cc:224] Handling call
0709 12:29:10.900774 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.911736 (+ 10962us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.912617 (+   881us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.910195 16106 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:10.945669  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1817) took 1707 ms. Trace:
W20260709 12:29:10.946211  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:10.950464  5702 rpcz_store.cc:276] 0709 12:29:09.232859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.924236 (+1691377us) service_pool.cc:224] Handling call
0709 12:29:10.938525 (+ 14289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.939346 (+   821us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.939785 (+   439us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.895955 16107 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:10.956957  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:10.952590  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:10.946496  5704 rpcz_store.cc:276] 0709 12:29:09.097428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.895862 (+1798434us) service_pool.cc:224] Handling call
0709 12:29:10.901599 (+  5737us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.921048 (+ 19449us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.921324 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:10.953394  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:10.906031 16127 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:10.936141 16138 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:10.904119  5701 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:10.961880  5630 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
W20260709 12:29:10.964769  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 139685us
I20260709 12:29:10.970209  5703 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 26 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:29:10.970924  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3663) took 1860 ms. Trace:
I20260709 12:29:10.971244  5703 rpcz_store.cc:276] 0709 12:29:09.110750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.895959 (+1785209us) service_pool.cc:224] Handling call
0709 12:29:10.925115 (+ 29156us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.970671 (+ 45556us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:10.970871 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:29:10.971671  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50263.6us
I20260709 12:29:10.974954  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:10.818351  5451 rpcz_store.cc:269] 0709 12:28:29.521692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:40.647733 (+11126041us) service_pool.cc:224] Handling call
0709 12:29:10.817696 (+30169963us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:28:51.496416 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:28:51.497127 (+   711us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:28:51.497187 (+    60us) write_op.cc:435] Acquired schema lock
0709 12:28:51.497206 (+    19us) tablet.cc:662] Decoding operations
0709 12:28:51.571008 (+ 73802us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:28:51.690352 (+119344us) write_op.cc:641] Partition lock acquired for write op
0709 12:28:51.698183 (+  7831us) tablet.cc:685] Acquiring locks for 34 operations
0709 12:28:52.197627 (+499444us) tablet.cc:701] Row locks acquired
0709 12:28:52.197681 (+    54us) write_op.cc:260] PREPARE: finished
0709 12:28:52.197907 (+   226us) write_op.cc:270] Start()
0709 12:28:52.198121 (+   214us) write_op.cc:276] Timestamp: P: 1783600132197823 usec, L: 0
0709 12:28:52.198164 (+    43us) op_driver.cc:348] REPLICATION: starting
0709 12:28:52.199196 (+  1032us) log.cc:844] Serialized 12965 byte log entry
0709 12:28:53.453909 (+1254713us) op_driver.cc:464] REPLICATION: finished
0709 12:28:53.764908 (+310999us) write_op.cc:301] APPLY: starting
0709 12:28:53.765024 (+   116us) tablet.cc:1367] starting BulkCheckPresence
0709 12:28:54.241329 (+476305us) tablet.cc:1370] finished BulkCheckPresence
0709 12:28:54.241447 (+   118us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:28:56.322193 (+2080746us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:28:56.337736 (+ 15543us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=68,key_file_lookups=68,delta_file_lookups=0,mrs_lookups=0
0709 12:28:56.337807 (+    71us) write_op.cc:312] APPLY: finished
0709 12:28:56.338769 (+   962us) log.cc:844] Serialized 292 byte log entry
0709 12:28:57.347220 (+1008451us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:28:57.347502 (+   282us) write_op.cc:454] Released schema lock
0709 12:28:57.347601 (+    99us) write_op.cc:341] FINISH: Updating metrics
0709 12:28:57.348178 (+   577us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:28:57.348222 (+    44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3710689,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":275023,"apply.run_cpu_time_us":107259,"apply.run_wall_time_us":3583243,"cfile_cache_hit":138,"cfile_cache_hit_bytes":146247,"num_ops":34,"prepare.queue_time_us":142040,"prepare.run_cpu_time_us":29884,"prepare.run_wall_time_us":948399,"raft.queue_time_us":251811,"raft.run_cpu_time_us":23819,"raft.run_wall_time_us":264066,"replication_time_us":1255554,"thread_start_us":1558873,"threads_started":5,"wal-append.queue_time_us":1213441,"wal-append.run_cpu_time_us":31029,"wal-append.run_wall_time_us":2801126}]]}
I20260709 12:29:10.988651  5553 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
W20260709 12:29:10.989972  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 86998.5us
I20260709 12:29:10.992799  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:10.933678 16118 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:10.957938 16026 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:10.993510  5633 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:10.997411  5636 raft_consensus.cc:2368] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26.
I20260709 12:29:10.985833 15932 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:10.968672 16021 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:11.000447  5710 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:11.004669  5635 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:10.973856 16202 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:11.005074  5630 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:29:11.010355  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:10.996253 16107 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:11.012861  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:11.016362  5633 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:11.016541  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:11.017243  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:11.015164  5703 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:10.961621 15921 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:11.017405  5707 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 34: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:10.999203 15932 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:11.017732  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1820) took 1784 ms. Trace:
I20260709 12:29:11.024513 15947 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.003s
I20260709 12:29:11.025462 15921 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:11.026101 16118 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:11.026463  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:11.030447  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:11.033317  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1812) took 2395 ms. Trace:
I20260709 12:29:11.033782  5701 rpcz_store.cc:276] 0709 12:29:08.637432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.764699 (+2127267us) service_pool.cc:224] Handling call
0709 12:29:10.903948 (+139249us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:10.966109 (+ 62161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.033080 (+ 66971us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b5400066150. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:11.033226 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34568448}
I20260709 12:29:11.035602  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1823) took 1101 ms. Trace:
I20260709 12:29:11.035650 16113 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:11.004195 16133 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:11.042776 16107 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:11.024832  5706 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 37 index: 22. (index mismatch)
I20260709 12:29:11.044328 15792 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:11.044864  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4063) took 1777 ms. Trace:
I20260709 12:29:11.045230  5706 rpcz_store.cc:276] 0709 12:29:09.267211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:11.015010 (+1747799us) service_pool.cc:224] Handling call
0709 12:29:11.019666 (+  4656us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:11.044610 (+ 24944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.044812 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:11.046805  5706 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:29:11.047554  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1824) took 1112 ms. Trace:
I20260709 12:29:11.021051  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:11.028301  5708 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 32 index: 16. (index mismatch)
I20260709 12:29:11.050433  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3665) took 1563 ms. Trace:
I20260709 12:29:11.050911  5708 rpcz_store.cc:276] 0709 12:29:09.486811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:11.027534 (+1540723us) service_pool.cc:224] Handling call
0709 12:29:11.027902 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:11.050152 (+ 22250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.050365 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:11.051645 15932 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 32.14, Last appended by leader: 14, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:11.053444 15932 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:11.017372  5524 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.095s (SENT) [suppressed 7 similar messages]
W20260709 12:29:11.060331  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53855.2us
I20260709 12:29:11.061640 16178 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:29:11.064970  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1819) took 1831 ms. Trace:
I20260709 12:29:11.047875  5706 rpcz_store.cc:276] 0709 12:29:09.937935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:11.046131 (+1108196us) service_pool.cc:224] Handling call
0709 12:29:11.046592 (+   461us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:11.047271 (+   679us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.047473 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:29:11.066494  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 141918us
I20260709 12:29:11.067503  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:11.066187  5702 rpcz_store.cc:276] 0709 12:29:09.233417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.951797 (+1718380us) service_pool.cc:224] Handling call
0709 12:29:11.000970 (+ 49173us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:11.063888 (+ 62918us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:11.063931 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:11.064202 (+   271us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:11.064292 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.064457 (+   165us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:11.064873 (+   416us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1605504}
I20260709 12:29:11.052446  5708 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:11.036077  5701 rpcz_store.cc:276] 0709 12:29:09.934581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:11.034697 (+1100116us) service_pool.cc:224] Handling call
0709 12:29:11.035053 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:11.035250 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:11.035276 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:11.035321 (+    45us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:11.035360 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.035427 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:11.035524 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:11.070221  5704 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:11.016799 16021 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:11.072125  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1821) took 1838 ms. Trace:
I20260709 12:29:11.073320  5704 rpcz_store.cc:276] 0709 12:29:09.233970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.991943 (+1757973us) service_pool.cc:224] Handling call
0709 12:29:11.069988 (+ 78045us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:11.071136 (+  1148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.072067 (+   931us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:11.072546  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:11.074678  5557 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:11.077811  5631 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 30: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:11.023797  5710 rpcz_store.cc:276] 0709 12:29:09.233669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.958946 (+1725277us) service_pool.cc:224] Handling call
0709 12:29:11.000216 (+ 41270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:11.015813 (+ 15597us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.017678 (+  1865us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:11.048287 15998 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:11.081314  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:11.082137  5554 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:11.086750  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3647) took 3324 ms. Trace:
I20260709 12:29:11.097667  5560 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:11.098233  5560 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:11.073292 16021 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:11.136052 16021 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:11.087617  5705 rpcz_store.cc:276] 0709 12:29:07.762094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.638072 (+2875978us) service_pool.cc:224] Handling call
0709 12:29:10.638609 (+   537us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:10.639212 (+   603us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:10.639249 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:10.639808 (+   559us) log.cc:844] Serialized 26 byte log entry
0709 12:29:10.883486 (+243678us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:11.080317 (+196831us) log.cc:844] Serialized 14 byte log entry
0709 12:29:11.080625 (+   308us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.080699 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:11.086161 (+  5462us) raft_consensus.cc:1685] finished
0709 12:29:11.086175 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:11.086525 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":165310,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":6401,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":5874,"thread_start_us":414831,"threads_started":2,"wal-append.queue_time_us":282949}
W20260709 12:29:11.145509  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68368.3us
W20260709 12:29:11.147432  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 140296us
I20260709 12:29:11.023756 16202 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.154062  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:11.155923  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:11.105525 16059 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:11.156663  5631 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:29:11.147750  5560 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:11.160735  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:11.164284  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 155839us
I20260709 12:29:11.169927  5552 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
W20260709 12:29:11.161559 16225 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:29:11.177047  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:11.176772  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 159436us
I20260709 12:29:11.159945 16162 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:11.185379  5709 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:29:11.187453  5708 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:29:11.188977  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:11.189250  5560 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:11.182896  5599 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:11.191231  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:11.199399  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1825) took 1249 ms. Trace:
I20260709 12:29:11.200745  5708 rpcz_store.cc:276] 0709 12:29:09.938525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:11.051815 (+1113290us) service_pool.cc:224] Handling call
0709 12:29:11.052255 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:11.188002 (+135747us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.188276 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:11.206117  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:11.209811  6273 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:11.210664  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51047us
I20260709 12:29:11.211802  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1813) took 2574 ms. Trace:
I20260709 12:29:11.209733  5561 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:29:11.159003 16134 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:11.215270  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:11.212292  5709 rpcz_store.cc:276] 0709 12:29:08.637734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.804234 (+2166500us) service_pool.cc:224] Handling call
0709 12:29:10.831269 (+ 27035us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:11.194901 (+363632us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.208769 (+ 13868us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b54002d14d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:11.211719 (+  2950us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":807162624}
I20260709 12:29:11.215329  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:11.217347  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:11.221316  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:11.165045 16059 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.220531  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:11.164081 16178 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.047s
I20260709 12:29:11.208973 16048 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:11.259366 16048 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:11.259883 16048 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:11.259992  5632 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:29:11.261968  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:11.263717  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:11.228765  5552 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 34: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:11.264482  5628 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:11.266093  5552 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
W20260709 12:29:11.266803 16223 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.297s	user 0.000s	sys 0.004s
I20260709 12:29:11.189153 15682 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.034s
I20260709 12:29:11.241641 16234 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:11.242556  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:11.243907 16231 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:11.244654 15930 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:11.245085 16172 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.112s
I20260709 12:29:11.248553 16220 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:11.249935 16237 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:11.249945  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:11.253842 16226 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:29:11.204365 16127 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:11.255157  5561 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:11.255364 16202 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:11.214752 16162 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:11.258697  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:11.269112  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57717.3us
I20260709 12:29:11.266086  5636 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
W20260709 12:29:11.277853  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 117340us
I20260709 12:29:11.269222  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:11.364729  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 95635.5us
I20260709 12:29:11.281968  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:11.214147 15998 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.015s
W20260709 12:29:11.288110  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:11.283852  5631 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:11.353936  5560 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 18. (term mismatch)
I20260709 12:29:11.355340  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1822) took 2007 ms. Trace:
I20260709 12:29:11.360392  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:11.372507  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 122629us
I20260709 12:29:11.377125  5634 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:11.366902  5560 pending_rounds.cc:85] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 17
I20260709 12:29:11.221410 16027 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:11.193260 16026 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.076s
W20260709 12:29:11.359776  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:11.353308  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:11.382759  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:11.283386 16232 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:29:11.363756 16200 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 12:29:11.379693  5560 pending_rounds.cc:107] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 28.18
I20260709 12:29:11.400054  5560 raft_consensus.cc:2889] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:29:11.383550  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 220061us
I20260709 12:29:11.376508 15929 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:11.279402 16236 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.118s
I20260709 12:29:11.371354 16234 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:11.360417 16021 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:11.419562  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:11.412767  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:11.424844  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 71661.1us
I20260709 12:29:11.399443 16237 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:11.414587 15921 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:11.426980 15921 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.396660 16232 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:11.428501  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 43754us
I20260709 12:29:11.426379  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:11.436514  5708 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:11.429358 15921 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:11.429762  6235 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.013s	sys 0.024s
W20260709 12:29:11.437704  6235 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.014s	sys 0.024s
I20260709 12:29:11.367013  5707 rpcz_store.cc:276] 0709 12:29:09.347786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:11.022617 (+1674831us) service_pool.cc:224] Handling call
0709 12:29:11.032913 (+ 10296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:11.033199 (+   286us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:29:11.158753 (+125554us) log.cc:844] Serialized 14 byte log entry
0709 12:29:11.278592 (+119839us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:11.278669 (+    77us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:11.354911 (+ 76242us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.355032 (+   121us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:11.355247 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":31440,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":30,"raft.queue_time_us":14431,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":6788,"thread_start_us":200116,"threads_started":2}
I20260709 12:29:11.438759  6273 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:11.366348 16220 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.440743  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:11.397903 15792 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:11.441963  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 183299us
W20260709 12:29:11.436812 15903 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 29 current term: 29 is higher.
I20260709 12:29:11.400337  5628 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e69273530292453d87614627debcc13e"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:11.448710  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:11.449357  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:11.451951  5628 raft_consensus.cc:606] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:11.452502  5709 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:11.413108 15929 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:11.411360 16244 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:29:11.459195  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:11.449272 16059 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:11.452544  5628 raft_consensus.cc:614] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:11.407791 16133 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:11.431945 15707 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:11.454566  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:11.453984  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:11.385393 15930 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.436993 16232 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:11.457592 16162 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:11.405342 16231 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.454133 16246 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:11.415323 16160 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:11.458145  5599 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:29:11.458469 16218 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:29:11.453893  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:11.463291  6272 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
W20260709 12:29:11.454536 16221 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.593s	user 0.013s	sys 0.000s
I20260709 12:29:11.467075 16118 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:29:11.468983 16172 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:29:11.403254 16027 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.457295 16250 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:11.473625  5561 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:11.475122  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:11.455003 16236 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:11.461058 15929 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:11.441246 15538 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.032s
I20260709 12:29:11.462312 15735 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:11.483148  6278 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:11.484014  6279 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:29:11.503196  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:11.504341  5705 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:11.455200 15792 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:11.446097 16235 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.046s
I20260709 12:29:11.500646  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:11.495379  6271 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 34: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:29:11.510097 16241 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
W20260709 12:29:11.510972 16239 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
W20260709 12:29:11.512724  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 236219us
I20260709 12:29:11.498525  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:11.513090 15735 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.519061  6278 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:11.473156 15707 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.496018 16250 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:11.462210 15690 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.032s
I20260709 12:29:11.525880 16228 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:11.525998 15930 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:11.526648 16248 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.062s
I20260709 12:29:11.527868 16234 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:11.533561 16021 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:11.534946 15792 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:11.535005 16106 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.571s	user 0.007s	sys 0.004s
I20260709 12:29:11.536723 16220 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:11.539276 16231 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:11.539860  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 81727us
I20260709 12:29:11.540261 16134 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:11.540589  5602 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:11.543087  5628 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:11.595691  5628 raft_consensus.cc:606] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:11.596199  5628 raft_consensus.cc:614] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:11.548241  5554 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:29:11.551162  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:11.563995  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 64746.8us
I20260709 12:29:11.503178 16133 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:11.570483  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:11.571118  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:29:11.580560  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 82350.4us
I20260709 12:29:11.605891  5561 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:11.501260 16160 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:11.607138  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56106.1us
W20260709 12:29:11.613713  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:11.560012 16228 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:11.559973 16250 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:11.614838  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:11.567719 15903 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:11.485275 16246 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:11.593665 16138 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.605s	user 0.010s	sys 0.002s
I20260709 12:29:11.625105  5705 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:11.628566  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:11.628899 16235 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:11.629271  5552 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:11.623637 16127 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:11.630187 16244 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:11.630914 16244 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:11.631894 16244 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 29.22, Last appended by leader: 22, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.633893 15707 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:11.635151 15735 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:11.642032  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:11.644531 16220 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 29.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:29:11.651969  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:11.652606  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:11.629977 16235 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:11.648711  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:11.660640  5634 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:29:11.606940 16021 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:11.617188 15792 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:11.667713  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:11.678601  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:11.681820  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 110810us
I20260709 12:29:11.682052  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:11.682823  5632 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:29:11.590498 16106 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:11.620669 16133 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:11.685774  5701 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 33: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:11.671922  5601 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:11.671999  5560 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:29:11.673741  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:11.687585  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:11.695824  5627 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:11.690407  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:11.673396  5702 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:11.698523  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:11.668594 16151 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:29:11.692586  5707 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:29:11.693156  5558 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:29:11.698122  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:11.675401  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:11.697980  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 157538us
I20260709 12:29:11.697690  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:11.703809  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3668) took 2132 ms. Trace:
I20260709 12:29:11.661058 16235 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:11.706408  5629 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:11.683333 15690 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:11.680382 15792 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:11.709154  5703 rpcz_store.cc:276] 0709 12:29:09.563242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:11.034009 (+1470767us) service_pool.cc:224] Handling call
0709 12:29:11.034426 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:11.034833 (+   407us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:11.034867 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:11.035690 (+   823us) log.cc:844] Serialized 26 byte log entry
0709 12:29:11.162923 (+127233us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:11.162991 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.163085 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:11.695209 (+532124us) raft_consensus.cc:1685] finished
0709 12:29:11.695238 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:11.695398 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":453,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"thread_start_us":126614,"threads_started":1,"wal-append.queue_time_us":181438}
I20260709 12:29:11.713832  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:11.671820 16257 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:29:11.716545  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:11.658107 15903 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.695631 16236 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:11.667510 15929 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.689455 16134 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:11.646602 16138 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.646704 16048 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:11.729214  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:11.716804  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:11.744674  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:11.704221 16113 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:11.685104 16021 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.753165  6276 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:11.731101 16151 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:11.697487 15707 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 31.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.046s
I20260709 12:29:11.780308  5636 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: []
W20260709 12:29:11.783327  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 125945us
I20260709 12:29:11.785953  5708 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:11.712669 15538 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:11.755831  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:11.757296  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:11.789029  6273 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:11.748659 16134 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:11.733139 15792 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:11.760478  5628 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:11.764778  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:11.795535  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:11.777659 16027 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:11.789600  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:11.787892  6279 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
W20260709 12:29:11.790503  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59071.9us
I20260709 12:29:11.714500 16260 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.073s
W20260709 12:29:11.797734  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 94259.7us
I20260709 12:29:11.759122 16178 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:11.795890 16134 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:11.807215  5702 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:29:11.773612 16127 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:11.875502  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:11.876427  5703 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 35 because replica has last-logged OpId of term: 33 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 21.
W20260709 12:29:11.868748  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:11.876334  5705 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:11.888386  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:11.749228 16234 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.077s
I20260709 12:29:11.894222  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:11.894632  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:11.897485  6279 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:11.872265 16244 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:29:11.911464  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:11.917502  6273 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
W20260709 12:29:11.916194  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 208285us
I20260709 12:29:11.921623  5627 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:11.922839  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:11.925204  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 125459us
W20260709 12:29:11.927435  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 229838us
I20260709 12:29:11.930243  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:11.934882  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:29:11.938771 16255 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:29:11.939149 16265 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:29:11.939944  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:11.940192 16255 raft_consensus.cc:2764] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:29:11.945336 16257 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:11.950593  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:11.952812 16246 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:11.953600 16106 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:11.954247 15903 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:11.871632 16235 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:11.910547 15538 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:11.951987  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:11.946952  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 212026us
W20260709 12:29:11.964501  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 69981.9us
I20260709 12:29:11.922293 16248 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:11.946123 16113 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:11.964066  5550 tablet_service.cc:1558] Processing DeleteTablet for tablet 26edf338d41e42f1a5ce0d4416ec67e1 with delete_type TABLET_DATA_TOMBSTONED (TS cea66aa15be3423880843974acdbf50f not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:49500
I20260709 12:29:11.943692 16127 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:11.935760 16160 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:11.981002 15707 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:11.986593  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:11.954283 16106 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:29:11.992336  5709 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.527s	user 0.004s	sys 0.005s
I20260709 12:29:11.965687 16234 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:11.925526 16178 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:12.003939  5709 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:11.977052 16244 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:11.903080 16133 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:12.008630  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56722.9us
I20260709 12:29:11.988516 16127 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:11.914175 16151 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:11.995534 16260 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:11.994593 15707 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:12.002641 16234 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:12.042455  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:11.994863 15932 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 12:29:12.063845  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:11.981477 16248 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:12.093544 16021 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:12.067798 16260 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:12.094260  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:12.094332 16021 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:12.095788 16260 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:12.040155 16138 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:12.096916 16269 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260709 12:29:12.099030 16138 raft_consensus.cc:2764] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:29:12.099984 16113 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.24, Last appended by leader: 24, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:12.032600 16048 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:12.105577 16178 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:12.117070  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:12.125486 16234 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:12.127128 16234 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:12.045162 15707 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:12.131709  5561 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:12.136555  5632 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:12.139367  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:12.136651  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:12.154783  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 50480.3us
I20260709 12:29:12.164227  6276 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
W20260709 12:29:12.148766 16258 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.355s	user 0.002s	sys 0.000s
I20260709 12:29:12.067263 16160 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.086s
I20260709 12:29:12.156589  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:12.148247 16021 raft_consensus.cc:2764] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:29:12.186902  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:29:12.154943 16266 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
I20260709 12:29:12.201292  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:12.155522 16048 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:29:12.235298  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 57667.5us
I20260709 12:29:12.237792 16279 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.040s
I20260709 12:29:12.110316 16257 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:12.248061  6276 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: [32.20-32.20]
I20260709 12:29:12.154460 16138 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.063s
W20260709 12:29:12.275748  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:12.219977 16237 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.794s	user 0.009s	sys 0.004s
I20260709 12:29:12.154532 16250 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:12.251368 15932 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:12.219696 16282 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:12.252161 16048 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:12.339425 16289 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260709 12:29:12.341532 16283 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.189s
W20260709 12:29:12.344856 16277 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260709 12:29:12.346674 16250 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:12.348248 16286 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:29:12.348582 16278 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:12.349220 16278 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:12.349653 16278 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:12.168267 16127 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:12.353404  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:12.354110  6278 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
W20260709 12:29:12.291071 16271 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.314s	user 0.001s	sys 0.000s
I20260709 12:29:12.359750  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:12.326718 15690 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.000s	sys 0.017s
I20260709 12:29:12.364498  5704 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 29: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:12.347710 16134 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:29:12.365626 16232 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.867s	user 0.009s	sys 0.004s
I20260709 12:29:12.366076 16232 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:12.367239  5633 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 31: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:12.368324 16134 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:12.369411 16113 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:12.370261 16113 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20260709 12:29:12.371189 15690 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.001s	sys 0.017s
I20260709 12:29:12.371601 15690 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:12.373756  5709 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:12.277086 16138 consensus_queue.cc:1243] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 36.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:29:12.379220  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3972) took 1284 ms. Trace:
I20260709 12:29:12.286157 15792 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:12.380986 16281 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.441s	user 0.006s	sys 0.007s
I20260709 12:29:12.379832  5556 rpcz_store.cc:276] 0709 12:29:11.094382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:11.134724 (+ 40342us) service_pool.cc:224] Handling call
0709 12:29:11.152865 (+ 18141us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:11.373771 (+220906us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:11.373819 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:11.387013 (+ 13194us) log.cc:844] Serialized 26 byte log entry
0709 12:29:11.656708 (+269695us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:29:11.656768 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.656862 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:12.378941 (+722079us) raft_consensus.cc:1685] finished
0709 12:29:12.378971 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:12.379146 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29104,"raft.queue_time_us":241561,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":15559,"spinlock_wait_cycles":1920,"thread_start_us":431968,"threads_started":2,"wal-append.queue_time_us":291503}
I20260709 12:29:12.382848  5708 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 33. Current term is 34. Ops: [33.21-33.21]
I20260709 12:29:12.383426 16107 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:29:12.379613 16270 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.308s	user 0.004s	sys 0.004s
I20260709 12:29:12.383870 16107 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:12.384897 16107 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:12.338600 15932 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:12.382412 15792 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:12.386270 16107 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:12.284685 16237 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:12.279747 16106 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:12.388281 16228 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:12.388827 15735 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:12.388756 16106 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:12.389442 15735 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:12.390457 15735 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 29.22, Last appended by leader: 22, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:12.393433 16248 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:12.388056  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:12.386147 15932 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:29:12.402172  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 166847us
I20260709 12:29:12.404603  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:12.408531  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:12.409969  5632 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:12.368185 16232 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:12.415791  5631 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:12.418395  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:12.419211  5632 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:12.420930  5634 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:12.422091  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:12.394906 15690 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:12.422979  5630 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:29:12.407202 16134 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:12.439218 16295 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.025s
I20260709 12:29:12.453092 16292 tablet_replica.cc:333] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:29:12.453171 15707 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:12.453399 16263 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.458s	user 0.000s	sys 0.013s
I20260709 12:29:12.464241  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:12.464815  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:12.465242  5702 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:12.467375  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:12.468801  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:12.469056  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:12.469717  5705 raft_consensus.cc:2368] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32.
I20260709 12:29:12.470291  5707 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:12.471956  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:12.473133  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:12.474128 16280 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:12.474758  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:12.475482 16280 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:12.475569  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:12.475919 16280 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:12.476758 16059 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:12.476840 16284 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:29:12.476956  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 413201us
I20260709 12:29:12.477944 16244 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:29:12.460207 15707 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:12.478689 16178 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:12.478842 16244 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:12.479286 16178 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:12.480437 16178 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:12.481539  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:12.479358 16244 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:12.482578 16285 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:29:12.483342 16244 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:12.483881 16250 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:12.484311 16250 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:12.478442 16255 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:29:12.485582 16255 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:12.486043  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:12.485482  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:12.498615  5704 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:12.487035  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:12.489876  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:12.499238  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.697s	user 0.009s	sys 0.029s
I20260709 12:29:12.491034  5709 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: [27.15-27.15]
W20260709 12:29:12.499871  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.009s	sys 0.029s
I20260709 12:29:12.412596 15998 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:29:12.492484  5702 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:12.372592 16048 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.15, Last appended by leader: 15, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:12.498800  5705 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:12.516840  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:12.526767  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:12.529309  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:12.527896  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:12.532122  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:12.532719  5553 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:12.532968  5632 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 28: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:12.536377  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:12.537003  5555 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:12.537887  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:12.498843  5602 connection.cc:256] RPC call timeout handler was delayed by -1.3780321170002026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:29:12.476280 16280 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:12.543460  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:12.411156 16160 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.095s
I20260709 12:29:12.548492  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:12.545542  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 67322.8us
I20260709 12:29:12.505463 15792 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:12.561498 16291 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:12.518347 15998 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:12.567375  6237 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:12.568120  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:12.570456  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:12.572593  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:12.578538  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:12.579325  5631 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:12.579483  5634 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 35 because replica has last-logged OpId of term: 33 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 21.
I20260709 12:29:12.571447  5703 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:12.564412  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:12.552899  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:12.574542  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 166076us
I20260709 12:29:12.568266  5710 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:12.546578  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:12.536185 16134 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:12.585258  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:12.591954  5632 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:29:12.588567  5635 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: []
W20260709 12:29:12.602036  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49214.9us
I20260709 12:29:12.602846  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:12.607998  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:12.609069  5636 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:29:12.574566 16300 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:12.485431 16250 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:12.585762 15998 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:12.641767  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:12.642527  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:12.646221  5705 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:12.587749 16282 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:12.648151  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 126465us
I20260709 12:29:12.569618 15930 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.059s
I20260709 12:29:12.601487 16295 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:12.656533  5705 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:29:12.653036  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:12.622505 16172 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:12.649914  5710 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:12.674036  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:12.629491 16300 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:29:12.694103  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:12.692917  5704 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:12.653256 16284 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:12.606240 16134 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:12.716939  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 101372us
I20260709 12:29:12.675817 16172 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:12.735224  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:12.735575  5599 leader_election.cc:400] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32.
I20260709 12:29:12.738894  5599 leader_election.cc:403] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:29:12.744897  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:12.746210  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 149656us
W20260709 12:29:12.747751  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 65350.3us
I20260709 12:29:12.747375  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:12.758071  5631 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:12.674059 16221 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:12.656505 16250 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:12.680764 16248 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:12.795576  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:12.795352  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:12.651906 15792 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.087s
I20260709 12:29:12.747057  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:12.760120 16262 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.009s	sys 0.012s
I20260709 12:29:12.811353  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:12.802814 15690 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:12.671241 16257 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.125s
I20260709 12:29:12.747519 16172 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:12.820204  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 166948us
I20260709 12:29:12.821457  5636 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: [25.13-27.14]
W20260709 12:29:12.825067  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 257842us
I20260709 12:29:12.735268 15682 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.109s
W20260709 12:29:12.815737 16262 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.877s	user 0.010s	sys 0.012s
I20260709 12:29:12.803202 16284 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:12.681866 16295 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:12.827942  5632 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:12.697659 16282 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:12.671890 16255 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:12.839138  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 155444us
I20260709 12:29:12.794735 16246 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.033s
I20260709 12:29:12.845669  5631 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
I20260709 12:29:12.848558  5632 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: [32.20-32.20]
W20260709 12:29:12.833495 16296 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.289s	user 0.001s	sys 0.001s
I20260709 12:29:12.852018  5631 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:12.851202  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:12.854821  5634 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 32 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 19.
I20260709 12:29:12.831332 16256 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:12.858695 16302 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.202s	user 0.001s	sys 0.000s
I20260709 12:29:12.862427  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3977) took 1391 ms. Trace:
I20260709 12:29:12.837327 16279 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32.
I20260709 12:29:12.866494  6272 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:12.869526  5560 rpcz_store.cc:276] 0709 12:29:11.471430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:11.471569 (+   139us) service_pool.cc:224] Handling call
0709 12:29:12.862255 (+1390686us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b540044f054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:12.862370 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":401599872}
I20260709 12:29:12.865909  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:12.863111  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:12.871213  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:12.878015  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 131107us
I20260709 12:29:12.856231 16294 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:29:12.874488 16106 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:12.879541  5632 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:12.889199  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:12.878795  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:12.832947 15690 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:12.884260 16293 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:29:12.929874 16248 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:29:12.931097 16304 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:29:12.954881  5555 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:12.883569 16106 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:12.882288 16294 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:12.934332 15690 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:12.934656 16307 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.278s	user 0.001s	sys 0.000s
I20260709 12:29:12.936885  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:13.001071  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 64267.6us
I20260709 12:29:12.937677 16313 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:12.938558 16312 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:29:12.938885  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 361563us
I20260709 12:29:12.939110 16322 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:12.939446 16284 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:12.941064 16311 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
W20260709 12:29:13.019369 16278 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.669s	user 0.014s	sys 0.000s
I20260709 12:29:13.019865 16278 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:13.020531 16234 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.892s	user 0.010s	sys 0.003s
I20260709 12:29:13.022850 16278 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:13.023208 16235 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:12.941865 16299 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:29:13.024791 16299 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:13.025741 16299 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.014590 15690 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:29:13.027287  5630 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 28: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:13.027478 15690 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:12.854745 16133 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.053s
I20260709 12:29:12.884814 16301 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:12.957412 16138 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:13.047053 16138 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:29:12.961061  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:12.962579 16295 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:12.962769 15998 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:12.991329 16294 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:13.055073 16294 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:12.964025 16282 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:12.880306 16256 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.031422 16312 consensus_queue.cc:1243] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 26.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20260709 12:29:12.967331 16298 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.338s	user 0.002s	sys 0.000s
I20260709 12:29:13.057678 16313 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:13.058120 16313 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:13.030409 16133 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:13.058772 16133 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:12.854826 16300 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.058979 16313 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.059504 16300 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:13.059594 16133 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:13.060700  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3817) took 1342 ms. Trace:
I20260709 12:29:13.030066 16301 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:29:13.061012 16021 consensus_queue.cc:1175] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "38736cc545eb4389b220a3bd21d3b466" responder_term: 29 status { last_received { term: 29 index: 20 } last_committed_idx: 20 last_received_current_leader { term: 29 index: 20 } }
I20260709 12:29:13.055080 16106 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:12.968469 16257 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:29:12.971009 16134 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:12.991740  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 95525.5us
I20260709 12:29:12.979398 15929 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:12.900794 16246 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.105s
W20260709 12:29:13.004407 16107 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.617s	user 0.013s	sys 0.001s
I20260709 12:29:12.883682 15932 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.009328 16314 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:13.009347  5553 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:13.094137  5633 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:29:13.002082  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:13.095577  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 93595.9us
I20260709 12:29:13.096568 16237 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:13.096709 16059 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:13.009107 16178 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:13.061097  5629 rpcz_store.cc:276] 0709 12:29:11.718023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:11.734751 (+ 16728us) service_pool.cc:224] Handling call
0709 12:29:11.735126 (+   375us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:11.918640 (+183514us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:29:12.125050 (+206410us) log.cc:844] Serialized 14 byte log entry
0709 12:29:12.417939 (+292889us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:12.451690 (+ 33751us) log.cc:844] Serialized 52 byte log entry
0709 12:29:12.451941 (+   251us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:29:12.451994 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:12.452100 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:13.060438 (+608338us) raft_consensus.cc:1685] finished
0709 12:29:13.060463 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:13.060630 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56630,"prepare.queue_time_us":344219,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":2963,"raft.queue_time_us":166008,"raft.run_cpu_time_us":346,"raft.run_wall_time_us":17445,"thread_start_us":577988,"threads_started":3,"wal-append.queue_time_us":419099}
I20260709 12:29:13.063810 16262 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:12.969376 16315 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:29:13.073204  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 194482us
I20260709 12:29:13.100670 16314 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.103565 16315 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:13.067939  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:13.090436 16292 raft_consensus.cc:2243] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:13.079960 16257 consensus_queue.cc:1243] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 29.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:29:13.061810 16301 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.092712 16247 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:13.036979 16113 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.056s
I20260709 12:29:13.028422 15690 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.097211 16237 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:13.073833 16313 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:13.104928 16292 raft_consensus.cc:2272] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:13.102275 16262 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:13.112370 16260 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:29:13.130565  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 301910us
W20260709 12:29:13.129536  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 226573us
I20260709 12:29:13.115422 16247 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:13.139775  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:13.105129 16312 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.020s
I20260709 12:29:13.144423  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:13.147599 15735 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:13.148236 16332 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.148895 16332 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.113865 16107 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.016s
I20260709 12:29:13.144627  5675 leader_election.cc:400] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26.
I20260709 12:29:13.154330 16247 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.155851 16247 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:13.157119  6272 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:29:13.157322  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:13.104581 16315 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.158331 16262 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.157119  6275 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 34 index: 24. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:29:13.160022 16262 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:29:13.163703  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:13.157105 16237 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.168183 15903 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:13.168762 15903 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:13.169796 15903 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:13.171275  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:13.171461 15903 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:13.159212 16315 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:13.178037  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:13.192024 16138 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.193843 16138 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:29:13.206751  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:13.229307  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3968) took 2142 ms. Trace:
W20260709 12:29:13.234081 16266 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.008s	user 0.003s	sys 0.000s
I20260709 12:29:13.235539  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3970) took 2141 ms. Trace:
I20260709 12:29:13.235948  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:13.235904  5559 rpcz_store.cc:276] 0709 12:29:11.093665 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:11.134793 (+ 41128us) service_pool.cc:224] Handling call
0709 12:29:11.155710 (+ 20917us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:11.220698 (+ 64988us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:29:11.482389 (+261691us) log.cc:844] Serialized 14 byte log entry
0709 12:29:11.661899 (+179510us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:11.684155 (+ 22256us) log.cc:844] Serialized 26 byte log entry
0709 12:29:11.684324 (+   169us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:29:11.684390 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:11.684461 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:13.233676 (+1549215us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b240013f658. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff740b9c4 00007f4ff7409acb 00007f4ff75c4dc9 00007f4ff75c4a55 00007f4ff75c409d 00007f4ffa1f610e 00007f4ffa20d88c 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221
0709 12:29:13.235229 (+  1553us) raft_consensus.cc:1685] finished
0709 12:29:13.235336 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:13.235481 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8780,"prepare.queue_time_us":340940,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":69,"raft.queue_time_us":281268,"raft.run_cpu_time_us":369,"raft.run_wall_time_us":23891,"spinlock_wait_cycles":27501824,"thread_start_us":409996,"threads_started":3,"wal-append.queue_time_us":275917}
I20260709 12:29:13.229828  5557 rpcz_store.cc:276] 0709 12:29:11.087266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:11.087434 (+   168us) service_pool.cc:224] Handling call
0709 12:29:11.148616 (+ 61182us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:11.660930 (+512314us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:11.660981 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:11.683861 (+ 22880us) log.cc:844] Serialized 26 byte log entry
0709 12:29:11.963180 (+279319us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:11.963238 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:13.228805 (+1265567us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b54001bd854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:13.228867 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:13.228929 (+    62us) raft_consensus.cc:1685] finished
0709 12:29:13.228946 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:13.229203 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13147,"raft.queue_time_us":92709,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"spinlock_wait_cycles":470650752,"thread_start_us":512716,"threads_started":2,"wal-append.queue_time_us":335854}
W20260709 12:29:13.248370  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 151280us
I20260709 12:29:13.141376 16301 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:29:13.248685  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 234295us
W20260709 12:29:13.247839  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 180022us
W20260709 12:29:13.247720  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 115849us
W20260709 12:29:13.255563  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 429603us
I20260709 12:29:13.250279  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:13.264051  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:13.265196 16285 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:13.265303  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:13.265744 16285 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:13.266175 16285 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:13.271479 16312 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:13.272150 16312 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:13.273375 16312 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.262319  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:13.301433  6275 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35. Ops: []
W20260709 12:29:13.290025  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:13.323536  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:13.324173  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:13.325834  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:13.304116  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:13.282070  5676 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:13.328202  5632 raft_consensus.cc:1240] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:13.329255 16134 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:13.329639 16134 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:13.329984 16134 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:13.276764 16059 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:13.332691 16279 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.333177 16279 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.333520 16059 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.250934 16133 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:29:13.336102  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 85895.6us
I20260709 12:29:13.323949  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:13.336372  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:13.328876  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46902.6us
I20260709 12:29:13.337628  5559 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 29: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:13.338356  6275 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: []
I20260709 12:29:13.348604 16273 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:13.349462 16279 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:13.349390 16273 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.351534  5635 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:13.352617 16260 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:13.352887  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:13.353088  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:13.353168 16332 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:13.353530  6271 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:13.355180  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:13.355420  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:13.348723  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:13.334410 16334 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.258800 16127 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:13.301940 16236 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.260040 16237 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:13.261703 16113 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.337875  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:13.356693  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:13.356290  5702 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260709 12:29:13.357069  6278 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 22. Preceding OpId from leader: term: 30 index: 23. (index mismatch)
I20260709 12:29:13.352650  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:13.351601 16273 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:13.397324  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 138057us
W20260709 12:29:13.398080  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 61085.7us
I20260709 12:29:13.356082  5701 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:13.403798  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:13.300724 15932 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:13.327894 16310 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:13.303217 15998 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:13.443933  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:13.426268  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:13.406636  5709 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:29:13.492602  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:13.496150  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 158329us
I20260709 12:29:13.396296 16021 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:13.497051  5676 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:13.462634  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:13.464902  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:13.465597  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:13.507874  5554 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:13.507890  5710 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:29:13.493029 16343 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.472308  6276 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:13.474399 16292 ts_tablet_manager.cc:1916] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:29:13.475174 16250 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.651s	user 0.009s	sys 0.006s
I20260709 12:29:13.476025 16314 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:13.479496 16319 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.493342 16273 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.483170 16285 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.358016 16259 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:13.484046  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:13.489323  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:13.492373 16232 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:29:13.435829  6271 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
W20260709 12:29:13.438509  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 112762us
I20260709 12:29:13.550408  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:13.500451  6273 raft_consensus.cc:2410] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 34 because replica has last-logged OpId of term: 33 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 20.
I20260709 12:29:13.532339 16332 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:29:13.459443  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 121066us
W20260709 12:29:13.402297 16280 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.813s	user 0.012s	sys 0.005s
I20260709 12:29:13.466245  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:13.532294 16021 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:13.528167  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:13.530498  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:13.362330 16260 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:13.438859 16151 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:29:13.482131 16244 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.998s	user 0.012s	sys 0.013s
I20260709 12:29:13.511926 15538 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:13.511544 16248 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.548498  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:13.553236  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:13.586990  5554 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:13.427507 16261 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:13.537634 16256 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:13.565267  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:13.511947 16236 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.405552 16334 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:13.590898  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:13.561333 16260 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:13.546550 16273 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.559096 16278 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:13.596179  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45862us
I20260709 12:29:13.510958 16113 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:13.563252 16172 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.731s	user 0.008s	sys 0.009s
I20260709 12:29:13.603400  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:13.541169 16283 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 21, Time since last communication: 0.022s
I20260709 12:29:13.544067 16319 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.532094 16343 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.654234  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:13.566592 16234 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:13.609642  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47055.6us
I20260709 12:29:13.661862  5554 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 37. Current term is 38. Ops: [35.21-37.22]
W20260709 12:29:13.609362  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 61002.6us
I20260709 12:29:13.596992 16299 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.590394 16282 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.645370 16228 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:13.647429 16118 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:29:13.678519  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 75199.1us
I20260709 12:29:13.680191  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:13.586303 16235 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.661195  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:13.665339  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:13.586354 16244 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.647751 16257 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 12:29:13.699863  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:13.685824  5554 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32. Ops: [31.19-31.19]
W20260709 12:29:13.678551 16262 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.516s	user 0.015s	sys 0.004s
I20260709 12:29:13.609874 16160 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:13.691957  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:13.684820  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:13.647372 16278 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.674233 16326 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.673501 15932 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.673978 16274 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:13.724640 16274 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.726137 16343 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:13.726778 16274 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:13.686137 16342 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:13.670001 16234 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.711750  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:13.673664 16299 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.674336 16338 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:13.555680 16280 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:13.717352  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:13.649878 16172 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.719847 15932 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:13.720510 16348 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:29:13.721469 16334 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:13.722584 16346 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:29:13.723305 16285 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:13.685473 16235 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.854818 15538 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:13.855840 15538 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:13.856830 16235 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:13.724045 16345 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 12:29:13.616253 16261 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.595650 16248 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:13.681337 16282 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.696185 16232 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.059s
W20260709 12:29:13.732257 16294 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.677s	user 0.006s	sys 0.007s
I20260709 12:29:13.823828  5561 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:13.702381 15929 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:13.743548  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:13.695153 16256 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:13.758143  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:13.733929  5552 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: [29.20-29.20]
I20260709 12:29:13.695895 16127 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:13.757956 16160 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:13.772001 16273 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:13.863773  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:13.700541 16151 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:13.694645 16021 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.710532 16250 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 34.24, Next index: 25, Last known committed idx: 22, Time since last communication: 0.174s
W20260709 12:29:13.822479 16320 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
W20260709 12:29:13.881569  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 138124us
W20260709 12:29:13.824894 16339 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:29:13.738965  5553 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:13.781021  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:13.861230 16246 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:13.758735 16283 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.890496 16319 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:13.890657 16133 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.541s	user 0.004s	sys 0.006s
W20260709 12:29:13.714206  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:13.892546  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:13.722584 16236 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:13.895232  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:13.858373 15932 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:13.879372  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 167680us
W20260709 12:29:14.026731  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 134292us
I20260709 12:29:13.919893  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:13.896920  5559 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:13.896826  5556 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:13.959646  5629 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:29:13.963050  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:14.027870  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:13.957890 16315 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.765s	user 0.012s	sys 0.003s
I20260709 12:29:13.733150 16326 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.877261 16342 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.887537 16127 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:14.034931 16127 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:13.879897 16338 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.036597 16363 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:14.037462 16363 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.038002 16363 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:13.887141 16256 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:14.039511 16256 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:13.885162 16259 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:29:13.981508  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:14.046758  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:13.896262 16332 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:13.970695 16151 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:13.701867 15792 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.187s
W20260709 12:29:14.040773  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 79945.1us
I20260709 12:29:14.046996  5708 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 37. Current term is 38. Ops: []
I20260709 12:29:14.043134  5552 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:14.060217  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.058677  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.049477  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:14.040706  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:14.046211  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:14.052594  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:14.056643  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.020900  5553 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 28 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 12.
W20260709 12:29:14.042991  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:14.019347 16313 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.893s	user 0.010s	sys 0.001s
W20260709 12:29:14.042846 16355 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:29:13.884686 16294 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.010696 16250 consensus_queue.cc:1243] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 35.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:29:14.073733  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:14.043946 16360 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:29:14.128265  5630 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 28: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:14.102252  5706 raft_consensus.cc:2410] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 33 because replica has last-logged OpId of term: 32 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 19.
I20260709 12:29:14.075134  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:13.879258 15998 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.088s
I20260709 12:29:14.031927 16133 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.134203  5705 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:14.133289  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:14.161669  6272 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:14.058004 16246 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:14.032342 16273 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:14.046286 16312 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:14.134573  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:14.068169 16301 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.799s	user 0.006s	sys 0.005s
I20260709 12:29:14.140107  5629 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:14.059319 16236 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:14.140831  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:14.040244 16363 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:14.041920 16327 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:29:14.152097  5558 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:14.052079 15930 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.049292 16172 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:14.171139 16246 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:14.164021  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:14.166572  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.010s	sys 0.031s
I20260709 12:29:14.172829  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:14.214810  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.010s	sys 0.031s
I20260709 12:29:14.214917  5709 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:14.188184  5633 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:14.188742  5708 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:14.188383  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:14.216665 16301 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.151674 16244 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.160065  5628 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:29:14.217310 16301 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:14.189388 16361 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:29:14.159739 16262 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:14.211194  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:14.219515 16301 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.219838  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:14.220621  6271 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:14.221078  5561 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:29:14.079794 16059 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.116s
I20260709 12:29:14.125474 16151 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.132777 16113 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:14.190711 16312 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 29 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:14.155510 16178 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.24, Next index: 25, Last known committed idx: 21, Time since last communication: 0.070s
I20260709 12:29:14.231256  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:14.025578 16283 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.242489  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:14.232244  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:14.256172  5558 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 31: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:14.232172 16234 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:14.257634 16338 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:14.258213 16353 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:14.176586 16133 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.219074 16262 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:14.264323  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 118179us
I20260709 12:29:14.207469 16278 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.263401  5629 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:14.171934 16370 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.074s
I20260709 12:29:14.213052 16160 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:14.271930  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:14.296751  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:14.266099 16172 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.050s
W20260709 12:29:14.304692  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 61161.7us
I20260709 12:29:14.302977  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:14.317389  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3829) took 1667 ms. Trace:
I20260709 12:29:14.214761 16332 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.265316 16244 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:29:14.353346  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:14.354614  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:14.300275  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:14.212908 15930 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.369690  5631 rpcz_store.cc:276] 0709 12:29:12.649515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:12.853355 (+203840us) service_pool.cc:224] Handling call
0709 12:29:13.066212 (+212857us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:13.090330 (+ 24118us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:29:13.090380 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:13.091042 (+   662us) log.cc:844] Serialized 26 byte log entry
0709 12:29:13.334006 (+242964us) spinlock_profiling.cc:243] Waited 2.96 ms on lock 0x7b18004f82e0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:29:13.500411 (+166405us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:13.500459 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:13.500542 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:13.970531 (+469989us) raft_consensus.cc:1685] finished
0709 12:29:13.970554 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:14.317210 (+346656us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b540018c250. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:14.317330 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2312,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":4063,"spinlock_wait_cycles":495482368,"thread_start_us":166121,"threads_started":1,"wal-append.queue_time_us":565310}
I20260709 12:29:14.301067  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:14.347216 16133 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:14.294564 15903 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.122s	user 0.013s	sys 0.001s
I20260709 12:29:14.362023  5634 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:14.365650 16283 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:14.366237 16134 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.036s	user 0.008s	sys 0.005s
I20260709 12:29:14.303314 16250 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.373202  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1700) took 1032 ms. Trace:
I20260709 12:29:14.373406  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:29:14.374809  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 74651.1us
W20260709 12:29:14.375885  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 74898us
I20260709 12:29:14.389647  6273 rpcz_store.cc:276] 0709 12:29:13.340645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:13.340784 (+   139us) service_pool.cc:224] Handling call
0709 12:29:14.372986 (+1032202us) spinlock_profiling.cc:243] Waited 39.1 ms on lock 0x7b5400771d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:14.373137 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":89998208}
I20260709 12:29:14.352336 16261 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:14.294443 16280 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.370252 16282 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:14.350435 16247 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.194s	user 0.020s	sys 0.002s
I20260709 12:29:14.383905  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:14.352195 16294 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.391672  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:14.411473 16341 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.704s	user 0.003s	sys 0.000s
I20260709 12:29:14.392670  6276 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:14.396742 16283 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:14.412779 16283 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:14.413189 16283 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:14.414168 16133 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:14.414908 16133 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:29:14.415566 16133 consensus_queue.cc:1243] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 36.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:29:14.381644 16235 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:29:14.416652 16235 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:14.417085 16235 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:14.391021 16134 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:14.421171 16356 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:29:14.296890 16259 consensus_queue.cc:1059] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.110s
W20260709 12:29:14.422333  5556 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 3986) took 1926 ms (client timeout 1602 ms). Trace:
I20260709 12:29:14.422863 16134 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:14.422784  5556 rpcz_store.cc:269] 0709 12:29:12.496473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:12.496664 (+   191us) service_pool.cc:224] Handling call
0709 12:29:14.422126 (+1925462us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54001bd354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:14.422261 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3130491648}
I20260709 12:29:14.424516  5704 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 33 index: 20. (term mismatch)
I20260709 12:29:14.425122  5704 pending_rounds.cc:85] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 19
I20260709 12:29:14.425387  5704 pending_rounds.cc:107] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 32.20
I20260709 12:29:14.425640  5704 raft_consensus.cc:2889] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:14.426945  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4096) took 1062 ms. Trace:
I20260709 12:29:14.427287  5704 rpcz_store.cc:276] 0709 12:29:13.364559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:13.404801 (+ 40242us) service_pool.cc:224] Handling call
0709 12:29:14.424102 (+1019301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:14.426109 (+  2007us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:14.426719 (+   610us) spinlock_profiling.cc:243] Waited 931 ms on lock 0x7b540033fc50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:14.426860 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2141948288}
I20260709 12:29:14.369546 16244 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:14.428604 16244 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:14.430852 16285 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:14.431318 16285 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:14.382962 16370 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:14.376940  5558 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:29:14.421011  5706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4075) took 1957 ms (client timeout 2152 ms). Trace:
W20260709 12:29:14.438627  5706 rpcz_store.cc:269] 0709 12:29:12.463956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:12.468496 (+  4540us) service_pool.cc:224] Handling call
0709 12:29:13.157149 (+688653us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b5800053198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754deee 000000000036eb39 00007f4fff5c11e0 00007f4fff68a0c4 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:29:14.420758 (+1263609us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b540033fc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:14.420946 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2198625920}
I20260709 12:29:14.424599 16259 raft_consensus.cc:3055] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:14.368245 16326 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.374898 16301 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:29:14.393507  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 327645us
I20260709 12:29:14.451608  5560 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:14.441768  5554 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:29:14.444129  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:14.388229 16127 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.317384 16299 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:14.441002  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 43726.3us
I20260709 12:29:14.465587  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:14.438071 16367 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.129s	user 0.000s	sys 0.006s
I20260709 12:29:14.476106  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:14.456314 16300 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.396s	user 0.022s	sys 0.000s
I20260709 12:29:14.444788 16259 raft_consensus.cc:740] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:14.489094  5635 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:29:14.442397 16273 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:14.455145 15682 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.446496 16278 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:14.318027 16338 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.150s
I20260709 12:29:14.432507 16372 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.047s
I20260709 12:29:14.457487 16280 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.454236 16370 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:14.432231 16285 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.469508 15998 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.438028 16250 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.389482 16312 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.486868 16134 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:14.491708 16342 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.502112 16333 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 37.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.025s
I20260709 12:29:14.528512 16262 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:14.529304 16262 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.529838 16262 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.532734  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:14.532897 16370 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.533591  6273 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:14.534601  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:14.523082 15998 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.537448 15903 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.538093 15903 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.457749 16160 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:14.545148 16228 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.853s	user 0.011s	sys 0.000s
I20260709 12:29:14.545840 16228 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:29:14.546257 16228 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:14.546411 16151 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:14.505872 16278 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.547369 16228 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 37.25, Last appended by leader: 25, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.548445 16237 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.548987 16237 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.505429 16314 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.550302  6278 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:29:14.552080 16250 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:14.514482 16280 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.552817  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:14.554533  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:14.562129 16127 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.503130 15682 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.565495 15682 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.568605 16259 consensus_queue.cc:260] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.570773 15930 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.575080  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.575968  5702 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
W20260709 12:29:14.576915  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:14.499228 16273 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.577513  5454 catalog_manager.cc:5697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 24 to 26, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 26 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:14.579242 16273 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:14.582257  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3573) took 1138 ms. Trace:
I20260709 12:29:14.582823  5560 rpcz_store.cc:276] 0709 12:29:13.443460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:13.464603 (+ 21143us) service_pool.cc:224] Handling call
0709 12:29:14.582020 (+1117417us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b540016ba54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:14.582174 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2019473152}
I20260709 12:29:14.584263  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:14.549971 16314 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.603086  6272 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:14.603446  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:14.604629  6276 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: [30.21-30.21]
I20260709 12:29:14.604118  5636 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:14.606928  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:14.607985  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.609997  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:14.601516  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.616459  5706 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:29:14.613497  5710 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:14.619475  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.622200  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.620278  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.626307  5702 raft_consensus.cc:2376] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31. Re-sending same reply.
I20260709 12:29:14.606393 16372 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.601264 16278 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.628808  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:14.635458  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.636179  6273 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:14.636461  5705 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 26.
I20260709 12:29:14.622145  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:14.629298  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:14.630025  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:14.641450  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:14.532197 16262 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.625736 15903 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:14.619897 16256 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.580s	user 0.006s	sys 0.006s
I20260709 12:29:14.642040  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.642746  5708 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:14.643756 16372 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.618909 16259 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:14.647017  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:14.655031  5702 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:14.652628 16278 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:14.656766 16278 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:14.657127 16278 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:14.657967 16248 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.658449 16282 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:14.649730  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:14.648823  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:14.661679  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.668437  5706 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:14.671614  5702 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:14.626434 16299 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:14.684763  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 2127) took 1307 ms. Trace:
I20260709 12:29:14.692687  5635 rpcz_store.cc:276] 0709 12:29:13.377913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:13.378094 (+   181us) service_pool.cc:224] Handling call
0709 12:29:14.684496 (+1306402us) spinlock_profiling.cc:243] Waited 993 ms on lock 0x7b54001de2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:14.684671 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2283721088}
I20260709 12:29:14.694990  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.697366  5708 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:14.697525  5705 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:14.709923  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.711061  5705 raft_consensus.cc:2368] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27.
I20260709 12:29:14.619518 16127 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.033s
I20260709 12:29:14.698921  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.718273  5707 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:14.652740 15682 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:29:14.726492  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77718.5us
I20260709 12:29:14.658951 16248 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:14.633728 16326 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.081s
I20260709 12:29:14.712219  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.727738  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:14.736611  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.735333  5705 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:14.738862 16366 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:14.746340 16366 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:14.746737 16366 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:14.749368 16314 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.753873 16235 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.738992 16248 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.756479 16244 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.758579 16248 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.759518 16244 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.761235 15998 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:14.729822 15682 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:14.762691 15682 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:14.717272 16299 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.730914 16246 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.766373 16299 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.766927 16246 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:14.768128  6271 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 26: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:14.777333 16151 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:14.777945 16280 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.785152 16278 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:14.787858 16377 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.240s	user 0.003s	sys 0.000s
I20260709 12:29:14.787940  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2104) took 1576 ms. Trace:
I20260709 12:29:14.788614  5557 rpcz_store.cc:276] 0709 12:29:13.249732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:13.249931 (+   199us) service_pool.cc:224] Handling call
0709 12:29:13.300902 (+ 50971us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:13.407360 (+106458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:13.407407 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:13.529411 (+122004us) log.cc:844] Serialized 26 byte log entry
0709 12:29:13.837429 (+308018us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:29:13.837529 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:13.837663 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:14.781012 (+943349us) spinlock_profiling.cc:243] Waited 7.4 ms on lock 0x7b54002312a0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e7dbe 00007f4ffa5f2930 00007f4ffa5f28d8 00007f4ffa5f2858 00007f4ffa5f2813 00007f4ffa5f1a9c 00007f50004dd2f6 00007f50004d7aa8 00007f4ffa62e7cb 00007f4ffa6312c0
0709 12:29:14.787560 (+  6548us) raft_consensus.cc:1685] finished
0709 12:29:14.787599 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:14.787867 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":112939,"raft.queue_time_us":210456,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":9428,"spinlock_wait_cycles":17031424,"thread_start_us":238507,"threads_started":1,"wal-append.queue_time_us":281824}
I20260709 12:29:14.791023 16301 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.791345  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:14.791558 16301 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.792382 16278 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.792778 16292 ts_tablet_manager.cc:1929] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 15.10
I20260709 12:29:14.741590  5708 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:14.793292 16292 log.cc:1199] T 26edf338d41e42f1a5ce0d4416ec67e1 P cea66aa15be3423880843974acdbf50f: Deleting WAL directory at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/wals/26edf338d41e42f1a5ce0d4416ec67e1
I20260709 12:29:14.794096  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.795079 16292 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49500 (request call id 0) took 3086 ms. Trace:
I20260709 12:29:14.795553  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:14.660837 16256 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.809267  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3583) took 1074 ms. Trace:
I20260709 12:29:14.814456  5554 rpcz_store.cc:276] 0709 12:29:13.734947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:13.754908 (+ 19961us) service_pool.cc:224] Handling call
0709 12:29:13.795397 (+ 40489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:14.453109 (+657712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:14.808829 (+355720us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b54001be754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:14.809177 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1476892416}
I20260709 12:29:14.796296  5708 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:14.798039  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:14.819074  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:14.733300 16260 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 1.079s	user 0.014s	sys 0.000s
I20260709 12:29:14.814450  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:14.824153  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:29:14.805289  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77695.8us
I20260709 12:29:14.803979  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:14.829807  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:14.833853  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 375282us
I20260709 12:29:14.834576  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:14.835433  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:14.836910  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:14.832933  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:14.795466 16292 rpcz_store.cc:276] 0709 12:29:11.742748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:11.800927 (+ 58179us) service_pool.cc:224] Handling call
0709 12:29:12.436220 (+635293us) ts_tablet_manager.cc:1151] Deleting tablet 26edf338d41e42f1a5ce0d4416ec67e1
0709 12:29:12.452417 (+ 16197us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:29:14.540832 (+2088415us) tablet_metadata.cc:665] Metadata flushed
0709 12:29:14.733986 (+193154us) tablet_metadata.cc:665] Metadata flushed
0709 12:29:14.794824 (+ 60838us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":631310,"tablet-delete.queue_time_us":409109,"thread_start_us":197009,"threads_started":1}
I20260709 12:29:14.799576 15538 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:14.835590  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:14.846464  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:14.841420  5709 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30. Ops: [27.17-29.18]
I20260709 12:29:14.829407 16133 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.807029 16237 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.851374  5705 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: [31.17-31.17]
I20260709 12:29:14.836109  5557 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:14.821759 16273 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.803617 16372 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:14.845659  5559 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:14.805589 16127 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:14.850195  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3993) took 1036 ms. Trace:
I20260709 12:29:14.860562  5560 rpcz_store.cc:276] 0709 12:29:13.813482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:14.607935 (+794453us) service_pool.cc:224] Handling call
0709 12:29:14.624122 (+ 16187us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:14.624427 (+   305us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:29:14.639448 (+ 15021us) log.cc:844] Serialized 14 byte log entry
0709 12:29:14.849767 (+210319us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:14.849839 (+    72us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:14.849899 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:14.849986 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:14.850129 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":350,"prepare.queue_time_us":20789,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":44,"raft.queue_time_us":1339,"raft.run_wall_time_us":6755,"thread_start_us":208525,"threads_started":1}
I20260709 12:29:14.796629 16366 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:14.843837 16138 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:14.844187 16248 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:14.848367 16332 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:14.842914  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:14.833797 16294 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.871049 16294 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:14.872720  5553 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3580) took 1219 ms (client timeout 1504 ms). Trace:
I20260709 12:29:14.873310 16294 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.874784 16301 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:14.843408  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 397963us
I20260709 12:29:14.858644  5705 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: [30.15-32.16]
I20260709 12:29:14.875795 16228 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:14.876751  5705 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: [29.20-31.21]
I20260709 12:29:14.841817 16338 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:14.880846  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:14.881780  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:14.841694 16134 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:14.842805 16326 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:14.850423  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:14.887224  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:14.861590 16127 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:14.892134  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:14.853229  5708 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:29:14.863128  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:14.863196  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:14.894495  5557 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:29:14.894733  5560 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:29:14.858111 16278 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:14.846807 15538 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:14.844038 16236 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:14.868427  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:14.869853  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:14.870426 16256 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.883987  5704 pending_rounds.cc:85] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 12
I20260709 12:29:14.884223  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:14.828637 16260 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.834980 16283 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.887132  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:14.887280  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:14.849107  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:14.888958  5629 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 28: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:14.888708  5706 raft_consensus.cc:2410] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 31 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:29:14.853997  5709 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:29:14.895726  5557 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
W20260709 12:29:14.873127  5553 rpcz_store.cc:269] 0709 12:29:13.653715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:13.653971 (+   256us) service_pool.cc:224] Handling call
0709 12:29:14.872377 (+1218406us) spinlock_profiling.cc:243] Waited 83.7 ms on lock 0x7b540023fd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:14.872532 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":192590208}
I20260709 12:29:14.867599 16332 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:14.855654 16133 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:14.913311  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:14.910183  5708 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:29:14.948860  5704 pending_rounds.cc:107] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 26.13
I20260709 12:29:14.956594  5704 raft_consensus.cc:2889] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:14.949354  5634 raft_consensus.cc:2376] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31. Re-sending same reply.
W20260709 12:29:14.955381  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 106529us
I20260709 12:29:14.959517  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:14.982488  5709 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:14.984889  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3994) took 1171 ms. Trace:
I20260709 12:29:14.985495  5557 rpcz_store.cc:276] 0709 12:29:13.813792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:14.895248 (+1081456us) service_pool.cc:224] Handling call
0709 12:29:14.895554 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:14.984542 (+ 88988us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:14.984819 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:14.885869 16021 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:14.862066 16273 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:14.991097  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:14.991418  5636 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 28: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:14.885483 16134 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:14.885731 16138 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:14.986886  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:14.888046 16326 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:15.008841  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:14.994764  6275 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 34: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:14.994942  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:14.897377 16127 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:14.999734  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:15.044529  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:15.045200  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:14.993003  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 124675us
I20260709 12:29:15.047075  5599 leader_election.cc:400] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27.
I20260709 12:29:15.036129  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:15.048609  5599 leader_election.cc:403] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:29:15.038136  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:14.993577  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:14.952725 16343 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:15.041487  5634 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:14.952962 16353 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:15.038697  5557 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:14.901541 16278 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:14.994083 16332 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:14.983865  5709 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.087514  5709 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:15.087610  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:15.086953  5634 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:29:15.035990 16234 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:15.036075 16256 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:14.900524 16366 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:14.902715 16257 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:15.057101  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:15.060063  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:15.147248  5557 raft_consensus.cc:2368] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27.
I20260709 12:29:15.147188  5553 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:15.005223 16134 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:15.070612 16353 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.072733  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:15.072589 16343 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:15.073105 16283 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:15.074388 16260 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:15.000890 16248 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:15.051003 16237 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:15.000294 16021 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:15.083732 16235 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:15.072335  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:15.049950  5632 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 31: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:15.001228 16299 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:15.100674  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:15.088469  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:15.106207  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:15.043135 16326 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:15.051215 15930 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:15.127975  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:15.065953 16278 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:15.133875 16300 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 27
W20260709 12:29:15.134373 16373 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.573s	user 0.005s	sys 0.001s
I20260709 12:29:15.139693  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:15.145491  6277 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:29:15.146407  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:15.004262 16236 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.161695  5709 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 19. (term mismatch)
I20260709 12:29:15.231369  5709 pending_rounds.cc:85] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 18
I20260709 12:29:15.231775  5709 pending_rounds.cc:107] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 25.19
I20260709 12:29:15.232136  5709 raft_consensus.cc:2889] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:15.163414  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:15.234031  5553 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:15.235415  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:15.093544 16280 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:15.168426  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:15.004951 16301 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.107s
W20260709 12:29:15.192399  5703 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.010s	sys 0.055s
W20260709 12:29:15.247786  5703 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.011s	sys 0.055s
I20260709 12:29:15.196676  5560 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
I20260709 12:29:15.201864 16127 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:15.186116 16248 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:15.122336 16256 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:15.288836 16248 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:15.290448 16256 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:15.204968 16021 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:15.109963 16234 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.292569  5553 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:15.294026 16234 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:15.209520  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:15.214051  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:15.294847  6276 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:15.295454  5524 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:15.295928  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:15.296435 16375 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.725s	user 0.006s	sys 0.002s
I20260709 12:29:15.297127  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:15.297899  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:15.298691  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:15.299520  6237 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:15.211966  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:15.300324  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:15.300873  5634 raft_consensus.cc:2410] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 32 because replica has last-logged OpId of term: 29 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:29:15.300817  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:15.301658  5552 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:15.301849  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:15.302464  5634 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:15.302537  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:15.302870 16237 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:15.303293 16237 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:15.303493 16235 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:15.303682 16237 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:15.303912  5552 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: []
I20260709 12:29:15.304842 16256 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:29:15.305748 16256 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:29:15.216581  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:15.243150  5557 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 32: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:15.225176  5636 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:15.307722  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:15.308298  5636 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
W20260709 12:29:15.308766  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:15.309125  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:15.309512  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:15.310185  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:15.310742  5636 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:15.248473  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:15.313386  5559 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:15.313926  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3999) took 1424 ms. Trace:
I20260709 12:29:15.314205  5559 rpcz_store.cc:276] 0709 12:29:13.891996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.312952 (+1420956us) service_pool.cc:224] Handling call
0709 12:29:15.313222 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:15.313728 (+   506us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.313877 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:15.315444  5559 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:29:15.315469 16388 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:15.316054  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3588) took 1288 ms. Trace:
I20260709 12:29:15.263024  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:15.317657  5559 rpcz_store.cc:276] 0709 12:29:14.027926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.314918 (+1286992us) service_pool.cc:224] Handling call
0709 12:29:15.315163 (+   245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:15.315836 (+   673us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.316006 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:15.318177  6277 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:15.318822  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:15.319312  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2107) took 1164 ms. Trace:
I20260709 12:29:15.265231  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:15.265936  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:15.324524  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59385.3us
I20260709 12:29:15.325730  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:15.227519 15930 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.329195  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3831) took 1284 ms. Trace:
I20260709 12:29:15.329622  5633 rpcz_store.cc:276] 0709 12:29:14.044928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:14.048464 (+  3536us) service_pool.cc:224] Handling call
0709 12:29:14.048766 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:14.215839 (+167073us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.328967 (+1113128us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54001de7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:15.329137 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":311625088}
I20260709 12:29:15.330183 15930 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:15.267568 16259 consensus_meta.cc:287] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.585s	user 0.011s	sys 0.002s
I20260709 12:29:15.268430  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:15.331137  5634 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:15.269098 16343 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:15.175541 15538 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:15.331816  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:15.330909 16283 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:29:15.272799 16332 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.341425  6278 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:29:15.342607 16332 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:15.343387  5708 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:15.343580 16236 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:15.345237 16353 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:15.345778 16283 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:29:15.346029 16280 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:15.332129  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:15.347409  5554 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:15.348912 16283 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:15.350878 16273 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:15.349521 16283 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.353565 16283 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:15.355026  5628 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:15.355937 16127 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:15.219224 16299 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:15.339959 16021 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:15.357216 16299 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:15.362560 16021 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:15.365733  5636 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:15.367139  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:15.371826  5636 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
W20260709 12:29:15.374358  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:15.273263 16391 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:15.188060 16257 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.217922 16138 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 32.16, Last appended by leader: 16, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:15.302626 16260 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:29:15.250626  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 162203us
I20260709 12:29:15.259945  5635 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:15.316195 16388 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:15.317191 16237 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.414149  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:15.321432  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:15.260063 15792 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260709 12:29:15.330842  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:15.271809 15682 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.508s	user 0.008s	sys 0.004s
I20260709 12:29:15.337153  5559 rpcz_store.cc:276] 0709 12:29:14.154938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.318530 (+1163592us) service_pool.cc:224] Handling call
0709 12:29:15.318846 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:15.319013 (+   167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:15.319036 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:15.319078 (+    42us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:15.319110 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.319167 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:15.319261 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:15.332197 15538 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:29:15.429282  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:15.415657  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:15.431996  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:15.263741 16133 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:29:15.379151  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3991) took 1571 ms. Trace:
I20260709 12:29:15.317302  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:15.439049  5629 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:29:15.438244  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 116908us
W20260709 12:29:15.388135  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 78703us
I20260709 12:29:15.392755  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3997) took 1513 ms. Trace:
I20260709 12:29:15.394608  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:15.376850 16021 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:15.445106  5709 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 31 because replica has last-logged OpId of term: 30 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 19.
I20260709 12:29:15.295473  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:15.446728  5556 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:29:15.415980  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:15.434135  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:15.426716  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:15.448537  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:15.449465  5635 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 33: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:15.416290  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:15.451071  5552 rpcz_store.cc:276] 0709 12:29:13.879693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.303489 (+1423796us) service_pool.cc:224] Handling call
0709 12:29:15.303756 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:15.392331 (+ 88575us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.392675 (+   344us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:15.409548 16256 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:29:15.413437 16260 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:15.436107  5555 rpcz_store.cc:276] 0709 12:29:13.808169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:14.085067 (+276898us) service_pool.cc:224] Handling call
0709 12:29:14.141210 (+ 56143us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:14.149027 (+  7817us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:29:14.177570 (+ 28543us) log.cc:844] Serialized 14 byte log entry
0709 12:29:14.898846 (+721276us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:14.898925 (+    79us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:14.898989 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.378776 (+479787us) spinlock_profiling.cc:243] Waited 7.52 ms on lock 0x7b54001bdd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:15.378848 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:15.379070 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":28312,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":5758,"raft.queue_time_us":38094,"raft.run_cpu_time_us":55,"raft.run_wall_time_us":55,"spinlock_wait_cycles":17287936,"thread_start_us":720735,"threads_started":2,"wal-append.queue_time_us":575146}
I20260709 12:29:15.459234  5710 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:15.462590  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2113) took 1250 ms. Trace:
I20260709 12:29:15.464402  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:15.409938 16332 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:15.497454 16338 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:15.431377 16373 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
I20260709 12:29:15.498035 16338 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:15.498035 16332 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.499168  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:15.500555 16332 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:15.500587 16338 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:15.465963  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 52021.2us
I20260709 12:29:15.501655  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:15.460253 16228 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:15.467280  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:15.503305  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:15.455837 16370 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:15.493005 16260 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:15.428745 16388 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:15.494192 16385 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:29:15.494948 16387 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:29:15.520138  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3992) took 1707 ms. Trace:
I20260709 12:29:15.496477 16178 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:29:15.496939  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:15.520852  5561 rpcz_store.cc:276] 0709 12:29:13.813131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:14.127726 (+314595us) service_pool.cc:224] Handling call
0709 12:29:14.210149 (+ 82423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:14.221483 (+ 11334us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.519743 (+1298260us) spinlock_profiling.cc:243] Waited 57.3 ms on lock 0x7b5400082350. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:15.520034 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":131770752}
I20260709 12:29:15.493000  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:15.409922 16391 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:15.437557 15682 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.500221  5708 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 27: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:15.504434  5559 rpcz_store.cc:276] 0709 12:29:14.212298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.461229 (+1248931us) service_pool.cc:224] Handling call
0709 12:29:15.461739 (+   510us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:15.462020 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:15.462059 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:15.462121 (+    62us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:15.462171 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.462244 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:15.462403 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:29:15.529210  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 427646us
I20260709 12:29:15.529291  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2114) took 1287 ms. Trace:
I20260709 12:29:15.503681  5706 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:15.468858  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:15.505107  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:15.510792  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:15.508946  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:15.467312 16343 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:15.516131  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:15.413573 16359 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.090s
I20260709 12:29:15.506593 16228 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:15.523770  5631 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:29:15.456323 16172 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:15.467118 16280 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:15.532766  5635 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 26: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:15.533913  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:15.513921  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:15.510416  5710 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:29:15.532249  5559 rpcz_store.cc:276] 0709 12:29:14.241817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.527756 (+1285939us) service_pool.cc:224] Handling call
0709 12:29:15.528375 (+   619us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:15.528634 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:15.528677 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:15.528745 (+    68us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:29:15.528798 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.529052 (+   254us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:15.529211 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:15.539286  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:15.507704 16338 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:15.531383  5703 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:15.512595 16257 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:15.528841 16385 consensus_queue.cc:1243] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 29.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:29:15.528657 16160 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:15.525189 16151 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:15.502826 16244 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.024s
I20260709 12:29:15.584697  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:15.575234  6279 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 33: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:15.544785  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:15.575246  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:15.575700  5556 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:15.575263  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4002) took 1125 ms. Trace:
I20260709 12:29:15.509925 16390 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:15.576910  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:15.585474  6277 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:15.584764 16151 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:15.467387 16021 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.527179 16391 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:15.535703 16178 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:15.609850  5559 rpcz_store.cc:276] 0709 12:29:14.449759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.574385 (+1124626us) service_pool.cc:224] Handling call
0709 12:29:15.574732 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:15.574940 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:15.574985 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:15.575028 (+    43us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:15.575058 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.575111 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:15.575205 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:15.540386 16228 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:15.574021 16261 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:15.508380 16370 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.525116 16353 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.057s
I20260709 12:29:15.618861  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:15.581180 16274 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:15.546284 16246 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Stepping down as leader of term 31
W20260709 12:29:15.623024  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:15.624192 16261 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:15.625209  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:15.610097 16390 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:15.576256 16332 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:15.629264  5635 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: [31.16-31.16]
I20260709 12:29:15.630805  6272 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 38.
W20260709 12:29:15.619472  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.009s	sys 0.054s
W20260709 12:29:15.684854  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.009s	sys 0.054s
W20260709 12:29:15.685252  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 638335us
I20260709 12:29:15.580601 16387 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:15.622784 16359 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:15.616264 16244 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:15.617148 16160 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.680725 16261 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:15.620066  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:15.580149 16343 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:15.685940  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:15.635915 16246 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:15.705536  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 86922.1us
I20260709 12:29:15.696571  5601 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:15.756860  5629 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 38.
W20260709 12:29:15.705142 16396 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.198s	user 0.000s	sys 0.000s
I20260709 12:29:15.697937 16370 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:15.761653 16237 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:15.698366 16151 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:15.702459 16160 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:15.705207 16353 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:29:15.765174 16353 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:15.766191 16353 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:15.769570 16236 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:15.770282 16236 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:15.770295 16326 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.543s	user 0.008s	sys 0.003s
I20260709 12:29:15.589836 16338 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.716773 16404 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:29:15.774322 16338 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:15.774102 16391 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.775398  5558 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:15.776473  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2108) took 1621 ms. Trace:
I20260709 12:29:15.776556  5635 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:29:15.776821  5558 rpcz_store.cc:276] 0709 12:29:14.155291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.325096 (+1169805us) service_pool.cc:224] Handling call
0709 12:29:15.325521 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:15.775749 (+450228us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.776228 (+   479us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b54001c43d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:15.776417 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1034252672}
I20260709 12:29:15.776958 16246 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:15.777864  5635 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 19. (term mismatch)
I20260709 12:29:15.778240  5635 pending_rounds.cc:85] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 18
I20260709 12:29:15.778471  5558 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:29:15.771767 16326 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.780223  5558 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:29:15.780910  5634 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
W20260709 12:29:15.781184 16403 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:29:15.770474 16248 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.782171  6274 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:15.782402  5703 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:29:15.784201 16326 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:15.784467  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3688) took 1151 ms. Trace:
I20260709 12:29:15.785946  5633 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: [25.13-27.14]
I20260709 12:29:15.784926  5704 rpcz_store.cc:276] 0709 12:29:14.633096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:14.859559 (+226463us) service_pool.cc:224] Handling call
0709 12:29:14.883728 (+ 24169us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:15.052693 (+168965us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:15.052737 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:15.053468 (+   731us) log.cc:844] Serialized 52 byte log entry
0709 12:29:15.439003 (+385535us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:29:15.439061 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.439152 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:15.784241 (+345089us) raft_consensus.cc:1685] finished
0709 12:29:15.784263 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:15.784414 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1859,"raft.queue_time_us":11746,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":9520,"thread_start_us":368413,"threads_started":1,"wal-append.queue_time_us":607256}
I20260709 12:29:15.787132 16248 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:15.787034  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:15.787041 16261 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.714313 16343 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:15.707516 16278 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.789976 16391 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:15.708850 16399 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260709 12:29:15.701200 15682 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:15.791671 16278 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:15.716665 16279 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:15.792634 16021 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:15.793082 16279 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:15.778451  5635 pending_rounds.cc:107] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 31.19
I20260709 12:29:15.794723  5635 raft_consensus.cc:2889] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:15.778528 16246 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:15.758306  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:15.804458  6273 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.002s	sys 0.027s
W20260709 12:29:15.811131  6273 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.002s	sys 0.027s
I20260709 12:29:15.811446  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2109) took 1651 ms. Trace:
I20260709 12:29:15.812057  5556 rpcz_store.cc:276] 0709 12:29:14.160160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.447963 (+1287803us) service_pool.cc:224] Handling call
0709 12:29:15.448291 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:15.612701 (+164410us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.811094 (+198393us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54001af254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:15.811363 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":286548864}
I20260709 12:29:15.823536  5635 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
W20260709 12:29:15.824407  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4103) took 1251 ms (client timeout 1628 ms). Trace:
W20260709 12:29:15.824858  5707 rpcz_store.cc:269] 0709 12:29:14.573385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:14.574643 (+  1258us) service_pool.cc:224] Handling call
0709 12:29:15.824193 (+1249550us) spinlock_profiling.cc:243] Waited 97.9 ms on lock 0x7b54004515d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:15.824307 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":225160448}
I20260709 12:29:15.680083 16273 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:15.906211  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 209793us
W20260709 12:29:15.906533  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 148314us
I20260709 12:29:15.687121 16372 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:15.793840 16279 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:15.906029  5635 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:15.684047 16390 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:15.929586 16279 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:15.909704  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:15.938083  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:15.938351  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:15.934593  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:15.939484  5710 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:15.912863 16343 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:15.940582 16391 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:29:15.723034 16134 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.548s	user 0.008s	sys 0.004s
I20260709 12:29:15.689888 16332 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:15.692914 16359 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:15.916776  5556 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:15.912703 16107 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:29:15.716773 16402 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.202s
I20260709 12:29:15.919710 16261 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:15.923786 16387 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 32.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:29:15.924137 16326 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:29:15.925859 16278 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:15.927290 15682 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:15.912779 16294 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:15.909219  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:15.957590  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:15.957823  5634 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:15.933713  5557 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.003s	sys 0.018s
W20260709 12:29:15.960858  5557 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.004s	sys 0.018s
I20260709 12:29:15.963691  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:15.941293  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:15.799801 16127 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:15.947433  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:15.910154 16283 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:15.941241 16391 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:15.793478 16178 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.952745  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:15.953609  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:15.985807  6272 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:15.940781 16244 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.017s
I20260709 12:29:15.966120  5707 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:15.947579 16343 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:15.957542 16385 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:15.978958  5556 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:29:15.998224  5634 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:29:15.913995 16273 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:15.956105 16326 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:15.955224  6275 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:29:16.031057  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:16.031865  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:16.033912  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:16.034709  5556 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:29:15.950811 16134 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:16.035219  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:16.042008  5710 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
W20260709 12:29:16.025614 16300 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.781s	user 0.011s	sys 0.001s
I20260709 12:29:15.967559 16133 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.073s
I20260709 12:29:16.021651 16343 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:15.969648 16278 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:16.072872  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:16.036324  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 82811.3us
I20260709 12:29:16.073782  5710 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:16.074542  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:16.052317  6272 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:16.067885  5556 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:29:15.985769 16257 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.060s
I20260709 12:29:16.088022  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:16.089931  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:16.090322  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:15.983433 16294 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:16.109125  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.021s	sys 0.036s
I20260709 12:29:15.989971 16283 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:16.109628  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.021s	sys 0.036s
I20260709 12:29:16.034963 16385 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:15.987135 16127 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:16.110484  5708 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:16.111140  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:16.040544 16236 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:16.115132  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 602795us
I20260709 12:29:16.045658 16407 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:29:16.059713 16366 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.014s	sys 0.024s
W20260709 12:29:16.118180 16366 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.014s	sys 0.024s
I20260709 12:29:16.117472  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:16.122849  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:16.123725  5710 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:16.093046 16300 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27.
I20260709 12:29:16.111068 16333 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:16.086750 16387 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.064s
I20260709 12:29:16.121101 16407 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:16.183398  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:16.184249  5702 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:16.186554  5710 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:29:16.187810  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:16.190506  5710 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:16.153268 16244 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:16.047914 16359 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:16.200825  5676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.000s	sys 0.029s
W20260709 12:29:16.203769  5676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.001s	sys 0.029s
W20260709 12:29:16.206534  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 672723us
I20260709 12:29:16.207674  5710 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:16.209661  5710 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:29:16.125859 16391 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 31.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:16.184784 16395 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
W20260709 12:29:16.217981  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.012s	sys 0.021s
W20260709 12:29:16.219866  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.013s	sys 0.021s
I20260709 12:29:16.202334 16261 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:16.189116 16314 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:29:16.226430  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 681733us
I20260709 12:29:16.185088 16407 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:16.234781  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:16.234957  5632 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
W20260709 12:29:16.235460  6271 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.004s	sys 0.016s
W20260709 12:29:16.235759  6271 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.004s	sys 0.016s
I20260709 12:29:16.237080  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:16.175100 16333 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:16.231971  5702 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
W20260709 12:29:16.243353 16412 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260709 12:29:16.126497 16326 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:16.249854  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:16.242643  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:16.253542  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:16.253219  5702 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 33: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:29:16.249800  6280 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.037s	sys 0.054s
W20260709 12:29:16.261597 16416 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
W20260709 12:29:16.269081 16413 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
W20260709 12:29:16.273025 16415 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 12:29:16.274507  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3840) took 1187 ms. Trace:
I20260709 12:29:16.275087  5636 rpcz_store.cc:276] 0709 12:29:15.087509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.407922 (+320413us) service_pool.cc:224] Handling call
0709 12:29:15.408389 (+   467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:15.408813 (+   424us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:15.408853 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:15.409380 (+   527us) log.cc:844] Serialized 26 byte log entry
0709 12:29:15.682373 (+272993us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:15.682702 (+   329us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.682778 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:16.274190 (+591412us) raft_consensus.cc:1685] finished
0709 12:29:16.274217 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:16.274415 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1606,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":14851,"thread_start_us":268969,"threads_started":1,"wal-append.queue_time_us":623571}
I20260709 12:29:16.233508 16314 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:16.242130 16332 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:16.282065  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:16.284325  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:16.246842 16160 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:29:16.279838  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:16.289188  5707 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 38: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:16.281203  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:16.221254 16404 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 31.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:29:16.233119 16310 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 30.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.047s
I20260709 12:29:16.290023  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:16.293740  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:16.294946  5634 raft_consensus.cc:2410] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 32 because replica has last-logged OpId of term: 29 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:29:16.295972  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:16.296680  5634 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:16.275529 16338 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:29:16.303382  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49946.7us
I20260709 12:29:16.304898  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:16.281059 16326 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:16.309798  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:16.312000  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:16.283281 16279 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:16.317063  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:16.298520 16387 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:16.329339  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:16.326880  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:16.313561 16388 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.786s	user 0.011s	sys 0.000s
I20260709 12:29:16.298686 16370 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:16.298405 15538 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:16.331570  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:16.332026  5702 raft_consensus.cc:2410] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 30 because replica has last-logged OpId of term: 29 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 20.
I20260709 12:29:16.248641 16385 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:16.318303 16338 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:16.336783 16333 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:16.340766 16314 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.16, Last appended by leader: 16, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:16.337785 15682 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:16.340312 16388 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:16.364740  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1639) took 1142 ms. Trace:
W20260709 12:29:16.370819  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:16.373872  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:16.322782  5634 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:16.326295  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:16.314114 16404 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:16.373925  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:16.317340 16134 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:16.383599  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:16.454382  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2140) took 1078 ms. Trace:
W20260709 12:29:16.398751  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 72585us
I20260709 12:29:16.389174  6276 rpcz_store.cc:276] 0709 12:29:15.222421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.298028 (+ 75607us) service_pool.cc:224] Handling call
0709 12:29:15.298310 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:15.391235 (+ 92925us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:15.391287 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:15.391352 (+    65us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:15.391401 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:16.364490 (+973089us) spinlock_profiling.cc:243] Waited 92.6 ms on lock 0x7b54006379d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:16.364540 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:16.364676 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":212990720}
I20260709 12:29:16.345850 16338 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:16.396304  5707 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:16.382021  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:16.328189 16160 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:16.456082  5636 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:16.328754 16273 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:16.372931  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:16.455966  5634 rpcz_store.cc:276] 0709 12:29:15.376087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:16.453375 (+1077288us) service_pool.cc:224] Handling call
0709 12:29:16.453774 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:16.453994 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:16.454018 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:16.454073 (+    55us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:29:16.454119 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:16.454176 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:16.454310 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:16.346485 16248 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:29:16.386746 16410 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:29:16.456332  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:16.466429  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2145) took 1013 ms. Trace:
I20260709 12:29:16.391402  5703 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:29:16.507910  5636 rpcz_store.cc:276] 0709 12:29:15.452908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:16.465349 (+1012441us) service_pool.cc:224] Handling call
0709 12:29:16.465833 (+   484us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:16.466059 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:16.466085 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:16.466131 (+    46us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:16.466160 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:16.466217 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:16.466363 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:16.509174  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:16.379698 16285 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:16.510481  5636 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
W20260709 12:29:16.374740 16228 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.741s	user 0.011s	sys 0.000s
W20260709 12:29:16.346890 16260 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.822s	user 0.009s	sys 0.002s
I20260709 12:29:16.505338  5634 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:16.528267  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2144) took 1116 ms. Trace:
I20260709 12:29:16.528795  5634 rpcz_store.cc:276] 0709 12:29:15.411967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:16.504634 (+1092667us) service_pool.cc:224] Handling call
0709 12:29:16.505078 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:16.527898 (+ 22820us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:16.528187 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:29:16.460621 16390 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.532s	user 0.010s	sys 0.004s
I20260709 12:29:16.346117 16294 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:16.344245 16387 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:16.344801 16422 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
W20260709 12:29:16.462100 16429 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.079s	user 0.002s	sys 0.000s
I20260709 12:29:16.530568  5634 raft_consensus.cc:1275] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
W20260709 12:29:16.459496  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 342114us
W20260709 12:29:16.355846 16417 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.245s	user 0.000s	sys 0.002s
W20260709 12:29:16.464607 16428 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.084s	user 0.000s	sys 0.002s
I20260709 12:29:16.546697  5636 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
W20260709 12:29:16.460103 16411 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.373s	user 0.000s	sys 0.002s
I20260709 12:29:16.338629 16424 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:16.380867 16314 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:16.579532  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:16.551052  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:16.460348 15792 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:29:16.471669 16425 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.085s	user 0.001s	sys 0.001s
I20260709 12:29:16.580727  5705 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:29:16.453284 16370 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:16.582571 16370 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:16.582949  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:16.581504 15792 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:16.584743 16370 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:16.584986 15792 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:16.344064 16418 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:29:16.468839 16278 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:16.386255 16388 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:16.573896  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:16.586934  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:16.588685  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2111) took 2391 ms. Trace:
I20260709 12:29:16.589135  5552 rpcz_store.cc:276] 0709 12:29:14.197188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.454099 (+1256911us) service_pool.cc:224] Handling call
0709 12:29:15.454469 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:15.455022 (+   553us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:15.455202 (+   180us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:15.519320 (+ 64118us) log.cc:844] Serialized 26 byte log entry
0709 12:29:15.986172 (+466852us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:15.986223 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:15.986309 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:16.588424 (+602115us) raft_consensus.cc:1685] finished
0709 12:29:16.588453 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:16.588615 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":129,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":9078,"thread_start_us":440550,"threads_started":1,"wal-append.queue_time_us":662125}
I20260709 12:29:16.591189 16257 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:16.592033 16257 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:16.592461  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:16.582108  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:16.601186  5702 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 35: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:16.603278 16257 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:16.604212 16257 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:16.607658 16257 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:16.549023 16273 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:16.619900  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 163662us
I20260709 12:29:16.617751 16256 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:16.622061 16256 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:16.624791  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 295536us
W20260709 12:29:16.574965 16246 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.667s	user 0.014s	sys 0.000s
I20260709 12:29:16.469251 16160 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:16.574307 16310 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:16.628250  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:16.640338  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:16.638859  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:16.640694  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:16.641083  5552 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:16.548626 16260 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:16.643684 16248 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:16.672679 16248 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:16.674881 16248 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:16.551991 16228 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:16.553426 16390 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:16.676062  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:16.676932 16228 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:16.677027 16390 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:16.653318  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:16.618245 16257 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:16.679615 16257 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:16.648998 15682 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:16.655804 16372 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:29:16.657111  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:16.660919  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3998) took 2774 ms. Trace:
I20260709 12:29:16.662032  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:16.663187  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3591) took 1714 ms. Trace:
I20260709 12:29:16.685742  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:16.698076  5708 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:16.698606  5553 rpcz_store.cc:276] 0709 12:29:14.949217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.234995 (+285778us) service_pool.cc:224] Handling call
0709 12:29:16.663013 (+1428018us) spinlock_profiling.cc:243] Waited 56.2 ms on lock 0x7b540044e154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:16.663119 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":129317120}
I20260709 12:29:16.701390  5555 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:16.702960  5553 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 37. Current term is 38. Ops: [35.21-37.22]
I20260709 12:29:16.703199  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:16.704921  5554 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 28 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 12.
W20260709 12:29:16.706614 16433 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:29:16.703856  5556 raft_consensus.cc:2376] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28. Re-sending same reply.
I20260709 12:29:16.707571  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:16.696995  5557 rpcz_store.cc:276] 0709 12:29:13.886670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.306630 (+1419960us) service_pool.cc:224] Handling call
0709 12:29:15.306907 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:15.409578 (+102671us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:29:15.967964 (+558386us) log.cc:844] Serialized 14 byte log entry
0709 12:29:16.342659 (+374695us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:16.342779 (+   120us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:16.342866 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:16.660679 (+317813us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5400082854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:16.660726 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:16.660858 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14062,"prepare.queue_time_us":709431,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":5812,"raft.queue_time_us":40447,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":117,"spinlock_wait_cycles":235476864,"thread_start_us":905619,"threads_started":2,"wal-append.queue_time_us":581466}
W20260709 12:29:16.680192 16423 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:29:16.717391  5552 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:16.709544  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:16.648823 16388 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:16.699185 16256 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:16.712994  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:16.723718  5599 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.628s (SENT) [suppressed 17 similar messages]
I20260709 12:29:16.706405  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:16.650622 16353 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:16.748625 16353 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:16.749150  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:16.750347 16420 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.507s	user 0.000s	sys 0.001s
I20260709 12:29:16.750602  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:16.750816 16353 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:16.718205  5554 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:16.752305  6279 raft_consensus.cc:2368] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35.
I20260709 12:29:16.755553 16283 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:16.756026  5602 leader_election.cc:400] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27.
W20260709 12:29:16.710335  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 635828us
I20260709 12:29:16.756702 16279 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:16.757221 16279 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:16.757670  5632 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3633) took 1655 ms (client timeout 1691 ms). Trace:
W20260709 12:29:16.758214  5632 rpcz_store.cc:269] 0709 12:29:15.101809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.216281 (+114472us) service_pool.cc:224] Handling call
0709 12:29:16.757450 (+1541169us) spinlock_profiling.cc:243] Waited 870 ms on lock 0x7b540008bbd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:16.757588 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2000386432}
I20260709 12:29:16.760044  5631 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: []
I20260709 12:29:16.766114  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:16.719729  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:16.719020  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:16.724400  6274 raft_consensus.cc:2368] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:16.700075 16310 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:16.728701  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:16.720709  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:16.729884 16247 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:16.731524 16244 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:29:16.717386 16021 raft_consensus.cc:2764] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election decision vote started in defunct term 37: won
W20260709 12:29:16.732326  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:16.732918 16237 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:16.789983  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:16.640179 16370 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.078s
I20260709 12:29:16.680119 16407 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:16.718541 16372 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 38 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:15.299840  5446 catalog_manager.cc:5028] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): tablet 26edf338d41e42f1a5ce0d4416ec67e1 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:29:15.576112  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:16.739809  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:16.648981 16134 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.072s
I20260709 12:29:16.632092 16273 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:16.648717 16424 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.082s
I20260709 12:29:16.740530  5454 catalog_manager.cc:5697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:16.741706  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4118) took 1023 ms. Trace:
W20260709 12:29:16.743829  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:16.746017 16257 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:16.759965  5628 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3630) took 1722 ms. Trace:
I20260709 12:29:16.764529  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3632) took 1716 ms. Trace:
I20260709 12:29:16.729323  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:16.769870  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:16.771657  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3631) took 1729 ms. Trace:
I20260709 12:29:16.770000  5555 raft_consensus.cc:2376] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29. Re-sending same reply.
I20260709 12:29:16.775283  5559 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 34 index: 24. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:29:16.775275  5554 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:16.782500  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:16.786731  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:16.762791 16127 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:16.842186  5628 rpcz_store.cc:276] 0709 12:29:15.037481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.037728 (+   247us) service_pool.cc:224] Handling call
0709 12:29:16.759709 (+1721981us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b5400438d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:16.759895 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":701872256}
W20260709 12:29:16.789889  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:16.791509  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:16.791643  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:16.777220 16234 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:16.789381 16294 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:16.759670 16279 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:16.787338  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:16.797911  5446 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 1.49807e+06us
I20260709 12:29:16.794209  5704 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:16.798756  6237 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:16.843045  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:16.849561  5556 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:16.880514  5557 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:16.845275  5705 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:16.843351  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:16.886572  5636 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:16.881270  5632 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:16.785295 16228 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:16.771409 16388 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:29:16.792420  5708 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
W20260709 12:29:16.843971  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 74188.1us
I20260709 12:29:16.779724 16247 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:16.717401 15792 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:16.920791  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:16.922184  5555 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 29 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:29:16.848157 16437 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:16.884296 16021 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:16.883940  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:16.842998  5630 rpcz_store.cc:276] 0709 12:29:15.048102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.048327 (+   225us) service_pool.cc:224] Handling call
0709 12:29:15.048793 (+   466us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:15.526342 (+477549us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:15.526373 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:15.577300 (+ 50927us) log.cc:844] Serialized 52 byte log entry
0709 12:29:15.961818 (+384518us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:15.989800 (+ 27982us) log.cc:844] Serialized 14 byte log entry
0709 12:29:16.041201 (+ 51401us) log.cc:844] Serialized 14 byte log entry
0709 12:29:16.359993 (+318792us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b180024eb60. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:29:16.708981 (+348988us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x7b180024eb60. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffa66a060 00007f4ffa67c716 00007f4ffa68d287 00007f4ffa68d208
0709 12:29:16.709186 (+   205us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:16.709253 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:16.749608 (+ 40355us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:16.764167 (+ 14559us) raft_consensus.cc:1685] finished
0709 12:29:16.764201 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:16.764447 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":677009,"prepare.queue_time_us":29551,"prepare.run_cpu_time_us":139,"prepare.run_wall_time_us":12254,"raft.queue_time_us":777077,"raft.run_cpu_time_us":489,"raft.run_wall_time_us":40312,"spinlock_wait_cycles":123410304,"thread_start_us":860950,"threads_started":2,"wal-append.queue_time_us":608009}
I20260709 12:29:16.845976  5627 rpcz_store.cc:276] 0709 12:29:15.042387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.042895 (+   508us) service_pool.cc:224] Handling call
0709 12:29:15.064503 (+ 21608us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:15.577199 (+512696us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:29:15.618796 (+ 41597us) log.cc:844] Serialized 14 byte log entry
0709 12:29:16.292013 (+673217us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:16.305955 (+ 13942us) log.cc:844] Serialized 26 byte log entry
0709 12:29:16.306150 (+   195us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:16.702097 (+395947us) log.cc:844] Serialized 14 byte log entry
0709 12:29:16.771317 (+ 69220us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:16.771424 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:16.771465 (+    41us) raft_consensus.cc:1685] finished
0709 12:29:16.771475 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:16.771594 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":797188,"prepare.queue_time_us":130392,"prepare.run_cpu_time_us":148,"prepare.run_wall_time_us":3562,"raft.queue_time_us":319138,"raft.run_cpu_time_us":384,"raft.run_wall_time_us":15748,"thread_start_us":1440679,"threads_started":4,"wal-append.queue_time_us":568952}
I20260709 12:29:16.891007  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1881) took 1309 ms. Trace:
I20260709 12:29:16.886802  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:16.793560 16422 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:29:16.716920 16314 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.126s
I20260709 12:29:16.889510  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:16.902367  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:16.970541  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4123) took 1182 ms. Trace:
I20260709 12:29:16.971016  5552 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:29:16.971396  5703 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 33: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:16.909744  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:16.971120  5710 rpcz_store.cc:276] 0709 12:29:15.788184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:16.220045 (+431861us) service_pool.cc:224] Handling call
0709 12:29:16.238849 (+ 18804us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:16.239057 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:29:16.629744 (+390687us) log.cc:844] Serialized 14 byte log entry
0709 12:29:16.970182 (+340438us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:16.970237 (+    55us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:16.970285 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:16.970351 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:16.970483 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":557985,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"thread_start_us":718196,"threads_started":2}
W20260709 12:29:16.913775  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 190089us
I20260709 12:29:16.792965 16237 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:16.800040 16273 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:16.927228 16178 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:16.929553  5633 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:29:17.014767  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3640) took 1750 ms. Trace:
I20260709 12:29:17.015172  5633 rpcz_store.cc:276] 0709 12:29:15.263964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.786768 (+522804us) service_pool.cc:224] Handling call
0709 12:29:15.787074 (+   306us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:16.929235 (+1142161us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b54001d5720. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5e9dc0 00007f4ffa6741c0 00007f4ffa675231 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c
0709 12:29:17.014445 (+ 85210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:17.014700 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":388223744}
I20260709 12:29:16.930948  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2139) took 1555 ms. Trace:
I20260709 12:29:17.016461  5635 rpcz_store.cc:276] 0709 12:29:15.375805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.918793 (+542988us) service_pool.cc:224] Handling call
0709 12:29:15.919127 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:16.149542 (+230415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:16.149590 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:16.149635 (+    45us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:16.149665 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:16.930616 (+780951us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b5400439254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:16.930679 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:16.930866 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":476377344}
I20260709 12:29:16.809958  5709 rpcz_store.cc:276] 0709 12:29:15.718434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.719365 (+   931us) service_pool.cc:224] Handling call
0709 12:29:15.824634 (+105269us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:15.932161 (+107527us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:29:15.932212 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:15.932895 (+   683us) log.cc:844] Serialized 26 byte log entry
0709 12:29:16.292572 (+359677us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:29:16.292614 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:16.292689 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:16.741470 (+448781us) raft_consensus.cc:1685] finished
0709 12:29:16.741491 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:16.741646 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":83974,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"thread_start_us":334224,"threads_started":1,"wal-append.queue_time_us":563678}
I20260709 12:29:16.932233  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:16.933291  5701 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4099) took 2910 ms (client timeout 1957 ms). Trace:
W20260709 12:29:17.019193  5701 rpcz_store.cc:269] 0709 12:29:14.022621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:14.022849 (+   228us) service_pool.cc:224] Handling call
0709 12:29:16.933030 (+2910181us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54001987d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:16.933211 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2596299136}
I20260709 12:29:16.934173  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 3995) took 3082 ms. Trace:
I20260709 12:29:17.020483  5560 rpcz_store.cc:276] 0709 12:29:13.851799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:14.917919 (+1066120us) service_pool.cc:224] Handling call
0709 12:29:14.998213 (+ 80294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:15.248804 (+250591us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:16.933896 (+1685092us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b540044f554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:16.934107 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":455328000}
I20260709 12:29:16.935565  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:16.940310  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:16.894805 16228 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:16.798916 16319 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.119s
I20260709 12:29:16.799088 16248 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.119s
I20260709 12:29:16.889292 16353 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:16.940469  6271 rpcz_store.cc:276] 0709 12:29:15.581451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.581650 (+   199us) service_pool.cc:224] Handling call
0709 12:29:15.619790 (+ 38140us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:15.620205 (+   415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:15.620234 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:15.623501 (+  3267us) log.cc:844] Serialized 26 byte log entry
0709 12:29:16.236075 (+612574us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:16.236113 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:16.236195 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:16.679605 (+443410us) spinlock_profiling.cc:243] Waited 4.91 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:16.853631 (+174026us) raft_consensus.cc:1685] finished
0709 12:29:16.853669 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:16.890939 (+ 37270us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2317,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":5987,"spinlock_wait_cycles":11293056,"thread_start_us":545214,"threads_started":1,"wal-append.queue_time_us":824508}
I20260709 12:29:16.941320  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:17.033665  5556 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:16.945081  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 153665us
I20260709 12:29:16.719964 16387 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:16.963660  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 324901us
I20260709 12:29:16.964862  5557 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
I20260709 12:29:17.036099  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:17.037086  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:16.907728  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:16.985859  5705 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
W20260709 12:29:16.987840  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 98411.9us
W20260709 12:29:17.038076  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 130457us
I20260709 12:29:17.038805  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:16.939330 16021 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:16.988896  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:16.983716  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:17.045636  5555 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
W20260709 12:29:17.045783  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62163.3us
I20260709 12:29:17.046531  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:16.884729 16294 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.059324  5702 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 26. Current term is 27. Ops: [24.16-26.17]
I20260709 12:29:17.059715 15792 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:16.992705  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 204088us
I20260709 12:29:17.060839 16283 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:29:17.061388 15792 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:17.061713  5635 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:29:16.996450 16247 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:17.061978 15792 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.062680 16247 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:17.061743 16283 consensus_queue.cc:1243] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 27.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:29:16.993863  5636 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 32. Current term is 33. Ops: []
W20260709 12:29:16.999449 16438 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260709 12:29:17.010982  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3590) took 2062 ms. Trace:
I20260709 12:29:17.013401 16257 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:29:17.025832  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 227150us
I20260709 12:29:17.065618  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:16.936996 15538 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.065s
I20260709 12:29:16.964954 16236 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:17.026239  5561 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 38: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:17.046722  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:16.969626 16244 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:17.068318  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:17.052417 16134 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:29:17.069934  5561 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
W20260709 12:29:17.070218 16443 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:29:17.070648 16134 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:17.071468 16127 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:17.072052 16127 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:17.052088  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:17.065543 16257 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:17.073040 16127 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.054118  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:17.053947  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:17.056212  5706 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:29:16.941447 16437 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:17.057403  5630 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:29:17.056121 16160 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:17.065310  5559 rpcz_store.cc:276] 0709 12:29:14.948902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.677217 (+728315us) service_pool.cc:224] Handling call
0709 12:29:15.677603 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:16.846778 (+1169175us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:17.010474 (+163696us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54002283d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:17.010851 (+   377us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2523403008}
I20260709 12:29:17.078912 16160 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:29:17.079226  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:17.059841 16294 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:17.073745 16257 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:17.080632 16021 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:17.046666  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:17.083413  6275 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:29:17.085667 16294 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:17.086367 16294 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:17.086449  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:17.087003 16294 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:17.087291  6279 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 24.
I20260709 12:29:17.085807 16299 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:17.068678 16244 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.090022  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:17.090459  5526 leader_election.cc:400] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35.
I20260709 12:29:17.099431  5526 leader_election.cc:403] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Cancelling election due to peer responding with higher term
I20260709 12:29:17.053215  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 30 index: 13 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:17.029758 16228 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:17.103657  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:17.104612  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:17.105002  6277 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:17.105278  6275 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:17.105837  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1647) took 1155 ms. Trace:
I20260709 12:29:17.106242  6274 rpcz_store.cc:276] 0709 12:29:15.950876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.951037 (+   161us) service_pool.cc:224] Handling call
0709 12:29:17.105652 (+1154615us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b54002a8354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:17.105767 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1747435392}
I20260709 12:29:17.047066 16106 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:17.104104 16332 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:17.107594 16332 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:17.108492 16332 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.109117 16244 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:17.109475 16332 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:17.099068 16312 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:17.111496  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:17.112217  5704 raft_consensus.cc:2368] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35.
I20260709 12:29:16.961108 16310 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:17.113056  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:17.114128  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:17.114852  5706 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:16.989614 16300 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.059s
I20260709 12:29:17.097389  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:17.096300  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:17.120725  6276 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
W20260709 12:29:17.121681  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 81872.2us
I20260709 12:29:17.122140  6272 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 31: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:17.125903  6273 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35. Ops: [34.23-34.23]
I20260709 12:29:17.052155  5629 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:29:17.116142  5710 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:17.117405  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:17.128695  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:17.092973 16326 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.774s	user 0.008s	sys 0.004s
I20260709 12:29:17.154469  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:17.124661  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:17.064246 15792 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:17.165723  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 247935us
I20260709 12:29:17.134644 16228 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:17.090775 16353 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:17.141078 16244 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election lost for term 34. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35.
I20260709 12:29:17.141991  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:17.128698 16279 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.015s
I20260709 12:29:17.131356 16319 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:17.148057  5560 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 35 because replica has last-logged OpId of term: 34 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 22.
I20260709 12:29:17.149444  6280 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:29:17.150034  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:17.158764  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:17.163443  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:17.087312 16407 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:16.963660 16372 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 37.25, Last appended by leader: 25, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:17.170186  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:17.172068  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:17.176098  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:17.178679  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 117590us
I20260709 12:29:17.179491 16366 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.117s
W20260709 12:29:17.129796 16415 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:29:17.318823  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:17.322765  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:17.030169 16178 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:17.274262  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:17.182628 16301 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:17.187076  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.181424 16326 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:17.164783 16246 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:17.194933 16244 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:17.300904  5630 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
W20260709 12:29:17.301192  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:17.333139  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:17.311172 16455 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:29:17.164148 16280 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:17.162257 16107 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:17.162614 16278 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:29:17.298192  5454 catalog_manager.cc:5697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 26 to 29, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 29 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:29:17.308034  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 149384us
W20260709 12:29:17.176137  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:17.269905 16408 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.113s	user 0.014s	sys 0.030s
I20260709 12:29:17.181537 16106 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.117s
I20260709 12:29:17.323096  6280 raft_consensus.cc:2368] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32.
I20260709 12:29:17.323547  6279 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:29:17.324121  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49999.7us
I20260709 12:29:17.379734  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:17.316162  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:17.305867 16319 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:17.181988 16283 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:17.273314  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:17.323149 16407 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:17.385174  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:17.329243 16333 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election won for term 35
W20260709 12:29:17.419729  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:17.329124 16326 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:17.433957  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:17.335354 16301 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:29:17.415140 16408 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.263s	user 0.014s	sys 0.030s
W20260709 12:29:17.372089 16435 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
W20260709 12:29:17.429176  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 253194us
I20260709 12:29:17.318334 16338 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.101s
I20260709 12:29:17.414783 16314 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:17.372292 16244 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:17.421386 16278 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:17.424477 16236 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:17.437209 16387 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:29:17.452550  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:17.453981  5554 pending_rounds.cc:85] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 16
I20260709 12:29:17.472541  5554 pending_rounds.cc:107] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 27.17
W20260709 12:29:17.454157  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 290812us
I20260709 12:29:17.472877  5554 raft_consensus.cc:2889] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:17.454856  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.456319  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.474772  5633 raft_consensus.cc:2368] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31.
I20260709 12:29:17.456511  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.475054  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:17.423122 16455 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:17.423741 16127 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:29:17.375777 16246 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:17.451411  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:17.514534  5552 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:17.515763  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:17.516490  5552 raft_consensus.cc:2376] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32. Re-sending same reply.
I20260709 12:29:17.377841 16280 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:17.443388 16465 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
W20260709 12:29:17.470340  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 91268.4us
I20260709 12:29:17.463809 16464 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:17.437328 16326 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:17.454490  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.437188 16333 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:17.485221 16278 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:17.487082 16457 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:29:17.465129 16236 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:17.567298 16236 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:17.567585 16301 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.466141 16314 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.571835 16314 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:17.492650 16404 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:17.492825  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.572678 16404 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.572907 16333 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:17.573751  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.574697  5629 raft_consensus.cc:2410] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 31 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
I20260709 12:29:17.575119 16404 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:17.567330 16353 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:17.575735  5629 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:17.576195  5631 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 3843) took 2488 ms. Trace:
I20260709 12:29:17.493228  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.576647  5631 rpcz_store.cc:276] 0709 12:29:15.088215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.578316 (+490101us) service_pool.cc:224] Handling call
0709 12:29:16.759852 (+1181536us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:16.761019 (+  1167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:17.576015 (+814996us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400438d50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:17.576143 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2711908608}
I20260709 12:29:17.494915 16178 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:17.500898 16244 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:17.437309 16250 consensus_queue.cc:1048] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.039s
I20260709 12:29:17.505962  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:17.593514  6272 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:17.506925  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:17.508224  6271 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:29:17.511169 16237 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:17.512305  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:17.469348 16343 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.376s	user 0.009s	sys 0.005s
I20260709 12:29:17.519884  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:17.523185  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:17.599329  5631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e69273530292453d87614627debcc13e"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:17.600065  5631 raft_consensus.cc:606] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:17.600519  5631 raft_consensus.cc:614] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:17.434320 16407 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:29:17.536896  5556 consensus_meta.cc:287] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.503s	user 0.011s	sys 0.000s
I20260709 12:29:17.535315  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:17.603178  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 67939.7us
I20260709 12:29:17.462122 16274 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:17.536105  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:17.544787  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.547220  5553 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:17.513512 16455 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.550268 16359 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:17.551378 16280 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:17.551708  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44405.5us
W20260709 12:29:17.553122 16446 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.583s	user 0.003s	sys 0.000s
I20260709 12:29:17.554046  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.622316  5627 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 38.
I20260709 12:29:17.444151 16390 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.070s
I20260709 12:29:17.557402  6280 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:29:17.557936  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:17.541882  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:17.624672  6277 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:17.577327  5628 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:29:17.500080 16283 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:17.542768  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 226653us
I20260709 12:29:17.595978  6275 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:17.542727 16464 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:17.605543 16138 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:17.535105  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:17.629508  5601 leader_election.cc:400] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32.
I20260709 12:29:17.629873  5601 leader_election.cc:403] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20260709 12:29:17.558135 16228 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:17.601835  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 82087us
I20260709 12:29:17.604447  5556 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
W20260709 12:29:17.606315  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 70295.2us
I20260709 12:29:17.606554  5630 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:17.607393  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:17.542397 16107 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.063s
I20260709 12:29:17.612848 16332 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:29:17.664826  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:17.613062 16441 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:29:17.610445  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:17.673892  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 63538.6us
I20260709 12:29:17.674698  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:17.615257 16455 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:17.580710 16021 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.541646 16426 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.063s
I20260709 12:29:17.565958 16294 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.630995  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:17.632077  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 78762.4us
W20260709 12:29:17.621695 16310 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.006s	sys 0.026s
I20260709 12:29:17.662300  5526 leader_election.cc:400] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:17.663017  6274 raft_consensus.cc:2410] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 31 because replica has last-logged OpId of term: 29 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 19.
I20260709 12:29:17.618326 16244 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:17.620205 16237 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:17.609985 16274 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.664768  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:17.678314  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:17.604260 16343 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.686618  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.728534  5630 raft_consensus.cc:2410] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 29 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 14.
I20260709 12:29:17.729640  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.729796  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.730695  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.658476 16464 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:17.712947  5526 leader_election.cc:403] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
W20260709 12:29:17.736642  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 74533.6us
W20260709 12:29:17.713357  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 82463.9us
I20260709 12:29:17.738283  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:17.699062  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:17.620234 16474 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:29:17.697355  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:17.741022  5631 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 33: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:17.665277 16234 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:17.666421 16280 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:17.629057 16283 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:17.659072 16138 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:17.714759  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:17.657176 16470 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32.
W20260709 12:29:17.756506  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57698us
I20260709 12:29:17.709311  5628 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: [31.16-31.16]
I20260709 12:29:17.668596 15538 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:17.742223  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:17.716629 16310 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.006s	sys 0.027s
I20260709 12:29:17.688709 16134 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:17.753011  5636 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:17.679565 16372 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:17.657445 16472 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.077s
I20260709 12:29:17.716325 16244 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:17.732511  5630 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:29:17.716940 16402 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:17.743264 16234 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:17.759320  5557 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 33: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:17.751006 16280 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:17.762933  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:17.751824 16353 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
W20260709 12:29:17.778232  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:17.814046  6279 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:17.716715 16256 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.048s
I20260709 12:29:17.743870 16474 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:17.757898  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:17.756407 16138 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:17.761024  5627 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:17.817685  5559 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:17.765897 16387 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:17.784164 16244 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:17.770403 16134 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:17.790612 16106 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:17.818992  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.771000 16419 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:17.824988  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.791397 16234 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:17.826814  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:17.827910  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70156.4us
I20260709 12:29:17.831425  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:17.771298 15538 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:17.847366  5627 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:29:17.851677  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:17.852980  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:17.827618 16134 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:17.819795 16474 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:17.829782 16314 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:29:17.831418 16385 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:17.860713  5524 leader_election.cc:400] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35.
I20260709 12:29:17.862807  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:17.864125  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 125397us
W20260709 12:29:17.864518  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 149827us
I20260709 12:29:17.865259  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:17.763886 16407 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 32.14, Last appended by leader: 14, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.869127  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:17.883055  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:17.850247 16106 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:17.880424  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:17.883541  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:17.892874  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:17.882488  5710 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
I20260709 12:29:17.894280  5708 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:29:17.922083  5704 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 33: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:17.921947  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:17.923700  5701 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
W20260709 12:29:17.927531  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 206445us
I20260709 12:29:17.929791  5706 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:17.858830 16256 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:17.832998 16419 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.864805 16250 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:17.859935 16314 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 26 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:17.884316 16279 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:17.917506 16178 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:17.916101 16285 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:17.919215 16247 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:17.943532 16488 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:29:17.929214 16470 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.019s
I20260709 12:29:17.949131 16106 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:17.951128 16483 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:29:17.980609  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:17.952214 16480 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
W20260709 12:29:17.953006 16476 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 12:29:17.938207 16021 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:17.955539 15538 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:17.855324 16280 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 30.23, Last appended by leader: 23, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:17.956741 16482 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
I20260709 12:29:17.959888 16260 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:17.960965 16314 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:17.866918 16390 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.077s
I20260709 12:29:17.875501 16294 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:17.964704  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:17.956175  6275 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:17.941494  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:17.951731 16236 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:18.014482  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 73185.3us
I20260709 12:29:18.015352  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:18.016554 16021 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:18.017279 16021 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:18.017625 16021 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:18.017966 16021 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:18.021339 16314 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:29:18.021790  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 57154.1us
I20260709 12:29:18.022550  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:18.023699  5675 leader_election.cc:400] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31.
I20260709 12:29:18.024580 15538 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:18.025496 15538 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:18.026974  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:18.027220 16236 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:18.028827 16294 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:18.029407 16294 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:18.029983 16294 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:17.968837  5627 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:18.031504  5627 raft_consensus.cc:606] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:18.031857  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:17.974543  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:17.968624 16488 consensus_queue.cc:1243] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer f613ba929b1046d3b5aed909414a95bf log is divergent from this leader: its last log entry 29.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:29:18.003024 16279 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:18.039968  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:18.052977  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:29:18.002485 16343 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:17.972919 16285 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:17.981374 16247 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.060843  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:18.061293  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:18.061617  5630 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 37 index: 25. Preceding OpId from leader: term: 38 index: 26. (index mismatch)
I20260709 12:29:17.982944 16134 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.053009 16479 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:18.061575  5627 raft_consensus.cc:2376] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27. Re-sending same reply.
I20260709 12:29:18.067667  5701 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:29:18.068631  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:18.010993 16474 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.047447 15538 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:18.074584  5706 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:18.042876 16426 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.029s
I20260709 12:29:18.126333  5707 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:18.057319 16279 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.127790  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:18.125030  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:18.076726 16326 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.501s	user 0.012s	sys 0.000s
I20260709 12:29:18.131510  5706 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 33: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:18.137629 16279 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:18.140945 16492 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:29:18.141170 16247 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:18.141984 16437 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:18.142987 16285 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:18.152057 16489 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:29:18.152891  6278 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:29:18.153529  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3603) took 3195 ms. Trace:
I20260709 12:29:18.154520  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:18.153914  5558 rpcz_store.cc:276] 0709 12:29:14.958429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:15.781113 (+822684us) service_pool.cc:224] Handling call
0709 12:29:15.781450 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:17.764113 (+1982663us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:17.764162 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:17.764224 (+    62us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:29:17.764272 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:18.153216 (+388944us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b540044e654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:18.153266 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:18.153460 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5275,"raft.queue_time_us":6797,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":12756,"spinlock_wait_cycles":4550098304}
W20260709 12:29:18.156966 16257 consensus_meta.cc:287] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 1.076s	user 0.013s	sys 0.000s
I20260709 12:29:18.157526 16257 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:18.161616 15682 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:18.162508 15682 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:18.165313 16134 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:18.147765 16138 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.156878  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3622) took 1430 ms. Trace:
I20260709 12:29:18.161883 16426 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:18.167567 16426 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:18.167892 16464 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:18.168828 16326 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:18.173887  5629 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: [32.19-32.19]
I20260709 12:29:18.174263 16138 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:18.173382 16426 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.175869 16426 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:18.074175 15538 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:18.177487  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:18.178169  5701 raft_consensus.cc:2368] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35.
I20260709 12:29:18.165706 16366 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:18.176754 16359 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.555s	user 0.010s	sys 0.002s
I20260709 12:29:18.193236  5703 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:18.166512 16372 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:18.195466  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:18.182937 15538 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:18.195349  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:18.201730  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:18.201040  5701 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:18.178684  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:18.166393  5555 rpcz_store.cc:276] 0709 12:29:16.726298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:17.068693 (+342395us) service_pool.cc:224] Handling call
0709 12:29:17.069093 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:17.130606 (+ 61513us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:29:17.130654 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:17.130715 (+    61us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:17.130760 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:18.156620 (+1025860us) spinlock_profiling.cc:243] Waited 40.7 ms on lock 0x7b540023f854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:18.156677 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:18.156812 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20395,"raft.queue_time_us":33050,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101,"spinlock_wait_cycles":93545728}
I20260709 12:29:18.216542 16257 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:18.216989  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:18.217643  5555 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:18.220047  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:18.181087  5558 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: [28.17-28.17]
I20260709 12:29:18.188369 16366 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.221179  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:18.198276 16449 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:18.225260 16449 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.226737  5555 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:18.210911  5703 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:29:18.229482  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:18.230343  5601 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:18.232785 16401 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:18.245872  5707 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:18.233683 16407 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:18.234375 16319 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:18.235356  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 570679us
I20260709 12:29:18.235720  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:18.243124  5705 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:18.259171  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:18.259644  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:18.260690  6275 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:18.261943  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:18.246599 16326 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:29:18.224146 16366 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:18.257148  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:18.270411  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:18.268901  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:18.271999  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3623) took 1545 ms. Trace:
I20260709 12:29:18.272228  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:18.272648  5557 rpcz_store.cc:276] 0709 12:29:16.726583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:17.036827 (+310244us) service_pool.cc:224] Handling call
0709 12:29:18.271729 (+1234902us) spinlock_profiling.cc:243] Waited 721 ms on lock 0x7b54000271d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:18.271917 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1659368704}
W20260709 12:29:18.278363  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:18.279044  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:18.279956  6278 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:18.246240 16333 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:18.254258 16464 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:18.292987  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 2124) took 1528 ms. Trace:
I20260709 12:29:18.287710  6277 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:29:18.299140  6275 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:18.299321  5560 rpcz_store.cc:276] 0709 12:29:16.770260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:17.021779 (+251519us) service_pool.cc:224] Handling call
0709 12:29:18.292715 (+1270936us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b540041a0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:18.292905 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":255270144}
I20260709 12:29:18.256335 16471 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:18.235311 16300 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.059s
I20260709 12:29:18.294997 16257 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:18.297250 16387 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:18.301522 16236 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:18.374815  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:18.375531  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:18.380663  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 111751us
I20260709 12:29:18.320600 16332 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:18.304445 16234 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:18.331846 16471 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:18.394701  5704 pending_rounds.cc:85] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 14
I20260709 12:29:18.398729  5704 pending_rounds.cc:107] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 29.15
I20260709 12:29:18.399227  5704 raft_consensus.cc:2889] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:18.368988 16401 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:29:18.389423 16283 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.623s	user 0.013s	sys 0.007s
I20260709 12:29:18.387950 16319 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:18.381716 16407 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:18.369382 16257 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.391705 16273 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:18.387502 16236 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.452109  5710 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:29:18.423326  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:18.405180 16401 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:18.388310 16387 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:18.414420 16244 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.582s	user 0.012s	sys 0.002s
I20260709 12:29:18.394097 16332 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.437381  6236 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:18.450927  5708 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 14. Preceding OpId from leader: term: 34 index: 15. (index mismatch)
I20260709 12:29:18.432857 16474 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:18.456333  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:18.414431 16283 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:18.482779  5701 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 24. Current term is 25. Ops: []
I20260709 12:29:18.443711 16260 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:18.477331 16244 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.444654 16319 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.488538 16283 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:18.489785 16244 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:18.490996 16319 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:18.494251 16502 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:29:18.494549 16260 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:18.496384 16498 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
W20260709 12:29:18.500685 16503 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.067s	user 0.002s	sys 0.000s
I20260709 12:29:18.478015 16512 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:18.504004 16512 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.509120  6274 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
W20260709 12:29:18.514458  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 77176.7us
I20260709 12:29:18.469405 16333 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:18.522241 16228 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:29:18.542694 16314 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.521s	user 0.008s	sys 0.004s
I20260709 12:29:18.545925  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:18.546291 16021 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.528s	user 0.012s	sys 0.002s
I20260709 12:29:18.546813 16021 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:18.547688 16387 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:18.548525 16407 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.549113 16512 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:18.549858 16512 raft_consensus.cc:2764] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election decision vote started in defunct term 30: won
W20260709 12:29:18.550870 16178 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.588s	user 0.007s	sys 0.006s
I20260709 12:29:18.552237 16333 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 29.22, Last appended by leader: 22, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.544246 16332 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:18.514351 16513 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.053s
I20260709 12:29:18.550554 16512 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:18.551270 16178 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:18.580124  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:18.557663 16504 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
W20260709 12:29:18.557008 16499 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
W20260709 12:29:18.555078 16294 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.509s	user 0.011s	sys 0.000s
I20260709 12:29:18.561477 16513 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Stepping down as leader of term 33
W20260709 12:29:18.586833  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 315014us
I20260709 12:29:18.519742 16471 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.590375  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:18.591758 16319 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:18.583384 16021 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:18.592757 16484 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.745s	user 0.005s	sys 0.000s
I20260709 12:29:18.592752  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:18.593747 16319 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:29:18.594417  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:18.595024 16319 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:18.596050 16246 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:18.598251  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:18.599246  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3725) took 1449 ms. Trace:
I20260709 12:29:18.599649  5709 rpcz_store.cc:276] 0709 12:29:17.149655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:17.167441 (+ 17786us) service_pool.cc:224] Handling call
0709 12:29:17.258818 (+ 91377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:18.195399 (+936581us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:18.195428 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:18.195472 (+    44us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:18.195505 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:18.599020 (+403515us) spinlock_profiling.cc:243] Waited 936 ms on lock 0x7b5400309654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:18.599067 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:18.599194 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5950,"raft.queue_time_us":2592,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":4718,"spinlock_wait_cycles":2153269504}
I20260709 12:29:18.600585  5705 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:29:18.603801  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:18.604471  5629 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 38.
I20260709 12:29:18.556429 16257 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:18.605464  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:18.605943  5707 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:29:18.607298  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:18.588263 16513 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:18.620154  5560 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:29:18.625404 16514 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:18.626029 16514 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.626509 16518 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:29:18.627199 16518 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:18.627622 16518 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:18.632740  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:18.633499  5633 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:29:18.635361  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:18.636264 16512 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:18.636709 16512 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:18.637065 16512 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:18.601830 16283 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:18.640664 16283 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:18.641153 16514 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:18.616317 16510 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 12:29:18.622965 16273 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.023s
I20260709 12:29:18.639510 15538 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:18.614923 16408 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:29:18.615307 16366 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.042s
I20260709 12:29:18.654352 16404 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:18.654773 15538 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:18.655061 16404 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:18.654414 16512 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.658404 16404 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:18.664570  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:18.665380  5633 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:18.535981 16401 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.674743 16475 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:18.674981 16512 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:18.675318 16475 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:18.677927 16501 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260709 12:29:18.682348 16512 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:18.683501 16512 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.651239 16513 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.689960 16513 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:18.689988  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:18.690485  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3645) took 1235 ms. Trace:
I20260709 12:29:18.691018  5635 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:29:18.690972  5636 rpcz_store.cc:276] 0709 12:29:17.455527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:17.455667 (+   140us) service_pool.cc:224] Handling call
0709 12:29:17.455932 (+   265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:17.783223 (+327291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:18.690225 (+907002us) spinlock_profiling.cc:243] Waited 280 ms on lock 0x7b5400438854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:18.690428 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":643042048}
I20260709 12:29:18.669095 16301 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:29:18.692889 16301 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:18.693086  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:18.694543 16301 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.667260 16283 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:18.696827 16449 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:18.599587 16294 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:18.699517  5601 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:18.701157  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:18.703794  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2126) took 1930 ms. Trace:
I20260709 12:29:18.706296  5561 rpcz_store.cc:276] 0709 12:29:16.773144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:17.071815 (+298671us) service_pool.cc:224] Handling call
0709 12:29:17.072094 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:17.258156 (+186062us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:17.258184 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:17.258218 (+    34us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:17.258248 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:18.703481 (+1445233us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54000267d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:18.703530 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:18.703714 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":425020288}
I20260709 12:29:18.696626 16514 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:18.697283 16512 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:18.723100  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 114837us
I20260709 12:29:18.704402 16279 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:18.718796 16510 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:18.716588 16514 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:18.718250 16401 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:29:18.720142 16487 consensus_queue.cc:1175] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f613ba929b1046d3b5aed909414a95bf" responder_term: 26 status { last_received { term: 26 index: 19 } last_committed_idx: 17 last_received_current_leader { term: 26 index: 19 } }
I20260709 12:29:18.737529 16514 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:18.728585 16279 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:18.767068  5633 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:29:18.768464  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:18.771013  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:18.778851  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:18.779922  5555 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:18.746371 16518 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.771474  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:18.796823  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:18.799942  5559 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:29:18.804801  5554 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 16. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
I20260709 12:29:18.700562 16475 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.812335  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:18.812695  5556 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:18.763036 16510 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:18.807886  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:18.809973  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:18.814585  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:18.810760  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:18.767446 16279 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:18.816428  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:18.818329  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:18.819707  5554 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 30 index: 13. Preceding OpId from leader: term: 34 index: 15. (index mismatch)
W20260709 12:29:18.826354  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 160127us
W20260709 12:29:18.829192  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:18.834432  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:18.835224  6273 raft_consensus.cc:2368] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31.
I20260709 12:29:18.835511  5555 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:18.839046  5554 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:29:18.813623 16512 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:29:18.835100  6277 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:18.848332  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:18.840611  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:18.844905  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:18.853549  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:18.847638  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:18.853636  6277 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 38. Current term is 39. Ops: []
I20260709 12:29:18.854406  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:18.854434  5634 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:18.846357  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:18.857863  6274 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:29:18.859401  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:18.861750  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:18.862080  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:18.863651  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:18.866906  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:18.869830  5524 leader_election.cc:400] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35.
I20260709 12:29:18.881471  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:18.877964  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:18.882561  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:18.878824  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:18.889521  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:18.890419  6274 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:18.841125 16407 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:18.892306 16372 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:18.849254 16475 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:18.892786  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:18.867828 16461 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:18.908507  5634 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
W20260709 12:29:18.913988  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:18.881541 16370 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:29:18.880793 16387 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:18.915887  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:18.892688 16359 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.009s
I20260709 12:29:18.928468  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:18.926256  6280 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:18.897382 16426 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:29:18.927196  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:18.877769 16134 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 33.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.038s
I20260709 12:29:18.881543 16178 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.037s
I20260709 12:29:18.881090 16449 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 13, Time since last communication: 0.042s
I20260709 12:29:18.884801 16332 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.039s
I20260709 12:29:18.893458 16314 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.039s
I20260709 12:29:18.925415 16257 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:18.932658 16294 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:18.902045 16407 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.973280  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:18.975865  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 112953us
I20260709 12:29:18.981889  5631 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 38.
I20260709 12:29:18.915974 16510 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.960513 16426 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:18.991540 16481 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.142s	user 0.003s	sys 0.000s
W20260709 12:29:18.993237 16523 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:29:18.997540 16532 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:18.998545 16532 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:18.981954 16518 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:19.009394 16520 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.130s	user 0.002s	sys 0.000s
I20260709 12:29:18.986873 16512 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:19.019333 16407 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:19.022331  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:19.022814 16274 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:19.022370  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3632) took 1569 ms. Trace:
I20260709 12:29:19.024123  5552 rpcz_store.cc:276] 0709 12:29:17.453292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:17.517201 (+ 63909us) service_pool.cc:224] Handling call
0709 12:29:17.518195 (+   994us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:17.536831 (+ 18636us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:17.536863 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:17.543288 (+  6425us) log.cc:844] Serialized 26 byte log entry
0709 12:29:17.697317 (+154029us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:17.697377 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:17.697494 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:18.990917 (+1293423us) spinlock_profiling.cc:243] Waited 3.97 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:19.022011 (+ 31094us) raft_consensus.cc:1685] finished
0709 12:29:19.022035 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:19.022310 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18177,"raft.queue_time_us":21183,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":21135,"spinlock_wait_cycles":9141504,"thread_start_us":153551,"threads_started":1,"wal-append.queue_time_us":243144}
I20260709 12:29:19.022421  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:19.027782 16426 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:19.017204 16512 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.029294 16280 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:19.036461  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:19.036691  6279 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:19.037180  5710 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:18.984477 16372 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:19.038810  5601 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:29:19.046415  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:19.055047  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 72554.5us
I20260709 12:29:19.069653  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:19.070802  5707 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:19.073953  5705 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:29:19.081724  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:19.082371  5629 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:19.037982 16273 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:19.066586 16526 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:29:19.089936  5707 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
W20260709 12:29:19.091706  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:19.095039  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:19.095137 16514 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:19.101596  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 208891us
I20260709 12:29:19.110296  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:19.112977  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 74247.6us
W20260709 12:29:19.112982  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 65064us
I20260709 12:29:19.124195  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:19.082901 16283 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:19.140327  5629 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 39.
I20260709 12:29:19.143781  5710 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:19.096822 16537 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:19.182004  5675 leader_election.cc:400] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31.
I20260709 12:29:19.182931  5675 leader_election.cc:403] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:29:19.132172 16228 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:19.197767  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:19.106097 16273 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:19.215240  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 120262us
I20260709 12:29:19.187014 16134 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:19.196135 16407 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:29:19.210590 16513 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.520s	user 0.010s	sys 0.009s
I20260709 12:29:19.190001 16542 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:19.237241  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 123534us
W20260709 12:29:19.251119  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:19.184993 16283 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.227871 16407 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:19.185271 16537 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:19.227423 16535 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:19.223376 16546 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:19.211436 16228 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.241302 16475 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:19.270330 16532 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:19.271286 16546 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:19.273954 16514 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:19.278611  6279 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:29:19.248857 16359 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.030s
W20260709 12:29:19.282089 16547 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260709 12:29:19.283073 16537 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:19.275358 16535 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.285044 16283 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:19.264463 16407 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:19.286404  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:19.287065  5557 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:29:19.286891 16279 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.288726  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:19.289513  5628 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:19.248746 16542 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.291971 16542 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:19.293422 16419 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:19.238905 16134 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:19.307212  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 220894us
W20260709 12:29:19.326320  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 139409us
I20260709 12:29:19.328177  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:19.331768  5629 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: [24.16-26.17]
I20260709 12:29:19.332590  5628 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 38. Current term is 39. Ops: [38.26-38.26]
I20260709 12:29:19.333027  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:19.333686  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:19.334378  5634 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: [32.19-32.19]
I20260709 12:29:19.333658  6273 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:19.260308 16541 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31.
I20260709 12:29:19.266480 16550 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:19.262522 16474 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.087s
I20260709 12:29:19.346778 16542 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:19.364050  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:19.353489  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:19.339592 16549 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:19.370405  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:19.372156  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:19.353211 16475 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:29:19.378661  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 161012us
I20260709 12:29:19.381582  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:19.354075 16021 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.022s
I20260709 12:29:19.368166 16542 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:19.356438 16513 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 18, Time since last communication: 0.024s
I20260709 12:29:19.375742 16535 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:19.382453 16557 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:19.397202  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:19.400074  6273 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:19.386241 16426 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:19.405840  6279 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 26. Current term is 27. Ops: [26.17-26.17]
I20260709 12:29:19.391341 16542 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:19.359136 16550 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.407330 16557 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:19.398451 16279 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:19.408867 16280 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:19.412319  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:19.413841  5634 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: [32.19-32.19]
I20260709 12:29:19.417555  6273 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:19.424784 16280 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:19.427973 16280 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:19.429265  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4025) took 1764 ms. Trace:
I20260709 12:29:19.433183 16257 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:19.433902 16257 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:19.437237 16273 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:19.443102  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:19.411345 16545 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:29:19.444799 16228 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:19.413436 16563 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:19.445746 16563 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:19.446264  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:19.446924  6276 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:19.446658 16563 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:19.447167  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:19.447887  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:19.448693  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:19.449565  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:19.450124  6277 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:19.450259  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:19.451275  5553 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 38: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:19.413266 16474 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:19.453042 16474 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.453475  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:19.454739 16560 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:19.454917  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:19.456084 16426 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:19.410732 16548 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:29:19.409067 16541 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.050s
I20260709 12:29:19.457176 16539 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:29:19.457075 16426 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:19.457758 16539 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 26 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:19.457891 16541 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:19.458384 16541 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:19.452363  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:19.459270  5557 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:19.458940 16539 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.414140 16359 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 37.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.049s
I20260709 12:29:19.460481 16539 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:19.402863 16510 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.058s
I20260709 12:29:19.461189 15682 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.463647 16545 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 38 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:19.464390 16557 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:19.464774 16545 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:19.467056  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 54809.9us
I20260709 12:29:19.468024 16407 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.468694 16565 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:19.469254 16565 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:19.469668 16565 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:29:19.416393 16401 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.657s	user 0.017s	sys 0.000s
I20260709 12:29:19.471971  5601 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:19.472144 16370 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:19.490056  5703 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: [31.21-31.21]
W20260709 12:29:19.491096  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44307.2us
I20260709 12:29:19.429826  5558 rpcz_store.cc:276] 0709 12:29:17.665055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:18.155612 (+490557us) service_pool.cc:224] Handling call
0709 12:29:18.156001 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:18.220787 (+ 64786us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:19.428742 (+1207955us) spinlock_profiling.cc:243] Waited 24.8 ms on lock 0x7b5400082354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:19.429173 (+   431us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":57152896}
I20260709 12:29:19.474702  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:19.475584  5561 raft_consensus.cc:2410] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 36 because replica has last-logged OpId of term: 34 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 22.
I20260709 12:29:19.480046  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:19.506292  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1724) took 1061 ms. Trace:
I20260709 12:29:19.506863  6275 rpcz_store.cc:276] 0709 12:29:18.445274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:18.446822 (+  1548us) service_pool.cc:224] Handling call
0709 12:29:18.447300 (+   478us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:18.534522 (+ 87222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:18.534577 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:18.535362 (+   785us) log.cc:844] Serialized 26 byte log entry
0709 12:29:18.716477 (+181115us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:18.716534 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:19.505566 (+789032us) spinlock_profiling.cc:243] Waited 60.9 ms on lock 0x7b54002ce7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:19.505645 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:19.505723 (+    78us) raft_consensus.cc:1685] finished
0709 12:29:19.505743 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:19.506205 (+   462us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15177,"raft.queue_time_us":28294,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":16004,"spinlock_wait_cycles":140095872,"thread_start_us":128605,"threads_started":1,"wal-append.queue_time_us":215974}
I20260709 12:29:19.510488 16550 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:19.480746  5526 connection.cc:256] RPC call timeout handler was delayed by -1.0679888409999876s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1778s for a response
I20260709 12:29:19.482446  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:19.518217  5526 connection.cc:256] RPC call timeout handler was delayed by -1.0481888409999556s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18000000000000002s for a response
I20260709 12:29:19.518626  5710 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:19.519763  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:19.527684 16390 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:19.529464 16332 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:19.530112 16332 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:19.484453  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:19.534602 16332 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:19.537005  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:19.539134  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:19.540699  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:19.541680  5553 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 36: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:19.542979  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:19.544732  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:19.546442  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:19.551621  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:19.472709 16475 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:19.566017  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:19.566540  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:19.570044  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 50377.6us
I20260709 12:29:19.570102  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:19.571013  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:19.571627  5634 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:19.565557  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:19.560463 16228 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.016s
W20260709 12:29:19.578708  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 94395us
W20260709 12:29:19.581183  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:19.582196  6271 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29. Ops: []
W20260709 12:29:19.573465  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 203048us
I20260709 12:29:19.586606  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:19.587662  5629 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:19.560087 16300 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:19.574796  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:19.488926 16370 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:19.599715  5558 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:19.604189  5710 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29. Ops: [27.15-28.16]
I20260709 12:29:19.604809  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:19.605736  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:19.611253  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:19.614555  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:19.615694  5631 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 39 because replica has last-logged OpId of term: 37 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 21.
I20260709 12:29:19.571959 16280 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:19.616768  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:19.621263 16478 log.cc:927] Time spent T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.905s	user 0.001s	sys 0.004s
I20260709 12:29:19.629298 16236 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:19.639652 15682 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:19.640574 16134 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:19.641337 16301 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:19.459187 16541 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.575354 16475 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.648579 16407 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:19.577316 16550 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:29:19.659420  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:19.660431  5558 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:19.662034  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:19.656446  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:19.664024  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:19.669301  6277 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:19.670934  5634 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:29:19.672427  6275 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:19.694715  5635 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:29:19.597579 16404 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.088s
W20260709 12:29:19.677811 16278 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.139s	user 0.008s	sys 0.034s
I20260709 12:29:19.696120  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:19.676380 16515 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.102s	user 0.005s	sys 0.001s
I20260709 12:29:19.606097 16390 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.087s
W20260709 12:29:19.706658 16278 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.168s	user 0.008s	sys 0.034s
I20260709 12:29:19.698001  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:19.699674  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:19.701365  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:19.653930 16419 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.050s
I20260709 12:29:19.721930 16475 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:19.722719 16273 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:19.723647 16273 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:19.724843 16273 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.750399 16565 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.752043  5707 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 36: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:19.758615 16565 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:19.739989 16542 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.769268  5703 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: [31.22-31.22]
I20260709 12:29:19.771420 16542 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:19.773939 16419 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:19.762581 16234 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:19.693061 16550 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 39 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:19.774688 16419 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:19.824149  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 351377us
I20260709 12:29:19.826342  5635 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:19.827251  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:19.828164  5627 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
W20260709 12:29:19.831943  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:19.719530 16370 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:19.796584 16257 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:19.860191 16550 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 37.25, Last appended by leader: 25, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.862566  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3736) took 1804 ms. Trace:
I20260709 12:29:19.865936 16419 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:19.856649 15538 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:19.894470  5706 rpcz_store.cc:276] 0709 12:29:18.057972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:18.266988 (+209016us) service_pool.cc:224] Handling call
0709 12:29:18.267433 (+   445us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:18.327792 (+ 60359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:18.327849 (+    57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:18.332766 (+  4917us) log.cc:844] Serialized 52 byte log entry
0709 12:29:18.455930 (+123164us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:18.507825 (+ 51895us) log.cc:844] Serialized 14 byte log entry
0709 12:29:18.508037 (+   212us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:18.508108 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:19.648594 (+1140486us) spinlock_profiling.cc:243] Waited 28.3 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:19.862227 (+213633us) raft_consensus.cc:1685] finished
0709 12:29:19.862334 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:19.862502 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":59726,"prepare.queue_time_us":54643,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":62,"raft.queue_time_us":62611,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":48221,"spinlock_wait_cycles":65063168,"thread_start_us":61430,"threads_started":1,"wal-append.queue_time_us":151352}
I20260709 12:29:19.901150 15538 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:19.907694  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:19.913903  5633 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:29:19.899839 16280 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:19.920432  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:19.926100  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 317665us
I20260709 12:29:19.918592 15538 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:19.928012  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:19.924445 16441 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:19.925244 16353 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:19.929631 16280 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:19.940124 16441 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.944566  6278 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:19.942518 16353 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:19.942044 16280 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:19.950655 16588 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20260709 12:29:19.947901 16506 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.407s	user 0.005s	sys 0.003s
I20260709 12:29:19.949033 16419 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.011s
I20260709 12:29:19.949517 16585 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:19.953176 16280 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:19.952695 16353 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:19.963503 16332 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:19.982558 16585 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:19.994164 16585 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:19.995751 16449 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:19.982738 16257 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:19.987751 16579 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:19.997745 16579 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:19.998122 16579 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:19.998875 16441 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:20.006084  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 387775us
I20260709 12:29:20.016927  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:20.017737  6277 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:20.018047  5635 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:29:20.021342  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:20.023891  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 98389.3us
I20260709 12:29:20.023972  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:20.024755  6279 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:20.012081 16300 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:20.027020  5703 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 35 index: 23. (term mismatch)
I20260709 12:29:20.034579  5703 pending_rounds.cc:85] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 22
I20260709 12:29:20.027904  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:20.034859  5703 pending_rounds.cc:107] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 34.23
I20260709 12:29:20.035223  5703 raft_consensus.cc:2889] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:20.022044  5628 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:20.039597  5702 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:29:20.039808  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:20.040571  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 214505us
I20260709 12:29:20.038795  6236 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:20.023272 15538 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:20.043381 16300 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.043823 15538 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:20.046047  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:20.046321 15538 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:20.047081 15682 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 13, Time since last communication: 0.008s
I20260709 12:29:20.020802 16449 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:20.053992  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:20.054327  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:20.054872  5704 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:20.054031  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:20.056468  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:20.057233  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:20.060552  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 123621us
I20260709 12:29:20.033560  6277 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:19.995848 16585 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:20.061400  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:20.085556  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:20.091976  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:20.092892  6271 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:20.073266 16585 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:20.072696 16461 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:20.095305  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:20.106844  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:20.106993  6277 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:20.107491  6274 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:20.104671 16461 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:20.098124 16449 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:20.100857 16585 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:20.111236 16449 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:20.111475 16585 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:20.109522 16590 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 30.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:20.109719 16461 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:20.120652  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:20.120841  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:20.098459 16319 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:20.106227 16510 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:20.116319 16301 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.025s
I20260709 12:29:20.119964 16106 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.026s
I20260709 12:29:20.124392 16470 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 35.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.023s
I20260709 12:29:20.141494 16510 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.172552 16590 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:20.173465 16590 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:20.173931 16590 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:20.175704 16579 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.228408  5454 catalog_manager.cc:5697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 20 to 24. New cstate: current_term: 24 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:18.815904  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:29:20.241838  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
W20260709 12:29:20.250602  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 129881us
I20260709 12:29:20.108162  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:20.261709 16134 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:20.262699 16134 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:20.263283 16134 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:20.265590  6273 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 39 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 38. Current term is 39. Ops: [38.26-38.26]
I20260709 12:29:20.267340  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:20.268635  6276 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:20.268702  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3670) took 1414 ms. Trace:
I20260709 12:29:20.269992  6274 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
W20260709 12:29:20.262122  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 176404us
I20260709 12:29:20.272534  5633 rpcz_store.cc:276] 0709 12:29:18.853974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:18.854153 (+   179us) service_pool.cc:224] Handling call
0709 12:29:20.268100 (+1413947us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540018d154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:20.268407 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2406880128}
I20260709 12:29:20.275475  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:20.278256  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 170093us
I20260709 12:29:20.272876  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:20.279937  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:20.180723 16419 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:20.177141 16510 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:20.310094 16604 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:29:20.310802  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:20.312817 16579 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:20.313395 16470 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:20.317953 16426 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:20.318454 16449 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:20.319103 16426 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.319077 16449 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:20.333940 16449 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:20.359989  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:20.384035  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 104504us
I20260709 12:29:20.385754  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:20.387442 16470 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:29:20.388751  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 100851us
I20260709 12:29:20.388855 16485 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:20.390596  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:20.391366 16485 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:29:20.394395  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 332942us
I20260709 12:29:20.396323 16607 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:20.396939  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:20.397207 16607 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:20.396950  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:20.336440 16604 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:20.384014  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:20.407191 16372 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:20.407765 16106 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:20.407210 16563 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:20.408737 16106 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:20.409904 16579 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:20.410758 16579 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.335988 16419 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:20.412653 16426 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:20.413823 16579 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:20.414332 16608 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:29:20.413852 16419 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 20, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.417955 16419 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:20.420576 16603 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:20.420688 16579 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:20.421510 16579 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:20.421643 16603 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:20.425055 16579 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.429210 16604 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.405098 16600 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:20.433422 16600 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.435839  5454 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 25 to 29, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 29 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:20.438961  5454 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 24 to 28, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 28 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:20.440809  6274 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:29:20.441069 16257 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:20.441455 16257 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:20.442693 16257 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:20.444293  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:20.444594  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:20.444769 16510 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:20.450763  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:20.451817 16138 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:20.451982 16554 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:20.410741  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:20.457391  6271 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 36.
W20260709 12:29:20.458823  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 181079us
I20260709 12:29:20.395879 16522 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.004s
I20260709 12:29:20.455406 16603 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:20.447850 16590 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:20.455899 16554 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:20.456763 16604 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:20.452323 16138 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:20.455432 16600 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:20.455452 16607 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:20.487124 16550 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:29:20.489185 16257 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:20.493163  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:20.494200  5561 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:29:20.496259  5710 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:29:20.507609 16138 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.510046 16138 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:20.500679 16563 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:29:20.484189 16554 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:20.523461  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 113019us
I20260709 12:29:20.524467 16609 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:29:20.525103  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:20.526127  5561 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
W20260709 12:29:20.527837  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 122742us
I20260709 12:29:20.505932 16370 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:20.534773  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:20.536965  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:20.542389  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:20.543123  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:20.544003 16127 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:20.544283 16106 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:20.544934 15792 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:29:20.546567  6273 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:29:20.551443  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:20.553829  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1732) took 1688 ms. Trace:
I20260709 12:29:20.558310  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1667) took 1112 ms. Trace:
W20260709 12:29:20.576160  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 180559us
I20260709 12:29:20.577996 15792 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 38 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:20.582695 15792 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.26, Last appended by leader: 26, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.583873 15792 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:20.586792  6277 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
W20260709 12:29:20.587482 16582 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.647s	user 0.001s	sys 0.001s
I20260709 12:29:20.578754  6278 rpcz_store.cc:276] 0709 12:29:19.445242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:19.448439 (+  3197us) service_pool.cc:224] Handling call
0709 12:29:20.557833 (+1109394us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b54002eed54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:20.558065 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1138953600}
I20260709 12:29:20.578418  6280 rpcz_store.cc:276] 0709 12:29:18.865866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:18.866288 (+   422us) service_pool.cc:224] Handling call
0709 12:29:18.889121 (+ 22833us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:18.934121 (+ 45000us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:20.553288 (+1619167us) spinlock_profiling.cc:243] Waited 33.4 ms on lock 0x7b5400343ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:20.553639 (+   351us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":76918656}
I20260709 12:29:20.511322 16134 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.593066  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:20.609764  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:20.611572 16546 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:20.611824 16106 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:20.612747 16106 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:20.615072  5601 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:20.617704  5635 raft_consensus.cc:2410] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:29:20.619341 16528 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:20.619939 16528 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:20.621237 16528 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.616230  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:20.594224 16609 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:20.629020  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 224396us
I20260709 12:29:20.629385  5631 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 37 index: 25. Preceding OpId from leader: term: 39 index: 26. (index mismatch)
I20260709 12:29:20.651010  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:20.629472 16609 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:20.629244 16475 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:20.665416 16257 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:20.665988 16257 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:20.666381 16257 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:20.669068 16542 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:20.670110 16542 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.672861 16614 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:20.673349 16614 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:20.673743 16614 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:20.675747 16542 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:20.672046 16260 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:20.680150  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 64038.9us
I20260709 12:29:20.653518 16546 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:20.686237 16278 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:20.686910 16278 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:20.686771 16508 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:29:20.686786  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:20.687412 16278 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:20.687597 16609 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:20.686291 16507 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:20.686491 16475 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.691658  5635 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:29:20.698491  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:20.703509 16546 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:20.710311 16283 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:20.713524 16475 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:20.714644  5631 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:29:20.713707 16106 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:20.717888  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 193118us
I20260709 12:29:20.719400  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:20.720119 16579 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:29:20.721149  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:20.724524  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 128609us
I20260709 12:29:20.725183 16228 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:20.725649 16228 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:20.726146 16228 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:20.726559 16283 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:20.733075  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:20.725559  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:20.700915 16260 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.043s
W20260709 12:29:20.751104  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:20.709005 16546 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:20.736104 16563 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:29:20.770147  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 184475us
I20260709 12:29:20.782625 16257 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:20.787209 16257 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:20.790222  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:20.801331  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:20.794500 16280 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.782s	user 0.008s	sys 0.016s
W20260709 12:29:20.804068  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:20.805754 16617 log.cc:927] Time spent T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:29:20.806762  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:20.809000 16475 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:20.810896  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:20.804248 16280 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:20.812117  5706 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 36: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:29:20.813082 16618 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:29:20.813980 16280 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:20.817126  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 75791.2us
I20260709 12:29:20.818208  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:20.819515 16260 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:20.779098 16134 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:20.826329  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 75842.5us
I20260709 12:29:20.823688  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:20.822402  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:20.819890 16461 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.698s	user 0.010s	sys 0.007s
I20260709 12:29:20.849381  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:20.819998 16260 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:20.854244  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1910) took 1319 ms. Trace:
I20260709 12:29:20.815984 16449 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:20.854521 16615 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:29:20.854712 16461 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.855881  6275 rpcz_store.cc:276] 0709 12:29:19.532441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:19.535978 (+  3537us) service_pool.cc:224] Handling call
0709 12:29:19.672232 (+136254us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:19.672841 (+   609us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:20.846024 (+1173183us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b54005f5d50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:20.851864 (+  5840us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":311993600}
I20260709 12:29:20.857627  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:20.861402  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:20.866572  5703 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:29:20.869074 16449 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:20.871373 16461 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:20.872184  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:20.872534  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:20.877496  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:20.879401 16300 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:20.844336 16408 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.122s
I20260709 12:29:20.880856  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:20.881778 16106 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:20.882246 16106 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:20.882334  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 80707us
W20260709 12:29:20.884616  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 61129.5us
I20260709 12:29:20.863999 16514 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.034s
I20260709 12:29:20.888829  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3669) took 2036 ms. Trace:
I20260709 12:29:20.889268  5636 rpcz_store.cc:276] 0709 12:29:18.851917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:18.852887 (+   970us) service_pool.cc:224] Handling call
0709 12:29:18.853338 (+   451us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:18.873076 (+ 19738us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:29:18.899525 (+ 26449us) log.cc:844] Serialized 14 byte log entry
0709 12:29:20.549186 (+1649661us) spinlock_profiling.cc:243] Waited 2.42 ms on lock 0x7b180042cc40. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:29:20.610575 (+ 61389us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:20.611378 (+   803us) log.cc:844] Serialized 26 byte log entry
0709 12:29:20.611585 (+   207us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:29:20.611624 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:20.623483 (+ 11859us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:20.884538 (+261055us) raft_consensus.cc:1685] finished
0709 12:29:20.888442 (+  3904us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:20.888650 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19221,"prepare.queue_time_us":26276,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":6394,"raft.queue_time_us":90140,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":3171,"spinlock_wait_cycles":5576064,"thread_start_us":58504,"threads_started":1,"wal-append.queue_time_us":1764375}
I20260709 12:29:20.882630 16106 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:20.861222 16260 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:20.892052  5709 raft_consensus.cc:1240] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 35. Current term is 36. Ops: [35.23-35.23]
I20260709 12:29:20.893110  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:20.897711  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 51084.1us
I20260709 12:29:20.899988 16475 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:20.900648 16475 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 36 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:20.901980 16475 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:20.903666  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:20.905566  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:20.907967 16294 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:20.908893  5454 catalog_manager.cc:5697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 21 to 23, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 23 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:20.910450  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:20.910571  5454 catalog_manager.cc:5697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:20.911587 16257 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:20.912552 16257 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:29:20.913189  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:20.914557 15792 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:20.915100 15792 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:20.916101 15792 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.917372 16541 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:20.917917 16539 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:20.919299 16539 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:20.918913  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:20.919708 16539 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:20.920712  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:20.920316  5636 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:20.900570  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:20.928485  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47760us
I20260709 12:29:20.934660  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:20.937237 16474 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:20.928188 16614 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.946897 16614 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:20.947184  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:20.948218 16471 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:20.948730  5559 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
W20260709 12:29:20.949995  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:20.950584 16554 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.950827  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:20.952039  5559 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 33: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:20.952870 16138 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:20.953226  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:20.953604 16138 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:20.964723  5559 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:20.922461  5454 catalog_manager.cc:5697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 29 to 31, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 31 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:20.966645  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:20.965901  5708 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:20.969274  5702 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:20.954844 16257 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:20.964603 16528 consensus_queue.cc:1059] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.012s
I20260709 12:29:20.976467  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:20.978349 16138 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:20.979683 16563 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:20.980232 16563 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:20.982786 16563 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:20.984498 16260 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.986423 16280 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:29:20.986912 16260 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:20.989218  5707 raft_consensus.cc:2410] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 34 because replica has last-logged OpId of term: 33 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 21.
I20260709 12:29:20.989094 16539 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:20.989246  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:20.990813 16539 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:20.991161  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:20.991894  6271 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:20.993877 16554 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:20.996300  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:20.965142  5706 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:21.000499  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:21.001406  5559 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:21.003142 16528 raft_consensus.cc:3055] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:21.003790 16528 raft_consensus.cc:740] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:21.000727  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:21.005059 16528 consensus_queue.cc:260] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.23, Last appended by leader: 23, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:21.008635  5554 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:29:20.990836  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:21.009631  6280 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:21.010319  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:21.012740  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:21.013089 16579 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:21.006096  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:21.006814  6278 raft_consensus.cc:2368] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34.
I20260709 12:29:21.019376  5454 catalog_manager.cc:5697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 21 to 24, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 24 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:21.048964 16278 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:21.051602 16278 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:21.052911  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:21.031764  5452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 181) took 73827 ms (client timeout 15000 ms). Trace:
I20260709 12:29:20.953343 16228 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:21.068385  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48478.8us
I20260709 12:29:21.057658  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4179) took 1019 ms. Trace:
I20260709 12:29:21.068974  5707 rpcz_store.cc:276] 0709 12:29:20.034711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:20.035290 (+   579us) service_pool.cc:224] Handling call
0709 12:29:21.032949 (+997659us) spinlock_profiling.cc:243] Waited 948 ms on lock 0x7b5400075654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:21.054388 (+ 21439us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2180963072}
I20260709 12:29:21.061097  5702 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:21.066565  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.057379 16528 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:21.079241  5701 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 34 index: 22. (term mismatch)
I20260709 12:29:21.080539  5701 pending_rounds.cc:85] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 21
I20260709 12:29:21.084317  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:21.088733  6278 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:21.085263  5454 catalog_manager.cc:5697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 26 to 29, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 29 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:21.093799  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:21.076612 16541 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:21.084787  5701 pending_rounds.cc:107] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 33.22
I20260709 12:29:21.097692  5701 raft_consensus.cc:2889] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:21.095355  5557 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:21.110164  5454 catalog_manager.cc:5697] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 24 to 27, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 27 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:21.120278  6280 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:21.097476 16541 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:21.121526  5454 catalog_manager.cc:5697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 29 to 30, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 30 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:29:21.135990  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 136481us
I20260709 12:29:21.106424 16236 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:21.120976 16257 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:21.128051 16541 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:21.146183  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:21.121551 16228 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:21.146876  5454 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 23 to 25. New cstate: current_term: 25 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
W20260709 12:29:21.153468  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 76392.2us
I20260709 12:29:21.143051 16236 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:21.155028 16573 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:21.155359  5557 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:29:21.155690 16573 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:21.148384  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:21.160105  5635 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:21.154417  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:21.162231  5557 pending_rounds.cc:85] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 15
I20260709 12:29:21.162637  5557 pending_rounds.cc:107] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 12:29:21.162940  5557 raft_consensus.cc:2889] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:21.163326  5557 pending_rounds.cc:107] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 26.17
I20260709 12:29:21.163636  5557 raft_consensus.cc:2889] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:21.182487 16573 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:21.183862  5633 raft_consensus.cc:1217] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Deduplicated request from leader. Original: 37.25->[38.26-38.26]   Dedup: 38.26->[]
W20260709 12:29:21.189406  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:21.180025  5454 catalog_manager.cc:5697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 23 to 26, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 26 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:21.196059  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:21.198813  5633 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 38. Current term is 39. Ops: [38.26-38.26]
I20260709 12:29:21.205420  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:21.200050  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:21.219090  5454 catalog_manager.cc:5697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 21 to 24, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 24 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
W20260709 12:29:21.054989  5452 rpcz_store.cc:269] 0709 12:28:07.192158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:17.798684 (+10606526us) service_pool.cc:224] Handling call
0709 12:29:21.018833 (+63220149us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:29:11.457408 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:29:11.458128 (+   720us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:29:11.458193 (+    65us) write_op.cc:435] Acquired schema lock
0709 12:29:11.458211 (+    18us) tablet.cc:662] Decoding operations
0709 12:29:11.459089 (+   878us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:29:11.459202 (+   113us) write_op.cc:641] Partition lock acquired for write op
0709 12:29:11.459234 (+    32us) tablet.cc:685] Acquiring locks for 5 operations
0709 12:29:11.459925 (+   691us) tablet.cc:701] Row locks acquired
0709 12:29:11.459981 (+    56us) write_op.cc:260] PREPARE: finished
0709 12:29:11.462164 (+  2183us) write_op.cc:270] Start()
0709 12:29:11.480723 (+ 18559us) write_op.cc:276] Timestamp: P: 1783600151462068 usec, L: 0
0709 12:29:11.493351 (+ 12628us) op_driver.cc:348] REPLICATION: starting
0709 12:29:11.545034 (+ 51683us) log.cc:844] Serialized 2024 byte log entry
0709 12:29:12.243636 (+698602us) op_driver.cc:464] REPLICATION: finished
0709 12:29:12.640673 (+397037us) write_op.cc:301] APPLY: starting
0709 12:29:12.655201 (+ 14528us) tablet.cc:1367] starting BulkCheckPresence
0709 12:29:12.882858 (+227657us) tablet.cc:1370] finished BulkCheckPresence
0709 12:29:12.882976 (+   118us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:29:12.970985 (+ 88009us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:29:12.971455 (+   470us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=10,key_file_lookups=10,delta_file_lookups=0,mrs_lookups=0
0709 12:29:12.971519 (+    64us) write_op.cc:312] APPLY: finished
0709 12:29:12.971908 (+   389us) log.cc:844] Serialized 57 byte log entry
0709 12:29:13.010741 (+ 38833us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:29:13.029624 (+ 18883us) write_op.cc:454] Released schema lock
0709 12:29:13.029724 (+   100us) write_op.cc:341] FINISH: Updating metrics
0709 12:29:13.030337 (+   613us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:29:13.030369 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6603079,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":367974,"apply.run_cpu_time_us":11654,"apply.run_wall_time_us":389695,"cfile_cache_hit":22,"cfile_cache_hit_bytes":25027,"num_ops":5,"prepare.queue_time_us":392609,"prepare.run_cpu_time_us":16714,"prepare.run_wall_time_us":194753,"raft.queue_time_us":272082,"raft.run_cpu_time_us":6470,"raft.run_wall_time_us":292457,"replication_time_us":750033,"thread_start_us":709317,"threads_started":4,"wal-append.queue_time_us":134173,"wal-append.run_cpu_time_us":64057,"wal-append.run_wall_time_us":2322894}]]}
I20260709 12:29:21.213529 16475 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:29:21.260792 16475 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:21.199630 16237 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.150s
W20260709 12:29:21.274322  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 153052us
W20260709 12:29:21.276952  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 122650us
W20260709 12:29:21.280460 16631 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260709 12:29:21.300180  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:21.303125 16474 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:21.303608 16474 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:21.304003 16474 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:21.304572 16546 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:29:21.304900 16475 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:21.305866 16546 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:21.306377 16546 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:21.308856 16611 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:21.309314 16611 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:21.310010 16592 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:29:21.311299 16630 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 12:29:21.312705  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:21.314137 16546 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:21.337493  5636 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
W20260709 12:29:21.339344  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:21.339428  5561 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:21.336377  5454 catalog_manager.cc:5697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 17 to 28, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 28 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } }
I20260709 12:29:21.351471  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:21.350297  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:21.355260  5707 raft_consensus.cc:2410] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 27 because replica has last-logged OpId of term: 26 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 16.
I20260709 12:29:21.355465  5559 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
W20260709 12:29:21.355989  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 152837us
I20260709 12:29:21.357038  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:21.368139  5454 catalog_manager.cc:5697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 18 to 24. New cstate: current_term: 24 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:21.371299  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:21.375015  5454 catalog_manager.cc:5697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 25 to 29, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 29 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:29:21.378245  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:21.375056  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.381992  5675 leader_election.cc:400] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34.
I20260709 12:29:21.382354  5675 leader_election.cc:403] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:29:21.382993  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:21.384869  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:21.372422 16554 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:21.387625 16370 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:29:21.393280  5627 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:29:21.393190  5454 catalog_manager.cc:5697] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 23 to 27, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 27 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:29:21.396188 16634 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:29:21.369570 16300 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.022s
I20260709 12:29:21.403121  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:21.401264  5560 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:21.404767  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:21.391275 16449 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:21.406623  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:21.411563 16273 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:21.411252  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:21.423051  5706 raft_consensus.cc:2410] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 39 because replica has last-logged OpId of term: 38 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 25.
I20260709 12:29:21.414392 16611 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:21.415102  5454 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 26 to 29, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 29 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:21.424436  5705 raft_consensus.cc:2410] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 27 because replica has last-logged OpId of term: 26 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 16.
I20260709 12:29:21.428598 16449 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:21.439749  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:21.442682  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 120011us
I20260709 12:29:21.445734  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:21.447942  5555 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:29:21.450544  6273 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:29:21.395042 15682 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:21.457444 16273 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:21.460062  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.465112  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:21.470568  6278 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:21.470800  5635 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
W20260709 12:29:21.461078 16106 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.569s	user 0.007s	sys 0.006s
I20260709 12:29:21.472502  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:21.475442  5560 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 29: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:21.474629  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:21.479206 16592 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:21.482059 16611 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:21.482422 16592 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:21.483383 16592 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:21.479959  5454 catalog_manager.cc:5697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 17 to 25. New cstate: current_term: 25 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:21.485988  5554 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:21.487673  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:21.488341  5557 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 32 because replica has last-logged OpId of term: 31 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 21.
I20260709 12:29:21.475873 16608 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:21.479276  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.490234  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:21.495409  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:21.491550  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:21.473762 15682 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:21.493244 16353 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:21.505045  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.475409 16106 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:21.511003  5705 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:21.514752  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.515813  5560 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 36: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:21.519414  5557 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 38.
I20260709 12:29:21.525556  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:21.504423 16353 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:21.539863  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61838.6us
I20260709 12:29:21.542768  5633 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: []
I20260709 12:29:21.545580  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:21.546178  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.547936  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.551584  5554 raft_consensus.cc:2368] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34.
I20260709 12:29:21.548982  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.554495  5557 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
W20260709 12:29:21.562013  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 85621.4us
I20260709 12:29:21.563323  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:21.564751 16607 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:29:21.565704 16639 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:29:21.569635  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:21.571051  5704 raft_consensus.cc:2410] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 39 because replica has last-logged OpId of term: 38 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 25.
I20260709 12:29:21.543912 16353 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:21.568090  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:21.568257  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:21.601642  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.603019  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:21.603802  5554 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:21.574446  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.617425  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.618355  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:21.618505  5557 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:21.619204  5705 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:21.547197 16563 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.058s
I20260709 12:29:21.609302 16616 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:21.620929 16616 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:21.624444  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61335.2us
I20260709 12:29:21.542128 16449 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:21.629282  5557 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:29:21.623407 16616 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:21.633129  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:21.626827  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:21.672818  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 129902us
W20260709 12:29:21.678735  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 163978us
I20260709 12:29:21.610419 16541 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:21.675191 16475 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:21.550625 15682 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 30.23, Last appended by leader: 23, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:21.673179 16514 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:21.697990  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:21.672994 16607 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:21.681650 16419 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:21.695681  5526 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:21.710937  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:21.722815  5705 raft_consensus.cc:2410] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
W20260709 12:29:21.709219 16539 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.718s	user 0.014s	sys 0.001s
I20260709 12:29:21.687170 16300 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.031s
I20260709 12:29:21.698313 16554 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:21.738096 16474 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:21.715008 16514 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:21.740561 16641 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:29:21.734369 16579 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:21.758203 16106 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:21.758597 16514 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:21.741986  5703 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: []
W20260709 12:29:21.749711  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 120046us
I20260709 12:29:21.762748  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:21.764505 16021 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.047s
I20260709 12:29:21.765069  5676 leader_election.cc:400] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34.
I20260709 12:29:21.765872  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:21.752305 16257 consensus_meta.cc:287] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.606s	user 0.013s	sys 0.000s
I20260709 12:29:21.766791 16257 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:21.734771 16138 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:21.767642  5555 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:21.767756 16138 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:21.768391 16138 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:21.771566 16273 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:21.774191  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:21.774434 16642 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:29:21.775909 16278 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:21.776500 16278 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:21.776991 16541 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:21.778765 16278 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:21.770722 16637 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.450s	user 0.002s	sys 0.000s
I20260709 12:29:21.785372  5554 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:21.780380  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:21.798099  6273 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 24.
I20260709 12:29:21.810446  5555 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:21.786448 16539 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:21.788372 16475 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:21.748495 16554 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:21.734539 16607 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:21.817363  5554 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
W20260709 12:29:21.831156  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 148331us
W20260709 12:29:21.834056  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 103099us
I20260709 12:29:21.837630 16021 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:21.816111 16475 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:21.840991  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:21.855312  5560 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:29:21.861433  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:21.865139  5557 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:21.836984 16370 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 33. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34.
I20260709 12:29:21.865819  5553 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:21.863615  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:21.865579  5557 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:21.841332 16475 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:21.871773  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:21.873203  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:21.877110  5710 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 32. Current term is 34. Ops: []
I20260709 12:29:21.877498  5561 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 36 index: 23. (term mismatch)
I20260709 12:29:21.872200 16294 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:21.878065  5561 pending_rounds.cc:85] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 22
I20260709 12:29:21.884724  5561 pending_rounds.cc:107] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 34.23
I20260709 12:29:21.885905  5561 raft_consensus.cc:2889] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:21.887272 16390 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:29:21.887770  5557 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:21.868799 16021 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:21.904912  5602 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:21.866865 16300 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.017s
I20260709 12:29:21.912261 16474 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:21.917508 16657 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 12:29:21.918416  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 66980.9us
I20260709 12:29:21.922461  5557 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:21.916445 16659 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:29:21.938493  5707 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 17. (term mismatch)
I20260709 12:29:21.938972  5707 pending_rounds.cc:85] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 16
I20260709 12:29:21.939211  5707 pending_rounds.cc:107] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 26.17
I20260709 12:29:21.939491  5707 raft_consensus.cc:2889] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:29:21.950287  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:21.956717  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:21.958631  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:21.960240  5557 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 34 index: 21. (term mismatch)
I20260709 12:29:21.960954  5557 pending_rounds.cc:85] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 20
I20260709 12:29:21.961282  5557 pending_rounds.cc:107] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 33.21
I20260709 12:29:21.961519  5557 raft_consensus.cc:2889] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:21.970500  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.974341 16353 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:21.975791  5710 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:29:21.977178 16353 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:21.977890 16353 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:21.977946  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.978611 16579 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:21.978772 16021 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:29:21.981040  5559 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:29:21.982239  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:21.982705 16528 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.894s	user 0.006s	sys 0.006s
I20260709 12:29:21.983125 15682 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:21.983695 15682 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:21.984108 15682 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:21.984311  5561 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:21.978497 16353 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:21.979239 16236 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:21.929678 16665 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:21.985880  5553 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:21.989432  5629 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 36 because replica has last-logged OpId of term: 35 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
I20260709 12:29:21.990031  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:21.949911 16659 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:21.968896 16651 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 12:29:22.004789  5707 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 38 index: 26. Preceding OpId from leader: term: 39 index: 26. (term mismatch)
I20260709 12:29:21.965741 16658 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:22.008566  5707 pending_rounds.cc:85] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 25
W20260709 12:29:22.011345  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 83298.8us
I20260709 12:29:22.012068  5707 pending_rounds.cc:107] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 38.26
I20260709 12:29:22.012583  5707 raft_consensus.cc:2889] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:29:22.014070 16608 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.022s	sys 0.015s
W20260709 12:29:22.019203 16608 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.515s	user 0.023s	sys 0.015s
I20260709 12:29:22.019556 16608 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:22.020838 16608 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:22.024892  6236 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:21.968730 16474 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 20, Time since last communication: 0.046s
I20260709 12:29:22.025957  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:22.006258 16659 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:22.005999 16665 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:21.989279 15792 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.036s
I20260709 12:29:22.012784 16236 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:22.033352  5627 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 25: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:21.992332 16541 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.034s
I20260709 12:29:22.034407  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:22.030470 16474 consensus_queue.cc:1243] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 36.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:29:22.030737 16665 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:22.040683 16616 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:22.042790 16138 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:22.027462 16607 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:22.056545  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:22.048432 16106 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:22.016713 16646 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.038s
I20260709 12:29:22.071848 16646 consensus_queue.cc:1243] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 31.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:29:22.064100 16260 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:22.079336  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:22.061858 16616 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:22.085814  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:22.067137 16106 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:22.116261 16106 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:22.117595 16106 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:22.119328 16644 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.575s	user 0.001s	sys 0.001s
I20260709 12:29:22.053982 16236 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.114845 16353 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:22.121312  5560 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:22.136955  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57968.4us
I20260709 12:29:22.124796 16260 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:29:22.144855  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 240087us
I20260709 12:29:22.138659  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:22.152237 16648 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:22.152848 16648 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:22.153613 16236 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:22.153806 16648 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.155241 16648 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:22.152961 16633 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.879s	user 0.000s	sys 0.002s
I20260709 12:29:22.161165 16138 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:22.162551 16474 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:22.163137 16474 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:22.166143 16663 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:29:22.167695 16260 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.169119  5703 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:29:22.172003  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:22.173044  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3682) took 1013 ms. Trace:
I20260709 12:29:22.173480 16539 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:22.173844  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:22.174202 16539 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:22.174738  6273 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 27: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:22.144680 16294 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:22.177316 16294 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:22.177805 16294 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:22.179114  6275 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:29:22.179733 16294 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:22.180673  6278 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
W20260709 12:29:22.180878  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 67175.9us
I20260709 12:29:22.182068  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:22.182015 15682 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.183027 16607 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:22.183398 16257 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:22.184984 15682 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:22.185231  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:22.173484  5629 rpcz_store.cc:276] 0709 12:29:21.159226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:21.167003 (+  7777us) service_pool.cc:224] Handling call
0709 12:29:22.172802 (+1005799us) spinlock_profiling.cc:243] Waited 672 ms on lock 0x7b54001cf2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:22.172950 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1545263488}
I20260709 12:29:22.186978  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:22.187578  5555 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 31
W20260709 12:29:22.187721  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49153.9us
I20260709 12:29:22.188733 16665 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:22.189275 16273 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:22.193518  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:22.189078 16607 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:22.208256  5555 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:22.202824  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:22.206179  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:22.208618  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 74773us
I20260709 12:29:22.214586  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:22.217023  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:22.211072  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:22.224232  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:22.244863  5706 raft_consensus.cc:2410] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 34 because replica has last-logged OpId of term: 32 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 18.
I20260709 12:29:22.245076  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:22.244889  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:22.180246 16294 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:22.245977  5705 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:22.255242  5554 pending_rounds.cc:85] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 20
I20260709 12:29:22.255709  5554 pending_rounds.cc:107] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 36.21
I20260709 12:29:22.256016  5554 raft_consensus.cc:2889] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:22.258224  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:22.250771  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:22.258953  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:22.250471  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:22.258949  5707 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:29:22.259038  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:22.201748 16539 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:22.291654  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:22.258414  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:22.286255  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:22.202468 16665 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:22.282226  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:22.219439 15792 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:22.314530 15682 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 31.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:29:22.245478 16608 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:22.318627  5629 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:22.324133  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:22.324935  5636 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 33: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:22.342692 15538 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:22.343035  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:22.344437  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:22.345641  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:22.290994 16616 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.323004 16510 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:22.347108  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:22.350207  5628 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:22.280606 16300 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.071s
I20260709 12:29:22.357232  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:22.360215  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 143406us
I20260709 12:29:22.360196  5631 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:22.366359  5627 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:22.273743 16563 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 18, Time since last communication: 0.080s
I20260709 12:29:22.374455  5708 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:22.316602 16646 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:22.371433  5631 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: [33.21-33.21]
I20260709 12:29:22.258478 16611 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.080s
I20260709 12:29:22.260901 16659 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:22.382742  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:22.386466  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 197845us
I20260709 12:29:22.389034  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:22.392105  5631 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:29:22.357577 16510 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:22.319231 16294 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:22.405627  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:22.408799  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:22.410498  5630 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:29:22.411561  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:22.384330 16475 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:22.414712 16475 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:22.415721  5636 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:22.394945 16134 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:22.416793 16134 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.383777 16408 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:29:22.417701  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:22.418973 16134 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:22.419243  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:22.419962  5627 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:22.416793  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:22.424548  5636 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:22.425221  5705 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
W20260709 12:29:22.444463  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 169278us
I20260709 12:29:22.388227 16554 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Stepping down as leader of term 31
W20260709 12:29:22.508950  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 86064.5us
I20260709 12:29:22.392098 15792 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:22.354071 16608 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.517057  6237 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:22.518190  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:22.516710  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:22.517298  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 87630.6us
I20260709 12:29:22.523742  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:22.515129 15538 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.004s
I20260709 12:29:22.519025  5628 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: []
I20260709 12:29:22.530997  5553 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:29:22.387660 16390 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.125s
I20260709 12:29:22.548862  6278 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:22.549391 16449 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:22.549803  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:22.549952 16449 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.551342 15792 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:22.517858  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:22.553537  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 160373us
I20260709 12:29:22.426913 16646 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:22.563362  5556 raft_consensus.cc:2410] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 32 because replica has last-logged OpId of term: 30 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 22.
I20260709 12:29:22.559094  6278 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:29:22.552688  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:22.568485  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:22.565124  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48160.1us
I20260709 12:29:22.505293 16474 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:22.509886 16607 raft_consensus.cc:2764] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election decision vote started in defunct term 33: won
I20260709 12:29:22.511297 16554 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:22.578280  5555 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 32 index: 22. (term mismatch)
I20260709 12:29:22.572551  5633 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:22.518638 16300 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:22.508237 16419 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:29:22.564005  5675 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 2.147s (SENT) [suppressed 14 similar messages]
I20260709 12:29:22.526885 16592 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:22.577009  5635 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: [32.23-32.23]
I20260709 12:29:22.552212 16449 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:22.519124 16541 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.061s
I20260709 12:29:22.555182 16667 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:29:22.583230 16673 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:29:22.578707  5555 pending_rounds.cc:85] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 21
I20260709 12:29:22.597689  5555 pending_rounds.cc:107] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 31.22
I20260709 12:29:22.598145  5555 raft_consensus.cc:2889] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:29:22.552678 16634 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260709 12:29:22.576313 16508 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:22.605296  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:22.605537  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:22.573817 16646 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.599845 16474 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:22.591892 16300 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.613884 16508 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:22.613976 16278 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:22.630605  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:22.623828  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:22.613656 16683 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:22.626310  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:22.641022  5556 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 31: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:29:22.641578  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:22.612396 16528 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.013s
I20260709 12:29:22.622392 16604 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 12:29:22.653053  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:22.655272  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:22.659081 16677 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.036s
W20260709 12:29:22.659961 16678 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.082s	user 0.001s	sys 0.001s
I20260709 12:29:22.660732 16236 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:29:22.607678 16510 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:22.663017 16679 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
W20260709 12:29:22.643828  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 224672us
I20260709 12:29:22.665313  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:22.641176 16556 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:22.679564  5560 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.545s	user 0.009s	sys 0.000s
I20260709 12:29:22.680038  5560 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
W20260709 12:29:22.680142  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53933.1us
I20260709 12:29:22.682361  5557 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: [32.19-32.19]
I20260709 12:29:22.683171  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:22.621224 16474 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:22.687498  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:22.689991  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:22.690096  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:22.688983  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:22.691373  5559 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 30 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 12.
I20260709 12:29:22.644807 15792 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:22.691447  5708 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:29:22.688601 16648 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.533s	user 0.009s	sys 0.005s
I20260709 12:29:22.732009  6278 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: [30.21-31.22]
I20260709 12:29:22.684226 16294 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:22.644858 16278 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:22.621881 16554 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.23, Last appended by leader: 23, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:22.746628  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 182639us
I20260709 12:29:22.747756  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:22.643467 16683 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.750159  6275 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: [33.21-33.21]
I20260709 12:29:22.751754  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:22.744688 16671 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.235s	user 0.002s	sys 0.000s
I20260709 12:29:22.758425  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:22.761785 16689 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:29:22.764909  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:22.692785 16556 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.765848  6273 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:22.780412 16278 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:22.780568 16556 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:22.781123 16278 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:22.781510 16278 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:22.782991  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:22.771626 16687 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:29:22.772724  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:22.775288  5634 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:22.740060 16508 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.777145 16646 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:22.778623 16683 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:22.786885  5707 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 27.
I20260709 12:29:22.791605 16662 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:22.792342 16688 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:22.753840 16228 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.039s
I20260709 12:29:22.794374  6275 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 36: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:22.756717 15792 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:22.793694 16138 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:29:22.783592 16608 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:22.809190  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:22.806617 16658 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:22.821123 16658 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:22.822260 16675 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:29:22.823805 16658 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:22.813323 16611 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 12:29:22.826730 16674 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:22.827425 16674 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:22.797847 16541 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:29:22.830329  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:22.782297 16278 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:22.787918 16475 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.023s
I20260709 12:29:22.835348  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:22.846688  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:22.847496  5632 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:22.783066 16689 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.030s
I20260709 12:29:22.862630  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:22.863869  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 103803us
I20260709 12:29:22.867223  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:22.870086  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:22.813755 16539 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.047s
I20260709 12:29:22.805757 16662 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.878721 15538 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:22.905217  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:22.907864  6271 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:22.842135 16611 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 31.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.029s
I20260709 12:29:22.868324 16688 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:29:22.865473 16278 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:22.868207 16475 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:22.918092  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:22.919684  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:22.871380 16514 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:22.905033 15538 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:22.926568  5553 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:22.927733  5636 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 36 because replica has last-logged OpId of term: 35 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
I20260709 12:29:22.906018 16370 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:22.930506  5630 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:29:22.930718  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:22.935111  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:22.906587 16300 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:22.961992 15538 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:22.929338 16514 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:22.964676 16514 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:22.964919  6271 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
W20260709 12:29:22.965494 16702 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.212s	user 0.000s	sys 0.001s
I20260709 12:29:22.968940 16662 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:22.975287 16554 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:22.921173 16563 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.028s
I20260709 12:29:22.913872 16677 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:22.953320  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:22.936059  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:22.956373 16711 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.109s	user 0.002s	sys 0.000s
I20260709 12:29:22.956771 16705 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:22.957219 16697 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:22.961308 16370 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:22.951706  5553 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31. Ops: []
W20260709 12:29:22.982286  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 46314.5us
I20260709 12:29:22.985518  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:22.991895  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:22.992776  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:22.851071 16674 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:23.000269  5553 pending_rounds.cc:85] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 15
W20260709 12:29:23.000317  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 95194.1us
I20260709 12:29:23.001880  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:23.002885  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:23.003546  5703 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:29:23.000674  5553 pending_rounds.cc:107] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 28.16
I20260709 12:29:23.005072  5553 raft_consensus.cc:2889] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:23.005861  5708 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 22. Preceding OpId from leader: term: 31 index: 24. (index mismatch)
I20260709 12:29:23.004303  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:23.007434  5706 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:23.008111  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:23.009367  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:22.981895 16475 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:29:23.009452  5630 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 36.
I20260709 12:29:23.019362 16716 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:22.995206 16563 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:29:22.999275 16474 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:23.040328  6277 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:29:22.994004 16677 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:23.047991  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:23.031273 16563 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:23.022657 16539 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:23.012017 16550 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.035s
I20260709 12:29:23.042034 16713 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:22.998847 16705 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:22.998235 16697 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.071931  5706 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:23.079033  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:23.079772  6277 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:23.053620 16539 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:23.085034 16659 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:23.087077 16539 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:23.094061 16705 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:23.087656 16677 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:23.087404 16684 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.462s	user 0.002s	sys 0.000s
I20260709 12:29:23.051712 16563 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:23.106504  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:23.106645  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:23.052860 16474 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.053s
I20260709 12:29:23.111097 16689 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:23.111321 16592 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:23.111575 16689 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:23.112515 16689 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:23.113358 16667 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:23.114307 16667 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.23, Last appended by leader: 23, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:23.116920  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 451662us
I20260709 12:29:23.117650 16588 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:23.118710  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:23.121246  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:23.107913 16579 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:23.117774  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:23.127053  5632 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:23.124142  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:23.125259  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:23.120187 16719 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:23.132005  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:23.135259  5559 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 31: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:23.133831  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:23.052924 16688 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:23.117694 16709 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.153s	user 0.000s	sys 0.003s
I20260709 12:29:23.142508  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:23.139114  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:23.147259 16692 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:23.052403 16715 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.085s
I20260709 12:29:23.156234  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:23.156939  6277 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:23.158941  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:23.145349 16474 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:23.148792 16611 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:23.164165  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:23.077641 16713 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.167717  5553 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:23.173828  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:23.155452 16419 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:29:23.184832  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:23.186782  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:23.189363  6278 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: [32.23-32.23]
I20260709 12:29:23.187287  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:23.144971 16579 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:23.195688  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:23.126842 16592 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.213263  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:23.188241 16697 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:23.180708 16474 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:23.201054  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:23.201354  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:23.199098 16705 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:23.207881 16611 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:23.196537 16703 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:23.199036 16692 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:23.210117 16688 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:23.215297  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 51262.6us
I20260709 12:29:23.187505 16550 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:23.234103  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:23.235551  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61773.1us
I20260709 12:29:23.247169 16607 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:23.233870 16719 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:23.253028 16579 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:23.256304  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:23.297680  5706 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:23.257302 16705 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:23.279289  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:23.307226  5702 raft_consensus.cc:2393] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 36: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:23.305351  5703 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:29:23.309201  6273 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 32 because replica has last-logged OpId of term: 31 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 22.
I20260709 12:29:23.305418  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:23.299382  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:23.328660  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:23.331737  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:23.310642 16705 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:23.331591  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:23.336859  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:23.272554 16703 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:23.299532 16550 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.306305 16607 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.338449 16667 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:29:23.338601 16608 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:29:23.379196  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:23.413323  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 123521us
W20260709 12:29:23.417001  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:23.390336 16608 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:23.419808 16727 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 12:29:23.428066  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:23.428869  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:23.435426  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:23.391244 16690 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:23.363600 16449 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:23.366309 16550 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:23.351897 16703 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:23.447434  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:23.358788 16607 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:23.357388 15792 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:23.430440 16689 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:23.433800 16608 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:23.341493 16474 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:23.346048 16611 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:23.467170 16690 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.467366 15792 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:23.477253 15792 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:23.478340 15792 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:23.479619  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:23.481240  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:23.482125 16370 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:23.482575  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:23.484432 16607 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:29:23.484848 16729 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:29:23.484920 16607 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:23.485402 16607 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:23.485512  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:23.465587 16676 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.101s	user 0.002s	sys 0.004s
I20260709 12:29:23.489872  5703 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:23.495118  5553 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 30 index: 23. Preceding OpId from leader: term: 31 index: 24. (index mismatch)
I20260709 12:29:23.492692 16690 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:23.457293 16579 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:23.454870 16449 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:23.509500  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:23.513674  5560 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
I20260709 12:29:23.360517  5630 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e69273530292453d87614627debcc13e"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:23.516919  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3894) took 1199 ms. Trace:
I20260709 12:29:23.517732  5630 raft_consensus.cc:606] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:23.521443  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:23.523417 16732 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260709 12:29:23.463074 16563 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.517968  5634 rpcz_store.cc:276] 0709 12:29:22.317955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:22.338530 (+ 20575us) service_pool.cc:224] Handling call
0709 12:29:23.516684 (+1178154us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b540018d654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:23.516838 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":550584704}
W20260709 12:29:23.553191  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:23.567061  5554 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:29:23.591253  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4058) took 1505 ms. Trace:
I20260709 12:29:23.601943  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:23.538331  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:23.619455  5629 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:23.613886  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:23.624050  5631 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:29:23.617183  5561 rpcz_store.cc:276] 0709 12:29:22.085820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:22.086154 (+   334us) service_pool.cc:224] Handling call
0709 12:29:22.116438 (+ 30284us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:22.116852 (+   414us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:22.116880 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:22.117324 (+   444us) log.cc:844] Serialized 26 byte log entry
0709 12:29:22.133503 (+ 16179us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:29:22.134157 (+   654us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:22.134232 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:23.365641 (+1231409us) spinlock_profiling.cc:243] Waited 6.96 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:23.590892 (+225251us) raft_consensus.cc:1685] finished
0709 12:29:23.590920 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:23.591166 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":112,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"spinlock_wait_cycles":16018688,"thread_start_us":15790,"threads_started":1,"wal-append.queue_time_us":116065}
I20260709 12:29:23.625821  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4061) took 1402 ms. Trace:
I20260709 12:29:23.626492  5554 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:23.627017  5709 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:29:23.511139 16370 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:29:23.519236 16717 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:29:23.626235  5555 rpcz_store.cc:276] 0709 12:29:22.223593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:22.250098 (+ 26505us) service_pool.cc:224] Handling call
0709 12:29:22.250361 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:22.603062 (+352701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:23.625601 (+1022539us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b54001bd854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:23.625769 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":753275776}
I20260709 12:29:23.536230 16719 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:23.538383 16713 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:23.628376 16514 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:23.516776 16592 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.133s
I20260709 12:29:23.628901 16370 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:23.567797 16353 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.083s
I20260709 12:29:23.629025 16667 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:23.560961 16690 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.089s
I20260709 12:29:23.662088 16370 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:23.668678 16667 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:23.691097 16674 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:23.692436 16419 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:23.693070 16419 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.693845 16563 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:23.694902 16579 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:29:23.695886 16419 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:23.697733 16667 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:23.660401 16514 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:23.697973 16607 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:23.698879  5561 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:23.721282  5561 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:23.721992  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72848.5us
I20260709 12:29:23.733348  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:23.720465 16514 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:23.741585  5556 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: [29.15-29.15]
W20260709 12:29:23.738925  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 92035.9us
I20260709 12:29:23.730705 16719 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:23.744341 15792 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:23.742026  5561 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.757794  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:23.738945 16692 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:23.768301  5561 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:23.770424 16719 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.772746 16719 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:23.776186 16611 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.009s
I20260709 12:29:23.780797  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3680) took 1250 ms. Trace:
I20260709 12:29:23.781332  5557 rpcz_store.cc:276] 0709 12:29:22.530377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:22.530507 (+   130us) service_pool.cc:224] Handling call
0709 12:29:22.682075 (+151568us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:22.682859 (+   784us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:23.780549 (+1097690us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54000ec4d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:23.780726 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":347770752}
I20260709 12:29:23.759857 16592 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:23.782240 16716 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:23.782783 16592 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:23.785283 16607 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:23.786942  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:23.787776 16703 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.764748 16721 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:23.789248 16721 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:23.791307 16721 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:23.783790 16592 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.23, Last appended by leader: 23, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.793612 16592 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:23.795044  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:23.802872  5557 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:29:23.804476  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:23.805929  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:23.804266  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:23.807322  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49597.8us
I20260709 12:29:23.809123  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:23.807875  5552 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
W20260709 12:29:23.810319  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 51004.6us
I20260709 12:29:23.810864  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:23.811066  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:23.797390 16739 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.070s	user 0.000s	sys 0.002s
I20260709 12:29:23.812909  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:23.811480  5559 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:23.814635  5561 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:29:23.816900  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:23.801138 16703 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:23.821669  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:23.824548  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:23.810058 16408 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:23.815241 15682 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:23.828588 15682 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:23.825559 16510 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:23.815343 16138 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:23.828972 15682 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:23.808296 16607 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:23.767846 15792 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.814181 16751 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:23.821861  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:23.800138 16608 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:23.825637 16658 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:23.820570 16257 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:29:23.839793  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:23.832651 16563 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:23.846215  5555 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:23.823103 16712 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.022s
I20260709 12:29:23.843590  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:23.833809 16408 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:23.842139 16749 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:23.869314  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:23.870304  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:23.870952  5634 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:23.871807  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:23.872186 16710 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.946s	user 0.003s	sys 0.000s
I20260709 12:29:23.872002 16138 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:29:23.872529  5629 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:23.873970  5630 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:29:23.874809 16228 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:23.875574  5632 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:23.876982 16713 raft_consensus.cc:993] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:29:23.876857  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:23.877714 16713 raft_consensus.cc:1081] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Signalling peer cea66aa15be3423880843974acdbf50f to start an election
W20260709 12:29:23.853499 16705 consensus_meta.cc:287] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.507s	user 0.008s	sys 0.005s
I20260709 12:29:23.850446 16703 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:23.837427 16607 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:23.884086  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:23.893472  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:23.896571  5627 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: [27.20-27.20]
I20260709 12:29:23.897434  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:23.899215 15538 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:23.863405  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:23.911342  5635 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 38. Current term is 39. Ops: [38.23-38.23]
I20260709 12:29:23.838478 16751 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.908293 16611 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:29:23.937003  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 73775.7us
I20260709 12:29:23.923889 16408 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:23.938767 16611 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:23.940428 16755 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:23.940618 16611 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:23.940797 15792 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:23.939042 16688 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:23.942636 16688 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:23.944805 16688 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:23.908094 16719 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 12:29:23.949102 16719 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:29:23.949494 16719 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 25 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:23.950307 16719 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.951591 16719 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:29:23.952702  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:23.953099 16608 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:23.953550 16607 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:23.955121 16689 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:23.955255 16751 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:23.955744 16689 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.958330 16689 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:23.961382 15792 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 37.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:29:23.910535 16716 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.028s
I20260709 12:29:23.965452 16419 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:23.891330 16705 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:23.975859  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 127421us
W20260709 12:29:23.951519 16746 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.263s	user 0.000s	sys 0.006s
I20260709 12:29:23.965978 16419 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:23.964270 16514 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:23.973039 16755 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:23.986248  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:23.960099 15682 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:24.002985  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:24.003561  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:23.966303 16370 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.004281  5559 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
W20260709 12:29:24.010036  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 166576us
I20260709 12:29:24.014243 15538 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:24.016688  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:24.026742 16715 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:24.027292 16715 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.028512 15682 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:24.032228  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:24.032946  5558 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:24.033293 16370 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:24.034093 16370 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:24.034580 16370 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.037590 16419 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.038870 16419 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:24.005894 16752 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.036s
I20260709 12:29:24.064716 15538 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:24.014448 16721 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:24.075150 15538 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:24.076478  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 57669.9us
I20260709 12:29:24.077358  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:24.077771 16707 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.125s	user 0.005s	sys 0.000s
I20260709 12:29:24.078191 16260 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:24.078630  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:24.079388 16260 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:24.080549  5555 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:29:24.080458 16260 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:29:24.081168 16757 log.cc:927] Time spent T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:29:24.081453 16514 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:24.069877 16665 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:24.082791 16665 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.085543  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:24.092515  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:24.095090 16514 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:24.095610 16713 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:24.094949 16750 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:24.096153 16713 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.040930 16755 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.100489 16713 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:24.094985 16665 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:24.102104 16766 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:24.106918  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:24.102682 16766 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.113973 16766 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:24.116588  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:24.072696 16370 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:24.084180 16716 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:29:24.119686  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 103083us
I20260709 12:29:24.119011 15538 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:24.121790 15538 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.124540  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:24.119760  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:24.103542 16539 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:24.104902 16750 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:24.126358  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:24.127384  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:24.125352  5704 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:24.130142  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:24.087384 16721 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.137359  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:24.138667 16760 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:29:24.139084 16752 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:24.138990  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:24.120731 16716 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:24.140779 16705 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:24.134073 16539 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:24.129824 16260 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:24.142103  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:24.118758 16688 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:24.126977 16665 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:24.161197 16688 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.161562 16665 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:24.164465 16742 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.572s	user 0.003s	sys 0.001s
I20260709 12:29:24.161229  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:24.167459  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4069) took 1044 ms. Trace:
I20260709 12:29:24.167896  5556 rpcz_store.cc:276] 0709 12:29:23.122887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:23.539228 (+416341us) service_pool.cc:224] Handling call
0709 12:29:23.539757 (+   529us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:23.743898 (+204141us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:24.167197 (+423299us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b540023fd54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:24.167397 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":458496896}
I20260709 12:29:24.169154  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:24.170286  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:24.163839 16688 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:24.148931  5558 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 36.
I20260709 12:29:24.149782  5708 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 35: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:24.156826  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:24.154528 15682 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:24.160559  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:24.150180 16541 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:29:24.172287  5556 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:29:24.173316  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:24.156054 16716 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:24.173039  6276 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:29:24.153963 16683 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:24.126817 16611 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:24.189824  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:24.191006  5553 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 33 index: 23. (term mismatch)
I20260709 12:29:24.193281  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:24.195375  5553 pending_rounds.cc:85] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 22
I20260709 12:29:24.195734  5553 pending_rounds.cc:107] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 32.23
I20260709 12:29:24.196012  5553 raft_consensus.cc:2889] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:24.197304  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:24.199038  6272 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:24.197866  5557 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:24.195544  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1934) took 1153 ms. Trace:
I20260709 12:29:24.200809  6275 rpcz_store.cc:276] 0709 12:29:23.042357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:23.046991 (+  4634us) service_pool.cc:224] Handling call
0709 12:29:23.053152 (+  6161us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:23.067068 (+ 13916us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:23.067089 (+    21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:23.067121 (+    32us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:29:23.067147 (+    26us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:24.195339 (+1128192us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b54006121d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:24.195384 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:24.195493 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31491840}
I20260709 12:29:24.178001 16449 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.163839 16665 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:24.184075 16510 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:24.212821  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:24.187551 16541 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 38 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:24.216215  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:24.217823  5553 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 29: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:24.220310  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:24.186780 15682 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:24.221545 16682 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:24.223294 15682 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:24.224653 16674 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:24.225982 16674 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:24.204576 16755 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:24.226513 16674 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:24.227718 16674 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:24.228686 16692 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:24.215885 16579 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.014s
I20260709 12:29:24.233687  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:24.234663  5553 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:24.220635 16510 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:24.240733  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:24.234145  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:24.229780 16674 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:24.247175 16510 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:24.255573  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:24.255573  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:24.247483 16721 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:24.256502  5553 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:24.259531  5705 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:29:24.243072 16715 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:24.269781  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:24.272109  5705 raft_consensus.cc:2410] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 26 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:29:24.273026  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:24.274174 16774 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:29:24.277966  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:24.283829  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:24.287060  6277 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 37: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:24.291868  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:24.292850  5706 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:24.314165  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:24.333809  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:24.335629  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:24.266769 16683 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.029s
I20260709 12:29:24.339226  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.341388  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.342453  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:24.343351  5628 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:24.343477  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.345356  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.346560  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.347728  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.343282  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.351354  5629 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:29:24.375162  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.415994  5628 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
W20260709 12:29:24.416428  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 113216us
I20260709 12:29:24.417137  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:24.352586  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:24.419420  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:24.419816  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:24.302985 16541 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.423281  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.424897  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.420495  5632 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:24.428947  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:24.429575  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:24.430315  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:24.431864  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 79374.8us
I20260709 12:29:24.351868 16700 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:24.434007  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.449823 16700 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.452873 16700 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:24.455031  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1915) took 1179 ms. Trace:
I20260709 12:29:24.435433  5627 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 27. Current term is 28. Ops: []
I20260709 12:29:24.435290  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:24.455502  5701 rpcz_store.cc:276] 0709 12:29:23.275522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:23.275672 (+   150us) service_pool.cc:224] Handling call
0709 12:29:23.307110 (+ 31438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:23.320411 (+ 13301us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:29:23.790219 (+469808us) spinlock_profiling.cc:243] Waited 993 us on lock 0x7b18004e9160. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:29:23.887373 (+ 97154us) log.cc:844] Serialized 14 byte log entry
0709 12:29:24.067135 (+179762us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:24.070377 (+  3242us) log.cc:844] Serialized 26 byte log entry
0709 12:29:24.070571 (+   194us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:24.071680 (+  1109us) log.cc:844] Serialized 14 byte log entry
0709 12:29:24.073057 (+  1377us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:24.083952 (+ 10895us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:24.162334 (+ 78382us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b18004e9160. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:29:24.279055 (+116721us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b18004e9160. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:29:24.454769 (+175714us) raft_consensus.cc:1685] finished
0709 12:29:24.454797 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:24.454936 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17110,"prepare.queue_time_us":92856,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":3067,"raft.queue_time_us":49990,"raft.run_cpu_time_us":381,"raft.run_wall_time_us":7830,"spinlock_wait_cycles":62156160,"thread_start_us":224432,"threads_started":2,"wal-append.queue_time_us":178654}
I20260709 12:29:24.458288  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:24.459041  6277 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:29:24.436297 16419 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:24.437429  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.440281  5631 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:29:24.445802 16541 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:24.463656  6274 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 23. Preceding OpId from leader: term: 32 index: 24. (index mismatch)
I20260709 12:29:24.448761  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:24.464922  5709 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:24.468339  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:24.395171 16751 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.054s
I20260709 12:29:24.420444 16611 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:24.474298  6274 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37. Ops: []
I20260709 12:29:24.452808  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:24.478025 15538 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:24.483506  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:24.485903 16667 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:24.486366 16667 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:24.487169 16667 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.489745  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3704) took 1308 ms. Trace:
I20260709 12:29:24.490125  5552 rpcz_store.cc:276] 0709 12:29:23.181802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:23.318810 (+137008us) service_pool.cc:224] Handling call
0709 12:29:24.489553 (+1170743us) spinlock_profiling.cc:243] Waited 475 ms on lock 0x7b54000ea1d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:24.489666 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1092673280}
W20260709 12:29:24.497144 16772 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.246s	user 0.001s	sys 0.000s
I20260709 12:29:24.415158 16510 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:29:24.499372  5628 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37. Ops: []
I20260709 12:29:24.504278  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:24.505420  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:24.505992  5701 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 22. Preceding OpId from leader: term: 32 index: 24. (index mismatch)
I20260709 12:29:24.507302  5701 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:29:24.507527  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:24.509042  5630 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:29:24.509254  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:24.510280  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:24.511049  5702 raft_consensus.cc:2410] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 30 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:29:24.504233 16764 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:24.512090  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:24.512012 16764 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:24.513967 16390 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:24.514290 16764 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:24.514415 16390 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:24.434216  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:24.507633  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.514973 16390 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:24.519321  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:24.516047  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:24.520560 16665 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:24.520644  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.521059 16665 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:24.521286 16747 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.925s	user 0.003s	sys 0.000s
I20260709 12:29:24.521495 16665 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:29:24.530261  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 96118.8us
W20260709 12:29:24.531016  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:24.531229  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:24.450179  5632 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:24.538034  5629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:24.549561  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:24.550529  5632 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:24.539567  5629 raft_consensus.cc:606] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:24.553474  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:24.554154  5561 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:24.560002  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:24.572690  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 46813us
I20260709 12:29:24.573616  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:24.574915  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:24.575037  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:24.576318 16771 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:24.577013 16771 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:24.577544  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:24.577708 16771 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:29:24.578092  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:24.578392 16715 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:24.578549 16771 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:24.578877 16715 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:24.578948 16771 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:24.579363 16715 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:24.580591 16716 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:24.581126 16750 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:24.582700  5560 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e69273530292453d87614627debcc13e"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:49044
I20260709 12:29:24.526670 16667 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:24.586315  5633 raft_consensus.cc:2410] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 36 because replica has last-logged OpId of term: 35 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 21.
I20260709 12:29:24.548790  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:24.589036  5554 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 35. Current term is 36. Ops: [35.22-35.22]
I20260709 12:29:24.590044  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:24.586460  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:24.545789 16751 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:24.604597 15538 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:24.603441  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:24.608613 15538 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:24.609143 15538 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:24.604393  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:29:24.550783 16588 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:29:24.617401  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 68710.6us
I20260709 12:29:24.623636  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:24.624162  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:24.642351 16716 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:24.643436 16770 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:24.588095 16550 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.057s
I20260709 12:29:24.644646  5704 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35. Ops: [34.19-34.19]
I20260709 12:29:24.603504 16419 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:24.646896 16768 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:24.647485 16768 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:24.647852 16768 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:24.656569  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:24.657390  5703 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:24.659058 16683 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:24.659943 16683 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 36 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:24.578868 16703 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:24.615387  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:29:24.668502  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 47427.4us
W20260709 12:29:24.668869  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44809.7us
I20260709 12:29:24.671512 15538 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.672042  6275 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:24.669749  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:24.672406 16764 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.063s
W20260709 12:29:24.759140  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 89467.2us
I20260709 12:29:24.675851  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:24.759845  5553 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:24.760972  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:24.680263 16767 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.562s	user 0.004s	sys 0.006s
I20260709 12:29:24.678323  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:24.671573 16752 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.012s
I20260709 12:29:24.607887 16751 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:24.610610 16713 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.079s
I20260709 12:29:24.662972 16579 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:24.711305  5709 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 37.
I20260709 12:29:24.713721  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:24.661180 16683 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:24.669737 16665 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:24.718043 16539 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:24.667136 16768 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.814116 16768 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:24.732767 16700 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:24.667898 16703 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:24.814924 16703 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:24.738179  5558 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:29:24.740790  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:24.748935 16449 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:24.820577 16514 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.038s
I20260709 12:29:24.814925 16700 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:24.820295  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 399923us
I20260709 12:29:24.769964 16688 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:24.753396 16390 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:24.755700 16771 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.673321 15792 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:24.759912  5552 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:29:24.681339  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:24.761760  5553 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:24.854486  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:24.764621  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:24.710949 16755 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.029s
I20260709 12:29:24.711265 16592 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.047s
I20260709 12:29:24.778829  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:24.779850 15538 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:24.793781 16715 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.794866 16705 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:24.829543  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:24.917708  5709 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:24.749524  5632 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:24.820668  5556 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:24.919322  5636 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: [32.19-33.20]
I20260709 12:29:24.827373 16700 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:24.928313  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:24.932758  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51387.9us
I20260709 12:29:24.940740 16716 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:24.781704 16579 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:24.867152  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 102672us
I20260709 12:29:24.949962 16579 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:24.950595 16688 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:24.951570 16688 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:24.769450 16719 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.100s
I20260709 12:29:24.851735 16697 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.105s
I20260709 12:29:24.958082 16677 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:24.958459 16677 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:24.851819  6275 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 37.
I20260709 12:29:24.770889 16751 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:24.963779  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:24.870548  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:24.977545  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 107103us
I20260709 12:29:24.959223 16677 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:24.981544  5554 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: [29.15-29.15]
I20260709 12:29:24.777252 16611 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:29:24.990850  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:24.920579 16419 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:24.865842 16539 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:25.057021 16539 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:25.062382 16390 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:25.064452 16692 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:29:24.914350 16510 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:25.067400 16510 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:25.069612 16510 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:25.076893 16771 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:25.076541 16419 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.079452 16419 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:24.998405 16700 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:25.080559 16700 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:25.081071 16700 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:24.913897 16768 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.170s
I20260709 12:29:25.085551 16719 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:25.087877 16719 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:25.088419 16588 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:29:25.088871 16588 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:25.089591 16611 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:25.090394 16611 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.060189 16539 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:25.096879  5552 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 35 because replica has last-logged OpId of term: 32 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 18.
I20260709 12:29:24.993358  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:25.052769 16278 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:25.100965  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 107729us
I20260709 12:29:24.917420 16705 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:25.055910 16665 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:25.105022  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:25.117836  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:25.113507  5556 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:25.119891  6279 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:29:25.128191  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:25.138863  5524 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:25.141114  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:25.140718 16667 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:25.151775  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:25.049031 16755 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:25.152982  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:25.155694  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:25.159492  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:25.163897  5553 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:29:25.164491  6277 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.068s	sys 0.113s
W20260709 12:29:25.164839  6277 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.068s	sys 0.113s
I20260709 12:29:25.120584 16539 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:29:25.165728  6271 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.876s	user 0.009s	sys 0.027s
W20260709 12:29:25.166182  6271 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.879s	user 0.010s	sys 0.028s
I20260709 12:29:25.175371  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:25.176268  5632 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:25.148669 16390 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.029s
I20260709 12:29:25.119804 16278 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.117023 16719 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.188731  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:25.188580  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:25.172907 15538 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:25.221284 15538 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:25.226253  5561 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.000s	sys 0.019s
I20260709 12:29:25.154911 16419 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 16, Time since last communication: 0.035s
W20260709 12:29:25.232074  5705 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.005s	sys 0.015s
W20260709 12:29:25.234550  5705 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.005s	sys 0.015s
I20260709 12:29:25.236940  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:25.155629 15792 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.061s
I20260709 12:29:25.170779 16755 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:25.228675  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44675.2us
I20260709 12:29:25.222618 16541 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election won for term 35
W20260709 12:29:25.249119  5559 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.010s	sys 0.013s
W20260709 12:29:25.249611  5559 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.010s	sys 0.013s
I20260709 12:29:25.249925  5636 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
W20260709 12:29:25.231720  6234 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.006s	sys 0.013s
I20260709 12:29:25.171666 16721 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.063s
W20260709 12:29:25.255542  6237 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.008s	sys 0.013s
W20260709 12:29:25.256043  6237 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.008s	sys 0.013s
W20260709 12:29:25.256420  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 594851us
W20260709 12:29:25.259013  5555 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.007s	sys 0.012s
W20260709 12:29:25.259783  5555 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.007s	sys 0.012s
I20260709 12:29:25.264562  5560 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:25.265146  5560 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:29:25.266738  6234 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.006s	sys 0.014s
W20260709 12:29:25.267407  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 592099us
I20260709 12:29:25.267691  5553 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
W20260709 12:29:25.227037  5561 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.000s	sys 0.020s
W20260709 12:29:25.240918 16712 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.000s	sys 0.020s
I20260709 12:29:25.273247  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:25.295168  5556 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
W20260709 12:29:25.295655  5635 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.002s	sys 0.018s
W20260709 12:29:25.296895  5635 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.002s	sys 0.019s
I20260709 12:29:25.305130  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:25.209731 16705 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.316852  5553 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: []
W20260709 12:29:25.283221 16712 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.000s	sys 0.021s
I20260709 12:29:25.271059 16278 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:25.248226 16667 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.051s
I20260709 12:29:25.270202 16611 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:25.349433 16755 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:25.275486 16541 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:25.372695 16783 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 12:29:25.376277  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1690) took 1133 ms. Trace:
I20260709 12:29:25.349324  5552 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35. Ops: [34.19-34.19]
I20260709 12:29:25.294958 16390 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:29:25.403331  6236 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.016s	sys 0.007s
I20260709 12:29:25.309476 16703 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.421409  5553 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:29:25.471464  6236 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.809s	user 0.016s	sys 0.008s
W20260709 12:29:25.472268  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 853429us
I20260709 12:29:25.416054  6277 rpcz_store.cc:276] 0709 12:29:24.242579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:24.459764 (+217185us) service_pool.cc:224] Handling call
0709 12:29:24.460147 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:24.475691 (+ 15544us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:24.475727 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:24.476262 (+   535us) log.cc:844] Serialized 26 byte log entry
0709 12:29:25.165377 (+689115us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:29:25.177236 (+ 11859us) log.cc:844] Serialized 14 byte log entry
0709 12:29:25.177458 (+   222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:25.177535 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:25.375940 (+198405us) raft_consensus.cc:1685] finished
0709 12:29:25.375971 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:25.376183 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":12108,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":27909,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":26132,"thread_start_us":688693,"threads_started":1,"wal-append.queue_time_us":896791}
W20260709 12:29:25.473551  5558 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.005s	sys 0.010s
W20260709 12:29:25.477980  5558 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.005s	sys 0.010s
W20260709 12:29:25.486302 16743 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.039s	sys 0.042s
W20260709 12:29:25.486968 16743 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.039s	sys 0.042s
I20260709 12:29:25.410648  5560 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:25.467427 16755 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Stepping down as leader of term 33
W20260709 12:29:25.503201  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 198290us
I20260709 12:29:25.468078 16611 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:25.512989 16260 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:25.522408 16703 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:25.592721 16755 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:25.623469 16755 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:25.625131 16755 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:25.626796 16766 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:25.627549 16766 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.630340 16766 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:25.523516 16784 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:29:25.635138  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:25.480008 16688 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:25.637123 16688 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:25.531720  5527 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.011s	sys 0.109s
I20260709 12:29:25.487246 16588 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.22, Last appended by leader: 22, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:25.544461  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.024s	sys 0.022s
W20260709 12:29:25.640173  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.024s	sys 0.022s
W20260709 12:29:25.641074  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 601084us
I20260709 12:29:25.478645  6236 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:25.598623  5601 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:29:25.643203  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44670.6us
I20260709 12:29:25.599937  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:25.600956 15792 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:25.601964  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:25.532451 16751 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:25.546967 16541 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:25.528878 16700 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.531014 16705 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:25.609431  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 1926) took 1161 ms. Trace:
W20260709 12:29:25.639858  5527 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.011s	sys 0.109s
W20260709 12:29:25.651062  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 172536us
I20260709 12:29:25.653899 15792 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.661731  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:25.663053 16750 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:25.653669 16278 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:29:25.653793 16789 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:29:25.642076  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:25.664547  5705 rpcz_store.cc:276] 0709 12:29:24.447893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:24.448025 (+   132us) service_pool.cc:224] Handling call
0709 12:29:24.448430 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:24.524654 (+ 76224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:24.524688 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:24.525133 (+   445us) log.cc:844] Serialized 26 byte log entry
0709 12:29:25.235123 (+709990us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:25.235167 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:25.235235 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:25.609184 (+373949us) raft_consensus.cc:1685] finished
0709 12:29:25.609211 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:25.609365 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75804,"raft.queue_time_us":76817,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":11803,"thread_start_us":709612,"threads_started":1,"wal-append.queue_time_us":872517}
W20260709 12:29:25.668506  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:25.670297 16700 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:25.611269 16260 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:25.613458 16721 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:25.652751 16771 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.575s	user 0.014s	sys 0.000s
I20260709 12:29:25.658483  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:25.683120  6274 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:25.678972 16260 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:25.707721  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.773s	user 0.016s	sys 0.031s
I20260709 12:29:25.639855 16611 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:25.705166 16683 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.024s	sys 0.067s
W20260709 12:29:25.711257  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.016s	sys 0.031s
I20260709 12:29:25.678354 16278 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:25.673171 16692 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.033s
I20260709 12:29:25.688604 16588 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:29:25.715703  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 780834us
W20260709 12:29:25.716235  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.015s	sys 0.054s
W20260709 12:29:25.716758  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.015s	sys 0.054s
W20260709 12:29:25.717877  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 563981us
I20260709 12:29:25.721972  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:25.714818 16683 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.024s	sys 0.067s
I20260709 12:29:25.742341  5553 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:25.689497 16721 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.718101 16750 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:25.752980  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 491112us
I20260709 12:29:25.719650 16688 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:25.757817  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:25.772876  5632 raft_consensus.cc:2410] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 34 because replica has last-logged OpId of term: 32 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 18.
I20260709 12:29:25.785091 16796 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:25.785735 16796 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.792637 16278 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.23, Last appended by leader: 23, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:25.793522  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:25.797137 16713 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:25.797438 16795 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:25.797029  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3712) took 1281 ms. Trace:
I20260709 12:29:25.798414  5555 rpcz_store.cc:276] 0709 12:29:24.516038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:24.535302 (+ 19264us) service_pool.cc:224] Handling call
0709 12:29:24.588468 (+ 53166us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:24.588726 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:29:24.607885 (+ 19159us) log.cc:844] Serialized 14 byte log entry
0709 12:29:25.781650 (+1173765us) log.cc:844] Serialized 14 byte log entry
0709 12:29:25.796670 (+ 15020us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:25.796723 (+    53us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:29:25.796769 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:25.796837 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:25.796966 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54197,"prepare.queue_time_us":20595,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":76,"raft.queue_time_us":349406,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":29063,"thread_start_us":1131758,"threads_started":2,"wal-append.queue_time_us":887453}
I20260709 12:29:25.782338 16688 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:25.806975  5561 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.006s	sys 0.037s
I20260709 12:29:25.800431 16796 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:25.812759  5599 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:25.801193 16683 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:25.820264  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 132492us
I20260709 12:29:25.825326  6271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 1941) took 1646 ms. Trace:
I20260709 12:29:25.807575 16419 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:25.777105 16750 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:25.854681  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 212722us
I20260709 12:29:25.858122  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3717) took 1260 ms. Trace:
I20260709 12:29:25.859149  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:25.858647  5561 rpcz_store.cc:276] 0709 12:29:24.597554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:24.597702 (+   148us) service_pool.cc:224] Handling call
0709 12:29:24.611047 (+ 13345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:24.611305 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:29:25.294980 (+683675us) log.cc:844] Serialized 14 byte log entry
0709 12:29:25.824020 (+529040us) log.cc:844] Serialized 14 byte log entry
0709 12:29:25.857733 (+ 33713us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:25.857795 (+    62us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:29:25.857859 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:25.857923 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:25.858048 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33465,"prepare.queue_time_us":34470,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":77,"thread_start_us":1182571,"threads_started":2}
I20260709 12:29:25.843858  6271 rpcz_store.cc:276] 0709 12:29:24.178820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:24.178944 (+   124us) service_pool.cc:224] Handling call
0709 12:29:24.195313 (+ 16369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:24.212162 (+ 16849us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:29:24.287197 (+ 75035us) log.cc:844] Serialized 14 byte log entry
0709 12:29:25.166660 (+879463us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:25.166734 (+    74us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:29:25.166800 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:25.824770 (+657970us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b54004f9cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:25.824842 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:25.825179 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":121774,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":103,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":11458,"spinlock_wait_cycles":25781504,"thread_start_us":946420,"threads_started":2,"wal-append.queue_time_us":1101753}
I20260709 12:29:25.861089  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:25.869719  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:25.800906 16795 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.854847 16700 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:29:25.857641 16771 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:25.817792 16688 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.862069 16751 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:25.877501 16797 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:25.859349 16683 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:25.896878 16799 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:25.915825  6274 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:25.902691 16797 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:25.917855  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:25.903844 16808 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:25.903559 16611 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:25.933230  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:25.905632 16813 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:25.896521 16771 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.928597  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:25.934846  6274 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: [26.19-27.20]
I20260709 12:29:25.926553 15538 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:25.933977  5707 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:25.945415  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:25.953022  6280 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 25.
I20260709 12:29:25.922488 16703 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:25.961701  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:25.962057 16801 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:29:25.929979 16797 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:29:25.965755  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:25.960467 15538 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:25.969373  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:25.926127 16804 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:25.981101 16804 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:25.957600 16677 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:25.934635 16592 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:25.985947 16592 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:25.989699 16751 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:25.990674 16721 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:25.991192 16794 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.240s	user 0.001s	sys 0.001s
W20260709 12:29:25.991335 16798 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 12:29:25.993196 16719 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:25.970849  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:25.988185 16419 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:26.005187 16419 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:26.015985 16816 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:26.016705 16816 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:25.977826 16611 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:29:26.017184 16816 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:26.017580 16611 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:26.018249 16611 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:25.942282 16808 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:25.979285 16807 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:29:25.979813 16803 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:25.925865 16805 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.042091  5709 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:29:25.943182 16799 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:29:25.984192 16804 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:26.035867  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 71562.7us
W20260709 12:29:26.036851  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 174376us
W20260709 12:29:26.042289  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 138820us
I20260709 12:29:26.033828 16697 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:26.045048  5560 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:26.043437 16719 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.064719  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.067173  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:49044 (request call id 3715) took 1518 ms. Trace:
I20260709 12:29:26.068895  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.035204 16815 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:26.078100  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:26.067878  5560 rpcz_store.cc:276] 0709 12:29:24.545102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:24.545449 (+   347us) service_pool.cc:224] Handling call
0709 12:29:26.006995 (+1461546us) spinlock_profiling.cc:243] Waited 662 ms on lock 0x7b540044dc54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa669a3d 00007f4fff639cd0 00007f4ff8bbe254 00007f4ff8bbe1e2 00007f4ff8bbe111 00007f4ff8bbe08c 00007f4ff8bbd342 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:26.054288 (+ 47293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1523228032}
I20260709 12:29:26.080058  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:26.080744  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:26.081198  5701 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:26.082739 15538 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:26.084244  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.084486 15538 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:26.085124  5556 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:26.090200 16803 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.091133  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:26.091969  5706 raft_consensus.cc:2410] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 39 because replica has last-logged OpId of term: 38 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 22.
I20260709 12:29:26.092824  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.093899  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:26.104740  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.063400 15792 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:26.105585  5558 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 33: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:26.098945 15538 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:26.111475  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:26.065804 16799 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:26.117421  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:26.118111  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:26.119150 16822 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:29:26.083808 16750 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.122175 16800 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:26.096422 16815 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.145248  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:26.127720 16683 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:26.130100  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 79728.1us
I20260709 12:29:26.136740  5553 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:26.142791  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.148881  5558 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:26.090852 16803 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.110285 16390 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.075531 16719 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.152262  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.151852  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.130287 16592 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.112632 16688 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:26.071928 16697 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.164731  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.165287 16697 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.165625  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:26.166004 16813 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:26.166603  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:26.171607 16795 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:26.173157 16821 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:29:26.173345 16808 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.174499 16771 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:26.175908 16815 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:26.177297 16755 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.552s	user 0.014s	sys 0.000s
I20260709 12:29:26.177547 16799 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:26.179595 16705 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.507s	user 0.004s	sys 0.008s
I20260709 12:29:26.150424 16800 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.181799 16719 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:26.181878 16800 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.186084 16755 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.186676 16797 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.186822 16755 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.177403 16611 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.157083  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.162340 16700 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:26.189577  5558 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 38.
I20260709 12:29:26.189651 16700 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.159461  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.090798 16260 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.193359  5552 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:26.155212  5706 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:26.193596  5705 raft_consensus.cc:2368] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27.
I20260709 12:29:26.161857 16812 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:26.064075 16805 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.196759  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.199851  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:26.202026  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:26.199836  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.202630  6274 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:26.207101  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.207895  5558 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:26.215389  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.218261  6276 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: [32.24-32.24]
I20260709 12:29:26.219967  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:26.222707  5553 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:29:26.192229 16700 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:26.229327  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:26.203817 16592 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.234910 16592 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.204205 16766 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:26.241669  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:26.241474 16766 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:26.247511  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:26.247960 16803 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.212406 16812 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.251354 16812 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.255331 16826 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:26.255901 16826 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.158289 16390 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.258419 16826 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:26.259682 16390 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.260706 16700 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:26.261188  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.195209 16771 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:26.262924 16771 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.265394 16771 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:26.266661  6234 leader_election.cc:400] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27.
I20260709 12:29:26.267048  6234 leader_election.cc:403] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:29:26.267839 16791 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 27
W20260709 12:29:26.272559 16588 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.552s	user 0.014s	sys 0.000s
I20260709 12:29:26.273272  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:26.243971 16766 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:26.275775  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:26.280202  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:26.239307 16592 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.290499  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.291332  5561 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:26.291874 16791 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27.
I20260709 12:29:26.214370 16697 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:26.296702 16697 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.296788 16260 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.297174 16697 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:26.240597 16805 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:26.304343  6275 raft_consensus.cc:2410] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 35 because replica has last-logged OpId of term: 34 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 18.
I20260709 12:29:26.306178 16755 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.280529 16588 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:29:26.197438 16611 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:26.311023  6272 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:26.313685 16816 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.315310  5557 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:26.317999 16419 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.319314 16715 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:26.319671  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1775) took 1210 ms. Trace:
I20260709 12:29:26.320267 16419 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.320097  6275 rpcz_store.cc:276] 0709 12:29:25.109028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:25.120758 (+ 11730us) service_pool.cc:224] Handling call
0709 12:29:26.319481 (+1198723us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54002eed54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:26.319612 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2636273920}
I20260709 12:29:26.328182  6274 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:26.329870  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:26.316056 16816 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:26.342334  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62549.5us
I20260709 12:29:26.343504  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.347811 16138 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:26.348403 16138 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.350449 16138 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.351950 16719 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.352679  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.352939  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.355417  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.356231  5709 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:26.357419  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.358209  5706 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:26.359508  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.353286  5701 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:26.353838  5708 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:26.365749  5704 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:26.361856  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.367167  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:26.340126 16419 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:26.341702 16682 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.370908  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.346226 16390 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:26.373507  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.382357  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.383174  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:26.382408  5703 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:26.384298  5636 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:29:26.385416  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:26.376552  5710 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:26.371747  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.377606  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.389770  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.390430  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.387710  5707 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:26.364202 16771 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:26.396016  5636 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:26.391219  5631 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:26.366732 16716 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:26.363797 16797 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.399668  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.399809  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.400398  5632 raft_consensus.cc:2368] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30.
I20260709 12:29:26.400789  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:26.389720  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 244622us
I20260709 12:29:26.398402  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:26.398402  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.415946 16588 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:26.416381 16588 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.416706  5627 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: [27.20-27.20]
I20260709 12:29:26.417356 16755 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:26.417866 16755 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.418622 16755 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:26.399376 16677 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 31.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.025s
I20260709 12:29:26.363781 16719 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.424621  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:26.426029 16835 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:26.426561 16814 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:26.426977 16814 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:26.427862 16814 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 31.24, Last appended by leader: 24, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.402035  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.430652  5629 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 28: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:26.370923 16688 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:26.431411 16688 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:26.431846 16688 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:26.401690  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.409672  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:26.387743 16682 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.432936  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.379541 16419 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.396483 16390 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:26.434281  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.434587 16390 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.435149  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.435829 16803 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:26.436249 16803 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.402287 16260 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.013s
I20260709 12:29:26.425923 16677 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:29:26.437054 16826 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:26.437530 16826 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:26.437860 16826 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:26.436662 16803 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:26.439327 16816 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:26.439710 16816 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:26.440110 16816 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:26.438714  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:26.397615 15538 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:26.442507 15538 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:26.442929 15538 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:26.393862  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:26.463698  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.467209 16805 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:26.467445 16667 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:26.467835 16805 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:26.467990 16667 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:26.468505 16805 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:26.468600 16667 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:29:26.473884  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55838.9us
I20260709 12:29:26.475108  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:26.476087 16278 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:26.476643 16278 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.478735 16278 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.480111 16832 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:26.482438  5629 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:29:26.416760 16588 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:26.480665  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:26.493108  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 91750.3us
W20260709 12:29:26.494201  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 105457us
I20260709 12:29:26.495048  5676 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.447297 16814 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:26.497766  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.502779  5634 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:26.501152  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.504393  5629 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:29:26.503614  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.455336 16796 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:26.398494 16771 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.514940  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:26.515798  5636 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:26.516212  5636 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:26.517351  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.518931  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:26.526628  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.521066  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.526607  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.527432  5635 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 26.
I20260709 12:29:26.527457  5634 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:26.530148  5629 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:26.531155  5631 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:26.531914 16802 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.532629 16802 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.535012 16802 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.541355 16419 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:26.542001 16768 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:26.542651 16768 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:26.547155 16667 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.509116 16700 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:26.549320 16700 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.498689 16719 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:26.511222 16592 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.522704  5636 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.508451 16138 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:26.513432 16832 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:29:26.563283  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.564879  5634 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 28: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:26.566027  5636 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:29:26.553248 16822 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:29:26.567529  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.582398 16138 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.585597 16138 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:26.581004  5629 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:26.581056 15538 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.587177  5632 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:26.543658 16768 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.592859  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.590652 15538 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:26.593858  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.595170  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.603040 16802 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:26.603710 16802 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.606400 16802 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:26.607808  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:26.564304 16682 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:26.586138 16697 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.613498  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.650164  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.636008  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.570223 16419 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.641755 16697 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.646302  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.647053  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.663201  5635 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:26.664635  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:26.667117  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.647229  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.668772  5635 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 30: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:26.668641  5627 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 39.
I20260709 12:29:26.647699  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:26.670054  5630 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:29:26.670228  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:26.649421  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.651189  5634 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:26.648715  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.671516  5629 raft_consensus.cc:2410] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 37 because replica has last-logged OpId of term: 35 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 21.
I20260709 12:29:26.672796  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.674397  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.674857  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:26.675499  5553 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:26.675534  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.677438  5631 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:26.678041  5559 raft_consensus.cc:2410] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 29 because replica has last-logged OpId of term: 27 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 19.
I20260709 12:29:26.679795  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:26.677911  5636 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:26.685580  5632 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:26.661329 16802 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:26.687994  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.690371  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:26.690613  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.587219 16592 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.698305  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:26.700613 16833 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:26.695919 16799 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:29:26.701061 16665 consensus_queue.cc:1059] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 33.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:29:26.671016 16768 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:26.661950 16716 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:26.707427  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:26.710043  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 216124us
W20260709 12:29:26.710179  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 215159us
I20260709 12:29:26.711637  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.687078 16831 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:26.715600  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.716936  5556 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:26.731382  5553 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:26.672123 16419 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.687984 16816 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.736797  5555 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:26.692024 16802 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.746320  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.747639  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.748857  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.749315  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:26.749905  5708 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:29:26.752466  5706 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 27. Current term is 28. Ops: []
I20260709 12:29:26.743796 16703 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:26.756546  5704 raft_consensus.cc:2410] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 29 because replica has last-logged OpId of term: 27 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 19.
I20260709 12:29:26.757378  5558 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:26.757650  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.759785  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:26.760376  5706 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:26.748173  5710 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:26.760730 16703 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:26.760891 16840 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:29:26.762039 16839 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:29:26.762491  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.758248  5704 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:26.765879  5709 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:29:26.770401 16667 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:26.773437 16846 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:29:26.777475  5630 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:26.781442 16766 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:26.782070  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 70519.1us
W20260709 12:29:26.782578  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 68982.1us
I20260709 12:29:26.782996 16541 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:29:26.784554 16716 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.786856 16802 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:26.787103 16390 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:26.789142  5554 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:26.790519  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.806789  5636 raft_consensus.cc:2368] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27.
I20260709 12:29:26.793367  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.797456  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:26.798450  5703 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:26.800146  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.800933  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.814267  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:26.815876  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.817826  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.819723  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.819770  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:26.820294  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:26.820927  5634 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:26.813454  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.821395  5635 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:26.796424 16841 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:26.834585  5708 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:26.835180  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:26.836212  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.800740 16541 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:26.850315 16541 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:29:26.853693  5634 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:26.856698 16688 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.839793  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.838613  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:26.848464 16816 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.806919 16771 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:26.811144 16766 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.806010 16697 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:26.812093 16797 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.043s
I20260709 12:29:26.859862  5707 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:26.798239 16805 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.837572 16842 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:29:26.838764 16832 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:26.832564 16719 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:26.838176 16592 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.838624 16715 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:26.842954 16721 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:26.881821  6236 leader_election.cc:400] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 26. Current term is 27.
I20260709 12:29:26.875753 16831 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:26.875394 16771 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:26.887507 16755 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.883411 16719 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.888975 16719 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:26.890343 16755 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.824285 16390 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.929531 16390 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:26.891428 16847 log.cc:927] Time spent T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 12:29:26.931321 16826 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.893647 16688 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:26.876014 16697 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.895087 16797 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:26.900444 16831 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:26.901613 16842 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:26.906281 16541 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.912665  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:26.922906 16805 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.885015 16832 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:26.842995 16841 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.865744 16138 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.060s
I20260709 12:29:26.866837 16278 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 21, Time since last communication: 0.060s
I20260709 12:29:26.891460 16766 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:26.882252 16716 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:26.893288 16721 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:26.933831  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 98744.1us
I20260709 12:29:26.939251 16831 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:26.941311 16797 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:26.903923 16719 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:26.944996  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 108856us
I20260709 12:29:26.945128  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:26.945554  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:26.927402  5561 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:26.955786  5628 pending_rounds.cc:85] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 21
I20260709 12:29:26.956138  5628 pending_rounds.cc:107] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 35.22
I20260709 12:29:26.956385  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:26.956415  5628 raft_consensus.cc:2889] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:26.948318  6274 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 35 index: 19. (term mismatch)
I20260709 12:29:26.958110  6274 pending_rounds.cc:85] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 18
I20260709 12:29:26.958366  6274 pending_rounds.cc:107] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 34.19
I20260709 12:29:26.958628  6274 raft_consensus.cc:2889] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:26.976960  5634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:26.977638  5634 raft_consensus.cc:606] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:26.977991  5634 raft_consensus.cc:614] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:26.947279 16832 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:26.981276  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:26.946868 16813 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:26.953523 16716 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:26.958583 16721 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:26.986864 16541 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:26.987757 16803 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.549s	user 0.008s	sys 0.005s
I20260709 12:29:26.989351 16588 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:26.991323 16705 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:26.993602  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:27.002693  5675 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:27.005813 16588 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:27.008630 16859 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:29:27.001825 16858 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:29:27.013406 16826 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:27.002015 16803 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.013998 16721 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.013473 16831 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.015367  5553 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:27.015830 16803 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:27.016281 16721 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:26.996237 16716 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:27.017114 16831 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:27.019490 16697 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:26.981950 16510 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:27.021564 16510 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:26.993566 16766 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.030s
I20260709 12:29:27.025543  5634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:27.026729  5634 raft_consensus.cc:606] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:27.028702 16700 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:26.996443 16842 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:27.034590  5631 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:27.037602 16797 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:27.039108  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:27.039178  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
W20260709 12:29:27.043644  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:27.049625 16859 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:29:27.049752  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47175us
I20260709 12:29:27.050266 16859 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:27.050704 16859 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:27.050875  5675 leader_election.cc:400] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30.
I20260709 12:29:27.051401  5675 leader_election.cc:403] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:29:27.052615 16768 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30.
I20260709 12:29:27.055294 16703 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:27.057194  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:27.058238 16611 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:29:27.059098 16719 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:27.059703 16858 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:27.060155 16858 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 28 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:27.059799 16721 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:27.061266 16820 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:27.061049 16858 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.061666 16820 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:27.062310 16858 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:27.062530 16820 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.063753 16820 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:27.059463 16611 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:27.067418 16841 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:27.069283  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:27.070046  5553 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:27.070875  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:27.072750  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:27.072774 16861 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:29:27.073730 16721 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:27.074093 16831 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:27.074147 16721 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:27.074769 16831 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:27.075065 16721 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:27.075732 16831 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.077144 16831 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:27.077680  5555 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:27.080968  5553 raft_consensus.cc:2410] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 38 because replica has last-logged OpId of term: 36 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 21.
I20260709 12:29:27.083055  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:27.084618  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:27.085738 16697 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:27.074088 16859 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.092247  6272 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:27.095453  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:27.097355 16716 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.099798 16713 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:27.100091 16716 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:27.100273 16713 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:27.102821  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:27.103150  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:27.108090  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:27.109254  5632 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 31 because replica has last-logged OpId of term: 29 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:29:27.023118 16510 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.113981  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:27.115383  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:27.119069  5634 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 28: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:27.115097  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:27.119879  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:27.121011  5636 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:27.115551  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:27.123896  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:27.124626  5630 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:29:27.125584  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:27.124791  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:27.137310  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:27.071828 16832 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.155087 16832 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:27.140096  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:27.156132  5675 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:27.156713  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:27.157382  6275 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:29:27.157492  6280 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 37.
I20260709 12:29:27.159121  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:27.159822 16541 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:27.161123  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:27.162010  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:27.162533  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:27.162564  5710 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:27.152887 16860 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:27.159463  5629 raft_consensus.cc:2368] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33.
I20260709 12:29:27.138036  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:27.172842  6274 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:29:27.151965  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:27.164641  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:27.172686  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:27.167646  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:27.176932  5707 raft_consensus.cc:2368] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33.
I20260709 12:29:27.169750  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:27.175415  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:27.142980  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:27.187652  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:27.190414  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52498.1us
I20260709 12:29:27.191020  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:27.196543  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:27.198894  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:27.199810  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:27.200825  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:27.160235 16541 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:27.128811 16841 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:27.215828  5636 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:27.202595  5702 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:27.197484 16805 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:27.218910 16805 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:27.219480 16805 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:27.220117  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:27.220788  6274 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:29:27.198315 16700 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:27.204522 16705 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:27.205694  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:27.205595  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:27.200891  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:27.218791  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:27.129238 16697 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:27.224112  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:27.162822 16826 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:27.163121 16716 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:27.192430 16719 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:27.227072  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:27.228965  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:27.229321  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:27.239449  5634 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:29:27.239580  5635 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 28: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:27.229934  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:27.229032  5636 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:29:27.241096  5630 raft_consensus.cc:2368] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33.
I20260709 12:29:27.240970  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:27.241990  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:27.207331  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:27.243001  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:27.233628  5702 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:27.234767  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:27.247243  6280 raft_consensus.cc:2410] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 38 because replica has last-logged OpId of term: 35 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 22.
I20260709 12:29:27.247687  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:27.198693 16800 raft_consensus.cc:2764] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
I20260709 12:29:27.243604  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:27.198565 16816 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:27.234915  5633 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:27.258998 15538 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:29:27.224538 16278 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:27.262066  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:27.249311  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:27.224776 16541 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:27.233711 16588 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:27.194553 16832 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.059s
W20260709 12:29:27.268733  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 61481.3us
I20260709 12:29:27.269523  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:27.236600 16716 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:27.274528  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:27.267488  5702 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:27.275195  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:27.276954 15538 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.041s
I20260709 12:29:27.286006  5675 leader_election.cc:400] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33.
I20260709 12:29:27.320417  5675 leader_election.cc:403] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
I20260709 12:29:27.275734  5634 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 40.
W20260709 12:29:27.322252  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47217.7us
I20260709 12:29:27.264412 16278 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:27.194334 16860 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:27.233220 16799 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 21, Time since last communication: 0.040s
I20260709 12:29:27.258720 16719 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:27.329793 15538 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33.
I20260709 12:29:27.275238 16861 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:27.267802 16592 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:29:27.332316  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 57873.6us
I20260709 12:29:27.242835 16826 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.333966  5710 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
I20260709 12:29:27.268226 16390 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:27.242364 16697 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:27.275028 16816 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:29:27.335608 16755 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:27.271613 16667 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:29:27.329731  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44100.2us
I20260709 12:29:27.331534 16278 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:27.232100 16713 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.330499 16716 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:27.339483 16797 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:27.429178 16797 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:27.429795 16797 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:27.354625  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:27.398617 16862 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:29:27.396793 16719 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.431905 16800 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:27.432381 16800 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:27.433321 16800 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:27.434825 16611 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:29:27.365428 16755 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:27.364753 16859 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:27.355682 16824 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.242s
I20260709 12:29:27.351260 16390 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:29:27.420434  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 77352us
I20260709 12:29:27.367424 16829 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Stepping down as leader of term 29
W20260709 12:29:27.434556  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 80018.4us
I20260709 12:29:27.436240 16611 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 40 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:27.436728 16755 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:27.354585 16703 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:27.434404 16800 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:27.445423  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:27.455492 16840 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:27.456081 16840 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:27.474254 16805 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:27.489786  5710 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:27.493865 16830 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:27.504736  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:27.456938  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:27.469264 16860 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:27.470129  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:27.509637 16683 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:27.509704 16611 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.510347 16683 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:27.511377 16683 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:27.513460  5631 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:27.515425 16830 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 33.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:27.467470  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:27.475389 16419 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:27.475932 16826 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:27.476994 16697 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:27.478438 16864 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:29:27.445048 16829 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:27.517789  5627 raft_consensus.cc:2368] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:27.518093 16829 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:27.519572 16829 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:27.479547  5553 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:27.484809 16870 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:27.516413 16752 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:27.521063 16752 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:27.522257 16390 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:27.523305  5701 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
W20260709 12:29:27.523526  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53530.8us
I20260709 12:29:27.521965 16752 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:27.494038  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:27.525348 16752 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:27.527410  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:27.528512  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 84197.4us
I20260709 12:29:27.529125  5710 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 20. (term mismatch)
I20260709 12:29:27.529529  5710 pending_rounds.cc:85] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 19
I20260709 12:29:27.529762  5710 pending_rounds.cc:107] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 27.20
I20260709 12:29:27.529997  5710 raft_consensus.cc:2889] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:27.530273  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:27.532976  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:27.533167  5630 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:29:27.534765 16870 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:27.471019  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:27.537230 16805 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:27.537472 16870 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:27.538066 16805 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:27.538542 16805 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:27.550354  6275 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:27.537289  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:27.544525  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:27.552943  5704 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 33: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:27.546608  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:27.540977 16278 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:27.554502  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:27.555840 16766 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:27.552671 16390 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:27.557080 16390 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:27.557929  5631 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:29:27.558426 16872 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:27.558696  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:27.558974 16872 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:27.559860  5526 leader_election.cc:400] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33.
I20260709 12:29:27.560293  5526 leader_election.cc:403] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
I20260709 12:29:27.562477  5526 leader_election.cc:400] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:27.562794  5526 leader_election.cc:403] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:29:27.562857  5627 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 33: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:27.564184  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:27.564836  6277 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
W20260709 12:29:27.565523  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 162017us
I20260709 12:29:27.567507  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:27.564083  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:27.572290  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:27.573575  5631 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:29:27.574187  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:27.574819  5630 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:27.576934  5524 leader_election.cc:400] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33.
I20260709 12:29:27.576941  6277 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:29:27.577932  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:27.570811 16419 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:27.579502 16863 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:29:27.583403  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:27.552453 16805 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:27.587488  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:27.558187 16390 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.588492  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:27.585301  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:27.585083  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:27.594396 15538 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:27.594911 15538 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:27.595484 15538 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:27.571420 16826 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:27.582660 16870 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:27.582247 16616 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:27.544544 16721 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.592937 16799 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:27.581986 16858 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.025s
I20260709 12:29:27.590022 16859 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.029s
I20260709 12:29:27.628782 15538 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:27.650043 16875 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:27.650751 16875 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:27.653945 16541 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.655294  5703 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:27.657945 16833 raft_consensus.cc:3055] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:27.658360 16833 raft_consensus.cc:740] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:27.659297 16833 consensus_queue.cc:260] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.662210 16872 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:27.663702 16833 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:27.651077 16721 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:27.672912  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:27.673673  6280 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:27.665675 16278 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:27.602069 16700 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.042s
W20260709 12:29:27.735894  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:27.643618 16870 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:27.757695  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:27.732913 16766 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:27.729120 16541 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:27.774359  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:27.772384 15538 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:27.732960 16841 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.040s
W20260709 12:29:27.791040  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:27.793082  6280 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 38.
I20260709 12:29:27.795544  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:27.795945  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:27.796602  5558 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:27.797845  5552 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:27.800415  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:27.802603  5555 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 34 index: 20. (term mismatch)
I20260709 12:29:27.802987  5555 pending_rounds.cc:85] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 19
I20260709 12:29:27.803213  5555 pending_rounds.cc:107] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 33.20
I20260709 12:29:27.803435  5555 raft_consensus.cc:2889] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:27.801036  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:27.807345  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:27.781194 16755 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:27.808719  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:27.744984 16870 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:27.816840  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 238978us
I20260709 12:29:27.817922  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:27.820945  5552 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:29:27.815850 16700 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:29:27.830211  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:27.831284  5560 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Advancing to term 40
I20260709 12:29:27.831737  6271 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:27.836527  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:27.834825  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:27.846342  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:27.847967  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:27.875028  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:27.878723  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:27.833921 16875 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:27.967288  5560 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 40.
W20260709 12:29:27.962250  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 144399us
I20260709 12:29:27.894824  5559 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:27.983018  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:27.864091 16858 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:27.870169 16700 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:28.017508  5556 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:29:27.841738 16755 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:28.034030  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:27.990825 16875 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:28.042109  5555 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:28.050863  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:28.054860  5559 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 33: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:28.016597 16858 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:28.056206  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:28.074342  5555 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:28.001530 16592 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:28.046536 16820 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.982s	user 0.016s	sys 0.004s
W20260709 12:29:28.092536  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:28.107652  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:28.093338 16887 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:29:28.094256 16878 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:29:28.094553 16700 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:28.099924 16884 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
I20260709 12:29:28.100788 16870 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:28.044914 16877 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:28.108466  5560 raft_consensus.cc:2393] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:28.110013  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:28.112466  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:28.140993  5627 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:28.087412 16588 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.808s	user 0.012s	sys 0.003s
I20260709 12:29:28.055294 16891 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:28.119920  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:28.127449  5559 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
W20260709 12:29:28.157577  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 146287us
I20260709 12:29:28.158497  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:28.162590  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:28.148028 16881 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.275s	user 0.000s	sys 0.001s
W20260709 12:29:28.112166 16419 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.532s	user 0.009s	sys 0.003s
W20260709 12:29:28.154227 16683 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.008s	sys 0.017s
I20260709 12:29:28.168138  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:28.168915  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:28.157279 16588 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:28.132222 16877 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:28.178476  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:28.179677  5555 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
W20260709 12:29:28.144596  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 161855us
I20260709 12:29:28.160852 16886 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:28.167208 16870 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:28.184446  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:28.175872 16683 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.009s	sys 0.018s
I20260709 12:29:28.117049 16592 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:28.171725 16419 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
W20260709 12:29:28.178299 16882 log.cc:927] Time spent T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.317s	user 0.001s	sys 0.001s
W20260709 12:29:28.194360  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:28.195442 16894 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:28.207067  6276 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:28.214231  5560 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:28.161055 16887 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.028s
I20260709 12:29:28.218693  5552 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
W20260709 12:29:28.220741  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 97950.4us
I20260709 12:29:28.189203 16896 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:28.234179  5636 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:28.214938 16870 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:29:28.237798  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44195.4us
I20260709 12:29:28.220332 16824 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:28.175915 16858 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 31.24, Last appended by leader: 24, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:28.240875  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 62522.5us
I20260709 12:29:28.242133  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:28.242341 16888 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:29:28.242888 16868 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:29:28.243223 16879 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:29:28.243315 16868 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:28.232038 16875 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:28.188112 16886 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:28.226920 16898 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:29:28.275303  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:28.258201  5557 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
W20260709 12:29:28.245425 16893 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:29:28.239732  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:28.269874  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:28.239337 16870 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:28.272213 16872 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:28.288194  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:28.273521 16875 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:28.291249 16904 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:29:28.291136 16889 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
W20260709 12:29:28.335847 16869 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.854s	user 0.001s	sys 0.003s
I20260709 12:29:28.299283 16896 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:29:28.299914 16840 consensus_meta.cc:287] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.825s	user 0.012s	sys 0.000s
I20260709 12:29:28.305387 16705 raft_consensus.cc:993] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: : Instructing follower cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:29:28.356091  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:28.359777 15538 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:28.305860  5627 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
W20260709 12:29:28.297438 16885 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.203s	user 0.001s	sys 0.000s
I20260709 12:29:28.285827 16898 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:28.315454 16665 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:28.363736 16665 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:28.364521 16898 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:28.365119 16665 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:28.318401 16829 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.798s	user 0.010s	sys 0.004s
I20260709 12:29:28.274523 16868 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:28.323073 16716 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.893s	user 0.009s	sys 0.003s
I20260709 12:29:28.367187 16868 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:28.367285 16716 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:28.369297  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:28.370194 16716 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:28.327610 16877 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:28.328114  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 3833) took 1123 ms. Trace:
I20260709 12:29:28.328963  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:28.329885 16886 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:28.331005 16592 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:28.331292  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:28.375165  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:28.376803  6274 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:29:28.377156  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:28.378268  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:28.379999 15538 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:28.331766 16833 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.667s	user 0.011s	sys 0.001s
W20260709 12:29:28.296895 16899 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
W20260709 12:29:28.298918 16797 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.853s	user 0.005s	sys 0.010s
I20260709 12:29:28.381903 16797 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:28.285144 16755 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:28.385054  5635 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:28.247723 16824 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:28.289868 16870 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:28.387213 16824 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:28.387496  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:28.345176  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:28.345547  5634 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:29:28.346984  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:28.391929  5631 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:28.352844 16842 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:28.392542 16829 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:28.392838 16842 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:28.393070 16829 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:28.352968 16902 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:28.394999 16842 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:28.395306 16902 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:28.353219 16897 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:28.396721  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:28.396646 16897 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:28.395018 16829 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:28.399132 16897 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:28.401665  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:28.371691  5705 rpcz_store.cc:276] 0709 12:29:27.204547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:27.205488 (+   941us) service_pool.cc:224] Handling call
0709 12:29:28.327768 (+1122280us) spinlock_profiling.cc:243] Waited 622 ms on lock 0x7b54004529d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:28.327911 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1430680960}
I20260709 12:29:28.403245 16870 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:28.405643 16902 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:28.406189 16870 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:28.417061  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:28.403273  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:28.418798  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:28.419667  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:28.420774  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:28.390463  5560 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:28.421094 16592 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:28.421777 16592 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:28.422024  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:28.420531 16755 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:29:28.427454 16797 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:28.428747  5560 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:29:28.380868 15538 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:28.471184  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 114229us
I20260709 12:29:28.470614 16541 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:28.481484  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 84844.9us
I20260709 12:29:28.483189  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:28.487033  5631 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:28.494338  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:28.495249  5705 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:29:28.473515 16805 consensus_meta.cc:287] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.877s	user 0.005s	sys 0.008s
I20260709 12:29:28.499796  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:28.497722  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:28.500324  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:28.470804 16703 raft_consensus.cc:1081] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Signalling peer cea66aa15be3423880843974acdbf50f to start an election
I20260709 12:29:28.524106 16858 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:28.528892 16883 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:28.529100  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3764) took 1299 ms. Trace:
I20260709 12:29:28.529464  5633 rpcz_store.cc:276] 0709 12:29:27.229494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:27.268986 (+ 39492us) service_pool.cc:224] Handling call
0709 12:29:27.269341 (+   355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:27.278630 (+  9289us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:27.278680 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:27.279304 (+   624us) log.cc:844] Serialized 26 byte log entry
0709 12:29:27.424228 (+144924us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:27.753533 (+329305us) log.cc:844] Serialized 14 byte log entry
0709 12:29:27.766490 (+ 12957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:27.766569 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:28.335101 (+568532us) spinlock_profiling.cc:243] Waited 43.9 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:28.521461 (+186360us) raft_consensus.cc:1685] finished
0709 12:29:28.528839 (+  7378us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:28.529036 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21455,"prepare.queue_time_us":61559,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":66176,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":12218,"spinlock_wait_cycles":100880768,"thread_start_us":473499,"threads_started":2,"wal-append.queue_time_us":200621}
I20260709 12:29:28.538601  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:28.539597 16833 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:28.498682 16895 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:28.479735 16541 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:28.555717 16541 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:28.496064 16705 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:28.557128 16705 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:28.559260 16705 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:28.357841 16840 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:28.561403 16840 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:28.506253 16875 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 32.24, Last appended by leader: 24, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:28.510872 16805 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:28.567487  5552 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:28.568122  5555 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 38: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:28.568961 16805 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:28.570106 16826 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.968s	user 0.007s	sys 0.006s
I20260709 12:29:28.570534 16826 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33.
W20260709 12:29:28.575300  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:28.580529  5553 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:28.587862  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:28.601970  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:28.602118  5561 raft_consensus.cc:1240] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:28.589041  5631 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
W20260709 12:29:28.591913  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:28.593933  5450 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 24 to 27, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 27 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:28.597239  5558 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360"
dest_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:49044
I20260709 12:29:28.612212  5558 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:28.607934  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:28.612654  5558 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:28.609772  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:28.614250  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:28.615484  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:28.620188  5705 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:28.622570  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:28.621140  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 260806us
I20260709 12:29:28.624192  5552 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 32: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:28.631985  5557 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:29:28.633632  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:28.635802  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:28.625325  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:28.639559  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 285594us
I20260709 12:29:28.644037  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:28.647198  6276 raft_consensus.cc:2393] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 38: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:28.653294  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:28.664438  5631 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:29:28.653878  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:28.682269  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:28.684044  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:28.612727 16803 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:28.611974 16831 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:28.684641  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:28.699457  5559 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 40 index: 24. (index mismatch)
I20260709 12:29:28.700320  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3759) took 1650 ms. Trace:
I20260709 12:29:28.696646 15538 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:28.685629  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:28.685962  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:28.700837  5561 rpcz_store.cc:276] 0709 12:29:27.050023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:27.084283 (+ 34260us) service_pool.cc:224] Handling call
0709 12:29:27.476140 (+391857us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:29:27.484399 (+  8259us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:28.605536 (+1121137us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:28.700074 (+ 94538us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54000271d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:28.700259 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2603150976}
I20260709 12:29:28.701866  5634 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:28.664345  5558 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:28.706187  5552 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:28.691207  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:28.653007 16816 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.210s	user 0.057s	sys 0.060s
I20260709 12:29:28.641966 16824 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:28.766528  5558 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:28.721608  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:28.728566  5634 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:28.779537 15538 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:28.624001 16858 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:28.690073 16800 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:29:28.792615  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 71127.4us
I20260709 12:29:28.780339 15538 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:28.795349  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:28.796974 15538 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:28.797663 15538 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:29:28.722445 16816 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.292s	user 0.057s	sys 0.061s
I20260709 12:29:28.818267 15538 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.002s
I20260709 12:29:28.690222 16859 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 34.20, Next index: 21, Last known committed idx: 17, Time since last communication: 0.127s
I20260709 12:29:28.713944 16840 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.112s
W20260709 12:29:28.845259  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 165286us
I20260709 12:29:28.846390  5601 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:28.720068 16831 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:28.849607  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 132307us
I20260709 12:29:28.719988 16803 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:28.867166  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:28.779201 16824 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:28.833668 16712 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:28.877661 16906 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:28.850880 16902 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:29:28.870839 16907 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.170s	user 0.001s	sys 0.000s
I20260709 12:29:28.881511  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:28.898330 16902 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:28.871361 16768 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:28.912667  5631 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:28.883597  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 4136) took 1083 ms. Trace:
I20260709 12:29:29.017230  5556 rpcz_store.cc:276] 0709 12:29:27.800008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:27.800136 (+   128us) service_pool.cc:224] Handling call
0709 12:29:28.883394 (+1083258us) spinlock_profiling.cc:243] Waited 53.7 ms on lock 0x7b5400026cd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:28.883537 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":123521408}
I20260709 12:29:28.864275 16897 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.036s
I20260709 12:29:29.026351 16824 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:28.956431 16906 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:28.935559 16813 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:28.882886 16866 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:28.898944 16902 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:28.913106  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:28.973953 16716 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:28.884207 16829 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:28.900712 16800 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:29:29.005318  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:29.039166  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4308) took 1248 ms. Trace:
I20260709 12:29:28.929639 16667 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.016934 16703 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.000058 16138 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.037s
W20260709 12:29:29.031520 16915 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
I20260709 12:29:28.883205 16712 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:29.044724  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 132351us
I20260709 12:29:29.068118  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.072254  5704 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 29: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:29:29.047430  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 136475us
I20260709 12:29:29.031927 16812 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.048071  5702 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:29.039721  5703 rpcz_store.cc:276] 0709 12:29:27.790324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:27.790473 (+   149us) service_pool.cc:224] Handling call
0709 12:29:27.802961 (+ 12488us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:27.803322 (+   361us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:29:27.803352 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:27.812164 (+  8812us) log.cc:844] Serialized 26 byte log entry
0709 12:29:28.316908 (+504744us) spinlock_profiling.cc:243] Waited 3.37 ms on lock 0x7b18004d1580. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:29:28.468975 (+152067us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:28.469049 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:28.469153 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:29.032595 (+563442us) raft_consensus.cc:1685] finished
0709 12:29:29.032636 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:29.039077 (+  6441us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5082,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":15479,"spinlock_wait_cycles":7745408,"thread_start_us":151790,"threads_started":1,"wal-append.queue_time_us":656847}
I20260709 12:29:29.049254  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:29.053112 16916 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20260709 12:29:29.044234 16829 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.054899 16800 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.067934  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.053682 16910 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.095927 16930 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.096208  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.096537 16930 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.096060 16910 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.098876 16930 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:29.084524 16700 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.099753 16910 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:29.099812 16700 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.102181 16700 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:29.038373 16924 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:29:29.104322 16919 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:29:29.106911 16875 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:29.107465 16831 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:29.044476 16866 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:29.070013  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.034240 16814 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:29.079677  5631 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:29.034610 16716 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:29.018473 16768 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.031s
I20260709 12:29:29.081571  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:29.155583  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:29.086882 16928 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.086s	user 0.004s	sys 0.000s
I20260709 12:29:29.086690 16812 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:29.087268 16917 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.247s	user 0.001s	sys 0.000s
I20260709 12:29:29.048707 16766 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:29.028272 16906 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:29.088379 16871 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.605s	user 0.013s	sys 0.008s
I20260709 12:29:29.089635 16803 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:29.090843  5703 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:29:29.092080  5710 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:29.093211  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.052973 16841 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:29.068539  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.068956  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:29.108925  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 315418us
I20260709 12:29:29.148063  5708 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 28: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:29.150695  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:29.151599 16843 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:29.085902  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.185865  5702 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:29.108011 16858 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:29.048604 16667 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.057726 16703 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.169227  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.150749 16814 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:29.171679  5709 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:29.180519  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:29.201660  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3767) took 1971 ms. Trace:
W20260709 12:29:29.175060  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:29.208797  5632 rpcz_store.cc:276] 0709 12:29:27.230340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:27.273465 (+ 43125us) service_pool.cc:224] Handling call
0709 12:29:27.325128 (+ 51663us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:27.325582 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:27.325623 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:27.334140 (+  8517us) log.cc:844] Serialized 26 byte log entry
0709 12:29:27.417153 (+ 83013us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:27.417220 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:27.417322 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:29.054651 (+1637329us) spinlock_profiling.cc:243] Waited 44.4 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:29.201318 (+146667us) raft_consensus.cc:1685] finished
0709 12:29:29.201344 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:29.201586 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1989,"raft.queue_time_us":13143,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":7683,"spinlock_wait_cycles":102135552,"thread_start_us":69050,"threads_started":1,"wal-append.queue_time_us":111891}
W20260709 12:29:29.219703  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59404.2us
I20260709 12:29:29.224390  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:29.200275 16936 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:29:29.173497 16942 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.230811  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:29.232156  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:29.234722  5635 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:29:29.239838  5561 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 24. Preceding OpId from leader: term: 32 index: 25. (index mismatch)
I20260709 12:29:29.240702  5555 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:29.242755 16766 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:29.246062  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:29.247192  5628 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:29.198598 16940 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:29.246204  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: true dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:29.250216  5633 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:29.250624  5633 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:29.251988  5634 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:29:29.252700  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:29.256645  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:29.175745 16541 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.058s
I20260709 12:29:29.261394 16941 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:29.261904 16953 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:29.263381 16768 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:29.265025 16914 raft_consensus.cc:2764] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election decision vote started in defunct term 28: won
I20260709 12:29:29.268623  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:29.271661  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:29.227979 16683 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:29.209096 16716 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:29.340070  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 65523.7us
I20260709 12:29:29.256493 16910 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:29.257282 16830 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:29.353420  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:29.186794 16841 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.354844  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:29.244300 16766 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.359445  5559 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:29.236613 16859 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.109s
I20260709 12:29:29.260630  5633 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.229544 16866 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.23, Last appended by leader: 23, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.351735 16592 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:29.386961  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:29.385998  5633 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:29.356630 16803 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.032s
I20260709 12:29:29.253180 16942 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.352056 16887 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.034s
I20260709 12:29:29.372020 16953 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:29.416441  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:29.340493 16814 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.385805 16716 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:29.360077 16913 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.073s
I20260709 12:29:29.341253 16940 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.430444 16952 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.431205 16952 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.431618  5676 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:29.432864 16842 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:29.433377 16842 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:29.433530 16952 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:29.433815 16842 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:29.434300 16932 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.159s
I20260709 12:29:29.436295 16956 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.014s
I20260709 12:29:29.396528 16592 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:29.439147 16954 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:29:29.440780 16812 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:29.440891 16887 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:29.441447 16887 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 28 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:29.441820 16667 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:29.440985 16592 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:29.442463 16942 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:29.372620 16683 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.443137 16887 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.402311 16712 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:29.446110 16683 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:29.420269 16927 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.446846 16814 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:29.447184 16927 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.448055 16841 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:29.390961 16768 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:29.449719 16927 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:29.449476 16768 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.451722  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:29.452059  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:29.453097  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:29.453996  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 167611us
I20260709 12:29:29.454538  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.455781  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.455169  5701 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:29.459447  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.459649  5633 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:29.460286  5710 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:29.462081  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:29.464761  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.465448  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:29.357677 16941 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.470016  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:29.472023  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:29.475283  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:29.477682  5633 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:29:29.377926 16910 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.478009  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:29.492549  5560 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 34 because replica has last-logged OpId of term: 32 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 23.
I20260709 12:29:29.495139 16683 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:29.495879 16683 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.471966 16886 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:29.498001 16683 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:29.499989  5630 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:29.500732  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:29.488085 16947 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:29:29.502645  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.445096 16887 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:29.503346  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.503731  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.504022  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:29.432866 16940 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:29.501863  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:29.506129  5706 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 34 because replica has last-logged OpId of term: 32 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 23.
I20260709 12:29:29.508423  5707 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:29.483911 16712 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:29.546420 16934 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:29.548640 16911 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.560724 16911 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.564785 16960 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:29.565701 16960 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:29.566143 16960 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:29.568830 16941 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:29.512200 16886 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:29.570750 16886 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.25, Last appended by leader: 25, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:29.544859 16902 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.501s	user 0.010s	sys 0.000s
I20260709 12:29:29.545225 16875 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:29:29.586040 16830 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:29.606884  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 96778.6us
I20260709 12:29:29.608114  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:29.618367  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:29.549963 16934 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:29.630947  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:29.631537  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:29.593479 16902 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.637470  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:29.639756  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:29.640203  6272 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:29:29.553454 16953 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.621397 16677 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:29.632140 16911 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:29.667160  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: true dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:29.618554 16830 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.673617  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:29.675218 16968 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:29:29.682789 16902 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:29.683108 16959 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:29:29.684165 16910 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:29.688726  6277 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:29.673091 16677 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:29.656702 16872 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.707458  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:29.716128  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.716591  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:29.727335  6275 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:29.717487  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:29.729228  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 227428us
I20260709 12:29:29.708071 16891 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.748585  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:29.766304  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:29.773442 16964 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 12:29:29.775338 16963 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
W20260709 12:29:29.777143 16966 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:29:29.778997  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3775) took 1182 ms. Trace:
I20260709 12:29:29.779425  5552 rpcz_store.cc:276] 0709 12:29:28.596889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:28.625214 (+ 28325us) service_pool.cc:224] Handling call
0709 12:29:28.625484 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:28.712602 (+ 87118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:29.778712 (+1066110us) spinlock_profiling.cc:243] Waited 80.5 ms on lock 0x7b54002283d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:29.778916 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":185196416}
I20260709 12:29:29.780112 16703 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:29.752938 16970 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:29.784559  6277 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:29.734045 16872 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.762701 16875 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.790247 16842 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.797899 16830 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:29.800829  5708 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:29.808166 16934 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:29.809770  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:29.814924  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:29.815898 16820 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:29.755782 16891 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.816341  5601 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:29.871362  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:29.874166  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:29.821622  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:29.868193  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:29.819978  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:29.879172  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 59318us
I20260709 12:29:29.886670  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:29.800045 16842 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:29.888770  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 72490.1us
I20260709 12:29:29.822965 16872 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:29.893857  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.894104  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.787003 16970 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.895176  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.817270 16927 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:29.874096 16278 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.837486 16953 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:29.903143  5704 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:29.893342  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:29.897694 16914 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:29.791342 16875 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:29.950079  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 216825us
I20260709 12:29:29.816485 16820 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:29.897099  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:29.890709 16946 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:29.970767  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:29.971547  6275 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:29.895951 16872 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:29:29.972599  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:29.897696 16956 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:29.891868 16957 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:29.896275 16831 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.080s
I20260709 12:29:29.905965 16887 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:29.951252 16953 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:29.987329 16886 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.002s
I20260709 12:29:29.994760 16951 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:29.995046  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:29.998113 16716 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.534s	user 0.011s	sys 0.001s
I20260709 12:29:29.949018 16278 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:29.998359 16970 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.000149 16987 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:29.986557 16713 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:29.985082 16797 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:29:30.001001 16870 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.034724 16951 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.041388  5703 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
W20260709 12:29:30.042477  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 149527us
W20260709 12:29:30.054373  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 157370us
I20260709 12:29:30.089390 16870 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.094861 16870 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.095553 16278 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:29.985405 16956 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.085120 16841 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.084909 16766 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.115021  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:30.115053  6275 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
W20260709 12:29:30.115187  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 131316us
I20260709 12:29:30.119882 16766 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.120489 16953 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.121860 16988 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:30.122148 16766 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:30.122684  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3849) took 1050 ms. Trace:
I20260709 12:29:30.123078  5705 rpcz_store.cc:276] 0709 12:29:29.071802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:29.082161 (+ 10359us) service_pool.cc:224] Handling call
0709 12:29:29.087718 (+  5557us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:29.426731 (+339013us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:29.426775 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:29.426825 (+    50us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:29.426876 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:30.122472 (+695596us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b54003ad4d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:30.122519 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:30.122632 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":778999040}
I20260709 12:29:30.114934 16992 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:30.124370 16797 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:29:30.124794 16829 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:30.124912 16992 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.122390 16988 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.048956 16887 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.133558 16992 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:30.138197 16960 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.571s	user 0.010s	sys 0.002s
I20260709 12:29:30.146982  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:30.152581  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:30.159091 16960 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.162685 16886 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:29:30.163682 16886 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:29:30.164508 16872 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:30.164909 16956 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:30.159839 16992 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:30.165747 16820 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:29:30.166072 16992 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:30.166472 16992 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:30.166842 16992 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:30.087209 16987 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.171098 16987 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.177620 16887 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.180192 16870 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:30.180845 16870 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:30.181209 16990 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:30.181823 16870 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.182467 16138 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:30.183079 16138 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.183513 16913 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:30.183965 16875 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.185197  6271 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:30.197873  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3784) took 1487 ms. Trace:
I20260709 12:29:30.199422  6273 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:30.185913 16138 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.183450 16870 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:29:30.186559  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:30.186290  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:30.211846  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:30.187158 16842 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:30.181845 16990 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.217439  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:30.160236 16833 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:30.223140  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:30.224085  5628 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:30.224639  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:30.225407  5561 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:30.191462 16992 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.194761 16988 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.118882 16841 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.198486  5627 rpcz_store.cc:276] 0709 12:29:28.710600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:28.710798 (+   198us) service_pool.cc:224] Handling call
0709 12:29:28.714813 (+  4015us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:28.910300 (+195487us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:28.910344 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:28.943772 (+ 33428us) log.cc:844] Serialized 52 byte log entry
0709 12:29:29.168311 (+224539us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:29:29.168387 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:30.197230 (+1028843us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b540018d154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:30.197311 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:30.197391 (+    80us) raft_consensus.cc:1685] finished
0709 12:29:30.197414 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:30.197779 (+   365us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36070,"raft.queue_time_us":119713,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":7279,"spinlock_wait_cycles":387561216,"thread_start_us":133679,"threads_started":1,"wal-append.queue_time_us":230554}
I20260709 12:29:30.210448  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:30.270113  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:30.270933  5553 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:30.213804  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:30.215039  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.215776  5557 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:30.273319  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.273698  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:30.274541  5557 raft_consensus.cc:2410] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 31 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 20.
I20260709 12:29:30.271910  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:30.275851  5553 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 33: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:30.276834 16829 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:30.277761 16829 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:30.282024 16986 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.165s	user 0.002s	sys 0.000s
I20260709 12:29:30.283025  5628 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:30.284507 16841 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.288151  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:30.289889  5627 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
I20260709 12:29:30.290226  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.191402 16925 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:30.221186  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.217530  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.291456  5633 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:30.219244 15538 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:30.215886  5636 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:30.292337 15538 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:30.228518  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.160642 16951 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:30.229574  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.229005 16833 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.204203 16875 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:30.236737  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:30.223573 16842 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.239168 16990 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.246059  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:30.256716  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:30.213878 16755 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.244248 16927 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:30.226531  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:30.152387 16891 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.298723  5555 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
I20260709 12:29:30.313660  5552 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:29:30.257809 16932 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:30.244705 16960 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:30.300798  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 154059us
I20260709 12:29:30.301818  5553 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 29 index: 21. (term mismatch)
W20260709 12:29:30.302874  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 66247.2us
I20260709 12:29:30.323235  5553 pending_rounds.cc:85] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 20
I20260709 12:29:30.323632  5553 pending_rounds.cc:107] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 28.21
I20260709 12:29:30.323942  5553 raft_consensus.cc:2889] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:30.306211  5561 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:30.326628  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:30.306723  5556 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 34: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:30.218164 16913 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.315235  5628 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
I20260709 12:29:30.320976  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.333801  5635 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:30.303911 15538 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.334517  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:30.308907  5629 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 28.
I20260709 12:29:30.335088  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.340278  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:30.339428  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:30.342805  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 190233us
I20260709 12:29:30.313081 16872 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:30.313283 16951 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:30.307271 16842 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.344921 16898 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:30.349676  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:30.329679 16829 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.321557 16683 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:30.360545  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:30.336390 16891 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:30.308398 16875 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.363795  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:30.367420  6277 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:30.307020 16755 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.347741 16914 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:30.342361  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:30.371505  6274 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 36 index: 26. (index mismatch)
I20260709 12:29:30.302945 16925 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.371367  5561 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
W20260709 12:29:30.375804  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:30.380666 16910 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:30.349279 16951 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:30.386684  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:30.364527 16960 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.347692 16911 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.028s
I20260709 12:29:30.402781  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.373759 16914 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.373976 16842 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:30.391273 16951 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:30.422559  6271 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 24. Preceding OpId from leader: term: 32 index: 25. (index mismatch)
I20260709 12:29:30.390267 16891 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:30.423276  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:30.424997  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:30.365257 16683 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.427402  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:30.438489  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51926.4us
I20260709 12:29:30.427716  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:30.428568  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:30.436326 16914 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:30.441876  6276 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 33: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:29:30.431965  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:30.422237  6275 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:30.443568  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:30.396936 16841 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 12:29:30.445111 16991 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:30.447844  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.458629  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:30.440807 16911 consensus_queue.cc:1243] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 34.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:29:30.418053 16840 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.035s
I20260709 12:29:30.490623  5635 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:29:30.475982  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:30.482590  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:30.495184  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:30.495954 16716 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:30.496567  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:30.498692  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:30.498759 16934 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:29:30.500005  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:30.500588 16677 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:30.501641 16677 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:30.502133 16677 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.502614 16677 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:30.504740 16957 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 35.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:29:30.505705 16716 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:30.438547 16891 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:30.506279 16860 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:30.506476 16891 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:30.506744 16860 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:30.507647 16860 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.509063 16860 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:30.414562 16960 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:30.516896  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 73403.6us
W20260709 12:29:30.517511  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 73570.4us
I20260709 12:29:30.447417 17006 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:30.448900 16988 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.519776  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:30.499857 16934 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:30.460039 16683 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.524183  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:30.480899 16956 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.488615 16941 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:30.440594 16842 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.496592 16830 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:30.539503  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:30.521878 16913 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.484772 16541 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.053s
I20260709 12:29:30.532289 16983 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:30.548836  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:30.550765 16924 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:29:30.551863 16997 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
I20260709 12:29:30.552577 16955 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:29:30.518013 16991 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.030s
I20260709 12:29:30.540748 16841 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:30.528276 17006 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.531754 16988 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.554603 16983 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:30.574604 16925 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.542716 16956 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.578574 16956 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:30.571051 17000 log.cc:927] Time spent T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.178s	user 0.000s	sys 0.002s
I20260709 12:29:30.583261 16988 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.572091 16999 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:30.576046 16983 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:30.590399 16999 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:30.596673 17006 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:30.596251 16914 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.597657 16960 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:30.592172 16999 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.598810 16999 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:30.600756 16951 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.602495 16951 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:30.598770 16914 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:30.606137 16983 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.642958  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4170) took 1191 ms. Trace:
I20260709 12:29:30.665536  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:30.576236 16875 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.666862  5553 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:30.668804 16875 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:30.669131 16541 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.670457 16541 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.656327  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:30.657393  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 138350us
W20260709 12:29:30.661464  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 112534us
I20260709 12:29:30.669509 16875 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:30.679075 16875 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:30.654585  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:30.643014  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 123218us
I20260709 12:29:30.680088 16541 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:30.689268  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:30.690557  5555 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:30.662575  5559 rpcz_store.cc:276] 0709 12:29:29.451660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:29.483242 (+ 31582us) service_pool.cc:224] Handling call
0709 12:29:29.497662 (+ 14420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:29.513291 (+ 15629us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:29:29.540816 (+ 27525us) log.cc:844] Serialized 14 byte log entry
0709 12:29:30.127542 (+586726us) spinlock_profiling.cc:243] Waited 8.4 ms on lock 0x7b18004543c0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a 00007f4ffa67ca2a
0709 12:29:30.302594 (+175052us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:30.303111 (+   517us) log.cc:844] Serialized 26 byte log entry
0709 12:29:30.303244 (+   133us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:29:30.303294 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:30.303388 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:30.641581 (+338193us) raft_consensus.cc:1685] finished
0709 12:29:30.641906 (+   325us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:30.642527 (+   621us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20504,"prepare.queue_time_us":27425,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":16237,"raft.queue_time_us":39742,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":11486,"spinlock_wait_cycles":29121152,"thread_start_us":174697,"threads_started":1,"wal-append.queue_time_us":802822}
I20260709 12:29:30.682629 16987 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.680476 16970 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.025s
W20260709 12:29:30.726548  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 46941.5us
I20260709 12:29:30.688042 16766 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:30.698994 16891 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.693670 16983 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.695072 16842 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.731341 16766 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.704241 16924 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:30.745249 16924 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.752084 16843 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:30.752725 16843 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:30.753451  5526 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.2:38587 timed out after 1.929s (SENT) [suppressed 4 similar messages]
I20260709 12:29:30.755856  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.757229 16992 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.758608 16541 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.016s
I20260709 12:29:30.760532  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:30.761662  5705 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:30.739055  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:30.695912 16987 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.741719 16766 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:30.770553 16541 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:30.772280  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:30.773663 16833 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.776065  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:30.792547 16891 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.794333 16842 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:30.802649 16990 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:30.803754 16990 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.805435  5707 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:29:30.808887 16990 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.809080  5560 raft_consensus.cc:2410] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 31 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
I20260709 12:29:30.815441  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:30.815807  5708 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:29:30.816529  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:30.816900  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 80689.3us
I20260709 12:29:30.817312  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 41 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:30.818255  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:30.818365  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:30.819094  6273 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:30.819588  5703 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:30.819761  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:30.820999  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:30.822001 17007 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:30.822677 17007 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.823196 17007 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:30.825379 16955 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:29:30.828958  5627 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:29:30.829388  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:30.829473  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:30.822042  6272 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:30.831163  5558 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:30.831250  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:30.832228  5556 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:29:30.803700  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:30.829180  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:30.855365 16840 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:30.856637 16840 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.857231  6237 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:30.859545  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:30.859813 16840 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.860481  5553 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:30.861126 16941 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:30.869362 16752 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:30.869967 16752 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:30.870513 16752 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:30.873396  5558 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:30.865171  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:30.877411  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 74351.3us
I20260709 12:29:30.882982  6273 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:30.863548 16941 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.430239  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:30.887362 16995 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:30.890367 16941 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:30.890689  5450 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 19 to 25, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 25 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:30.893332 16814 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:30.894450 16995 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:30.896250  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:30.896556 16990 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:30.897149 16990 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:30.897591 16990 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:30.903784  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.904613 16872 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:30.904800 16983 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:30.904845  5555 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:30.905742 16872 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:30.905678 16983 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.907383  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3769) took 2517 ms. Trace:
I20260709 12:29:30.907498  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.907811  5554 rpcz_store.cc:276] 0709 12:29:28.389428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:28.426099 (+ 36671us) service_pool.cc:224] Handling call
0709 12:29:28.429897 (+  3798us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:28.939722 (+509825us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:28.939760 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:28.939806 (+    46us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:29:28.939839 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:30.906889 (+1967050us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b5400025dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:30.906975 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:30.907181 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1171811200}
I20260709 12:29:30.908587 16983 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.884599 16956 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.912626  5635 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:29:30.904810  5627 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
W20260709 12:29:30.921314  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:30.909885 16956 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.933220 16665 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:30.942724 16812 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:30.943488 16812 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:30.946302 17023 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.090s	user 0.002s	sys 0.002s
I20260709 12:29:30.947086 16812 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.949958 16843 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.23, Last appended by leader: 23, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.950714 16987 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.951491 16843 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:30.957708  5629 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 3961) took 2613 ms (client timeout 1929 ms). Trace:
I20260709 12:29:30.958091 16956 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:30.958379  5629 rpcz_store.cc:269] 0709 12:29:28.344426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:28.344865 (+   439us) service_pool.cc:224] Handling call
0709 12:29:30.957214 (+2612349us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b5400438d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:30.957428 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4410641152}
I20260709 12:29:30.975647 16755 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.977538  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:30.947938 16872 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.989475 16872 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:30.990594  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.993201  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.994225  5628 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:30.994521  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.995572  5629 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
W20260709 12:29:30.996554  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:30.999296  5559 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 16. Preceding OpId from leader: term: 33 index: 17. (index mismatch)
I20260709 12:29:30.977752 16995 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:30.979056 16705 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:31.000665 16705 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:30.980896 16924 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:30.982199  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:30.983762  5555 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:30.985338  5554 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:29:31.003875 16705 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:31.006268  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:31.007100  6274 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:31.014861  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:31.019587  5554 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34. Ops: [33.23-33.23]
I20260709 12:29:30.995990  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:31.016439  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:31.036216  5628 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:31.037568  5554 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:31.042001  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:31.002118 16924 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:31.017472  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:31.019737  5559 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:31.020499 16705 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:31.015904  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:31.053620  5560 raft_consensus.cc:2410] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 31 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
I20260709 12:29:31.053666  5558 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:31.022632 17007 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:31.026218  5555 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 38 index: 22. (term mismatch)
I20260709 12:29:31.025982  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:31.059604  5555 pending_rounds.cc:85] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 21
I20260709 12:29:31.029436  5450 catalog_manager.cc:5697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:31.032593  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:31.044849  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:31.057971  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:31.061709  6274 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:31.059937  5555 pending_rounds.cc:107] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 36.22
I20260709 12:29:31.065999  5555 raft_consensus.cc:2889] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:30.947870 16752 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:30.976118 16941 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:31.040051 16990 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:31.081229  6272 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:31.081615  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:31.082661  5558 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:31.087762 16705 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:31.092950  5450 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 14 to 23, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 23 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:31.095023 16705 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:31.097038  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 186319us
W20260709 12:29:31.097854  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 221336us
I20260709 12:29:31.098973  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:31.099152  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:31.100715 16858 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:31.101027 16955 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:29:31.101835  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 49622.2us
I20260709 12:29:31.103796 16898 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:31.096903  5628 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 36 index: 26. (index mismatch)
I20260709 12:29:31.104589 16898 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:31.105631  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:31.106081  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:31.105712 16898 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 32.24, Last appended by leader: 24, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:31.106930  5558 raft_consensus.cc:2393] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:31.107913  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:31.108783  5554 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:31.109880 16752 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:31.110121  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:31.110991 16868 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:31.111446 16868 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:31.112318  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:31.113677  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:31.115477 16703 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:31.111524  5556 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:31.064553 16906 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:31.116142 16703 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:31.116842 16906 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:31.117247 16906 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:31.119191  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 41 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:31.111847 16868 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:31.122397 16924 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:31.138919  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:31.155557  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:31.159325  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:31.163017  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:31.157651  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:31.165447  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:31.156951  5450 catalog_manager.cc:5697] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 19 to 26, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 26 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:31.168572  5558 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 29 index: 15. (index mismatch)
I20260709 12:29:31.148602 16703 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:31.181496  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:31.170770 16872 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:31.246606  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:31.243523  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 64649.5us
I20260709 12:29:31.247551  5558 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 29: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:29:31.242841  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 106897us
I20260709 12:29:31.257414  5555 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:31.257807  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:31.257382 16833 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:29:31.259853  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 120633us
I20260709 12:29:31.264472 16861 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:31.264678 16883 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:31.265568 15538 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:29:31.265450 16861 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:31.251482 16703 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:31.247081  5450 catalog_manager.cc:5697] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:31.272823  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:31.255426 16875 consensus_meta.cc:287] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.576s	user 0.015s	sys 0.004s
I20260709 12:29:31.252087 16872 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:31.280822  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:31.289037  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:31.298038  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 75559us
I20260709 12:29:31.296867  5560 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:31.282163 16703 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:31.300889  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:31.313088  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:31.318253 16677 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.755s	user 0.010s	sys 0.014s
I20260709 12:29:31.259233 16924 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:29:31.325649  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 176195us
I20260709 12:29:31.329578  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:31.328172  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:31.341969  5703 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 29: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:31.341877  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:31.342120 16953 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:31.316016 16703 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:31.287851 16875 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:31.280143 16705 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.045s
I20260709 12:29:31.351395  6277 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: [32.25-32.25]
I20260709 12:29:31.341995  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:31.350484  5708 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:31.351699 16953 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:31.359591 16953 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:31.359791  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:31.360064  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:31.360520  5704 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:29:31.362403 16949 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:31.362980 16842 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:31.363436 16842 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:31.363931  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:31.364311  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:31.365779  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:31.334957 16941 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:31.373217  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:31.374902  5629 raft_consensus.cc:2368] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30.
I20260709 12:29:31.318799 16677 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:31.385456  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:31.380245  5708 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:31.363940 16842 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:31.385350  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:31.386236  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:31.363169 16875 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:31.385113 16138 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.034s
I20260709 12:29:31.354650 16990 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:31.373545 16755 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:31.398015  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3786) took 1131 ms. Trace:
I20260709 12:29:31.400794  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:31.401012  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:31.401636  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:31.403416 17025 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.266s	user 0.000s	sys 0.002s
I20260709 12:29:31.403764 16278 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.097s
W20260709 12:29:31.405115 17026 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
W20260709 12:29:31.406036 17028 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.153s	user 0.000s	sys 0.000s
W20260709 12:29:31.406548  5454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35600 (request call id 282) took 33799 ms (client timeout 15000 ms). Trace:
I20260709 12:29:31.396764  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:31.442169  5702 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 35 because replica has last-logged OpId of term: 33 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 22.
I20260709 12:29:31.445863  5635 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:31.357774 17007 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:31.435369  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:31.438999  5555 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:29:31.434016  5632 rpcz_store.cc:276] 0709 12:29:30.212387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:30.212952 (+   565us) service_pool.cc:224] Handling call
0709 12:29:30.228727 (+ 15775us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:30.229042 (+   315us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:29:30.829517 (+600475us) log.cc:844] Serialized 14 byte log entry
0709 12:29:31.258494 (+428977us) log.cc:844] Serialized 14 byte log entry
0709 12:29:31.277691 (+ 19197us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:31.287166 (+  9475us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:31.297679 (+ 10513us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:31.310944 (+ 13265us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:31.343386 (+ 32442us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15324,"prepare.queue_time_us":373240,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":119,"raft.queue_time_us":35218,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":9926,"spinlock_wait_cycles":99456,"thread_start_us":783959,"threads_started":2,"wal-append.queue_time_us":499362}
I20260709 12:29:31.463042  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:31.384841 16924 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.068s
I20260709 12:29:31.468508  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:31.450219 16990 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:31.469235  5629 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 27.
I20260709 12:29:31.468122  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:31.481182 17027 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.191s	user 0.000s	sys 0.001s
I20260709 12:29:31.489163  6276 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:31.494290  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:31.494611  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:31.495083  5633 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:31.496210  5629 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:29:31.497061  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:31.499018  5710 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:31.498548  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:31.430660 16872 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 32.25, Last appended by leader: 25, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:31.529224  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:31.531534  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:31.534951  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:31.535808  5629 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:31.530145  5627 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:31.510320 16833 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:31.560600 16833 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:31.545215 16883 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:31.546027  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:31.546288 16927 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:31.550565 16138 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:29:31.576223  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44807.8us
I20260709 12:29:31.590330  6275 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:31.594235  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:31.573594 16927 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:31.573767 16705 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:31.606029  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:31.607894  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:31.610554  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:31.468581 16755 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:31.577553 16138 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:31.618454 17029 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 12:29:31.521857 16278 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.24, Next index: 25, Last known committed idx: 22, Time since last communication: 0.071s
I20260709 12:29:31.569093 16883 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:31.522428 16990 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:31.653189  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:31.629895 16138 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:29:31.667147 17031 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.233s	user 0.004s	sys 0.000s
I20260709 12:29:31.599676 16927 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:31.622123 16861 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:31.705606  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:31.717696  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:31.721544  5553 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:31.727746  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:31.726877 16861 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:31.728716  5561 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:31.728749 16861 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:31.719496 16766 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.873s	user 0.013s	sys 0.003s
I20260709 12:29:31.727128  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:31.753398  5552 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:31.730885 16861 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:31.749598  5629 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:29:31.812235  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:31.726881 16866 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.103s
I20260709 12:29:31.822798 16677 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:31.838856 16946 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:31.777195 16766 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:31.441185  5454 rpcz_store.cc:269] 0709 12:28:57.566648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:07.467377 (+9900729us) service_pool.cc:224] Handling call
0709 12:29:31.354817 (+23887440us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:29:22.245010 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:29:22.548749 (+303739us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:29:22.548799 (+    50us) write_op.cc:435] Acquired schema lock
0709 12:29:22.548822 (+    23us) tablet.cc:662] Decoding operations
0709 12:29:22.550606 (+  1784us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:29:22.550719 (+   113us) write_op.cc:641] Partition lock acquired for write op
0709 12:29:22.550747 (+    28us) tablet.cc:685] Acquiring locks for 22 operations
0709 12:29:22.583847 (+ 33100us) tablet.cc:701] Row locks acquired
0709 12:29:22.600759 (+ 16912us) write_op.cc:260] PREPARE: finished
0709 12:29:22.616667 (+ 15908us) write_op.cc:270] Start()
0709 12:29:22.631950 (+ 15283us) write_op.cc:276] Timestamp: P: 1783600162607577 usec, L: 0
0709 12:29:22.643033 (+ 11083us) op_driver.cc:348] REPLICATION: starting
0709 12:29:22.797703 (+154670us) log.cc:844] Serialized 8448 byte log entry
0709 12:29:23.304833 (+507130us) op_driver.cc:464] REPLICATION: finished
0709 12:29:23.674325 (+369492us) write_op.cc:301] APPLY: starting
0709 12:29:23.698909 (+ 24584us) tablet.cc:1367] starting BulkCheckPresence
0709 12:29:24.096717 (+397808us) tablet.cc:1370] finished BulkCheckPresence
0709 12:29:24.096861 (+   144us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:29:24.795696 (+698835us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:29:24.796145 (+   449us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=44,key_file_lookups=44,delta_file_lookups=0,mrs_lookups=0
0709 12:29:24.796215 (+    70us) write_op.cc:312] APPLY: finished
0709 12:29:24.796806 (+   591us) log.cc:844] Serialized 196 byte log entry
0709 12:29:25.493347 (+696541us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:29:25.501876 (+  8529us) write_op.cc:454] Released schema lock
0709 12:29:25.502001 (+   125us) write_op.cc:341] FINISH: Updating metrics
0709 12:29:25.513153 (+ 11152us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:29:25.513203 (+    50us) write_op.cc:454] Released schema lock
0709 12:29:25.990622 (+477419us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
Metrics: {"mutex_wait_us":8831257,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":355093,"apply.run_cpu_time_us":121340,"apply.run_wall_time_us":1828252,"cfile_cache_hit":90,"cfile_cache_hit_bytes":96087,"num_ops":22,"prepare.queue_time_us":208800,"prepare.run_cpu_time_us":18686,"prepare.run_wall_time_us":708427,"raft.queue_time_us":173602,"raft.run_cpu_time_us":51605,"raft.run_wall_time_us":371646,"replication_time_us":614125,"spinlock_wait_cycles":54899968,"thread_start_us":1094763,"threads_started":5,"wal-append.queue_time_us":950596,"wal-append.run_cpu_time_us":23113,"wal-append.run_wall_time_us":2510529}]]}
I20260709 12:29:31.874214  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:31.900426  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.001s	sys 0.027s
W20260709 12:29:32.006596  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.005s	sys 0.042s
W20260709 12:29:31.927361  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
W20260709 12:29:32.007272  5676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.006s	sys 0.015s
W20260709 12:29:32.007647  5676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.006s	sys 0.015s
W20260709 12:29:31.948822  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50247.9us
I20260709 12:29:31.969082 16990 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:32.009132  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 489057us
W20260709 12:29:31.916973 16868 consensus_meta.cc:287] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.779s	user 0.005s	sys 0.010s
I20260709 12:29:31.944505 16278 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:32.007987  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 645294us
W20260709 12:29:31.899605 16949 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.000s	sys 0.019s
I20260709 12:29:32.030701  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:32.015718 16840 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.006s	sys 0.013s
I20260709 12:29:32.016175  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:32.041229  5633 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.503s	user 0.010s	sys 0.003s
I20260709 12:29:31.835093 16883 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:31.997915 16755 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:32.041728  5633 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:32.049448  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:32.032989 16949 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.001s	sys 0.019s
W20260709 12:29:32.040177 16840 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.007s	sys 0.014s
I20260709 12:29:32.022720 16875 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:32.045650 16906 consensus_meta.cc:287] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.928s	user 0.017s	sys 0.000s
I20260709 12:29:32.070914  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:32.071807 16875 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.057788 16949 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:32.099375  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:32.055086 16752 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:29:32.121348  5635 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.663s	user 0.010s	sys 0.003s
I20260709 12:29:32.121881  5635 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:32.023787 16868 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.075249 16906 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.132664  5554 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 33: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:32.133181 16987 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:32.133708 16906 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:32.133837 16987 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.135521  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:32.136471  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 69705.8us
I20260709 12:29:31.899078 16946 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.139496 16946 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:32.141777 16953 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.782s	user 0.013s	sys 0.000s
I20260709 12:29:32.142272 16953 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.143141 16924 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:32.148746  5454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 279) took 21201 ms (client timeout 15000 ms). Trace:
W20260709 12:29:32.149170  5454 rpcz_store.cc:269] 0709 12:29:10.947513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:32.148493 (+21200980us) service_pool.cc:211] Skipping call since client already timed out
0709 12:29:32.148658 (+   165us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:29:32.150208 16703 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.761s	user 0.012s	sys 0.001s
I20260709 12:29:32.150799 16703 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.069007 16956 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.059s
I20260709 12:29:32.152879 16956 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:32.153403 16956 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.154569 16868 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.155380 16956 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:32.155987 16956 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:32.093981 16278 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.133870 16940 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.118s
I20260709 12:29:32.157927 16940 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:32.158424 16278 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.158821 16995 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:32.159298 16766 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:32.159476 16995 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.134402  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 2252) took 1004 ms. Trace:
I20260709 12:29:32.162824  5554 rpcz_store.cc:276] 0709 12:29:31.130205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:31.130366 (+   161us) service_pool.cc:224] Handling call
0709 12:29:32.134179 (+1003813us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b54002279d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:32.134324 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2282669824}
W20260709 12:29:32.148088 16833 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.587s	user 0.011s	sys 0.000s
W20260709 12:29:32.164047  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 67430.2us
I20260709 12:29:32.176416  5553 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
I20260709 12:29:32.175395 16138 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.177779  5636 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: []
W20260709 12:29:32.157961 16842 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.749s	user 0.006s	sys 0.008s
I20260709 12:29:32.143859 16924 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.158447 16940 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:32.183969  5559 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.011s	sys 0.033s
W20260709 12:29:32.189395 16898 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.828s	user 0.006s	sys 0.030s
W20260709 12:29:32.192646 16843 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.013s	sys 0.057s
I20260709 12:29:32.193917 16138 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:32.197942  5559 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.012s	sys 0.033s
I20260709 12:29:32.185232 16953 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:32.219322  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3787) took 1114 ms. Trace:
I20260709 12:29:32.226203  5559 rpcz_store.cc:276] 0709 12:29:31.105371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:31.107418 (+  2047us) service_pool.cc:224] Handling call
0709 12:29:31.107794 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:31.108117 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:29:31.435852 (+327735us) log.cc:844] Serialized 14 byte log entry
0709 12:29:32.218767 (+782915us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:32.218845 (+    78us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:32.218927 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:32.219032 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:32.219235 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":165974,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":11013,"thread_start_us":1066086,"threads_started":2}
W20260709 12:29:32.206910 16898 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.006s	sys 0.031s
W20260709 12:29:32.212294 16843 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.013s	sys 0.057s
I20260709 12:29:32.177208 16833 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:32.163852 16941 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.029s	sys 0.011s
W20260709 12:29:32.240691  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 59641.1us
I20260709 12:29:32.242496 16927 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:32.243160 16995 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:32.231037 16875 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:32.199637 16842 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.247843 16949 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.272522 17037 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:32.273137 17037 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.273703 16812 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:32.274185 16812 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:32.275395 17037 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:32.275058 16812 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.276561  6236 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:32.277602 16677 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:32.277796 16842 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:32.235641 17038 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:32.278177 16677 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:32.278445 17038 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:32.279136  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4184) took 2044 ms. Trace:
I20260709 12:29:32.279778 17038 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.281095 17038 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:29:32.262774 16941 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.772s	user 0.030s	sys 0.011s
W20260709 12:29:32.287351  5552 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.505s	user 0.006s	sys 0.004s
I20260709 12:29:32.289700  5552 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:32.278065  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:32.276141 16812 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:29:32.288475  5710 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.717s	user 0.008s	sys 0.004s
I20260709 12:29:32.279965  5557 rpcz_store.cc:276] 0709 12:29:30.234584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:30.295752 (+ 61168us) service_pool.cc:224] Handling call
0709 12:29:30.800843 (+505091us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b5800002d98. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff64bfc0 00007f4fff636929 00007f4fff6363d2 00007f4ff8badbd4
0709 12:29:30.801399 (+   556us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:30.837480 (+ 36081us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:30.837518 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:30.838776 (+  1258us) log.cc:844] Serialized 26 byte log entry
0709 12:29:31.067929 (+229153us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:29:31.067989 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:31.068066 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:32.271675 (+1203609us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b240013f4a8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff740b9c4 00007f4ff7409acb 00007f4ff75c4dc9 00007f4ff75c4a55 00007f4ff75c3fb0 00007f4ffa23a488 00007f4ffa23abe8 00007f4ffa212398 00007f4ffa20d76a 00007f4ffa20cbb6
0709 12:29:32.278641 (+  6966us) raft_consensus.cc:1685] finished
0709 12:29:32.278664 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:32.278801 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":487,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":157,"spinlock_wait_cycles":45926016,"thread_start_us":228693,"threads_started":1,"wal-append.queue_time_us":292314}
I20260709 12:29:32.292343  5710 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
W20260709 12:29:32.299333  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:32.294793 16988 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.014s	sys 0.029s
I20260709 12:29:32.312549 16897 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:29:32.307096 16988 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.015s	sys 0.030s
I20260709 12:29:32.292006 16940 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.325381  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:32.335896  6273 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
W20260709 12:29:32.324636  5602 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.013s	sys 0.014s
W20260709 12:29:32.337253  5602 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.013s	sys 0.014s
I20260709 12:29:32.306696 17046 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:32.279978 16677 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.325632  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:32.342790  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 64812.3us
I20260709 12:29:32.335608 16995 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:32.356213 17046 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.360419 16924 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:32.420567  5631 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.007s	sys 0.025s
I20260709 12:29:32.352905 16946 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:32.432695 16755 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:32.448828  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.450314  5705 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:32.452066  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:32.455137  5703 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:32.453157  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.462858  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:32.463825  5709 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:32.475625  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 289322us
W20260709 12:29:32.455756  5450 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.005s	sys 0.006s
W20260709 12:29:32.476739  5450 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.005s	sys 0.006s
I20260709 12:29:32.480195  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:32.480988  5704 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:32.483680  5709 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:32.473937  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:32.485347  5701 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:32.488689  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 51226us
W20260709 12:29:32.493117  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 347761us
I20260709 12:29:32.444336 16995 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.497615  5709 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34. Ops: []
W20260709 12:29:32.495352  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 357686us
W20260709 12:29:32.488451 16861 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.678s	user 0.012s	sys 0.002s
I20260709 12:29:32.493774  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:32.511509 16138 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:32.511435  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 440625us
I20260709 12:29:32.491811 16898 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:32.493713 16875 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:32.521039  5552 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:32.513679  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:32.497323 16897 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:32.524478 16138 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.522055  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:32.500695 17038 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:32.491825 17007 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:29:32.505353 16705 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:32.505177 16988 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:32.485212 16755 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.535840  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:32.536918  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:32.516727 16906 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:32.563949 16138 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.492213 16946 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.532394  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:32.535180 16875 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.538911 16898 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:29:32.546061 17036 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 12:29:32.547139  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62204.1us
I20260709 12:29:32.547573 16897 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:32.548666  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:32.548501 16988 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.549011 17037 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:32.550316 16860 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:32.584625 16752 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:32.566931  5559 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 33: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:32.568315 16866 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:32.570289 16883 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:32.571336 16703 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.571570 16812 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 34.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:29:32.612653 16755 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.613194 16812 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:32.613770 16812 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:32.582690 16906 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.614672 16812 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 34.15, Last appended by leader: 15, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.616004 16812 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:32.572021 16987 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:32.572458 17042 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.249s	user 0.000s	sys 0.001s
I20260709 12:29:32.616444 16906 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:32.574239  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3817) took 1035 ms. Trace:
W20260709 12:29:32.616997 16927 consensus_queue.cc:1175] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cea66aa15be3423880843974acdbf50f" responder_term: 34 status { last_received { term: 34 index: 15 } last_committed_idx: 15 last_received_current_leader { term: 34 index: 15 } }
I20260709 12:29:32.574110 16833 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.617244  5634 rpcz_store.cc:276] 0709 12:29:31.539179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:31.790627 (+251448us) service_pool.cc:224] Handling call
0709 12:29:31.836611 (+ 45984us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:31.949355 (+112744us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:31.949407 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:31.950101 (+   694us) log.cc:844] Serialized 26 byte log entry
0709 12:29:32.305018 (+354917us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:29:32.305064 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:32.305126 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:32.574001 (+268875us) raft_consensus.cc:1685] finished
0709 12:29:32.574025 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:32.574170 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16262,"raft.queue_time_us":45953,"raft.run_cpu_time_us":50,"raft.run_wall_time_us":51,"thread_start_us":354263,"threads_started":1,"wal-append.queue_time_us":533439}
I20260709 12:29:32.573565  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:32.613775 17040 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.168s	user 0.000s	sys 0.001s
W20260709 12:29:32.619493  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45996.2us
I20260709 12:29:32.529237 17051 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:32.621871 16138 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:32.566159  5701 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:32.525285 16843 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.056s
I20260709 12:29:32.548053 16705 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.478008  5636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:32.598206 17039 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:32.601382  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.578836 16840 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:32.604570  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:32.645320  5553 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 34: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:32.583504 17057 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:32.613281 16755 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:32.561841 17038 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.662981 17038 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:32.656992 16840 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.631207 16752 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.665733 16840 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:32.666769 16898 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:32.667518 16752 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.667379 17057 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.669028  5709 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 34 index: 23. (term mismatch)
I20260709 12:29:32.669389  5709 pending_rounds.cc:85] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 22
I20260709 12:29:32.669622  5709 pending_rounds.cc:107] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 33.23
I20260709 12:29:32.669847  5709 raft_consensus.cc:2889] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:32.539084 16872 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:32.670696 16872 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.672724 16872 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:32.508404 16861 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.629060  5636 raft_consensus.cc:606] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:32.675390  5636 raft_consensus.cc:614] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:29:32.600629  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 67577.9us
I20260709 12:29:32.630774 16138 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:29:32.641695  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44923.9us
I20260709 12:29:32.647256  5702 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 35: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:29:32.659404  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52477.1us
I20260709 12:29:32.649930  5599 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:32.660933  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.673102  5703 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:32.679625  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.680184  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:32.680771  5553 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:32.676122  5704 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:32.630697 16866 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:32.688784  5707 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 27. Current term is 28. Ops: []
I20260709 12:29:32.687180  5599 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:32.696022  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:32.697726  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 204187us
I20260709 12:29:32.645684 17050 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 33.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.048s
I20260709 12:29:32.697991 16138 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.707945 17053 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:29:32.708218 17058 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:29:32.644474 17051 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.708603 17053 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:32.709451 17045 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:32.710008 16788 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:32.710788 17051 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.710630 16788 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.688889 16677 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:32.648975 17037 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.716346  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.724759  5702 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:32.727705  5553 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
W20260709 12:29:32.728986  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 84261.8us
I20260709 12:29:32.702487 17052 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:32.732797  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.726745  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:32.713034 16833 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:29:32.702967 17046 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.712942 17068 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:32.712710 16677 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:32.742911  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.745412  5560 raft_consensus.cc:2393] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 36: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:32.709869 17045 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:32.658159 17039 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.754240  5704 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:32.762522  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.763155  5704 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:32.743693 16995 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.744107 17052 raft_consensus.cc:697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:32.781205  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:32.779690  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:32.709841 17053 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.765909  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.783562  5703 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 35: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:32.760171 16705 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.758452 16833 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:32.782145  5706 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:32.791188 17070 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:32.791843 17070 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:32.792834 16946 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.794734 17039 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:32.795851 17042 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.179s	user 0.000s	sys 0.002s
I20260709 12:29:32.797984 16988 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:32.799358  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.800510  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.801712  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.802047 16861 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.803074  5631 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:32.802837 17052 consensus_queue.cc:237] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.803794  5631 raft_consensus.cc:606] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:32.804147  5631 raft_consensus.cc:614] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:32.789737 16875 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.807757  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.810750  5710 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:32.762280 16866 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.819099  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.757416 17074 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.056s
I20260709 12:29:32.819773 16812 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:32.820389 17051 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:32.893146 17051 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:32.821115  5561 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:29:32.821193  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:32.821488  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:32.895215  5554 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:32.757225 17068 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:32.898263  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 157330us
I20260709 12:29:32.821215  5702 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:29:32.820432  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:32.900928  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 255998us
I20260709 12:29:32.900996  5553 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:32.819621 17057 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:32.825403  5705 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:32.863837  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:32.819154  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:32.866838  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:32.913545  5710 raft_consensus.cc:2368] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30.
I20260709 12:29:32.913681  5555 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:32.868641  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.915076  5709 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
W20260709 12:29:32.868755  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 111217us
I20260709 12:29:32.917249  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:32.869728 17081 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:29:32.788389 16788 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:32.872876 17037 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:32.919396  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.865386 17039 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 32.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.018s
I20260709 12:29:32.889807  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:32.890480  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:29:32.890669  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:32.891276  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:32.894127  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:32.819114  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:32.899195  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:32.879518  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:32.958420  6277 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 36.
I20260709 12:29:32.924873  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:32.959290  5710 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:29:32.957216  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44985.2us
I20260709 12:29:32.962184  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.964922  5706 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:32.957121  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:32.966877  6279 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 34: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:29:32.967482  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 77805.8us
W20260709 12:29:32.967851  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 148826us
I20260709 12:29:32.966046  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.969508  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.972265  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:32.972481  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:32.972896  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61170.3us
I20260709 12:29:32.974408  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:32.972081 17057 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:32.975328  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:32.975641 17057 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:32.975939  5704 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:32.976099 17057 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:32.981572 17079 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
W20260709 12:29:32.983601  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 66526.5us
I20260709 12:29:32.985895  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:32.986865 17074 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:32.987584 17074 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.989527  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:32.920495 17086 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:32.993402  5561 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 32: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:33.001543  5710 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
W20260709 12:29:33.005679  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:33.006059 16138 raft_consensus.cc:2764] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:29:33.006579  5704 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:33.008386  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:29:33.010419  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:32.962256 16897 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:33.013103  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:32.955403 17037 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:33.024680  5559 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:29:33.018361  5553 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:32.894263 17051 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.17, Last appended by leader: 17, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:32.988739 16924 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:32.969631 17073 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:33.039472  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:33.026165 16843 raft_consensus.cc:2764] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:29:33.038085 16842 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.003s
I20260709 12:29:33.041674  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:33.026592 16897 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.045312 16677 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.044075 16897 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:33.046815 16677 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:29:33.046926 17054 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.496s	user 0.006s	sys 0.003s
I20260709 12:29:33.049639  5701 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:33.052670  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:32.903121 16833 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:32.972265 17039 consensus_queue.cc:1243] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer cea66aa15be3423880843974acdbf50f log is divergent from this leader: its last log entry 33.26 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
W20260709 12:29:33.056228  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:33.017781 17070 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.24, Last appended by leader: 24, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.057475 17072 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:33.058420  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:33.058780  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:33.036212 17038 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:33.036918 17037 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:33.049299 16941 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.012s
I20260709 12:29:33.062098 17090 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:33.064579 16924 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:33.015264 17086 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.069103  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:33.071753 17051 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:33.075547  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 86099.8us
I20260709 12:29:33.075794  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:33.076334 16138 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:33.128968 16138 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:33.129364 16138 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:33.077178  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.021872 17050 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:33.026374 16861 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:33.049154 17053 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:33.095150  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:33.097436  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:33.105109  5560 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:29:33.094758  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:33.108115  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 122331us
W20260709 12:29:33.109369  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 119501us
I20260709 12:29:33.125212  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:33.125650  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:33.125950  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:33.126747  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:33.119246  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:33.130118  5553 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:33.079373 16868 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:33.107440 16953 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:33.085191 17038 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:33.113720 16875 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:33.145021  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:33.078208 17073 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.144019 17053 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:33.179497 17086 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:33.153685 16843 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:33.113411 17072 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:33.197194  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 156997us
I20260709 12:29:33.191743  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:33.159569 16875 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:33.205130 16875 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:33.206470 17074 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:33.208557 16924 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 34.15, Last appended by leader: 15, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:33.209951 17072 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:33.211979  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:33.213339  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:33.214504  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:33.216534 17053 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:33.218835 17037 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:33.219797 17045 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 32.25, Last appended by leader: 25, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.221625 17045 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:33.158049 17050 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:33.146919 16861 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.196534 16988 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.029s
I20260709 12:29:33.226310 16861 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:33.198900  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53980.8us
I20260709 12:29:33.201577 16868 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:33.193502 16843 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:33.204548 16705 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:33.181448 17069 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:33.125982 17090 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:33.234767  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 269990us
I20260709 12:29:33.249328  5524 leader_election.cc:400] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30.
I20260709 12:29:33.249187  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:33.249712  5524 leader_election.cc:403] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Cancelling election due to peer responding with higher term
W20260709 12:29:33.252259  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 102326us
I20260709 12:29:33.256533  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:33.160130 17038 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:33.238493 16703 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:33.239398 16843 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:33.237923 16138 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:33.245167 16705 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.240165 16866 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:33.238219 17052 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:33.277230  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:33.245959  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 99320.2us
I20260709 12:29:33.276041  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:33.289779 16138 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:33.251753 16868 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:33.288642  5676 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:33.290640  5634 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
W20260709 12:29:33.302906  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 56773us
I20260709 12:29:33.298029  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:33.304387  5559 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
W20260709 12:29:33.305286  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 90873.8us
W20260709 12:29:33.306916  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 200336us
I20260709 12:29:33.308182  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:33.277974 17090 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:33.237658 17077 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:33.260547 16872 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election lost for term 29. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30.
I20260709 12:29:33.309479  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:33.312335  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.293471 16995 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:33.314509  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:33.317361  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:33.251698 17069 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.322412  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.277017 17050 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:33.324838  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.325616  5561 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 21. (term mismatch)
I20260709 12:29:33.330871  5561 pending_rounds.cc:85] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 20
I20260709 12:29:33.331240  5561 pending_rounds.cc:107] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 30.21
I20260709 12:29:33.295678 16988 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:33.331547  5561 raft_consensus.cc:2889] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:33.324852  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.326421  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.332481  5555 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
I20260709 12:29:33.289848 17037 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.040s
I20260709 12:29:33.305088 16953 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:29:33.367141  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 86267.6us
W20260709 12:29:33.405575  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:33.324543 17074 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:33.333333 16995 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:33.318342 17072 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:33.321816 17077 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:33.420478  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:33.422480  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 85872.9us
I20260709 12:29:33.334836 16988 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:33.360222 17091 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:33.433641 17091 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:33.434178 17091 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:33.313863 16755 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.109s
I20260709 12:29:33.365239 16883 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:29:33.435377  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.438086  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:33.438866  5553 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:33.324678 16875 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:33.443610  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 129232us
I20260709 12:29:33.422407 17090 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:33.445643  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 128088us
I20260709 12:29:33.428134 16953 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:33.431027 16995 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:33.429309 17072 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:33.435527 16988 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:33.435943 17077 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:33.468834  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 112032us
I20260709 12:29:33.419207 17105 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.093s
W20260709 12:29:33.501219  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:33.511750  5630 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:29:33.467875 17072 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:33.438279 16924 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.037s
I20260709 12:29:33.475172  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:33.507565 17057 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.531s	user 0.013s	sys 0.000s
W20260709 12:29:33.523711  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72487.7us
W20260709 12:29:33.528926  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 95651.8us
I20260709 12:29:33.529444  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:33.529304  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:33.439841 16868 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:33.515540 17105 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:33.533912  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58880.8us
I20260709 12:29:33.553654  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:33.568938 17118 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.018s
I20260709 12:29:33.570082 17116 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:33.584738 16705 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:33.584564 16953 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:33.571015 16875 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:33.573144 17107 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:29:33.573607 17068 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:33.574005 17100 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
I20260709 12:29:33.464071 17090 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.577602  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:33.533854 17099 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.274s	user 0.000s	sys 0.001s
I20260709 12:29:33.595136 16705 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:33.553999 16703 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:33.613131 17069 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:33.600319  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:33.603614  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:33.588369 16677 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.541s	user 0.011s	sys 0.005s
I20260709 12:29:33.618222  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:33.621415  5526 leader_election.cc:400] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 29. Current term is 30.
I20260709 12:29:33.626012 16703 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:33.626694 16703 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:33.627179 16703 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:33.632961  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.633750  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:33.533682 17057 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:33.646553  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:33.648927  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:33.652637 16875 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:29:33.655589  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:33.659114  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:33.590674 17105 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.633152 17050 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:33.663892  5636 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 33: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:33.534579 17111 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.118s
W20260709 12:29:33.666148  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 149547us
I20260709 12:29:33.528133 17052 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.114s
I20260709 12:29:33.635679 16755 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:33.672632  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:33.672887  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:33.653472 16875 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:33.677284  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:33.685038  5628 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:33.684101  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:33.679461  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:33.653800 16924 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:33.692320  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:33.700567  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:33.701225  5635 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:33.638769 17077 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:33.710163  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:33.711097  5628 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:33.712139  5636 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:29:33.717686  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:33.718914  5635 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:33.697887 17120 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:33.725806  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:33.730958  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:33.743656  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.749781  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:33.753921  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.706743 17068 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.036s
I20260709 12:29:33.762784  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:33.765877  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 120495us
I20260709 12:29:33.747347 17120 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:33.706951 16924 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.774931  5635 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
W20260709 12:29:33.755307 16833 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.011s	sys 0.007s
I20260709 12:29:33.775825  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:33.740533 16995 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:33.778421  5636 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:33.836503  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:33.840885  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:33.837653  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:33.841813  5628 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:33.837754  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:33.838827  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 72000.5us
W20260709 12:29:33.822212 16833 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.749s	user 0.013s	sys 0.007s
I20260709 12:29:33.847220  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:33.849054  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:33.844724 17045 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.621s	user 0.010s	sys 0.005s
I20260709 12:29:33.861915  5560 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 35: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:33.866291 16924 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:33.869894  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:33.870458  5632 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:33.871740  5635 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:33.873185 17090 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:33.874212 17120 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:33.835667 17038 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:33.876750 17038 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:33.850203 16897 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:33.877647 16897 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.879796 16897 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:33.883893  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.885818  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.886785 17057 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:33.887109  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.889269  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:33.890002  5557 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 34: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:29:33.890529 16843 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.596s	user 0.011s	sys 0.000s
I20260709 12:29:33.891079  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.890976 16843 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.891285 17072 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.891755  5557 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:33.892722  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.893702  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.893798  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.895064  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.896164  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:33.897475  5557 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:33.897802  5557 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:33.898689  5557 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:33.848686 17106 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:33.900236  5627 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:33.860543 17111 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:33.909839  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:33.912103  5559 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:29:33.910281  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:33.900602  5557 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:33.861657 17052 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:33.926290  5630 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:33.933069  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:33.918929 17077 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:33.959012 16941 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:33.969856 17072 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:33.973099 16843 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:33.980110  5675 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:33.915550 16988 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:33.883920 16703 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:33.982296  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:33.979178 17091 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.543s	user 0.014s	sys 0.000s
I20260709 12:29:33.900496 17106 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:33.990505  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:33.990629  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:33.982722 16941 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:33.994560 17131 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:33.995078 17131 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:33.995505 17131 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:33.997735 16988 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:33.999894 16703 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:33.991227 17091 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.002501 16995 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:34.003410 17091 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:33.990772 16843 raft_consensus.cc:2764] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election decision vote started in defunct term 35: won
I20260709 12:29:34.008545  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:34.009989  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:34.011251  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:33.993583 17105 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:34.012337  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:34.013331 17127 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.172s	user 0.003s	sys 0.001s
I20260709 12:29:34.016330 17045 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:34.019692  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:34.020058 17045 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:34.020576 17045 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:34.017769 17106 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:34.018390 17105 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:34.023097 17105 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.037024 17091 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:34.037874 17091 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.040887 17091 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:34.041131 17131 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.043489 17131 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:34.033989 16941 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.011s
I20260709 12:29:34.033735 16924 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.015s
W20260709 12:29:34.052084  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:34.054984  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3883) took 1254 ms. Trace:
I20260709 12:29:34.062958  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:34.063611  6277 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:34.064204  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:34.066897  5631 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:34.055418  5709 rpcz_store.cc:276] 0709 12:29:32.812676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:32.977469 (+164793us) service_pool.cc:224] Handling call
0709 12:29:32.977755 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:33.871453 (+893698us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:33.871494 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:33.871553 (+    59us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:29:33.871587 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:34.054720 (+183133us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b5400324454. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:34.054763 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:34.054898 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":249,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"spinlock_wait_cycles":2054619904}
I20260709 12:29:34.067965  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:34.070988  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:34.076781  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:34.077502  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:34.056033 17045 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:34.082149  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:34.084758  5557 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:29:34.084184  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:34.085944  5628 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:34.089118 17045 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:34.094585  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:34.096834  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.098843  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.100911  5557 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:29:34.103355  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.105811  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:34.110613 17105 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:34.104810  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:34.112694  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:34.112929 17113 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:34.113468  5708 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:34.102492 17051 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:34.114365  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:34.115111  5708 raft_consensus.cc:2393] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 35: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:34.100190 16941 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:34.115728 16941 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:34.116925  5707 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35. Ops: [34.23-34.23]
I20260709 12:29:34.113533  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:34.118495  5701 raft_consensus.cc:2368] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37.
I20260709 12:29:34.110651 16868 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:34.121001 17073 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:34.116012  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:34.098078 17118 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 34 current term: 34 is higher.
I20260709 12:29:34.122087  5708 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 31: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:34.117862 16941 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:34.124831  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:34.125864  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:34.126565 16868 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:34.128305  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:34.129252  5703 raft_consensus.cc:2376] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32. Re-sending same reply.
I20260709 12:29:34.129626  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:34.132097  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:34.134794  6277 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 37.
I20260709 12:29:34.135110  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.138733  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.140709  5628 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 36.
I20260709 12:29:34.143486 16868 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:34.147861 16946 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:34.142691 17104 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:34.113523 17113 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.160490 16703 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:34.162176 17113 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:34.138289 17051 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:34.162786 17051 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:34.163167 17051 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:34.166292  5524 leader_election.cc:400] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37.
I20260709 12:29:34.166756  5524 leader_election.cc:403] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Cancelling election due to peer responding with higher term
I20260709 12:29:34.168637  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:34.169137  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:34.169785  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:34.169884  6277 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 28.
I20260709 12:29:34.173512 16995 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:34.174002 16946 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:34.174436 16995 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:34.174854 16995 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:34.174531 16946 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.180778  5703 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:34.180871  5553 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:29:34.182801 16946 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:34.184082  5704 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:29:34.175789 16995 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:34.184366  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.191026 16946 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:34.191574  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:34.192804  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:34.193233  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.194387  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:34.184952  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:34.185350  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:34.186478  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:34.187163  5709 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:29:34.189101  5701 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:29:34.200968  5634 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 33 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:29:34.188376  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:34.203637  5561 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:34.199869  5710 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 36: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:34.216048  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.221973  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:34.219928  5710 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:34.228596  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:34.238799 17121 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.643s	user 0.003s	sys 0.000s
I20260709 12:29:34.247692  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:34.264454  5633 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:34.265539  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:34.266744  5633 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:34.267807  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:34.273332  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:34.274540  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:34.275547  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:34.279369  5633 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 36.
I20260709 12:29:34.289767  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:34.296797  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:34.298020  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 69686.9us
I20260709 12:29:34.200537 16988 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:34.189086 16995 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:34.300179  5706 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 37: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:34.303805  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:34.200834 17118 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:34.301537  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:34.200913 17106 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:34.303956 16872 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:29:34.301740  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:34.204586 17060 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:34.320194  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:34.323356  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:34.325536  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:34.328909  5558 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 34 index: 15. Preceding OpId from leader: term: 35 index: 16. (index mismatch)
I20260709 12:29:34.330293  5599 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.334816  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:34.336531  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 88945.4us
I20260709 12:29:34.338339  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:34.342569  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 64129.6us
I20260709 12:29:34.345438  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:34.303803 17037 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:34.349668  6279 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 23. Preceding OpId from leader: term: 32 index: 24. (index mismatch)
I20260709 12:29:34.349903  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:34.341162 16860 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.009s
I20260709 12:29:34.350553 17072 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:34.351272  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:34.352118  6274 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:34.352054  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:34.341321 16941 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:34.353380  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:34.354228 16940 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:34.354952 16940 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:34.356600 17069 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 35.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:29:34.357856 17068 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:34.358242 17068 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:34.359072 16833 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:29:34.359184 17068 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.23, Last appended by leader: 23, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.360139 16833 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:29:34.361140 17123 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:29:34.361966 16861 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:34.362358 16861 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:34.363137 16861 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:34.309084 17114 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:34.351109 17072 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.361204 16990 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:29:34.368018  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 152100us
I20260709 12:29:34.378186  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:34.359985  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.338618  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:34.374403  5628 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:29:34.296100 17102 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 33.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.102s
I20260709 12:29:34.365468  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:34.374305  5709 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:29:34.375655  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:34.386404  5558 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:34.386487  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.377488  5554 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 33 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:29:34.370011  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:34.389456  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 51006.4us
I20260709 12:29:34.328074 17070 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:34.393292  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:34.375103 17128 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:29:34.388149  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:34.390062  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:34.397532  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.392953  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:34.364459 16868 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:34.393584  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:34.364204 17091 raft_consensus.cc:2764] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election decision vote started in defunct term 36: won
W20260709 12:29:34.382009  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47280.3us
I20260709 12:29:34.401456  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:34.370126 17116 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:34.417471  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.403820  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:34.404233  5633 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 34 index: 15. Preceding OpId from leader: term: 35 index: 16. (index mismatch)
I20260709 12:29:34.377859 16872 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:34.377043 17068 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:34.369628 17126 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:34.475987  6280 raft_consensus.cc:2368] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37.
I20260709 12:29:34.392812 17037 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 36 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:34.398537 16946 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:34.382557 16860 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:34.426761  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:34.403820 17125 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:34.430274 16755 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:34.432651 16861 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:34.379948 16897 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.056s
I20260709 12:29:34.398874 17115 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:34.327244 17106 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.438407 17120 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:34.446307 17072 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:34.353412 16941 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:34.458328  5709 raft_consensus.cc:1275] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:29:34.499230 17125 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:34.463570 17104 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:29:34.465726 16940 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:34.398690 17045 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election won for term 37
W20260709 12:29:34.530040  5450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 182) took 66782 ms (client timeout 15000 ms). Trace:
I20260709 12:29:34.479619  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:34.531730  5634 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:34.417655  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:34.488894 17117 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:29:34.496488  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 110149us
I20260709 12:29:34.535508  5632 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:34.498555  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:34.538075  6275 raft_consensus.cc:2368] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37.
I20260709 12:29:34.499882  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.539691  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.469983 16924 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:34.540131 16866 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:34.540598 16924 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:34.541201 16924 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:34.541774  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.543431  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.543982 17072 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:29:34.544520 16788 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:34.545341 17072 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:34.473902 17114 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:34.458913 17038 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:34.547448 17038 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:34.547227  5459 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 21 to 24, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 24 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:34.548470 17038 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.17, Last appended by leader: 17, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:34.549961 17038 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:34.549602  5459 catalog_manager.cc:5697] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 21 to 28, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 28 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:34.528434 17125 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:34.403676 17070 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.422413  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:34.564245  6274 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:34.493860 17113 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.036s
I20260709 12:29:34.421018 17077 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:34.533608  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:34.581688  5559 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:29:34.501294 16843 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:34.560521  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 142937us
I20260709 12:29:34.477388 16872 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:34.553400  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.575987  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:34.531775  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 176072us
I20260709 12:29:34.580080  5558 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:29:34.545599 17105 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:34.528553 17115 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:34.596342 17060 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:34.597828 17115 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.596565 17045 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 37 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:34.601135 17072 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:34.598114 17060 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.576979 16940 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:34.601754 17045 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:34.597213  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 63685.2us
I20260709 12:29:34.604815 16842 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:34.605418 16842 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.597432 17102 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:34.609289 17102 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:34.609678 17102 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:34.610188  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:34.609828 17037 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.23, Last appended by leader: 23, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.604456 16843 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.602447 16940 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:34.613903 16940 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:34.618462  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 65129.5us
I20260709 12:29:34.619252  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.622757  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:34.468217 17051 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.634050  5459 catalog_manager.cc:5697] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:34.577277 17118 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:34.530686  5450 rpcz_store.cc:269] 0709 12:28:27.747573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:37.797661 (+10050088us) service_pool.cc:224] Handling call
0709 12:28:38.295704 (+498043us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b5400002490. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754e640 00007f5000425846 00007f500052f90e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc 00007f4ffe9ef392 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:34.529936 (+56234232us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:29:32.568586 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:29:32.569302 (+   716us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:29:32.569353 (+    51us) write_op.cc:435] Acquired schema lock
0709 12:29:32.569370 (+    17us) tablet.cc:662] Decoding operations
0709 12:29:32.570170 (+   800us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:29:32.570280 (+   110us) write_op.cc:641] Partition lock acquired for write op
0709 12:29:32.570312 (+    32us) tablet.cc:685] Acquiring locks for 6 operations
0709 12:29:32.571029 (+   717us) tablet.cc:701] Row locks acquired
0709 12:29:32.571073 (+    44us) write_op.cc:260] PREPARE: finished
0709 12:29:32.571239 (+   166us) write_op.cc:270] Start()
0709 12:29:32.571392 (+   153us) write_op.cc:276] Timestamp: P: 1783600172571169 usec, L: 0
0709 12:29:32.571423 (+    31us) op_driver.cc:348] REPLICATION: starting
0709 12:29:32.572131 (+   708us) log.cc:844] Serialized 2394 byte log entry
0709 12:29:33.228053 (+655922us) op_driver.cc:464] REPLICATION: finished
0709 12:29:33.523540 (+295487us) write_op.cc:301] APPLY: starting
0709 12:29:33.550476 (+ 26936us) tablet.cc:1367] starting BulkCheckPresence
0709 12:29:33.582573 (+ 32097us) tablet.cc:1370] finished BulkCheckPresence
0709 12:29:33.582942 (+   369us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:29:33.894642 (+311700us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:29:33.894973 (+   331us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=12,key_file_lookups=12,delta_file_lookups=0,mrs_lookups=0
0709 12:29:33.895040 (+    67us) write_op.cc:312] APPLY: finished
0709 12:29:33.895395 (+   355us) log.cc:844] Serialized 65 byte log entry
0709 12:29:33.895593 (+   198us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:29:33.895715 (+   122us) write_op.cc:454] Released schema lock
0709 12:29:33.895774 (+    59us) write_op.cc:341] FINISH: Updating metrics
0709 12:29:33.908374 (+ 12600us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:29:33.908411 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3327083,"reported_tablets":32,"spinlock_wait_cycles":25884416,"child_traces":[["op",{"apply.queue_time_us":228834,"apply.run_cpu_time_us":13901,"apply.run_wall_time_us":394353,"cfile_cache_hit":26,"cfile_cache_hit_bytes":29207,"num_ops":6,"prepare.queue_time_us":628026,"prepare.run_cpu_time_us":50427,"prepare.run_wall_time_us":195936,"raft.queue_time_us":191873,"raft.run_cpu_time_us":20191,"raft.run_wall_time_us":150304,"replication_time_us":625368,"thread_start_us":839043,"threads_started":4,"wal-append.queue_time_us":205024}]]}
I20260709 12:29:34.637404  5632 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 34 index: 24. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:29:34.647855  5636 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
W20260709 12:29:34.660089  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 263073us
I20260709 12:29:34.662397  6237 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:34.631556 16897 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:34.658679  5459 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 23 to 24, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 24 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:34.680430  5459 catalog_manager.cc:5697] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 22 to 24, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 24 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:34.587177 16868 consensus_queue.cc:1048] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.066s
I20260709 12:29:34.616323 16946 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.040s
I20260709 12:29:34.688906  5459 catalog_manager.cc:5697] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 19 to 26, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 26 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:29:34.689560  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 44578.9us
I20260709 12:29:34.708137  5706 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:29:34.717144  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:34.651352 17118 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.736130  5459 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 26 to 30, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 30 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:29:34.780835  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 363404us
I20260709 12:29:34.782083  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.794217  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:34.690465 17123 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.051s
I20260709 12:29:34.807173  5557 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:34.735673 16843 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:34.818230  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 155922us
I20260709 12:29:34.825194  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.752283 17105 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:29:34.829093  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 202192us
I20260709 12:29:34.829803  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:34.831931  5560 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 36. Current term is 37. Ops: []
W20260709 12:29:34.836897  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 93182.2us
I20260709 12:29:34.795599 17070 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:34.841868  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:34.844820  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:34.828474  5459 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 21 to 25, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 25 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:29:34.852420  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 219624us
I20260709 12:29:34.816740 17106 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:34.851907  5459 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 25 to 28, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 28 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:34.849448  5558 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:34.827021 17120 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:34.877987  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:34.882652 17070 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:34.859084 17155 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:34.924623  5636 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 33: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:34.822873 17115 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.062s
W20260709 12:29:34.937827  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 155850us
I20260709 12:29:34.831866 17157 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.096s
W20260709 12:29:34.914793 17167 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:29:34.889989 16842 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:34.885239 16941 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:34.888033 17105 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:34.853520 16897 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 36 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:34.943738  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.964942  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:34.916589 17120 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:34.920262 17114 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:34.965231  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:34.965428  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:34.940698 17051 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:34.935732 16924 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:34.973462  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:34.990139 16872 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:35.006836  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:34.958459 17151 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.330s	user 0.000s	sys 0.001s
W20260709 12:29:35.013123 17160 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 12:29:34.996160 16755 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:29:35.055424  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48647.1us
W20260709 12:29:35.057247  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:34.990710 17120 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:34.996482 17158 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:34.948191 17180 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.080s
I20260709 12:29:35.025236 17057 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:35.026823 16924 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:35.027199 16788 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:35.027247 17060 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:34.944114 17178 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.101s
I20260709 12:29:34.948478 17106 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.080s
W20260709 12:29:35.028599 17161 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:29:35.028664 17114 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:35.029685 17162 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.141s	user 0.000s	sys 0.000s
I20260709 12:29:35.073160 17158 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:35.029837 16995 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.714s	user 0.012s	sys 0.002s
W20260709 12:29:35.031996 17068 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.538s	user 0.006s	sys 0.006s
W20260709 12:29:35.034003 17154 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
I20260709 12:29:35.035638  5706 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:35.074138  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:35.177664  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 103628us
I20260709 12:29:35.178507  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:35.179104  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:35.179862  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:35.024631 17164 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20260709 12:29:35.121152  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:35.034929 17118 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:35.123719  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 201317us
W20260709 12:29:35.130024  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 276237us
I20260709 12:29:35.154281  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:35.116287 17038 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.566s	user 0.012s	sys 0.002s
I20260709 12:29:34.991324 17105 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:35.030371 17153 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.283s	user 0.000s	sys 0.001s
I20260709 12:29:35.081590 16872 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:35.118064 17057 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:35.175275  5561 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 35 because replica has last-logged OpId of term: 34 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 22.
W20260709 12:29:35.118781  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 499588us
I20260709 12:29:35.202147  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:35.182739  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:35.189476  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 248247us
I20260709 12:29:35.179941  5459 catalog_manager.cc:5697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 22 to 26, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 26 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:35.186071 16995 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:35.190842  5676 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:35.181229  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:35.205201  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3842) took 1141 ms. Trace:
I20260709 12:29:35.171041 17158 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:35.206493  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:35.207607  5632 raft_consensus.cc:2393] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:35.205627  5627 rpcz_store.cc:276] 0709 12:29:34.063396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:34.068438 (+  5042us) service_pool.cc:224] Handling call
0709 12:29:34.068735 (+   297us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:34.073592 (+  4857us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:29:34.073620 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:34.076000 (+  2380us) log.cc:844] Serialized 26 byte log entry
0709 12:29:34.383186 (+307186us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:34.383223 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:34.383285 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:35.204984 (+821699us) raft_consensus.cc:1685] finished
0709 12:29:35.205005 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:35.205141 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":108,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":4673,"thread_start_us":249512,"threads_started":1,"wal-append.queue_time_us":422432}
I20260709 12:29:35.187325 17123 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:35.176709 17111 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:35.388010  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3906) took 1224 ms. Trace:
W20260709 12:29:35.213667 17125 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.662s	user 0.013s	sys 0.000s
I20260709 12:29:35.388384 17111 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:35.388846 17125 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:35.390201  5552 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: [31.21-31.21]
I20260709 12:29:35.390832 17111 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:35.391003 17125 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:35.200907 17057 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:29:35.218858 16861 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.709s	user 0.013s	sys 0.000s
I20260709 12:29:35.201085 16872 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:35.392284 16872 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:35.393705 16861 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:35.394263 16861 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:35.283042 17172 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
W20260709 12:29:35.266924 16988 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.951s	user 0.011s	sys 0.005s
I20260709 12:29:35.396343 16861 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:35.352943 17166 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.404s	user 0.001s	sys 0.000s
I20260709 12:29:35.396943  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:35.259132 16995 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:35.401407 16995 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:35.358058 17182 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.098s	user 0.000s	sys 0.002s
I20260709 12:29:35.397289  5705 rpcz_store.cc:276] 0709 12:29:34.163554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:34.176467 (+ 12913us) service_pool.cc:224] Handling call
0709 12:29:34.176791 (+   324us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:34.177004 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:34.177029 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:34.177455 (+   426us) log.cc:844] Serialized 26 byte log entry
0709 12:29:34.516144 (+338689us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:29:34.516205 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:34.516319 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:35.387749 (+871430us) raft_consensus.cc:1685] finished
0709 12:29:35.387777 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:35.387933 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":338357,"threads_started":1,"wal-append.queue_time_us":523923}
I20260709 12:29:35.397847 16988 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader election lost for term 36. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37.
W20260709 12:29:35.415462  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 213372us
W20260709 12:29:35.372309 17187 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.106s	user 0.004s	sys 0.000s
I20260709 12:29:35.372108 17178 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:29:35.243394 17115 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:35.373224 17116 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:29:35.374430  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2313) took 1110 ms. Trace:
W20260709 12:29:35.375221 17102 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.765s	user 0.011s	sys 0.004s
I20260709 12:29:35.185895 16897 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:35.376323 17159 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.402s	user 0.004s	sys 0.000s
I20260709 12:29:35.552964  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:35.375950 16755 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:35.378530 16940 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.764s	user 0.011s	sys 0.001s
I20260709 12:29:35.378551 17105 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:35.380934 17124 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:35.381733 16843 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:35.382440 17114 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:35.225296 17181 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:35.388236 17123 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:35.420322  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 213891us
W20260709 12:29:35.420955  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 213793us
I20260709 12:29:35.442744  5553 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4246) took 1120 ms. Trace:
I20260709 12:29:35.437819  5628 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
I20260709 12:29:35.436604  5629 rpcz_store.cc:276] 0709 12:29:34.264104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:34.270412 (+  6308us) service_pool.cc:224] Handling call
0709 12:29:34.270743 (+   331us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:34.271109 (+   366us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:34.271138 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:34.271618 (+   480us) log.cc:844] Serialized 26 byte log entry
0709 12:29:34.641307 (+369689us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:34.641365 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:34.641454 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:35.374169 (+732715us) raft_consensus.cc:1685] finished
0709 12:29:35.374198 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:35.374357 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1831,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":92,"thread_start_us":369277,"threads_started":1,"wal-append.queue_time_us":437984}
I20260709 12:29:35.422144 17155 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:35.422215 16788 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:35.534271  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:35.591606  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 147111us
I20260709 12:29:35.535346  5555 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 37.
I20260709 12:29:35.537827  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:29:35.525390  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 345558us
I20260709 12:29:35.541615  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:35.541733  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:35.555543  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 345496us
I20260709 12:29:35.571914  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:35.602155  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:35.575930  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:35.603231  6273 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:29:35.526396 17115 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:35.576733 16940 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:35.617681  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:35.618526  5558 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 40.
I20260709 12:29:35.619560  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:35.619603  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:35.620290  6274 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:35.588042  5553 rpcz_store.cc:276] 0709 12:29:34.322644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:34.323653 (+  1009us) service_pool.cc:224] Handling call
0709 12:29:34.365106 (+ 41453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:34.374663 (+  9557us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:34.374696 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:34.384572 (+  9876us) log.cc:844] Serialized 26 byte log entry
0709 12:29:34.804215 (+419643us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:29:34.804274 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:34.804388 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:35.428451 (+624063us) raft_consensus.cc:1685] finished
0709 12:29:35.428554 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:35.434848 (+  6294us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4674,"raft.queue_time_us":504,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":15204,"thread_start_us":332225,"threads_started":1,"wal-append.queue_time_us":435643}
I20260709 12:29:35.577267 17124 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:35.625144  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:35.627534  6273 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:35.602198 17123 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:35.560999 17102 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:35.613631 16860 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:29:35.576893 16755 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:35.570791 17057 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:35.563812 17120 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:35.630195 17124 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:35.644644  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4403) took 1198 ms. Trace:
I20260709 12:29:35.582652 16843 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:35.629271 16861 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:35.657722  5554 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:35.654850  5709 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: []
W20260709 12:29:35.666852  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 109472us
I20260709 12:29:35.650898  5708 rpcz_store.cc:276] 0709 12:29:34.445951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:34.447034 (+  1083us) service_pool.cc:224] Handling call
0709 12:29:34.447399 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:34.447774 (+   375us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:34.447802 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:34.448261 (+   459us) log.cc:844] Serialized 26 byte log entry
0709 12:29:34.703051 (+254790us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:29:34.703093 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:34.703154 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:35.644443 (+941289us) raft_consensus.cc:1685] finished
0709 12:29:35.644463 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:35.644582 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1184,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"thread_start_us":253535,"threads_started":1,"wal-append.queue_time_us":427293}
I20260709 12:29:35.673204  6274 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:29:35.621814 17115 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:35.690629  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 88579us
I20260709 12:29:35.655802 17120 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:35.657598 17057 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:35.670257 16843 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:35.688969 17051 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:35.653033 17195 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:35.696798  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:35.672210 16988 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:35.674175 17181 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:35.654150 17118 consensus_queue.cc:1059] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 33.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.039s
I20260709 12:29:35.690397 16860 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:35.676520 17158 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:35.674293 16861 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:35.751405  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:35.752193  5561 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 4252) took 1371 ms (client timeout 1505 ms). Trace:
I20260709 12:29:35.728080 17195 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:35.752740  5561 rpcz_store.cc:269] 0709 12:29:34.373625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:34.375374 (+  1749us) service_pool.cc:224] Handling call
0709 12:29:35.734609 (+1359235us) spinlock_profiling.cc:243] Waited 774 ms on lock 0x7b540023f854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:35.744767 (+ 10158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1779785984}
I20260709 12:29:35.783471  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:35.690706 17175 consensus_queue.cc:1059] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 36.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.069s
W20260709 12:29:35.796042  5526 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.512s (SENT) [suppressed 11 similar messages]
W20260709 12:29:35.782895 17171 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:29:35.781497 17057 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:35.801673  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3825) took 1230 ms. Trace:
I20260709 12:29:35.810041  5556 rpcz_store.cc:276] 0709 12:29:34.571560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:34.571731 (+   171us) service_pool.cc:224] Handling call
0709 12:29:34.590563 (+ 18832us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:34.595186 (+  4623us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:34.595214 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:34.629185 (+ 33971us) log.cc:844] Serialized 26 byte log entry
0709 12:29:34.878896 (+249711us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:34.878961 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:34.879066 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:35.801270 (+922204us) raft_consensus.cc:1685] finished
0709 12:29:35.801307 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:35.801573 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4199,"raft.queue_time_us":10982,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"thread_start_us":205282,"threads_started":1,"wal-append.queue_time_us":1095154}
I20260709 12:29:35.728945 17051 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:35.806525  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 110846us
W20260709 12:29:35.813400  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 127538us
I20260709 12:29:35.732173 16988 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:35.783113 17195 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:35.833439  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:35.838063  6274 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:29:35.845156  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:35.751256 16872 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:35.734604 17181 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:35.853610  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:35.860960  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 266940us
I20260709 12:29:35.826182 17118 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:35.868865 17118 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:35.883015 17201 log.cc:927] Time spent T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.176s	user 0.000s	sys 0.000s
I20260709 12:29:35.828433 17057 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:35.867159 16940 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:35.875718 17102 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:35.903808  6234 connection.cc:256] RPC call timeout handler was delayed by -1.2510897779998231s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1523s for a response
I20260709 12:29:35.898797 16924 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.010s
W20260709 12:29:35.941452  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 272726us
W20260709 12:29:35.948766  6234 connection.cc:256] RPC call timeout handler was delayed by -1.2483897779998188s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
I20260709 12:29:35.913758 17039 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:35.958608  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:35.956205  6274 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 24. Preceding OpId from leader: term: 33 index: 26. (index mismatch)
I20260709 12:29:35.957376  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:35.954557  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 158549us
I20260709 12:29:35.960112  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:35.965207  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:35.974133  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:35.974550  6276 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:29:35.977278 17221 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:35.978251 17221 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:35.999493  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:36.017629 17057 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.021377  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:36.025879  5526 leader_election.cc:400] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37.
I20260709 12:29:36.027680  5554 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:36.030154  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:35.972564 17115 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:35.968328 17120 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:35.976109 16755 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.042841  5630 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 35. Current term is 36. Ops: []
W20260709 12:29:36.060907  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
W20260709 12:29:36.048563  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 110162us
I20260709 12:29:36.055486  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:36.002463 16995 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:36.069984  6279 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:36.071120  5705 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
W20260709 12:29:36.071250  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 97196.7us
I20260709 12:29:36.071923  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:36.072566  5630 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 36
I20260709 12:29:36.073017  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:36.055282  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:36.073160  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:36.073719  5701 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:36.050770  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:36.074558  5628 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:36.074621  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:36.075176  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:36.075810  6279 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 33: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:36.076529  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:36.077143  5631 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:36.080428  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:36.081056  5704 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
W20260709 12:29:36.038741 17212 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.247s	user 0.001s	sys 0.001s
I20260709 12:29:36.088825  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:36.089222  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:36.086158  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:36.038311 17194 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:36.090205  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:36.088608  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:36.086989  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:36.089149  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:36.091957  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:36.094789  5632 raft_consensus.cc:2368] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33.
I20260709 12:29:36.041450 16861 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:36.094498  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:36.098038  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:36.098745  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4409) took 1253 ms. Trace:
I20260709 12:29:36.099462  5627 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:35.974545 17039 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.044889 17069 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:36.133469  5706 rpcz_store.cc:276] 0709 12:29:34.845211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:34.845930 (+   719us) service_pool.cc:224] Handling call
0709 12:29:34.846556 (+   626us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:35.184086 (+337530us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:36.098443 (+914357us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54002d14d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:36.098686 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":434307456}
W20260709 12:29:36.160063  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 65656.4us
I20260709 12:29:36.018643 17198 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.075s
I20260709 12:29:36.165302  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:36.166173  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1848) took 1124 ms. Trace:
I20260709 12:29:36.166446  5636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:36.167137  5636 raft_consensus.cc:606] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:36.167585  5636 raft_consensus.cc:614] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:36.033768 17126 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 35.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.082s
I20260709 12:29:36.172417  5636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:36.166594  6280 rpcz_store.cc:276] 0709 12:29:35.036970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:35.037435 (+   465us) service_pool.cc:224] Handling call
0709 12:29:36.097816 (+1060381us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:29:36.161576 (+ 63760us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7033600}
I20260709 12:29:36.104904 16995 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:36.190869 16995 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:36.191540 16995 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:36.193748 17039 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:36.195182 17209 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:36.195761 17209 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:36.165331 16860 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:36.072952 17120 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:36.185122 17196 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:36.088699 16755 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.085s
I20260709 12:29:36.187178  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:36.249101  6271 raft_consensus.cc:2368] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37.
I20260709 12:29:36.179924 17125 raft_consensus.cc:3055] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:36.184631 17220 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:29:36.205962  5636 raft_consensus.cc:606] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:36.251827  5636 raft_consensus.cc:614] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:29:36.252597  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 206575us
W20260709 12:29:36.198244 17213 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.322s	user 0.001s	sys 0.000s
I20260709 12:29:36.187819 16872 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.172652 17051 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.249897  5628 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:36.259668  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:36.173375 17102 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:36.220769 17181 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.198187 17209 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:36.262871  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:29:36.199224 17221 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:36.251848 17125 raft_consensus.cc:740] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:36.267361  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 162357us
I20260709 12:29:36.268157  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:36.247678 17196 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:36.253791 17220 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:36.218940  5631 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:36.202524 16860 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:36.277745  5601 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:36.265477  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:36.271865 17181 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:36.262382 17126 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.015s
I20260709 12:29:36.288882  5630 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:29:36.277319 17196 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:36.266873 17102 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:36.285669  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 198138us
I20260709 12:29:36.296146  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:36.293335  5601 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:36.277496 17220 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:29:36.288915 16861 raft_consensus.cc:2764] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election decision vote started in defunct term 35: won
I20260709 12:29:36.300552  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:36.289465 17181 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:36.302085 17210 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.363s	user 0.000s	sys 0.001s
I20260709 12:29:36.303081  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:36.304425 17241 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.004s
I20260709 12:29:36.305160  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:36.306035  5630 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:36.306066  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:36.307439 16872 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:36.310398 17102 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:36.311544  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:36.312563 17057 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:36.312165  5556 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:36.315780  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:36.316655  5631 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:36.316821  5628 raft_consensus.cc:1240] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:36.321499  5630 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:36.297345 17125 consensus_queue.cc:260] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:36.293877 17237 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:29:36.317178 17181 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:36.328970 17239 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.007s
I20260709 12:29:36.325883 17241 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:36.321573 16860 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:36.325713 17051 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:36.329128 17125 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:36.329564 17221 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:29:36.358949  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 58466.4us
I20260709 12:29:36.361099 16843 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:36.379105  5708 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:36.372816  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:36.378911  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:36.373687  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:36.378310  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 215392us
I20260709 12:29:36.380069  5710 raft_consensus.cc:2393] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 35: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:36.383071  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:36.383738  5710 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:36.360857 17120 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.356470 17238 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:36.368258 16861 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:36.398808  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:36.359166 17221 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:36.403275  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:36.355688 17126 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 35.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.031s
I20260709 12:29:36.401985  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:36.360250 17051 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:36.448236  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 123788us
I20260709 12:29:36.392053 16843 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.417315 17221 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:36.394840 16860 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:36.454325  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:36.450074  5526 leader_election.cc:400] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33.
I20260709 12:29:36.456126  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:36.462054  5526 leader_election.cc:403] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
I20260709 12:29:36.437151 17104 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:29:36.462984  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45813.6us
I20260709 12:29:36.415861 16861 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:36.411834 17238 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.493081  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:36.458431 16703 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:36.455020 16866 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:36.470191 17090 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:36.468468 16860 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.528905 17239 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:36.529577 17239 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:36.530414 16860 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:36.531211 17104 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:36.532598 17239 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.513983 17120 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:36.534540 17245 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:36.496320 17123 consensus_meta.cc:287] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.854s	user 0.040s	sys 0.033s
W20260709 12:29:36.499662 17124 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.828s	user 0.015s	sys 0.001s
I20260709 12:29:36.455261 16995 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.511065  5707 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:36.511747 16703 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:36.516041 16866 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:36.519200 16872 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:36.497769 17248 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33.
I20260709 12:29:36.521322 17238 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:36.522872 17240 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260709 12:29:36.501724 16941 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:36.524273 17090 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:36.538477  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:36.574414 17070 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:36.583751  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:36.589781  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:36.538337  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:36.595003  6278 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
I20260709 12:29:36.600368  6279 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:36.603024  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3832) took 1051 ms. Trace:
I20260709 12:29:36.604008  5554 rpcz_store.cc:276] 0709 12:29:35.552406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:35.552635 (+   229us) service_pool.cc:224] Handling call
0709 12:29:36.602810 (+1050175us) spinlock_profiling.cc:243] Waited 95.6 ms on lock 0x7b540023ffd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:36.602947 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":219861888}
I20260709 12:29:36.608628  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:36.609975  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:36.584969 17196 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:36.611078  5561 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:36.613330 17196 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.613463  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:36.614609  5556 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:36.627442  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:36.628835  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:36.602398 17195 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.743s	user 0.040s	sys 0.025s
W20260709 12:29:36.604519  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 66106.8us
I20260709 12:29:36.637346  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:36.596814 16941 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.640225  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:36.573232 17123 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.625332 16872 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:36.581494 17124 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:36.636400 16995 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:36.643671 17196 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.014s
I20260709 12:29:36.657624  5702 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:36.650265 17051 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.658528 17124 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:36.652918 16861 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.606173 17070 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.661662 17220 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:36.651630 16866 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.649139 16843 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:36.666181 17123 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:36.667335 16872 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:36.640705 17195 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.668291 16872 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.669543  5632 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: [31.21-31.21]
I20260709 12:29:36.670264 17195 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:36.671442  6273 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 33: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:36.636682 17194 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:36.658694 17248 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:36.633244 17193 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:36.672657 17248 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:36.666857  5629 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 36: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:36.671422  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3909) took 2064 ms. Trace:
I20260709 12:29:36.673099 17248 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:36.673400  5709 rpcz_store.cc:276] 0709 12:29:34.607033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:34.609003 (+  1970us) service_pool.cc:224] Handling call
0709 12:29:34.618710 (+  9707us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:35.667764 (+1049054us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:36.671191 (+1003427us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540026fa54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:36.671365 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2375956864}
I20260709 12:29:36.652621 17229 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:36.674618  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:36.674973 17229 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.536756 17245 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:36.677479  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:36.678064 17245 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.649807 16941 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:36.679296 17229 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:36.662039 17070 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.680775 17221 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:36.681885 17228 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.607s	user 0.000s	sys 0.006s
I20260709 12:29:36.688692  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:36.689065  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:36.690258 17220 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 41 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.691048 17158 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:36.692329  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3838) took 1136 ms. Trace:
I20260709 12:29:36.692785  5560 rpcz_store.cc:276] 0709 12:29:35.555721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:35.556257 (+   536us) service_pool.cc:224] Handling call
0709 12:29:35.556524 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:35.590339 (+ 33815us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:29:35.716421 (+126082us) log.cc:844] Serialized 14 byte log entry
0709 12:29:35.728555 (+ 12134us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:35.728631 (+    76us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:29:35.728700 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:36.692094 (+963394us) spinlock_profiling.cc:243] Waited 33.6 ms on lock 0x7b54003c4bd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:36.692153 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:36.692274 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42783,"prepare.queue_time_us":27343,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":2687,"raft.queue_time_us":12308,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":10601,"spinlock_wait_cycles":77204352,"thread_start_us":125643,"threads_started":1}
I20260709 12:29:36.683045 17221 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:36.707185  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:36.708289  5636 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 33: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:29:36.715595 17190 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.335s	user 0.000s	sys 0.008s
W20260709 12:29:36.721704  5555 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 4251) took 2348 ms (client timeout 1906 ms). Trace:
W20260709 12:29:36.722339  5555 rpcz_store.cc:269] 0709 12:29:34.373331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:34.375300 (+  1969us) service_pool.cc:224] Handling call
0709 12:29:36.721412 (+2346112us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54001af254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:36.721559 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2318920064}
W20260709 12:29:36.729647 17232 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.654s	user 0.003s	sys 0.000s
I20260709 12:29:36.745510  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2291) took 1914 ms. Trace:
I20260709 12:29:36.745986  5552 rpcz_store.cc:276] 0709 12:29:34.831049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:34.831231 (+   182us) service_pool.cc:224] Handling call
0709 12:29:34.853102 (+ 21871us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:35.390568 (+537466us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:36.745205 (+1354637us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b54000eb0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:36.745442 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1234689792}
I20260709 12:29:36.748562 16703 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.751206  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:36.713737  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:36.642043 17125 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:36.758528  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:36.758924  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49666.5us
I20260709 12:29:36.759455  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:36.764765  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:36.769575  5631 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:36.774077  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:36.775168  5636 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:36.780822  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:36.782914  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:36.789621 17249 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:36.766916 17158 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:36.791592  5526 leader_election.cc:400] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37.
I20260709 12:29:36.767473 17070 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:36.820026  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:36.822252  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:36.820791  5552 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:36.787040 17206 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:36.827126  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:36.787678 17125 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:36.785986 17077 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:36.844542 17120 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:36.815165 17158 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:36.784189 16897 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.046s
W20260709 12:29:36.851260 17181 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.502s	user 0.012s	sys 0.002s
I20260709 12:29:36.829169 17249 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:36.863452  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:36.867957  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:36.870860  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:36.850431 16988 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.821367 17070 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.868441  5636 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:36.876950  5602 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:36.852394 17120 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:36.871680  5554 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:29:36.897742  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:36.885066  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:36.898432  5558 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:36.898277  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:36.903131 17264 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:36.903847 17220 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:36.904224 17220 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:36.904456  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:36.905591  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:36.906659 17229 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:36.906901  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:36.907044 17229 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:36.907428 17229 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:36.904603 17220 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:36.908632 17070 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:36.910439 16995 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 35.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:36.914402 16866 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:36.914999 16866 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:36.915374 16866 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:36.866122 17181 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.916829  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:36.906247 17123 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:36.918799 17181 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:36.906392  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:36.920485  5553 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:36.919009 17123 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:36.923135  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:36.925565  6273 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:36.878401 17221 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:36.858639 17077 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.900480 17120 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:36.885951 17237 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:36.949049  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 164984us
I20260709 12:29:36.961186  5554 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 41.
I20260709 12:29:36.962563  5558 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:36.969074  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:37.011526  6275 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1847) took 1974 ms (client timeout 2031 ms). Trace:
I20260709 12:29:37.011430  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3914) took 1857 ms. Trace:
I20260709 12:29:36.964782  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:29:37.014207  6275 rpcz_store.cc:269] 0709 12:29:35.036633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:35.037347 (+   714us) service_pool.cc:224] Handling call
0709 12:29:36.968887 (+1931540us) spinlock_profiling.cc:243] Waited 2.73 ms on lock 0x7b58000d4198. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754db71 00007f4ff754da22 00007f50004668d9 00007f4fff5c0ee0 00007f4fff68a0ab 00007f4fff68a131 00007f4fff6380a0 00007f4fff637984 00007f4ff8bb26d4 00007f4ff8bb2662
0709 12:29:37.011457 (+ 42570us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6269184}
I20260709 12:29:37.014289  5703 rpcz_store.cc:276] 0709 12:29:35.156228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:35.156419 (+   191us) service_pool.cc:224] Handling call
0709 12:29:35.156875 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:35.158721 (+  1846us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:35.158757 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:36.469628 (+1310871us) spinlock_profiling.cc:243] Waited 4.46 ms on lock 0x7b2c0008bd48. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff8f6ba11 00007f4ff8f7b372 00007f4ffa6ada01 00007f4ffa67243a 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2
0709 12:29:36.495641 (+ 26013us) log.cc:844] Serialized 26 byte log entry
0709 12:29:36.631667 (+136026us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:36.631715 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:36.631785 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:36.963711 (+331926us) raft_consensus.cc:1685] finished
0709 12:29:36.963731 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:37.011360 (+ 47629us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6665,"raft.queue_time_us":12993,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":8739,"spinlock_wait_cycles":10255744,"thread_start_us":135573,"threads_started":1,"wal-append.queue_time_us":290446}
I20260709 12:29:37.017190  6273 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
W20260709 12:29:37.017390  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48688.1us
I20260709 12:29:37.019198  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:37.023134  5628 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
W20260709 12:29:37.027419  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62726us
I20260709 12:29:37.028566  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:37.029325  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:37.016134 16897 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:37.012809 17239 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:36.851442 17206 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.055724 16897 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.871161 17249 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.057816 17206 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.058069 16897 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.058326 17249 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.058490  5555 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
W20260709 12:29:37.048864 17258 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
I20260709 12:29:37.049741 17077 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:37.035778 16861 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:29:37.055869 17239 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:37.041481 17123 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.051900 17090 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.063336 17239 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:29:37.023610 16995 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:37.064709 16995 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:36.963250 17125 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:37.070333  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:37.076246  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.077345  5630 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 29.
I20260709 12:29:36.960767 17221 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:36.963243 17237 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.087476  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.088905  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:37.089517  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 60230.1us
I20260709 12:29:36.952944 17158 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.069795 16861 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:37.065686 16995 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.129599  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.130535  5555 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:37.091593 17266 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:37.093726 17090 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election won for term 36
W20260709 12:29:37.140012  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48153.3us
I20260709 12:29:37.136268  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.141767  5558 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 35: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:37.119293  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:37.118754 17077 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:37.124596 17268 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:37.160252  5631 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: [31.21-31.21]
I20260709 12:29:37.145074 17246 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:37.142330 17184 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:37.165026 17184 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:37.167198 17261 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
I20260709 12:29:37.167578 17184 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.152447 17090 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:37.155247 17223 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:29:37.170696  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:37.140403 17248 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.142442 17266 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.168848 17268 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.188247  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:37.189038 17227 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.146s	user 0.001s	sys 0.005s
I20260709 12:29:37.154490 17077 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.192823 17216 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:37.193449 17216 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.180554 16866 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.194891 17116 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:37.195282 17116 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:37.195608 17216 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:37.195868 16866 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:37.189937 17262 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.317s	user 0.000s	sys 0.001s
I20260709 12:29:37.196188 17116 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.175063 17223 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:37.169338 17229 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.200091  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.201643  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.188155 17239 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 30.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.015s
I20260709 12:29:37.201174 17268 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.208746  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.212452  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.216261  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:37.201422 17223 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:37.199298 17216 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:37.219794  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 41 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:37.199862 17220 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.221230 17237 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.223053 17229 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.225010  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:37.227272 17176 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 2.154s	user 0.008s	sys 0.003s
I20260709 12:29:37.181519 16861 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.24, Last appended by leader: 24, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.229842  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.231521 17241 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:37.247673  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 4029) took 1084 ms. Trace:
I20260709 12:29:37.246399  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.232470 17118 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.248132  5627 rpcz_store.cc:276] 0709 12:29:36.163109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:37.229472 (+1066363us) service_pool.cc:224] Handling call
0709 12:29:37.247596 (+ 18124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:37.250934  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40390 (request call id 2333) took 1087 ms. Trace:
I20260709 12:29:37.252183  5634 rpcz_store.cc:276] 0709 12:29:36.163399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:37.224478 (+1061079us) service_pool.cc:224] Handling call
0709 12:29:37.250819 (+ 26341us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:37.254379  5708 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:37.239267 17123 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:37.248916 17241 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:37.259382 17123 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:37.260042 17241 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.260385 17123 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.261412 17241 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:37.261799 17123 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:37.244781 16866 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:37.263811 16866 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.258628  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.270915  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.249118 17221 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:37.281447  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.244398 17216 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.291857  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.211429 17090 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.291976  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:37.285732 16866 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.293931  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.297915  5553 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:37.298822  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:37.299806  5627 raft_consensus.cc:2368] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:37.301169  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:37.301936  5553 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:37.307480  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:37.308387  5557 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:37.295063  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.311060  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:37.321802  5708 raft_consensus.cc:2410] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 37 because replica has last-logged OpId of term: 36 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 20.
I20260709 12:29:37.322856  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3860) took 2694 ms. Trace:
I20260709 12:29:37.328152  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.323431  5635 rpcz_store.cc:276] 0709 12:29:34.619807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:34.619934 (+   127us) service_pool.cc:224] Handling call
0709 12:29:34.642000 (+ 22066us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:34.642464 (+   464us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:34.642506 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:34.643098 (+   592us) log.cc:844] Serialized 26 byte log entry
0709 12:29:34.888889 (+245791us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:29:34.888955 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:34.889057 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:37.226597 (+2337540us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:37.301704 (+ 75107us) raft_consensus.cc:1685] finished
0709 12:29:37.301728 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:37.313835 (+ 12107us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1485,"raft.queue_time_us":119,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":7360,"spinlock_wait_cycles":36612352,"thread_start_us":186046,"threads_started":1,"wal-append.queue_time_us":359422}
I20260709 12:29:37.330274  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.332687  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:37.314993 17173 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.138s	user 0.020s	sys 0.000s
I20260709 12:29:37.337658  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:37.304454 17184 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:37.298305 17158 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:37.303385 17077 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.346026  5553 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:37.322968 16866 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:29:37.337990 17173 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.177s	user 0.020s	sys 0.000s
I20260709 12:29:37.371328  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.376111  5553 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:37.378531  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.379216  5704 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:37.385874  5557 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:37.370829 16866 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:37.396880 16866 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:37.367092 17184 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:37.396147  5599 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:37.415477  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:37.416579 17052 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:29:37.417286 17052 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:29:37.423655 17211 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 1.479s	user 0.006s	sys 0.000s
I20260709 12:29:37.426862 17052 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:37.429970  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 4021) took 1380 ms. Trace:
I20260709 12:29:37.430558  5629 rpcz_store.cc:276] 0709 12:29:36.049794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:36.049950 (+   156us) service_pool.cc:224] Handling call
0709 12:29:37.429699 (+1379749us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b54000c1a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:37.429885 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1317600640}
I20260709 12:29:37.434576 17125 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:37.435117 17183 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:37.435851  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 31 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.435139 17125 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.437101  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:37.437775  5561 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 33: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:37.439182 17125 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.435675 17183 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.440860  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4418) took 1069 ms. Trace:
I20260709 12:29:37.441773 17183 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:37.442802 17278 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:29:37.441829  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4419) took 1069 ms. Trace:
I20260709 12:29:37.447180  5707 rpcz_store.cc:276] 0709 12:29:36.372095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:36.378635 (+  6540us) service_pool.cc:224] Handling call
0709 12:29:37.441644 (+1063009us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b54004524d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:37.441750 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":301840128}
I20260709 12:29:37.449082  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.449324  5561 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 35. Current term is 36. Ops: [34.24-35.25]
I20260709 12:29:37.441310  5702 rpcz_store.cc:276] 0709 12:29:36.371822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:36.374394 (+  2572us) service_pool.cc:224] Handling call
0709 12:29:36.374753 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:36.658027 (+283274us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:37.440581 (+782554us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b54002f5154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:37.440789 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":650047744}
I20260709 12:29:37.453465  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:37.454955  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 127085us
I20260709 12:29:37.443598  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.458369  5636 raft_consensus.cc:2410] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 36 because replica has last-logged OpId of term: 35 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 15.
I20260709 12:29:37.449955 17266 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.459870  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.461915  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.462977  5629 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:37.407917 17184 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:37.464507  5633 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 35. Current term is 36. Ops: [35.25-35.25]
I20260709 12:29:37.466137  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.371694 17158 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.469874  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.467075  5561 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:37.465790 16861 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:29:37.460639 17261 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:29:37.474865 16861 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:37.533239 16861 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:37.536613 17072 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:29:37.537076 17072 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:37.537997 17072 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.22, Last appended by leader: 22, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.538481 17223 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.539131 17072 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:37.542460  6278 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 36 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:37.545645  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:37.546263  5561 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:37.474579  5553 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: []
W20260709 12:29:37.472636 17224 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.432s	user 0.024s	sys 0.052s
W20260709 12:29:37.551110 17224 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.520s	user 0.024s	sys 0.053s
I20260709 12:29:37.476786 17090 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:29:37.477021 17120 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.464751 17220 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.552309  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:37.484706  5552 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:37.486135 17158 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:37.488704  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.552529  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.594642  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.574416  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:37.603395  5553 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:29:37.522574 17216 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:37.524672 17116 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:37.530139 17248 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.581382 17246 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:37.581602 17178 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:37.553234 17268 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.037s
I20260709 12:29:37.554934 17090 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:37.558722  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:37.565296  5556 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 36: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:29:37.597524  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:37.506242 17253 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:37.641017  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.641562  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.603453  6273 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:37.509121  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.574741  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:37.624140  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 180967us
I20260709 12:29:37.507879  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:37.473755  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:37.645509  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:37.553586 17115 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.056s
I20260709 12:29:37.650679  5560 raft_consensus.cc:2410] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 35 because replica has last-logged OpId of term: 34 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 22.
W20260709 12:29:37.577986  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58753.9us
I20260709 12:29:37.651360  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:37.651485  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:37.652287  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 178626us
I20260709 12:29:37.606788  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:37.611559  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:37.611654  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:37.572834 17219 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.399s	user 0.000s	sys 0.006s
I20260709 12:29:37.556777 17120 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.662487  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.631258  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:37.662919  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:37.639523  5629 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:37.571341  6275 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:37.643891  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:37.574815  5635 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:29:37.646718  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.667441  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.667929  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:37.668540 17104 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:37.669355 17104 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.669768  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:37.671135  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:37.671591  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.602325 16866 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.609673 17193 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.050s
I20260709 12:29:37.660090  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:37.660638  5701 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:37.634358 17246 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:37.675767  5602 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:37.689188  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.669232  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:37.683399  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:37.675465  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3834) took 2121 ms. Trace:
I20260709 12:29:37.690061  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:37.706777  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 88803us
I20260709 12:29:37.672756  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.673029  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:37.661276  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:37.689065  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:37.700691  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54255.9us
I20260709 12:29:37.677676  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.672200  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.686143  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:37.669025  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.731941  5555 raft_consensus.cc:2376] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 37. Re-sending same reply.
I20260709 12:29:37.736873  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:37.737908  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62242.7us
I20260709 12:29:37.638840 17090 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:37.750628  5552 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 36: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:37.753458  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.754148  5701 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:37.739074  5602 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:37.757046  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.640023 17253 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:37.623504 17158 consensus_queue.cc:1059] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.068s
I20260709 12:29:37.763968  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:37.764223  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:37.757046  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:37.759692  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:37.764617  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:37.760551  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 153841us
I20260709 12:29:37.727620  5559 rpcz_store.cc:276] 0709 12:29:35.554310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:35.554501 (+   191us) service_pool.cc:224] Handling call
0709 12:29:35.554938 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:35.572112 (+ 17174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:35.572150 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:35.594787 (+ 22637us) log.cc:844] Serialized 26 byte log entry
0709 12:29:35.954400 (+359613us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:29:35.954465 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:35.954577 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:37.572168 (+1617591us) spinlock_profiling.cc:243] Waited 22.6 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:37.675146 (+102978us) raft_consensus.cc:1685] finished
0709 12:29:37.675172 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:37.675402 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66878,"raft.queue_time_us":7285,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":10696,"spinlock_wait_cycles":51874560,"thread_start_us":308716,"threads_started":1,"wal-append.queue_time_us":446785}
I20260709 12:29:37.767570  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.768160  5559 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 36.
I20260709 12:29:37.672127 17104 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.768857  6274 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:37.769006  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.769536  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:37.769724  5559 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 37.
I20260709 12:29:37.660279 17199 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:37.674896 17224 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:37.687790 17220 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:37.773447  5526 leader_election.cc:400] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:37.734647 17216 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:37.687497 17204 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:37.777482  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:37.678992 17223 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:37.779636  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 90816.2us
I20260709 12:29:37.803133  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:37.804939  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:37.805958 17179 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:37.806306 17169 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 34.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:29:37.806649  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:37.807111 17169 consensus_queue.cc:1243] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 36.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:29:37.807173 17179 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:37.807727 17179 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:37.807754  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:37.808506 17070 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:37.808977 17070 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:37.808822 17197 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:29:37.669812 16788 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.108s
I20260709 12:29:37.810052 17070 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.25, Last appended by leader: 25, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.810858 17197 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:29:37.638652 17178 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.812386 16866 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.813917 17178 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:37.788992  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59856.9us
I20260709 12:29:37.777233 17052 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:37.815027 17052 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:37.815181  6237 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:37.815452 17052 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:37.816529 17266 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:37.817107  5627 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 40. Current term is 41. Ops: []
I20260709 12:29:37.817191  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:37.818634 17193 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:37.819043 17193 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:37.819670  5556 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: []
W20260709 12:29:37.820653  6305 heartbeater.cc:646] Failed to heartbeat to 127.4.65.62:38921 (8 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.4.65.62:38921 timed out after 15.000s (SENT)
I20260709 12:29:37.796464  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.672554 17123 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:37.822404  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.822603 17123 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.823176  5556 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:37.784442  6273 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:29:37.824803 17123 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.787285  5559 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:37.737105 17268 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:37.758225 17287 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:37.826844 17287 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:37.828176  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.828958  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.802529  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:37.830709  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.806030 17173 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:37.832624 17061 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:37.832976  5630 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:37.833142 17061 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:37.833592 17173 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:29:37.834260 17204 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:37.819585 17193 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:37.834728 17204 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:37.835305 17173 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:37.835929 17173 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:37.813843  5629 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:37.851574  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.820535 17224 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:37.822947  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:37.794540 17199 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.796778  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.811316 17070 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:37.852260 17234 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:37.859434  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.853061 17178 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:37.837574  5634 raft_consensus.cc:1240] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 27. Current term is 28. Ops: []
I20260709 12:29:37.860409 17178 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:37.860744  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.861012 17178 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:37.861368  5634 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:37.771981 17120 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:37.861876 17120 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:37.862288 17120 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:37.808260 17179 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:37.840960  5556 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 41 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 40. Current term is 41. Ops: []
I20260709 12:29:37.833549 17061 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:37.794384 17220 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:37.841744 17184 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.842823 17249 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:37.843160 16861 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.844974  5631 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:37.846371 17216 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.878598 16861 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:37.879868 17216 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:37.794584  5526 leader_election.cc:403] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:29:37.800427  5633 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:37.848295  5630 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:37.774094  5704 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:37.849960  5561 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:37.819528  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:37.852965  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.870002  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:37.841477  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:37.872393 17215 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.841s	user 0.006s	sys 0.021s
W20260709 12:29:37.884950  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 43611.5us
W20260709 12:29:37.893662  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 127662us
I20260709 12:29:37.895088  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:37.915337  5629 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:37.918165  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:37.873236 17220 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:37.951783  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:37.951956  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:37.951375 16988 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:37.865418 17226 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.052s
I20260709 12:29:37.921873  5628 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: []
W20260709 12:29:37.892663 17215 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.869s	user 0.006s	sys 0.021s
I20260709 12:29:37.939175  5704 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:37.828145 17287 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:37.965456  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47338.3us
I20260709 12:29:37.967926  5704 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:37.973141  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:37.974810  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:37.975077  5703 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:37.975605  5703 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:37.920221 17037 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:29:37.973650  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.986500  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.987797  5554 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 36: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:37.983052  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.994015  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.995352  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:37.980726  5703 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.866226 17090 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.998907  5703 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:38.000007  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:37.997124  5704 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.004604  5704 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:37.896183 17224 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:37.883708 17249 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.021075  5554 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:38.021894 16988 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:38.023494 17193 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:37.969664 17237 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:38.025449 17184 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:38.041106  5560 raft_consensus.cc:1275] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 37 index: 20. (index mismatch)
I20260709 12:29:37.982432 17205 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:37.971690 17183 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:38.036203  5554 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36. Ops: []
W20260709 12:29:38.047456  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:37.998121 17196 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:29:38.048239  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:38.029464  5704 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
W20260709 12:29:38.030470 17275 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:29:38.010973 17158 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:29:38.062937  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:38.076200  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:38.085814  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:38.082456 17037 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:38.061867 17090 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:38.059439 17183 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:38.082715 17231 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:38.098472  5557 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:29:38.111239  5703 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:29:38.112201  6275 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 35 index: 25. Preceding OpId from leader: term: 36 index: 25. (term mismatch)
I20260709 12:29:38.112756  6275 pending_rounds.cc:85] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 24
I20260709 12:29:38.113047  6275 pending_rounds.cc:107] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 35.25
I20260709 12:29:38.113335  6275 raft_consensus.cc:2889] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:38.115309 17223 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:29:38.066552 17205 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.093425 17183 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:38.082260 17191 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 40.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.035s
I20260709 12:29:38.135032  5559 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:38.091190 17226 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.041s
I20260709 12:29:38.101097 17231 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:38.150643  5555 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:38.151572  5559 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 36. Current term is 37. Ops: []
I20260709 12:29:38.152590 17118 raft_consensus.cc:3055] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:38.158313 17205 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:38.118815 17102 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:38.139845 17038 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 34.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.010s
I20260709 12:29:38.088850 17246 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:38.093349 17204 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.172323  5559 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34. Ops: []
W20260709 12:29:38.152246 17291 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 12:29:38.170149 17118 raft_consensus.cc:740] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:38.146641 17072 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:38.140321 17233 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 35.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.036s
I20260709 12:29:38.187826 17246 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:38.172679 17199 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:38.188381 17120 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.130460 17037 consensus_queue.cc:1059] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.041s
I20260709 12:29:38.170542 17231 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:38.142081 17216 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.033s
I20260709 12:29:38.171365 17266 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:38.146634 17197 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.035s
I20260709 12:29:38.169581  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:38.190069 17169 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:38.199780 17052 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:38.225376 17169 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:38.227131 17052 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:38.227557 17169 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:38.193187 17072 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:38.229333  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 254610us
I20260709 12:29:38.230542  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:38.232697  5559 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:38.234663  5558 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:29:38.209940  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 232554us
I20260709 12:29:38.253371  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:38.258845  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:38.310389  5556 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:38.320502  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:38.322396  6274 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
W20260709 12:29:38.313580  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60369.6us
I20260709 12:29:38.326906  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:38.319742 17249 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:38.311702 17253 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:38.313869  6275 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:29:38.331353  5557 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:38.263835 17069 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:38.336169  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1771) took 1556 ms. Trace:
I20260709 12:29:38.215073 17179 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:38.318480  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:38.279676  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:38.336668  6277 rpcz_store.cc:276] 0709 12:29:36.773321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:36.773473 (+   152us) service_pool.cc:224] Handling call
0709 12:29:36.773812 (+   339us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:36.780047 (+  6235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:36.780078 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:36.789374 (+  9296us) log.cc:844] Serialized 52 byte log entry
0709 12:29:37.544701 (+755327us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b18003f5bc0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:29:37.968329 (+423628us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:29:38.245492 (+277163us) log.cc:844] Serialized 14 byte log entry
0709 12:29:38.256068 (+ 10576us) log.cc:844] Serialized 14 byte log entry
0709 12:29:38.256458 (+   390us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:38.256637 (+   179us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:38.329373 (+ 72736us) raft_consensus.cc:1685] finished
0709 12:29:38.329418 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:38.329614 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5818,"prepare.queue_time_us":192490,"prepare.run_cpu_time_us":156,"prepare.run_wall_time_us":10467,"raft.queue_time_us":6653,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":61749,"spinlock_wait_cycles":141096192,"thread_start_us":670990,"threads_started":2,"wal-append.queue_time_us":1282423}
I20260709 12:29:38.325380 17193 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:38.349915  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70336.8us
I20260709 12:29:38.325879 17158 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:38.229481 17266 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:38.334928 17253 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:38.333385 16995 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:38.360075  5705 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: [32.20-32.20]
W20260709 12:29:38.368311  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 225857us
I20260709 12:29:38.355497 17229 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:38.377115  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:38.381966  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 55168.7us
W20260709 12:29:38.381956  5632 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.019s	sys 0.034s
W20260709 12:29:38.383627  5632 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.019s	sys 0.034s
I20260709 12:29:38.365190 17249 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:38.374058 17191 consensus_queue.cc:1243] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 41.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20260709 12:29:38.390859  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:38.406893  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:38.407812  5631 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:38.410064  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:38.400388  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:38.418190  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:38.419754  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:38.422984  5707 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 29: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:38.420488  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:38.425165  5631 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:29:38.384312 17246 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:38.395838 17061 consensus_meta.cc:287] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.525s	user 0.016s	sys 0.005s
I20260709 12:29:38.366616 16995 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:38.429986  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:38.431102  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:38.428942  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:38.442408  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:38.434396  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:38.434883  5628 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:29:38.383471 17301 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:38.444492  5706 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 40 LEADER]: Stepping down as leader of term 40
I20260709 12:29:38.396916 17191 raft_consensus.cc:3055] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:29:38.394704 16755 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:38.441875  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:38.447250  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:38.435690  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:38.446058  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:38.443940  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:38.437749  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:38.449651  5706 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 40 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:38.398320 17233 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:38.452436 17191 raft_consensus.cc:740] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 27 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:38.454418  5703 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:38.454501 17116 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:38.455585 17233 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.457046 17191 consensus_queue.cc:260] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:38.437454 17158 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:29:38.450645 17297 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260709 12:29:38.483338  5703 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:38.449961 17115 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:38.445937 17057 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:38.485210  5599 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:38.395295 17249 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.477876  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:38.489850  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:38.449829 17303 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:38.451752 17310 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:38.502529  5627 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:38.440712 17196 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.046s
I20260709 12:29:38.459493 17120 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:38.505256  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:38.440452 17246 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.510496  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:38.481640 17179 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:38.440852 17061 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:38.497092 17295 log.cc:927] Time spent T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
I20260709 12:29:38.521484  5627 raft_consensus.cc:1240] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 28. Current term is 29. Ops: []
W20260709 12:29:38.526237  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 84489.6us
I20260709 12:29:38.526813  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:38.528038  5559 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:38.495396 17116 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:38.496897 16755 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:38.502903 17229 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:38.497970 17072 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:38.503046 17115 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:38.539052 17115 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.535132  5633 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:29:38.522872 17057 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:29:38.537647 17299 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.146s	user 0.000s	sys 0.000s
I20260709 12:29:38.537607 17229 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:38.541831 17057 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:29:38.542073 17229 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.17, Last appended by leader: 17, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.518833 17303 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:38.543246 17057 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:29:38.532727 16755 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:38.544622 16755 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:38.545810  5635 raft_consensus.cc:1240] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:38.547067  5635 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: [31.21-31.21]
W20260709 12:29:38.547883 17306 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:29:38.549845  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:38.552574 17038 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:38.552692 17179 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:38.555512  5634 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:38.532369 17120 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:38.557948  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:38.553097 17216 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.006s
I20260709 12:29:38.564687 17266 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:38.560101 17120 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:38.569370 17294 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.397s	user 0.004s	sys 0.000s
I20260709 12:29:38.568991 16755 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.571815 17233 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:38.518237 17313 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:38.531970 17191 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:38.562530  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:38.557188 17178 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.696s	user 0.013s	sys 0.002s
I20260709 12:29:38.541906 17061 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:38.578910  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:38.610438 17178 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:38.611228  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:38.578233  5635 raft_consensus.cc:2368] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 34. Current term is 35.
I20260709 12:29:38.567452 17118 consensus_queue.cc:260] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.573757 17301 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:38.582757 17120 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.566900 17199 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:38.584769 17246 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:38.601045 17313 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:38.601876 17220 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.637s	user 0.011s	sys 0.004s
I20260709 12:29:38.604504  6276 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 21. (term mismatch)
I20260709 12:29:38.609596  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:38.634732  6276 pending_rounds.cc:85] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 20
I20260709 12:29:38.635058  6276 pending_rounds.cc:107] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 31.21
I20260709 12:29:38.625560  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:38.635313  6276 raft_consensus.cc:2889] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:38.626202 17234 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:38.628295  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 91348.3us
I20260709 12:29:38.586210 17231 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.664485 17178 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:38.665468 17313 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:38.665943 17037 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:38.666417 17037 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:38.667377 17231 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:38.667719 17037 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:38.633055  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:38.633627  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:38.672205  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:38.673964  5630 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:38.642141 16755 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:38.639670  5602 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:38.675288 17246 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:38.675770 17246 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:38.676596 17246 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.677330  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:38.647621  5701 raft_consensus.cc:1240] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:38.630337 17199 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:38.679540 17199 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.681725  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:38.652740 17234 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:38.653644 17183 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.508s	user 0.008s	sys 0.003s
I20260709 12:29:38.653882  5632 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 40. Current term is 41. Ops: []
I20260709 12:29:38.637122 17220 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:38.654543 17118 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:38.630930 17301 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:38.662920  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:38.738220  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:38.673633 17231 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:38.652009  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:38.649806 17308 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:38.714285  5634 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:38.706223 17037 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:38.741370 17199 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:38.741051 16755 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:38.793133 16755 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:38.741920  5709 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:38.743040  5559 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 41 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 40 index: 24. Preceding OpId from leader: term: 41 index: 25. (index mismatch)
I20260709 12:29:38.740820  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:38.743906  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:38.719293 17241 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:38.750164 17204 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:38.746057  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:38.733107 17183 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:38.752424 17298 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.414s	user 0.000s	sys 0.013s
I20260709 12:29:38.757623 16995 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:38.757822 17313 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:29:38.759094 17226 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Stepping down as leader of term 29
W20260709 12:29:38.802637  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61952.3us
I20260709 12:29:38.760373 17303 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:38.774160 17220 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:38.747520 17173 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:38.780789 17327 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:29:38.782790 17231 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:38.800573  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:38.741153 17308 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:38.757654  5675 leader_election.cc:400] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 34. Current term is 35.
I20260709 12:29:38.814152  5559 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
I20260709 12:29:38.814455  5675 leader_election.cc:403] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Cancelling election due to peer responding with higher term
I20260709 12:29:38.803619  5557 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:29:38.721609 17328 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.078s
I20260709 12:29:38.779788 17301 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:38.796298 16755 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:38.817282  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 59769.4us
I20260709 12:29:38.818302  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:38.820123  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:38.815864  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:38.808535 17320 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:29:38.824599  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:38.807436 17226 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:38.829399  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:38.833408  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:38.833338  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:38.816260 17197 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.008s	sys 0.005s
I20260709 12:29:38.844973  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:38.843549  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:38.844419  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:38.848667  5555 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 34. Ops: []
I20260709 12:29:38.849402  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:38.849602  5676 leader_election.cc:336] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.906s (SENT)
I20260709 12:29:38.850154  5561 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:38.850934  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:38.857549  5676 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:38.837117  5706 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 40.24, Last appended by leader: 24, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.861529  5706 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:29:38.861445  5559 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:38.866011  5553 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 31: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:38.870139 17312 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:38.871306 17312 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:38.871672 17312 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:38.823197 17233 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:38.876187  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:38.882169  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:38.884490  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:38.884619  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:38.885747 17234 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:38.839031 17316 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:38.891701  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:38.892745  5553 raft_consensus.cc:3060] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:38.901858 17183 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:38.902341  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:38.903017  5632 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:38.902920 17316 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:38.904750 17037 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:38.905802 17231 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:38.905889 17316 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.907166 17308 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:38.909086 17327 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:29:38.846617 17320 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:38.912472  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:38.913345  5632 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:38.913739  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:38.913563 17226 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.914225  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:38.914489  5526 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:38.914702 17226 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:29:38.918689  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55364.8us
I20260709 12:29:38.921065  5553 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:38.923926  5632 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
W20260709 12:29:38.862486 17197 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.009s	sys 0.006s
W20260709 12:29:38.927264  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:38.929113  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:38.932832  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:38.934273  5706 raft_consensus.cc:1275] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 40 index: 24. Preceding OpId from leader: term: 41 index: 25. (index mismatch)
I20260709 12:29:38.934325  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:38.926605 17120 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:38.949038  5706 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:38.948302  5703 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:38.951274  5706 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:38.951787  5703 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:38.939584 17061 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:38.953198  5703 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.954105  5631 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:38.943277 17179 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:38.932758 17317 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:38.963403  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:38.963581  5703 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:38.970013  5707 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:29:38.952010 17120 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:38.972527  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:38.973297  5553 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:38.975533 17120 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.23, Last appended by leader: 23, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:38.978461 17300 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.525s	user 0.001s	sys 0.001s
I20260709 12:29:38.980865  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4264) took 1507 ms. Trace:
I20260709 12:29:38.981364  5552 rpcz_store.cc:276] 0709 12:29:37.473113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:37.867950 (+394837us) service_pool.cc:224] Handling call
0709 12:29:37.875068 (+  7118us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:37.875611 (+   543us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:37.875655 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:37.919000 (+ 43345us) log.cc:844] Serialized 26 byte log entry
0709 12:29:38.210254 (+291254us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:29:38.210633 (+   379us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:38.211227 (+   594us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:38.977812 (+766585us) spinlock_profiling.cc:243] Waited 9.1 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:38.980048 (+  2236us) raft_consensus.cc:1685] finished
0709 12:29:38.980072 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:38.980335 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47844,"raft.queue_time_us":138,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":12249,"spinlock_wait_cycles":20921088,"thread_start_us":226321,"threads_started":1,"wal-append.queue_time_us":425132}
I20260709 12:29:38.925778 17312 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:38.986622  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:38.989275  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:38.990227  5703 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:38.992887  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:38.993085  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:38.961701 17197 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 31.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.024s
I20260709 12:29:38.997655  5553 raft_consensus.cc:2393] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 37: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:38.999032  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:39.002292  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:39.003528  5703 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:38.966058 17313 consensus_queue.cc:1048] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.025s
I20260709 12:29:39.005107  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:39.005764  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:39.006070  5703 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:39.006385  5524 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:39.007336 16755 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:38.990317 17118 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:39.008947  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:38.991007 17234 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:38.988631 17115 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:39.012791  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:39.012472  5602 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:38.971210 17332 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 35.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.041s
W20260709 12:29:38.994331 17178 consensus_queue.cc:1175] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cea66aa15be3423880843974acdbf50f" responder_term: 31 status { last_received { term: 31 index: 20 } last_committed_idx: 20 last_received_current_leader { term: 31 index: 20 } }
I20260709 12:29:39.023132 17249 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:39.025238 17070 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:39.026222 17070 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.026548 17090 raft_consensus.cc:2804] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:39.027534 17090 raft_consensus.cc:697] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:39.008736 17221 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:39.028853 17090 consensus_queue.cc:237] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:39.029968  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:39.002645 17264 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.021s
I20260709 12:29:39.031417  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:39.032176  6280 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:39.011556 17038 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.023s
I20260709 12:29:39.001596 17229 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:39.008339 17204 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.008s
I20260709 12:29:39.033707  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1782) took 1408 ms. Trace:
I20260709 12:29:39.034756  5627 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:39.034188  6279 rpcz_store.cc:276] 0709 12:29:37.625788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:37.625976 (+   188us) service_pool.cc:224] Handling call
0709 12:29:37.626357 (+   381us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:37.660727 (+ 34370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:37.660758 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:37.762847 (+102089us) log.cc:844] Serialized 52 byte log entry
0709 12:29:38.101687 (+338840us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:29:38.353154 (+251467us) log.cc:844] Serialized 14 byte log entry
0709 12:29:38.582786 (+229632us) spinlock_profiling.cc:243] Waited 2.75 ms on lock 0x7b18003c65e0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff74ffc0a 00007f5000c3f219 00007f4ff7587a40 00007f4ff75824e4 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:29:38.583009 (+   223us) log.cc:844] Serialized 14 byte log entry
0709 12:29:39.033267 (+450258us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:39.033349 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:39.033469 (+   120us) raft_consensus.cc:1685] finished
0709 12:29:39.033484 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:39.033644 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1165490,"prepare.queue_time_us":287118,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":62,"raft.queue_time_us":72462,"raft.run_cpu_time_us":376,"raft.run_wall_time_us":43105,"spinlock_wait_cycles":6315264,"thread_start_us":517401,"threads_started":2,"wal-append.queue_time_us":467232}
I20260709 12:29:39.016215 17115 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.017190 17199 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:39.047659  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:39.018522 17191 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:39.025502  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:39.015000  5602 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:39.001502 17220 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:38.959568 17320 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:39.038388  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:39.043756  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:39.009301 17312 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:39.045554  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:29:39.017441 17333 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
W20260709 12:29:39.050247  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 649520us
I20260709 12:29:39.085177  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:39.054013 17179 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:39.078982 17180 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:39.105405 17180 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:39.105825 17180 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:39.062487 17191 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:39.109581 17191 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:39.065448  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:39.121323  6280 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:39.063973  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:39.069571  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 54653.3us
I20260709 12:29:39.125242  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:39.007969 17241 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.044s
I20260709 12:29:39.126655 17221 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:39.127067 17221 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:39.127457 17221 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:39.061870 17070 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:39.047463 17199 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:39.071121 17246 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:39.128526 17199 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:29:39.093807  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:39.080854 16988 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:39.095963 17169 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:39.143579  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:39.098234 17226 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:29:39.100803 17337 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.105s	user 0.004s	sys 0.000s
I20260709 12:29:39.013469 17118 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:39.147830  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 84021us
I20260709 12:29:39.150202  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:39.152289  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:39.145355  6279 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:29:39.165033  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:39.165735  6272 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
W20260709 12:29:39.142918 17319 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:29:39.131418 17193 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:39.150944 17308 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:39.201574  6237 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:39.205865  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:39.210343  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:39.212251  5705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3945) took 1409 ms. Trace:
I20260709 12:29:39.212818  5705 rpcz_store.cc:276] 0709 12:29:37.802713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:37.839091 (+ 36378us) service_pool.cc:224] Handling call
0709 12:29:37.839486 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:38.360729 (+521243us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:39.211946 (+851217us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x7b5400323054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:39.212179 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1192950016}
I20260709 12:29:39.215302  6276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1783) took 1589 ms. Trace:
I20260709 12:29:39.215691  6276 rpcz_store.cc:276] 0709 12:29:37.627461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:37.627614 (+   153us) service_pool.cc:224] Handling call
0709 12:29:37.627997 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:38.635989 (+1007992us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:39.211656 (+575667us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b54002a8d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:39.215226 (+  3570us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2244647168}
I20260709 12:29:39.187042 17191 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:39.188999 17069 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:39.207329 17287 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.981s	user 0.009s	sys 0.007s
I20260709 12:29:39.155745 17268 consensus_queue.cc:1059] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.060s
I20260709 12:29:39.239771  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:39.198503 17193 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:39.225476 17070 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:39.253358 17072 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:39.253988 17069 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.154301 16988 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:39.191871 17266 consensus_queue.cc:1059] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.054s
I20260709 12:29:39.285252  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4274) took 1804 ms. Trace:
I20260709 12:29:39.250694  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:39.192116 17253 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.054s
W20260709 12:29:39.245116 17287 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.019s	user 0.009s	sys 0.008s
I20260709 12:29:39.290319  6275 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 37.
I20260709 12:29:39.241405 17233 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election lost for term 34. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 34. Current term is 35.
I20260709 12:29:39.285967  5560 rpcz_store.cc:276] 0709 12:29:37.480281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:38.048499 (+568218us) service_pool.cc:224] Handling call
0709 12:29:38.048932 (+   433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:38.064142 (+ 15210us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:38.064190 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:38.064834 (+   644us) log.cc:844] Serialized 26 byte log entry
0709 12:29:38.363673 (+298839us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:38.383508 (+ 19835us) log.cc:844] Serialized 14 byte log entry
0709 12:29:38.389588 (+  6080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:38.833486 (+443898us) spinlock_profiling.cc:243] Waited 4 ms on lock 0x7b180048bce0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419
0709 12:29:39.284799 (+451313us) spinlock_profiling.cc:243] Waited 3.25 ms on lock 0x7b5400227ed4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:39.284888 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:39.284962 (+    74us) raft_consensus.cc:1685] finished
0709 12:29:39.284984 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:39.285164 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46198,"prepare.queue_time_us":22023,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":53,"raft.queue_time_us":16370,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":5366,"spinlock_wait_cycles":16675584,"thread_start_us":267696,"threads_started":1,"wal-append.queue_time_us":408687}
I20260709 12:29:39.283427 17193 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:39.347182  5636 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:39.293413 17115 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:39.294292 17220 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:39.300307 17183 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.040s
W20260709 12:29:39.376168  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 332576us
I20260709 12:29:39.245445 17315 consensus_queue.cc:1059] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 35.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.137s
I20260709 12:29:39.391512  5704 raft_consensus.cc:2468] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
W20260709 12:29:39.397877  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 382903us
I20260709 12:29:39.389497 16788 raft_consensus.cc:3055] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:39.453326  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:39.454859  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:39.420719  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 219207us
I20260709 12:29:39.370921 17268 raft_consensus.cc:3055] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:39.481925  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:39.490095  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 196375us
I20260709 12:29:39.505482  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:39.461350 16788 raft_consensus.cc:740] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:29:39.509935 17336 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.513s	user 0.000s	sys 0.002s
I20260709 12:29:39.510219 16788 consensus_queue.cc:260] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.462328 17313 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:39.511899 17313 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.514565 17313 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:39.515933  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:39.517062  5707 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:29:39.393218 17220 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:39.527019  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 308782us
I20260709 12:29:39.516049  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:39.509001  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 290694us
W20260709 12:29:39.534754  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 252759us
I20260709 12:29:39.536914  5707 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:39.470439 17268 raft_consensus.cc:740] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:39.539530  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:39.535945  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:39.544727  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:39.550072  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:39.529204 17118 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:39.530982 17120 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:39.532212 17072 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:29:39.599517  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:39.612682  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:39.565960 17350 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260709 12:29:39.545739 17342 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:39.552789 17226 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:29:39.566555 17343 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:29:39.564263 17220 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:39.581105 17241 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:39.561355 17356 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:39.634788 17061 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.675s	user 0.012s	sys 0.013s
I20260709 12:29:39.650098  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:39.642109 17334 log.cc:927] Time spent T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.648s	user 0.003s	sys 0.000s
W20260709 12:29:39.654104  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 187652us
I20260709 12:29:39.659744  5707 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 28.
I20260709 12:29:39.645447 17069 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:39.578066 17180 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.572600 17120 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:39.683265  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 131037us
I20260709 12:29:39.692863  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:39.659065 17220 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:39.677443 17061 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:29:39.709096 17221 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.581s	user 0.017s	sys 0.002s
I20260709 12:29:39.709626 17221 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.711884 17221 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:39.713835 17366 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.009s
I20260709 12:29:39.673018 17241 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:39.721513  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 171973us
I20260709 12:29:39.691767 17363 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:29:39.719123  5705 raft_consensus.cc:1240] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:39.695930 16988 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:39.729130  6234 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:39.707752 17179 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.604s	user 0.011s	sys 0.004s
W20260709 12:29:39.731030  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:39.732313  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 195285us
I20260709 12:29:39.656853 17268 consensus_queue.cc:260] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.683883 17356 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.701707 17220 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:39.718992 17221 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:39.776563  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:39.732081 17179 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:39.733827  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:39.734943  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:39.772578  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 69251us
I20260709 12:29:39.744228 17320 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:39.775626  5631 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:29:39.778008  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:39.790429 17320 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:39.791955 17233 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:39.792063  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:39.792462 17233 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:39.792657 17320 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:39.792961 17233 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:39.793519 17314 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:39.793783  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:39.773769 17360 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:29:39.728384 17072 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.776427  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:39.778038  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:39.797006  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:39.797253 17361 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:39.793783  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:39.798352 17216 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:39.798506 17361 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:39.798985 17361 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:39.799350 17361 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:39.799599  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:39.785434  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:39.801291  5634 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 28: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:39.796293 17266 raft_consensus.cc:3055] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Stepping down as leader of term 35
W20260709 12:29:39.802366 17102 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 41 current term: 41 is higher.
I20260709 12:29:39.802444 17266 raft_consensus.cc:740] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:39.804280  5627 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: []
W20260709 12:29:39.805300  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70609.3us
I20260709 12:29:39.806032  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:39.806960  5635 raft_consensus.cc:2393] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:29:39.809706  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 75975.1us
I20260709 12:29:39.810784  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:39.833550  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:39.833910  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:39.833366  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:39.803503 17070 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:39.834982  6277 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 27.
I20260709 12:29:39.775048 17355 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:39.836301 17070 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:39.842098 17180 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:39.843341 17370 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:29:39.844772 17371 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:39.845471 17371 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.847750 17371 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:39.841143  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:39.852419 17241 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:39.853015 17379 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:39.854141 17070 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:39.855207 17199 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.726s	user 0.005s	sys 0.008s
I20260709 12:29:39.855279 17361 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.856643 17070 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:39.857514 17365 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:29:39.858052 17220 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
W20260709 12:29:39.858418 17312 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.775s	user 0.008s	sys 0.004s
I20260709 12:29:39.803519 17266 consensus_queue.cc:260] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.813974 17303 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:39.865656  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:39.815062 17102 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:39.866858 17102 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:39.865814 17266 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:39.867307 17102 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:39.837662  5601 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:39.783011 17193 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.832464 17362 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:39.873205 17362 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:39.874534 17193 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:39.875352 17362 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:39.875284 17193 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:29:39.875705 17191 consensus_meta.cc:287] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.626s	user 0.009s	sys 0.000s
I20260709 12:29:39.876538 17377 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:39.876971 17377 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:39.877278 17356 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:39.808197 17376 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:39.878105 17287 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:39.877938 17377 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.878638 17287 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:39.879091 17287 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:39.879071 17355 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.879487 17377 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:39.881536 17355 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:39.882022 17220 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:39.882624 17220 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:39.883225  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:39.858104 17365 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:39.884778  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:39.885632 17328 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:39.886075 17328 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:39.886457 17328 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:39.866441  5707 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:39.885277 17365 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:29:39.890295  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:39.891788 17220 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:39.855758 17199 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:39.894847 17308 consensus_meta.cc:287] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.679s	user 0.009s	sys 0.003s
I20260709 12:29:39.895347 17199 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:39.895750  5709 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
W20260709 12:29:39.896461  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 58908.1us
I20260709 12:29:39.897601 17372 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:39.898178 17372 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.901078  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:39.901394  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:39.901779  6273 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:39.903679  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:39.905211  5631 raft_consensus.cc:1240] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 36. Current term is 37. Ops: []
I20260709 12:29:39.906551  6273 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 37. Current term is 38. Ops: []
I20260709 12:29:39.907583  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:39.907799  5632 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
W20260709 12:29:39.909097  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 68074.6us
I20260709 12:29:39.910303 17362 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:39.910339  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:39.910859 17362 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:39.911285 17195 consensus_queue.cc:1059] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:39.911911 17362 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:39.916798 17372 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:39.918318  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49030 (request call id 4295) took 1085 ms. Trace:
I20260709 12:29:39.918924  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:39.918727  5557 rpcz_store.cc:276] 0709 12:29:38.832546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:38.833067 (+   521us) service_pool.cc:224] Handling call
0709 12:29:39.918117 (+1085050us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b540016ba54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:39.918258 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1999806720}
I20260709 12:29:39.920737 17120 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:39.922222  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:39.923241  5701 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:39.911372 17249 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:39.908133  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:39.934067  5629 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:29:39.940562  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:39.942885  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:39.943462  5632 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
I20260709 12:29:39.930645  5708 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:39.945513  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:39.935546 17224 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:39.968014 17224 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:39.970273 17224 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:39.946470 16995 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:39.934669 17237 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.037s
I20260709 12:29:39.971776  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:39.972206  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:39.971833 16995 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:39.947501 17287 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.974761 16995 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:39.950991 17179 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:39.975920  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:39.976378  5701 raft_consensus.cc:2393] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 33: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:39.976169 17179 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.963322 17355 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:39.948920 17308 consensus_queue.cc:1059] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 37.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.019s
I20260709 12:29:39.948549 17226 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:39.977349  5709 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:29:39.977560 17226 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:39.978086 17226 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:39.978325 17179 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:39.952457 17312 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 31.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.027s
I20260709 12:29:39.979462 17312 consensus_queue.cc:1243] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 32.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:29:39.976538  5707 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:39.983443 17328 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.962162 17361 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:39.987208 16788 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:39.989176  6275 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 34. Ops: []
I20260709 12:29:39.990516  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:39.991297  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3880) took 2197 ms. Trace:
I20260709 12:29:39.991384 17355 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:39.992543 17355 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.993300  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:39.944485 17233 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:39.995673  6273 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:39.995831  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:39.996042  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:39.991768  5636 rpcz_store.cc:276] 0709 12:29:37.794094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:37.794245 (+   151us) service_pool.cc:224] Handling call
0709 12:29:37.794592 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:39.365523 (+1570931us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:39.991079 (+625556us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54000c2954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:39.991242 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3569938688}
I20260709 12:29:39.997021  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:39.996935  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:39.999121  5629 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:39.999169  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:39.999300  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:39.998309  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:40.003448  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:40.004546  5707 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:40.005329  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:40.013260  5632 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:40.018986  6277 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 37.
I20260709 12:29:40.021478  5633 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:40.021737  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:40.024413  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:40.025451 17180 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:40.027912 17268 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:40.028798 17357 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.429s	user 0.000s	sys 0.002s
I20260709 12:29:40.030272  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 3959) took 1046 ms. Trace:
I20260709 12:29:40.030721  5703 rpcz_store.cc:276] 0709 12:29:38.983651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:39.009873 (+ 26222us) service_pool.cc:224] Handling call
0709 12:29:39.010145 (+   272us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:39.148532 (+138387us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:39.148565 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:39.168344 (+ 19779us) log.cc:844] Serialized 26 byte log entry
0709 12:29:39.399232 (+230888us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:39.399280 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:39.399377 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:40.028127 (+628750us) spinlock_profiling.cc:243] Waited 35.4 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:40.029906 (+  1779us) raft_consensus.cc:1685] finished
0709 12:29:40.029935 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:40.030203 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31102,"raft.queue_time_us":3973,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":87,"spinlock_wait_cycles":81407872,"thread_start_us":299625,"threads_started":2,"wal-append.queue_time_us":316486}
I20260709 12:29:40.033111  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:40.004866 17351 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.302s	user 0.001s	sys 0.000s
I20260709 12:29:40.046042  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:40.047662  5629 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:39.985391 17287 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:40.022696 17317 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:29:40.058892  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:40.003012 17361 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:40.060784  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:40.057758  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:40.065999  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:40.049729 17376 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:40.073594  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:40.055236 17234 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:40.060487 17233 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:40.076736 17234 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.062502 17316 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.015s
I20260709 12:29:40.062207 16995 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.015s
I20260709 12:29:40.074414 17317 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:40.078673 17317 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.079160 17234 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:40.081275 17311 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:40.081732 17311 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:40.082595 17311 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.083623 17173 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:40.083956 17311 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:40.030157 17365 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.086324 17233 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:40.100715 17233 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.102044 17365 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:40.102722  6271 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:40.102922 17233 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:40.104946  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:40.108892  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:40.109347  5676 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:40.109956  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:40.110262 17380 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:29:40.111362  5599 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:40.111778 17380 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:40.112221 17368 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:40.113624 17368 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:40.114048 17368 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:40.115280  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:40.116536 17264 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:29:40.072327 17301 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:40.117090 17264 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:40.117245  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:40.117991 17264 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.123900  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4468) took 1250 ms. Trace:
I20260709 12:29:40.128700  5704 rpcz_store.cc:276] 0709 12:29:38.873947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:38.913512 (+ 39565us) service_pool.cc:224] Handling call
0709 12:29:40.123718 (+1210206us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b54001991d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:40.123838 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":982957312}
I20260709 12:29:40.130860 17346 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:40.090773 17362 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:40.131474 17374 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.385s	user 0.000s	sys 0.003s
I20260709 12:29:40.133050 17362 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:40.134963 17340 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.038s	user 0.002s	sys 0.002s
I20260709 12:29:40.089960  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:40.060253 17115 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.036s
I20260709 12:29:40.048791 17366 consensus_queue.cc:1048] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.028s
I20260709 12:29:40.092417  6272 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 36. Current term is 37. Ops: []
I20260709 12:29:40.096798 17371 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:40.141866 17371 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:40.142323 17371 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:40.143141 17372 raft_consensus.cc:3055] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:29:40.143567 17372 raft_consensus.cc:740] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:40.143746  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:40.144492 17372 consensus_queue.cc:260] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.145306  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4466) took 1271 ms. Trace:
I20260709 12:29:40.145593  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:37.981297  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:29:40.146981  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:40.147294  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:40.147995  5704 raft_consensus.cc:2393] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 33: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:40.145656  5708 rpcz_store.cc:276] 0709 12:29:38.873459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:38.913995 (+ 40536us) service_pool.cc:224] Handling call
0709 12:29:40.145156 (+1231161us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b5400451ad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:40.145253 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2171590400}
I20260709 12:29:40.149523  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:40.112586 17241 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:40.151659  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 42 candidate_status { last_received { term: 40 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:40.151856 17241 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:40.152292  5701 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 41.
I20260709 12:29:40.065614 17361 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.152745 17233 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:40.153194  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:40.152839 17241 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.153518 17233 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:29:40.113015 17220 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:29:40.154311 17361 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:40.154371  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:40.145740 17372 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:40.118392  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:40.054867 17120 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.042s
I20260709 12:29:40.156155  5708 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:40.157407  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:40.158890  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:40.088551  5706 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.16, Last appended by leader: 16, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.153788  5701 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:40.164335  5706 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:40.168665 17310 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:40.168849  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:40.169304 17310 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.171504  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:40.081486 17317 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:40.154340 17241 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:40.185294  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:40.187086  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:40.188093  5704 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:40.190718  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:40.189652  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:40.192476  5708 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:40.195216  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:40.193552  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 120056us
I20260709 12:29:40.196379  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:40.196671  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:40.196404  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:40.197887  5560 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:40.198562  5554 raft_consensus.cc:2368] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 37. Current term is 38.
I20260709 12:29:40.198859  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:40.201306  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:40.204141  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:40.205020  6271 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 29.
I20260709 12:29:40.205428  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:40.205624  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:40.206585  5708 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:40.210209  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:40.154593 17233 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:29:40.217911  5631 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:40.213629  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:40.220227  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:40.112195 17380 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:40.072633 17287 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:29:40.235852  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:40.237763  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:40.211509 16755 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:40.249387 16755 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:40.249758 16755 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:40.214741 17237 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.036s
I20260709 12:29:40.251359 17237 consensus_queue.cc:1243] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 35.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:29:40.251854 17237 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:40.242962 17380 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:40.253276  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:40.252184 17237 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:40.261305 17237 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.262589 17237 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:40.265859  5557 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:40.131476 17328 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:40.269457 17310 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:40.268903 17226 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.270416  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:40.271536  5709 raft_consensus.cc:2368] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30.
I20260709 12:29:40.271636 17226 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:40.212471  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:40.275784  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:40.276405  6274 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:40.276799  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:40.268981  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4467) took 1395 ms. Trace:
W20260709 12:29:40.281730  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 45993.7us
I20260709 12:29:40.238471  5707 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 17. Preceding OpId from leader: term: 35 index: 18. (index mismatch)
I20260709 12:29:40.241528  5708 raft_consensus.cc:2368] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 28. Current term is 29.
I20260709 12:29:40.223858  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:40.092195 17102 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.211301 17249 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:40.214684 17233 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 36 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:40.314955  5552 raft_consensus.cc:1275] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:29:40.278285  5706 rpcz_store.cc:276] 0709 12:29:38.873698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:38.943891 (+ 70193us) service_pool.cc:224] Handling call
0709 12:29:38.948908 (+  5017us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:40.088155 (+1139247us) spinlock_profiling.cc:243] Waited 918 ms on lock 0x7b5400040c20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5f006f 00007f4ffa668f60 00007f4ffa66a7de 00007f4ffa674b49 00007f4ffa675416 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62
0709 12:29:40.163531 (+ 75376us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b58008bdcd0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa5ce0aa 00007f4ffa650447 00007f4ffa668fe7 00007f4ffa66a7de 00007f4ffa674b49 00007f4ffa675416 00007f4ffa671d8f 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4
0709 12:29:40.264497 (+100966us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:40.264535 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:40.268694 (+  4159us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:29:40.268743 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:40.268810 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:40.268927 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1069,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":101,"spinlock_wait_cycles":2114904832}
I20260709 12:29:40.333382  5708 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:40.287353  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:40.352615  5627 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:40.334779  5707 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
W20260709 12:29:40.282546  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 70466.7us
I20260709 12:29:40.282145  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:40.299675  5557 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
W20260709 12:29:40.329938  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 106189us
W20260709 12:29:40.344645  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:40.346727  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:40.367170  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:40.368477  6234 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:40.370841  5676 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:40.274268 17057 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:40.378103  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:40.393028  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62918.9us
I20260709 12:29:40.287467 17328 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:40.381845  5560 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:40.274924 17377 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:40.379755  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:40.348930 17310 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
W20260709 12:29:40.427747  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 148829us
I20260709 12:29:40.367337 17224 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:40.375123 17069 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:40.368175 17287 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:40.388265 17379 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:29:40.433071  5627 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:40.342211 16995 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.056s
I20260709 12:29:40.349795 17320 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.059s
W20260709 12:29:40.400262 17384 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:29:40.383101 17268 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:29:40.459059  6234 leader_election.cc:400] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 28. Current term is 29.
I20260709 12:29:40.433918  6274 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 38.
I20260709 12:29:40.392777 17367 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:40.402215 17328 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:40.487107 17328 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:40.433598 17377 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:40.496620 17358 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:29:40.366268 17249 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.476665  6234 leader_election.cc:403] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:29:40.434445 17346 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:40.393167 17368 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.391993 17057 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.509249 17224 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:40.510238 17224 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.514830 17102 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:40.516925  5635 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37. Ops: [36.22-36.22]
I20260709 12:29:40.508903 17229 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:29:40.482786  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:40.552636 16995 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:40.553229 16995 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:40.554370 16995 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.550097 17226 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:40.555941 17226 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.550416 17178 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:40.559248 17178 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.564110 17368 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:40.565171 17362 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:40.566242 17362 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.462311 17233 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.595427 17312 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:40.594844 17287 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.22, Last appended by leader: 22, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.596007 17365 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 28. Current term is 29.
I20260709 12:29:40.556057 16995 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:40.474931 17310 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:40.612258  6280 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.051s	sys 0.078s
W20260709 12:29:40.615793  6280 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.051s	sys 0.078s
I20260709 12:29:40.617234  6280 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1795) took 1084 ms. Trace:
I20260709 12:29:40.628876 17317 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:40.629650 17317 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:40.630770 17317 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.571167 17301 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:29:40.650089  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 191189us
W20260709 12:29:40.654779  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 172190us
I20260709 12:29:40.658874  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:40.620592  6280 rpcz_store.cc:276] 0709 12:29:39.533003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:39.533105 (+   102us) service_pool.cc:224] Handling call
0709 12:29:39.533405 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:39.748280 (+214875us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:39.748329 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:39.778962 (+ 30633us) log.cc:844] Serialized 26 byte log entry
0709 12:29:40.616709 (+837747us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:40.616770 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:40.616883 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:40.616933 (+    50us) raft_consensus.cc:1685] finished
0709 12:29:40.616954 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:40.617158 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":192368,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":8504,"thread_start_us":909563,"threads_started":2,"wal-append.queue_time_us":554774}
W20260709 12:29:40.670133  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:40.619544 17220 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:40.678505  5633 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:29:40.596093 17312 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.653724 17379 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:40.618122 17373 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:40.689119  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 372273us
I20260709 12:29:40.673107 17057 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:40.675458 17377 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:40.690341  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:40.637706 17310 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.681728 17226 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:40.698797  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 279138us
I20260709 12:29:40.684242 17368 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:40.697369  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 340721us
I20260709 12:29:40.707728  5559 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:29:40.713193  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:40.720979 17365 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:29:40.722208  5636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:40.723182  5636 raft_consensus.cc:606] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:40.723747  5636 raft_consensus.cc:614] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:40.724759  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:40.709339 17057 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:40.683288 17220 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.692121 17215 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:40.767204  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55027us
I20260709 12:29:40.748114  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:40.753237  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:40.769970  5636 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:29:40.727687  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:40.771896  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44318.6us
I20260709 12:29:40.689538 17379 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:40.741128 17395 log.cc:927] Time spent T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
W20260709 12:29:40.742395 17311 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.642s	user 0.009s	sys 0.009s
I20260709 12:29:40.688966 17373 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.749388  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:40.747162 17249 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:40.717173 17368 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.781219  5627 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:40.786741 17312 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:40.754006 17396 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260709 12:29:40.786295 17320 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:40.805124  5702 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:40.788506 17226 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:40.764464 17371 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.622s	user 0.007s	sys 0.013s
W20260709 12:29:40.789812 17380 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.537s	user 0.008s	sys 0.008s
W20260709 12:29:40.791742 17372 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.636s	user 0.011s	sys 0.006s
W20260709 12:29:40.792312 16755 consensus_meta.cc:287] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.542s	user 0.008s	sys 0.008s
I20260709 12:29:40.792429 17346 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:40.749325 17362 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:40.792815 17215 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.795790 17301 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:29:40.834271 16755 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.803946 17402 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 33.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.028s
W20260709 12:29:40.847361 17173 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.755s	user 0.012s	sys 0.007s
I20260709 12:29:40.861351  5710 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:40.799937  6272 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.007s	sys 0.039s
I20260709 12:29:40.773881  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:29:40.867246  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 93495.3us
I20260709 12:29:40.804317  6274 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:40.744956 17102 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.045s
I20260709 12:29:40.827593 17226 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:40.852938 17037 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:29:40.835295 17301 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 41 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:40.825129 17371 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:40.879740 17195 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:40.832286 17380 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:40.902796 17380 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:29:40.893128 17195 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:40.928038 17407 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.055s	user 0.003s	sys 0.000s
I20260709 12:29:40.933657 17195 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:40.936683  6279 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:29:40.938072 17371 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:40.890991 17301 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:29:40.944278 17367 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:40.944862 17367 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:40.945950 17237 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.683s	user 0.030s	sys 0.000s
I20260709 12:29:40.947024 17367 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:40.948029 17368 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:40.948789 16755 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:40.948921 17220 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:40.949417 17328 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.952296 17328 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:40.947062 17226 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:40.950037 17379 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:40.955747 17178 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:40.964016 17233 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:29:40.972368 17411 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 12:29:40.973395 17405 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.098s	user 0.000s	sys 0.002s
I20260709 12:29:40.967713 17403 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:40.976747 17403 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:40.977545 17403 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 33.26, Last appended by leader: 26, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.978886 17403 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:40.983379 17310 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:40.970975 17373 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:40.984441 17342 raft_consensus.cc:3055] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:29:40.984795 17342 raft_consensus.cc:740] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 37 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:40.985653 17342 consensus_queue.cc:260] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.20, Last appended by leader: 20, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:40.986614 17342 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:29:40.874495  6272 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.007s	sys 0.040s
I20260709 12:29:40.972259 17215 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.014127 17195 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.016811 17195 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.064757  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:41.070363  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:41.074050  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:41.074612  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:41.140741  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70637.3us
I20260709 12:29:41.141292  5557 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:41.076905  5554 raft_consensus.cc:2368] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34.
I20260709 12:29:41.077334  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.077929  5561 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 36: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:41.078517  5558 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 35 index: 18. (term mismatch)
I20260709 12:29:41.147363  5558 pending_rounds.cc:85] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 17
I20260709 12:29:41.147833  5558 pending_rounds.cc:107] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 34.18
I20260709 12:29:41.148173  5558 raft_consensus.cc:2889] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:41.083588  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:41.148801  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 75235.6us
I20260709 12:29:41.150636  5526 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:41.092056  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:41.101495  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:41.164216  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:41.164932  5701 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:41.103703  5707 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:29:41.173913  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:41.174479  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:41.175194  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.175532  5707 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:41.103729  5706 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:41.176046  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:41.176929  6271 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:29:41.177002  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 43802.5us
I20260709 12:29:41.177609  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:41.179005  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.179375  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:41.180564  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48246.7us
I20260709 12:29:41.181010  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:41.099484  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 351301us
I20260709 12:29:41.183298  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.114941  5703 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30. Ops: []
W20260709 12:29:41.184818  5459 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 282) took 62523 ms (client timeout 15000 ms). Trace:
I20260709 12:29:41.185045  5550 tablet_service.cc:1558] Processing DeleteTablet for tablet 26edf338d41e42f1a5ce0d4416ec67e1 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 11)) from {username='slave'} at 127.0.0.1:49500
I20260709 12:29:41.115130  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:41.121699  5710 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:41.136106  5632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e20e319f46504838a20c558a66692da6"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:41.137923  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.188534  5629 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:41.189169  5709 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:29:41.190214  5507 auto_rebalancer.cc:288] Could not retrieve cluster info: Not found: tserver f613ba929b1046d3b5aed909414a95bf not available for placement
I20260709 12:29:41.190680 17204 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:41.193084 16788 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:41.194442 17215 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:41.194988 17069 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.738s	user 0.007s	sys 0.017s
I20260709 12:29:41.195684 17069 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:41.196729 17037 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:41.197566 17037 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:41.198179  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:41.200629 17061 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:41.201037 16995 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:41.201519 16995 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:41.201426 17061 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.205039 17057 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:41.205581 17115 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:41.205718 17184 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:29:41.206754 17120 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:41.202006 16995 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:41.208813  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:41.209614  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:41.195302 17215 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.225704  5706 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:41.225973  5632 raft_consensus.cc:606] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:41.233057  5632 raft_consensus.cc:614] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:41.234920 17253 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:41.235510 17253 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:41.236132 17061 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:41.236968 17253 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.238660 17253 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:41.208271 17115 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:29:41.242569  6274 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:41.244124  6272 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:41.019133 17301 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:41.249404  5710 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 2025) took 2829 ms (client timeout 2709 ms). Trace:
I20260709 12:29:41.249541 17226 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:41.249749  5701 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 37.
W20260709 12:29:41.249835  5710 rpcz_store.cc:269] 0709 12:29:38.419973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:38.420136 (+   163us) service_pool.cc:224] Handling call
0709 12:29:41.249147 (+2829011us) spinlock_profiling.cc:243] Waited 2.44 s on lock 0x7b5400309654. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:41.249304 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5604934400}
I20260709 12:29:41.250357 17226 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.251669  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:41.253144  5704 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:29:41.254880 17215 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.210561 17224 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:41.256073 17224 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.259347 17224 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.262256  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.262549  5627 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:29:41.263249  5629 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 33: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:41.268932 17037 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:41.269677  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:41.269899 17037 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:41.270758  5632 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 36: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:41.270704 17037 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.267266  6272 raft_consensus.cc:1240] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 37. Current term is 38. Ops: []
I20260709 12:29:41.273138 17115 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 38 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:41.273350  5707 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:41.277733 17115 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.279865  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:41.281040  5631 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 34
W20260709 12:29:41.282492  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 52279.6us
I20260709 12:29:41.191618 17204 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.305831  5709 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:41.250413 17301 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 42 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.308504 17037 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:41.308658  6234 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 2.709s (SENT) [suppressed 16 similar messages]
I20260709 12:29:41.313088  6271 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
W20260709 12:29:41.310817  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 66244.6us
I20260709 12:29:41.315342  5710 raft_consensus.cc:2393] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:41.316331  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:41.335489 17317 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:41.336621  5559 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.000s	sys 0.025s
W20260709 12:29:41.337379  5559 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.001s	sys 0.025s
I20260709 12:29:41.340762  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.341632  5553 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:41.342281 17204 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.342943 17204 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:41.323120 17069 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:41.344003 17069 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.344712 17413 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.011s
I20260709 12:29:41.345372 17409 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.019s
I20260709 12:29:41.346873 17069 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:41.348897  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:41.350257  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61548.8us
I20260709 12:29:41.350744  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:41.352804 17369 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 34.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:29:41.336522 17178 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:41.357298 17178 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.358479 17069 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:41.359256 17069 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:41.360255  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.361268  5561 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:41.343792 17204 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.365379  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.365582  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.366185  5553 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
I20260709 12:29:41.351967  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:41.367852 17204 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:41.368988 17379 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:41.370751 17057 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 35.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
W20260709 12:29:41.362658  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 78313.5us
I20260709 12:29:41.366356  5633 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
W20260709 12:29:41.399664  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47930.8us
I20260709 12:29:41.375950 16755 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:41.372972  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:41.361997  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:41.401216  6237 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:41.412186  5706 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:41.363086 17069 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:41.413416 17241 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:29:41.426801  5555 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.015s	sys 0.022s
W20260709 12:29:41.427325  5555 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.016s	sys 0.022s
I20260709 12:29:41.427913 17241 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:41.428524 17241 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:41.428740  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3920) took 1139 ms. Trace:
I20260709 12:29:41.429337  5555 rpcz_store.cc:276] 0709 12:29:40.288995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:40.398327 (+109332us) service_pool.cc:224] Handling call
0709 12:29:40.457506 (+ 59179us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:40.457750 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:29:40.804548 (+346798us) log.cc:844] Serialized 14 byte log entry
0709 12:29:41.428209 (+623661us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:41.428304 (+    95us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:41.428408 (+   104us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:41.428504 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:41.428676 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":397873,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"thread_start_us":954639,"threads_started":2,"wal-append.queue_time_us":549386}
W20260709 12:29:41.432773  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 82197us
I20260709 12:29:41.433202 17316 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:41.433899 17316 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.434485  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:41.437263  5455 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 179) took 137889 ms (client timeout 15000 ms). Trace:
I20260709 12:29:41.437568  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:41.437736  5455 rpcz_store.cc:269] 0709 12:27:23.547600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:23.547987 (+   387us) service_pool.cc:224] Handling call
0709 12:27:23.938189 (+390202us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5dc090 00007f4ffa67af3a 00007f5000430e9c 00007f500052f04e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc 00007f4ffe9ef392 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:41.437055 (+137498866us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":972344,"reported_tablets":32,"spinlock_wait_cycles":700304640}
I20260709 12:29:41.418941 17226 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:41.437613  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 42 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:41.447121  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:41.447703  5629 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:41.449411  5629 raft_consensus.cc:606] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:41.449849  5629 raft_consensus.cc:614] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:41.422688  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.451152  5632 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:41.436285 17316 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:41.458403  5676 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:41.456094  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 55148.2us
W20260709 12:29:41.470821  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70379.6us
I20260709 12:29:41.404307 17317 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.476029 17373 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:41.476282 17408 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:41.476605 17234 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:41.480921 17369 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:41.481390 17037 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:41.481525 17061 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:41.485426 17316 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:41.488593  5707 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:29:41.489555  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:41.490378  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 71988.5us
I20260709 12:29:41.447296 17413 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:41.494858  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.495127 17408 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:41.493741 17373 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:41.516577  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 208074us
I20260709 12:29:41.496261 17234 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:41.518191  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 80912.9us
I20260709 12:29:41.457389 17379 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.502290 17061 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:41.499861 17037 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:41.523636  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.525733  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.517776 17408 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:41.537462  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:41.522985 17234 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:41.543358  5561 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 34 because replica has last-logged OpId of term: 32 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 19.
I20260709 12:29:41.548898  5602 leader_election.cc:400] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 37. Current term is 38.
I20260709 12:29:41.549206 17301 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:29:41.550827 17310 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:41.551388  5446 catalog_manager.cc:5028] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): tablet 26edf338d41e42f1a5ce0d4416ec67e1 (table test-workload [id=53989103653948c0b4757ec009e63460]) successfully deleted
I20260709 12:29:41.517617 17373 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:41.523952 17037 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:41.524235 17061 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:41.592891  5446 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 76334.9us
I20260709 12:29:41.572049  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:41.185415  5459 rpcz_store.cc:269] 0709 12:28:38.660941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:28:49.793196 (+11132255us) service_pool.cc:224] Handling call
0709 12:28:50.488912 (+695716us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5dc090 00007f4ffa67af3a 00007f5000430e9c 00007f500052f04e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc 00007f4ffe9ef392 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:41.184546 (+50695634us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:29:35.744503 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:29:35.910077 (+165574us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:29:35.912557 (+  2480us) write_op.cc:435] Acquired schema lock
0709 12:29:35.921549 (+  8992us) tablet.cc:662] Decoding operations
0709 12:29:36.193888 (+272339us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:29:36.193955 (+    67us) write_op.cc:641] Partition lock acquired for write op
0709 12:29:36.193983 (+    28us) tablet.cc:685] Acquiring locks for 10 operations
0709 12:29:36.194828 (+   845us) tablet.cc:701] Row locks acquired
0709 12:29:36.194849 (+    21us) write_op.cc:260] PREPARE: finished
0709 12:29:36.195046 (+   197us) write_op.cc:270] Start()
0709 12:29:36.195187 (+   141us) write_op.cc:276] Timestamp: P: 1783600176194989 usec, L: 0
0709 12:29:36.195208 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 12:29:36.195901 (+   693us) log.cc:844] Serialized 3909 byte log entry
0709 12:29:36.875222 (+679321us) op_driver.cc:464] REPLICATION: finished
0709 12:29:37.170550 (+295328us) write_op.cc:301] APPLY: starting
0709 12:29:37.174140 (+  3590us) tablet.cc:1367] starting BulkCheckPresence
0709 12:29:37.241531 (+ 67391us) tablet.cc:1370] finished BulkCheckPresence
0709 12:29:37.241682 (+   151us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:29:37.579955 (+338273us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:29:37.580213 (+   258us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=20,key_file_lookups=20,delta_file_lookups=0,mrs_lookups=0
0709 12:29:37.580254 (+    41us) write_op.cc:312] APPLY: finished
0709 12:29:37.580570 (+   316us) log.cc:844] Serialized 97 byte log entry
0709 12:29:37.626283 (+ 45713us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:29:37.635778 (+  9495us) write_op.cc:454] Released schema lock
0709 12:29:37.651333 (+ 15555us) write_op.cc:341] FINISH: Updating metrics
0709 12:29:37.689510 (+ 38177us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:29:37.689549 (+    39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":936390,"reported_tablets":49,"spinlock_wait_cycles":25308032,"child_traces":[["op",{"apply.queue_time_us":271423,"apply.run_cpu_time_us":20313,"apply.run_wall_time_us":528447,"cfile_cache_hit":42,"cfile_cache_hit_bytes":45927,"num_ops":10,"prepare.queue_time_us":135767,"prepare.run_cpu_time_us":20405,"prepare.run_wall_time_us":564880,"raft.queue_time_us":248501,"raft.run_cpu_time_us":38201,"raft.run_wall_time_us":246857,"replication_time_us":663444,"thread_start_us":434139,"threads_started":4,"wal-append.queue_time_us":136825,"wal-append.run_cpu_time_us":21324,"wal-append.run_wall_time_us":2301722}]]}
I20260709 12:29:41.570410  5631 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:41.553285  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:29:41.598462  6234 leader_election.cc:400] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 29. Current term is 30.
W20260709 12:29:41.599260  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 68886.6us
W20260709 12:29:41.601708  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61824.1us
I20260709 12:29:41.557996 17301 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:41.592329 17195 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:41.611644  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:41.532078 17316 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.616961  5526 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:41.554648 17369 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.628914  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:41.629545  6271 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:41.634407 17380 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:41.620751  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:41.632205  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:41.639072  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:41.640691  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:41.642079 17426 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.167s	user 0.000s	sys 0.001s
W20260709 12:29:41.650200 17433 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260709 12:29:41.650724 17316 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:41.631812 17241 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.652256 17317 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:41.652034 17434 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:29:41.656893 17301 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.22, Last appended by leader: 22, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.658136 17301 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:41.658140 17241 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.600960 17310 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.667523  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:41.667699 17204 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:29:41.668071  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53436.1us
I20260709 12:29:41.620106 17069 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.674923  5631 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:41.639581  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:41.626179 17195 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.638957 17120 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:41.698048  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:41.698601  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:41.699033  5555 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 36: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:41.699705  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.701094  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.706318 17403 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:29:41.706840  5560 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:41.707279 17403 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:41.707687 17403 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:41.708173  5554 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:41.708506 17403 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.709807 17403 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:41.712181 17115 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:41.712893 17115 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.704937  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.722886 17191 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:41.683173 17380 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:41.729321  5599 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:41.730098  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 102518us
I20260709 12:29:41.735423  5636 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
W20260709 12:29:41.731431  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 203830us
I20260709 12:29:41.737234  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.747181  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.744186  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:41.745122  5705 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:41.749189  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.749567  5554 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:29:41.723618 17342 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:41.691318 17317 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:41.720180 17115 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:41.747062  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 62125.5us
I20260709 12:29:41.759315  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.749467  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:41.750099  5559 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:41.732769 17120 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:41.767208  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.786643 17072 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:41.787513 17072 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.779631  5557 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:41.790365  5706 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:29:41.793963  5631 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37. Ops: [36.22-36.22]
W20260709 12:29:41.799934  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53463.4us
I20260709 12:29:41.804569  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.746045 16755 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:41.806005  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.742803 17057 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:41.818852  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 84047.6us
I20260709 12:29:41.815725  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:41.794306 17226 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:41.810920 17241 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:29:41.839919  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:41.818315 17316 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:41.844084 17191 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:41.844826 17191 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.845412 17237 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:41.846096 17237 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.847122 17191 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:41.837463 16755 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.849545  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.850410 16755 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:41.851696 17195 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.856640 17057 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.857489 17234 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.859251 17057 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.859735 17234 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:41.862043  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 376428us
I20260709 12:29:41.864107 17379 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.837759 17342 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.864974 17369 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:41.864782 17316 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:41.866828 17432 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.323s	user 0.002s	sys 0.000s
I20260709 12:29:41.867722 17316 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:41.830287 17115 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:41.868820 17115 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.869829 17373 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.872401 17115 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.872561 17373 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:41.861909  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.866791 17342 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:41.861804 17072 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:41.884542 17178 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:41.886183 17115 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:41.886735 17361 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:41.887298 17361 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:41.886986 17115 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.888952  5627 raft_consensus.cc:2393] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 37: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:41.889690 17115 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:41.890002 17342 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:41.890616 17342 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:41.891307 17120 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.891752 17115 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:41.892726  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.892630 17115 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:41.889739  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 56877.2us
I20260709 12:29:41.889014 16755 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:41.894644  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:41.894551 16755 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.895112 17115 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:41.895936  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:41.886184  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:41.896631 17316 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:41.897614 17233 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:41.898092 17233 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:41.888270 17361 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.893128  5558 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 37 because replica has last-logged OpId of term: 36 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 21.
I20260709 12:29:41.896153  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.899705  5556 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:41.901111 17361 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:41.901153 17402 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:41.901855 17402 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:41.902261 17402 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:41.898569 17233 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:41.891675 17342 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.23, Last appended by leader: 23, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:41.897030 16755 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.909303  5675 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:41.911844 17316 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:41.913009 17115 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:41.913789 17301 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:41.914552 17402 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.914680 17301 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.916314 17310 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:41.917042 17380 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:41.920876 17069 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:41.807325 17317 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.923440 17317 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:41.925415  5709 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
W20260709 12:29:41.924654  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 129835us
W20260709 12:29:41.933527  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:41.933514  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:41.935235  5558 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:29:41.933697  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:41.875147 16995 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.634s	user 0.005s	sys 0.011s
I20260709 12:29:41.939208  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:41.945230  5553 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:41.943687  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.946470  5631 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:41.848246 17237 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.958513  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.924826 17379 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.040s
I20260709 12:29:41.958062 17301 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.967623 17061 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.970994  5710 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:41.975452  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.976325  5631 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:29:41.981657  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.981750  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:41.936818 17402 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:41.948017 17234 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:29:41.989908  5705 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:41.943665 17069 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:41.913877 17115 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:41.992959  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:41.994618  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 188440us
I20260709 12:29:42.003014  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:42.003867  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:42.004035  5709 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:42.016875  5636 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:42.021741  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:41.974278 17057 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:42.035595 17301 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:42.036296 17301 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:42.036682 17061 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:42.037532  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:42.038998  5602 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:42.041311  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:42.042899  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:42.043694  5556 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:42.043772  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:42.044728  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:42.048467 17406 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:42.049139 17406 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.051451 17406 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:42.052740  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:42.055780  5556 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:42.056845 17372 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:42.058095 17373 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:42.057713 17372 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.058811 17373 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.062695  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:42.066471 17409 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:42.044780  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:42.072397  5559 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:42.043283  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:41.958444 16995 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:42.101679  5558 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:29:42.036716  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:42.086508  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47703.2us
I20260709 12:29:41.986418 17379 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:42.005921 17069 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:42.108193  5676 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:42.109689  5703 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:42.016422 17361 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:42.117385  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:42.114082  6237 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:42.036711 17301 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:42.121891  5561 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:42.128029  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:42.128978  5559 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 36.
I20260709 12:29:42.110041 17069 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:42.127322 17253 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.888s	user 0.012s	sys 0.006s
I20260709 12:29:42.109654 17379 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.154465  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:42.155792 17379 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:42.156977  5553 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
I20260709 12:29:42.156697 17361 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.073194 17037 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:42.119171 17057 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:42.142202  6237 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:42.145059 17372 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:42.146442 17443 raft_consensus.cc:2804] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:42.148025 17115 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:42.150323  5705 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37. Ops: [36.22-36.22]
I20260709 12:29:42.151059  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:42.152957 17367 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:42.160025  5708 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 34: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:42.171219  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:42.171722  6274 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 30 index: 21. (term mismatch)
I20260709 12:29:42.167918  6237 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:42.136214 17408 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.529s	user 0.013s	sys 0.000s
I20260709 12:29:42.172691  6274 pending_rounds.cc:85] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 20
I20260709 12:29:42.178241  6274 pending_rounds.cc:107] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 29.21
I20260709 12:29:42.178642  6274 raft_consensus.cc:2889] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:42.179188  5552 raft_consensus.cc:1240] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 42 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 41. Current term is 42. Ops: []
I20260709 12:29:42.180344  5555 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:42.176229  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:42.176837  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:42.157483 17379 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:42.184401  5704 raft_consensus.cc:2393] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 36: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:42.160641 17253 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:42.159822 17057 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:42.159430 17361 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:42.180117 17241 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:29:42.186021  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 190678us
I20260709 12:29:42.159339 17037 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:42.171921 17317 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:42.200894 17441 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:42.169260 17443 raft_consensus.cc:697] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:42.199759  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:42.202914  5599 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:42.211877  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:42.193007 17342 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:42.189682 17253 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:42.179718 17408 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:42.234604  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:42.208385 17372 raft_consensus.cc:2804] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:42.183223 17233 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.239459  5552 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 34
W20260709 12:29:42.239907 17442 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.054s	user 0.001s	sys 0.001s
W20260709 12:29:42.240696  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 60346.2us
W20260709 12:29:42.245463  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 381154us
I20260709 12:29:42.245774  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:42.241417  5710 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:42.212989 17379 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:42.241228  5629 raft_consensus.cc:3055] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:29:42.251379  5629 raft_consensus.cc:740] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:42.234400 17253 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:42.229692 17342 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:42.257220  5634 raft_consensus.cc:2368] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33.
I20260709 12:29:42.240170 17440 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:42.267117  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:42.223819 17317 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:42.244737 17441 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:42.254519 17037 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:42.207803 17367 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.257457 17372 raft_consensus.cc:697] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:42.263089 17342 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:42.284884  5704 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:42.261422 17361 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:42.270161  5629 consensus_queue.cc:260] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.23, Last appended by leader: 23, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.378353  5629 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:42.270203 17440 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:42.383037  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:42.386096  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:42.386914 17433 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:29:42.371642 17237 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:42.388365  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:42.391069  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 236703us
I20260709 12:29:42.393213  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:42.371300 17361 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:42.392421  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 127854us
I20260709 12:29:42.416659  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:42.385197 17440 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:42.387895 16995 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:42.459551  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:42.458750 17446 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:42.395941 17057 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:42.408978 17361 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:42.388221 17037 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.529253  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 42 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:42.536792 17195 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:42.552322 17195 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:42.541467 17455 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:42.541059 17372 consensus_queue.cc:237] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.541460 17057 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.543885 17317 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:42.546761  5710 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:42.428172 17237 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.549124 17229 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.178s
W20260709 12:29:42.549924 17448 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.272s	user 0.000s	sys 0.001s
I20260709 12:29:42.397823 17379 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:42.555330 17450 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:29:42.549137  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:42.548444  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:42.541049 17446 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.398887 17241 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:42.583424  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:42.565019 17195 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:42.393412 17443 consensus_queue.cc:237] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:42.600638  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 140430us
I20260709 12:29:42.594738 17233 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:42.614861  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:42.618660 17458 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 12:29:42.600718 17195 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:29:42.609164 17447 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.348s	user 0.000s	sys 0.001s
I20260709 12:29:42.599520 17406 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:42.632675  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:42.644933  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:42.647622  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:42.649210  5552 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
W20260709 12:29:42.649627  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 274979us
I20260709 12:29:42.652180  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:42.653810  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:42.621583 17373 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:42.650640 17178 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:42.634326 17455 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:42.629801 17468 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:42.582185 17441 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:42.656369 17367 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:42.656730 17459 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:42.717140  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:42.722476  5706 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 35: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:42.725641 17453 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:42.726320 17453 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.732986  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4515) took 1248 ms. Trace:
W20260709 12:29:42.733290 17461 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260709 12:29:42.737944  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:42.658758 17178 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.733496  5708 rpcz_store.cc:276] 0709 12:29:41.484733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:41.489043 (+  4310us) service_pool.cc:224] Handling call
0709 12:29:42.732785 (+1243742us) spinlock_profiling.cc:243] Waited 651 ms on lock 0x7b5400074c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:42.732895 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1497103616}
I20260709 12:29:42.703972 17406 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.769847  5602 leader_election.cc:400] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34.
I20260709 12:29:42.770283  5602 leader_election.cc:403] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:29:42.777616 17367 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:42.778038 17057 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:42.778144 17367 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:42.778738 17367 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:42.780745 17456 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 12:29:42.780915 17037 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:42.781199 17446 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:42.782214 17457 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:29:42.782922 17406 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:42.783433 17454 consensus_queue.cc:1048] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.127s
I20260709 12:29:42.768121 17459 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.784915 17237 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:42.785835 17459 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:42.712934 17468 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.718021 17455 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.788278  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:42.799628  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:42.806787  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:42.807883  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 87279.3us
I20260709 12:29:42.808655 17440 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:42.810876 17440 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:42.812674 17464 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election lost for term 33. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34.
I20260709 12:29:42.812873  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:42.813797  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:42.814158 17361 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:42.814925  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:42.815205 17069 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.643s	user 0.003s	sys 0.006s
I20260709 12:29:42.815650  5702 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:42.797616 17455 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:42.816488  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:42.805071 17037 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:42.817845  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:42.818048 17037 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.819245  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:42.819862  6271 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:42.806715  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:42.821017  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:42.823441  5707 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:29:42.790879 17468 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:42.829452  5703 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:29:42.792522 17178 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:42.817274  6272 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:42.815755 17069 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.835278  5704 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:29:42.833846  5706 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
W20260709 12:29:42.831393  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 110664us
W20260709 12:29:42.842800  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:42.830927 17253 consensus_meta.cc:287] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.557s	user 0.012s	sys 0.000s
I20260709 12:29:42.820287 17037 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:42.836552 17301 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.703s	user 0.002s	sys 0.010s
I20260709 12:29:42.866744  5633 raft_consensus.cc:1240] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
W20260709 12:29:42.867782 17460 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.326s	user 0.003s	sys 0.000s
I20260709 12:29:42.868615  5628 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:29:42.872041  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3926) took 1168 ms. Trace:
I20260709 12:29:42.872567  5555 rpcz_store.cc:276] 0709 12:29:41.703946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:41.704463 (+   517us) service_pool.cc:224] Handling call
0709 12:29:42.871829 (+1167366us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b5400082354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:42.871950 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1065894144}
I20260709 12:29:42.867833  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:42.877118  5632 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 24.
I20260709 12:29:42.834673 17453 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:42.877943 17069 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:42.790495 17342 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:42.879791  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:42.880533 17301 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:42.880695 17342 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:42.882345  5701 raft_consensus.cc:2393] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 36: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:42.883328 17301 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:42.834508 17452 consensus_queue.cc:1059] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.166s
I20260709 12:29:42.884183  5560 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:42.885195 17452 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:42.885659 17452 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:42.887001 17452 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.857039 17408 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:42.863366  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:42.892117  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44440.6us
I20260709 12:29:42.895933  5629 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 38 index: 23. (term mismatch)
I20260709 12:29:42.896445  5629 pending_rounds.cc:85] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 22
I20260709 12:29:42.896741  5629 pending_rounds.cc:107] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 37.23
I20260709 12:29:42.897035  5629 raft_consensus.cc:2889] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:42.902880  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:42.913506  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:42.914198  6277 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:42.920383 17440 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:42.921115 17440 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:42.925882 17440 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:42.927608 17440 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:42.928673  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:42.929402  6271 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:42.893800 17178 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:42.930444 17178 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:42.932837 17178 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:42.853564 17253 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.928056 17440 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:42.908993 17317 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:42.901036 17037 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:42.893848 17342 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:42.936687  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:42.941315  6274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1890) took 1287 ms. Trace:
I20260709 12:29:42.943964  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:42.945787  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 486284us
I20260709 12:29:42.944212  5526 leader_election.cc:400] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33.
W20260709 12:29:42.947021  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:42.945761  6274 rpcz_store.cc:276] 0709 12:29:41.653700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:41.668788 (+ 15088us) service_pool.cc:224] Handling call
0709 12:29:41.692980 (+ 24192us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:42.184874 (+491894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:42.941045 (+756171us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b54006bdad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:42.941234 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1075073920}
I20260709 12:29:42.947144  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:42.946595  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:42.948916  5708 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
W20260709 12:29:42.949359  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56402.7us
I20260709 12:29:42.936331  5450 catalog_manager.cc:5697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 26 to 29, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 29 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:42.951071  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:42.951941 17234 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:42.951556  5450 catalog_manager.cc:5697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 32 to 39, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 39 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:42.953728  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:42.954103  5526 leader_election.cc:403] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
I20260709 12:29:42.955938  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:42.956552  5635 raft_consensus.cc:3060] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:42.957173  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:42.957262  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:42.957870  5629 raft_consensus.cc:3060] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:42.911679 17408 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:42.960587  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:42.960604  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51112.4us
I20260709 12:29:42.942296 17440 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:42.965751  5450 catalog_manager.cc:5697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 29 to 32, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 32 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:42.942188 17253 leader_election.cc:290] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:42.952431 17233 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:42.975611  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:42.970268  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:42.971095  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:42.973220 17375 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.073s
I20260709 12:29:42.973438 17037 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.972664  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:42.978516  5629 raft_consensus.cc:2468] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:29:42.971753  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:42.980526  5633 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 37.
I20260709 12:29:42.979642  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:42.960640 17452 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:42.984566  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:42.978461  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:42.967103 17120 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:42.970259 17178 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:42.975365 16755 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.017s
I20260709 12:29:42.992774 17178 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:42.993328 17178 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:42.994702  5709 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 38.
I20260709 12:29:43.000203 17415 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:43.003954 17361 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:42.983640  5450 catalog_manager.cc:5697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 24 to 29, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 29 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:42.989081 17474 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:42.988910 17111 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.009s
I20260709 12:29:42.992602 17416 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:42.975708 17233 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:42.942698 17317 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.006587 17441 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:43.007968 17234 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:43.008435  5708 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:43.008579  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:43.008831  5703 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:42.963382 17449 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:42.957017 17342 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.055168  5632 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:43.055590 17342 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:43.029950 17372 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:43.060794 17372 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:42.984242 17115 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.061s
I20260709 12:29:43.062011 17317 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:42.960335 17237 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.025s
I20260709 12:29:43.063723 17372 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:42.980683 17403 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:29:43.064236 17441 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.064847 17403 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:43.065557 17403 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:43.066931 17441 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:43.031630 17301 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:43.068032 17301 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:43.068536 17301 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:43.033965 17253 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:43.071419 17253 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.073472 17237 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:29:43.073976 17253 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:43.074779 17237 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:43.075165 17237 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:43.076081 17237 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.23, Last appended by leader: 23, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.078091 17443 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:43.078776 17443 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.041359 17367 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.080744 17069 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:43.081204 17367 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:43.081292 17069 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.081890 17367 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:43.082247 17443 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:43.082803  6279 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 31: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:43.083586 17069 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:43.062079  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77614.6us
I20260709 12:29:42.977923 17408 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.084997  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:43.086789 17408 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:43.037497 17416 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:43.087888 17416 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:43.089764 17406 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:43.091300  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.032806  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:42.985461 17440 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:43.038980  5676 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:43.092105 17440 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:43.041532 17233 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:29:43.092406  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53498.4us
I20260709 12:29:43.092649 17440 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:43.093221  5676 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:43.097285  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:43.097373  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:43.097785  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.042737  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:42.976214 17057 consensus_queue.cc:1059] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 31.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.058s
I20260709 12:29:43.047708  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:43.108493  5553 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:43.049849  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:43.052585 17453 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:43.052726 17120 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33.
I20260709 12:29:43.053072 17184 consensus_queue.cc:1059] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 41.25, Next index: 26, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:29:43.078120 17237 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:43.038650  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:43.138468  5560 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:43.099382  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:43.043964  5450 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 23 to 31, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 31 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.104912  5635 raft_consensus.cc:2468] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
I20260709 12:29:43.056118  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:43.139679  5632 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:43.086748  5524 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:43.031742 17474 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:43.106782  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 74094.5us
I20260709 12:29:43.144922  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:43.145644  5553 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:43.109548  5552 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:29:43.148839  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 49533.5us
I20260709 12:29:43.083712 17234 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.129729  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.121645  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:43.118884  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:43.113382  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:43.120853  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 136860us
I20260709 12:29:43.130232  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.118683  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 42 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:43.112003  5627 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 31: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:43.143607  5631 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:29:43.151855  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.154012  5558 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 37.
I20260709 12:29:43.160049  5555 raft_consensus.cc:2410] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 38 because replica has last-logged OpId of term: 36 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 21.
I20260709 12:29:43.155625  5450 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 29 to 34, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 34 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.162976  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:43.163218  5561 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 42: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:43.163748  5707 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 37.
I20260709 12:29:43.154625  5703 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:43.116575 17317 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:43.082397 17367 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.181203 17317 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:43.182622 17317 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.112141 17453 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:43.183840 17453 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:43.183970  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:43.185071  5561 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:43.186354  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:43.129717 17253 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:43.187213  5561 raft_consensus.cc:3060] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 38
I20260709 12:29:43.188812  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:43.189563  5558 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 36.
I20260709 12:29:43.190155 17178 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.191427  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:43.192169  5559 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 38.
I20260709 12:29:43.137569 17441 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:43.193123 17441 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:43.193414 17195 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.546s	user 0.014s	sys 0.006s
I20260709 12:29:43.193531  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:43.193864 17195 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.194320  5556 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 36.
I20260709 12:29:43.196748  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:43.144126 17111 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:43.197935 17234 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:43.144398 17406 raft_consensus.cc:3055] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:43.198262  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:43.198586 17406 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:43.171736  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:43.199067  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.199402  5702 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:43.174979  5632 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 38.
I20260709 12:29:43.171675 16755 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:43.183202 17367 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:43.190512  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:43.196082 17195 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:43.179749  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:43.213505  6272 raft_consensus.cc:2368] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
I20260709 12:29:43.213671  5553 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:43.213984  6274 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:43.152099 17474 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:43.176533  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.218667  5636 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:29:43.178574  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.193147  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:43.220834  5629 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:43.206748  6236 leader_election.cc:304] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:43.207538  5450 catalog_manager.cc:5697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 28 to 32, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 32 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.114022 17037 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:43.210258  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:43.196660  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:43.197650 17111 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:43.215683  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 271635us
I20260709 12:29:43.203521  5552 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 36.
I20260709 12:29:43.153834 17449 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:43.229537  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:43.231676  5634 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:43.232074  6271 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:29:43.234273  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.168933 17342 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:43.236613  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:43.193554 17441 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:43.241644  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:43.245114  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:43.245548  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:43.246905  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:43.233064  5450 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 24 to 28, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 28 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:29:43.255532  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:43.246021  6279 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 38.
I20260709 12:29:43.260300  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:43.213234 17115 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:43.259274 17195 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:43.267848  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.229048 17120 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:29:43.229408 16755 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:43.233238 17367 raft_consensus.cc:2804] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:29:43.272603  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:43.276202  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:43.262531  5558 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:43.213305 17444 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 29.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260709 12:29:43.279047  6279 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:43.286273  5450 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 21 to 29, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 29 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.290629  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:43.291452  5558 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 37.
I20260709 12:29:43.292680  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:43.295459  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:43.292809  6237 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:43.263454 17178 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:43.301512  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:43.260506 17342 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.314461  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:43.314592 17111 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:43.232946 17301 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.260730 17449 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.316725  6273 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 37. Current term is 38. Ops: []
I20260709 12:29:43.316906  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:43.317276 17449 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:43.318459  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:43.319080  6278 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:43.229369 17406 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.308636 17372 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:43.341303  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:43.314730 17342 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:43.317112 17301 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:43.236780 17452 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 35.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.044s
I20260709 12:29:43.293633  5706 raft_consensus.cc:2376] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34. Re-sending same reply.
I20260709 12:29:43.322626  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.323755  5450 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:43.322223  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:43.342468  6271 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 38: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:43.333666  5560 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:43.352013  5636 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:43.343150  5702 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:43.301645 17367 raft_consensus.cc:697] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 38 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:43.361886  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:43.346321 17372 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:43.365295  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 278612us
I20260709 12:29:43.364794  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:43.276675 17195 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.363551  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:43.369451  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:43.368947  5450 catalog_manager.cc:5697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 26 to 33. New cstate: current_term: 33 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.298458 17115 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.390661 17115 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:43.370072 17443 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:29:43.395269  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.395555 17443 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:43.395991 17443 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:43.396741  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.397903  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:43.398936  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:43.373808 17372 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:43.407006 17233 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.321650 17416 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.377404  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:43.419185  5704 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:43.392302  5555 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 34: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:43.393087 17367 consensus_queue.cc:237] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.373248 17406 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:43.409729 17408 raft_consensus.cc:3055] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:29:43.466341 17408 raft_consensus.cc:740] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:43.467231 17408 consensus_queue.cc:260] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 41.25, Last appended by leader: 25, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.468580 17408 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:29:43.472692 17440 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.474915 17440 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:43.477442 17403 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.490193  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:43.491160  5552 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:43.492025  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:43.494059  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.495280  5634 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:43.495389  5558 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 24. Current term is 25. Ops: []
I20260709 12:29:43.415143  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.412797  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:43.417641 17453 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.386304  5450 catalog_manager.cc:5697] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 20 to 33, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 33 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.431499  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:43.372546 17072 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:43.433885  5561 raft_consensus.cc:2468] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 38.
I20260709 12:29:43.455113  5560 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:43.456400  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:43.516456  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:43.517086  5556 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:43.518484 17474 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.519650  5629 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:43.520980  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:43.522600  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.523816  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.524248 17444 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:43.524767 17444 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:43.521071  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:43.525556  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.525841 17444 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.526731  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.527180 17444 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:43.527757  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.443921 17178 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:43.528661  5554 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 23. Preceding OpId from leader: term: 32 index: 24. (index mismatch)
I20260709 12:29:43.528766 17178 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.525918  5629 raft_consensus.cc:2368] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
I20260709 12:29:43.529625  5633 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:43.531002 17178 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:43.369127 16755 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.533560  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:43.534291 17178 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:43.535347  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:43.456947  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:43.537994  6271 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:43.459113 17195 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:43.459757  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:43.540522  5706 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 39.
I20260709 12:29:43.464044  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.464919 17472 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:43.420320 17416 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:43.548624  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:43.542397  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:43.495724  5707 raft_consensus.cc:2368] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35.
I20260709 12:29:43.378818  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:43.549152  5450 catalog_manager.cc:5697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 23 to 27, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 27 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.497018  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 42 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.465880 17233 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:43.503072  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:43.385905 17069 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:43.512898  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:43.494419 17468 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:29:43.514448  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:43.539510  5556 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:43.549335  5708 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 38.
W20260709 12:29:43.506891  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 63837us
I20260709 12:29:43.548375 17472 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:43.514020 17072 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:43.528154 17443 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.541149 17253 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:43.552634 17057 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
W20260709 12:29:43.552799  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 174105us
I20260709 12:29:43.557062  5635 raft_consensus.cc:2393] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 31: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:43.586508 17178 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:43.551436  5628 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:43.587033 17178 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:43.587507 17178 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:43.558019 17453 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:43.559151  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:43.605406  6271 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 38. Current term is 39. Ops: []
I20260709 12:29:43.591866  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:43.606848  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 283312us
I20260709 12:29:43.609611  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:43.611287  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:43.613958  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:43.622532  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:43.633445  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 74361.1us
I20260709 12:29:43.564662  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:43.634513  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:43.635231  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:43.573235  5450 catalog_manager.cc:5697] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:43.577412  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:43.581707  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:43.582381  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:43.649544  5701 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:43.636992  5450 catalog_manager.cc:5697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 29 to 32, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 32 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:29:43.635161  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70579.7us
I20260709 12:29:43.561342 17488 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:29:43.559401 17468 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:43.569165 17072 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:43.643846  5636 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:29:43.569828 17403 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:43.669477  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:43.670827  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:43.688468  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:43.688452  5627 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:29:43.671108  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:43.636865  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:43.646313  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:43.707300  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61100us
I20260709 12:29:43.699587  5450 catalog_manager.cc:5697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 28 to 34. New cstate: current_term: 34 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.637209  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:29:43.710211  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 73090.5us
I20260709 12:29:43.708204  5676 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:43.691027  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:43.729533  5633 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:29:43.730038  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:43.715036  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44033us
I20260709 12:29:43.577076 17443 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:43.566422 17069 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:43.710896  6236 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:43.724713 17416 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:43.691915 17468 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:43.732146 17490 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:43.657384 17452 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:29:43.760298  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:43.563422 17372 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.666694 17413 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:43.645023 17195 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:43.660706 17301 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:43.660580 17237 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:43.643139 17484 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:43.735271 17037 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:29:43.661000 17440 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:43.568655 17472 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.670064 17496 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:43.692431 17342 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:43.687659 17474 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:43.574790 17441 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.763414  5450 catalog_manager.cc:5697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 30 to 34, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 34 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.765619  5701 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:43.767470  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:43.769065  6279 raft_consensus.cc:2368] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 37. Current term is 38.
I20260709 12:29:43.770514 17372 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:43.763674 17413 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:43.750031 17443 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:43.695106 17463 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 33.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.066s
I20260709 12:29:43.579725 17253 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.792105  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:43.792560  5557 raft_consensus.cc:1275] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:29:43.808921  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:43.809854  5560 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:43.792668 17488 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:43.792936 17449 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:43.792177  5450 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 25 to 29. New cstate: current_term: 29 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.772409 17195 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:43.771952 17416 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 36 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:43.818634  5556 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:29:43.819368  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:43.785521 17440 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:43.820915  5634 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 42.
I20260709 12:29:43.793872 17452 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:43.797899 17413 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:43.794207 17037 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:43.828771  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:43.831979  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:43.803504 17443 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:43.836659  5450 catalog_manager.cc:5697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 27 to 32. New cstate: current_term: 32 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.814615 17367 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:43.838807  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:43.818199 17195 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:43.816048 17488 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:43.824404 17440 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:43.795507 17342 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:43.843366  5450 catalog_manager.cc:5697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 24 to 29. New cstate: current_term: 29 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:43.829670 17301 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:43.833695 17472 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:43.836374 17237 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:43.849514  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:43.853485  5560 raft_consensus.cc:2468] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 39.
I20260709 12:29:43.825261 17496 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:43.839771 17443 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:43.864840  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:43.865016 17500 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 12:29:43.865518 17403 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:43.865617 17253 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:43.837776 17115 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.031s
I20260709 12:29:43.866411 17037 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.873310  5450 catalog_manager.cc:5697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 18 to 30. New cstate: current_term: 30 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.890621  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:43.853935 17367 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:29:43.874539  5526 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.674s	user 0.027s	sys 0.066s
W20260709 12:29:43.857779  6276 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.023s	sys 0.054s
W20260709 12:29:43.902889  5526 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.027s	sys 0.066s
W20260709 12:29:43.902987  6276 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.024s	sys 0.055s
I20260709 12:29:43.845582 17120 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:43.888079  6279 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 23. Preceding OpId from leader: term: 32 index: 24. (index mismatch)
I20260709 12:29:43.851833 16755 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:43.866613  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:43.912796  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:43.894907  5450 catalog_manager.cc:5697] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 29 to 31, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 31 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.874073  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:43.892601  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:43.901602  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:43.869148 17237 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:43.920217  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:43.877396 17468 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.923476 17468 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:43.924230  6275 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 39: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:43.924605 17237 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:43.927620 17474 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:43.928692 17057 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:43.917811 17403 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:43.929536 17403 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:43.929244 17057 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.929962 17403 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:43.931567 17057 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:43.928059 17474 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:43.934336  5450 catalog_manager.cc:5697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 24 to 30, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 30 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.938812 17449 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:43.939388 17449 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.936106  5450 catalog_manager.cc:5697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 26 to 33. New cstate: current_term: 33 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
W20260709 12:29:43.940976 17502 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:29:43.941624 17449 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:43.941929  5450 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 29 to 33, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 33 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:43.943406  5450 catalog_manager.cc:5697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 28 to 35. New cstate: current_term: 35 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.934177 17474 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.944866 17342 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:43.945484 17474 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:43.946643 17072 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.947816  6280 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 36. Current term is 37. Ops: []
I20260709 12:29:43.953243  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:43.953929 17452 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.843819 17069 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 33.26, Last appended by leader: 26, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:43.964700 17468 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 34.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.006s
I20260709 12:29:43.971127  5450 catalog_manager.cc:5697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 25 to 32. New cstate: current_term: 32 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:43.972893  5450 catalog_manager.cc:5697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 25 to 28. New cstate: current_term: 28 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:43.973960 17484 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:29:43.886258 17416 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 35.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:43.975430 17072 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:43.976094 17441 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:43.908751 16755 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:43.978044 17372 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:43.978071 16755 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:43.974479  5450 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 19 to 27, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 27 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:29:44.001579  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 292595us
W20260709 12:29:44.000797  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 273209us
I20260709 12:29:44.012377  5636 raft_consensus.cc:1275] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:29:44.002728  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:44.030565  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 292198us
I20260709 12:29:44.032042  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:44.034415  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:44.034957  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:44.029129  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:44.056248 17504 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:44.056762  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:44.056850 17504 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:44.059180 17504 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:44.059725  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:44.061282  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:44.062106 17415 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:44.063263 17415 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:29:44.066279  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:44.067565  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:44.026115 17234 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:29:44.068856 17449 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:44.069115 17234 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:44.069866 17234 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:29:44.070183 17449 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:44.071784 17449 raft_consensus.cc:2764] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader pre-election decision vote started in defunct term 38: won
I20260709 12:29:44.072717 17115 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:44.073243 17115 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:44.073642 17115 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:43.997186 17367 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.062119  5634 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:44.078305  5701 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:29:43.990796 17509 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.089s
I20260709 12:29:44.105674  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:44.031388 17441 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.055s
I20260709 12:29:44.083386 17120 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Stepping down as leader of term 30
W20260709 12:29:44.125948  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:44.130941  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:44.131738  5634 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:44.086951 17342 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:44.050608 17508 consensus_queue.cc:1048] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.077s
W20260709 12:29:44.140846  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 229759us
I20260709 12:29:44.117306 17233 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:44.151794  6237 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:44.161199 17253 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:29:44.161650 17253 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:44.162138  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:44.162832  5560 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 37.
I20260709 12:29:44.162493 17253 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.163853  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:44.130255 17120 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 30 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:44.164342  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:44.164842  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:44.173576  5630 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:29:44.173940  5636 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:44.166184  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:44.176460 17509 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:29:44.178023  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:44.179603 17440 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:44.180665  5709 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: []
W20260709 12:29:44.181274 17505 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
I20260709 12:29:44.186857 17507 consensus_queue.cc:1048] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:44.188035 17440 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:44.189146 17507 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20260709 12:29:44.198885  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:44.210021 17443 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.212906 17415 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:44.213513 17415 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:44.215749 17415 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:44.165448 17178 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.577s	user 0.012s	sys 0.000s
W20260709 12:29:44.223994  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 349983us
I20260709 12:29:44.226068  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:44.173332 17342 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:44.250402  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:44.233850  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:44.142764 17241 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.081s
I20260709 12:29:44.257234  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:44.258211  5558 raft_consensus.cc:2393] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 34: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:44.249440  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:44.200103 17516 raft_consensus.cc:2804] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:29:44.166325 17413 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:44.269549  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 43584.5us
I20260709 12:29:44.271983  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:44.273998  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:44.223237 17443 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:44.239559 17490 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:44.260154 17072 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:29:44.313271  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 278937us
I20260709 12:29:44.315055  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:44.321425  5709 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:44.238317 17178 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:44.323048  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:44.322916  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:44.325898  5561 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:44.341156  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:44.342803  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 239716us
I20260709 12:29:44.347545  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:44.347975  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 319125us
I20260709 12:29:44.348667  5558 raft_consensus.cc:2393] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 30: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:44.167136 17120 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.352039  5629 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:29:44.388195  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:44.401906  5561 raft_consensus.cc:2393] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:44.403684  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:44.404587  5561 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:44.411412  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:44.299736 17072 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:44.416265  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:44.417629  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:44.296864 17516 raft_consensus.cc:697] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 39 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:44.320917 17507 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:44.420114  5629 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:29:44.297152 17518 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:44.422762  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:44.427459  5708 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:44.348278 17416 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:44.432792  5703 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 2052) took 1329 ms. Trace:
I20260709 12:29:44.297333 17403 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.289710 17234 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.412258 17449 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:44.442938  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 173124us
I20260709 12:29:44.415470 17440 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:44.484474 17474 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.538s	user 0.009s	sys 0.004s
I20260709 12:29:44.496037 17440 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.497189 17504 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:44.497622 17504 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:44.497848 17234 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:44.497984 17504 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:44.498929 17440 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:44.460208  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 162538us
I20260709 12:29:44.500537 17403 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:44.502583  5557 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:44.484532  5558 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 36.
I20260709 12:29:44.503504  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:44.485520  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 317790us
W20260709 12:29:44.484407 17444 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.957s	user 0.013s	sys 0.003s
W20260709 12:29:44.444466 17237 consensus_meta.cc:287] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.519s	user 0.013s	sys 0.000s
I20260709 12:29:44.496107 17520 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:44.483816 17195 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.631s	user 0.009s	sys 0.002s
I20260709 12:29:44.470140  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:44.447991  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:44.459501 17072 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:44.518290  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:44.524266  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:44.483379  5703 rpcz_store.cc:276] 0709 12:29:43.095674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:43.312932 (+217258us) service_pool.cc:224] Handling call
0709 12:29:43.349946 (+ 37014us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:43.361110 (+ 11164us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:43.361149 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:43.374322 (+ 13173us) log.cc:844] Serialized 26 byte log entry
0709 12:29:43.871725 (+497403us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:29:44.223025 (+351300us) log.cc:844] Serialized 14 byte log entry
0709 12:29:44.223324 (+   299us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:44.223407 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:44.223452 (+    45us) raft_consensus.cc:1685] finished
0709 12:29:44.223467 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:44.424704 (+201237us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x7b5400309650. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:44.424875 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6653,"prepare.queue_time_us":544246,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":3140,"raft.queue_time_us":10091,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":33642,"spinlock_wait_cycles":83560704,"thread_start_us":781791,"threads_started":2,"wal-append.queue_time_us":634460}
I20260709 12:29:44.537520  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:44.540913  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:44.541090  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:44.526388  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:44.558727  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 110843us
I20260709 12:29:44.487510 17416 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:44.542192  5555 raft_consensus.cc:1240] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 38 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 37. Current term is 38. Ops: []
I20260709 12:29:44.541793  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:44.559690  5675 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:44.540537  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 129207us
I20260709 12:29:44.543473  5701 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 37 index: 22. (index mismatch)
I20260709 12:29:44.581095  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:44.542388  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:44.542308  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:44.583235  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:44.583957  5708 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 37.
I20260709 12:29:44.584483 17463 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:29:44.566617 17416 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:44.588586  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:44.556176 16755 consensus_meta.cc:287] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.562s	user 0.013s	sys 0.000s
I20260709 12:29:44.567956  5561 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
W20260709 12:29:44.574162  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52684.2us
I20260709 12:29:44.553215  5704 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:44.567494  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:44.590214  5601 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:44.520577 17367 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.069s
I20260709 12:29:44.442013 17342 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 33.26, Last appended by leader: 26, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:44.656030  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 76858.1us
I20260709 12:29:44.687332  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:44.646217  5553 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:44.698540  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:44.603582  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 87661.6us
I20260709 12:29:44.698784  5676 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:44.700455  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:44.700893 17111 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:44.705407 17120 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:44.720293  5633 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.041s	sys 0.080s
I20260709 12:29:44.705796 17367 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:44.591131 17406 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:44.706809 17509 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:44.591300 17496 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:44.534137 17474 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.123s
I20260709 12:29:44.708308  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:44.776801  5554 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 37.
I20260709 12:29:44.513145 17449 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.589736 17440 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:44.784235  5555 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
W20260709 12:29:44.788393 17527 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
W20260709 12:29:44.732175  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 142064us
I20260709 12:29:44.789587  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:44.791342  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:44.750272 17367 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.793251 17444 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:44.794062 17367 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:44.575855 17490 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:44.795238 17474 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:44.542953 17237 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.580202 17415 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:44.741957  5558 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:44.719610 17057 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:44.566632 17195 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.737367  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:44.751531  5553 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32. Ops: [31.20-31.20]
I20260709 12:29:44.656013 17526 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:29:44.820138  5630 consensus_meta.cc:287] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.560s	user 0.005s	sys 0.002s
I20260709 12:29:44.719573 17253 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:44.744882 17233 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:29:44.731034  5633 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.048s	sys 0.083s
I20260709 12:29:44.793810 17444 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:44.797258  5704 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 38.
I20260709 12:29:44.849228  5630 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:44.573768 17115 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.795650 17474 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:44.800558  5556 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:29:44.800817  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 59838.9us
I20260709 12:29:44.731338 17234 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:44.655673 16755 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.656461  5675 leader_election.cc:400] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
I20260709 12:29:44.558255 17520 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.815490 17195 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:44.767747 17406 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:44.796787 17496 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:44.816207  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 78985us
W20260709 12:29:44.904685  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49113.8us
I20260709 12:29:44.905313  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:44.817301  6275 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: [30.21-30.21]
I20260709 12:29:44.817694 17367 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:44.718469 17516 consensus_queue.cc:237] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.822578 17057 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:44.821554 17525 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:29:44.784446 17440 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:44.925558  5556 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 38.
I20260709 12:29:44.824919 17449 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:44.790784 17451 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:44.821408 17532 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 38.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:44.734653  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:44.855562 17526 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:44.858443 17528 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:44.861768  5554 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:29:44.876590 17233 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:44.877879 17541 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:29:44.877627  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:44.880185 17115 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:44.884555  5560 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 37: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:44.800093 17237 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:44.928337 17241 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.030s
I20260709 12:29:44.900398  5675 leader_election.cc:403] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
W20260709 12:29:44.901794  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 49436.6us
I20260709 12:29:44.890872 17444 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:44.971925  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:44.828030 17504 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.910358  6234 leader_election.cc:400] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35.
W20260709 12:29:44.974783  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 64616.2us
I20260709 12:29:44.976116  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:44.732689 17111 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:44.851641 17178 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:44.930639  5561 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:44.777982 17509 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.901966 17372 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:44.934350  5704 raft_consensus.cc:1275] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:29:44.956976 17449 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:44.903026 17408 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.034s
I20260709 12:29:44.928239 17440 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:44.923090 17468 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:29:44.951412  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 216833us
I20260709 12:29:44.953508  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:44.960032 17057 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:44.962692 17530 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.244s	user 0.004s	sys 0.000s
I20260709 12:29:44.961145  5555 raft_consensus.cc:2376] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32. Re-sending same reply.
W20260709 12:29:44.899633  5559 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.547s	user 0.012s	sys 0.000s
I20260709 12:29:44.997531  5559 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:44.941190 17452 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:44.952553 17451 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:45.009397  5602 leader_election.cc:336] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Timed out: RequestConsensusVote RPC to 127.4.65.1:43285 timed out after 1.600s (SENT)
I20260709 12:29:44.985525  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:44.989593  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 397266us
W20260709 12:29:44.935372  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:44.997851  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:45.012221  5554 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 34: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:45.001318  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:45.005491  5676 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:45.020207  5557 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.508s	user 0.012s	sys 0.000s
I20260709 12:29:45.021078  5557 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 39.
I20260709 12:29:45.022432 17490 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:45.023046 17490 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:45.022792 17037 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:45.023749 17490 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:45.026036 17037 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:45.026598 17037 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:44.987509 17195 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:44.991166 17526 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
I20260709 12:29:44.981653 17372 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:44.989161 17449 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:45.039721  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:45.039942  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:45.030128  5602 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:44.999083 17472 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.130s	user 0.009s	sys 0.005s
I20260709 12:29:45.036273  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:45.036317  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:45.028630  5552 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:44.961429 17528 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:44.965736 17474 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.019382 17408 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:45.019539 17451 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:45.043535  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:45.035772 17544 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.014s
I20260709 12:29:45.042675 17449 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:29:45.106609  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 160742us
I20260709 12:29:45.051355 17520 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:45.052719  6276 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:29:45.111136  6276 raft_consensus.cc:606] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:45.111521  6276 raft_consensus.cc:614] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:45.029201 17550 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:45.056970  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:45.058871 17416 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.059796 17037 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:45.105201 17548 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 12:29:45.061246 17504 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:45.062000 17178 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:29:45.064546 17547 log.cc:927] Time spent T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 12:29:45.066524 17111 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:45.067651 17233 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.070937  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:45.137326  5629 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:45.071655 17413 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:45.071884  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:45.139501  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:45.140111  5630 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:29:45.073148 17537 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.289s	user 0.000s	sys 0.003s
I20260709 12:29:45.072942  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:45.142414  5633 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:45.074124 17406 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:45.019258 17452 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:44.993335 17444 consensus_queue.cc:1059] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.048s
I20260709 12:29:45.050329 17372 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.107800 17241 consensus_queue.cc:1059] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.033s
I20260709 12:29:45.107949 17532 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.033s
I20260709 12:29:45.034569 17507 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:45.009035 17072 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.113258  5554 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:45.056082 17528 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:45.117465  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:45.119529  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:45.119733  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:45.160648  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:45.161463  6271 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 37: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:45.161713 17072 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:45.162307  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:45.106442 17488 consensus_meta.cc:287] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 1.248s	user 0.010s	sys 0.004s
W20260709 12:29:45.121208  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 404035us
I20260709 12:29:45.176420  6277 raft_consensus.cc:2368] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 37. Current term is 38.
I20260709 12:29:45.174482  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:45.179922  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:45.180073  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:45.180685  6271 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:45.127385  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 153445us
I20260709 12:29:45.078536 17408 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:45.104657 17472 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:29:45.112246 17555 log.cc:927] Time spent T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 12:29:45.139036  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 109621us
I20260709 12:29:45.184679  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:45.140210  6274 raft_consensus.cc:2368] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 38. Current term is 39.
I20260709 12:29:45.146483 17552 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:45.141537  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62182.4us
I20260709 12:29:45.143077  5710 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:45.113054 17550 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:45.144680  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 2358) took 1294 ms. Trace:
I20260709 12:29:45.145814 17037 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:29:45.193770 17037 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:45.193888  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:45.194527  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:45.194766 17037 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.195524 17551 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:45.196056 17037 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:45.183732 17488 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.202988 17488 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:45.157833  5629 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:45.119983  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:45.203814  5709 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 39: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:45.168990 17444 raft_consensus.cc:3055] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:29:45.183691 17072 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:45.204566 17444 raft_consensus.cc:740] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:45.204633  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:45.205696 17444 consensus_queue.cc:260] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 36.25, Last appended by leader: 25, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.206983 17444 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:45.176237 17507 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.209528  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:45.210131  5553 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:45.169762 17416 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:45.169848  6272 raft_consensus.cc:2368] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 37. Current term is 38.
I20260709 12:29:45.171089  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:45.171144 17406 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:45.176052  5628 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:45.184772 17111 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:45.169184 16755 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:45.191617  5630 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:29:45.192005  5554 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:29:45.192595  5627 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 39. Current term is 40. Ops: []
I20260709 12:29:45.191392  5556 rpcz_store.cc:276] 0709 12:29:43.850555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:43.850769 (+   214us) service_pool.cc:224] Handling call
0709 12:29:45.144498 (+1293729us) spinlock_profiling.cc:243] Waited 928 ms on lock 0x7b540041a0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:45.144618 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2133693824}
I20260709 12:29:45.192401  5602 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:45.214454  6280 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:45.050676 17496 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:45.196844 17408 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.197558 17518 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:45.199244  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:45.211997  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:45.189412  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:45.216828  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:45.216135  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:45.269685  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 77392us
I20260709 12:29:45.245620  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:45.271955  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:45.271056  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 81703.6us
W20260709 12:29:45.272858  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:45.276536  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:45.273895  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:45.278113  5706 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:29:45.280405  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:45.280670  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:45.281010  5629 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:45.281370  5636 raft_consensus.cc:2368] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
I20260709 12:29:45.282681  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:45.214011 17372 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:45.284853  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:45.285655 17556 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:45.214195 17552 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.215360 17072 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.289697  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:45.290520 17565 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:29:45.291275  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:45.308490  5554 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:45.276228 17342 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:29:45.278861 17496 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:45.295794  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:45.316494  5553 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:45.299597  5556 raft_consensus.cc:2393] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 35: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:45.208823 17550 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:45.270880 17111 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.219301 17415 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.089s
I20260709 12:29:45.298573  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:45.307525  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:45.317083  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:45.342185  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 43716.3us
W20260709 12:29:45.342945  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:45.343704  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:45.290472 17559 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:45.328125  5629 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 36.
I20260709 12:29:45.334378 17507 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:45.316730 17342 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:45.336413  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:45.356897  5601 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:45.337886 17552 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:45.300511 17551 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:29:45.300146 17509 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:45.295600 17561 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:45.246802 17488 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.065s
I20260709 12:29:45.305244  6271 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:45.297261 17472 raft_consensus.cc:697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:45.247228 17406 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.317611  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:45.383430  5553 raft_consensus.cc:2393] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:45.385027 17451 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.388648  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:45.390367  5632 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 31: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:45.388144 17451 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:45.392832  5631 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: [30.21-30.21]
I20260709 12:29:45.395362  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:45.392091  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:45.398860  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4368) took 1045 ms. Trace:
I20260709 12:29:45.343492  5526 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:45.350348  5554 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:45.335026 17523 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:29:45.348773 17496 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:45.353361  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:45.404590  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 51403.9us
I20260709 12:29:45.399250  5559 rpcz_store.cc:276] 0709 12:29:44.353190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.009482 (+656292us) service_pool.cc:224] Handling call
0709 12:29:45.009957 (+   475us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:45.017750 (+  7793us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:45.017806 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:45.039891 (+ 22085us) log.cc:844] Serialized 26 byte log entry
0709 12:29:45.141859 (+101968us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:29:45.141912 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:45.142005 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:45.398631 (+256626us) raft_consensus.cc:1685] finished
0709 12:29:45.398651 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:45.398804 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15831,"raft.queue_time_us":7535,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":12194,"thread_start_us":36535,"threads_started":1,"wal-append.queue_time_us":227905}
I20260709 12:29:45.362658 17342 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:29:45.381888  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:45.403579  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60146.4us
I20260709 12:29:45.379545  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:45.406476  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:45.411316  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:45.411834  6278 raft_consensus.cc:2393] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 39: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:45.407500  5676 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:45.410436  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:45.412969  5553 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:45.411801  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:45.414536  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:45.414860  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:45.415109  5559 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:45.416594  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:45.417241  5704 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:45.359427 17572 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
W20260709 12:29:45.420809  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 64040.2us
I20260709 12:29:45.420698  5526 leader_election.cc:400] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 38. Current term is 39.
I20260709 12:29:45.423056  5526 leader_election.cc:403] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Cancelling election due to peer responding with higher term
I20260709 12:29:45.423136  5634 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:29:45.380450 17532 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:45.429039  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:45.409559 17551 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:45.412807 17552 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:45.438336  5631 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:29:45.439616  5630 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:29:45.378461 17449 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.409785 17496 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:45.442572  5553 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 38.
I20260709 12:29:45.417474 17451 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:45.452261  6274 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:29:45.352275 17559 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.470079  6274 raft_consensus.cc:606] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:45.470773  6274 raft_consensus.cc:614] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:45.434712  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:45.451316  5552 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:45.452087 17406 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:45.452988 17072 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:45.453317  6275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1907) took 1500 ms. Trace:
I20260709 12:29:45.453964 17452 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:45.454909 17571 log.cc:927] Time spent T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:29:45.455293 17566 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:45.457085 17111 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:45.460439  5560 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 2361) took 1232 ms (client timeout 1604 ms). Trace:
I20260709 12:29:45.461283 17408 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:45.462059  5556 raft_consensus.cc:1240] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 36. Current term is 37. Ops: []
I20260709 12:29:45.461898 17472 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.448755 17520 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:45.450044 17496 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:45.438151 17115 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader election lost for term 38. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 38. Current term is 39.
I20260709 12:29:45.471903  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3943) took 1031 ms. Trace:
I20260709 12:29:45.494601  5555 raft_consensus.cc:2368] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32.
I20260709 12:29:45.478718  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:45.483196  6275 rpcz_store.cc:276] 0709 12:29:43.953139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:43.953281 (+   142us) service_pool.cc:224] Handling call
0709 12:29:43.953639 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:44.907438 (+953799us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:45.453047 (+545609us) spinlock_profiling.cc:243] Waited 863 ms on lock 0x7b54006bdad4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:45.453253 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1985668864}
I20260709 12:29:45.423933 17518 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:45.435879 17523 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 38 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:45.492838  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3944) took 1014 ms. Trace:
I20260709 12:29:45.519392  5556 rpcz_store.cc:276] 0709 12:29:44.478711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.461498 (+982787us) service_pool.cc:224] Handling call
0709 12:29:45.461869 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:45.492520 (+ 30651us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:45.492770 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:29:45.491957  5560 rpcz_store.cc:269] 0709 12:29:44.228123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:44.228320 (+   197us) service_pool.cc:224] Handling call
0709 12:29:45.460220 (+1231900us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b54003a89d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:45.460376 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1455161600}
I20260709 12:29:45.496990  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4371) took 1001 ms. Trace:
I20260709 12:29:45.498476 17566 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:45.503803 17449 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:45.505658  5559 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 29. Current term is 30. Ops: []
I20260709 12:29:45.521267  5554 rpcz_store.cc:276] 0709 12:29:44.495752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.468210 (+972458us) service_pool.cc:224] Handling call
0709 12:29:45.496406 (+ 28196us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:45.496611 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:45.496642 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:45.496692 (+    50us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:29:45.496743 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:45.496808 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:45.496926 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:45.477821 17516 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:45.483132 17451 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:45.513538 17253 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.659s	user 0.012s	sys 0.000s
W20260709 12:29:45.514185  5557 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 2360) took 1353 ms (client timeout 1587 ms). Trace:
W20260709 12:29:45.539565  5557 rpcz_store.cc:269] 0709 12:29:44.160763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:44.163563 (+  2800us) service_pool.cc:224] Handling call
0709 12:29:45.514019 (+1350456us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b54001af254. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:45.514127 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":777149184}
W20260709 12:29:45.541543 17440 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.531s	user 0.009s	sys 0.001s
I20260709 12:29:45.542462  5557 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 38. Current term is 39. Ops: []
I20260709 12:29:45.542148 17440 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:45.543704  5708 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 36. Current term is 37. Ops: [36.21-36.21]
I20260709 12:29:45.544006  5557 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 36 index: 22. Preceding OpId from leader: term: 37 index: 22. (term mismatch)
I20260709 12:29:45.515348  6277 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1821) took 1003 ms. Trace:
I20260709 12:29:45.544938  6277 rpcz_store.cc:276] 0709 12:29:44.512147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:44.512543 (+   396us) service_pool.cc:224] Handling call
0709 12:29:45.515180 (+1002637us) spinlock_profiling.cc:243] Waited 590 ms on lock 0x7b5400771d54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:45.515296 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1356953344}
I20260709 12:29:45.546528  5557 pending_rounds.cc:85] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 21
I20260709 12:29:45.546957  5557 pending_rounds.cc:107] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 36.22
I20260709 12:29:45.547289  5557 raft_consensus.cc:2889] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:45.494454  5552 rpcz_store.cc:276] 0709 12:29:44.455466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.450794 (+995328us) service_pool.cc:224] Handling call
0709 12:29:45.451130 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:45.471653 (+ 20523us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:45.471861 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:45.549273  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:45.549960 17490 consensus_meta.cc:287] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.526s	user 0.011s	sys 0.000s
I20260709 12:29:45.550141  5629 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:45.550644  5552 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:29:45.550455 17490 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.480845 17551 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:45.551683  5707 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 38: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:45.552613  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:45.527525  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3945) took 1007 ms. Trace:
I20260709 12:29:45.518827 17520 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.528622 17072 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:29:45.529417 17573 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 39.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:29:45.529937 17567 raft_consensus.cc:3055] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:45.484262 17552 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:45.525951  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:45.532272 17361 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:45.538440 17451 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:45.514827  5561 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 35. Current term is 36. Ops: [35.18-35.18]
I20260709 12:29:45.577678  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3946) took 1039 ms. Trace:
I20260709 12:29:45.577797  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:45.554450  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:45.554579  5634 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 35. Current term is 36. Ops: [35.24-35.24]
I20260709 12:29:45.554260  5559 rpcz_store.cc:276] 0709 12:29:44.520253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.505002 (+984749us) service_pool.cc:224] Handling call
0709 12:29:45.505458 (+   456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:45.527230 (+ 21772us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:45.527462 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:29:45.555342  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:45.568614  6279 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25.
I20260709 12:29:45.580260  5557 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 33 index: 17. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:29:45.571183  5554 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 32. Current term is 33. Ops: [32.21-32.21]
W20260709 12:29:45.573693  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47813.7us
I20260709 12:29:45.485008 17415 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:45.578037  5561 rpcz_store.cc:276] 0709 12:29:44.538743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.512021 (+973278us) service_pool.cc:224] Handling call
0709 12:29:45.514608 (+  2587us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:45.577395 (+ 62787us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:45.577624 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:29:45.577116 17057 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:45.578485  5629 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
W20260709 12:29:45.587967  5526 leader_election.cc:336] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.571s (SENT)
I20260709 12:29:45.592906  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:45.593746  5629 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:45.595247  5634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "458143f1c77141febe7d9725625617c2"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "cea66aa15be3423880843974acdbf50f"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:45.593086  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:45.596040  5634 raft_consensus.cc:606] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Received request to transfer leadership to TS cea66aa15be3423880843974acdbf50f
I20260709 12:29:45.598557  5634 raft_consensus.cc:614] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:45.572970 17567 raft_consensus.cc:740] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:45.620869  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:45.622613  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:45.596329 17057 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:45.581343 17509 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:45.610085  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:45.610899  5636 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:45.611943  5559 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:45.575780 17361 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:45.585830 17415 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:45.632863  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 416079us
I20260709 12:29:45.635272  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:45.641055  5710 raft_consensus.cc:2368] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 35. Current term is 36.
I20260709 12:29:45.636520  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:45.641582  5705 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:45.641870  5601 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:45.645391  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:45.646153  5710 raft_consensus.cc:2368] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 37. Current term is 38.
I20260709 12:29:45.651206  5709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 2070) took 1140 ms. Trace:
I20260709 12:29:45.640039  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:45.656314  5559 raft_consensus.cc:2393] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 38: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:45.659139  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:45.596590 17572 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:45.613586 17468 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:45.655860  6237 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:45.666159  5629 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:45.667759  5710 raft_consensus.cc:2393] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 34: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:45.671037  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:45.671947  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:45.673185  5554 raft_consensus.cc:1275] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:29:45.651762  5709 rpcz_store.cc:276] 0709 12:29:44.511307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:44.517957 (+  6650us) service_pool.cc:224] Handling call
0709 12:29:45.650928 (+1132971us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b54002f6054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:45.651119 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1512429312}
I20260709 12:29:45.675065  5553 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:45.624693 17057 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:45.681700  5703 raft_consensus.cc:2376] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34. Re-sending same reply.
I20260709 12:29:45.624935 17544 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:45.684763 17508 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:29:45.685001  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:45.684666 17574 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:45.632807 17451 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:45.645974 17415 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:45.644199 17233 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:45.644452 17037 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:45.698560  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:45.624904 17237 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 34.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.073s
I20260709 12:29:45.670409 17408 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:45.701449  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:45.702936  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:45.671468 17115 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:45.705729  6274 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:45.670821 17578 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:45.706918  6237 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:45.680764 17468 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:45.736995 17408 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:45.710445 17037 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.777808 17408 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:29:45.729465  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:45.711910  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:45.744171 17580 consensus_queue.cc:1048] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.281s
I20260709 12:29:45.779930 17037 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:45.780217  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:45.780921  5709 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:45.709556  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:45.711055  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:45.741886  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:45.782254  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 72767.4us
I20260709 12:29:45.686378 17416 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.058s
I20260709 12:29:45.765498 17115 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:45.789671 17115 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:45.693423 17463 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 38.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.067s
I20260709 12:29:45.710461 17508 raft_consensus.cc:697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 38 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:29:45.767222  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 64376us
W20260709 12:29:45.770278  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63439.6us
I20260709 12:29:45.706342 17234 raft_consensus.cc:3055] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:29:45.699870 17544 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.686561 17551 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.774032 17237 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:45.776664 17443 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:29:45.644577 17523 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.697988 17111 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.070s
I20260709 12:29:45.633587 17550 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.657240 17509 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.782500  5701 raft_consensus.cc:2368] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 38. Current term is 39.
I20260709 12:29:45.786183 17580 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:45.624655 17552 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.766911  6275 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 39 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 38. Current term is 39. Ops: []
I20260709 12:29:45.698206 17570 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.073s
I20260709 12:29:45.624527 17518 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.710072 17490 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:45.806354  6280 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8c6474db7a70405694509f6976d850ce"
dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:46870
I20260709 12:29:45.829408 17561 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:45.857698 17408 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.846050 17342 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 42 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.826962 17578 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:45.859570 17578 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.829967 17507 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.030s
I20260709 12:29:45.860705 17342 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 42 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:45.862082 17578 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:45.777567 17468 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:45.710107 17233 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.770329 17072 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:45.783643 17474 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:45.808106 17537 log.cc:927] Time spent T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
W20260709 12:29:45.830928 17553 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 12:29:45.831069 17583 log.cc:927] Time spent T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
I20260709 12:29:45.831358 17443 raft_consensus.cc:697] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 39 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:45.834424  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:45.678393 17566 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.815747 17234 raft_consensus.cc:740] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:45.905238 17452 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:45.819175 17237 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:45.849318  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:45.981424 17237 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.24, Last appended by leader: 24, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.982016  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:45.847507  5675 leader_election.cc:400] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
I20260709 12:29:45.982848 17237 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:29:45.983366  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 136006us
I20260709 12:29:45.983671  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:45.853269  6280 raft_consensus.cc:606] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:45.984560  6280 raft_consensus.cc:614] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:45.681305 17572 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.855368 17508 consensus_queue.cc:237] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.832540 17559 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:45.843186 17580 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:45.764631 17576 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:29:45.864142 17444 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:45.865124  5627 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 39. Current term is 40. Ops: []
I20260709 12:29:45.865834 17562 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.036s
I20260709 12:29:45.872419 17512 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.197s
I20260709 12:29:45.869256 17115 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.879642  5705 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:45.990748  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:45.984571  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:45.991657  5705 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:45.884641  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:45.884714 17544 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:45.883843  5709 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:45.887796 16755 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:45.894526  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4563) took 1380 ms. Trace:
I20260709 12:29:45.908576  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:45.991695  5710 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:45.994866  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:45.995225  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:45.995504  5559 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:45.996330  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:45.997007  5559 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:45.993546  5708 rpcz_store.cc:276] 0709 12:29:44.514461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:44.584627 (+ 70166us) service_pool.cc:224] Handling call
0709 12:29:44.584985 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:45.544232 (+959247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:45.894155 (+349923us) spinlock_profiling.cc:243] Waited 958 ms on lock 0x7b54000e01d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:45.894444 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2204264192}
W20260709 12:29:45.923508  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 313538us
I20260709 12:29:45.708122 17574 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:46.002419  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:46.002343  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:45.934057  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:45.944986 17443 consensus_queue.cc:237] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 37.20, Last appended by leader: 20, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:45.949784 17408 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:45.910565 17570 consensus_queue.cc:1059] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 38.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.041s
W20260709 12:29:45.929473 17372 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.623s	user 0.006s	sys 0.006s
I20260709 12:29:45.952628 17233 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:45.955798 17415 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:46.003129 17582 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.212s	user 0.002s	sys 0.000s
I20260709 12:29:45.966578 17567 consensus_queue.cc:260] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.969337 17507 consensus_queue.cc:1048] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.000s
W20260709 12:29:45.971789  5524 leader_election.cc:336] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.571s (SENT)
I20260709 12:29:45.974728  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:45.975903  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49044 (request call id 3955) took 1110 ms. Trace:
I20260709 12:29:46.024681  5555 rpcz_store.cc:276] 0709 12:29:44.865768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.009189 (+143421us) service_pool.cc:224] Handling call
0709 12:29:45.975605 (+966416us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b540044f054. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:45.975822 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1028333952}
I20260709 12:29:45.976066  5552 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: [31.22-31.22]
I20260709 12:29:45.977867  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3947) took 1398 ms. Trace:
I20260709 12:29:45.984380  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:45.982851  5701 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:46.026789  5556 rpcz_store.cc:276] 0709 12:29:44.579367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.520553 (+941186us) service_pool.cc:224] Handling call
0709 12:29:45.545044 (+ 24491us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:45.579538 (+ 34494us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:45.579579 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:45.579625 (+    46us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:29:45.579658 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:45.977584 (+397926us) spinlock_profiling.cc:243] Waited 34.3 ms on lock 0x7b54000271d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:45.977641 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:45.977790 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":78877952}
I20260709 12:29:46.028537  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:46.030272  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:45.832703 17440 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:45.927951  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:45.992539  5561 raft_consensus.cc:2393] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 25: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:45.923813 17578 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:46.000707  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:46.000921  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:46.001668  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:46.004736 17057 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:45.987357 17452 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:46.002664 17342 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.012s
I20260709 12:29:46.011957  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:46.014073  5554 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 34. Current term is 35. Ops: [34.23-34.23]
I20260709 12:29:46.016708  5559 raft_consensus.cc:2468] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 37.
I20260709 12:29:46.017256  6271 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:45.963959  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:46.044581  5599 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:46.032014  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:46.051757  5705 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31. Ops: []
I20260709 12:29:46.032169  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:46.037179  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 52937.5us
I20260709 12:29:45.998790 17449 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:46.005901  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:46.064052  5708 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:46.065970  5708 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 35. Current term is 36. Ops: []
W20260709 12:29:46.046185  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:46.046033  5636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de"
dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
mode: GRACEFUL
new_leader_uuid: "f613ba929b1046d3b5aed909414a95bf"
 from {username='slave'} at 127.0.0.1:39496
I20260709 12:29:46.042922  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:46.067512  5636 raft_consensus.cc:606] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Received request to transfer leadership to TS f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:46.050838  5707 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:45.993976 17580 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:46.062181  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:45.960032 17361 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:46.068037  5636 raft_consensus.cc:614] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:29:46.077811  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:46.069591  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:46.079316  5555 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:46.070755  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:46.069481  5561 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:46.080849  5701 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:46.023382 17544 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:46.075007  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:46.021095 17598 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:46.096978  5709 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:46.003091 17444 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:46.050276 17567 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:46.007475 16755 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:46.091163  5710 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35. Ops: []
W20260709 12:29:46.097059  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 125301us
W20260709 12:29:46.145185  5524 leader_election.cc:336] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.632s (SENT)
I20260709 12:29:46.097642 17596 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:29:46.097714 17468 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:46.046908 17233 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:29:46.025820 17570 consensus_queue.cc:1243] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer 38736cc545eb4389b220a3bd21d3b466 log is divergent from this leader: its last log entry 39.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
W20260709 12:29:46.104336  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:46.151954  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:46.152930 17570 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:46.086769  5634 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 42.
I20260709 12:29:46.153317 17570 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:46.153622  5602 leader_election.cc:400] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32.
I20260709 12:29:46.153736 17570 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:46.153975  5602 leader_election.cc:403] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20260709 12:29:46.155100 17472 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32.
I20260709 12:29:46.101344 17463 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:46.042855 17578 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:46.132720  5707 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:46.040524 17241 consensus_queue.cc:1059] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 34.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.066s
I20260709 12:29:46.081818 17111 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:46.105104  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:46.100924 17516 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 39 LEADER]: Stepping down as leader of term 39
I20260709 12:29:46.089846 17526 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:29:46.119800  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:46.082595 17449 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:46.156795  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:46.309571  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 152863us
I20260709 12:29:46.310613  5602 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:46.084682  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:46.134160  6279 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1825) took 1595 ms (client timeout 1789 ms). Trace:
I20260709 12:29:46.159725  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:46.134625 17595 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 12:29:46.287375  5704 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:46.149873  6273 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:46.081662 17413 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.075s
W20260709 12:29:46.313681  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 208687us
W20260709 12:29:46.316521  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:46.312820  6279 rpcz_store.cc:269] 0709 12:29:44.514620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:44.514742 (+   122us) service_pool.cc:224] Handling call
0709 12:29:46.109995 (+1595253us) spinlock_profiling.cc:243] Waited 991 ms on lock 0x7b5400534dd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:46.110170 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2278540032}
W20260709 12:29:46.319674  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 235094us
I20260709 12:29:46.109227 17598 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:46.321373  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:46.323102  5526 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.789s (SENT) [suppressed 18 similar messages]
I20260709 12:29:46.344333  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:46.349174  5556 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 42.
I20260709 12:29:46.143080 17550 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:46.349771  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:46.312731  5602 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:46.181507 17361 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:46.106742 17544 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:46.323287  5507 auto_rebalancer.cc:629] Failed to schedule move for tablet 52cee1d09af749489de4689a67047497: Illegal state: Replica cea66aa15be3423880843974acdbf50f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 37 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:46.159219 17463 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:46.275672 17253 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:46.304831 17516 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:46.313620 17406 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:46.375885  5557 raft_consensus.cc:1240] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32. Ops: [31.20-31.20]
I20260709 12:29:46.304646 17563 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.345s
I20260709 12:29:46.345794 17566 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:29:46.346002 17552 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:46.358101 17513 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:29:46.373927 17463 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:46.392994  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:46.358180 17472 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:29:46.397492 17601 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.253s	user 0.001s	sys 0.000s
W20260709 12:29:46.398200 17592 log.cc:927] Time spent T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.374s	user 0.000s	sys 0.001s
I20260709 12:29:46.399842 17578 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:46.387651  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:46.372829 17233 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:29:46.409945  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2364) took 1751 ms. Trace:
I20260709 12:29:46.421589 17444 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:46.422086 17472 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:46.455618 17472 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:46.379453 17253 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:46.462981 17253 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:46.423153 17241 raft_consensus.cc:2804] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader election won for term 37
W20260709 12:29:46.384845 17496 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.852s	user 0.012s	sys 0.002s
I20260709 12:29:46.368685  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:46.464037 17496 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:46.465384  5707 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34. Ops: [33.20-33.20]
I20260709 12:29:46.466440 17496 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:46.468451  5558 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:46.447414 17552 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:46.473177 17552 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:46.454267  5560 rpcz_store.cc:276] 0709 12:29:44.658137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.532037 (+873900us) service_pool.cc:224] Handling call
0709 12:29:45.534223 (+  2186us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:45.536142 (+  1919us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:45.536182 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:45.536707 (+   525us) log.cc:844] Serialized 26 byte log entry
0709 12:29:45.790875 (+254168us) spinlock_profiling.cc:243] Waited 7.68 ms on lock 0x7b180037b340. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:29:45.996212 (+205337us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:29:45.996277 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:45.996397 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:46.409602 (+413205us) raft_consensus.cc:1685] finished
0709 12:29:46.409628 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:46.409873 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":276,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":108,"spinlock_wait_cycles":17669504,"thread_start_us":205098,"threads_started":1,"wal-append.queue_time_us":571883}
I20260709 12:29:46.375924 17440 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.054s
I20260709 12:29:46.426100 17115 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:46.432586 17512 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:29:46.515031 17598 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:46.519763  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:46.521413  5705 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: [30.22-30.22]
I20260709 12:29:46.389592 17406 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:46.532032  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 219967us
I20260709 12:29:46.536196  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:46.491369 17241 raft_consensus.cc:697] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:46.544450 17512 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:46.543712  5602 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:46.523821 17233 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:46.571106 17444 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:46.571038 17233 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:46.573690 17233 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:46.577587  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:46.580564  5675 leader_election.cc:400] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32.
I20260709 12:29:46.552644 17057 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:46.582157  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:46.586295  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 221701us
I20260709 12:29:46.590070  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:46.595117  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:46.597537  5602 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:46.599900  5526 leader_election.cc:400] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 37. Current term is 38.
I20260709 12:29:46.533303 17516 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 39.26, Last appended by leader: 26, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:46.576293 17572 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:46.605436  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:46.600884  5526 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:46.609223  5555 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.529s	user 0.013s	sys 0.000s
I20260709 12:29:46.609647  5555 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 37.
I20260709 12:29:46.610350  5556 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 36. Current term is 37. Ops: [36.21-36.21]
I20260709 12:29:46.591285 17233 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:46.593395 17444 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:46.607154  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:46.641530  5524 leader_election.cc:400] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 38. Current term is 39.
I20260709 12:29:46.646557 17570 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:46.625874 17449 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:46.643687 17444 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:46.645226 17504 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:46.626919 17544 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:46.650979  5526 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:46.599033 17440 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:46.611055  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:46.657156  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:29:46.711944  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61072.5us
I20260709 12:29:46.714054  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:29:46.680824  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:46.611474 17526 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:46.599720 17413 consensus_queue.cc:1059] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.057s
I20260709 12:29:46.626492 17550 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:46.727308 17550 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:46.727873 17550 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:46.595144 17241 consensus_queue.cc:237] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 35.18, Last appended by leader: 18, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:46.709452 17444 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:29:46.732671 17567 consensus_meta.cc:287] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.597s	user 0.006s	sys 0.006s
I20260709 12:29:46.733289 17504 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:46.624037 17561 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:46.716714 17616 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:29:46.658814 17516 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:29:46.722425 17513 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:29:46.656788 16755 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:46.725099 17507 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:46.726502 17406 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:46.733537 17608 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.228s	user 0.001s	sys 0.000s
I20260709 12:29:46.650226  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:46.735232  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 124271us
I20260709 12:29:46.744639  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:46.717615 17496 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:29:46.738202  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 128340us
I20260709 12:29:46.740434  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:46.746809  6274 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:46.748485  6278 raft_consensus.cc:2376] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 25. Re-sending same reply.
I20260709 12:29:46.737003 17616 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:46.739904 17528 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election won for term 37
W20260709 12:29:46.755021  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 157575us
I20260709 12:29:46.755671  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:46.772239  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:46.661761 17342 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.084s
I20260709 12:29:46.747933 17440 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 36 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:46.758831 17561 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20260709 12:29:46.777412  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:46.760282  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:46.766386  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:46.768692 17616 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:29:46.744746 17057 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:46.770959  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 120791us
I20260709 12:29:46.783524  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:46.759349  5709 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:46.778079  5705 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 42.
I20260709 12:29:46.745237 17544 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:46.785127  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:46.779261  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:46.780550  5628 raft_consensus.cc:1240] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: [31.22-31.22]
I20260709 12:29:46.734476 17504 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:46.787599  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 42 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:46.790194  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:46.790776  5705 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:46.781574  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:46.746891 17507 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:46.788501 17609 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:46.794178  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:46.793220  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:46.795640  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:46.796245  5628 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 36.
I20260709 12:29:46.797809  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:46.782454 17237 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.799s	user 0.013s	sys 0.001s
I20260709 12:29:46.823776  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:46.831560  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:46.775394 17472 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:46.852648 17619 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:29:46.857277 16755 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:46.857867 16755 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:46.858228 16755 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:46.777474 17526 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 20, Time since last communication: 0.083s
I20260709 12:29:46.861128  5636 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:46.776979 17496 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:46.898161  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:46.783938 17610 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:46.789528 17528 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 37 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:46.923389  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 66212.4us
I20260709 12:29:46.928527  5632 raft_consensus.cc:2368] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35.
I20260709 12:29:46.929270  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:46.931093  6280 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:46.933350  5554 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4383) took 1406 ms. Trace:
I20260709 12:29:46.933792  5554 rpcz_store.cc:276] 0709 12:29:45.515772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.703612 (+187840us) service_pool.cc:224] Handling call
0709 12:29:45.766052 (+ 62440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:46.050099 (+284047us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:46.921847 (+871748us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b54001be754. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:46.922050 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":569837312}
I20260709 12:29:46.794230 17544 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:46.941494  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:46.919523  5705 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
W20260709 12:29:46.945119  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 147372us
I20260709 12:29:46.788455 17057 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:46.950770  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:46.904565 17598 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:29:46.953653  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:46.951828  5709 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:29:46.904477 17620 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:46.958343  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:46.960125  5709 raft_consensus.cc:2410] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 33 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 19.
I20260709 12:29:46.954766  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:46.909032 17610 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:46.972849  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:46.904464 17449 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.053s
I20260709 12:29:46.979338  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:46.980587  5631 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:46.975090  5705 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
W20260709 12:29:46.971437 17474 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.055s	user 0.008s	sys 0.008s
I20260709 12:29:46.987882  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:46.988905  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:46.990669  5705 raft_consensus.cc:2376] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35. Re-sending same reply.
I20260709 12:29:46.992039  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:46.990958  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:47.006546  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 71933.1us
I20260709 12:29:46.970569 17610 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:47.014592  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:47.007717  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:47.020632  5524 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:47.033589  5635 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:29:46.985474 17596 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 34.25, Next index: 26, Last known committed idx: 25, Time since last communication: 0.038s
W20260709 12:29:47.033437 17361 consensus_meta.cc:287] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.658s	user 0.016s	sys 0.000s
I20260709 12:29:46.971743 17620 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:46.963984 17440 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:47.022884 17570 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:47.033812 17580 consensus_meta.cc:287] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.920s	user 0.014s	sys 0.000s
I20260709 12:29:47.045564 17626 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:29:47.227821  5709 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:29:47.104127  5631 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:47.034544 17408 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:47.292466  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:47.298796  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:46.993461 17496 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:47.334183  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 373087us
I20260709 12:29:47.035290 17598 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:47.236217 17550 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.507s	user 0.007s	sys 0.003s
I20260709 12:29:47.280845 17626 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:47.013342 17528 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:47.228473 17567 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:47.426566  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:47.427383  5631 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:29:47.431420  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:47.431927  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:47.239378 17580 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:47.447639  6275 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.013s	sys 0.029s
I20260709 12:29:47.418576 17609 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Stepping down as leader of term 34
W20260709 12:29:47.450361  6275 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.013s	sys 0.030s
I20260709 12:29:47.418642 17629 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:47.425071 17626 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:29:47.296705 17408 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:47.452844  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:47.435516 17474 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:47.453610  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:47.422279 17472 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:47.460408  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:47.475404 17234 consensus_queue.cc:260] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 22, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.476913 17580 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.477622 17580 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:47.478122 17580 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.479246 17615 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:47.479689 17615 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:47.480036 17615 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:47.447410 17618 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:47.481594  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4580) took 1663 ms. Trace:
I20260709 12:29:47.418293 17550 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.454403 17609 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:47.483958  5704 rpcz_store.cc:276] 0709 12:29:45.818161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:46.140460 (+322299us) service_pool.cc:224] Handling call
0709 12:29:46.140855 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:46.314010 (+173155us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:47.481349 (+1167339us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54002d19d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:47.481536 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":336312192}
I20260709 12:29:47.452237 17361 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:47.490161  6276 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47932 (request call id 1837) took 2368 ms (client timeout 2848 ms). Trace:
I20260709 12:29:47.435627 17567 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:47.492451  6271 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1910) took 2111 ms (client timeout 1546 ms). Trace:
I20260709 12:29:47.491951  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4377) took 2275 ms. Trace:
I20260709 12:29:47.511813  5552 rpcz_store.cc:276] 0709 12:29:45.216443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.551381 (+334938us) service_pool.cc:224] Handling call
0709 12:29:45.975778 (+424397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:46.026827 (+ 51049us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:47.491747 (+1464920us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b540044f050. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:47.491885 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":975316352}
W20260709 12:29:47.511734  6271 rpcz_store.cc:269] 0709 12:29:45.380715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.405977 (+ 25262us) service_pool.cc:224] Handling call
0709 12:29:47.492249 (+2086272us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b5400773154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:47.492376 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1393964416}
I20260709 12:29:47.497136  5631 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:47.491768  6278 raft_consensus.cc:3055] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:47.459962 17509 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:47.463446 17622 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:47.491390  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:47.488476 17550 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:47.556538  5628 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:29:47.435423 17623 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.048s
I20260709 12:29:47.515981  6278 raft_consensus.cc:740] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:47.520939  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:47.559845  5631 raft_consensus.cc:2410] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 33 because replica has last-logged OpId of term: 31 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 19.
I20260709 12:29:47.526677  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 4157) took 1524 ms. Trace:
I20260709 12:29:47.529490 17574 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:47.532281 17579 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 1.763s	user 0.020s	sys 0.020s
I20260709 12:29:47.534281 17609 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 34.25, Last appended by leader: 25, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.535728 17566 raft_consensus.cc:3055] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:47.596881  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:47.539875  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:47.597711  5636 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:29:47.540889  6273 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47920 (request call id 2119) took 1838 ms (client timeout 1870 ms). Trace:
I20260709 12:29:47.542043  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4367) took 3189 ms. Trace:
I20260709 12:29:47.521170 17577 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:29:47.544405  5627 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 4155) took 1543 ms. Trace:
I20260709 12:29:47.545002 17636 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:47.545667 17544 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.481076 17057 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:47.548167  5635 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3972) took 2266 ms (client timeout 1975 ms). Trace:
W20260709 12:29:47.489264 17516 consensus_meta.cc:287] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.744s	user 0.010s	sys 0.000s
W20260709 12:29:47.616011  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 184161us
W20260709 12:29:47.517220 17444 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.787s	user 0.009s	sys 0.001s
I20260709 12:29:47.485631 17637 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:47.514247 17361 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.560719  5634 rpcz_store.cc:276] 0709 12:29:46.002051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:46.002205 (+   154us) service_pool.cc:224] Handling call
0709 12:29:47.526452 (+1524247us) spinlock_profiling.cc:243] Waited 83.7 ms on lock 0x7b540032d7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:47.526578 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":192411392}
I20260709 12:29:47.480947 17474 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:47.558254  6278 consensus_queue.cc:260] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.459751 17598 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.538573 17468 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.600247  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:47.600379  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:47.507815  6276 rpcz_store.cc:269] 0709 12:29:45.122135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.187450 (+ 65315us) service_pool.cc:224] Handling call
0709 12:29:47.489847 (+2302397us) spinlock_profiling.cc:243] Waited 732 ms on lock 0x7b54002e2f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:47.490019 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1682748160}
W20260709 12:29:47.598456  6273 rpcz_store.cc:269] 0709 12:29:45.702307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.702463 (+   156us) service_pool.cc:224] Handling call
0709 12:29:47.540730 (+1838267us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b54006383d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:47.540836 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":844706048}
I20260709 12:29:47.645583  5632 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:47.598596  5558 rpcz_store.cc:276] 0709 12:29:44.352928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:44.836167 (+483239us) service_pool.cc:224] Handling call
0709 12:29:45.513639 (+677472us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:46.468920 (+955281us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:46.772866 (+303946us) spinlock_profiling.cc:243] Waited 954 ms on lock 0x7b540044e154. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:47.541814 (+768948us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x7b540044e150. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:47.541958 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3752206464}
I20260709 12:29:47.600618  5627 rpcz_store.cc:276] 0709 12:29:46.000633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:46.000775 (+   142us) service_pool.cc:224] Handling call
0709 12:29:46.001054 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:46.322447 (+321393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:46.322491 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:46.322552 (+    61us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:29:46.322597 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:47.544121 (+1221524us) spinlock_profiling.cc:243] Waited 321 ms on lock 0x7b54001dddd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:47.544171 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:47.544302 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":738684160}
W20260709 12:29:47.616288  5635 rpcz_store.cc:269] 0709 12:29:45.281571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.282504 (+   933us) service_pool.cc:224] Handling call
0709 12:29:47.547949 (+2265445us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b540032cdd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:47.548071 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3934027008}
I20260709 12:29:47.510762 17633 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:47.634443  6278 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:47.600521 17577 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:47.596928 17627 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:47.597373 17566 raft_consensus.cc:740] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:47.496660 17618 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.599359 17550 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:47.639026  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 226933us
I20260709 12:29:47.621325 17636 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:47.557852 17622 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.694836  5709 raft_consensus.cc:2368] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 37. Current term is 38.
I20260709 12:29:47.696022 17622 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:47.697114 17234 raft_consensus.cc:3060] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:47.699313  5627 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
W20260709 12:29:47.699959 17616 consensus_meta.cc:287] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.910s	user 0.007s	sys 0.003s
I20260709 12:29:47.693125 17574 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:47.701192 17574 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.703526 17574 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:47.636152 17367 consensus_queue.cc:1175] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" responder_term: 35 status { last_received { term: 33 index: 22 } last_committed_idx: 22 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:29:47.679414 17516 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:47.630722 17609 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:47.657776 17625 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:47.687005 17566 consensus_queue.cc:260] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.687819 17567 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:47.688812  5632 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
W20260709 12:29:47.706184  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 252636us
I20260709 12:29:47.686474 17550 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:47.708324 17616 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:47.713145 16755 consensus_meta.cc:287] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.854s	user 0.012s	sys 0.002s
I20260709 12:29:47.726766  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:47.675782  5636 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:47.632226 17444 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:47.677505 17544 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:47.717142  6236 leader_election.cc:400] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35.
I20260709 12:29:47.728410  6236 leader_election.cc:403] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Cancelling election due to peer responding with higher term
I20260709 12:29:47.728434  5631 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:47.717945  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:47.719481  5627 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 34. Current term is 35. Ops: [34.23-34.23]
I20260709 12:29:47.694916 17526 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:47.717316 17516 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.721903 17580 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:47.690143 17636 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:47.727702  5632 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 25.
I20260709 12:29:47.733140  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:47.747339  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:47.749019  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:47.723131 17620 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:47.752188  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:47.757682  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:47.758775  5634 raft_consensus.cc:1240] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:47.758410  6275 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:47.752586  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:47.762070  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:29:47.734259 17551 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.866s	user 0.010s	sys 0.024s
W20260709 12:29:47.755259  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:47.743638 17623 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:47.736723 17650 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:47.735123 17640 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:47.774051  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:47.778661  5632 raft_consensus.cc:1240] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: [30.16-30.16]
W20260709 12:29:47.779711  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:47.731580 16755 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:47.769810 17551 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.901s	user 0.010s	sys 0.024s
I20260709 12:29:47.762718 17504 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:29:47.737193 17544 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.744858 17526 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:47.800827 17524 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.261s	user 0.015s	sys 0.042s
W20260709 12:29:47.801229 17524 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.319s	user 0.016s	sys 0.042s
I20260709 12:29:47.803660 17604 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:47.807787  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:47.808257  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:47.774374 17563 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:29:47.775918 17452 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.810639  5628 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:47.800339 17634 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:47.785066 17449 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:47.819502 17516 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.813668 17655 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:47.819876 17643 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:47.811674 17563 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:47.716753 17625 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.634089 17637 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.685256 17627 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.693888 17577 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:47.677309 17633 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:47.826031 17544 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:47.804744 17604 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:47.829523  5628 raft_consensus.cc:1240] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 30. Current term is 31. Ops: [30.21-30.21]
I20260709 12:29:47.787521 17057 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:47.777575 17629 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:47.814422 17463 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 1.364s	user 0.005s	sys 0.010s
I20260709 12:29:47.808257 17550 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.778525 17623 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.841090 17615 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.843546 17440 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:47.838269  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:47.847652 17563 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:47.817808 17634 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:47.823807 17626 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:47.831734 17474 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.837633 17642 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:47.931689  5629 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
W20260709 12:29:47.868562  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:47.845758  5633 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 3978) took 2562 ms (client timeout 1576 ms). Trace:
I20260709 12:29:47.884814 17629 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:47.887014 17520 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.890175 17623 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:47.844310 17644 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:47.891595  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 2076) took 2245 ms. Trace:
I20260709 12:29:47.941219  5708 rpcz_store.cc:276] 0709 12:29:45.646004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:46.102790 (+456786us) service_pool.cc:224] Handling call
0709 12:29:46.141615 (+ 38825us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:46.149792 (+  8177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:46.149822 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:46.149866 (+    44us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:29:46.149902 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:47.891293 (+1741391us) spinlock_profiling.cc:243] Waited 7.94 ms on lock 0x7b54002d0fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:47.891355 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:47.891508 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18252032}
I20260709 12:29:47.892670 17633 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.896481 17440 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.863229 17463 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:47.858341 17057 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:47.905450 17233 raft_consensus.cc:3055] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:47.908659 17652 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:47.909013 17637 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.909763  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:47.912532 17551 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:47.913534  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 42 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:48.013167  5554 raft_consensus.cc:2368] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 42. Current term is 43.
I20260709 12:29:47.914633  5552 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 33 index: 26. Preceding OpId from leader: term: 35 index: 27. (index mismatch)
I20260709 12:29:47.915237 17550 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:47.916229 17115 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:47.916929 17361 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:47.917155 17552 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:47.917773 17604 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.917984  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:47.920535 17408 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.920835 17656 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:48.035171  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:47.921541  5557 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 36. Current term is 37. Ops: [35.18-36.19]
I20260709 12:29:47.922226 17625 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.922104  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:47.922974 17444 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.923048  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:48.047860  5559 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:48.011819 17610 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.977s	user 0.011s	sys 0.002s
W20260709 12:29:47.964839 17593 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 2.106s	user 0.009s	sys 0.034s
W20260709 12:29:48.050881  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 128853us
I20260709 12:29:47.924822 17627 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:47.844398 17655 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:47.925377 17616 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 43 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:47.924917  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:47.846695 17580 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:47.838948 17449 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:47.929098  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:47.938156 17642 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:47.925225 17563 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:47.934453  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:47.932828  5633 rpcz_store.cc:269] 0709 12:29:45.282982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.336098 (+ 53116us) service_pool.cc:224] Handling call
0709 12:29:47.845432 (+2509334us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b540032d2d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:47.845679 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1450114816}
I20260709 12:29:47.864149 17577 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:47.935757 17508 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:47.929411 17634 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:47.936726 17640 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:48.017127  5558 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:47.853281 17643 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.015280  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.021453  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:47.943059 16755 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:48.099186  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.100061  5552 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 38.
I20260709 12:29:48.101822  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:47.961062  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:48.022616  5628 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:48.024027  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:48.103482  5703 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 42.
I20260709 12:29:48.024828  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:48.104772  5706 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:48.036062  5709 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:48.041357  5524 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:48.106737  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 65541.6us
I20260709 12:29:48.053822  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.108860  5557 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:48.054006  5560 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
W20260709 12:29:48.055399  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:48.109895  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:48.056699  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.110102  5704 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:47.945104 17623 consensus_queue.cc:1059] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.090s
I20260709 12:29:48.059998  5555 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:29:47.930490  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 144755us
I20260709 12:29:48.060565  5559 raft_consensus.cc:2468] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:48.021507 17551 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.021184 17115 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 31 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:48.024617 17496 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:48.052295 17593 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 2.193s	user 0.023s	sys 0.044s
I20260709 12:29:48.054785 17361 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:48.091811  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:47.944761 17644 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.094583  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:48.095430  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.113657  5559 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:48.130693  5705 raft_consensus.cc:3060] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:29:48.016721 17233 raft_consensus.cc:740] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:48.114431  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:48.134400  5635 raft_consensus.cc:1240] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 36. Current term is 37. Ops: [36.19-36.19]
I20260709 12:29:48.116986  5557 raft_consensus.cc:1275] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 39 index: 23. (index mismatch)
I20260709 12:29:48.135599  5554 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:29:48.054371 17652 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.056227 17610 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.089726 17625 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:48.132973  5558 raft_consensus.cc:1240] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34. Ops: [33.20-33.20]
W20260709 12:29:48.118899  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 210888us
I20260709 12:29:48.129516  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:48.100648 17656 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:48.140652  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.024380 17550 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:48.162344  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:48.146684  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:48.147814  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.151199  5708 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 37 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 36. Current term is 37. Ops: [36.21-36.21]
I20260709 12:29:48.062258 17234 consensus_queue.cc:1059] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.063s
I20260709 12:29:48.181811  5554 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:29:48.120831 17659 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:48.152151  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:48.065506 17474 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.061s
I20260709 12:29:48.157193  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:48.162096  5552 raft_consensus.cc:1240] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:48.054376 17552 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 35.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.071s
I20260709 12:29:48.171200  5706 raft_consensus.cc:2468] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:48.122134 17544 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:48.187970  5705 raft_consensus.cc:2468] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 43.
I20260709 12:29:48.136991 17408 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:48.161176 17650 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election lost for term 34. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35.
I20260709 12:29:48.099427 17508 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.214938  5709 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:48.217428  5558 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35. Ops: [34.27-34.27]
I20260709 12:29:48.062747 17580 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:48.196312  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49717.7us
I20260709 12:29:48.225157  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:48.159250 16755 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:48.221510  5599 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:48.181573 17361 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:48.191445 17528 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:48.229322  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3974) took 2073 ms. Trace:
I20260709 12:29:48.229200 17622 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:48.230562  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:48.280889  5708 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:48.283613  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:48.216020 17640 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.236639 17551 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:48.238685 17550 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:48.239336 17626 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 43 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.240159 17440 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:48.209582 17633 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:48.248493 17610 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.255079  5552 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:29:48.261552 17655 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:48.129027 17496 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.263754 17652 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:48.266132 17508 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.268013  6273 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 40.
I20260709 12:29:48.201514 17616 raft_consensus.cc:3055] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:48.277457 17528 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:48.271416  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:48.275410  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48235.4us
W20260709 12:29:48.276659  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51608.6us
I20260709 12:29:48.277596  5556 rpcz_store.cc:276] 0709 12:29:46.155863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:46.350069 (+194206us) service_pool.cc:224] Handling call
0709 12:29:46.610118 (+260049us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:46.610720 (+   602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:48.229062 (+1618342us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b54003c41d0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:48.229231 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":541819776}
I20260709 12:29:48.280210 17659 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:48.355999  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:48.216404 17615 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:48.181377 17625 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.331416  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.210019 17656 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.349247  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.280278 17650 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:48.274060 17361 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:48.355899  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:48.385624  6276 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:48.142983 17449 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 35.16, Last appended by leader: 16, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:48.388494  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:48.388236 17528 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.389710 17528 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:48.390388  5558 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:29:48.241250 17636 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.391033 17625 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:48.391682  6279 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 36. Current term is 37. Ops: []
W20260709 12:29:48.392453 17664 log.cc:927] Time spent T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:29:48.393045 17636 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.200443 17544 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.395252 17615 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:48.395946 17544 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.357894  5599 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:48.418262  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60513.3us
W20260709 12:29:48.331549 17657 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:29:48.357867 17616 raft_consensus.cc:740] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:48.199604 17115 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.224099 17408 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.382407  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:48.381866  5552 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 37.
I20260709 12:29:48.427098  5630 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3982) took 1648 ms. Trace:
I20260709 12:29:48.295626 17443 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:48.366343 17508 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.021s
I20260709 12:29:48.285452 17623 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:48.363343 17463 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:48.399762  5561 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:48.381036 17057 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.401626 17496 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.435663 17115 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:48.437302 17618 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:48.402047 17588 log.cc:927] Time spent T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 2.547s	user 0.010s	sys 0.005s
I20260709 12:29:48.439486  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:48.440382  5701 raft_consensus.cc:2393] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:48.424074 17532 raft_consensus.cc:2804] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:29:48.442044 17532 raft_consensus.cc:697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 43 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:48.442386  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:48.409991 17659 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:48.388032  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:48.420424  5708 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:48.421197  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:48.278407 17234 raft_consensus.cc:2804] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:48.294256 17622 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.331625 17440 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:48.465303 17415 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:48.496757 17440 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:48.498047 17440 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 35.16, Last appended by leader: 16, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.499402 17440 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:48.424111  6276 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:48.302078 17637 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.118s
I20260709 12:29:48.409240 17643 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:48.429433  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47205.9us
I20260709 12:29:48.503316  5524 leader_election.cc:400] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 37. Current term is 38.
I20260709 12:29:48.503706  5524 leader_election.cc:403] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: Cancelling election due to peer responding with higher term
I20260709 12:29:48.415267 17361 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:48.427512  5630 rpcz_store.cc:276] 0709 12:29:46.778451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:46.778729 (+   278us) service_pool.cc:224] Handling call
0709 12:29:46.793464 (+ 14735us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:46.793668 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:29:46.942359 (+148691us) log.cc:844] Serialized 14 byte log entry
0709 12:29:48.424744 (+1482385us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b1800407680. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ffa21a4bc 00007f4ff756848b 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa21223a
0709 12:29:48.426735 (+  1991us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:48.426799 (+    64us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:29:48.426850 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:48.426916 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:48.427036 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1755,"prepare.queue_time_us":194615,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":2756,"spinlock_wait_cycles":24232192,"thread_start_us":222076,"threads_started":2,"wal-append.queue_time_us":109704}
I20260709 12:29:48.435111 17663 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:48.441288  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:48.387036 17655 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:48.506923 17655 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.509373 17655 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:48.443059  5703 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:48.510612  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:48.511279  5703 raft_consensus.cc:2468] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
W20260709 12:29:48.457957  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:48.513127  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:48.306244 17233 consensus_queue.cc:260] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.518261 17622 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:48.528854 17532 consensus_queue.cc:237] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 41.25, Last appended by leader: 25, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.532377 17524 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:48.533041 17524 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:48.533965 17524 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.535310 17524 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:48.532450 17474 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:48.537425 17474 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:48.538357 17474 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.541308 17507 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 34.27, Last appended by leader: 27, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.542438 17507 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:48.462602  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4588) took 2456 ms. Trace:
I20260709 12:29:48.421515 17610 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:29:48.466120  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44976.7us
I20260709 12:29:48.544552  5707 rpcz_store.cc:276] 0709 12:29:46.006578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:46.467389 (+460811us) service_pool.cc:224] Handling call
0709 12:29:46.467999 (+   610us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:46.480824 (+ 12825us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:46.480876 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:46.480949 (+    73us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:46.481009 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:48.462232 (+1981223us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b54002d23d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:48.462325 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:48.462525 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28772608}
I20260709 12:29:48.545980  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 3993) took 1594 ms. Trace:
I20260709 12:29:48.546304  5707 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:48.547109 17444 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:48.547621 17444 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.512265  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:48.549077  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:48.549744 17444 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:48.551416  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:48.551591  6272 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 34 index: 27. Preceding OpId from leader: term: 35 index: 27. (term mismatch)
I20260709 12:29:48.551928  6272 pending_rounds.cc:85] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 26
I20260709 12:29:48.552150  6272 pending_rounds.cc:107] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 34.27
I20260709 12:29:48.552395  6272 raft_consensus.cc:2889] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:48.552580  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:48.556524 17463 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:48.557142 17463 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.559499 17570 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:48.560484  6272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47934 (request call id 1915) took 1774 ms. Trace:
I20260709 12:29:48.560964 17570 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:48.560863  6272 rpcz_store.cc:276] 0709 12:29:46.786289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:46.786466 (+   177us) service_pool.cc:224] Handling call
0709 12:29:46.852719 (+ 66253us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:48.552862 (+1700143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:48.560235 (+  7373us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54005eef54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:48.560424 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3906233600}
W20260709 12:29:48.563349  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:48.564330  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:48.565220  5636 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:29:48.565469 17656 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:48.563511 17570 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:48.567193 17661 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:29:48.567478  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:48.568608  5628 raft_consensus.cc:2410] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 34 because replica has last-logged OpId of term: 32 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 20.
I20260709 12:29:48.569479 17644 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:48.466521  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 179670us
I20260709 12:29:48.467130  5556 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:48.419533 17650 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.469856  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:48.606376  5708 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:48.395973 17615 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.480558 17526 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:48.486743 17566 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:48.489951 17408 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.490599 17234 raft_consensus.cc:697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:48.490531  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:48.637540  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:48.506202  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:48.492188  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2428) took 2786 ms. Trace:
I20260709 12:29:48.572671  5676 leader_election.cc:400] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 42 election: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 42. Current term is 43.
W20260709 12:29:48.576234  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 55885.5us
I20260709 12:29:48.639257  5676 leader_election.cc:403] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 42 election: Cancelling election due to peer responding with higher term
I20260709 12:29:48.546362  5635 rpcz_store.cc:276] 0709 12:29:46.951801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:47.646991 (+695190us) service_pool.cc:224] Handling call
0709 12:29:47.647456 (+   465us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:48.134841 (+487385us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:48.545710 (+410869us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b540008c0d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:48.545922 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1119212928}
I20260709 12:29:48.641266  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:48.642825  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:48.594295  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:48.606488  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:48.473868  6278 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47932 (request call id 1848) took 1549 ms. Trace:
W20260709 12:29:48.607031  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:48.576575  6234 connection.cc:256] RPC call timeout handler was delayed by -1.0742410540001401s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1738s for a response
I20260709 12:29:48.645346  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:48.615669  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 317833us
W20260709 12:29:48.617844  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 114732us
I20260709 12:29:48.621037  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:48.630851  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:48.647538  5631 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 40: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:48.634967  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:48.648727  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:48.649554  5632 raft_consensus.cc:2376] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33. Re-sending same reply.
I20260709 12:29:48.644548  6278 rpcz_store.cc:276] 0709 12:29:46.924576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:46.924778 (+   202us) service_pool.cc:224] Handling call
0709 12:29:47.491628 (+566850us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:47.928444 (+436816us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:29:47.928494 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:47.928541 (+    47us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:29:47.928581 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:47.928657 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:48.473614 (+544957us) spinlock_profiling.cc:243] Waited 566 ms on lock 0x7b54002e2f50. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:48.473782 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1116,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":3321,"spinlock_wait_cycles":1302669184}
I20260709 12:29:48.653716  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:48.654285  5702 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:48.654595  5708 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 39.
I20260709 12:29:48.638751  5629 rpcz_store.cc:276] 0709 12:29:45.705513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:45.707335 (+  1822us) service_pool.cc:224] Handling call
0709 12:29:45.707583 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:47.932230 (+2224647us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:48.491834 (+559604us) spinlock_profiling.cc:243] Waited 2.22 s on lock 0x7b54001f2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:48.492070 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5113833728}
I20260709 12:29:48.639078  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:48.659257  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:48.665521  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.665891  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.666393  6272 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 38.
I20260709 12:29:48.667487  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:48.668979  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 97063.4us
I20260709 12:29:48.669310  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:48.669783  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:48.670181  5636 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:48.570782 17563 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:48.666411  6274 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 37.
I20260709 12:29:48.672430  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:48.670482  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:48.559506 17463 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.676275  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:48.704731  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.670320  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:48.679399  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.680121  5634 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 40.
I20260709 12:29:48.684675 17552 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:48.679368  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:48.700519  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:48.433673 17623 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.615671 17658 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:48.719461  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:48.719256 17659 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.720067  5629 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:48.719784 17658 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.721951 17658 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:48.615695 17620 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:48.633404 17663 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election lost for term 37. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 37. Current term is 38.
I20260709 12:29:48.722627 17620 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.630446 17642 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.705482  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.732637  5553 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 37.
I20260709 12:29:48.733713  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.734751  5553 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 37 because replica has last-logged OpId of term: 35 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
I20260709 12:29:48.655026 17408 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:48.708508  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.712781  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.713909  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.714953  6279 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 39 LEADER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 38. Current term is 39. Ops: []
I20260709 12:29:48.718469  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:48.718716  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.739269  5558 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:48.740365  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:48.722923 17636 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:48.707073  5561 raft_consensus.cc:1240] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 38. Current term is 39. Ops: []
I20260709 12:29:48.723803  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:48.742009  5630 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 37.
I20260709 12:29:48.724298  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:48.743201  5707 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
W20260709 12:29:48.729180  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58920.5us
I20260709 12:29:48.744599  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:48.730275  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.745827  6276 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 40.
I20260709 12:29:48.703078 17443 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:48.705580 17637 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:48.749065  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.735918  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:48.751044  5558 raft_consensus.cc:2393] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 43: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:29:48.722141 17634 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.602s	user 0.008s	sys 0.003s
I20260709 12:29:48.752315  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:48.753535  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.678269 17656 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Leader election lost for term 42. Reason: Vote denied by peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285) with higher term. Message: Invalid argument: T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 42. Current term is 43.
I20260709 12:29:48.756480  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:48.782809  6274 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:48.759151  5629 raft_consensus.cc:2468] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 26.
I20260709 12:29:48.726455 17463 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:48.764576 17665 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:48.788447 17463 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.781435 17443 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.790730 17463 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.782689  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.755769  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.704412 17526 consensus_queue.cc:1059] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 33.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.060s
I20260709 12:29:48.791744 17443 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.726172 17516 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:48.792199  6275 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:48.792537 17676 raft_consensus.cc:3055] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:29:48.792656 17516 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.793057 17676 raft_consensus.cc:740] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 36 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:48.793735  6273 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:48.784024  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.794597  5553 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 32: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:48.794092 17676 consensus_queue.cc:260] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 36.19, Last appended by leader: 19, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.794879 17516 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:48.795625 17676 raft_consensus.cc:3060] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:48.759553  6271 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:48.785053 17234 consensus_queue.cc:237] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.796509 17620 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:48.791944  5561 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:48.798269  5561 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:29:48.790740  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:48.791684  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.703367 17644 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.061s
I20260709 12:29:48.808221  5676 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:48.809099 17526 raft_consensus.cc:2804] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:48.809630 17526 raft_consensus.cc:697] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:48.810622 17526 consensus_queue.cc:237] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.760382 17408 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.813426 17671 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 32.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:29:48.815390 17408 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.786787 17656 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:48.816005 17656 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:48.787305 17650 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.816665 17656 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:48.817689  6274 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:48.787133 17634 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.632131 17616 consensus_queue.cc:260] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.821072  5560 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:48.754361 17563 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.034s
I20260709 12:29:48.823383 17623 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:48.788506 17665 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:48.826031 17665 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:48.827669 17634 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:48.831012 17577 consensus_meta.cc:287] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.739s	user 0.006s	sys 0.004s
I20260709 12:29:48.832429  5561 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 17. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:29:48.785641 17507 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:29:48.746833 17658 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:48.840911  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:48.833855  5675 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.026s	sys 0.039s
W20260709 12:29:48.846378  5675 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.027s	sys 0.039s
I20260709 12:29:48.764811 17622 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:29:48.791682 17615 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:48.859189  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.863181  6276 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:29:48.863592  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:48.865108  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:48.881676  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:48.889268  6276 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:48.813465 17233 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:48.893136  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.836833 17577 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:48.734644 17552 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.894129  5556 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 37.
I20260709 12:29:48.913112  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:48.913894  6273 raft_consensus.cc:2393] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:48.913892  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.899798  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.902726  5560 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 35 index: 24. Preceding OpId from leader: term: 36 index: 24. (term mismatch)
I20260709 12:29:48.854796 17623 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:48.912066  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:48.853252 17658 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:48.764940 17636 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.906167  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.929982  5553 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:48.919134  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:48.845943 17528 consensus_queue.cc:1059] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.069s
I20260709 12:29:48.926963  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:48.836519 17637 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:48.914278 17516 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:48.930975  5552 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36. Ops: []
W20260709 12:29:48.921794  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:48.892867 17580 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:48.935138  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:48.869324 17671 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.064s
I20260709 12:29:48.916080  5560 pending_rounds.cc:85] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Aborting all ops after (but not including) 23
W20260709 12:29:48.970777  6235 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:48.941071  5555 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.014s	sys 0.032s
I20260709 12:29:48.869854 17622 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:48.919936  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 245577us
I20260709 12:29:48.948257  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.922165 17623 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:48.929189 17658 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:48.972901  5560 pending_rounds.cc:107] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Aborting uncommitted NO_OP operation due to leader change: 35.24
I20260709 12:29:48.983254  5560 raft_consensus.cc:2889] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:29:48.974342  5555 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.014s	sys 0.032s
W20260709 12:29:48.940474  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 438675us
I20260709 12:29:48.911430 17615 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:48.977643  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.929584 17508 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:48.992053  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:48.990836  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:48.994052  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:48.916888 17660 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.056s
I20260709 12:29:48.981182 17622 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:49.000319 17504 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:49.000838 17504 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:49.001197 17690 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:29:49.001480 17504 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:49.001972 17690 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:49.002472 17690 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:29:49.004777  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:49.005244  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:49.005545  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:49.006002  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:49.006860 17616 raft_consensus.cc:3060] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:49.009080 17408 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:49.009516 17408 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:49.009876 17408 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:49.010881  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.012326  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.012758  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.013513  5708 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:49.014384  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:48.952757 17516 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:48.947690 17634 raft_consensus.cc:2804] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:48.911027 17672 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.087s
I20260709 12:29:48.994730  5556 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:48.921684 17577 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:49.021024  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:49.027634  6277 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:49.021911  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:49.022663  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:48.954584 17679 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:48.990661 16755 raft_consensus.cc:2804] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:49.029601  6274 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:48.991145 17615 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:49.023557 17634 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:49.023968 17620 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:49.057691  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 385705us
I20260709 12:29:48.996610 17563 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.045s
I20260709 12:29:49.033643 17671 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:49.037696 17660 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:49.032950 16755 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:49.042421 17615 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:49.089094  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:49.090061  5560 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
W20260709 12:29:49.103147  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:49.117028  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:49.054642 17634 raft_consensus.cc:3060] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:49.120093  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:49.123633  6237 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:49.124646  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:49.124795  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:49.125708  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:49.138727  5553 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
I20260709 12:29:49.139741  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:49.126760  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:49.130936  5628 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
W20260709 12:29:49.132452  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:49.059777 17415 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:49.135643  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:49.140575  5556 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:49.147351  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:49.151904  5634 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 31: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:49.139930  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:49.153836  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:49.153913  5552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:49.154860  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:49.158203  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:49.119264 16755 raft_consensus.cc:3060] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:49.076525 17660 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:49.158872  5630 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
W20260709 12:29:49.174820  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 77815.1us
I20260709 12:29:49.175123  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:49.160925  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:49.190563  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:49.181120  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:29:49.206336  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 45525.8us
I20260709 12:29:49.209653  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:49.210295  5627 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:49.205983 17504 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:49.213153  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:49.064291 17620 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:49.158020 17367 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:49.075675 17671 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:49.204771 17443 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:49.284677 17443 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:49.285494 17443 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:49.053967 17623 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 32.24, Last appended by leader: 24, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:49.237107 17367 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:49.248936 17682 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.317s	user 0.000s	sys 0.007s
I20260709 12:29:49.264607 17650 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:49.264743 17552 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:49.266458 17625 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:49.273628 17660 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:49.222983 17566 consensus_meta.cc:287] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.587s	user 0.010s	sys 0.000s
I20260709 12:29:49.206769 17665 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:49.281778 17504 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:49.137565 17679 consensus_queue.cc:1048] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.087s
I20260709 12:29:49.282611  5632 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 30: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:29:49.283878 17361 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.763s	user 0.012s	sys 0.000s
I20260709 12:29:49.306712 16755 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:49.307534  5557 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.019s	sys 0.027s
I20260709 12:29:49.289435 17577 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:49.349623 17671 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:49.315035 17440 consensus_meta.cc:287] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.815s	user 0.012s	sys 0.001s
I20260709 12:29:49.290613 17672 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:49.305197 17367 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:49.318188  5627 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:49.319181 17620 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:49.388635 17676 consensus_meta.cc:287] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.592s	user 0.012s	sys 0.006s
I20260709 12:29:49.398877  5555 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2374) took 1449 ms. Trace:
I20260709 12:29:49.309295 17234 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:49.421665  5707 raft_consensus.cc:1275] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 36 index: 19. Preceding OpId from leader: term: 37 index: 19. (term mismatch)
I20260709 12:29:49.308166 17650 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:49.328891  5705 raft_consensus.cc:2410] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 33 because replica has last-logged OpId of term: 31 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 21.
I20260709 12:29:49.327700 17636 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:49.336536 17660 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:49.345896  5704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 4047) took 1048 ms. Trace:
I20260709 12:29:49.427124  5704 rpcz_store.cc:276] 0709 12:29:48.297473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:48.521187 (+223714us) service_pool.cc:224] Handling call
0709 12:29:48.521535 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:48.521762 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:29:48.956479 (+434717us) log.cc:844] Serialized 14 byte log entry
0709 12:29:49.345398 (+388919us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:49.345473 (+    75us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:29:49.345555 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:49.345649 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:49.345819 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":267779,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"raft.queue_time_us":113,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"thread_start_us":813213,"threads_started":2}
W20260709 12:29:49.351084  5557 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.019s	sys 0.027s
W20260709 12:29:49.429373  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:49.313411 17615 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:49.355412  5701 raft_consensus.cc:2410] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 32 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 20.
I20260709 12:29:49.331825 17625 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:49.362032 16755 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:49.362983 17672 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:49.343446 17665 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:29:49.358666 17524 consensus_meta.cc:287] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.823s	user 0.013s	sys 0.004s
I20260709 12:29:49.357295 17361 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:49.354321 17618 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:49.421692 17408 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:49.425956 17234 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:49.388675 17690 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:49.399391  5555 rpcz_store.cc:276] 0709 12:29:47.952180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:48.118350 (+166170us) service_pool.cc:224] Handling call
0709 12:29:48.141413 (+ 23063us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:48.141798 (+   385us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:29:48.415993 (+274195us) log.cc:844] Serialized 14 byte log entry
0709 12:29:49.026116 (+610123us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:49.034602 (+  8486us) log.cc:844] Serialized 26 byte log entry
0709 12:29:49.034776 (+   174us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:29:49.035128 (+   352us) log.cc:844] Serialized 14 byte log entry
0709 12:29:49.035663 (+   535us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:49.035763 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:49.398642 (+362879us) raft_consensus.cc:1685] finished
0709 12:29:49.398663 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:49.398815 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47320,"prepare.queue_time_us":515763,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":1293,"raft.queue_time_us":45458,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":31255,"spinlock_wait_cycles":8741888,"thread_start_us":786530,"threads_started":2,"wal-append.queue_time_us":736076}
I20260709 12:29:49.422326  5707 pending_rounds.cc:85] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 18
I20260709 12:29:49.535130  5707 pending_rounds.cc:107] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 36.19
I20260709 12:29:49.535482  5707 raft_consensus.cc:2889] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:49.431921  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:49.537447  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.439677  5710 raft_consensus.cc:1275] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:29:49.426307 17650 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:49.388659 17622 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:49.322079 17658 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:49.521376  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:49.544975  5708 raft_consensus.cc:2368] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35.
I20260709 12:29:49.523149  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.430999 17676 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:49.524371  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.549768  5703 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
I20260709 12:29:49.441730 17625 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:49.526835  5559 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2382) took 1345 ms. Trace:
I20260709 12:29:49.561008  5559 rpcz_store.cc:276] 0709 12:29:48.181175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:48.182026 (+   851us) service_pool.cc:224] Handling call
0709 12:29:48.182355 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:48.194815 (+ 12460us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:29:48.421096 (+226281us) log.cc:844] Serialized 14 byte log entry
0709 12:29:48.926834 (+505738us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:48.926944 (+   110us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:48.926996 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:49.056338 (+129342us) spinlock_profiling.cc:243] Waited 26.3 ms on lock 0x7b18004f5940. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75532a6 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419
0709 12:29:49.526165 (+469827us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b54001c57d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:49.526250 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:49.526453 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48192,"prepare.queue_time_us":559432,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":64,"raft.queue_time_us":4020,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":7483,"spinlock_wait_cycles":88345088,"thread_start_us":691875,"threads_started":2,"wal-append.queue_time_us":729322}
W20260709 12:29:49.446352 17526 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.000s	sys 0.024s
I20260709 12:29:49.576303  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.504204 17620 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:49.577744 17620 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:49.573210  5703 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:49.573403  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.584100  5702 raft_consensus.cc:2468] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 39.
I20260709 12:29:49.584601  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 43 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:49.550367  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 342574us
W20260709 12:29:49.525719 17656 consensus_meta.cc:287] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.709s	user 0.020s	sys 0.000s
W20260709 12:29:49.574442  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 433316us
I20260709 12:29:49.578265 17665 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:49.575608  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:49.590678  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:49.587924  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:49.591280  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:49.592448  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.593209  5708 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 28.
W20260709 12:29:49.572124 17513 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.716s	user 0.011s	sys 0.015s
I20260709 12:29:49.547643 17440 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:49.607594  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.597700  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:49.583632 17526 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.001s	sys 0.024s
W20260709 12:29:49.616802 17697 log.cc:927] Time spent T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.220s	user 0.000s	sys 0.000s
I20260709 12:29:49.616792 17440 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:49.599212  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 418145us
I20260709 12:29:49.619424  6236 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:49.580325 17620 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:49.610059 17513 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.011s	sys 0.016s
I20260709 12:29:49.620231 17440 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:49.629106 17691 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.233s	user 0.001s	sys 0.000s
I20260709 12:29:49.630170 17408 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:49.634219 17367 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:49.572569 17676 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:49.637552  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:49.640986  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:49.641618 17690 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 41 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:49.642644 17663 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:49.643342 17672 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:49.643561 17234 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:49.594194 17640 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.001s	sys 0.017s
I20260709 12:29:49.649809  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.653110  5704 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:29:49.608564  5702 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 39.
I20260709 12:29:49.611928 17361 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:49.620381 17524 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:29:49.637957 16755 consensus_queue.cc:1059] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 33.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.010s
W20260709 12:29:49.658756  5601 leader_election.cc:336] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.870s (SENT)
I20260709 12:29:49.647874  6236 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:49.654501  5599 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.036s	sys 0.028s
W20260709 12:29:49.659606  5599 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.036s	sys 0.028s
W20260709 12:29:49.659901  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 506533us
I20260709 12:29:49.661134  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.595082 17577 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:49.657306  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:49.534530 17618 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:49.654554  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.640115 17676 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:49.664182  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:49.667673  5634 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 33. Current term is 34. Ops: []
W20260709 12:29:49.656327 17616 consensus_meta.cc:287] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.645s	user 0.011s	sys 0.007s
I20260709 12:29:49.608513 17665 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:49.640132 17659 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:29:49.683035  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:49.685858  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:49.686404  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:49.683003  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.687988  5630 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:49.594038 17656 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:49.700789 16755 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:29:49.701082  5708 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 35 index: 16. Preceding OpId from leader: term: 36 index: 17. (index mismatch)
W20260709 12:29:49.655508 17640 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.001s	sys 0.017s
I20260709 12:29:49.696239 17711 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:49.696610 17415 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:49.719000  5524 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.011s	sys 0.028s
I20260709 12:29:49.721400  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:49.721784  5524 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.011s	sys 0.028s
I20260709 12:29:49.625411 17650 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:49.668571 17692 raft_consensus.cc:2804] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:29:49.691817 17504 raft_consensus.cc:3055] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:49.659228 17524 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 40 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:49.738049  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:49.707100  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:49.743772  5630 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
I20260709 12:29:49.719090 17676 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:49.723946 17415 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:49.714434  5710 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 30.
I20260709 12:29:49.718912  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:49.719901  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:49.701766  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:49.723093  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:49.705897  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:49.745148  5708 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Advancing to term 41
W20260709 12:29:49.765659  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 59820.8us
I20260709 12:29:49.766363  5635 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 33: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:49.766686  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:49.767035  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:49.767262  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:49.768399 17526 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:49.769765  5702 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:49.727020  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49044 (request call id 3983) took 1025 ms. Trace:
W20260709 12:29:49.686347 17634 consensus_meta.cc:287] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.532s	user 0.011s	sys 0.003s
I20260709 12:29:49.728324  5704 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 32 index: 22. (term mismatch)
I20260709 12:29:49.793074  5704 pending_rounds.cc:85] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 21
I20260709 12:29:49.793386  5704 pending_rounds.cc:107] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 31.22
I20260709 12:29:49.793627  5704 raft_consensus.cc:2889] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:49.689824 17659 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:49.731920  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:49.795110  5705 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:49.794802  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:49.739620  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:49.684482 17672 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.044s
I20260709 12:29:49.671474 17577 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:49.742556  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:49.683914  5676 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:49.698770 17443 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:49.744187 17692 raft_consensus.cc:697] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:49.734170 17504 raft_consensus.cc:740] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:49.786082  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:49.787075  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:49.740175 17524 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:29:49.731359 17367 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:49.788470  5557 rpcz_store.cc:276] 0709 12:29:48.701442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:48.701645 (+   203us) service_pool.cc:224] Handling call
0709 12:29:48.702077 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:48.702361 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:29:49.429056 (+726695us) log.cc:844] Serialized 14 byte log entry
0709 12:29:49.726475 (+297419us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:49.726542 (+    67us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:29:49.726613 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:49.726813 (+   200us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:49.726954 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13867,"prepare.queue_time_us":85823,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":13971,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":82,"thread_start_us":983382,"threads_started":2,"wal-append.queue_time_us":339462}
I20260709 12:29:49.724932 17440 consensus_queue.cc:1059] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 29.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
W20260709 12:29:49.807852  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 124004us
I20260709 12:29:49.746908 17656 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:49.762686  5602 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:49.754550 17615 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:49.748767 17703 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:49.813097  5629 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 37 index: 21. (term mismatch)
I20260709 12:29:50.191113  5629 pending_rounds.cc:85] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 20
I20260709 12:29:50.191437  5629 pending_rounds.cc:107] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 36.21
I20260709 12:29:50.191682  5629 raft_consensus.cc:2889] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:50.191854  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.193118  6274 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 37: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:49.816109  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:49.803534 17449 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:50.196995 17449 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:50.197444 17449 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:50.197826 17552 raft_consensus.cc:3055] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:29:50.198207 17552 raft_consensus.cc:740] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 30 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:50.199002 17552 consensus_queue.cc:260] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.200203 17552 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:29:50.201184  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 438723us
I20260709 12:29:50.052199 17658 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:50.020793  5560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:50.202263  5560 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:50.202607 17444 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:29:49.821286  5705 raft_consensus.cc:2468] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
W20260709 12:29:49.934087 17706 log.cc:927] Time spent T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260709 12:29:49.819044 17640 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:50.016259 17516 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.205403 17526 raft_consensus.cc:2764] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election decision vote started in defunct term 32: won
I20260709 12:29:50.205672 17367 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:29:50.206620 17367 consensus_queue.cc:1243] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 33.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:29:50.207350 17516 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:49.962689 17650 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:49.908515  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 141568us
I20260709 12:29:50.209764  5526 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.205165  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:50.212172 17629 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:50.212666  5705 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:29:50.215423  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:50.057902  5709 raft_consensus.cc:1240] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
W20260709 12:29:50.216539  5507 auto_rebalancer.cc:629] Failed to schedule move for tablet abd24f2e500a44729c9111fae83c25ed: Illegal state: Replica 4a1c5e6dd8e249bf82d38f90fa37b43d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 35 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:50.216958  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.217666 17115 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:49.950947 17443 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:50.139986  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:50.216565 16755 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:50.219440  5631 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:50.205447 17640 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.220448 17644 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:50.222023  5632 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 35 index: 16. Preceding OpId from leader: term: 36 index: 17. (index mismatch)
I20260709 12:29:50.221971  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:50.220976 17644 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.223644  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:50.224578 17644 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:49.932955 17722 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
W20260709 12:29:49.931985 17710 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.146s	user 0.000s	sys 0.000s
W20260709 12:29:50.075232 17698 log.cc:927] Time spent T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.553s	user 0.000s	sys 0.003s
I20260709 12:29:49.941646 17692 consensus_queue.cc:237] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.097764  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:50.226864 17570 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:50.227314 17570 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:50.227691 17570 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:29:50.232664  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 135018us
W20260709 12:29:50.234556  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:50.235493  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:49.904983 17622 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:49.949465 17577 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:29:50.185137 17672 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:49.800279 17659 raft_consensus.cc:3060] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:49.771974 17676 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:49.820993 17551 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:50.017279  5707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4619) took 1008 ms. Trace:
I20260709 12:29:50.010387 17717 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:49.816910 17701 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:29:50.124011 17623 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:50.117612  5704 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:50.259657  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.094184 17637 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:49.949215 17625 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:50.046720 17660 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.617s	user 0.010s	sys 0.000s
I20260709 12:29:49.992938  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:50.269309  5703 raft_consensus.cc:2410] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 34 because replica has last-logged OpId of term: 33 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 17.
I20260709 12:29:50.270104 17625 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:49.923126 17715 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
W20260709 12:29:50.062181 17709 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.276s	user 0.003s	sys 0.000s
I20260709 12:29:49.917347  5701 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38468 (request call id 4614) took 1254 ms. Trace:
I20260709 12:29:49.803148 17634 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.106997 17721 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 33.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:29:50.275641  5701 rpcz_store.cc:276] 0709 12:29:48.663038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:48.670240 (+  7202us) service_pool.cc:224] Handling call
0709 12:29:49.917106 (+1246866us) spinlock_profiling.cc:243] Waited 675 ms on lock 0x7b54002d14d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:49.917271 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1553130368}
I20260709 12:29:49.951205 17504 consensus_queue.cc:260] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:49.963459  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:50.007573 17655 raft_consensus.cc:3055] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:50.091787 17415 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.174853 17618 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:50.194473  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 378371us
I20260709 12:29:50.194555  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.286190  6276 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:29:50.196077  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:50.286538  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:50.191138 17703 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.212697 17629 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:49.914742  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 157614us
I20260709 12:29:50.225826  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:50.294001  5629 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:29:49.749498 17704 consensus_queue.cc:1059] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.163s
I20260709 12:29:49.924603  5557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:50.235906  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:50.012992  5554 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 2385) took 1747 ms (client timeout 1927 ms). Trace:
I20260709 12:29:49.911715  5702 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:50.172052  5708 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 41.
W20260709 12:29:50.304069  5554 rpcz_store.cc:269] 0709 12:29:48.265359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:48.271084 (+  5725us) service_pool.cc:224] Handling call
0709 12:29:50.012716 (+1741632us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b54001bd854. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:50.012912 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1987170560}
I20260709 12:29:50.183588 17524 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:49.819213 17528 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:50.185315 17713 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:50.261409 17717 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.257964  5559 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 37 index: 21. (index mismatch)
I20260709 12:29:50.317051  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:50.251482  5602 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:50.318019  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 66667.1us
I20260709 12:29:50.267501  5631 raft_consensus.cc:2468] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 38.
I20260709 12:29:50.272461  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:50.273720  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:50.280718  5553 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:50.282761  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:50.249981 17656 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.027s
I20260709 12:29:50.324774  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.326109  5710 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
W20260709 12:29:50.275756 17728 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.094s	user 0.009s	sys 0.000s
W20260709 12:29:50.275509 17723 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.100s	user 0.004s	sys 0.001s
/apache/dev/git/kudu/src/kudu/master/auto_leader_rebalancer-test.cc:461: Failure
Failed
Bad status: Illegal state: leader_transfer_task size should be 0, but 5
W20260709 12:29:50.287232  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 91284.7us
I20260709 12:29:50.369968  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:50.261998 17551 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:50.288264  5634 raft_consensus.cc:1240] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:50.290226  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:50.296334  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.298949 17415 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:50.299003 17634 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:50.281775 17655 raft_consensus.cc:740] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:50.387290 17655 consensus_queue.cc:260] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.388789  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.389979  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.390947  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.391489  6278 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:50.299474 17504 raft_consensus.cc:3060] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:50.392868  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.396615  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:50.300657  5709 raft_consensus.cc:1240] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 37. Current term is 38. Ops: []
I20260709 12:29:50.400710  5559 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:50.401398  5559 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:50.304308  5599 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:50.309348 16755 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:50.402863 16755 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.297369 17644 consensus_queue.cc:1059] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.107s
I20260709 12:29:50.404834 17660 raft_consensus.cc:2764] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election decision vote started in defunct term 34: won
I20260709 12:29:50.405591 17524 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 41 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:50.405720 16755 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:50.310185  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:50.406955 17717 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:50.407131  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 97030.6us
I20260709 12:29:50.407868 17655 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:50.311774  5633 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: []
I20260709 12:29:50.411173 17637 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:50.412783 17625 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:50.413220 17625 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:50.255977  5707 rpcz_store.cc:276] 0709 12:29:49.009081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:49.010107 (+  1026us) service_pool.cc:224] Handling call
0709 12:29:49.010471 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:49.535953 (+525482us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:50.017017 (+481064us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x7b5400323a54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:50.017199 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":944540544}
I20260709 12:29:50.313704  5554 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:50.257679 17672 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.320245  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:50.322708  6236 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:29:50.418841  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 114665us
I20260709 12:29:50.330699  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.332477  6277 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 41: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:50.334976  5558 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2388) took 1496 ms. Trace:
I20260709 12:29:50.330461 17616 raft_consensus.cc:2764] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election decision vote started in defunct term 37: won
I20260709 12:29:50.341930  5561 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4410) took 1421 ms. Trace:
I20260709 12:29:50.424556  5561 rpcz_store.cc:276] 0709 12:29:48.920460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:48.972104 (+ 51644us) service_pool.cc:224] Handling call
0709 12:29:48.979554 (+  7450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:48.982148 (+  2594us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:48.982180 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:48.997499 (+ 15319us) log.cc:844] Serialized 26 byte log entry
0709 12:29:49.530710 (+533211us) raft_consensus.cc:1645] Marking committed up to 26
0709 12:29:49.530779 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:49.530885 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:50.061568 (+530683us) spinlock_profiling.cc:243] Waited 4.28 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:50.341538 (+279970us) raft_consensus.cc:1685] finished
0709 12:29:50.341568 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:50.341848 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2206,"raft.queue_time_us":10509,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":24211,"spinlock_wait_cycles":9842048,"thread_start_us":490520,"threads_started":1,"wal-append.queue_time_us":675140}
I20260709 12:29:50.309222 17711 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:50.342383 17629 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 32.24, Last appended by leader: 24, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.343374 17234 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:50.327515 17713 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.345192 17703 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:50.347405  4356 tablet_server.cc:179] TabletServer@127.4.65.1:0 shutting down...
I20260709 12:29:50.352162 17676 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:50.353072 17658 raft_consensus.cc:3055] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:50.414130 17625 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.467739 17658 raft_consensus.cc:740] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:50.354605 17640 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:50.356236 17449 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.470705 17449 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:50.301468 17526 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:50.471668 17526 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:50.466516 17650 raft_consensus.cc:2804] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:29:50.472632 17526 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.473420 17650 raft_consensus.cc:697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 38 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:50.464242 17618 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.020s
I20260709 12:29:50.474313 17650 consensus_queue.cc:237] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 36.25, Last appended by leader: 25, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.312173 17665 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:50.358172  5706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38468 (request call id 4607) took 1784 ms. Trace:
I20260709 12:29:50.359102  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.477340  6274 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 38.
I20260709 12:29:50.361105  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.478355  6271 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:29:50.361768  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:50.360944  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.302924 17622 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:50.373790  6276 raft_consensus.cc:2410] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 36 because replica has last-logged OpId of term: 34 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 22.
I20260709 12:29:50.377724  5601 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:50.382525  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 92362.4us
I20260709 12:29:50.483789  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:50.383433  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
W20260709 12:29:50.485872  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 108574us
I20260709 12:29:50.487273  5601 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.419173  6278 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
W20260709 12:29:50.323087  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:50.420338  5558 rpcz_store.cc:276] 0709 12:29:48.839079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:48.843269 (+  4190us) service_pool.cc:224] Handling call
0709 12:29:48.843594 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:48.843965 (+   371us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:48.843991 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:48.857893 (+ 13902us) log.cc:844] Serialized 26 byte log entry
0709 12:29:49.358837 (+500944us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:29:49.358898 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:49.359001 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:50.334743 (+975742us) raft_consensus.cc:1685] finished
0709 12:29:50.334766 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:50.334913 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2184,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":7988,"thread_start_us":432930,"threads_started":1,"wal-append.queue_time_us":662317}
I20260709 12:29:50.489861  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 35 index: 16 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:50.432750 17714 raft_consensus.cc:2804] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:50.433197 17644 raft_consensus.cc:3055] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:29:50.437978 17672 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:50.516239  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:50.438012 17415 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:50.439615  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:50.440732  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:50.441725  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:50.442659  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:50.443619  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:50.444862  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 33 index: 26 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:50.446512  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 123980us
W20260709 12:29:50.447414  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 116839us
I20260709 12:29:50.448146  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
W20260709 12:29:50.455271  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:50.519098 16755 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:50.525261 16755 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:50.525691  5702 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:50.526976  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:50.527151  5602 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.527731  5702 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:50.456589  5704 raft_consensus.cc:1240] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 38. Current term is 39. Ops: [38.24-38.24]
I20260709 12:29:50.528644  5635 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:50.528997  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:50.457127  5703 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:50.530472  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:50.457610 17474 consensus_queue.cc:1059] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:29:50.460330  5599 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:50.460270 17241 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:29:50.461457 17580 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:50.542593  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.483052 17622 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:50.466753  5676 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:50.468405 17625 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:50.515991 17616 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:50.508952  5555 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:50.513648  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:50.477106  5706 rpcz_store.cc:276] 0709 12:29:48.573752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:48.576738 (+  2986us) service_pool.cc:224] Handling call
0709 12:29:48.577149 (+   411us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:48.577618 (+   469us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:29:48.820118 (+242500us) spinlock_profiling.cc:243] Waited 2.88 ms on lock 0x7b18004ecb20. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7582455 00007f4ffe06a3c7 00007f4ffa67c9a7 00007f4ffa68c0bf 00007f4ffa68c038
0709 12:29:49.178475 (+358357us) log.cc:844] Serialized 14 byte log entry
0709 12:29:49.503493 (+325018us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:49.524442 (+ 20949us) log.cc:844] Serialized 26 byte log entry
0709 12:29:49.524703 (+   261us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:29:49.524764 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:49.524859 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:50.357861 (+833002us) raft_consensus.cc:1685] finished
0709 12:29:50.357958 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:50.358112 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8597,"prepare.queue_time_us":212084,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":25,"raft.queue_time_us":1834,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":46103,"spinlock_wait_cycles":6627200,"thread_start_us":682518,"threads_started":2,"wal-append.queue_time_us":489320}
I20260709 12:29:50.515441  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:50.568228  5629 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 35: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:50.516848  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:50.568851  5631 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:50.518399  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:50.525767 16755 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:29:50.570207  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 51898us
I20260709 12:29:50.522177  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:50.535107  5627 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:29:50.573125  5628 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:50.535465  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.536237  5708 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:50.519220  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:50.530972  5703 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:50.507601 17618 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:50.585811 17713 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:50.550321 17718 consensus_queue.cc:1059] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.037s
I20260709 12:29:50.587641 17718 raft_consensus.cc:3055] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:29:50.588234 17718 raft_consensus.cc:740] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:50.545694  5556 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58854 (request call id 2394) took 1422 ms. Trace:
I20260709 12:29:50.519873 17714 raft_consensus.cc:697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:50.546679  5561 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 35: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
W20260709 12:29:50.549410  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 82747.5us
I20260709 12:29:50.544690 17580 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:50.544757 17721 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:50.517270 17623 raft_consensus.cc:2764] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Leader pre-election decision vote started in defunct term 39: won
I20260709 12:29:50.590090  5556 rpcz_store.cc:276] 0709 12:29:49.123403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:49.148447 (+ 25044us) service_pool.cc:224] Handling call
0709 12:29:49.149440 (+   993us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:49.548094 (+398654us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:49.548155 (+    61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:49.581648 (+ 33493us) log.cc:844] Serialized 26 byte log entry
0709 12:29:49.795147 (+213499us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:29:49.795872 (+   725us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:49.795951 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:50.545440 (+749489us) raft_consensus.cc:1685] finished
0709 12:29:50.545462 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:50.545605 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23477,"raft.queue_time_us":54888,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":10185,"thread_start_us":532641,"threads_started":2,"wal-append.queue_time_us":577971}
I20260709 12:29:50.508097  5558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:50.455955  5553 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
I20260709 12:29:50.604521  5556 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:50.601817  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:50.605661  5553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f"
I20260709 12:29:50.557273  5559 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:50.514513 17644 raft_consensus.cc:740] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 36 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:50.565582  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:50.565915  5707 raft_consensus.cc:1240] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:50.530865  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:50.607517  5635 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 41: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:50.576802  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.578625  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.579509  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.566407 17616 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:50.581359 17660 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:50.608341  6279 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:50.600652 17580 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:50.612906  5561 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "cea66aa15be3423880843974acdbf50f" is_pre_election: true
I20260709 12:29:50.613998 17659 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.608768  6276 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:50.609517 17616 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:50.548074 17551 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.17, Last appended by leader: 16, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.585870 17692 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:50.616739 17714 consensus_queue.cc:237] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.597509 17704 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:50.617908 17704 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.589164 17718 consensus_queue.cc:260] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.609681 17660 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:50.620743  5557 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 38: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:50.620978 17660 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:50.621081 17718 raft_consensus.cc:3060] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:29:50.609974 17615 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.013s
I20260709 12:29:50.624027 17659 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:50.494691 17665 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.519721 17703 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.064s
I20260709 12:29:50.533579 17676 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:50.626072 17570 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.626503 17676 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.626960 17665 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:50.551173 17622 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:29:50.627982  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 92593.3us
I20260709 12:29:50.534353 17415 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.551786 17234 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:50.614406  5561 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 36. Current term is 37. Ops: [36.24-36.24]
I20260709 12:29:50.620131 17704 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:50.494222 17504 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.054s
W20260709 12:29:50.590572  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 130314us
I20260709 12:29:50.581409  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:50.566552 17672 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:50.600920 17642 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:50.465236  5554 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:50.628558  5555 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:50.629899  5632 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 32: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:50.651228  5599 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:50.632280 17474 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:50.600262  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:50.639714  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.671844  6276 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 35 because replica has last-logged OpId of term: 32 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 19.
I20260709 12:29:50.602795 17721 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.641273  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:50.679030  6279 raft_consensus.cc:2410] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 32 because replica has last-logged OpId of term: 30 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 21.
I20260709 12:29:50.634387  5602 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.659825  5703 raft_consensus.cc:2410] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 32 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 20.
I20260709 12:29:50.601773 16755 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:50.683769 16755 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:50.635746  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 276024us
I20260709 12:29:50.636412 17675 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:50.675612  6237 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:50.684849 17675 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:50.685379 17675 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:50.689849  5559 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: []
W20260709 12:29:50.690757  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 56442.4us
I20260709 12:29:50.691831  5602 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:50.652580 17672 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:50.700659 17672 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:50.702023  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:50.703119  5703 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 35: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:50.590076 17658 consensus_queue.cc:260] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.706946  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:50.707772  5705 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 35 index: 23. (term mismatch)
I20260709 12:29:50.658234 17713 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.047s
I20260709 12:29:50.696950 17234 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:29:50.710214  5705 pending_rounds.cc:85] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 22
I20260709 12:29:50.708084  5703 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:29:50.713583  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:50.714983  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:50.717626  5705 pending_rounds.cc:107] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 34.23
I20260709 12:29:50.718565  5705 raft_consensus.cc:2889] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:50.719596  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:50.708583  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:50.721002  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:50.721562  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:50.724102  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:50.725175  5709 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:29:50.721778  5703 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:50.726972  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:50.701685 17570 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:50.703634 17703 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:50.730767  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:50.731859  5705 raft_consensus.cc:2393] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 38: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:29:50.745978  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 94871.3us
I20260709 12:29:50.746893  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:50.753902  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 78379.3us
I20260709 12:29:50.755995  6237 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:50.779443  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 236935us
W20260709 12:29:50.782966  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77792us
I20260709 12:29:50.700019  5555 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 38.
I20260709 12:29:50.783746  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:50.790297  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.743332 17734 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:50.793246 17734 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.717481 17577 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:29:50.797765  5524 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.798333 16755 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20260709 12:29:50.803196  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.731442 17241 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:50.805207 17115 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:50.806509 17415 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:50.808908 17665 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:50.809466 17665 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.805167 17241 consensus_queue.cc:1059] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 37.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:29:50.812078 17665 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:50.742954 17234 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:50.788303 17526 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:29:50.828028  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.835763 16755 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
W20260709 12:29:50.836117  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 229280us
W20260709 12:29:50.856402  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 164633us
I20260709 12:29:50.704161 17644 consensus_queue.cc:260] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 36.24, Last appended by leader: 24, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.799265 17734 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:50.816820 17570 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:50.818408 17634 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:50.849098  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:50.883067  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:50.886777 17650 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:50.887910 17650 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:50.870525  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.897495  5676 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:50.891191  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 142587us
W20260709 12:29:50.891203  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:50.797135 17703 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.813144 17629 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.060s
W20260709 12:29:50.914808  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 132489us
I20260709 12:29:50.917475  6234 leader_election.cc:400] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 34. Current term is 35.
W20260709 12:29:50.898192  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 138371us
I20260709 12:29:50.870028 17415 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:50.909238  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:50.821667 17449 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.074s
I20260709 12:29:50.828395 17663 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.078s
I20260709 12:29:50.825511 17577 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.076s
I20260709 12:29:50.879150 17570 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:50.902073 17704 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:50.931042 17415 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.937179 17616 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.938575 17415 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:50.939385 17616 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:50.933079 17570 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:50.944505 17754 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
W20260709 12:29:50.922184  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62945.5us
I20260709 12:29:50.945815  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.946676 17513 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:50.947088 17513 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:50.948053 17513 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.949399 17513 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:50.922288  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:50.895874 17713 raft_consensus.cc:3055] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:29:50.952150 17713 raft_consensus.cc:740] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:50.953038 17713 consensus_queue.cc:260] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.896235 17526 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:50.954267 17526 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:50.955128 17526 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.936290 17644 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:50.957571 17234 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 39.26, Last appended by leader: 26, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:50.963462 17516 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.755s	user 0.011s	sys 0.000s
I20260709 12:29:50.964334  6273 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:29:50.965356 17634 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:29:50.966961 17634 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:29:50.939214 17241 consensus_queue.cc:1048] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:29:50.970192 17672 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:50.970606 17580 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:50.899312 17734 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.073s
I20260709 12:29:50.945019 17721 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:50.909755 17443 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:50.920433  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:50.975694  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 66487.2us
I20260709 12:29:50.916220 17440 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:51.014083 17552 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 }, Status: INVALID_TERM, Last received: 34.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:29:51.015197 17552 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 36 LEADER]: Stepping down as leader of term 36
W20260709 12:29:50.988517  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 232643us
W20260709 12:29:51.017227  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 70236.1us
I20260709 12:29:51.015197 17642 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
W20260709 12:29:51.032709  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:51.040766  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 118532us
I20260709 12:29:50.972290 17672 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:51.044420  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 124101us
I20260709 12:29:51.044937  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:51.045621  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:51.052289  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:51.045532  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:51.036279 17704 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.011s
I20260709 12:29:51.047264  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:51.047948 17711 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:51.052460  5703 raft_consensus.cc:2468] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 38.
I20260709 12:29:51.049021 17443 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:51.031435 17637 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:51.060462  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:51.064716  5701 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:51.047310 17616 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:29:51.076802  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:51.082306  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:51.089216  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:51.090902  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:51.091710  5634 raft_consensus.cc:2393] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 38: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:51.071075  5628 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:29:51.087644  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:51.052078 17449 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:29:51.027503 17516 consensus_queue.cc:1048] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.037s
I20260709 12:29:51.050029 17618 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:51.100090  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:51.137991  5632 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:51.105816  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:51.139477  5636 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 30.
I20260709 12:29:51.140650  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 35 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:51.107169  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:51.097375  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:51.142647  5635 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:51.114579  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:51.155118  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:51.147785  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:51.158030  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:51.015581 17552 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 36 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:51.143257  5524 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:51.175470  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:51.097473 17616 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:51.149031 17718 consensus_meta.cc:287] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.523s	user 0.009s	sys 0.002s
I20260709 12:29:51.055532 17650 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 34.25, Last appended by leader: 25, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:51.149230 17660 consensus_meta.cc:287] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.528s	user 0.005s	sys 0.007s
W20260709 12:29:51.220853 17745 log.cc:927] Time spent T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260709 12:29:51.222671 17658 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:51.096694 17622 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:51.214989  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:51.202502 17777 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:51.220320 17660 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:51.188539  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:51.209493  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:51.194480  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51338.8us
W20260709 12:29:51.242470  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 131766us
W20260709 12:29:51.241681  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 225670us
I20260709 12:29:51.137391 17570 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:51.210749 17580 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:51.096843 17637 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:51.228906 17524 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:51.203599 17616 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:51.203763 17734 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:51.284667  5634 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:51.307251  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:51.318027  5636 raft_consensus.cc:2393] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 37: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:51.285102 17241 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:51.285285  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:51.203873 17759 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:29:51.290040  5599 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:51.293385  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 175494us
I20260709 12:29:51.266637  5632 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 38.
W20260709 12:29:51.234962 17625 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.662s	user 0.008s	sys 0.003s
I20260709 12:29:51.157965 17675 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:51.260707 17731 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:29:51.308386  5704 raft_consensus.cc:2368] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 35. Current term is 36.
I20260709 12:29:51.302171 17778 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:51.341984  5634 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36. Ops: [35.22-35.22]
I20260709 12:29:51.343809  5633 raft_consensus.cc:2410] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 32 because replica has last-logged OpId of term: 30 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 21.
I20260709 12:29:51.361482  6273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47920 (request call id 2147) took 1044 ms. Trace:
I20260709 12:29:51.372752  6273 rpcz_store.cc:276] 0709 12:29:50.316596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:50.316857 (+   261us) service_pool.cc:224] Handling call
0709 12:29:50.317123 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:50.964746 (+647623us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:51.330920 (+366174us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b54005f8f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:51.361382 (+ 30462us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1487637760}
W20260709 12:29:51.384344  5556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.007s	sys 0.048s
W20260709 12:29:51.385805  5556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.007s	sys 0.048s
I20260709 12:29:51.176919 17642 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:51.302412 17524 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:51.329381 17734 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:51.309463 17767 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:51.397961  5709 raft_consensus.cc:2368] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:51.318907 17570 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:51.403384  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 110333us
W20260709 12:29:51.410903  5602 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 33 similar messages]
W20260709 12:29:51.413620  5602 leader_election.cc:336] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:51.414707  5602 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:51.369048 17778 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:51.287144 17777 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:51.376163 17703 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:51.277076 17443 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:51.396692 17504 raft_consensus.cc:2804] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:29:51.299743 17552 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.24, Last appended by leader: 24, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:51.402851 17524 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:51.411362 17734 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:51.382459 17676 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:51.459259 17774 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 12:29:51.410701 17767 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:51.474216  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:51.481865  5631 raft_consensus.cc:1275] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
W20260709 12:29:51.480255  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:51.437077 17778 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:51.415645 17367 consensus_queue.cc:1059] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.064s
I20260709 12:29:51.468017 17444 raft_consensus.cc:3055] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Stepping down as leader of term 34
W20260709 12:29:51.530462 17115 consensus_meta.cc:287] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.724s	user 0.011s	sys 0.000s
I20260709 12:29:51.531136 17637 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:51.531488 17115 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:51.537212 17675 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:51.490707 17644 consensus_meta.cc:287] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.534s	user 0.011s	sys 0.000s
I20260709 12:29:51.505471 17552 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:29:51.526686 17474 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.850s	user 0.008s	sys 0.002s
I20260709 12:29:51.565368 17743 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:51.534873 17513 consensus_meta.cc:287] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.585s	user 0.011s	sys 0.000s
I20260709 12:29:51.543502 17444 raft_consensus.cc:740] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:51.480121 17767 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:29:51.560467 17791 log.cc:927] Time spent T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:29:51.564129 17756 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:29:51.578842 17714 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.959s	user 0.020s	sys 0.033s
W20260709 12:29:51.587040  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:51.571226 17474 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:29:51.562433 17774 log.cc:927] Time spent T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:29:51.592641 17622 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:51.565904 17743 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:51.644758 17743 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:51.648576 17660 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:51.577394 17516 raft_consensus.cc:3055] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:29:51.650477 17516 raft_consensus.cc:740] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:51.634110 17444 consensus_queue.cc:260] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:51.655714 17714 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.011s	user 0.021s	sys 0.033s
I20260709 12:29:51.651675 17516 consensus_queue.cc:260] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:51.680919 17778 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:51.683688 17778 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:51.697847 17767 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:51.673319 17444 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:51.768976 17644 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:51.770236 17644 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:51.733825  5633 raft_consensus.cc:1240] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
W20260709 12:29:51.768870  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 483443us
I20260709 12:29:51.762302 17660 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:51.762581 17516 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:51.761726 17773 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:51.785943 17773 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:51.782737 17622 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:51.832132 17660 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:51.835104 17660 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:51.845289 17773 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:51.844192 17767 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:51.870992  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:51.882377  5635 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:51.886284  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:51.891912  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:51.903347 17642 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:51.905910 17644 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:51.913589 17642 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:51.855003 17507 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.230s	user 0.005s	sys 0.026s
W20260709 12:29:51.917420 17507 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.300s	user 0.005s	sys 0.027s
I20260709 12:29:51.918140  5557 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58854 (request call id 2399) took 1994 ms. Trace:
I20260709 12:29:51.918725  5557 rpcz_store.cc:276] 0709 12:29:49.923447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:49.924303 (+   856us) service_pool.cc:224] Handling call
0709 12:29:51.917464 (+1993161us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b54003a89d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:51.917819 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":731415936}
I20260709 12:29:51.846844 17622 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:51.922508  5526 connection.cc:256] RPC call timeout handler was delayed by -1.037216640999759s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20579999999999998s for a response
W20260709 12:29:51.923549  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:51.923942 17622 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:51.926749 17629 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:51.927171 17771 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:51.928120 17771 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:51.928252 17772 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:51.945942  4356 thread.cc:527] Waited for 1000ms trying to join with rpc worker (tid 5562)
I20260709 12:29:51.931166 17629 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:51.947532 17115 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:51.948516 17115 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:51.949196  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:51.950440  5705 raft_consensus.cc:2410] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 32 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 20.
I20260709 12:29:51.950677 17771 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:51.951179 17629 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:51.951728 17115 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:51.952744 17616 consensus_meta.cc:287] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.594s	user 0.018s	sys 0.007s
I20260709 12:29:51.953229 17642 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:51.953933 17773 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:51.965108 17773 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:51.967825 17773 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:51.954591 17771 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:51.968938 17771 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:51.970335  5602 consensus_peers.cc:597] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:51.971311  5602 leader_election.cc:336] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:51.971688 17771 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:51.972812 17767 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:51.971961  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:51.973558 17767 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:51.978572 17616 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:51.930581 17772 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:51.963888 17769 log.cc:927] Time spent T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.417s	user 0.007s	sys 0.000s
I20260709 12:29:51.984997  5629 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34. Ops: [33.22-33.22]
I20260709 12:29:51.986069  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:51.990336 17616 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:51.991035 17772 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:51.991588  5630 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 36: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:51.992063 17767 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:51.994100  5602 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:51.994999  5602 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:51.995685 17733 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 1.078s	user 0.005s	sys 0.000s
W20260709 12:29:51.997509  5602 leader_election.cc:336] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:51.999454 17767 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:29:51.999706  5602 leader_election.cc:336] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:52.000453 17767 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:52.002118  5602 leader_election.cc:336] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:52.003180  5602 consensus_peers.cc:597] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:52.004017 17767 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:52.004092  5602 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:52.005079  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:52.007519 17616 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:29:52.009797  5602 leader_election.cc:336] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:52.010330 17504 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:52.012430  5552 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:59260 (request call id 1) took 1793 ms. Trace:
I20260709 12:29:52.013566 17714 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:29:52.000011 17777 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:52.017661  5552 rpcz_store.cc:276] 0709 12:29:50.219187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:50.288148 (+ 68961us) service_pool.cc:224] Handling call
0709 12:29:52.011686 (+1723538us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54003c4bd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa678735 00007f4fff638d29 00007f4ff8bb71d4 00007f4ff8bb7162 00007f4ff8bb7091 00007f4ff8bb700c 00007f4ff8bb62c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:52.012081 (+   395us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":421386624}
W20260709 12:29:52.022729  5602 consensus_peers.cc:597] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:52.002444 17577 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:29:52.032857  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 249363us
I20260709 12:29:52.023643 17504 raft_consensus.cc:3060] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:29:52.047276  5602 leader_election.cc:336] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:52.055887  5507 auto_rebalancer.cc:629] Failed to schedule move for tablet 34a1556938b2414c9e883f1659d8fa0d: Illegal state: Replica cea66aa15be3423880843974acdbf50f is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 37 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:29:52.056443  5602 consensus_peers.cc:597] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:52.058297  5602 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:52.060078  5602 consensus_peers.cc:597] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:52.060712  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:52.061271  5602 leader_election.cc:336] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:52.061767  5507 auto_rebalancer.cc:629] Failed to schedule move for tablet 90c64867bae6493c846d941815aa1074: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:52.067548 17763 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:29:52.075266  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:52.075734  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:52.082347  5634 raft_consensus.cc:1240] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37. Ops: [36.17-36.17]
W20260709 12:29:52.084832  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:52.085657 17717 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:52.034832 17767 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:52.092587  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:52.107908  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 43803.6us
I20260709 12:29:52.096566 17763 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.114799  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:52.115727  5709 raft_consensus.cc:2393] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 31: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:52.117735  5560 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49030 (request call id 4415) took 2190 ms. Trace:
I20260709 12:29:52.118162  5560 rpcz_store.cc:276] 0709 12:29:49.926980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:50.203123 (+276143us) service_pool.cc:224] Handling call
0709 12:29:50.203499 (+   376us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:50.203890 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:50.203923 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:50.237255 (+ 33332us) log.cc:844] Serialized 52 byte log entry
0709 12:29:50.660406 (+423151us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:29:50.660470 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:50.660567 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:51.994587 (+1334020us) spinlock_profiling.cc:243] Waited 6.3 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:52.117204 (+122617us) raft_consensus.cc:1685] finished
0709 12:29:52.117228 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:52.117501 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7160,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"spinlock_wait_cycles":14499584,"thread_start_us":382695,"threads_started":1,"wal-append.queue_time_us":562672}
I20260709 12:29:52.120543  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:52.120600 17763 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.121603  5627 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 36.
I20260709 12:29:52.122879  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:52.123337 17763 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:52.123942 17763 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:52.086333 17717 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:52.136651 17658 consensus_meta.cc:287] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.913s	user 0.011s	sys 0.016s
I20260709 12:29:52.137698  5702 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:52.112205 17767 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.146107 17743 consensus_queue.cc:1059] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:52.147254 17767 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:52.148012 17767 raft_consensus.cc:3055] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:52.148494 17655 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.148957  6234 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:52.150374 17717 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:52.158092  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 1.66239e+06us
I20260709 12:29:52.158610  6279 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47934 (request call id 1938) took 1114 ms. Trace:
I20260709 12:29:52.159341 17655 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.112700  5526 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:52.124292 17763 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:52.162550  6279 rpcz_store.cc:276] 0709 12:29:51.044502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:51.044658 (+   156us) service_pool.cc:224] Handling call
0709 12:29:52.158170 (+1113512us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54006379d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:52.158371 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2300384640}
I20260709 12:29:52.166653 17763 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:52.174758 17507 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:29:52.165860  5602 consensus_peers.cc:597] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:52.176585  5602 leader_election.cc:336] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:52.156414 17767 raft_consensus.cc:740] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:52.180878  5632 raft_consensus.cc:1240] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 36. Current term is 37. Ops: []
I20260709 12:29:52.181989  6278 raft_consensus.cc:1240] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 25. Current term is 26. Ops: []
W20260709 12:29:52.183171  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 89190.5us
I20260709 12:29:52.180258 17767 consensus_queue.cc:260] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.184409  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:52.191553  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:52.184664 17772 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:52.194384 17655 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:52.184322  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 4022) took 1075 ms. Trace:
I20260709 12:29:52.204430  6279 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:52.185048  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:52.185712  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:52.184906 17717 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 20, Time since last communication: 0.016s
I20260709 12:29:52.207187  6280 raft_consensus.cc:2410] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 35 because replica has last-logged OpId of term: 34 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 22.
W20260709 12:29:52.209693  5602 leader_election.cc:336] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:52.207363  5634 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 4221) took 1092 ms. Trace:
W20260709 12:29:52.208055  5702 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 4079) took 1501 ms (client timeout 1699 ms). Trace:
I20260709 12:29:52.204704  5632 rpcz_store.cc:276] 0709 12:29:51.109107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:51.374309 (+265202us) service_pool.cc:224] Handling call
0709 12:29:51.374718 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:52.181556 (+806838us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:52.183952 (+  2396us) spinlock_profiling.cc:243] Waited 806 ms on lock 0x7b540009e7d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:52.184257 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1852813184}
W20260709 12:29:52.206532  5704 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42568 (request call id 2104) took 1743 ms (client timeout 1957 ms). Trace:
I20260709 12:29:52.212945  5634 rpcz_store.cc:276] 0709 12:29:51.115500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:51.393206 (+277706us) service_pool.cc:224] Handling call
0709 12:29:52.080141 (+686935us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:52.083109 (+  2968us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:52.207154 (+124045us) spinlock_profiling.cc:243] Waited 683 ms on lock 0x7b54000c1050. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6718ad 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:52.207307 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1570659584}
W20260709 12:29:52.213963  5702 rpcz_store.cc:269] 0709 12:29:50.706509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:50.706638 (+   129us) service_pool.cc:224] Handling call
0709 12:29:52.207871 (+1501233us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b5400451fd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:52.207998 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3283266304}
W20260709 12:29:52.208026 17734 consensus_meta.cc:287] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.716s	user 0.008s	sys 0.017s
I20260709 12:29:52.224524  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:52.222256  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:52.217185  5704 rpcz_store.cc:269] 0709 12:29:50.463424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:50.528724 (+ 65300us) service_pool.cc:224] Handling call
0709 12:29:52.206115 (+1677391us) spinlock_profiling.cc:243] Waited 773 ms on lock 0x7b54001a8954. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:52.206310 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1778202880}
I20260709 12:29:52.235441  6276 pending_rounds.cc:85] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 22
W20260709 12:29:52.237403  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:52.225144 17618 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:52.240777  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:52.235985  6276 pending_rounds.cc:107] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 34.23
I20260709 12:29:52.244110 17772 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:52.242990  6276 raft_consensus.cc:2889] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:52.254153  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:52.256018  5627 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:52.234138 17734 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:52.259568  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 124161us
I20260709 12:29:52.271629  6279 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:29:52.274989  5629 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 4020) took 1166 ms. Trace:
I20260709 12:29:52.275549  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:52.276067  5710 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38460 (request call id 4090) took 1176 ms. Trace:
I20260709 12:29:52.278815  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:52.277014  5629 rpcz_store.cc:276] 0709 12:29:51.108531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:51.316377 (+207846us) service_pool.cc:224] Handling call
0709 12:29:51.316874 (+   497us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:51.990649 (+673775us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:52.270469 (+279820us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b540009ecd4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:52.274898 (+  4429us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1524995328}
I20260709 12:29:52.266382 17713 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:52.283982  5627 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 36. Current term is 37. Ops: [36.24-36.24]
I20260709 12:29:52.285746 17679 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:52.286237 17679 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:52.286592 17679 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:52.287017  6271 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:52.288975 17655 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.257679 17618 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:52.289892 17655 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.292858 17618 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:52.295028  5633 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35630 (request call id 4026) took 1138 ms. Trace:
I20260709 12:29:52.295621  5633 rpcz_store.cc:276] 0709 12:29:51.157171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:51.391819 (+234648us) service_pool.cc:224] Handling call
0709 12:29:51.422858 (+ 31039us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:51.796799 (+373941us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:52.294584 (+497785us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b54003afa54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:52.294933 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":713505024}
I20260709 12:29:52.302626 17504 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.305897  5635 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35630 (request call id 4018) took 1198 ms. Trace:
I20260709 12:29:52.306209 17504 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:52.306377  5635 rpcz_store.cc:276] 0709 12:29:51.107895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:51.147456 (+ 39561us) service_pool.cc:224] Handling call
0709 12:29:52.305637 (+1158181us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x7b5400438354. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:52.305814 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":972696832}
I20260709 12:29:52.310468  5627 raft_consensus.cc:1275] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 36 index: 25. Preceding OpId from leader: term: 38 index: 26. (index mismatch)
I20260709 12:29:52.280400 17767 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:29:52.315146  5601 connection.cc:256] RPC call timeout handler was delayed by -1.1913502980000885s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
I20260709 12:29:52.315990 17577 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:52.318748 17577 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:52.319286  6279 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 24. Preceding OpId from leader: term: 33 index: 25. (index mismatch)
W20260709 12:29:52.320627  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 111737us
I20260709 12:29:52.280447  5710 rpcz_store.cc:276] 0709 12:29:51.045984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:51.049447 (+  3463us) service_pool.cc:224] Handling call
0709 12:29:51.049837 (+   390us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:51.050227 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:51.050261 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:51.050735 (+   474us) log.cc:844] Serialized 26 byte log entry
0709 12:29:51.927557 (+876822us) spinlock_profiling.cc:243] Waited 4.16 ms on lock 0x7b18000eff80. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:29:52.156866 (+229309us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:29:52.162228 (+  5362us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:52.169891 (+  7663us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:52.208649 (+ 38758us) raft_consensus.cc:1685] finished
0709 12:29:52.208672 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:52.222091 (+ 13419us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58401,"raft.queue_time_us":525,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":14677,"spinlock_wait_cycles":9570560,"thread_start_us":172134,"threads_started":1,"wal-append.queue_time_us":1112729}
I20260709 12:29:52.326720 17650 consensus_queue.cc:1048] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:29:52.330842  5627 raft_consensus.cc:1240] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 37. Current term is 38. Ops: [37.21-37.21]
I20260709 12:29:52.332252  5702 raft_consensus.cc:1275] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:29:52.334159  5702 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:29:52.334939 17807 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:52.335255  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:52.335668  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:52.336072  5702 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:52.337227  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:52.335682 17807 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.332373  5708 raft_consensus.cc:1240] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 35. Current term is 36. Ops: [35.21-35.21]
I20260709 12:29:52.342649  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:52.343243 17577 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:52.343359  5701 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:52.342676 17241 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.325373 17618 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.324050 17655 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:52.348742  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:52.348730 17241 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.354189 17650 consensus_queue.cc:1048] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:29:52.354820 17734 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:52.356930 17552 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.730s	user 0.004s	sys 0.024s
I20260709 12:29:52.361728 17790 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:52.361959  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:52.362818  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:52.366376  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:52.366473 17241 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:52.346897 17504 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.354967 17807 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:52.370163  6279 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
W20260709 12:29:52.370612  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55410.4us
I20260709 12:29:52.381568  5599 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:52.376754 17618 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:52.383798 17787 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:52.385776 17618 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:52.388710  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:52.390072 17734 raft_consensus.cc:2804] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:29:52.390733 17734 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:52.391176 17734 raft_consensus.cc:3060] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:52.392845 17552 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.393505 17552 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.393885  5676 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:52.394850 17658 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.395336 17658 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.395810 17552 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:52.396502 17552 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:29:52.397378 17552 raft_consensus.cc:697] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 38 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:52.397485 17658 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:52.398562 17790 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.398525 17552 consensus_queue.cc:237] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:52.399433 17790 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:52.402000 17790 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:52.407356 17524 consensus_meta.cc:287] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.923s	user 0.007s	sys 0.017s
I20260709 12:29:52.410070  5635 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:29:52.410591  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:52.411603  5633 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:52.412930  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:52.420722  5602 leader_election.cc:336] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:52.425828  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
W20260709 12:29:52.433151  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:52.435043  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:52.437229  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:52.437376  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:52.465168  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:52.407905 17524 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:52.470028  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52366.7us
I20260709 12:29:52.474399  5633 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
W20260709 12:29:52.478793  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 65037us
I20260709 12:29:52.471158  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 41 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:52.484714  5601 leader_election.cc:336] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.538s (SENT)
I20260709 12:29:52.485145  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:52.485620  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:52.490113  6279 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:52.482450  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 44 candidate_status { last_received { term: 43 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:52.494076  5704 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 41: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:52.499140 17449 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.499923 17449 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:52.502586  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:52.502688 17449 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:52.493844  5601 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:52.504508  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:52.506903  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 42 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:52.438234 17504 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.520102  6274 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 35. Current term is 36. Ops: []
W20260709 12:29:52.520443 17813 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20260709 12:29:52.520928  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:52.521404  5526 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:52.521827  5635 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
I20260709 12:29:52.523523 17551 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:29:52.524057 17551 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 37 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:52.524751 17660 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.525358 17660 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:52.525143 17551 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 24, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:52.527535 17660 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:52.527812 17772 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:29:52.547385  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 42 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:52.519364 17655 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.552587  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:52.553709  6273 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 41 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 41.
I20260709 12:29:52.520450 17714 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:52.538749 17660 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:52.542965 17772 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 38 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:52.572239  5628 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 38.
I20260709 12:29:52.572465 17577 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
W20260709 12:29:52.594084  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47927.2us
W20260709 12:29:52.597623  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 131486us
I20260709 12:29:52.570951 17763 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:52.602985  6237 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:52.606868  6234 connection.cc:256] RPC call timeout handler was delayed by -1.5895523549997961s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1957s for a response
W20260709 12:29:52.615103  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47696.5us
I20260709 12:29:52.618046  6271 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37. Ops: []
W20260709 12:29:52.624712  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 146084us
I20260709 12:29:52.627007  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:52.629907 17504 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:52.635689  6237 consensus_peers.cc:597] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:52.639711  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:52.638500 17763 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:52.645928  6273 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 32: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:52.654773  6276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:52.643347  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 152742us
I20260709 12:29:52.599319 17655 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.662358  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:52.668083  5633 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:52.666261 17772 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:52.672266  6273 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:52.657470 17444 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:52.635802 17679 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.645171 17516 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:52.605510 17714 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.681511  6279 raft_consensus.cc:2468] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 36.
W20260709 12:29:52.683032  5601 leader_election.cc:336] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.566s (SENT)
I20260709 12:29:52.620782 17660 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.701525  6278 raft_consensus.cc:2368] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32.
W20260709 12:29:52.725998  6234 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:52.725901 17692 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.726884 17692 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.716018 17524 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:52.729830 17692 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:52.715425  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:52.743511 17655 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:52.748811 17714 leader_election.cc:290] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:52.750672  6237 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:52.762881  6275 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 41 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 39 index: 26. Preceding OpId from leader: term: 41 index: 27. (index mismatch)
W20260709 12:29:52.763648  6237 leader_election.cc:336] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:52.764168  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 86044.1us
I20260709 12:29:52.764322  6237 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:52.764954  5601 leader_election.cc:336] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.607s (SENT)
W20260709 12:29:52.766225  5601 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Timed out: RequestConsensusVote RPC to 127.4.65.4:46059 timed out after 1.684s (SENT)
I20260709 12:29:52.766582  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:52.766861  6237 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:52.767185  5450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35600 (request call id 283) took 26642 ms (client timeout 15000 ms). Trace:
I20260709 12:29:52.706391 17516 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.769033 17770 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:52.771571 17660 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:52.772260  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 64403us
W20260709 12:29:52.777104  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 65423.4us
I20260709 12:29:52.761493 17655 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:52.787339  5676 leader_election.cc:336] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:52.790972  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:52.791560  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:52.791837  5629 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:29:52.786274  5628 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:35614 (request call id 4219) took 1690 ms (client timeout 2198 ms). Trace:
I20260709 12:29:52.793279  5524 leader_election.cc:400] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:52.793751  5524 leader_election.cc:403] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:29:52.801273 17655 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.802114  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:52.803709 17655 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:52.792538  5628 rpcz_store.cc:269] 0709 12:29:51.096645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:51.096891 (+   246us) service_pool.cc:224] Handling call
0709 12:29:52.776754 (+1679863us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54001dd8d4. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:52.786190 (+  9436us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3275981568}
I20260709 12:29:52.782754 17787 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:29:52.806641  5676 consensus_peers.cc:597] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:52.809494  5676 consensus_peers.cc:597] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:52.817008 17767 consensus_meta.cc:287] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Time spent flushing consensus metadata: real 0.502s	user 0.011s	sys 0.003s
I20260709 12:29:52.817739  6280 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
W20260709 12:29:52.820868  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 54734.9us
I20260709 12:29:52.807039 17241 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:52.821907  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 195089us
W20260709 12:29:52.811133  5676 consensus_peers.cc:597] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:52.790916 17444 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.022s
I20260709 12:29:52.785905 17734 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:52.824335  6237 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:52.791095 17714 consensus_queue.cc:1048] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.022s
I20260709 12:29:52.815956 17679 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:52.841881  5676 leader_election.cc:336] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:52.812888 17577 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
W20260709 12:29:52.879153  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53735.8us
W20260709 12:29:52.863003  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:52.846760  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 55285.8us
W20260709 12:29:52.888792  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46066.4us
I20260709 12:29:52.882776  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:52.887590  5676 leader_election.cc:336] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:52.897056 17577 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:52.896878 17827 consensus_queue.cc:1059] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 35.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:29:52.897308  5676 leader_election.cc:336] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:52.897698 17577 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:52.847039 17449 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:52.898224 17577 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:52.899586  5601 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:52.904875  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 184146us
I20260709 12:29:52.841879 17241 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:52.906715  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 41 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:52.907611 17241 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:52.908524 17650 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 41 FOLLOWER]: Leader election lost for term 41. Reason: could not achieve majority
I20260709 12:29:52.914058  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:52.915561  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:52.916466  5709 raft_consensus.cc:1240] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:52.916534 17577 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:52.917201  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:52.918031  5708 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:52.920281 17577 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:52.920941  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:52.926904  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:52.927768  5705 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:29:52.929454 17516 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:52.889937  6237 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:52.933038  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:52.933583  6237 leader_election.cc:336] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:52.933959  5706 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
W20260709 12:29:52.934423  6237 leader_election.cc:336] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:52.935155  6237 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:52.937273 17734 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:52.942091 17516 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.942804 17516 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:52.894771  5599 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:52.953989  6237 leader_election.cc:336] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:52.955199  5633 raft_consensus.cc:2468] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 37.
I20260709 12:29:52.957929 17777 raft_consensus.cc:2764] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election decision vote started in defunct term 36: won
I20260709 12:29:52.958230  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:52.959172  5705 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:52.960213  5708 raft_consensus.cc:2468] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
W20260709 12:29:52.973330  5602 leader_election.cc:336] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:52.977311  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 82681.1us
I20260709 12:29:52.979168  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:52.992839  5706 raft_consensus.cc:1240] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:52.996937  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:52.998279  5708 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:29:53.003883  6236 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:53.005942 17692 raft_consensus.cc:2804] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:29:53.006471 17692 raft_consensus.cc:697] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 37 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:29:53.009511  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:53.008831  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 109347us
I20260709 12:29:52.979338 17577 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:52.984903 17790 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:52.984719 17650 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:53.048388  5629 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:53.008379 17717 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:53.052146  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:53.053115  5701 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:53.067584  5705 raft_consensus.cc:2468] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 36.
W20260709 12:29:53.097008  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:53.073271  5601 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:52.767560  5450 rpcz_store.cc:269] 0709 12:29:26.125083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:34.703408 (+8578325us) service_pool.cc:224] Handling call
0709 12:29:52.766924 (+18063516us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:29:44.793717 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:29:44.794380 (+   663us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:29:44.794437 (+    57us) write_op.cc:435] Acquired schema lock
0709 12:29:44.794458 (+    21us) tablet.cc:662] Decoding operations
0709 12:29:44.799117 (+  4659us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:29:44.799216 (+    99us) write_op.cc:641] Partition lock acquired for write op
0709 12:29:44.799245 (+    29us) tablet.cc:685] Acquiring locks for 29 operations
0709 12:29:45.076239 (+276994us) tablet.cc:701] Row locks acquired
0709 12:29:45.076296 (+    57us) write_op.cc:260] PREPARE: finished
0709 12:29:45.076500 (+   204us) write_op.cc:270] Start()
0709 12:29:45.076660 (+   160us) write_op.cc:276] Timestamp: P: 1783600185076432 usec, L: 0
0709 12:29:45.076700 (+    40us) op_driver.cc:348] REPLICATION: starting
0709 12:29:45.183910 (+107210us) log.cc:844] Serialized 11075 byte log entry
0709 12:29:45.858850 (+674940us) op_driver.cc:464] REPLICATION: finished
0709 12:29:47.844788 (+1985938us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ff75680ac 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffe0a8ef0
0709 12:29:48.230730 (+385942us) write_op.cc:301] APPLY: starting
0709 12:29:48.230826 (+    96us) tablet.cc:1367] starting BulkCheckPresence
0709 12:29:49.496282 (+1265456us) tablet.cc:1370] finished BulkCheckPresence
0709 12:29:49.527083 (+ 30801us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:29:50.359405 (+832322us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:29:50.359836 (+   431us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=58,key_file_lookups=58,delta_file_lookups=0,mrs_lookups=0
0709 12:29:50.359894 (+    58us) write_op.cc:312] APPLY: finished
0709 12:29:50.360812 (+   918us) log.cc:844] Serialized 252 byte log entry
0709 12:29:50.776400 (+415588us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:29:50.798421 (+ 22021us) write_op.cc:454] Released schema lock
0709 12:29:50.823450 (+ 25029us) write_op.cc:341] FINISH: Updating metrics
0709 12:29:50.910776 (+ 87326us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:29:50.917012 (+  6236us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1212,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":2371590,"apply.run_cpu_time_us":110403,"apply.run_wall_time_us":2650280,"cfile_cache_hit":118,"cfile_cache_hit_bytes":125347,"num_ops":29,"prepare.queue_time_us":537362,"prepare.run_cpu_time_us":45064,"prepare.run_wall_time_us":628411,"raft.queue_time_us":270632,"raft.run_cpu_time_us":67971,"raft.run_wall_time_us":2284313,"replication_time_us":781963,"spinlock_wait_cycles":82090880,"thread_start_us":3343110,"threads_started":5,"wal-append.queue_time_us":632497,"wal-append.run_cpu_time_us":18088,"wal-append.run_wall_time_us":2474430}]]}
I20260709 12:29:53.104416  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:53.111295  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 489795us
I20260709 12:29:53.112243  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:53.115191  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:53.116044  5705 raft_consensus.cc:2368] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 35. Current term is 36.
I20260709 12:29:53.120585 17692 consensus_queue.cc:237] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 36.26, Last appended by leader: 26, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:53.140902  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 129624us
W20260709 12:29:53.142134  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 261830us
I20260709 12:29:53.139796 17444 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:53.077796 17790 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:53.073336 17650 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:53.138679 17552 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:53.141762 17516 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:53.104878 17717 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:53.144235 17551 raft_consensus.cc:3055] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:29:53.160908 17825 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:53.162106 17552 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:53.167495  5701 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
W20260709 12:29:53.168236  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 56105.3us
I20260709 12:29:53.195509  5599 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:53.206231  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:53.169154 17551 raft_consensus.cc:740] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:53.258556  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 56613us
I20260709 12:29:53.209354 17713 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:29:53.263038  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 215595us
W20260709 12:29:53.273818  5676 leader_election.cc:336] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:53.274514  5675 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:53.209775 17444 consensus_queue.cc:1059] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 33.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.060s
I20260709 12:29:53.265579 17655 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:53.266121 17713 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:53.270283 17241 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:53.314271  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:53.392598  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:53.301267 17713 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:29:53.393642  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 111459us
I20260709 12:29:53.275874 17835 raft_consensus.cc:2764] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election decision vote started in defunct term 37: won
I20260709 12:29:53.394894  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:53.293076 17618 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:53.257468 17825 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:53.413666 17828 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:53.414407  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 140174us
W20260709 12:29:53.420423  6237 consensus_peers.cc:597] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:53.430538  6237 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:53.431327 17850 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.005s
I20260709 12:29:53.435232 17655 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:53.442399 17844 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:53.442997 17844 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:53.443424 17844 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:53.444554 17655 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Leader pre-election won for term 42
I20260709 12:29:53.444986 17655 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:53.445340 17655 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:29:53.451084  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:53.452461 17828 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:53.454114  5602 consensus_peers.cc:597] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:53.454632  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 474977us
W20260709 12:29:53.455247  5602 leader_election.cc:336] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:53.455705 17828 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:53.457621 17825 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:53.436182  6237 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:53.458750  6237 leader_election.cc:336] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:53.459841  6237 leader_election.cc:336] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:53.460748 17846 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:53.461951  5675 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:53.463366 17835 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:53.462899  5451 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 24 to 28. New cstate: current_term: 28 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:53.463816 17655 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 42 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:53.463915 17835 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:53.460757  6237 leader_election.cc:336] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:53.465421  6237 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:53.466189 17655 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 42 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:53.455816  5602 leader_election.cc:336] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:53.466506 17835 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:53.467011  5602 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:53.471148  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 189602us
W20260709 12:29:53.442582  5456 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 180) took 124785 ms (client timeout 15000 ms). Trace:
W20260709 12:29:53.472589  5456 rpcz_store.cc:269] 0709 12:27:48.657556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:56.956206 (+8298650us) service_pool.cc:224] Handling call
0709 12:27:57.913667 (+957461us) spinlock_profiling.cc:243] Waited 4.88 ms on lock 0x7b5400002490. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff754e640 00007f5000425846 00007f500052f90e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc 00007f4ffe9ef392 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:53.442373 (+115528706us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":211401,"reported_tablets":32,"spinlock_wait_cycles":11221888}
I20260709 12:29:53.479442  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:53.480317  5701 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:53.417578 17504 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:53.477063  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 47872.8us
W20260709 12:29:53.485430  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 316501us
I20260709 12:29:53.485819  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:53.397094 17241 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:53.488739  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 32 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:53.488911  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:53.490267  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:53.495518  5708 raft_consensus.cc:2410] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 36 because replica has last-logged OpId of term: 35 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 22.
I20260709 12:29:53.496047  5703 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:53.493460  5451 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 24 to 25, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 25 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:29:53.500979 17449 consensus_meta.cc:287] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.602s	user 0.010s	sys 0.003s
I20260709 12:29:53.501405 17449 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 30. Current term is 31.
I20260709 12:29:53.502146 17552 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:53.502636  5707 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 42 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 41. Current term is 42. Ops: []
I20260709 12:29:53.498659  5451 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 25 to 27, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 27 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:53.507412  5451 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 28 to 30, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 30 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:53.509267  5451 catalog_manager.cc:5697] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:53.396574 17551 consensus_queue.cc:260] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 36.17, Last appended by leader: 17, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:53.469390 17618 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:53.512223  5706 raft_consensus.cc:2410] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 32 because replica has last-logged OpId of term: 30 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 15.
I20260709 12:29:53.529232  5701 raft_consensus.cc:1240] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35. Ops: [32.20-34.21]
W20260709 12:29:53.529343  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 63388.7us
I20260709 12:29:53.533674  5703 raft_consensus.cc:2468] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 37.
I20260709 12:29:53.535480  5627 raft_consensus.cc:1240] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37. Ops: []
I20260709 12:29:53.504199 17650 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:53.543823  5629 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:29:53.546464  5708 raft_consensus.cc:1240] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 35. Current term is 36. Ops: [35.24-35.24]
I20260709 12:29:53.547739  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:53.495659 17655 raft_consensus.cc:2804] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:53.564036  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:53.574558  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:53.575207  5708 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 24. Preceding OpId from leader: term: 33 index: 25. (index mismatch)
I20260709 12:29:53.575532  5628 raft_consensus.cc:2410] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 35 because replica has last-logged OpId of term: 33 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 19.
I20260709 12:29:53.576687  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:53.580268  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 315578us
I20260709 12:29:53.489459 17504 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:53.534966 17692 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:29:53.581429  5676 consensus_peers.cc:597] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:53.589473  5676 leader_election.cc:336] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:53.589859  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:53.609211  5708 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 34 index: 18. (term mismatch)
I20260709 12:29:53.609982  5708 pending_rounds.cc:85] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 17
I20260709 12:29:53.610296  5708 pending_rounds.cc:107] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 33.18
I20260709 12:29:53.610580  5708 raft_consensus.cc:2889] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:53.559314 17790 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:53.620385  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:53.650655  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:53.673811  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 195647us
I20260709 12:29:53.678650  5629 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37. Ops: []
W20260709 12:29:53.678473  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 89025.7us
I20260709 12:29:53.683410  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:53.585973 17717 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:53.697913  5676 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:53.701977  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:53.702791  5634 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:53.702557  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:53.705489  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:53.703651  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 44 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:53.711731  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 166856us
I20260709 12:29:53.723443  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:53.724722  6234 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:53.726629  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 230993us
I20260709 12:29:53.730911  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:53.710179  5676 consensus_peers.cc:597] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:53.696748 17658 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:29:53.736083  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:53.740666  5676 leader_election.cc:336] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:53.716753 17516 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:53.610309 17844 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:53.727105 17655 raft_consensus.cc:697] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 36 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:53.746377  5676 leader_election.cc:336] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:53.754792  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:53.717041 17763 consensus_queue.cc:1059] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.038s
I20260709 12:29:53.760946 17856 raft_consensus.cc:3055] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Stepping down as leader of term 35
W20260709 12:29:53.755669  5676 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:53.783361  5706 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34. Ops: []
W20260709 12:29:53.785111  5676 consensus_peers.cc:597] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:53.789211  5676 leader_election.cc:336] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:53.753679 17787 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:53.793035  5526 connection.cc:256] RPC call timeout handler was delayed by -1.1843197429998327s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1706s for a response
I20260709 12:29:53.791608  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:53.802342  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77706.4us
I20260709 12:29:53.803974  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:53.797904  5526 connection.cc:256] RPC call timeout handler was delayed by -1.1097422879997794s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19740000000000002s for a response
W20260709 12:29:53.796837  5676 consensus_peers.cc:597] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:53.776058 17856 raft_consensus.cc:740] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:53.814353  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:53.791262 17679 raft_consensus.cc:2804] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:29:53.853554  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:53.855967  5635 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
W20260709 12:29:53.856988  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:53.804586 17861 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:53.868866  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:53.809350 17516 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:53.873621  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:53.874256  5635 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
W20260709 12:29:53.875829  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 91277.2us
I20260709 12:29:53.876526  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:53.907148  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 95151.9us
I20260709 12:29:53.910446  5707 raft_consensus.cc:2393] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 39: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:53.816237 17241 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:53.806870 17787 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:53.865073 17618 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:53.877954 17658 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:53.911505 17444 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:53.942082  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:53.942906 17504 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:53.947870  5676 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:53.940107 17449 consensus_queue.cc:1059] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:53.959725 17449 raft_consensus.cc:3055] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:53.960197 17449 raft_consensus.cc:740] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:53.955288  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:53.961354 17863 log.cc:927] Time spent T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:29:53.961884 17844 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:53.948938  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:53.943794 17516 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:53.951035  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:53.910125 17847 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:53.964620 17847 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:53.965023 17847 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:53.961673 17449 consensus_queue.cc:260] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.21, Last appended by leader: 21, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:53.966707  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 473523us
I20260709 12:29:53.955641 17658 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:53.969599 17658 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:53.911558 17679 raft_consensus.cc:697] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:53.941707 17862 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:53.962706  5701 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 39.
I20260709 12:29:53.993232  5709 raft_consensus.cc:1240] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37. Ops: []
I20260709 12:29:53.966569 17449 raft_consensus.cc:3060] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:53.877642 17861 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:53.999152  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 135872us
I20260709 12:29:54.003854  6234 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:54.002379  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 57880us
W20260709 12:29:53.998522  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 271930us
I20260709 12:29:53.990324 17862 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:53.875962 17655 consensus_queue.cc:237] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.015986  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:54.018450 17873 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:54.018922 17842 consensus_queue.cc:1059] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.063s
I20260709 12:29:54.018635 17679 consensus_queue.cc:237] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:53.997961 17835 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:54.023654  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 81880.9us
I20260709 12:29:54.024058 17842 raft_consensus.cc:2804] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:54.002486 17516 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:54.023307 17876 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:54.024494 17866 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:53.975150 17856 consensus_queue.cc:260] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.022347  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:54.024540 17717 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:29:54.017050 17862 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:54.025765 17879 raft_consensus.cc:3055] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:54.025851 17787 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:54.023696 17873 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.015129 17241 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.026s
I20260709 12:29:54.043031  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:54.036289 17858 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.025s
I20260709 12:29:54.047751  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.060601  5705 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:54.035431 17879 raft_consensus.cc:740] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:54.065372 17879 consensus_queue.cc:260] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.26, Last appended by leader: 26, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.066685 17879 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:54.045717  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.067612  5708 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:54.040791  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 27 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.041829  5601 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:54.085124  5601 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:54.058408 17714 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.027s
I20260709 12:29:54.030391 17692 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:29:54.031080 17835 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.087145  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.084642 17717 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:54.090809  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:54.103677  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:54.034415 17866 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:29:54.028704 17877 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:54.091614  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:54.092716  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:54.085597 17867 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:54.053140  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.114732  5707 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 36: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:54.113263  6236 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:54.106580  5636 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:54.094628  5705 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:54.120548  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:54.066677 17873 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.042119 17504 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.054s
I20260709 12:29:54.052708  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 44 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.125345  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.125988  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.127091  5705 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 33: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:54.068992  5701 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
W20260709 12:29:54.059274 17713 consensus_meta.cc:287] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.665s	user 0.000s	sys 0.022s
I20260709 12:29:54.106909 17787 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:54.101428  5634 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
W20260709 12:29:54.099851  5676 leader_election.cc:336] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:54.102608  5601 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:54.136511  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 95806.7us
I20260709 12:29:54.027741 17842 raft_consensus.cc:697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 36 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:54.101527 17872 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:54.101711 17449 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:54.138397  5676 leader_election.cc:336] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:54.138725  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.142962  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.136750  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.143868  5701 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 38 because replica has last-logged OpId of term: 36 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
I20260709 12:29:54.036289 17876 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:54.142843  5676 leader_election.cc:336] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:54.149457  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.150766  5709 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 37.
W20260709 12:29:54.141611  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 137805us
I20260709 12:29:54.153554  5632 raft_consensus.cc:1275] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 36 index: 26. Preceding OpId from leader: term: 37 index: 27. (index mismatch)
I20260709 12:29:54.153798  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.154376  5705 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:54.137967 17516 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:54.157624  6234 leader_election.cc:400] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for earlier term 35. Current term is 36.
W20260709 12:29:54.156708  5676 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:54.143357 17658 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:54.140556 17713 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.147019 17449 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:54.172804  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 81306.7us
I20260709 12:29:54.185604  5701 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 36 index: 24. Preceding OpId from leader: term: 37 index: 24. (term mismatch)
I20260709 12:29:54.141749 17717 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.192938  5708 raft_consensus.cc:3055] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:29:54.194861  5708 raft_consensus.cc:740] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 37 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:54.186626  5701 pending_rounds.cc:85] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 23
I20260709 12:29:54.195483  5701 pending_rounds.cc:107] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 36.24
I20260709 12:29:54.195853  5701 raft_consensus.cc:2889] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:54.197038  5629 raft_consensus.cc:1240] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37. Ops: []
W20260709 12:29:54.197660  5676 leader_election.cc:336] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:54.202732  5676 leader_election.cc:336] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:54.189990 17516 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:29:54.206230  5676 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:54.149207 17872 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:54.214964  6237 leader_election.cc:336] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:54.215771 17822 consensus_queue.cc:1059] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
I20260709 12:29:54.216974 17822 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:54.217085 17872 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.217552 17822 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.219372 17717 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.206797 17847 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.208169 17876 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:54.210290  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:54.210206 17842 consensus_queue.cc:237] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.237133 17847 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.204838  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:54.241287  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:54.226370  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:54.232623 17713 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.232789 17891 raft_consensus.cc:3055] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:29:54.219990 17717 consensus_queue.cc:1059] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.012s
I20260709 12:29:54.234251 17866 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:29:54.234879 17862 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.235162 17516 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:54.262390  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 108329us
I20260709 12:29:54.263511  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:54.264976  5524 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Timed out: RequestConsensusVote RPC to 127.4.65.3:45979 timed out after 1.699s (SENT)
I20260709 12:29:54.265726  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:54.242883 17658 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:29:54.270112  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:54.243582  5708 consensus_queue.cc:260] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 37.21, Last appended by leader: 21, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:29:54.241345  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 56532.5us
I20260709 12:29:54.219609 17822 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.219801 17835 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.274917  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:54.221680 17832 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.949s	user 0.006s	sys 0.000s
I20260709 12:29:54.274823 17879 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:29:54.277171 17879 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:54.277544 17879 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:29:54.277663  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:54.268997 17876 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:54.281265 17692 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:54.282047 17692 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.283239 17642 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 LEADER]: Stepping down as leader of term 41
I20260709 12:29:54.283674 17642 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:54.284459 17692 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.284607 17642 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 41.27, Last appended by leader: 27, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.285820  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:54.286466  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:54.287017  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:54.281283 17880 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:54.288198  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:54.288787  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:54.289448  6278 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:54.289121 17880 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.261396 17891 raft_consensus.cc:740] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:54.289942  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 27 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:54.290654  6275 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:54.290859 17891 consensus_queue.cc:260] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 36.26, Last appended by leader: 26, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.291967 17891 raft_consensus.cc:3060] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:54.292126  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:54.292977  6275 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 36 because replica has last-logged OpId of term: 35 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 25.
I20260709 12:29:54.292886 17880 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.276520 17772 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:54.295608 17772 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:54.297366 17822 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:54.297458 17772 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:54.298956 17862 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.300669  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:54.305156  6237 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:54.305835  6237 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:54.306860  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 27 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:54.288105  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:54.275240 17717 raft_consensus.cc:3055] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:54.309110 17717 raft_consensus.cc:740] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:54.311510 17449 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.312788  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:54.310042 17717 consensus_queue.cc:260] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.24, Last appended by leader: 24, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.313484  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:54.321895  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:54.322677  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:54.326956  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:54.327625 17524 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:54.321412 17861 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.329232  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:54.330663 17847 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:29:54.328420 17524 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:54.332257 17524 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:54.332737 17524 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:54.334789 17870 raft_consensus.cc:3055] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:54.335232 17870 raft_consensus.cc:740] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:29:54.313881  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:54.329115 17770 consensus_queue.cc:1048] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
W20260709 12:29:54.328307  6237 leader_election.cc:336] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:54.319259 17822 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:54.337939  5599 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:54.338719  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.328749  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.322808  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:54.340008  5702 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 37.
I20260709 12:29:54.340184  5629 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 37.
I20260709 12:29:54.285995 17642 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:29:54.337092  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:54.352293  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 142171us
I20260709 12:29:54.314061  5708 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:54.349736  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:54.358875 17879 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:54.359562  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:54.361836 17241 raft_consensus.cc:2804] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:29:54.362562 17873 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:54.373602 17241 raft_consensus.cc:697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:54.373591 17873 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.374576 17241 consensus_queue.cc:237] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.362957 17880 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:54.337205 17717 raft_consensus.cc:3060] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:29:54.376271  5600 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:54.336091 17870 consensus_queue.cc:260] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.320271 17551 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:54.364612  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.365341  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:54.367676  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 265150us
I20260709 12:29:54.368605 17877 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:29:54.370085  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:54.370884 17444 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:54.385262  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:54.379614  5601 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:54.382279  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:54.386950  5702 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:54.400249  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:54.377053 17856 raft_consensus.cc:3060] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:29:54.407500  5602 leader_election.cc:336] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:54.402618  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:54.400159  6237 leader_election.cc:336] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:54.377564 17870 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:54.376343 17873 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.408686  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:54.409054  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 111983us
I20260709 12:29:54.409682  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 37 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:54.410671  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:54.404502  6279 raft_consensus.cc:1240] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: [33.25-33.25]
W20260709 12:29:54.404289  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 139360us
I20260709 12:29:54.404793  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:54.398015 17840 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.787s	user 0.021s	sys 0.018s
I20260709 12:29:54.394199 17770 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:54.413086 17858 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:54.394615 17444 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:54.394330 17660 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:54.429111  5633 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
I20260709 12:29:54.429431  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:54.420143  6237 leader_election.cc:336] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:54.422338  5631 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:54.425305  6280 raft_consensus.cc:2368] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 39. Current term is 40.
I20260709 12:29:54.433225  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:54.433846  5629 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:54.435176  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:54.435288 17449 consensus_queue.cc:1059] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 36.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:29:54.436599  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:54.437811 17888 raft_consensus.cc:3055] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:29:54.438210 17888 raft_consensus.cc:740] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:54.439195 17888 consensus_queue.cc:260] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.429642 17822 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.019s
I20260709 12:29:54.423372 17770 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:54.446014 17770 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:54.447122  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:54.399593 17524 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.447924 17895 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:54.448263 17888 raft_consensus.cc:3060] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:54.448433 17895 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:54.448939 17895 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:54.450601  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:54.423259 17846 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:54.452363 17449 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:54.452833 17449 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:54.453217 17449 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:54.453461  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:54.454561 17873 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:29:54.422016 17840 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.021s	sys 0.019s
W20260709 12:29:54.458076  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 78532.9us
I20260709 12:29:54.454838  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:54.454941 17873 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:54.466275 17873 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:54.467839 17879 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 40 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.469106 17866 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:29:54.469182 17772 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.471601 17772 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:54.473011  5675 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:54.474829  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:54.477505 17524 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:54.477418  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 40 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:54.478451  5701 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:29:54.481005 17844 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:54.483271 17844 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:54.485023  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:54.487277 17444 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.488579 17873 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.490523  5601 leader_election.cc:400] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32.
I20260709 12:29:54.490841  5601 leader_election.cc:403] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20260709 12:29:54.491446 17877 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32.
I20260709 12:29:54.495146 17895 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.497249  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:54.497907  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:54.498917  5702 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:54.501969 17840 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:54.506814  5708 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 37 index: 21. Preceding OpId from leader: term: 38 index: 21. (term mismatch)
I20260709 12:29:54.516307  5708 pending_rounds.cc:85] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 20
W20260709 12:29:54.516521  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 61764.2us
I20260709 12:29:54.511260  6234 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:54.516719  5708 pending_rounds.cc:107] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 37.21
I20260709 12:29:54.517445  5708 raft_consensus.cc:2889] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:54.517910  5526 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:54.513082  5701 raft_consensus.cc:2468] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 40.
W20260709 12:29:54.522410  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:54.513636  6237 leader_election.cc:336] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:54.539172  6234 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:54.541638  6279 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: []
W20260709 12:29:54.566016  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48183.5us
I20260709 12:29:54.516119 17840 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:54.522620 17655 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:54.541281 17504 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:29:54.575166  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:54.578503  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:54.546948 17880 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:29:54.582180  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 172072us
I20260709 12:29:54.550765 17516 raft_consensus.cc:2804] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 40 FOLLOWER]: Leader election won for term 40
W20260709 12:29:54.584898  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 99988.5us
W20260709 12:29:54.587735  5602 leader_election.cc:336] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:54.589012  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.591789  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:54.577709 17504 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:54.581030 17835 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:54.592274  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:54.600435  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:54.593514  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:54.584743 17872 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:54.607587  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:54.608297  5702 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:54.605988  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:54.602999 17504 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:54.588791 17880 raft_consensus.cc:697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:54.620235  5601 leader_election.cc:304] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:54.624536 17787 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:54.625157 17787 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.627385 17787 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.627934 17787 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:54.612150 17835 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:54.629343 17835 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.584860 17827 consensus_queue.cc:1059] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 36.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.025s
I20260709 12:29:54.591449 17516 raft_consensus.cc:697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 40 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
W20260709 12:29:54.639647 17897 log.cc:927] Time spent T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 12:29:54.639918 17516 consensus_queue.cc:237] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.24, Last appended by leader: 24, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.641340 17840 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.642709 17840 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:54.635955 17787 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.630810  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.644227  5708 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:54.613641  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:54.635263  6279 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:54.635246 17858 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:54.612560 17772 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:54.657356  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:54.592734 17891 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.039s
I20260709 12:29:54.612764 17873 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:54.660146  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:54.664501  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 50991.4us
I20260709 12:29:54.613381 17872 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.674108 17872 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.658291 17660 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:54.676997 17660 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:54.677407 17660 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:54.677599  6278 raft_consensus.cc:2410] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 33 because replica has last-logged OpId of term: 32 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 21.
W20260709 12:29:54.677935  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:54.665587  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:54.681097  5702 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 38.
I20260709 12:29:54.616786 17885 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:54.635423 17842 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:54.684438  5708 raft_consensus.cc:1275] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:29:54.685617  5599 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:54.688876  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.690025  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:54.663038 17866 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:54.660740 17852 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:54.708489  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:54.664240 17772 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:54.679082 17577 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.049s
I20260709 12:29:54.711849 17444 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:54.713160 17902 log.cc:927] Time spent T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:29:54.714884  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:54.738561  5701 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:54.707599 17713 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:29:54.719692 17504 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.710165 17827 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:54.664569 17858 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.720180 17880 consensus_queue.cc:237] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 32.24, Last appended by leader: 24, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.721004 17892 raft_consensus.cc:3055] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:29:54.709935 17842 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:54.726305  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:54.726786  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:54.743155  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 155424us
I20260709 12:29:54.692540 17877 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:54.745994  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:54.700381 17790 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:29:54.767056  5602 leader_election.cc:336] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:54.706492 17891 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:54.757931  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:54.737684 17772 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:54.751511 17885 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:29:54.753381 17827 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:54.731945 17655 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 34.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.038s
I20260709 12:29:54.762334 17713 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:54.797896  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:54.756768 17892 raft_consensus.cc:740] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:54.811081 17892 consensus_queue.cc:260] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 36.24, Last appended by leader: 24, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:54.800740  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 69523.6us
I20260709 12:29:54.801761 17504 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.802306 17658 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:29:54.802739  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:54.769867 17877 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:54.770406 17891 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:54.804706  5708 raft_consensus.cc:3060] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:54.808750 17844 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:54.808971 17790 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:54.819590  5701 raft_consensus.cc:2468] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:54.824070 17660 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.824530 17892 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:54.826570 17660 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:54.827670 17842 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.828507 17852 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:54.829012 17852 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.834118 17449 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.818810 17877 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:54.836935 17713 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 36.24, Last appended by leader: 24, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:54.839334  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 106476us
I20260709 12:29:54.795425 17827 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:54.852818  5675 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:54.820781 17891 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:54.821141 17844 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:54.867141  5708 raft_consensus.cc:2468] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:54.871232 17844 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.873190 17844 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:54.873584 17772 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:54.846679 17770 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.877815  6280 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 38: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:54.878494 17770 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:54.879285 17907 log.cc:927] Time spent T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:29:54.850862 17835 consensus_queue.cc:1059] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.034s
I20260709 12:29:54.883328 17835 raft_consensus.cc:3055] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:54.883744 17835 raft_consensus.cc:740] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:54.859628 17660 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:54.884685 17835 consensus_queue.cc:260] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.25, Last appended by leader: 25, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:54.886014 17835 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:54.903039  5676 leader_election.cc:336] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:54.911346 17917 consensus_queue.cc:1059] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:54.912731 17909 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:29:54.914896  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:54.915963 17858 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:54.916999 17919 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:29:54.937417  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.940014  5708 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:54.876080 17772 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:54.925401 17913 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.117s	user 0.003s	sys 0.000s
I20260709 12:29:54.917047 17714 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:29:54.947120  5701 raft_consensus.cc:1275] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
W20260709 12:29:54.948495  5602 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:54.950644  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.951563  5705 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:54.956235  6274 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:54.957023  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:54.960183  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:54.961937  5703 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 38: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:54.960671 17842 consensus_queue.cc:1048] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.005s
I20260709 12:29:54.966676 17655 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:29:54.970546 17852 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:54.974438  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 130088us
I20260709 12:29:54.976455  5526 leader_election.cc:400] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 39. Current term is 40.
I20260709 12:29:54.976821  5526 leader_election.cc:403] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:29:54.978618 17915 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.033s
I20260709 12:29:54.948869 17714 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:54.959733 17449 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:54.983318  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:54.991461  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:55.003065  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:55.006924  5630 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
W20260709 12:29:55.007934  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 93195.7us
W20260709 12:29:55.007881  6237 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:55.016934  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:55.004063  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:55.030498  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:55.013120  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:54.992933 17770 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 41 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:55.066968  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 53999.4us
I20260709 12:29:55.006078 17891 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.074549 17891 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:55.066954 17772 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:55.004407 17449 raft_consensus.cc:3060] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Advancing to term 40
I20260709 12:29:55.075518 17772 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:55.078094 17772 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:55.078779 17892 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:29:55.079551 17892 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:55.080034 17892 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:55.080945  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 67525.8us
I20260709 12:29:55.081846  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:55.082233 17892 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:55.083063  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:29:55.084493  6237 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:55.084859 17925 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:29:55.085613 17925 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:29:55.085980 17449 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Leader pre-election lost for term 39. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 39. Current term is 40.
I20260709 12:29:55.086119 17925 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:55.084530 17917 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:29:55.086615 17770 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 41 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.087149 17925 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:55.088328  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 84383.3us
I20260709 12:29:55.088984 17770 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 42 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:55.090504 17912 log.cc:927] Time spent T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.305s	user 0.001s	sys 0.000s
I20260709 12:29:55.009800  5627 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 32.23, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:29:55.091647 17785 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:55.093405  5629 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:29:55.084256  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:55.097018  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:55.104564  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:54.990244 17852 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.080s
I20260709 12:29:55.108294  5599 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:55.113587  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 42 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:55.114054  6237 leader_election.cc:336] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:55.114517  5635 raft_consensus.cc:2410] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 42 because replica has last-logged OpId of term: 41 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 26.
I20260709 12:29:55.100236 17767 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:55.119154  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:55.120160  5629 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:55.121556  5599 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:55.123626  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 42 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.125797 17859 raft_consensus.cc:2804] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:55.127092 17842 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:29:55.032538 17846 consensus_queue.cc:1048] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.095s
I20260709 12:29:55.127575 17767 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:55.144800  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:55.145717  6272 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:55.145642 17827 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.148147 17827 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:55.148859  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:55.141449 17777 raft_consensus.cc:2804] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:55.154697  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:55.155583  5632 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:29:55.166730  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:55.129176 17877 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:55.115367 17920 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:55.127133  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:55.131652  6280 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:55.125437  5706 raft_consensus.cc:2393] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 42 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 42: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:55.134951  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:55.143697  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:29:55.186110  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 59079.8us
I20260709 12:29:55.187155  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:55.202158  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:55.202939  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:55.203214  6280 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: [34.24-34.24]
I20260709 12:29:55.203130  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:55.204867  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 70046.6us
I20260709 12:29:55.145889 17767 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:55.249815  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:55.250277  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:55.249773  5633 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:29:55.149770 17859 raft_consensus.cc:697] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:55.118652 17785 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.067s
I20260709 12:29:55.253197  6280 raft_consensus.cc:2393] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 35: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:55.247852 17925 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:29:55.263378 17859 consensus_queue.cc:237] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:55.264231 17872 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:29:55.266054  5708 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:29:55.267581 17655 raft_consensus.cc:3055] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:55.268041  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:55.200219 17777 raft_consensus.cc:697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:55.269958  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:55.273960 17895 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:55.249366 17926 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:55.286083 17877 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:55.292708  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:55.294030  5632 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:55.257267  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:55.294997  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.295202  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:55.299917  5628 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 31.
I20260709 12:29:55.296525  5629 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:55.283895  6272 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:55.297286  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 27 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:55.299047  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.253888 17909 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:55.302304  5633 raft_consensus.cc:1275] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:29:55.307003  6279 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:55.307431  5710 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:55.310228  6280 raft_consensus.cc:2393] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 38: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:55.314031  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:55.316562  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:55.317333  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:55.324074  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.325307  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.326488  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 42 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:55.325878  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:55.320986  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 27 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.299301 17785 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:55.330376  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:55.295969 17655 raft_consensus.cc:740] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 33 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:55.329949  5627 raft_consensus.cc:2410] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 42 because replica has last-logged OpId of term: 41 index: 27, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 26.
I20260709 12:29:55.333496  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.333730  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.334196  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:55.335031  6274 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: [32.23-32.23]
I20260709 12:29:55.335059  5629 raft_consensus.cc:2393] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 33: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:55.311897 17660 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:55.331259  5526 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:55.344515  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:55.353055  5526 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:55.354353  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.314769 17909 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:55.339244  5601 consensus_peers.cc:597] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 785e79f6fbd0463aafaca5498918a142. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:55.363515  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 46574.4us
I20260709 12:29:55.362999  5706 raft_consensus.cc:2368] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 39. Current term is 40.
I20260709 12:29:55.362499  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:55.372740  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:55.305778 17835 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.035s
I20260709 12:29:55.370010  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:55.374742  6275 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 36 index: 24. Preceding OpId from leader: term: 38 index: 25. (index mismatch)
I20260709 12:29:55.375301  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.378539  5526 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:55.381883  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:55.389900  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.390949  5705 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 36: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:55.394341  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:55.393357  6279 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:55.396968  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 42 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:55.397540  6275 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 41 FOLLOWER]: Advancing to term 42
I20260709 12:29:55.397910  5710 raft_consensus.cc:2393] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:55.371503 17909 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:55.400898  6274 raft_consensus.cc:1240] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36. Ops: []
I20260709 12:29:55.366979 17449 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:55.400086  5526 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:55.372236 17551 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:55.377043 17892 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:29:55.408324  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:55.408504  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:55.408954  5629 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 37.
I20260709 12:29:55.337212 17790 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.058s
I20260709 12:29:55.383528 17717 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:55.410318  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.409370  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:55.423969  5706 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:55.418527  5633 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:55.412082  5627 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:29:55.412315  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:55.391983 17827 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:29:55.406675 17852 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:55.435806  6279 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 28. Current term is 29. Ops: []
I20260709 12:29:55.436034  5629 pending_rounds.cc:85] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 19
I20260709 12:29:55.436478  5629 pending_rounds.cc:107] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 33.20
I20260709 12:29:55.436796  5629 raft_consensus.cc:2889] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:55.433874  5701 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 32 index: 24. (index mismatch)
I20260709 12:29:55.411588 17892 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:55.437964  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:55.432885  5702 raft_consensus.cc:1275] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 37 index: 22. (index mismatch)
I20260709 12:29:55.406944 17551 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:55.449164  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 121361us
I20260709 12:29:55.407068 17914 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.044s
I20260709 12:29:55.462917  6275 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 42.
I20260709 12:29:55.463994  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:55.442037 17892 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:55.333226 17785 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.446506 17551 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:55.436422 17717 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:55.406215 17655 consensus_queue.cc:260] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:55.474270  5633 raft_consensus.cc:1240] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 33. Current term is 34. Ops: [33.22-33.22]
I20260709 12:29:55.404620 17777 consensus_queue.cc:237] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.21, Last appended by leader: 21, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:55.473150  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:55.444921 17895 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.032s
I20260709 12:29:55.490463  5708 raft_consensus.cc:2393] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 36: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:55.483273  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:55.493052  6272 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:29:55.495091  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:55.499152  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:55.457659 17877 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.034s
W20260709 12:29:55.501096 17934 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.208s	user 0.002s	sys 0.000s
I20260709 12:29:55.511268  5627 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: [32.18-32.18]
I20260709 12:29:55.513053  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.497951 17655 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:55.507699 17933 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260709 12:29:55.559183  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.567950  5636 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: []
W20260709 12:29:55.581207  5673 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:55.586537  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:55.588840  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:55.591049  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 27 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:55.597730 17717 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:55.598578  6279 raft_consensus.cc:1240] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 34. Current term is 35. Ops: [34.24-34.24]
I20260709 12:29:55.599267 17935 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:55.599795 17935 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:55.600255 17895 consensus_queue.cc:1059] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 33.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:29:55.602013 17935 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:55.520737 17922 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:55.603531 17858 raft_consensus.cc:3055] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:55.604251 17922 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:55.605299 17922 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 35.27, Last appended by leader: 27, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.606946  5708 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 36 index: 24. Preceding OpId from leader: term: 38 index: 25. (index mismatch)
I20260709 12:29:55.604276 17858 raft_consensus.cc:740] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:55.616612  5706 raft_consensus.cc:2468] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:55.618937  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:55.618728  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 108745us
I20260709 12:29:55.622051 17926 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:55.620641 17915 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:55.615942 17858 consensus_queue.cc:260] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.622972 17935 consensus_queue.cc:1048] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:55.624125 17858 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:55.622776 17915 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.624534 17926 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:55.624619 17785 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:55.620136  5602 leader_election.cc:336] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:55.625533  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:55.516803 17449 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:55.631470  5602 leader_election.cc:336] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:55.635789  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:55.639334  5525 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:55.637117  5675 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:55.644286 17915 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:55.645406 17935 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:55.646006 17935 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:55.646456 17935 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:55.646534  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.649477 17444 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:55.650511  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:55.650724  5602 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:55.651510  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:55.652796 17915 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:29:55.654637  5601 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:55.625562 17835 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:55.659351  5602 leader_election.cc:336] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:55.659605 17835 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:55.660917 17926 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:55.661249 17785 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:55.661449 17926 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:55.661736 17785 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.662362 17926 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.664772 17785 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:55.666749  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:55.666229 17714 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.668236 17714 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:55.659987  5602 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:55.678826  5602 leader_election.cc:336] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:55.679371  4356 thread.cc:527] Waited for 1000ms trying to join with heartbeat (tid 5586)
I20260709 12:29:55.679278  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:55.680711  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:29:55.681816  5602 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:55.682363  5602 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:55.683676 17660 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:29:55.685043  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:55.662631 17835 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:55.688333  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:55.690649  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 42 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:55.691545  5602 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:55.691956  5602 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:55.697526  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:55.697710  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:55.596197 17767 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.705312  5675 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:55.701560  5602 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:55.714834  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.715003  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:55.713800  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.716050 17840 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:55.695683 17660 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:55.689420 17939 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:55.695621 17642 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:55.695892 17713 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:29:55.727193  5675 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:55.700744 17925 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:55.711156 17658 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:55.711686 17914 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:55.698856 17785 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 42 FOLLOWER]: Leader pre-election lost for term 42. Reason: could not achieve majority
I20260709 12:29:55.713810 17444 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:55.731920  6277 raft_consensus.cc:2368] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33.
I20260709 12:29:55.700621 17844 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:29:55.732302  5602 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:55.712987 17846 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:55.713837 17516 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.024s
I20260709 12:29:55.755261 17787 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:55.756542 17941 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:55.738700 17917 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:29:55.751730  5602 leader_election.cc:336] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:55.776286 17892 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:55.750083 17449 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.777181 17852 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:55.778297 17852 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:55.778510 17892 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:55.780004 17941 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.768949 17846 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.782327 17941 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:55.787514  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:55.788828 17941 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:29:55.740119 17658 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.774950  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:55.791122 17777 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:29:55.791410  5602 leader_election.cc:336] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:55.750685 17920 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:55.795948 17920 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.799595 17920 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:55.760520 17928 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.615s	user 0.006s	sys 0.000s
W20260709 12:29:55.792433  5602 leader_election.cc:336] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:55.792389  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:55.809751  5602 leader_election.cc:336] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:55.827754  5602 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:55.831079  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 105007us
I20260709 12:29:55.733349 17939 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.807502 17892 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:55.836694  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:55.837146  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 44841.6us
I20260709 12:29:55.839372  5524 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:55.841150  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 114088us
W20260709 12:29:55.838316  5676 leader_election.cc:336] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:55.817899 17920 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:55.841820  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:55.846349  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:55.846973  5706 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:29:55.847836  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:55.848454  5710 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:55.842262  5675 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:55.849126  5601 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:55.803180 17551 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:55.849954  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:55.837764 17949 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:55.866657  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:55.863796  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:55.871057  5630 raft_consensus.cc:2410] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 37 because replica has last-logged OpId of term: 35 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 20.
I20260709 12:29:55.873567  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:55.882108  5676 consensus_peers.cc:597] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:55.884202  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:55.887161  5705 raft_consensus.cc:1275] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:29:55.853818 17642 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:29:55.859045 17844 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:29:55.884888  5627 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
I20260709 12:29:55.852856 17790 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:29:55.865474 17787 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:55.870144 17929 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:55.875814 17835 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:55.916491  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:55.917078  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 67308.7us
I20260709 12:29:55.913729  5599 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:55.919695  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:55.920495  5632 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:55.917292  5631 raft_consensus.cc:2468] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 38.
I20260709 12:29:55.880189 17877 consensus_queue.cc:1048] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.036s
I20260709 12:29:55.853304 17920 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:55.909965  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 68648.7us
I20260709 12:29:55.874614 17655 consensus_queue.cc:1059] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 21, Time since last communication: 0.040s
I20260709 12:29:55.875100 17949 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.907321 17917 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 42 FOLLOWER]: Leader election won for term 42
W20260709 12:29:55.950765  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 49791.5us
W20260709 12:29:55.943985  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 82899.6us
I20260709 12:29:55.944434  5524 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:55.906131 17449 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.032s
I20260709 12:29:55.977717  5632 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:55.981225  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:55.976939  5676 leader_election.cc:336] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:55.984830  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 60093.2us
I20260709 12:29:55.909744 17909 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:55.997049  5706 raft_consensus.cc:1275] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 32 index: 24. Preceding OpId from leader: term: 34 index: 25. (index mismatch)
I20260709 12:29:55.944818 17872 raft_consensus.cc:2804] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:55.991493  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:55.919248 17859 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.062s
I20260709 12:29:55.919442 17948 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.079s
I20260709 12:29:55.975848 17954 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:55.915941 17787 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:55.983912 17922 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:56.021261  5707 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 32: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:55.983951 17777 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:56.026178 17917 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 42 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:56.027441 17947 log.cc:927] Time spent T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
W20260709 12:29:56.028704 17945 log.cc:927] Time spent T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.204s	user 0.000s	sys 0.002s
I20260709 12:29:56.030144 17939 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:56.036068  5710 raft_consensus.cc:2468] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:29:56.032106 17954 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:56.014256 17660 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:56.012729 17935 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:56.028704 17946 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
W20260709 12:29:56.038570  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51239.1us
I20260709 12:29:56.067214  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:55.988072 17926 consensus_queue.cc:1059] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 35.27, Next index: 28, Last known committed idx: 26, Time since last communication: 0.045s
W20260709 12:29:56.050835 17951 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
W20260709 12:29:56.069833  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 150478us
I20260709 12:29:56.044628 17872 raft_consensus.cc:697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:56.074819  5599 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.056595 17964 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:56.065279 17925 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:56.059686 17954 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:56.090214  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 99467.4us
I20260709 12:29:56.083197  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.083504  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:56.079555  5676 leader_election.cc:336] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:56.091166  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:56.043946 17777 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.073567 17551 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.081089 17655 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:56.107475  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 133036us
I20260709 12:29:56.122325  5524 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.080626 17660 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:56.084154 17877 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:56.105530 17840 consensus_queue.cc:1048] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.022s
I20260709 12:29:56.075609 17449 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.044s
I20260709 12:29:56.078833 17939 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.043s
I20260709 12:29:56.153515  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:56.154537  6277 raft_consensus.cc:2410] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 37 because replica has last-logged OpId of term: 35 index: 26, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 25.
I20260709 12:29:56.133934  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.088363 17964 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.144793 17968 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:29:56.146476 17965 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:56.075011 17956 consensus_queue.cc:1059] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 33.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.056s
W20260709 12:29:56.173059  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 89620.8us
W20260709 12:29:56.191941  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 58098.1us
I20260709 12:29:56.203204  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.148151 17972 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:29:56.153573 17944 log.cc:927] Time spent T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.282s	user 0.000s	sys 0.003s
I20260709 12:29:56.156477 17679 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:56.208861  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 86970.4us
I20260709 12:29:56.129362 17655 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.216828 17949 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.218184 17926 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:56.220252  5629 raft_consensus.cc:1275] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 36 index: 21. (term mismatch)
I20260709 12:29:56.245453  5629 pending_rounds.cc:85] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 20
I20260709 12:29:56.245826  5629 pending_rounds.cc:107] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 35.21
I20260709 12:29:56.246160  5629 raft_consensus.cc:2889] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:56.199023 17965 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:56.229411 17950 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.229s	user 0.000s	sys 0.002s
W20260709 12:29:56.192775  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54431.3us
I20260709 12:29:56.156015 17967 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:56.250490  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:56.253584  6237 consensus_peers.cc:597] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:56.241140  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:56.137311 17660 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.198328 17969 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:56.248565 17926 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:56.263363 17967 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.277572 17926 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.278136 17964 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.279068 17926 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:56.279255 17967 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.268484  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.248777 17679 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.269223 17949 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:56.281939 17949 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:29:56.282334 17949 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:56.282546 17679 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.282727 17949 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:56.233237 17956 consensus_queue.cc:1243] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer 4a1c5e6dd8e249bf82d38f90fa37b43d log is divergent from this leader: its last log entry 34.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
W20260709 12:29:56.284289  6237 consensus_peers.cc:597] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:56.275429 17655 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:56.242630 17920 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:56.265897  5627 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:56.287452  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:56.153575 17917 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 39.26, Last appended by leader: 26, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.288962  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:56.274796 17660 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:56.249039 17972 raft_consensus.cc:2804] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:29:56.293215  5628 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:29:56.252967 17965 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:56.295013  5630 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 34: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:56.296187  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:56.311545  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 108420us
I20260709 12:29:56.313118  5524 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:56.300379 17952 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.283s	user 0.001s	sys 0.000s
I20260709 12:29:56.315742  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:56.321559 17964 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:56.322005 17964 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:56.322362 17964 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:56.209975 17872 consensus_queue.cc:237] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.298070 17449 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:56.302297 17939 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:56.306073 17972 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:56.342562  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.332851 17846 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.346711  5524 leader_election.cc:400] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Vote denied by peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979) with higher term. Message: Invalid argument: T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 39. Current term is 40.
I20260709 12:29:56.347569  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.336516 17717 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:56.353238 17960 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:56.353806 17960 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:29:56.355827 17852 consensus_meta.cc:287] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.577s	user 0.013s	sys 0.000s
I20260709 12:29:56.356288 17852 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.364796 17983 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:56.365401 17983 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.366539 17852 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.342049 17972 raft_consensus.cc:3060] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:29:56.367631 17959 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20260709 12:29:56.368424 17959 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:56.372562 17717 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.373246 17767 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.375114 17717 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.334998 17984 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:56.382406  5701 raft_consensus.cc:1275] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:29:56.374744 17954 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:56.359033 17858 consensus_meta.cc:287] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Time spent flushing consensus metadata: real 0.734s	user 0.006s	sys 0.006s
I20260709 12:29:56.378233 17787 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:56.394877 17914 consensus_queue.cc:1048] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:56.395705  5635 raft_consensus.cc:2410] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 36 because replica has last-logged OpId of term: 34 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 23.
W20260709 12:29:56.359978 17892 consensus_meta.cc:287] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.518s	user 0.010s	sys 0.000s
I20260709 12:29:56.397708 17892 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:56.398701 17973 raft_consensus.cc:3055] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Stepping down as leader of term 35
W20260709 12:29:56.398538  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 48657us
I20260709 12:29:56.400657 17909 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:56.399199 17973 raft_consensus.cc:740] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:56.403987  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 56474.7us
I20260709 12:29:56.408735  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.406248 17892 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:29:56.415289  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 181050us
I20260709 12:29:56.417882  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:29:56.419091  5676 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:56.424822  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.425112  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:56.422355  5676 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 183 similar messages]
W20260709 12:29:56.443624  5676 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:56.451819  5701 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:29:56.453560  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:56.454362  5701 raft_consensus.cc:2393] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 33: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:56.455832  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:56.461632  5676 leader_election.cc:336] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:56.464181  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:56.464848  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:56.465029  5706 raft_consensus.cc:2376] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33. Re-sending same reply.
I20260709 12:29:56.467760  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:56.469215  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:56.469946  5706 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:56.433531 17939 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:29:56.429189 17956 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:29:56.477725  5676 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:56.473266  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:56.481419  5701 raft_consensus.cc:1240] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 37. Current term is 38. Ops: [37.24-37.24]
I20260709 12:29:56.482790  5701 raft_consensus.cc:3055] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:56.483202  5701 raft_consensus.cc:740] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:29:56.442624 17827 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.991s	user 0.009s	sys 0.020s
I20260709 12:29:56.428925 17984 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:56.433894 17922 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:56.490804  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:56.491647  5627 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:56.492756  5706 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 38.
W20260709 12:29:56.496290 17955 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.003s	sys 0.023s
W20260709 12:29:56.496742 17955 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.003s	sys 0.024s
I20260709 12:29:56.476944 17964 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.500728 17986 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:29:56.502362 17964 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.503123 17983 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:56.489950 17827 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.046s	user 0.009s	sys 0.020s
I20260709 12:29:56.512421  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:56.513103  5706 raft_consensus.cc:2393] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 36: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:56.441881 17973 consensus_queue.cc:260] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 35.27, Last appended by leader: 27, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.495669  5701 consensus_queue.cc:260] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.23, Last appended by leader: 23, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.516222  5701 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:56.511605 17827 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:56.528872 17827 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.532557 17984 raft_consensus.cc:3055] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:56.519884 17956 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:56.534453 17984 raft_consensus.cc:740] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:56.535259 17956 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.535389 17984 consensus_queue.cc:260] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.536655 17984 raft_consensus.cc:3060] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:29:56.537042 17996 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:29:56.537128 17993 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:56.537742 17993 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.526221 17965 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.542338 17965 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.543426 17993 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:56.544862  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:56.545151  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:56.545634  5706 raft_consensus.cc:2468] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:29:56.546111  5526 leader_election.cc:400] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33.
I20260709 12:29:56.546507  5526 leader_election.cc:403] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
I20260709 12:29:56.548379  5709 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:56.563926  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:56.564981  5632 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:56.569416  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:56.546609 17717 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:56.546679 17960 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:56.574368 17717 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.575941 17995 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:56.546713 17991 log.cc:927] Time spent T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
I20260709 12:29:56.576687 17995 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:56.579546  5675 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:56.550271 17892 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:56.550927 17965 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:56.587809  5701 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 33 index: 23. (term mismatch)
I20260709 12:29:56.549499 17954 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.590739  5701 pending_rounds.cc:85] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 22
I20260709 12:29:56.593972  5701 pending_rounds.cc:107] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 32.23
I20260709 12:29:56.594388  5701 raft_consensus.cc:2889] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:56.592689  5706 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:56.579200 17995 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:56.569957 17955 raft_consensus.cc:3060] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:56.596585  5632 raft_consensus.cc:2468] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 33.
I20260709 12:29:56.593357 17965 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:56.599888 17990 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260709 12:29:56.553107 17993 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:56.598232  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:56.606016  6236 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:56.607347 17954 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:56.607854 17954 raft_consensus.cc:697] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:56.608548 17965 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.608727 17954 consensus_queue.cc:237] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.613507  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:56.614506  5701 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
I20260709 12:29:56.599207 17949 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.615710  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:56.547324 17972 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.620882 17993 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.630666  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:56.632884  5627 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 38.
I20260709 12:29:56.634554  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:56.646488  5634 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 36: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:29:56.648155 17984 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:56.648861 17984 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.649432 17972 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.651419 17984 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:56.652746 17717 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:56.654170  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:56.654994  5628 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:29:56.656139 17914 raft_consensus.cc:2955] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Committing config change with OpId 32.24: config changed from index -1 to 24, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
W20260709 12:29:56.657470 17844 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 33 current term: 33 is higher.
I20260709 12:29:56.627826 17960 raft_consensus.cc:2804] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:56.663033 17960 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:56.663447 17960 raft_consensus.cc:3060] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:56.647444  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:56.673148  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:56.674374  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:56.675956  5629 raft_consensus.cc:2393] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate cea66aa15be3423880843974acdbf50f in this term.
I20260709 12:29:56.677438  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:56.678242  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:56.677441  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:56.679873  4356 thread.cc:527] Waited for 2000ms trying to join with heartbeat (tid 5586)
I20260709 12:29:56.680327  6279 raft_consensus.cc:1240] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: []
I20260709 12:29:56.681227  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:56.684278  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:56.685678  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:56.686849  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:56.688838  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:56.692901  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:56.693897  5634 raft_consensus.cc:1240] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 34. Current term is 35. Ops: []
I20260709 12:29:56.694265  6277 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
W20260709 12:29:56.681540  5602 leader_election.cc:336] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:56.694983  5602 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:56.696573  5602 consensus_peers.cc:597] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:56.701685  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:56.702842 17787 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:56.706079 17993 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:56.707509 17949 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.709677 17658 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:56.720613  6234 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:56.727746 17914 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:54228 (request call id 1) took 1728 ms. Trace:
I20260709 12:29:56.728888 17949 consensus_queue.cc:1048] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:29:56.730563 17955 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33.
I20260709 12:29:56.731389 17954 raft_consensus.cc:2804] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:29:56.732565 17993 consensus_queue.cc:1059] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:29:56.737723 17954 raft_consensus.cc:697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:56.738644 17954 consensus_queue.cc:237] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:56.739084  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 253075us
I20260709 12:29:56.742913  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:56.744580 17926 consensus_queue.cc:1048] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:29:56.736330  5634 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: []
I20260709 12:29:56.746054 17968 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:29:56.747022  5704 raft_consensus.cc:1275] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 40 index: 25. (index mismatch)
W20260709 12:29:56.743319  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 113218us
I20260709 12:29:56.750425  6274 raft_consensus.cc:1240] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 32. Current term is 33. Ops: [32.23-32.23]
I20260709 12:29:56.752108  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:56.754491  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:56.758458  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:56.758862  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:56.759972  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:56.760565 17844 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:56.740914 18003 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.020s
I20260709 12:29:56.762281  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:56.764261  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:56.778573  5706 raft_consensus.cc:2410] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 for term 36 because replica has last-logged OpId of term: 34 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 23.
I20260709 12:29:56.761390 17925 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:56.761699 17772 raft_consensus.cc:3055] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:29:56.764541 17968 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:29:56.766412  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:56.767423  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:56.768138  5675 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:56.768857  5629 raft_consensus.cc:1240] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: []
I20260709 12:29:56.770414 17935 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:56.772683  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:56.772931  5602 consensus_peers.cc:597] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:56.786638  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:56.788863  5602 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:56.774291  5701 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 36.
W20260709 12:29:56.715979  5451 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 183) took 56027 ms (client timeout 15000 ms). Trace:
I20260709 12:29:56.735540 17960 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:56.790331  5602 consensus_peers.cc:597] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:56.806422  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.808706  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:56.809808  5704 raft_consensus.cc:3060] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:56.782277 17772 raft_consensus.cc:740] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:56.814384  5709 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 33: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:29:56.812141  5602 leader_election.cc:336] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:56.815042  5602 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:56.816632  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:56.813050 17955 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:56.817503  6277 raft_consensus.cc:2393] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 34: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
W20260709 12:29:56.818755  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 528677us
I20260709 12:29:56.819723  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.820771  6274 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:56.821162  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:56.822054 17679 consensus_queue.cc:1048] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
W20260709 12:29:56.826020  5602 consensus_peers.cc:597] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:56.814944  5524 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.831080 17859 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:56.831238 17967 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:56.832736 17967 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:56.833122 17967 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:56.835358  6274 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:29:56.835916 17949 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:29:56.839008  5602 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:56.841236  6274 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 35 index: 27. Preceding OpId from leader: term: 36 index: 28. (index mismatch)
I20260709 12:29:56.842291  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:29:56.790714  5451 rpcz_store.cc:269] 0709 12:29:00.715672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:10.980626 (+10264954us) service_pool.cc:224] Handling call
0709 12:29:56.715871 (+45735245us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:29:54.207178 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:29:54.207876 (+   698us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:29:54.207926 (+    50us) write_op.cc:435] Acquired schema lock
0709 12:29:54.207945 (+    19us) tablet.cc:662] Decoding operations
0709 12:29:54.208715 (+   770us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:29:54.208805 (+    90us) write_op.cc:641] Partition lock acquired for write op
0709 12:29:54.208832 (+    27us) tablet.cc:685] Acquiring locks for 5 operations
0709 12:29:54.210020 (+  1188us) tablet.cc:701] Row locks acquired
0709 12:29:54.210087 (+    67us) write_op.cc:260] PREPARE: finished
0709 12:29:54.210413 (+   326us) write_op.cc:270] Start()
0709 12:29:54.210609 (+   196us) write_op.cc:276] Timestamp: P: 1783600194210221 usec, L: 0
0709 12:29:54.210648 (+    39us) op_driver.cc:348] REPLICATION: starting
0709 12:29:54.211559 (+   911us) log.cc:844] Serialized 2024 byte log entry
0709 12:29:54.453068 (+241509us) op_driver.cc:464] REPLICATION: finished
0709 12:29:54.622337 (+169269us) write_op.cc:301] APPLY: starting
0709 12:29:54.622456 (+   119us) tablet.cc:1367] starting BulkCheckPresence
0709 12:29:54.626110 (+  3654us) tablet.cc:1370] finished BulkCheckPresence
0709 12:29:54.626222 (+   112us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:29:54.788625 (+162403us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:29:54.854306 (+ 65681us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=10,key_file_lookups=10,delta_file_lookups=0,mrs_lookups=0
0709 12:29:54.854380 (+    74us) write_op.cc:312] APPLY: finished
0709 12:29:54.876753 (+ 22373us) log.cc:844] Serialized 57 byte log entry
0709 12:29:54.876965 (+   212us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:29:54.877072 (+   107us) write_op.cc:454] Released schema lock
0709 12:29:54.877134 (+    62us) write_op.cc:341] FINISH: Updating metrics
0709 12:29:54.877645 (+   511us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:29:54.877673 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":466355,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":151956,"apply.run_cpu_time_us":10012,"apply.run_wall_time_us":255291,"cfile_cache_hit":22,"cfile_cache_hit_bytes":25027,"num_ops":5,"prepare.queue_time_us":431172,"prepare.run_cpu_time_us":42605,"prepare.run_wall_time_us":181052,"raft.queue_time_us":93591,"raft.run_cpu_time_us":1294,"raft.run_wall_time_us":91457,"replication_time_us":242232,"thread_start_us":570895,"threads_started":4,"wal-append.queue_time_us":144118,"wal-append.run_cpu_time_us":5450,"wal-append.run_wall_time_us":1524646}]]}
I20260709 12:29:56.846689  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.811820 17920 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:56.850131  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:56.847527 17847 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:56.851111  5675 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:56.851642 17847 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.854590  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:56.881546  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:56.855329  5524 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:56.884792  5676 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:29:56.813175 18003 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.042s
I20260709 12:29:56.895179  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
I20260709 12:29:56.899008  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:29:56.902484  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 47973.4us
I20260709 12:29:56.890719 17984 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 26, Time since last communication: 0.013s
W20260709 12:29:56.906042 17977 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.632s	user 0.000s	sys 0.004s
I20260709 12:29:56.856925 17985 raft_consensus.cc:3060] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:56.882659 17827 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:56.925866 17929 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:29:56.926586 17929 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:56.927039 17929 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.893028 17920 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:56.928824 17920 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.931034 17920 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:56.734071 17914 rpcz_store.cc:276] 0709 12:29:54.999789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:55.008001 (+  8212us) service_pool.cc:224] Handling call
0709 12:29:55.093384 (+ 85383us) log.cc:844] Serialized 485 byte log entry
0709 12:29:56.653849 (+1560465us) spinlock_profiling.cc:243] Waited 3.47 ms on lock 0x7b54001da798. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa62e7bc 00007f4ffa6312c0 00007f4ffa631248 00007f4ffa6311c8 00007f4ffa631183 00007f4ffa63044c 00007f50004dd2f6 00007f50004d7aa8 00007f4ffa219b41 00007f4ffa20cf2a
0709 12:29:56.727423 (+ 73574us) log.cc:844] Serialized 14 byte log entry
0709 12:29:56.727643 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48310,"raft.queue_time_us":58598,"raft.run_cpu_time_us":1856,"raft.run_wall_time_us":8546,"spinlock_wait_cycles":7973632,"thread_start_us":117240,"threads_started":1,"wal-append.queue_time_us":195741}
I20260709 12:29:56.934774  5704 raft_consensus.cc:2468] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 34.
I20260709 12:29:56.888833 17717 consensus_queue.cc:1059] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 34.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.052s
I20260709 12:29:56.936692 17717 raft_consensus.cc:3055] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:29:56.936621 17984 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:56.937172 17717 raft_consensus.cc:740] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 37 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:56.938033 17717 consensus_queue.cc:260] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.938807  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:56.939218 17717 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:56.939569 17984 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:29:56.939961 17984 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:56.940304 17984 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:56.944159 17960 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:56.949530 17772 consensus_queue.cc:260] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:56.951717 18016 ts_tablet_manager.cc:933] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:56.952280  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:56.953037 17995 raft_consensus.cc:2804] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:56.961102  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:56.951282 17984 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:56.858006 17917 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.139s
W20260709 12:29:56.993273  5676 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:56.996647  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:56.991870  5631 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 31. Current term is 32. Ops: [31.18-31.18]
I20260709 12:29:56.989792  6277 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 42 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 39 index: 26. Preceding OpId from leader: term: 42 index: 27. (index mismatch)
I20260709 12:29:56.997153  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:57.015758  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:56.893359 17993 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:57.017495  5627 raft_consensus.cc:2393] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 33: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:57.018419  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 43 candidate_status { last_received { term: 41 index: 27 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:57.015930  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:57.024434  5635 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:29:56.953510 17965 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election won for term 36
W20260709 12:29:56.999521 17951 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:29:56.901530 17449 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:29:57.040596 17449 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 38 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:29:57.041937  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:57.041570 17449 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 36.17, Last appended by leader: 17, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:57.021363  5633 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 41 index: 27. Preceding OpId from leader: term: 42 index: 27. (term mismatch)
I20260709 12:29:57.047472  5633 pending_rounds.cc:85] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting all ops after (but not including) 26
I20260709 12:29:57.047781  5633 pending_rounds.cc:107] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: Aborting uncommitted NO_OP operation due to leader change: 41.27
I20260709 12:29:57.048581  5633 raft_consensus.cc:2889] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:29:57.029570  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51738.9us
I20260709 12:29:56.965675 17985 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:29:57.031092 17984 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:56.965013 17859 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:57.025784  6279 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 42 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 42.
I20260709 12:29:56.968784 17827 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:56.992535 17777 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 43 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:57.038031  6234 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:57.070950  5701 raft_consensus.cc:1275] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:29:57.071578 17770 raft_consensus.cc:2804] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:29:57.072427 17770 raft_consensus.cc:697] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:57.073196 17957 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:57.073324 17770 consensus_queue.cc:237] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:57.073917 17957 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:57.038791 17965 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:57.048074 17993 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:57.056393 17929 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:57.057104 17973 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:56.991153 17847 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:57.062387 17844 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:29:57.060312 17967 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:57.063318  5707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38460 (request call id 4138) took 1640 ms (client timeout 1733 ms). Trace:
I20260709 12:29:57.076331 17957 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:57.097277  5526 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:57.076857  5701 raft_consensus.cc:2955] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Committing config change with OpId 32.24: config changed from index -1 to 24, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } } }
I20260709 12:29:57.086380  5636 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40390 (request call id 2513) took 1223 ms. Trace:
I20260709 12:29:57.100440  5636 rpcz_store.cc:276] 0709 12:29:55.863155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:55.863281 (+   126us) service_pool.cc:224] Handling call
0709 12:29:55.863526 (+   245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:29:55.912145 (+ 48619us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:55.912176 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:29:55.912219 (+    43us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:29:55.912249 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:57.086133 (+1173884us) spinlock_profiling.cc:243] Waited 48.4 ms on lock 0x7b54001f2554. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa672e01 00007f4ffa67183e 00007f4fff636419 00007f4ff8badbd4 00007f4ff8badb62 00007f4ff8bada91 00007f4ff8bada0c 00007f4ff8baccc2 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:29:57.086178 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:57.086296 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":111347072}
I20260709 12:29:57.076973 17859 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:57.091995 17844 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 31 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:57.102902  5634 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:29:57.102783 17844 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:57.103809 17844 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:29:57.104063 17859 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:57.078478 17965 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:57.105404 17772 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:57.080188  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:57.108757 17915 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:29:57.110198 17915 consensus_queue.cc:1048] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:29:57.083474 17993 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:57.112104  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:57.094009 17967 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:57.114600  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:57.116106 17767 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:57.118752  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 43 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:57.047778  5676 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:57.092311  5707 rpcz_store.cc:269] 0709 12:29:55.423376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:55.423503 (+   127us) service_pool.cc:224] Handling call
0709 12:29:57.062967 (+1639464us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b54002f4c54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa6762fa 00007f4fff637b17 00007f4ff8bb26d4 00007f4ff8bb2662 00007f4ff8bb2591 00007f4ff8bb250c 00007f4ff8bb17c2 00007f4ff89ad48c 00007f4ff89ac8c6 00007f4ff89ac26f
0709 12:29:57.063254 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1339329792}
I20260709 12:29:57.123801  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:57.124634  6279 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:57.119918  5676 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:57.126113  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 78367.2us
W20260709 12:29:57.126291  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 138106us
W20260709 12:29:57.131295  5676 consensus_peers.cc:597] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:29:57.127563  6237 leader_election.cc:336] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:57.135730  6237 leader_election.cc:336] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:57.136504  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:57.139593  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:57.140869  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:57.123865  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:57.120656  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:57.137445  5636 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 29.
I20260709 12:29:56.964686 17995 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:56.965214 17968 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:57.177382  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:57.186962  6279 raft_consensus.cc:2368] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30.
W20260709 12:29:57.179023  6237 consensus_peers.cc:597] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:57.180148  5709 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 18. (term mismatch)
W20260709 12:29:57.183553  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 63019.5us
I20260709 12:29:57.203483  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:57.204111  5709 pending_rounds.cc:85] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 17
I20260709 12:29:57.204658  5709 pending_rounds.cc:107] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 32.18
I20260709 12:29:57.205056  5709 raft_consensus.cc:2889] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:57.206338  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:57.152324 17909 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:29:57.204375  5706 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 38
I20260709 12:29:57.219660  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:57.213045  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:57.221312  5636 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:29:57.221642  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:57.218994  5507 auto_rebalancer.cc:629] Failed to schedule move for tablet 733a5b54e4dd4f86999821705033529f: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:57.217691  6237 leader_election.cc:336] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:57.158330 17935 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:29:57.175330 17917 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:57.224460 17975 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:57.238065  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:57.187645 17995 raft_consensus.cc:3060] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:57.182045 17859 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Leader pre-election won for term 43
I20260709 12:29:57.222283  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 35 index: 26 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:57.299999  5707 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:57.156522 17980 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.069s
I20260709 12:29:57.231942  5456 catalog_manager.cc:5697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 30 to 37. New cstate: current_term: 37 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:57.298581  5701 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:29:57.185835 17955 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:57.218937 17984 raft_consensus.cc:3060] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:57.207700 17993 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:57.219635 17909 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:57.305394  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:57.306030  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:57.330936  5627 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:57.230659 17987 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:29:57.306746  5456 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 31 to 35, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 35 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:57.308606  5706 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 38.
I20260709 12:29:57.226176 17959 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:29:57.306638  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:57.231289 17940 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:29:57.311995  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58303.5us
I20260709 12:29:57.334522  5632 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:29:57.335695  5456 catalog_manager.cc:5697] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 34 to 38, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 38 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:29:57.303313 17551 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.008s	sys 0.036s
I20260709 12:29:57.344854  5629 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 15. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:29:57.302259 17859 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:29:57.326575 17946 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:29:57.339797  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:57.342022  5456 catalog_manager.cc:5697] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:57.359161  5456 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 29 to 30, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 30 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:57.360845  5456 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 30 to 32. New cstate: current_term: 32 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:57.362185  5456 catalog_manager.cc:5697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 33 to 34, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 34 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:29:57.339633 17551 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.009s	sys 0.041s
I20260709 12:29:57.367920  5632 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:29:57.362694  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:57.335067 17909 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:57.381515 17827 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:29:57.385005 17827 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:29:57.371253  5456 catalog_manager.cc:5697] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 31 to 38, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 38 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:57.390283  5456 catalog_manager.cc:5697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 32 to 34. New cstate: current_term: 34 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:57.302204 17975 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:57.329815 17963 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.050s
I20260709 12:29:57.354166 17859 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:29:57.392727  5456 catalog_manager.cc:5697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 34 to 36, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 36 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:57.426468  5456 catalog_manager.cc:5697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 34 to 35, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 35 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:57.326218 17955 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:57.429350 17988 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:57.430030 17988 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:57.431409 17955 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:57.418150 18022 log.cc:927] Time spent T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.169s	user 0.000s	sys 0.001s
I20260709 12:29:57.418119 17975 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:57.421479 18031 log.cc:927] Time spent T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260709 12:29:57.422065 18019 log.cc:927] Time spent T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
I20260709 12:29:57.351397 18003 consensus_queue.cc:1059] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 34.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.049s
I20260709 12:29:57.451965 18003 raft_consensus.cc:3055] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:57.424571 17859 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 43 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:57.436246  5636 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:29:57.428234  5456 catalog_manager.cc:5697] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 25 to 35. New cstate: current_term: 35 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:57.334424 17993 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:57.452067 18021 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:29:57.452490 18003 raft_consensus.cc:740] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:57.373072  5627 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 38.
I20260709 12:29:57.433704 17988 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:57.455260  5456 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 29 to 31. New cstate: current_term: 31 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:57.498502  5456 catalog_manager.cc:5697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 26 to 33. New cstate: current_term: 33 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:57.503219  5456 catalog_manager.cc:5697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 32 to 34. New cstate: current_term: 34 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:57.506215  5456 catalog_manager.cc:5697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 28 to 36, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 36 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:57.508175  5456 catalog_manager.cc:5697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 29 to 32, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 32 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:57.537740  5456 catalog_manager.cc:5697] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 27 to 37. New cstate: current_term: 37 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:57.496136 17915 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:57.540149  5456 catalog_manager.cc:5697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 19 to 29, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 29 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:57.574692  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:57.584709  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 357204us
I20260709 12:29:57.585165  5710 raft_consensus.cc:1275] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 33 index: 21. (term mismatch)
I20260709 12:29:57.571142  5456 catalog_manager.cc:5697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 30 to 33, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 33 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:57.585703  5710 pending_rounds.cc:85] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 20
I20260709 12:29:57.585996  5710 pending_rounds.cc:107] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 30.21
I20260709 12:29:57.586279  5710 raft_consensus.cc:2889] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:57.595836  5456 catalog_manager.cc:5697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 30 to 32. New cstate: current_term: 32 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:57.598142  5456 catalog_manager.cc:5697] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 33 to 35. New cstate: current_term: 35 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:57.602063  5456 catalog_manager.cc:5697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 29 to 38, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 38 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:57.510390 18016 tablet_copy_client.cc:323] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Beginning tablet copy session from remote peer at address 127.4.65.2:38587
I20260709 12:29:57.578948 17965 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:57.578787 17241 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:57.578918 17859 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 43 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:57.622907 17449 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.007s	sys 0.028s
I20260709 12:29:57.580163 17915 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:57.634610  5456 catalog_manager.cc:5697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 35 to 36, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 36 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:29:57.649616  5705 raft_consensus.cc:1275] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
W20260709 12:29:57.630190 17449 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.008s	sys 0.028s
I20260709 12:29:57.650370  5456 catalog_manager.cc:5697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 32 to 36, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 36 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:57.667927  5707 raft_consensus.cc:2468] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 37.
I20260709 12:29:57.665421  5456 catalog_manager.cc:5697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 28 to 31. New cstate: current_term: 31 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:57.672560  5456 catalog_manager.cc:5697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 20 to 36. New cstate: current_term: 36 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:57.638484 17967 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.033s
I20260709 12:29:57.674496  5456 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 27 to 28, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 28 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:29:57.677023  5601 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:57.680442  4356 thread.cc:527] Waited for 3000ms trying to join with heartbeat (tid 5586)
I20260709 12:29:57.608760 18003 consensus_queue.cc:260] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:57.629379 17965 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:57.672046 17772 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.561s	user 0.007s	sys 0.006s
W20260709 12:29:57.714310  5704 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.025s	sys 0.103s
W20260709 12:29:57.716109  5704 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.025s	sys 0.104s
I20260709 12:29:57.631345 17241 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:57.714941 17859 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:29:57.726156 18025 log.cc:927] Time spent T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.224s	user 0.001s	sys 0.000s
I20260709 12:29:57.676555 17909 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:57.718598 17984 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:57.722203 17975 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:57.735396 18003 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:57.736761 17859 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:57.738117 17993 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:57.756031 17973 consensus_meta.cc:287] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.672s	user 0.006s	sys 0.006s
I20260709 12:29:57.763913 17859 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:29:57.771111 18032 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
W20260709 12:29:57.790323 18041 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 12:29:57.758061 17975 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:57.788291 17717 consensus_meta.cc:287] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.847s	user 0.010s	sys 0.006s
I20260709 12:29:57.812152  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:57.813935 17965 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:57.814575  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:57.815269  6279 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:57.815459 17241 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:57.816673  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 43 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:57.817241  6279 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:29:57.825428  6279 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 43.
I20260709 12:29:57.831430  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:57.833460 17241 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:57.833828 17915 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:57.833904 17241 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:57.834542 17241 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:29:57.825692 18020 log.cc:927] Time spent T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.512s	user 0.003s	sys 0.000s
I20260709 12:29:57.826200  5708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 2162) took 1008 ms. Trace:
I20260709 12:29:57.827450 17975 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:57.828292 17909 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:57.859354  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:57.860038  5707 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:29:57.849539  5708 rpcz_store.cc:276] 0709 12:29:56.818020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:56.820248 (+  2228us) service_pool.cc:224] Handling call
0709 12:29:56.820650 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:29:56.883311 (+ 62661us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:56.883344 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:29:56.884312 (+   968us) log.cc:844] Serialized 26 byte log entry
0709 12:29:57.446039 (+561727us) spinlock_profiling.cc:243] Waited 7.9 ms on lock 0x7b1800257ec0. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7567e07 00007f50003c29a9 00007f4ff758469b 00007f4ff758293f 00007f4ff7584c02 00007f4ffa20c20b 00007f4ffa211d23 00007f4ffa2120d1 00007f4ffa62da0f
0709 12:29:57.733039 (+287000us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:29:57.733080 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:57.733143 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:57.825870 (+ 92727us) raft_consensus.cc:1685] finished
0709 12:29:57.825892 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:57.826136 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70817,"raft.queue_time_us":66123,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":8775,"spinlock_wait_cycles":18160896,"thread_start_us":286789,"threads_started":1,"wal-append.queue_time_us":910258}
I20260709 12:29:57.858053  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:57.856691  5601 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
W20260709 12:29:57.849213 17995 consensus_meta.cc:287] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.535s	user 0.008s	sys 0.004s
I20260709 12:29:57.869755  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 43 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:57.839573 17859 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:57.895789 17975 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 43 FOLLOWER]: Leader election won for term 43
I20260709 12:29:57.898572 17975 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 43 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:57.899500 17975 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 39.26, Last appended by leader: 27, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:57.919503 17859 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:57.896585 17995 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:57.927496 17995 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:57.931082  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:57.953074 17241 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:57.975498 17241 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:57.975695  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:57.976306  6280 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:57.985620  6280 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:57.987160  5601 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:29:58.000881  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:58.015914  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.024423  5704 raft_consensus.cc:2410] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 37 because replica has last-logged OpId of term: 35 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 21.
I20260709 12:29:58.015556  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:58.014712  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.026455  5708 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:29:58.029997  5628 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 35 index: 16. Preceding OpId from leader: term: 36 index: 17. (index mismatch)
W20260709 12:29:58.030601  5527 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:58.030682  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:58.031523  5708 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:29:58.032230  6237 leader_election.cc:336] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:58.033448  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:58.036994  5646 tablet_copy_service.cc:140] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Received BeginTabletCopySession request for tablet 785e79f6fbd0463aafaca5498918a142 from peer 38736cc545eb4389b220a3bd21d3b466 ({username='slave'} at 127.0.0.1:40390)
I20260709 12:29:58.037768  5646 tablet_copy_service.cc:161] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Beginning new tablet copy session on tablet 785e79f6fbd0463aafaca5498918a142 from peer 38736cc545eb4389b220a3bd21d3b466 at {username='slave'} at 127.0.0.1:40390: session id = 38736cc545eb4389b220a3bd21d3b466-785e79f6fbd0463aafaca5498918a142
I20260709 12:29:58.038321 17980 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:29:58.038944 17980 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:58.039448 17980 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:29:57.988204 17241 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:29:57.996912 17948 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.053684  5706 raft_consensus.cc:1275] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:29:58.004966 17713 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:58.055276  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:58.055948  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:58.056588  5631 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:58.058452 18035 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:29:58.058575  6277 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 43 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 42. Current term is 43. Ops: [42.27-42.27]
I20260709 12:29:58.058946 17713 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.061275  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:58.062980  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:58.067888  5675 leader_election.cc:400] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30.
I20260709 12:29:58.061628 17948 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.071516 17714 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:58.072080 17714 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:58.072626 17714 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:58.073289 17948 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:58.087963  5704 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 35 index: 27. Preceding OpId from leader: term: 36 index: 28. (index mismatch)
W20260709 12:29:58.064384 18057 log.cc:927] Time spent T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:29:58.092963  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.096103  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 35 index: 27 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:58.097927  5676 leader_election.cc:336] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:58.095050  6234 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:58.054889 17909 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.035s
I20260709 12:29:58.064927 17241 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:29:58.076476 17995 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.128507  5708 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:58.130672  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:58.142112 17847 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:29:58.143549  5507 auto_rebalancer.cc:629] Failed to schedule move for tablet 458143f1c77141febe7d9725625617c2: Illegal state: Replica 4a1c5e6dd8e249bf82d38f90fa37b43d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 35 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:58.147535 17827 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:58.147811  5524 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:58.101809 18050 consensus_queue.cc:1048] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.036s
I20260709 12:29:58.111809 18051 consensus_queue.cc:1048] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.050s
I20260709 12:29:58.092286 17948 consensus_queue.cc:1059] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 39.26, Next index: 27, Last known committed idx: 26, Time since last communication: 0.035s
I20260709 12:29:58.126966 18040 raft_consensus.cc:2804] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:29:58.159897  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:58.149910  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 58290.9us
I20260709 12:29:58.161005  5702 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42568 (request call id 2164) took 1274 ms. Trace:
I20260709 12:29:58.223052  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:58.223711  5636 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:29:58.236299  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.186182 17847 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:58.227460  5629 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 34 index: 21. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:29:58.227383  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:58.149355 17713 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:58.230065  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:58.236701  5702 rpcz_store.cc:276] 0709 12:29:56.886684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:56.887099 (+   415us) service_pool.cc:224] Handling call
0709 12:29:56.887462 (+   363us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:56.901666 (+ 14204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:56.901703 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:56.908410 (+  6707us) log.cc:844] Serialized 52 byte log entry
0709 12:29:57.074886 (+166476us) spinlock_profiling.cc:243] Waited 6.58 ms on lock 0x7b48001701d8. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4fff59baa1 00007f4fff592b37 00007f4fff5929ca 00007f4fff68a26f 00007f4fff689f37 00007f4fff69e55c 00007f4fff69e508 00007f4fff69e488 00007f4fff69e443 00007f4fff69d6ec 00007f4ffe5c7a96
0709 12:29:57.252275 (+177389us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:29:57.324371 (+ 72096us) log.cc:844] Serialized 14 byte log entry
0709 12:29:57.327043 (+  2672us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:57.327176 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:58.154934 (+827758us) spinlock_profiling.cc:243] Waited 7.43 ms on lock 0x7b18000b4e80. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff75853f3 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
0709 12:29:58.160829 (+  5895us) raft_consensus.cc:1685] finished
0709 12:29:58.160849 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:58.160954 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37670,"prepare.queue_time_us":111393,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":41479,"raft.run_cpu_time_us":3135,"raft.run_wall_time_us":153279,"spinlock_wait_cycles":32236032,"thread_start_us":343423,"threads_started":1,"wal-append.queue_time_us":472615}
I20260709 12:29:58.231731  5705 raft_consensus.cc:1240] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 35. Current term is 36. Ops: []
W20260709 12:29:58.243336  6237 leader_election.cc:336] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:58.126582 17995 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:58.195093  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 47361.3us
I20260709 12:29:58.148175 17827 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:58.218516 17985 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:29:58.235598 17940 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.148643 17955 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.095s
I20260709 12:29:58.149103 17998 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.093s
I20260709 12:29:58.148499 17449 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.239315 17968 raft_consensus.cc:3055] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 42 LEADER]: Stepping down as leader of term 42
I20260709 12:29:58.240126  5706 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:29:58.235944 17993 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:58.158612  5676 leader_election.cc:336] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:58.218369 17859 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:58.245002  6237 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:58.136502 17957 consensus_queue.cc:1048] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.102s
I20260709 12:29:58.235782 17892 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.241353  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:29:58.279307  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 120721us
W20260709 12:29:58.292820  5676 leader_election.cc:336] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:58.293926  5676 leader_election.cc:336] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:58.294374  5676 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:58.309793 17948 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.254196 17968 raft_consensus.cc:740] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 42 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:58.316818 17984 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:58.310657 17772 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.067s
I20260709 12:29:58.242861 17241 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 34.24, Last appended by leader: 24, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.388885  5631 raft_consensus.cc:2468] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 35.
I20260709 12:29:58.287859 17993 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:58.287467 17940 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:58.352443 17960 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:58.290719 17859 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:58.361001 18061 log.cc:927] Time spent T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.196s	user 0.001s	sys 0.000s
I20260709 12:29:58.285902 17449 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.310261 17892 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:58.383781  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 136517us
I20260709 12:29:58.366114 17985 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 38 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:58.347968 17714 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:58.401604  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 160333us
I20260709 12:29:58.351186 17948 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:58.392863 17917 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:58.414203  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:58.411473 18060 log.cc:927] Time spent T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
I20260709 12:29:58.389137 17847 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.409874 17917 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.418780 18069 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.034s
I20260709 12:29:58.450313 17917 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:58.451113 17993 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:58.441941  5646 tablet_copy_source_session.cc:215] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:29:58.444556 18046 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:58.421041 18068 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.015s
I20260709 12:29:58.411546 17985 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 36.24, Last appended by leader: 24, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:58.451193 18069 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:29:58.436713 17827 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:58.458240 17449 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:58.458849  5676 leader_election.cc:336] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:58.459299 17714 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:58.460777 17892 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:58.462342 17948 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:58.454911 17995 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:58.407454 17968 consensus_queue.cc:260] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 42.27, Last appended by leader: 27, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.463689 18046 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:58.456734 18003 consensus_queue.cc:1059] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 35.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:29:58.466082 18003 raft_consensus.cc:3055] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:29:58.466660 18003 raft_consensus.cc:740] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:58.467619 18003 consensus_queue.cc:260] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.23, Last appended by leader: 23, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:58.470309  5676 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:58.463725 17859 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:58.466125 18046 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:58.481707  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.482915  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.488927  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.476182 18003 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:58.494228  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.504948 18074 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.505617 18074 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:58.506569 18059 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.372s	user 0.003s	sys 0.020s
I20260709 12:29:58.508088 17995 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.508577 18074 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:58.508997  5710 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 32 index: 18. (term mismatch)
I20260709 12:29:58.508594 17995 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.509370  5710 pending_rounds.cc:85] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 17
I20260709 12:29:58.509636  5710 pending_rounds.cc:107] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 31.18
I20260709 12:29:58.509876  5710 raft_consensus.cc:2889] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:58.470360 17940 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:58.510876 17995 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:58.510685 17892 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:29:58.491499 17980 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.513582 17980 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:58.489585 17859 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:58.516947 17859 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.519541 17859 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:58.522084  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 353483us
W20260709 12:29:58.524516  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 281184us
I20260709 12:29:58.525000 18050 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:29:58.525612  6237 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:58.526130  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:29:58.527302  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 110635us
I20260709 12:29:58.531487  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.532697  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:29:58.536612  6237 leader_election.cc:336] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:58.497737 18069 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.542686  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:29:58.544643  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:58.556273  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:58.562805  5675 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:58.574373  5675 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:58.534556 18040 raft_consensus.cc:697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 37 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:29:58.539335 17957 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:29:58.596969  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 50742.6us
I20260709 12:29:58.597764  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:58.561055 17717 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:58.581440 17960 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:58.608250  5526 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:58.583281 17917 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:58.584565 17909 raft_consensus.cc:2804] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election won for term 35
W20260709 12:29:58.615975  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 46525.3us
I20260709 12:29:58.596654 17772 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.544737 18069 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.560683 17940 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.048s
W20260709 12:29:58.627035  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44676us
I20260709 12:29:58.608295 18050 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:58.641044  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:58.647271  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:58.652323  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:58.612483 18068 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:29:58.680732  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 49738us
I20260709 12:29:58.685045  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:58.686600  5630 raft_consensus.cc:1240] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 33. Current term is 34. Ops: [33.18-33.18]
I20260709 12:29:58.686614  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:58.686947  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:58.689975  5526 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:58.603641 17984 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.079s
I20260709 12:29:58.704888  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:58.705789  5633 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 32.
I20260709 12:29:58.706837  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:58.705210  4356 thread.cc:527] Waited for 4000ms trying to join with heartbeat (tid 5586)
I20260709 12:29:58.617219 17892 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.681648 17717 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:29:58.635957 17960 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:58.692904 17852 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
W20260709 12:29:58.712232 18080 log.cc:927] Time spent T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:29:58.681005 17980 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:58.693027 17985 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:58.703926 17909 raft_consensus.cc:697] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:58.687433 17955 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.079s
I20260709 12:29:58.748092  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:58.688892 17772 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.727943  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:58.729319 18016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785e79f6fbd0463aafaca5498918a142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:58.754060  5635 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 36.
I20260709 12:29:58.744417  5630 raft_consensus.cc:1240] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 34. Current term is 35. Ops: [34.20-34.20]
I20260709 12:29:58.749100  5636 raft_consensus.cc:2368] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30.
W20260709 12:29:58.722358  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 144478us
I20260709 12:29:58.715416 18074 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:58.691687 18050 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:58.769377  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 185468us
I20260709 12:29:58.775532  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:58.716171 17892 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:58.790335  5675 leader_election.cc:400] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30.
I20260709 12:29:58.790723  5675 leader_election.cc:403] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 29 election: Cancelling election due to peer responding with higher term
I20260709 12:29:58.730108 17852 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:58.795950 17852 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:58.797103 17844 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:29:58.797638 17844 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:58.798053 17844 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:58.800570 17917 consensus_queue.cc:1059] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:29:58.801573 17917 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.802047 17917 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:58.809092 17929 raft_consensus.cc:3055] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:29:58.809626 17929 raft_consensus.cc:740] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Follower/Learner. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:29:58.810645  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.810729 17929 consensus_queue.cc:260] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.815392  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:58.816720  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.817534  5701 raft_consensus.cc:2393] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 36: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
W20260709 12:29:58.829439 18077 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
W20260709 12:29:58.830253  5507 auto_rebalancer.cc:629] Failed to schedule move for tablet 637a73babedf4b1aa47b3b50937db9ef: Illegal state: Replica 4a1c5e6dd8e249bf82d38f90fa37b43d is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 35 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:29:58.830693 17909 consensus_queue.cc:237] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:58.832654 18037 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:58.731426 17980 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:58.865391  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 75217.9us
I20260709 12:29:58.865478  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:58.726315 17960 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:58.709735 18040 consensus_queue.cc:237] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 35.26, Last appended by leader: 26, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:58.888973  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:58.891032  5628 raft_consensus.cc:1240] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 43 LEADER]: Rejecting Update request from peer f613ba929b1046d3b5aed909414a95bf for earlier term 42. Current term is 43. Ops: [42.27-42.27]
I20260709 12:29:58.872459 17940 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.812254 17929 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:29:58.905535  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 312142us
W20260709 12:29:58.916671  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51319.3us
I20260709 12:29:58.917469  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:58.919554  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.873271 17973 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:29:58.921984  6237 consensus_peers.cc:597] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:58.922811  6237 consensus_peers.cc:597] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:58.833623 18037 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:29:58.928900  5602 consensus_peers.cc:597] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:58.914126 18086 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.940752 18086 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:58.915457 17940 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.945485 17940 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:58.945569 17892 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:58.946834 17772 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:58.949203 17917 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:58.952817  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.953549  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.954109  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:58.954236 18037 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:58.928093 17954 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:58.893079 18076 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:58.968690  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 51325.4us
I20260709 12:29:58.974009  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:58.977304 17892 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.921885 17948 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:58.918380 17915 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:58.920420 18075 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:29:58.969241  5676 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:58.979705  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:58.993877  5634 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
W20260709 12:29:58.989768  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 216991us
W20260709 12:29:58.996867  5676 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:58.979470 18068 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:29:58.978791 17955 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:59.010385  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:29:58.989764 17940 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.955364 17954 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:58.994613 17827 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:58.980630 17984 consensus_queue.cc:1059] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.032s
I20260709 12:29:59.024783  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:59.007202 18069 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:58.969139 18076 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:59.009222 18068 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:59.028179  5636 raft_consensus.cc:2393] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 35: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:59.021098 17940 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:59.039517  5627 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 36 index: 24. Preceding OpId from leader: term: 38 index: 25. (index mismatch)
I20260709 12:29:58.988293 17892 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:58.997246 17915 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:59.017817 18091 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:59.017489 18066 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:59.054428 17984 raft_consensus.cc:3055] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:29:58.997502 17948 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:59.027623 17963 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:29:59.046861 18068 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:59.049880 18069 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:59.025378 17827 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:59.208029  5705 raft_consensus.cc:2410] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 36 because replica has last-logged OpId of term: 35 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 21.
I20260709 12:29:59.173648 17984 raft_consensus.cc:740] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:29:59.182093 17787 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:59.205284 17909 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:59.172274 18091 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:59.244714 17844 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:59.173808 18066 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:59.261946 17844 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:59.262076 18066 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:59.262612 17844 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:59.263104 17844 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:59.234766 17993 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.029s
I20260709 12:29:59.244450 17787 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:59.266561 17787 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:59.257817 17948 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:59.253404 18064 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.850s	user 0.002s	sys 0.006s
I20260709 12:29:59.205714 18069 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:59.296965 18069 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:59.297443 17965 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:59.298146 18102 log.cc:927] Time spent T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:29:59.298082 17965 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:59.298460 17940 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:59.300014 17827 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:59.300248 17965 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:59.301146 18098 log.cc:927] Time spent T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.245s	user 0.001s	sys 0.000s
W20260709 12:29:59.329115  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 154565us
I20260709 12:29:59.330359  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:59.337359  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:59.338048  5635 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:29:59.336144  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:59.345906  5632 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:35614 (request call id 4307) took 1304 ms. Trace:
W20260709 12:29:59.353229  5602 leader_election.cc:336] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:59.354466  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:59.354543  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:59.254737 17909 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:59.346400  5632 rpcz_store.cc:276] 0709 12:29:58.041813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:58.041976 (+   163us) service_pool.cc:224] Handling call
0709 12:29:58.057805 (+ 15829us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:29:58.065155 (+  7350us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:29:58.065204 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:29:58.077562 (+ 12358us) log.cc:844] Serialized 52 byte log entry
0709 12:29:58.243749 (+166187us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:29:58.291342 (+ 47593us) log.cc:844] Serialized 14 byte log entry
0709 12:29:58.291731 (+   389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:29:58.291986 (+   255us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:29:59.023329 (+731343us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x7b4400022b4c. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ff740ddb1 00007f4ff74abc85 00007f4ff74ac6a2 00007f4ffa21e1db 00007f4ffa219aaf 00007f4ffa20d86e 00007f4ffa20cbb6 00007f4ffa20c7d8 00007f4ffa216221 00007f4ffa2161d9 00007f4ffa216169
0709 12:29:59.330063 (+306734us) raft_consensus.cc:1685] finished
0709 12:29:59.340486 (+ 10423us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:29:59.345839 (+  5353us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33110,"prepare.queue_time_us":41873,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":1810,"raft.queue_time_us":11498,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":16153,"spinlock_wait_cycles":2865408,"thread_start_us":129661,"threads_started":1,"wal-append.queue_time_us":260581}
I20260709 12:29:59.361934  5702 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
W20260709 12:29:59.362385  5674 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:29:59.363435  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:59.366329  5702 raft_consensus.cc:2393] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 37: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:59.366598  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:59.367739  5708 raft_consensus.cc:2410] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f for term 37 because replica has last-logged OpId of term: 36 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 24.
I20260709 12:29:59.327325 17827 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:59.335130 18038 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:59.339956 17965 raft_consensus.cc:3060] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:29:59.339655 18113 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:29:59.392230  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:59.395936  5702 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37. Ops: []
W20260709 12:29:59.398005  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44656.7us
W20260709 12:29:59.409427  5602 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:59.411386  5602 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:59.411988  5602 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:59.377002 17948 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:29:59.420717  5702 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 36 index: 24. Preceding OpId from leader: term: 38 index: 25. (index mismatch)
I20260709 12:29:59.421917  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:59.422405  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.423137  6280 raft_consensus.cc:2393] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 33: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:29:59.423024  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:59.423894  5702 raft_consensus.cc:2393] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 38: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:29:59.336069 17984 consensus_queue.cc:260] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:59.430753  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:59.434333  5709 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:29:59.413031 17787 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:59.437897  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1af549c25a724a2f9d4f65cc1c59f22a" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.437578  6280 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 43 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 39 index: 26. Preceding OpId from leader: term: 43 index: 27. (index mismatch)
W20260709 12:29:59.441154 18106 log.cc:927] Time spent T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260709 12:29:59.427418 18094 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.255s	user 0.001s	sys 0.000s
I20260709 12:29:59.389530 17827 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:59.443701 17915 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:59.446233  6237 consensus_peers.cc:597] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:29:59.453748 18099 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:59.458539 18091 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:59.459453 18076 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:59.460013 17973 consensus_meta.cc:287] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.528s	user 0.008s	sys 0.001s
I20260709 12:29:59.397821 18113 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:59.504042 17973 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election lost for term 29. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 29. Current term is 30.
I20260709 12:29:59.505229 18050 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:59.505559 18113 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:59.506321  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:59.507889 18046 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:59.508250 18046 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:59.508615 18046 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:59.509563  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:59.465539  5602 consensus_peers.cc:597] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:29:59.512586  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47096.5us
W20260709 12:29:59.513441  5602 leader_election.cc:336] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:59.513806  5602 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:29:59.514250 17995 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:59.473351 17909 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:59.472859 18099 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:59.514775 17995 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:59.425499 17954 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:29:59.480602  6237 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:29:59.516892 17995 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:59.517058 17844 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:59.516927 18099 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:59.514633 18103 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:29:59.520273  5635 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
W20260709 12:29:59.521322  6237 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:29:59.527150  5602 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:59.527635  5602 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:59.528941  5602 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:59.530565  5602 leader_election.cc:336] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:59.533363 18086 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:59.537189 17787 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:29:59.539496 17968 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 42 FOLLOWER]: Advancing to term 43
I20260709 12:29:59.539347  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:59.530952  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:59.541564  5602 leader_election.cc:336] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:59.542027  5602 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:59.546180 17827 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:59.547412 18035 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:59.552711  5706 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 42 index: 27. Preceding OpId from leader: term: 43 index: 27. (term mismatch)
I20260709 12:29:59.553170  5706 pending_rounds.cc:85] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Aborting all ops after (but not including) 26
I20260709 12:29:59.553580  5706 pending_rounds.cc:107] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: Aborting uncommitted NO_OP operation due to leader change: 42.27
I20260709 12:29:59.553841  5706 raft_consensus.cc:2889] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:29:59.554495 18103 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:59.555032 17859 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:29:59.555627 17859 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:59.481624 17975 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:29:59.484256 17984 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:29:59.486495 17960 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:59.394794 18038 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:59.494249 17980 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:59.500735 17963 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:29:59.481055 18111 log.cc:927] Time spent T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:29:59.562757  6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.564193  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.565793  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:59.571394  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:59.571995  5705 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 34 index: 24. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
I20260709 12:29:59.576493  6279 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
I20260709 12:29:59.577921  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:59.583837  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:59.599483 17915 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:59.559994  5602 consensus_peers.cc:597] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:29:59.613356  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 53405.5us
I20260709 12:29:59.586177  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:59.592342  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.595153  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:59.428195 17948 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:59.593225  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 41 candidate_status { last_received { term: 40 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.583464 18075 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:59.600200  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.609334  5634 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
W20260709 12:29:59.619685  5602 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:59.630342  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:59.620549  6237 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:29:59.611099 17915 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:59.627658 17787 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.006s
I20260709 12:29:59.594486 17965 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:29:59.605016 17963 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:59.649389  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.650250  6274 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:29:59.638618  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.612042 18035 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:59.587517  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:59.643627  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.645375  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:59.646791  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:29:59.647535  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:59.651161  6237 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:29:59.657351  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:59.658228  5631 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:29:59.659412  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:29:59.662107  5676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.013s	sys 0.052s
W20260709 12:29:59.662513  5676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.013s	sys 0.052s
I20260709 12:29:59.662317  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:59.663511  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:29:59.666834  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 79401.5us
W20260709 12:29:59.665500  6237 leader_election.cc:336] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:29:59.652318 17714 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:29:59.672466  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.647934 18016 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:47920 (request call id 2190) took 2939 ms. Trace:
W20260709 12:29:59.687238  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 83587.5us
I20260709 12:29:59.640465 18103 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.043s
I20260709 12:29:59.637717 18075 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:59.676659 18016 rpcz_store.cc:276] 0709 12:29:56.699946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:56.700120 (+   174us) service_pool.cc:224] Handling call
0709 12:29:56.952043 (+251923us) ts_tablet_manager.cc:934] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466: Initiating tablet copy from peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
0709 12:29:58.688678 (+1736635us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:29:59.264315 (+575637us) tablet_metadata.cc:665] Metadata flushed
0709 12:29:59.284735 (+ 20420us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:29:59.531621 (+246886us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:29:59.531947 (+   326us) tablet_replica.cc:206] Creating consensus instance
0709 12:29:59.612085 (+ 80138us) ts_tablet_manager.cc:989] Replica registered.
0709 12:29:59.621804 (+  9719us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":135015,"tablet-copy.queue_time_us":185206,"thread_start_us":507313,"threads_started":2}
I20260709 12:29:59.666452 17892 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:59.678606 18074 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:29:59.695938  5526 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:29:59.735963  5634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:59.689020 17980 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:29:59.702183 17929 consensus_meta.cc:287] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: Time spent flushing consensus metadata: real 0.789s	user 0.007s	sys 0.005s
I20260709 12:29:59.703380 17859 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:59.695717 18046 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:59.704334  5631 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:59.705492  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.705783  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.706072  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.747455 18046 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:29:59.730865  5676 leader_election.cc:336] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:59.709444  4356 thread.cc:527] Waited for 5000ms trying to join with heartbeat (tid 5586)
I20260709 12:29:59.748581  5676 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:59.706391  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.731940  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:29:59.725837 18016 tablet_copy_client.cc:806] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 0 data blocks...
I20260709 12:29:59.695933  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:59.738199  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:29:59.722991 18068 consensus_meta.cc:287] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Time spent flushing consensus metadata: real 0.513s	user 0.010s	sys 0.000s
W20260709 12:29:59.751976  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 52707.7us
I20260709 12:29:59.710870  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.752390  5633 raft_consensus.cc:2368] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36.
I20260709 12:29:59.722854 18086 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:59.753842  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:29:59.734783 18074 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:59.713927 18038 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:29:59.715900 17915 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:29:59.757361  5710 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 30.
I20260709 12:29:59.767900  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:29:59.760953  5704 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 36
I20260709 12:29:59.717377 17948 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:29:59.725458 18075 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:29:59.761885  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 66406.5us
I20260709 12:29:59.743912 17892 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:29:59.745265 18039 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:29:59.743247 17980 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:29:59.794225  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60238.1us
I20260709 12:29:59.808883  5629 raft_consensus.cc:1275] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:29:59.761336 17929 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:59.813225  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.813845  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.764246 17998 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:59.775194 18074 raft_consensus.cc:3060] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:29:59.778919 18016 tablet_copy_client.cc:670] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Starting download of 1 WAL segments...
I20260709 12:29:59.806170 17980 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:59.839111  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:29:59.788380 17892 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:59.830906  5524 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:59.823222  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:29:59.763790 18068 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:59.850656  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:59.785849 17985 consensus_queue.cc:1059] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 21, Time since last communication: 0.045s
I20260709 12:29:59.778697 18086 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:29:59.806018 18103 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:29:59.829205  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:59.795696 17993 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.043s
I20260709 12:29:59.824349 17948 consensus_queue.cc:1048] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.018s
W20260709 12:29:59.840847 18126 log.cc:927] Time spent T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.126s	user 0.002s	sys 0.002s
I20260709 12:29:59.873734  5675 leader_election.cc:400] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36.
I20260709 12:29:59.874082  5675 leader_election.cc:403] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:29:59.840915 17892 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:29:59.877345  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 48220.7us
I20260709 12:29:59.878216  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:59.879455  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:29:59.881405  5704 raft_consensus.cc:2468] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:29:59.864317 18075 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:29:59.806480 18039 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:29:59.886336  6237 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:29:59.868117 18103 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:59.872704 18038 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:29:59.894078  6237 leader_election.cc:336] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:59.892207  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 135814us
I20260709 12:29:59.897943  5526 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:59.900846 17995 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:29:59.899175  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:59.904237  5526 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:59.905375  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 137626us
I20260709 12:29:59.883019 18127 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 35. Reason: Vote denied by peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587) with higher term. Message: Invalid argument: T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for earlier term 35. Current term is 36.
I20260709 12:29:59.885207 17965 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:29:59.913748  5526 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:59.913997  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:29:59.906463  5602 consensus_peers.cc:597] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:29:59.885490 17449 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:29:59.885802 17772 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:59.885672 17985 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:29:59.948546  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:29:59.891503 18103 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:29:59.946748  5601 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:29:59.946297  5602 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:29:59.951296  5602 consensus_peers.cc:597] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:29:59.878876 18122 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.032s
I20260709 12:29:59.888844 18116 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:59.959640  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:59.960803  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:59.908689 17717 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:29:59.970377  5706 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
W20260709 12:29:59.953766 18130 log.cc:927] Time spent T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:29:59.981124  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:29:59.948567 18075 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:29:59.955494 18064 log.cc:927] Time spent T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:29:59.972110 18115 raft_consensus.cc:3060] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:29:59.963922 17827 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:29:59.973961 18135 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:29:59.954160 17772 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:29:59.971921 17787 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.025s
W20260709 12:30:00.008910 18131 log.cc:927] Time spent T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
I20260709 12:30:00.010095 17963 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:00.011822 18086 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:00.037974  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:30:00.012426 18128 log.cc:927] Time spent T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
I20260709 12:30:00.065665  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:00.072304  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:00.015480 18119 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:30:00.078308 17772 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:00.084615 17846 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:30:00.087548 17963 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:00.095830 17714 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:30:00.088301 17963 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:00.100921 17963 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:00.102963 17963 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:00.071379 18122 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:30:00.071712 18050 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:30:00.073235 18024 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.037s
I20260709 12:30:00.109969  5627 raft_consensus.cc:2410] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf for term 39 because replica has last-logged OpId of term: 37 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
I20260709 12:30:00.110106 17892 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:00.111644 18068 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:00.107872 18103 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:00.116207 18103 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:00.114357 18115 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:30:00.120663  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 48448.7us
I20260709 12:30:00.112213 17892 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:00.128654  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 143644us
I20260709 12:30:00.130829 18143 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:30:00.142978  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:00.151753 18135 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:30:00.152001  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 201668us
I20260709 12:30:00.153749 18074 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:00.153918 17980 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:00.155908  5708 raft_consensus.cc:1240] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 32. Current term is 33. Ops: []
I20260709 12:30:00.157605 18039 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:00.158586 18016 tablet_copy_client.cc:538] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:30:00.160319  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:00.160943  5630 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:30:00.166738  5599 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:00.169498 17241 raft_consensus.cc:2804] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:30:00.170174 17241 raft_consensus.cc:697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:30:00.171159 17241 consensus_queue.cc:237] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 34.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:00.178202  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 44 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:00.178478 17980 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:00.182159 18074 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:00.182736  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:30:00.183323  6274 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:30:00.187824  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:00.188457  5704 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:30:00.190691  6274 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:30:00.192564  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:00.204883  5632 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: [30.19-30.19]
I20260709 12:30:00.207386  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:00.208410  5636 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 35: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:30:00.211270  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:30:00.218436 18039 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:30:00.219369 18039 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:30:00.219519 18074 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:00.219755  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:00.220743 18074 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:00.221339 17998 raft_consensus.cc:3055] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:30:00.221848 17998 raft_consensus.cc:740] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:30:00.222644 17998 consensus_queue.cc:260] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:00.223574 17998 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:30:00.221416  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:00.223470 18074 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:00.224617  5629 raft_consensus.cc:2393] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 34: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:30:00.225577  5602 leader_election.cc:336] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:00.226210  5602 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:00.227375 17859 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:30:00.219758 18039 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:30:00.229146 18039 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:00.230690 18039 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:30:00.232837  5701 raft_consensus.cc:1240] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 32. Current term is 33. Ops: []
I20260709 12:30:00.233906  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:00.234596  5705 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 38.
I20260709 12:30:00.238525  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:00.248065  5705 raft_consensus.cc:1275] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 34 index: 25. Preceding OpId from leader: term: 37 index: 27. (index mismatch)
I20260709 12:30:00.254827 18075 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:00.255436 18075 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:00.256763  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:00.257455  5701 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
I20260709 12:30:00.258435 18075 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:00.260169  5676 leader_election.cc:336] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:00.260705  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:00.261446 17980 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:00.262624  5704 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 31.
I20260709 12:30:00.264613  5702 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Rejecting Update request from peer 38736cc545eb4389b220a3bd21d3b466 for earlier term 30. Current term is 31. Ops: [30.19-30.19]
I20260709 12:30:00.272409  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:00.273389  5708 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:30:00.275818  5630 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
W20260709 12:30:00.272780  5676 leader_election.cc:336] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:00.278364  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 44 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:00.283823  5676 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:30:00.294162  5676 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:00.294621  5676 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:00.296684 17980 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:00.300411  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
W20260709 12:30:00.303802  6237 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:00.305823  5710 raft_consensus.cc:2393] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 36: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
W20260709 12:30:00.313098  6237 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:00.312130  5602 leader_election.cc:336] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:00.331099  5602 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:30:00.314344  5599 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466
W20260709 12:30:00.335501  5676 leader_election.cc:336] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:00.336164  5676 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:30:00.344532  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:00.347352 17980 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:30:00.348538  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:30:00.350466  5702 raft_consensus.cc:1240] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Rejecting Update request from peer cea66aa15be3423880843974acdbf50f for earlier term 36. Current term is 37. Ops: []
I20260709 12:30:00.359947  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:30:00.399752 18067 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.016s	sys 0.019s
W20260709 12:30:00.397668  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 62208.7us
I20260709 12:30:00.400288 17980 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:30:00.362007  6237 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:00.420583  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 50374.4us
W20260709 12:30:00.498140  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 138284us
I20260709 12:30:00.498768  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:00.385480 18137 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:30:00.416194  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:00.502584  5704 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
W20260709 12:30:00.369277  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 57183.3us
W20260709 12:30:00.503825  5602 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:00.367434 18016 tablet_bootstrap.cc:492] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap starting.
I20260709 12:30:00.368595 18135 raft_consensus.cc:2749] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:00.436563 18074 consensus_queue.cc:1059] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.007s
I20260709 12:30:00.337409 18140 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:30:00.387487  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 73211.5us
W20260709 12:30:00.409675  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 93305.7us
I20260709 12:30:00.431124  5708 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 39.
W20260709 12:30:00.497104  5676 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:30:00.499466  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 137498us
I20260709 12:30:00.506294  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:30:00.497462 18067 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.779s	user 0.028s	sys 0.026s
W20260709 12:30:00.523221  5676 leader_election.cc:336] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:00.533329  5676 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:30:00.521896  5526 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
W20260709 12:30:00.534611  5676 leader_election.cc:336] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:00.535058  5676 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:00.536024 17948 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:30:00.536314 17717 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:30:00.536545 17998 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:30:00.536729 17948 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:30:00.533320 18086 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:30:00.532819  5452 catalog_manager.cc:5697] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 19 to 27. New cstate: current_term: 27 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:00.505662 18115 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:30:00.538273  5524 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:00.541730  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:30:00.553020  6237 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:00.544485  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:00.566596  5524 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:30:00.566533  5456 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35600 (request call id 284) took 11282 ms (client timeout 15000 ms). Trace:
W20260709 12:30:00.567945  5456 rpcz_store.cc:269] 0709 12:29:49.290856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:53.518629 (+4227773us) service_pool.cc:224] Handling call
0709 12:30:00.539392 (+7020763us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:29:57.954142 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:29:57.970464 (+ 16322us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:29:57.970501 (+    37us) write_op.cc:435] Acquired schema lock
0709 12:29:57.970520 (+    19us) tablet.cc:662] Decoding operations
0709 12:29:57.972592 (+  2072us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:29:57.972659 (+    67us) write_op.cc:641] Partition lock acquired for write op
0709 12:29:57.972686 (+    27us) tablet.cc:685] Acquiring locks for 28 operations
0709 12:29:57.974848 (+  2162us) tablet.cc:701] Row locks acquired
0709 12:29:57.974863 (+    15us) write_op.cc:260] PREPARE: finished
0709 12:29:57.975020 (+   157us) write_op.cc:270] Start()
0709 12:29:57.975130 (+   110us) write_op.cc:276] Timestamp: P: 1783600197974972 usec, L: 0
0709 12:29:57.975149 (+    19us) op_driver.cc:348] REPLICATION: starting
0709 12:29:57.988162 (+ 13013us) log.cc:844] Serialized 10697 byte log entry
0709 12:29:58.312240 (+324078us) op_driver.cc:464] REPLICATION: finished
0709 12:29:58.432865 (+120625us) write_op.cc:301] APPLY: starting
0709 12:29:58.442928 (+ 10063us) tablet.cc:1367] starting BulkCheckPresence
0709 12:29:59.265665 (+822737us) tablet.cc:1370] finished BulkCheckPresence
0709 12:29:59.265771 (+   106us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:29:59.503082 (+237311us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:29:59.503515 (+   433us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=56,key_file_lookups=56,delta_file_lookups=0,mrs_lookups=0
0709 12:29:59.503563 (+    48us) write_op.cc:312] APPLY: finished
0709 12:29:59.670681 (+167118us) log.cc:844] Serialized 244 byte log entry
0709 12:29:59.897065 (+226384us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:29:59.913094 (+ 16029us) write_op.cc:454] Released schema lock
0709 12:29:59.953090 (+ 39996us) write_op.cc:341] FINISH: Updating metrics
0709 12:29:59.994480 (+ 41390us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:29:59.996891 (+  2411us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":112357,"apply.run_cpu_time_us":66890,"apply.run_wall_time_us":1552880,"cfile_cache_hit":114,"cfile_cache_hit_bytes":121167,"num_ops":28,"prepare.queue_time_us":142535,"prepare.run_cpu_time_us":30809,"prepare.run_wall_time_us":93505,"raft.queue_time_us":130804,"raft.run_cpu_time_us":26545,"raft.run_wall_time_us":148379,"replication_time_us":276758,"thread_start_us":445866,"threads_started":5,"wal-append.queue_time_us":301886,"wal-append.run_cpu_time_us":13196,"wal-append.run_wall_time_us":1173503}]]}
I20260709 12:30:00.566440  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:00.585191  5636 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 36.
W20260709 12:30:00.586298  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 54817.3us
I20260709 12:30:00.515138 18137 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:30:00.597693  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:30:00.606702  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 72129.6us
W20260709 12:30:00.607436  5676 consensus_peers.cc:597] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:00.608994  5452 catalog_manager.cc:5697] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 24 to 29, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 29 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:00.581023 17998 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:30:00.607945  5676 leader_election.cc:336] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:00.584383 18074 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:30:00.610736  5452 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:30:00.616267  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49776.2us
I20260709 12:30:00.614897  5676 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:30:00.604944 18035 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:00.519328 18140 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:00.622439  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:30:00.624230  6236 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:30:00.626240 17948 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:30:00.627348  5602 consensus_peers.cc:597] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:00.631856 17940 raft_consensus.cc:3055] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:30:00.631358  5452 catalog_manager.cc:5697] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 28 to 29, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 29 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:00.615649 18112 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:00.671244 17892 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:30:00.673544 17998 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:00.615586 18074 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:00.628584 18067 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:30:00.692728 18112 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:00.694257  5452 catalog_manager.cc:5697] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 22 to 29, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 29 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:00.624053 17993 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:00.725611 17993 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:00.727844 17993 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:00.729557  5452 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 25 to 27, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 27 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:00.731045  5452 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:00.635443 17940 raft_consensus.cc:740] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Becoming Follower/Learner. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:30:00.734458  5452 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 30 to 33, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 33 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:00.737520 18140 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:00.737692  5452 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 30 to 32. New cstate: current_term: 32 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:00.709307 18003 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:30:00.740044 18003 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:00.633952 17985 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:00.718411 18074 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:30:00.742153 18003 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:00.708884 18137 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:00.723225 18067 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 39 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:30:00.696570 18159 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:30:00.746111 18067 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:00.725031 18112 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:00.749861  4356 thread.cc:527] Waited for 6000ms trying to join with heartbeat (tid 5586)
I20260709 12:30:00.735340 17940 consensus_queue.cc:260] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:00.635037 18035 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:00.761587  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 134933us
W20260709 12:30:00.764786  6237 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:00.770524  5676 leader_election.cc:336] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:00.770972  5676 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:00.741035  5452 catalog_manager.cc:5697] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 27 to 28, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 28 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:30:00.778102  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 107088us
I20260709 12:30:00.790448  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:00.791328  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:00.794234  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:00.795648  5452 catalog_manager.cc:5697] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 26 to 31, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 31 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:00.762365 18112 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:30:00.797331  5452 catalog_manager.cc:5697] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 26 to 31, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 31 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:30:00.802500  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 163631us
I20260709 12:30:00.798903  5708 raft_consensus.cc:1275] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:30:00.772954 18068 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:00.806301  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:00.807816  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:00.720399 17892 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:00.815605  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:00.810051  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:00.789108 17948 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:30:00.829071  5452 catalog_manager.cc:5697] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 22 to 26, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 26 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:30:00.833410 17241 consensus_queue.cc:1048] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:30:00.764938 17985 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:00.842257  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:30:00.845816  5524 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:30:00.845075  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:00.817045 18068 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:00.815973 18112 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:00.870504 18140 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:00.868016 18074 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:00.903316 18074 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:00.921715  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 73536.7us
I20260709 12:30:00.922801  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:00.926640 17892 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:00.938656  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:00.939775  6279 raft_consensus.cc:2410] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 36 because replica has last-logged OpId of term: 34 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 19.
I20260709 12:30:00.956629  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:30:00.956596  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 179295us
I20260709 12:30:00.930415 18140 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:30:00.938037 18035 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:00.969485  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:00.969746  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:00.970083  5636 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:30:00.970973  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:00.973165  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:30:00.962467  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 85501.7us
I20260709 12:30:00.992328 18068 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:00.997395 17985 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:00.998766 18112 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:00.965883 18074 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:30:01.032286  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:01.033857  5526 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:01.043143  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:01.044744  5709 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:30:01.052788  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 77633.7us
I20260709 12:30:01.056291  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:01.057133  5702 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:30:01.062657  5636 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:30:01.062677  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:01.067539  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:01.067708  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:30:01.101584  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 240768us
I20260709 12:30:01.082355 18003 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:30:01.082338 18075 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:30:01.145427  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 82818.4us
I20260709 12:30:01.088893 18038 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:30:01.070142 18105 consensus_queue.cc:1059] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: INVALID_TERM, Last received: 32.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.076s
I20260709 12:30:01.194450  5524 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:01.075915 17847 consensus_queue.cc:1048] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 25, Time since last communication: 0.077s
I20260709 12:30:01.109004 18140 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:01.195992  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:01.195011  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:01.202268  5632 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:30:01.204640  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:01.191075 18161 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:01.150583 18177 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:01.214188  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:01.219671  5704 raft_consensus.cc:2410] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for term 36 because replica has last-logged OpId of term: 34 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 19.
I20260709 12:30:01.215474  5526 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:30:01.193985 18038 raft_consensus.cc:493] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting leader election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:30:01.221818  5526 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:30:01.227737  5602 consensus_peers.cc:597] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:01.200214 18162 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:01.241503 18162 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:01.233868  5602 consensus_peers.cc:597] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:01.221403 18161 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:01.216713 18112 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:30:01.207926 17892 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:30:01.230618 18068 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:30:01.209713 18140 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:01.259547  5599 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
W20260709 12:30:01.249256 18173 log.cc:927] Time spent T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:30:01.275949 18140 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:01.252410 17940 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:30:01.276708 18068 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 37 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:30:01.277787 18068 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.23, Last appended by leader: 23, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:01.258164 17993 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:30:01.281615 18174 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:30:01.261133 18161 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:01.269301  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:01.270344  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:01.260170 17892 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:01.254905 17714 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:01.233415 18038 raft_consensus.cc:3060] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:30:01.285717  5708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:01.291280  5709 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:30:01.244230 18162 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:01.294737 18046 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:01.303099 17892 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:30:01.304765 18035 raft_consensus.cc:2749] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:01.319587 18161 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:30:01.380033  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 52806.7us
I20260709 12:30:01.306874 17714 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:01.325443 18182 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:30:01.384766 18178 log.cc:927] Time spent T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:30:01.410714 17714 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:01.396410 18182 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:01.332726 18046 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:01.381549 18161 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:01.427665 18046 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:01.429013 18161 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:01.434712  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:30:01.444586  5602 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 159 similar messages]
I20260709 12:30:01.457018 18182 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:01.466351 18180 raft_consensus.cc:3055] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:30:01.470853  5602 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:30:01.488154  5602 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:01.495715  5705 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
W20260709 12:30:01.500314  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 179440us
I20260709 12:30:01.469928 18180 raft_consensus.cc:740] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:30:01.513052  6274 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 31 index: 19. (term mismatch)
I20260709 12:30:01.513550  6274 pending_rounds.cc:85] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Aborting all ops after (but not including) 18
I20260709 12:30:01.513655  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:01.514276  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:01.513775  6274 pending_rounds.cc:107] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: Aborting uncommitted NO_OP operation due to leader change: 30.19
I20260709 12:30:01.515880  6274 raft_consensus.cc:2889] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:30:01.516923  5676 leader_election.cc:336] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:01.517478  5676 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
W20260709 12:30:01.518697  5676 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:01.519330  5676 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:01.520886  6278 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:30:01.525861  5630 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:30:01.526194  5676 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:01.499219 18170 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:30:01.526970  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:01.507911 18127 raft_consensus.cc:493] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:30:01.534770  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:01.539899  5636 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:30:01.543035  5630 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 39 index: 25. (index mismatch)
I20260709 12:30:01.544987  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:01.525569 18161 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:01.548336  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 28 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:01.552706  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:01.553493  5704 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 37: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:30:01.524681 18086 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:30:01.524828 18167 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:30:01.557799 17940 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:30:01.558784  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:01.528331 17892 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:01.559762 18172 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:30:01.535246 18166 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:30:01.522552 18038 raft_consensus.cc:515] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:01.562151  5526 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:01.563074 18172 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:01.561278 17892 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:01.564949 18038 leader_election.cc:290] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:01.565004 17892 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:30:01.568141 18190 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:01.569337  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:01.569005 18190 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:01.579439  5701 raft_consensus.cc:2393] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in current term 33: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:30:01.585749 17915 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:30:01.532675 18170 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:01.551935 18180 consensus_queue.cc:260] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 32.23, Last appended by leader: 23, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:01.589788 18180 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:30:01.589903 18170 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:01.593680  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:01.594568  5636 raft_consensus.cc:2393] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 38736cc545eb4389b220a3bd21d3b466 in current term 35: Already voted for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in this term.
I20260709 12:30:01.596971 18190 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:01.572968  5676 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:01.547883 18127 raft_consensus.cc:515] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:01.577653  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:01.604432  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:01.607273  5635 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:30:01.604775  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:01.608842  5710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf758b480ecf41619471236195bf1838" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:01.631899  5526 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:01.636581 18162 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:30:01.637529 18162 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:30:01.646010  5708 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:30:01.647055  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:01.647832  5703 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 31.
I20260709 12:30:01.641283  5526 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:01.652339  5635 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:30:01.651607  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:30:01.654062  5676 leader_election.cc:336] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:01.555174 18161 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:01.664789  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 110236us
I20260709 12:30:01.667033  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:01.669879  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:01.670810  5675 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:30:01.669948  5710 raft_consensus.cc:2468] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
I20260709 12:30:01.673745  6234 leader_election.cc:304] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:01.655059 17940 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:30:01.672478  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:01.677403  5599 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:30:01.682794  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 129241us
I20260709 12:30:01.682116  5526 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:30:01.680907 18167 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:30:01.691131 18143 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:30:01.693425  6237 consensus_peers.cc:597] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:01.694278  6237 consensus_peers.cc:597] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:01.694078 18135 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:30:01.698545 18161 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:01.650274 18172 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.026s
I20260709 12:30:01.707557  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:01.680193 18190 raft_consensus.cc:2804] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:30:01.692174 18166 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:30:01.744673  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:30:01.750612  4356 thread.cc:527] Waited for 7000ms trying to join with heartbeat (tid 5586)
I20260709 12:30:01.704665 17998 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:30:01.760155  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:01.725806 17940 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:30:01.776041 18190 raft_consensus.cc:697] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:30:01.777416 18190 consensus_queue.cc:237] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:01.761834 18161 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:01.785271 18161 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:01.787387 18161 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:01.809679  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:30:01.810873  5602 consensus_peers.cc:597] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:01.813735  5707 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:30:01.762720 18191 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
W20260709 12:30:01.817354  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 49982us
I20260709 12:30:01.823772  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
W20260709 12:30:01.825242  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 65188us
I20260709 12:30:01.828856  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:01.818789 17940 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
I20260709 12:30:01.837888 17998 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:30:01.838317 18199 log.cc:927] Time spent T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 12:30:01.848567  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:01.842338 18135 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:30:01.871660  5707 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 35.
I20260709 12:30:01.853185 18169 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:30:01.901386  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 69747.1us
I20260709 12:30:01.875629 18127 leader_election.cc:290] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:01.838219 17940 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 33.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:01.863690 18162 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:30:01.896636 18200 log.cc:927] Time spent T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 12:30:01.876808  6275 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9efccdcce1164ab5b57aa40caea54717" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:30:01.908329  5676 consensus_peers.cc:597] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:01.917676  5675 leader_election.cc:304] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:01.928016 17963 raft_consensus.cc:2749] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:30:01.914364 18162 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:01.961290  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 216891us
I20260709 12:30:01.912616 18169 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:02.015702 18162 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:02.078433 18169 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:30:02.071847 18207 log.cc:927] Time spent T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:30:02.085495 18143 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:30:02.099532 18135 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:30:02.140045  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 277594us
W20260709 12:30:02.155248  6277 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.006s	sys 0.029s
I20260709 12:30:02.164765 17772 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:30:02.166428  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 200754us
I20260709 12:30:02.290841 17772 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:02.292260  6237 leader_election.cc:336] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:02.293473  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:02.295187 18182 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:30:02.295516 17772 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:02.300150  5702 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:30:02.300324 18172 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:02.301198 18172 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:02.303489  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:02.229787 18189 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:02.306367 18189 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:02.244549 17940 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:02.260512 18166 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:30:02.289834  6277 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.007s	sys 0.030s
I20260709 12:30:02.306972 18172 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:02.308882 17940 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:02.322500 18166 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:02.325213 18166 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:02.326946 18082 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:30:02.332634 17940 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:02.342958 18189 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:02.356943  5676 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:02.364636 17859 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:02.366017 17859 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:02.368429  5705 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:02.373365 17859 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:02.374708 18166 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:02.392922 18166 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:02.402091  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:02.395924 18166 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:02.420264 18082 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:02.430377 18082 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:30:02.431736  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:02.432803  6277 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:30:02.438879 18140 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:02.446790 17993 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:02.449285 17993 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:02.454748 18140 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:02.457502 18140 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:02.467562 18190 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.013s	sys 0.017s
W20260709 12:30:02.468092 18190 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.013s	sys 0.018s
I20260709 12:30:02.479805  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:02.487551 18166 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:30:02.488068 18166 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:02.482296 17993 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:02.488560 18166 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:30:02.493474  5632 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:30:02.495344  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:02.496097  5632 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:30:02.497778  5632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:02.501067  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:02.492043  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:02.523707  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:02.535702  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:02.536585  5636 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:30:02.537554  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:02.541841  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:02.542518  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:30:02.544806  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 54781.9us
I20260709 12:30:02.548897  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:02.543687  5636 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:30:02.558369 17717 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:30:02.558197 18082 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:02.559653  6272 raft_consensus.cc:1240] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Rejecting Update request from peer 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 31. Current term is 32. Ops: []
I20260709 12:30:02.565469  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:02.572978 18082 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:02.527133  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:02.581218 18086 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:30:02.582348 18086 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:02.589344  5709 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 33.
I20260709 12:30:02.640213 17993 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.006s
I20260709 12:30:02.660918 18166 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:02.662070  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:02.662669 18086 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:02.662822  5701 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 37.
I20260709 12:30:02.663506 18166 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:02.660991  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:02.668125  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:02.668463  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:30:02.669030  6278 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:30:02.669420  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 143187us
I20260709 12:30:02.687188  6278 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:30:02.692775  5675 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:02.708027  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:30:02.708271  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 50710.8us
I20260709 12:30:02.711440  5636 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 37.
W20260709 12:30:02.715278  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 43962.5us
I20260709 12:30:02.703028 18166 raft_consensus.cc:2804] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:30:02.723960 17915 consensus_queue.cc:1059] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.013s
I20260709 12:30:02.735188  5632 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:30:02.735737  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:02.749756  5706 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:02.753000  5627 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:30:02.763327  4356 thread.cc:527] Waited for 8000ms trying to join with heartbeat (tid 5586)
I20260709 12:30:02.763260 17915 raft_consensus.cc:3055] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:30:02.762065  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:02.765980 17915 raft_consensus.cc:740] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:30:02.769742 17915 consensus_queue.cc:260] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:02.730026 18166 raft_consensus.cc:697] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:30:02.782119 18166 consensus_queue.cc:237] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 32.23, Last appended by leader: 23, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:02.786024  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:02.786928  5629 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:30:02.788879  6236 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:02.789204 18182 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:30:02.791231  6236 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:02.791215 18038 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:30:02.792219 18082 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:30:02.793949 18137 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:30:02.794613 18137 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:02.795547 18164 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:30:02.797138 18137 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:02.796309 18164 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:02.798143  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:02.799253 17940 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:02.799613 18164 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:02.803309  6277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:02.804208  6277 raft_consensus.cc:2368] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34.
I20260709 12:30:02.807334  5601 leader_election.cc:400] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34.
I20260709 12:30:02.807694  5601 leader_election.cc:403] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:30:02.767856  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:02.808923  5630 raft_consensus.cc:3060] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 34
W20260709 12:30:02.813452  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47629.6us
I20260709 12:30:02.818975  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "378854044c0e42f4b0a737cdbdbcc464" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:30:02.815383  5601 consensus_peers.cc:597] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059): Couldn't send request to peer 38736cc545eb4389b220a3bd21d3b466. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:02.828799  5707 raft_consensus.cc:1275] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:30:02.823879 17717 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:02.830895 17717 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:02.835048 17915 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:30:02.835443 18038 consensus_queue.cc:1048] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:30:02.838063 17717 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:02.838743  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 96079.9us
I20260709 12:30:02.850739  5524 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:30:02.835273  5676 leader_election.cc:336] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:02.852564  5676 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:30:02.855731 18127 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:30:02.856937 18162 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:30:02.857380 18162 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:02.857746 18162 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:30:02.860738  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:02.841168 18164 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:30:02.862504 17993 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:30:02.862882 17993 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:02.863232 17993 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:30:02.841614 18016 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:30:02.875828  5630 raft_consensus.cc:2468] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:30:02.876214  5526 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:02.883406 18210 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.003s
I20260709 12:30:02.887944 18035 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election lost for term 33. Reason: Vote denied by peer 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059) with higher term. Message: Invalid argument: T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a1c5e6dd8e249bf82d38f90fa37b43d for earlier term 33. Current term is 34.
I20260709 12:30:02.890244  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:02.896955  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:02.904260 18210 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:02.905925  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:02.919123  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:02.909636 18164 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:30:02.922806 18210 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:02.928472 17963 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:30:02.928937 17963 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:02.929288 17963 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:30:02.930188 18210 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:02.918726 18140 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:30:02.916997 18172 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.003s
I20260709 12:30:02.952327  5708 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:30:02.971786 18068 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:30:02.964799 18210 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:02.976054 18210 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:02.978368 18210 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:02.962988 17963 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:03.012552  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 94018.8us
I20260709 12:30:03.028262  5454 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 29 to 31. New cstate: current_term: 31 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:03.029794 17993 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:03.032038 17993 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:03.040720 17963 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:03.042773  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:03.043463  5630 raft_consensus.cc:3060] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:30:03.053236  6237 consensus_peers.cc:597] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:03.048548  5454 catalog_manager.cc:5697] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 31 to 35, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 35 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:03.055377  5630 raft_consensus.cc:2468] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 38.
I20260709 12:30:03.056619  5675 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:03.058238 17963 raft_consensus.cc:2804] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:30:03.058805 17963 raft_consensus.cc:697] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 38 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
W20260709 12:30:03.061698  5676 leader_election.cc:336] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:03.066828  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:03.067216  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdcede9bc9104440a7ce1e3a4eb3a86f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:03.067780 17963 consensus_queue.cc:237] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 36.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:03.068640  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:03.052086 18162 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:03.073832 18162 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:03.083886  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:03.087492  6278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:03.099936  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:03.104993  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:30:03.076223  5452 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 283) took 52230 ms (client timeout 15000 ms). Trace:
I20260709 12:30:03.077116  5454 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:03.113094  5633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:03.119665  5633 raft_consensus.cc:2393] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate cea66aa15be3423880843974acdbf50f in current term 38: Already voted for candidate f613ba929b1046d3b5aed909414a95bf in this term.
I20260709 12:30:03.123927  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:03.125062 18016 tablet_bootstrap.cc:492] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=3 applied=24 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:03.126518 18016 tablet_bootstrap.cc:492] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466: Bootstrap complete.
I20260709 12:30:03.126245  5526 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:03.126840 17717 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:30:03.127611  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:03.145780  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:03.146510  5701 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:30:03.146572  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:03.148769  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:03.151413 17717 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:30:03.153911  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:03.154206 18164 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:03.154546  5628 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:30:03.117168 18210 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.042s
I20260709 12:30:03.158215 18140 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:03.161585  5454 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 27 to 30. New cstate: current_term: 30 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:30:03.108639  5452 rpcz_store.cc:269] 0709 12:29:10.845584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:21.317547 (+10471963us) service_pool.cc:224] Handling call
0709 12:30:03.075931 (+41758384us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:30:00.940768 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:30:00.941572 (+   804us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:30:00.941632 (+    60us) write_op.cc:435] Acquired schema lock
0709 12:30:00.941661 (+    29us) tablet.cc:662] Decoding operations
0709 12:30:00.943178 (+  1517us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:30:00.943288 (+   110us) write_op.cc:641] Partition lock acquired for write op
0709 12:30:00.943330 (+    42us) tablet.cc:685] Acquiring locks for 13 operations
0709 12:30:01.023072 (+ 79742us) tablet.cc:701] Row locks acquired
0709 12:30:01.023095 (+    23us) write_op.cc:260] PREPARE: finished
0709 12:30:01.023284 (+   189us) write_op.cc:270] Start()
0709 12:30:01.023426 (+   142us) write_op.cc:276] Timestamp: P: 1783600201023218 usec, L: 0
0709 12:30:01.023454 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:30:01.024169 (+   715us) log.cc:844] Serialized 5031 byte log entry
0709 12:30:01.443037 (+418868us) op_driver.cc:464] REPLICATION: finished
0709 12:30:01.577470 (+134433us) write_op.cc:301] APPLY: starting
0709 12:30:01.577567 (+    97us) tablet.cc:1367] starting BulkCheckPresence
0709 12:30:01.875916 (+298349us) tablet.cc:1370] finished BulkCheckPresence
0709 12:30:01.892678 (+ 16762us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:30:02.082043 (+189365us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:30:02.082338 (+   295us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=0,mrs_lookups=0
0709 12:30:02.082384 (+    46us) write_op.cc:312] APPLY: finished
0709 12:30:02.082845 (+   461us) log.cc:844] Serialized 121 byte log entry
0709 12:30:02.175969 (+ 93124us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:02.176594 (+   625us) write_op.cc:454] Released schema lock
0709 12:30:02.176715 (+   121us) write_op.cc:341] FINISH: Updating metrics
0709 12:30:02.177698 (+   983us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:02.177746 (+    48us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":237477,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":134168,"apply.run_cpu_time_us":25341,"apply.run_wall_time_us":600045,"cfile_cache_hit":54,"cfile_cache_hit_bytes":58467,"num_ops":13,"prepare.queue_time_us":83054,"prepare.run_cpu_time_us":130939,"prepare.run_wall_time_us":261757,"raft.queue_time_us":169005,"raft.run_cpu_time_us":31249,"raft.run_wall_time_us":85243,"replication_time_us":419410,"thread_start_us":365084,"threads_started":4,"wal-append.queue_time_us":240448,"wal-append.run_cpu_time_us":10662,"wal-append.run_wall_time_us":1818979}]]}
W20260709 12:30:03.181648  5602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 65495.6us
I20260709 12:30:03.127628 18016 ts_tablet_manager.cc:1403] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466: Time spent bootstrapping tablet: real 2.760s	user 0.078s	sys 0.031s
I20260709 12:30:03.200032  5526 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:03.201476 18068 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
W20260709 12:30:03.237030  5602 consensus_peers.cc:597] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:03.265606  5446 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 138282us
I20260709 12:30:03.242242 18164 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:03.278857 17892 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:03.279542 17892 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:03.262974 18068 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:03.284006 18068 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:03.286230  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:30:03.289072  5602 consensus_peers.cc:597] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:03.289772  5602 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:03.290642  5602 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:30:03.291442 18214 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:30:03.282989 18164 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:03.315805  5628 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:30:03.318804  5526 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f; no voters: 
I20260709 12:30:03.326546  5701 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cea66aa15be3423880843974acdbf50f in term 34.
I20260709 12:30:03.347064 18244 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:30:03.355726  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:03.325796 17892 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:03.319667 17717 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:30:03.317111  5446 catalog_manager.cc:4755] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 26edf338d41e42f1a5ce0d4416ec67e1: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer
I20260709 12:30:03.366794  5526 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:03.371025  5630 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:30:03.373735  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:30:03.375280  5446 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 61904.6us
W20260709 12:30:03.381978  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 51361.7us
I20260709 12:30:03.379949  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:03.386200 18244 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Becoming Leader. State: Replica: cea66aa15be3423880843974acdbf50f, State: Running, Role: LEADER
W20260709 12:30:03.415580  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 53074us
I20260709 12:30:03.420622 18250 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:30:03.423130  6237 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:03.429476  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 99531.4us
I20260709 12:30:03.407696 18068 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:03.430542  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:03.390343 17717 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:03.396466 18016 raft_consensus.cc:359] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
W20260709 12:30:03.449910  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 57847.9us
I20260709 12:30:03.509902 17717 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:03.513706 18164 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:03.501098 18244 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:03.533560  5526 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:03.492609 18016 raft_consensus.cc:740] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEARNER]: Becoming Follower/Learner. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Initialized, Role: LEARNER
I20260709 12:30:03.548914  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:03.451294 18251 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.092s
I20260709 12:30:03.562198  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:30:03.564494 18239 log.cc:927] Time spent T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:30:03.572849 18256 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:03.588002 18256 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:03.573968 18255 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:30:03.577235 18172 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:03.593657 18172 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:03.595323  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:03.612680 18172 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:03.614260 18256 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:03.615763  5524 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:03.615475 18016 consensus_queue.cc:260] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 24, Last appended: 32.24, Last appended by leader: 24, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } }
I20260709 12:30:03.617997  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:30:03.629750  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 86742us
I20260709 12:30:03.629961 18210 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:30:03.635237  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d816d235ce5a41c796c8949c48c0a53c" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:03.644781  6272 raft_consensus.cc:1275] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:30:03.646170 18016 ts_tablet_manager.cc:1434] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466: Time spent starting tablet: real 0.459s	user 0.014s	sys 0.007s
I20260709 12:30:03.647953 18127 consensus_queue.cc:1048] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
W20260709 12:30:03.647329  5453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 273) took 227924 ms (client timeout 15000 ms). Trace:
I20260709 12:30:03.666805  6277 raft_consensus.cc:1217] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEARNER]: Deduplicated request from leader. Original: 32.23->[32.24-32.24]   Dedup: 32.24->[]
I20260709 12:30:03.668929  5646 tablet_copy_service.cc:342] P 4a1c5e6dd8e249bf82d38f90fa37b43d: Request end of tablet copy session 38736cc545eb4389b220a3bd21d3b466-785e79f6fbd0463aafaca5498918a142 received from {username='slave'} at 127.0.0.1:40390
I20260709 12:30:03.669834  5646 tablet_copy_service.cc:434] P 4a1c5e6dd8e249bf82d38f90fa37b43d: ending tablet copy session 38736cc545eb4389b220a3bd21d3b466-785e79f6fbd0463aafaca5498918a142 on tablet 785e79f6fbd0463aafaca5498918a142 with peer 38736cc545eb4389b220a3bd21d3b466
W20260709 12:30:03.658224  5676 consensus_peers.cc:597] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:03.681792  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:03.679903  5524 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:03.702656 18164 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:30:03.697721  5602 consensus_peers.cc:597] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:03.705695  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:03.708494  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:03.703359 18164 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:03.733530  6237 consensus_peers.cc:597] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:03.734531  6237 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:03.734835 18164 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:03.744493  6237 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:03.694917 18068 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:30:03.745208  6237 consensus_peers.cc:597] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:03.666778  5453 rpcz_store.cc:269] 0709 12:26:15.722497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.722933 (+   436us) service_pool.cc:224] Handling call
0709 12:30:03.647036 (+227924103us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":630579,"reported_tablets":50}
I20260709 12:30:03.740528  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:30:03.750201  6237 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:03.753836  5526 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:03.764050 18068 raft_consensus.cc:493] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:30:03.764994 18162 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:30:03.764860 18068 raft_consensus.cc:515] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:03.766746  4356 thread.cc:527] Waited for 9000ms trying to join with heartbeat (tid 5586)
I20260709 12:30:03.770067 17993 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:30:03.772454 18068 leader_election.cc:290] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:03.780709  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:30:03.783588  6237 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:03.788902  5602 consensus_peers.cc:597] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:03.798034 18162 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:03.798905 18162 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:03.801298 18162 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:03.805614  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:03.819779  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "abd24f2e500a44729c9111fae83c25ed" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:03.823513  5629 raft_consensus.cc:1275] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:30:03.824772  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "edd23660363f48bf95cee8a70c5a2f26" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:03.826978  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:03.831830  5524 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:03.833266 18127 consensus_queue.cc:1048] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:30:03.835253  5524 leader_election.cc:304] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:03.833293 18210 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:30:03.854904  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:03.855654 18255 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:30:03.859035 18255 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:03.859619 18210 raft_consensus.cc:2749] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:30:03.859058 18162 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:03.861706 18255 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:03.871742  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:03.893149  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:03.899899  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:03.912477 18256 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:30:03.908365  6237 consensus_peers.cc:597] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:03.936651 18068 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:03.948493 18068 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:03.950836 18068 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:03.957787  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:03.986953  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:03.988770  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:03.990115 18255 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:30:03.996117 18166 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:03.996907 18166 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:03.999404 18166 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:04.000192  5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:04.000532  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:04.001101  5630 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:30:04.001435  6272 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:30:04.002465  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:04.009227 18166 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:30:04.009837 18166 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:04.010344 18166 raft_consensus.cc:3060] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:30:04.044780  5602 consensus_peers.cc:597] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:04.065214  5676 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:04.073148  5676 consensus_peers.cc:597] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:04.085606 18166 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:04.089399  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
W20260709 12:30:04.096833  5454 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 184) took 41482 ms (client timeout 15000 ms). Trace:
W20260709 12:30:04.097781  6237 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:04.101995  5629 raft_consensus.cc:2468] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:30:04.103895  5675 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:04.108014 18255 raft_consensus.cc:493] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:04.108623 18255 raft_consensus.cc:515] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:04.110862 18255 leader_election.cc:290] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:04.114282  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:04.114701 18166 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:04.117223 18251 raft_consensus.cc:2804] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:30:04.117784 18251 raft_consensus.cc:697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:30:04.118710 18251 consensus_queue.cc:237] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:04.121564  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6adc8ae91dfe46039ab5bd2bc4c69708" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:30:04.121937  5629 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:30:04.122576  6272 raft_consensus.cc:2393] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 32: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:30:04.124392  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b585af92ddc4f4fa5bada0f9b198495" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:04.129664  5702 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer cea66aa15be3423880843974acdbf50f: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:30:04.138245 18255 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:30:04.139642  5524 leader_election.cc:304] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:04.141655 18244 raft_consensus.cc:2749] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:30:04.142370 18244 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:30:04.102165  5454 rpcz_store.cc:269] 0709 12:29:22.641051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:32.149942 (+9508891us) service_pool.cc:224] Handling call
0709 12:30:04.096730 (+31946788us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:30:03.278857 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:30:03.279490 (+   633us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:30:03.279541 (+    51us) write_op.cc:435] Acquired schema lock
0709 12:30:03.279560 (+    19us) tablet.cc:662] Decoding operations
0709 12:30:03.280234 (+   674us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:30:03.280323 (+    89us) write_op.cc:641] Partition lock acquired for write op
0709 12:30:03.280375 (+    52us) tablet.cc:685] Acquiring locks for 4 operations
0709 12:30:03.280876 (+   501us) tablet.cc:701] Row locks acquired
0709 12:30:03.280923 (+    47us) write_op.cc:260] PREPARE: finished
0709 12:30:03.281208 (+   285us) write_op.cc:270] Start()
0709 12:30:03.281354 (+   146us) write_op.cc:276] Timestamp: P: 1783600203281033 usec, L: 0
0709 12:30:03.281392 (+    38us) op_driver.cc:348] REPLICATION: starting
0709 12:30:03.287022 (+  5630us) log.cc:844] Serialized 1650 byte log entry
0709 12:30:03.624087 (+337065us) op_driver.cc:464] REPLICATION: finished
0709 12:30:03.699059 (+ 74972us) write_op.cc:301] APPLY: starting
0709 12:30:03.699194 (+   135us) tablet.cc:1367] starting BulkCheckPresence
0709 12:30:03.728303 (+ 29109us) tablet.cc:1370] finished BulkCheckPresence
0709 12:30:03.728445 (+   142us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:30:03.765032 (+ 36587us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:30:03.765374 (+   342us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=8,key_file_lookups=8,delta_file_lookups=0,mrs_lookups=0
0709 12:30:03.765439 (+    65us) write_op.cc:312] APPLY: finished
0709 12:30:03.765814 (+   375us) log.cc:844] Serialized 49 byte log entry
0709 12:30:03.766046 (+   232us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:03.766167 (+   121us) write_op.cc:454] Released schema lock
0709 12:30:03.766212 (+    45us) write_op.cc:341] FINISH: Updating metrics
0709 12:30:03.766751 (+   539us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:03.766779 (+    28us) write_op.cc:454] Released schema lock
0709 12:30:03.809112 (+ 42333us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b180011ad00. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f500044dab1 00007f500045a281 00007f4ff7585522 00007f4ff75877c1 00007f4ff7587779 00007f4ff7587709 00007f4ff75876d1 00007f4ff75869cd 0000000000388634 0000000000388479 00007f4ff756883a
Metrics: {"mutex_wait_us":157804,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":74649,"apply.run_cpu_time_us":9861,"apply.run_wall_time_us":67646,"cfile_cache_hit":18,"cfile_cache_hit_bytes":20847,"num_ops":4,"prepare.queue_time_us":99539,"prepare.run_cpu_time_us":19935,"prepare.run_wall_time_us":102498,"raft.queue_time_us":174831,"raft.run_cpu_time_us":23914,"raft.run_wall_time_us":95021,"replication_time_us":342527,"spinlock_wait_cycles":29971584,"thread_start_us":275362,"threads_started":4,"wal-append.queue_time_us":92836}]]}
I20260709 12:30:04.160928 18213 raft_consensus.cc:1064] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: attempting to promote NON_VOTER 38736cc545eb4389b220a3bd21d3b466 to VOTER
I20260709 12:30:04.165940 18213 consensus_queue.cc:237] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 32.24, Last appended by leader: 22, Current term: 32, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:04.200002  6237 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:04.237833  6237 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:04.281088  5602 consensus_peers.cc:597] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:04.282006  6272 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [term 32 LEARNER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 24. Preceding OpId from leader: term: 32 index: 25. (index mismatch)
I20260709 12:30:04.286603 18248 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:30:04.292764  5703 raft_consensus.cc:1275] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 24. Preceding OpId from leader: term: 32 index: 25. (index mismatch)
I20260709 12:30:04.297493 18137 consensus_queue.cc:1048] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
W20260709 12:30:04.299055  6237 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:04.314378 18273 log.cc:927] Time spent T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: Append to log took a long time: real 0.057s	user 0.004s	sys 0.000s
W20260709 12:30:04.332734 18270 log.cc:927] Time spent T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: Append to log took a long time: real 0.075s	user 0.002s	sys 0.000s
W20260709 12:30:04.349458  5457 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 274) took 197881 ms (client timeout 15000 ms). Trace:
W20260709 12:30:04.349975  5457 rpcz_store.cc:269] 0709 12:26:46.409972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:46.418553 (+  8581us) service_pool.cc:224] Handling call
0709 12:30:04.285739 (+197867186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4976,"reported_tablets":50}
I20260709 12:30:04.396500 18210 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
W20260709 12:30:04.396924  6237 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:04.397264 18210 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:04.397120  5453 catalog_manager.cc:5697] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 30 to 38. New cstate: current_term: 38 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:04.399371  5453 catalog_manager.cc:5697] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 32 to 43. New cstate: current_term: 43 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:04.400889  5453 catalog_manager.cc:5697] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 37 to 38, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 38 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:30:04.403093  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:04.405258  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:04.408092  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:04.408903 18256 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:30:04.408990 18210 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:04.405342  5453 catalog_manager.cc:5697] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 39 to 43, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 43 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:04.426831 18035 raft_consensus.cc:2955] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Committing config change with OpId 32.25: config changed from index 24 to 25, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:04.433941  6279 raft_consensus.cc:2955] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Committing config change with OpId 32.25: config changed from index 24 to 25, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:04.439740  5453 catalog_manager.cc:5697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 32 to 36, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 36 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:30:04.441402  5453 catalog_manager.cc:5697] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 27 to 32. New cstate: current_term: 32 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:30:04.442715  5453 catalog_manager.cc:5697] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 29 to 33. New cstate: current_term: 33 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:04.444641  5453 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 30 to 32. New cstate: current_term: 32 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:04.452492  5453 catalog_manager.cc:5697] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 34 to 36, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 36 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:04.454084  5453 catalog_manager.cc:5697] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 33 to 36, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 36 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:04.455390  5453 catalog_manager.cc:5697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 27 to 32, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 32 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:30:04.456139  5676 consensus_peers.cc:597] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:04.473955  5453 catalog_manager.cc:5697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 24 to 32, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 32 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:30:04.475725  5453 catalog_manager.cc:5697] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 23 to 33, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 33 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:04.485776  5453 catalog_manager.cc:5697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 36 to 38, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 38 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } }
I20260709 12:30:04.487629  5453 catalog_manager.cc:5697] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:04.489223  5453 catalog_manager.cc:5697] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 36 to 38, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 38 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:04.491032  5453 catalog_manager.cc:5697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 35 to 38, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 38 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:30:04.498366 18210 raft_consensus.cc:493] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:04.499480 18210 raft_consensus.cc:515] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:04.504105  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:04.508206  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f7bab74248a4aeda157f7a50fa82233" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:04.508854  5453 catalog_manager.cc:5697] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 33 to 34, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 34 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:30:04.509913  5526 leader_election.cc:304] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:04.510687  5453 catalog_manager.cc:5697] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 27 to 37, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 37 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:04.516506 18255 raft_consensus.cc:2749] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:30:04.514783  5453 catalog_manager.cc:5697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index -1 to 24, term changed from 24 to 32, NON_VOTER 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) added. New cstate: current_term: 32 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: NON_VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:04.522280  5453 catalog_manager.cc:5697] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 28 to 35, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 35 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:04.529656  5704 raft_consensus.cc:2955] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Committing config change with OpId 32.25: config changed from index 24 to 25, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:04.513595 18210 leader_election.cc:290] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:04.524237  5453 catalog_manager.cc:5697] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 38 to 39, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 39 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:04.538012  5453 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 33 to 36, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 36 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:30:04.550604  5602 consensus_peers.cc:597] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:04.539783  5453 catalog_manager.cc:5697] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 36 to 37, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 37 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:04.561568  5453 catalog_manager.cc:5697] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 29 to 36, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 36 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:04.563319  5453 catalog_manager.cc:5697] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 32 to 35, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 35 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:30:04.570417  5453 catalog_manager.cc:5697] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 36 to 38. New cstate: current_term: 38 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:30:04.572055  5453 catalog_manager.cc:5697] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 36 to 37, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 37 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
W20260709 12:30:04.586107  6237 consensus_peers.cc:597] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:04.579456  5453 catalog_manager.cc:5697] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 34 to 36. New cstate: current_term: 36 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:04.593469  5453 catalog_manager.cc:5697] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 26 to 34, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 34 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:04.595645  5453 catalog_manager.cc:5697] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 33 to 34, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 34 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:04.609483  5453 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 28 to 31. New cstate: current_term: 31 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:04.611141  5453 catalog_manager.cc:5697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 31 to 33, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 33 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:30:04.626549  5602 consensus_peers.cc:597] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:04.654810  6279 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:30:04.656790 18251 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:30:04.677273 18255 raft_consensus.cc:493] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:04.678015 18255 raft_consensus.cc:515] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:04.683076  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:04.687139 18255 leader_election.cc:290] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:04.688730  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca6496163ec40978a2dfd744edddb18" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:04.691526  5526 leader_election.cc:304] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:04.693487 18256 raft_consensus.cc:2749] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:30:04.694852  5629 raft_consensus.cc:1275] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:30:04.699064 18251 consensus_queue.cc:1048] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
W20260709 12:30:04.718459  5602 consensus_peers.cc:597] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:04.731185  5602 consensus_peers.cc:597] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:04.767254  4356 thread.cc:527] Waited for 10000ms trying to join with heartbeat (tid 5586)
W20260709 12:30:04.772647  5446 catalog_manager.cc:4755] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 26edf338d41e42f1a5ce0d4416ec67e1: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer
I20260709 12:30:04.779052 18210 raft_consensus.cc:493] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:04.779701 18210 raft_consensus.cc:515] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:04.783020  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:30:04.788479  5602 consensus_peers.cc:597] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:04.790120  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f0f7b2d2a7e34d629491c96dbfbc7fdc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:04.795219  5524 leader_election.cc:304] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:04.796770 18255 raft_consensus.cc:2749] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:30:04.802789 18210 leader_election.cc:290] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:04.810182 18210 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:04.810772 18210 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:04.814360  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:30:04.820989  6237 consensus_peers.cc:597] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:04.825090 18210 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:04.826651  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:04.829118  5524 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:04.830919 18255 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:30:04.924777  5602 consensus_peers.cc:597] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:04.953361 18255 raft_consensus.cc:493] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:30:04.962621 18255 raft_consensus.cc:515] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:04.968737  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 72603.6us
I20260709 12:30:04.984207  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:04.984207  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69273530292453d87614627debcc13e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:04.988094 18255 leader_election.cc:290] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:04.989244  5526 leader_election.cc:304] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:04.993837 18210 raft_consensus.cc:2749] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:30:05.023586 18210 raft_consensus.cc:493] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:05.024307 18210 raft_consensus.cc:515] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:05.051820 18210 leader_election.cc:290] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:05.052922  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:05.052910  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addbcf2919194d2e866b5e8826422851" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:05.056514  5526 leader_election.cc:304] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:05.057632 18210 raft_consensus.cc:2749] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:30:05.157296 18210 raft_consensus.cc:493] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:05.157892 18210 raft_consensus.cc:515] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:05.166208 18210 leader_election.cc:290] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:05.172432  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:05.178963 18256 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:05.179437  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "637a73babedf4b1aa47b3b50937db9ef" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:05.179502 18256 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:05.182350  5526 leader_election.cc:304] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:05.183945 18255 raft_consensus.cc:2749] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:05.184544  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:05.186480  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:05.183018 18256 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:05.188438  5526 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:05.196552 18210 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:30:05.240036 18210 raft_consensus.cc:493] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:05.240592 18210 raft_consensus.cc:515] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:05.242833 18210 leader_election.cc:290] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:05.245378  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:05.246503  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "832269e5980743c48521740b22eefa9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:05.252286  5524 leader_election.cc:304] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:05.254201 18210 raft_consensus.cc:2749] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:30:05.280025 18210 raft_consensus.cc:493] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:05.280958 18210 raft_consensus.cc:515] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:05.283181 18210 leader_election.cc:290] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:05.284574  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:05.287196  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d439f3f9c7c40288efc981cec05e7ab" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 34 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:05.288836  5524 leader_election.cc:304] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:05.295969  5453 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35600 (request call id 285) took 3718 ms. Trace:
I20260709 12:30:05.308859 18210 raft_consensus.cc:2749] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:05.314656  5453 rpcz_store.cc:276] 0709 12:30:01.577099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:30:03.746164 (+2169065us) service_pool.cc:224] Handling call
0709 12:30:05.295846 (+1549682us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:30:04.668631 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:30:04.672960 (+  4329us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:30:04.673006 (+    46us) write_op.cc:435] Acquired schema lock
0709 12:30:04.673025 (+    19us) tablet.cc:662] Decoding operations
0709 12:30:04.675630 (+  2605us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:30:04.675725 (+    95us) write_op.cc:641] Partition lock acquired for write op
0709 12:30:04.675761 (+    36us) tablet.cc:685] Acquiring locks for 33 operations
0709 12:30:04.698509 (+ 22748us) tablet.cc:701] Row locks acquired
0709 12:30:04.698543 (+    34us) write_op.cc:260] PREPARE: finished
0709 12:30:04.698724 (+   181us) write_op.cc:270] Start()
0709 12:30:04.698870 (+   146us) write_op.cc:276] Timestamp: P: 1783600204698656 usec, L: 0
0709 12:30:04.698897 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 12:30:04.699625 (+   728us) log.cc:844] Serialized 12652 byte log entry
0709 12:30:04.713745 (+ 14120us) op_driver.cc:464] REPLICATION: finished
0709 12:30:04.725321 (+ 11576us) write_op.cc:301] APPLY: starting
0709 12:30:04.725433 (+   112us) tablet.cc:1367] starting BulkCheckPresence
0709 12:30:04.882725 (+157292us) tablet.cc:1370] finished BulkCheckPresence
0709 12:30:04.882873 (+   148us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:30:05.098151 (+215278us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:30:05.098768 (+   617us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=66,key_file_lookups=66,delta_file_lookups=0,mrs_lookups=0
0709 12:30:05.098833 (+    65us) write_op.cc:312] APPLY: finished
0709 12:30:05.099481 (+   648us) log.cc:844] Serialized 284 byte log entry
0709 12:30:05.120894 (+ 21413us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:05.121199 (+   305us) write_op.cc:454] Released schema lock
0709 12:30:05.121285 (+    86us) write_op.cc:341] FINISH: Updating metrics
0709 12:30:05.121793 (+   508us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:05.121827 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":128565,"reported_tablets":44,"spinlock_wait_cycles":2432,"child_traces":[["op",{"apply.queue_time_us":11278,"apply.run_cpu_time_us":46731,"apply.run_wall_time_us":396412,"cfile_cache_hit":134,"cfile_cache_hit_bytes":142067,"num_ops":33,"prepare.queue_time_us":5961,"prepare.run_cpu_time_us":7776,"prepare.run_wall_time_us":41916,"replication_time_us":14664,"thread_start_us":36348,"threads_started":2}]]}
I20260709 12:30:05.325647  5451 catalog_manager.cc:5697] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 35 to 38. New cstate: current_term: 38 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:05.331640  5451 catalog_manager.cc:5697] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 32 to 39. New cstate: current_term: 39 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:05.355343  5458 catalog_manager.cc:5244] ChangeConfig:REMOVE_PEER RPC for tablet 26edf338d41e42f1a5ce0d4416ec67e1 with cas_config_opid_index 3: aborting the task: latest config opid_index 11; task opid_index 3
I20260709 12:30:05.356302  5451 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 31 to 32, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 32 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:05.370383  5451 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 29 to 32. New cstate: current_term: 32 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:05.382272  5451 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 32 to 33, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 33 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:05.393536  5451 catalog_manager.cc:5697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 29 to 35, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 35 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:05.386329 18256 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:30:05.368922  5458 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 177) took 213136 ms (client timeout 15000 ms). Trace:
I20260709 12:30:05.404317 18256 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:05.404378  5458 rpcz_store.cc:269] 0709 12:26:32.232990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.250638 (+ 17648us) service_pool.cc:224] Handling call
0709 12:30:05.368830 (+213118192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106533,"reported_tablets":32}
I20260709 12:30:05.404801 18255 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:05.406134 18255 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:05.406962 18256 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:05.408308 18255 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:05.409514  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:05.409920  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:05.410722  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:05.410594  5451 catalog_manager.cc:5697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 29 to 35. New cstate: current_term: 35 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:30:05.411201  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b21feec8630479b94ed9384256483fc" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:05.413112  5526 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:05.414780  5526 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:05.417789 18256 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:30:05.420493 18255 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:30:05.451810  5602 consensus_peers.cc:597] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:05.515821  5602 consensus_peers.cc:597] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:05.527128  5453 catalog_manager.cc:5697] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 32 to 39, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 39 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:30:05.529670  5453 catalog_manager.cc:5697] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 35 to 37. New cstate: current_term: 37 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
W20260709 12:30:05.578423  6237 consensus_peers.cc:597] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:05.531451  5453 catalog_manager.cc:5697] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 32 to 34, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 34 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:05.579872  5453 catalog_manager.cc:5697] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 33 to 35, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 35 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:05.581445  5453 catalog_manager.cc:5697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 32 to 34. New cstate: current_term: 34 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:30:05.582770  5453 catalog_manager.cc:5697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 32 to 34. New cstate: current_term: 34 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:05.614627  5453 catalog_manager.cc:5697] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: config changed from index 24 to 25, 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) changed from NON_VOTER to VOTER. New cstate: current_term: 32 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:05.617558  5453 catalog_manager.cc:5697] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 36 to 38, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 38 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:30:05.619678  5453 catalog_manager.cc:5697] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 31 to 32, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 32 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:05.621183  5453 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 32 to 35, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 35 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } }
I20260709 12:30:05.641036  5451 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 281) took 10728 ms. Trace:
I20260709 12:30:05.641706  5451 rpcz_store.cc:276] 0709 12:29:54.912737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:56.912476 (+1999739us) service_pool.cc:224] Handling call
0709 12:30:05.640941 (+8728465us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:30:05.461398 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:30:05.461932 (+   534us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:30:05.461970 (+    38us) write_op.cc:435] Acquired schema lock
0709 12:30:05.461989 (+    19us) tablet.cc:662] Decoding operations
0709 12:30:05.462771 (+   782us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:30:05.462839 (+    68us) write_op.cc:641] Partition lock acquired for write op
0709 12:30:05.462865 (+    26us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:30:05.463437 (+   572us) tablet.cc:701] Row locks acquired
0709 12:30:05.463454 (+    17us) write_op.cc:260] PREPARE: finished
0709 12:30:05.463592 (+   138us) write_op.cc:270] Start()
0709 12:30:05.463708 (+   116us) write_op.cc:276] Timestamp: P: 1783600205463545 usec, L: 0
0709 12:30:05.463734 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 12:30:05.476553 (+ 12819us) log.cc:844] Serialized 2782 byte log entry
0709 12:30:05.489948 (+ 13395us) op_driver.cc:464] REPLICATION: finished
0709 12:30:05.490361 (+   413us) write_op.cc:301] APPLY: starting
0709 12:30:05.490483 (+   122us) tablet.cc:1367] starting BulkCheckPresence
0709 12:30:05.494187 (+  3704us) tablet.cc:1370] finished BulkCheckPresence
0709 12:30:05.494282 (+    95us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:30:05.503894 (+  9612us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:30:05.504258 (+   364us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=0,mrs_lookups=0
0709 12:30:05.504322 (+    64us) write_op.cc:312] APPLY: finished
0709 12:30:05.504702 (+   380us) log.cc:844] Serialized 73 byte log entry
0709 12:30:05.508549 (+  3847us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:05.508678 (+   129us) write_op.cc:454] Released schema lock
0709 12:30:05.508724 (+    46us) write_op.cc:341] FINISH: Updating metrics
0709 12:30:05.509197 (+   473us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:05.509240 (+    43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":46734,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":141,"apply.run_cpu_time_us":9719,"apply.run_wall_time_us":18735,"cfile_cache_hit":28,"cfile_cache_hit_bytes":29260,"num_ops":7,"prepare.queue_time_us":17769,"prepare.run_cpu_time_us":3638,"prepare.run_wall_time_us":22584,"replication_time_us":25978,"thread_start_us":27789,"threads_started":1}]]}
W20260709 12:30:05.697350  6237 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:05.720883  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 90226.6us
W20260709 12:30:05.767781  4356 thread.cc:527] Waited for 11000ms trying to join with heartbeat (tid 5586)
I20260709 12:30:05.773028 18255 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:05.773751 18255 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:05.776903  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 44 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:05.780128 18255 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:05.780339  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d78f6c4aaa14250ba8528ff2fe6af34" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 44 candidate_status { last_received { term: 41 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:05.785730  5524 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:05.786870 18210 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:30:05.817060 18210 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:05.830312 18210 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:05.833469 18210 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:05.834604  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:05.837589  5455 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 25 to 31. New cstate: current_term: 31 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:05.841455  5455 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 33 to 36. New cstate: current_term: 36 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:05.858479  5455 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 30 to 31, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 31 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:30:05.873435  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 44628.1us
I20260709 12:30:05.874056  5455 catalog_manager.cc:5697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 26 to 32. New cstate: current_term: 32 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:05.880836  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "feb9fb12cc5a41e6958575c17961252f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:05.899055  5526 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:05.899861 18256 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:30:05.927258  5446 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 104728us
I20260709 12:30:05.929566 18255 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:05.930146 18255 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:05.931849  5446 catalog_manager.cc:4755] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 26edf338d41e42f1a5ce0d4416ec67e1: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer
I20260709 12:30:05.935828  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:05.936110 18256 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:05.936681 18256 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:05.937000 18255 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:05.938781 18256 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:05.939544  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52ccafecca54436198d9d455d076ef8d" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 37 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:05.940649  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:05.941835  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "733a5b54e4dd4f86999821705033529f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:05.942795  5526 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:05.943253  5524 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:05.944495 18255 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:30:05.944630 18256 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:30:06.029208  5455 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 284) took 33047 ms (client timeout 15000 ms). Trace:
W20260709 12:30:06.029665  5455 rpcz_store.cc:269] 0709 12:29:32.982363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:41.495827 (+8513464us) service_pool.cc:224] Handling call
0709 12:30:06.029134 (+24533307us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:30:05.932634 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:30:05.933154 (+   520us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:30:05.933188 (+    34us) write_op.cc:435] Acquired schema lock
0709 12:30:05.933204 (+    16us) tablet.cc:662] Decoding operations
0709 12:30:05.933796 (+   592us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:30:05.933849 (+    53us) write_op.cc:641] Partition lock acquired for write op
0709 12:30:05.933873 (+    24us) tablet.cc:685] Acquiring locks for 4 operations
0709 12:30:05.934265 (+   392us) tablet.cc:701] Row locks acquired
0709 12:30:05.934283 (+    18us) write_op.cc:260] PREPARE: finished
0709 12:30:05.934421 (+   138us) write_op.cc:270] Start()
0709 12:30:05.934529 (+   108us) write_op.cc:276] Timestamp: P: 1783600205934378 usec, L: 0
0709 12:30:05.934550 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 12:30:05.936394 (+  1844us) log.cc:844] Serialized 1646 byte log entry
0709 12:30:05.938516 (+  2122us) op_driver.cc:464] REPLICATION: finished
0709 12:30:05.939993 (+  1477us) write_op.cc:301] APPLY: starting
0709 12:30:05.940249 (+   256us) tablet.cc:1367] starting BulkCheckPresence
0709 12:30:05.943733 (+  3484us) tablet.cc:1370] finished BulkCheckPresence
0709 12:30:05.943840 (+   107us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:30:05.949027 (+  5187us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:30:05.949159 (+   132us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=8,key_file_lookups=8,delta_file_lookups=0,mrs_lookups=0
0709 12:30:05.949187 (+    28us) write_op.cc:312] APPLY: finished
0709 12:30:05.949365 (+   178us) log.cc:844] Serialized 49 byte log entry
0709 12:30:05.949710 (+   345us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:05.949848 (+   138us) write_op.cc:454] Released schema lock
0709 12:30:05.949909 (+    61us) write_op.cc:341] FINISH: Updating metrics
0709 12:30:05.950512 (+   603us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:05.950547 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":1165,"apply.run_cpu_time_us":6268,"apply.run_wall_time_us":10545,"cfile_cache_hit":16,"cfile_cache_hit_bytes":16720,"num_ops":4,"prepare.queue_time_us":935,"prepare.run_cpu_time_us":2889,"prepare.run_wall_time_us":4023,"replication_time_us":3819}]]}
W20260709 12:30:06.046263  5602 consensus_peers.cc:597] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:06.062553 18256 raft_consensus.cc:493] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:30:06.063261 18256 raft_consensus.cc:515] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:06.066475 18256 leader_election.cc:290] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:06.068446  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:06.069034  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e20e319f46504838a20c558a66692da6" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:06.071014  5526 leader_election.cc:304] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:06.072175 18256 raft_consensus.cc:2749] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:30:06.062565  6237 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:06.091166  6237 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:06.097210  5459 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 31 to 34, leader changed from 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 34 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:06.099424 18256 raft_consensus.cc:493] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:06.113097 18256 raft_consensus.cc:515] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:06.115355 18256 leader_election.cc:290] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:06.116993  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:06.121594  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "021ceb67cc2f4876b5dbdf1ca21c5f07" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:06.125360  5524 leader_election.cc:304] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:06.126405 18256 raft_consensus.cc:2749] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:30:06.193864  6237 consensus_peers.cc:597] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:06.195422  6237 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:06.218916  5602 consensus_peers.cc:597] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:06.222386 18256 raft_consensus.cc:493] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:06.223693 18256 raft_consensus.cc:515] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:06.230602  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:06.231675  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35344119b8ff4cdebc4810661ff94f79" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 33 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:06.233599  5524 leader_election.cc:304] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:06.234256 18256 leader_election.cc:290] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:06.235292 18295 raft_consensus.cc:2749] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:30:06.247195  6237 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:06.251744  6237 consensus_peers.cc:597] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:06.258400  5450 catalog_manager.cc:5697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 32 to 34, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 34 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:06.282013 18256 raft_consensus.cc:493] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:06.282658 18256 raft_consensus.cc:515] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:06.287755  5676 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:06.289031  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:06.296103 18256 leader_election.cc:290] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:06.298062  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa0101c69f7746009c5d77c9616f3360" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 26 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:06.304503  5526 leader_election.cc:304] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:06.305899 18256 raft_consensus.cc:2749] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:30:06.307333  6237 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:06.372596  5459 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 280) took 29651 ms (client timeout 15000 ms). Trace:
W20260709 12:30:06.373317  5459 rpcz_store.cc:269] 0709 12:29:36.720800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:41.623862 (+4903062us) service_pool.cc:224] Handling call
0709 12:30:06.372491 (+24748629us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:30:06.129130 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:30:06.129884 (+   754us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:30:06.129924 (+    40us) write_op.cc:435] Acquired schema lock
0709 12:30:06.129948 (+    24us) tablet.cc:662] Decoding operations
0709 12:30:06.130521 (+   573us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:30:06.130601 (+    80us) write_op.cc:641] Partition lock acquired for write op
0709 12:30:06.130653 (+    52us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:30:06.130887 (+   234us) tablet.cc:701] Row locks acquired
0709 12:30:06.130908 (+    21us) write_op.cc:260] PREPARE: finished
0709 12:30:06.131079 (+   171us) write_op.cc:270] Start()
0709 12:30:06.131204 (+   125us) write_op.cc:276] Timestamp: P: 1783600206131024 usec, L: 0
0709 12:30:06.131227 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:30:06.151657 (+ 20430us) log.cc:844] Serialized 527 byte log entry
0709 12:30:06.166810 (+ 15153us) op_driver.cc:464] REPLICATION: finished
0709 12:30:06.169488 (+  2678us) write_op.cc:301] APPLY: starting
0709 12:30:06.169598 (+   110us) tablet.cc:1367] starting BulkCheckPresence
0709 12:30:06.170657 (+  1059us) tablet.cc:1370] finished BulkCheckPresence
0709 12:30:06.170778 (+   121us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:30:06.171611 (+   833us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:30:06.171811 (+   200us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0709 12:30:06.171872 (+    61us) write_op.cc:312] APPLY: finished
0709 12:30:06.172132 (+   260us) log.cc:844] Serialized 25 byte log entry
0709 12:30:06.172355 (+   223us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:06.174971 (+  2616us) write_op.cc:454] Released schema lock
0709 12:30:06.175023 (+    52us) write_op.cc:341] FINISH: Updating metrics
0709 12:30:06.186839 (+ 11816us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa66de28 00007f4ffe0a8581 00007f4ffe0a7d58 00007f4ffe0ac9f1 00007f4ffe0ac9a9 00007f4ffe0ac939 00007f4ffe0ac901 00007f4ffe0abb2d 0000000000388634 0000000000388479
0709 12:30:06.187081 (+   242us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:06.187117 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":25484,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":2340,"apply.run_cpu_time_us":3505,"apply.run_wall_time_us":5949,"cfile_cache_hit":4,"cfile_cache_hit_bytes":4180,"num_ops":1,"prepare.queue_time_us":2242,"prepare.run_cpu_time_us":6256,"prepare.run_wall_time_us":58071,"replication_time_us":35328,"spinlock_wait_cycles":44862208}]]}
W20260709 12:30:06.417439  5676 consensus_peers.cc:597] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:06.450244  5450 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 185) took 22592 ms (client timeout 15000 ms). Trace:
W20260709 12:30:06.457387  5450 rpcz_store.cc:269] 0709 12:29:43.864389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:29:53.276623 (+9412234us) service_pool.cc:224] Handling call
0709 12:30:06.450134 (+13173511us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:30:06.269416 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:30:06.269986 (+   570us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:30:06.270025 (+    39us) write_op.cc:435] Acquired schema lock
0709 12:30:06.270042 (+    17us) tablet.cc:662] Decoding operations
0709 12:30:06.270369 (+   327us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:30:06.270420 (+    51us) write_op.cc:641] Partition lock acquired for write op
0709 12:30:06.270451 (+    31us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:30:06.270662 (+   211us) tablet.cc:701] Row locks acquired
0709 12:30:06.270679 (+    17us) write_op.cc:260] PREPARE: finished
0709 12:30:06.270814 (+   135us) write_op.cc:270] Start()
0709 12:30:06.270932 (+   118us) write_op.cc:276] Timestamp: P: 1783600206270769 usec, L: 0
0709 12:30:06.270955 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:30:06.274759 (+  3804us) log.cc:844] Serialized 527 byte log entry
0709 12:30:06.276692 (+  1933us) op_driver.cc:464] REPLICATION: finished
0709 12:30:06.278263 (+  1571us) write_op.cc:301] APPLY: starting
0709 12:30:06.278346 (+    83us) tablet.cc:1367] starting BulkCheckPresence
0709 12:30:06.279148 (+   802us) tablet.cc:1370] finished BulkCheckPresence
0709 12:30:06.279238 (+    90us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:30:06.279908 (+   670us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:30:06.280064 (+   156us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0709 12:30:06.280109 (+    45us) write_op.cc:312] APPLY: finished
0709 12:30:06.280301 (+   192us) log.cc:844] Serialized 25 byte log entry
0709 12:30:06.280482 (+   181us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:06.280573 (+    91us) write_op.cc:454] Released schema lock
0709 12:30:06.280613 (+    40us) write_op.cc:341] FINISH: Updating metrics
0709 12:30:06.304561 (+ 23948us) spinlock_profiling.cc:243] Waited 3.13 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5db571 00007f4ffa66de28 00007f4ffe0a8581 00007f4ffe0a7d58 00007f4ffe0ac9f1 00007f4ffe0ac9a9 00007f4ffe0ac939 00007f4ffe0ac901 00007f4ffe0abb2d 0000000000388634 0000000000388479
0709 12:30:06.304748 (+   187us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:06.304775 (+    27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":62671,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":1345,"apply.run_cpu_time_us":2673,"apply.run_wall_time_us":2674,"cfile_cache_hit":4,"cfile_cache_hit_bytes":4180,"num_ops":1,"prepare.queue_time_us":4461,"prepare.run_cpu_time_us":3924,"prepare.run_wall_time_us":35319,"replication_time_us":5610,"spinlock_wait_cycles":7199104}]]}
I20260709 12:30:06.456051  5456 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 36 to 38, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 38 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
W20260709 12:30:06.472118  5602 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 275 similar messages]
W20260709 12:30:06.485775  6237 consensus_peers.cc:597] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:06.491003  5456 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f reported cstate change: term changed from 31 to 33. New cstate: current_term: 33 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:30:06.558037  6237 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:06.566097  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 59923.3us
W20260709 12:30:06.584996  5602 consensus_peers.cc:597] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:06.601554  5676 consensus_peers.cc:597] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:06.612877  6237 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:06.645135 18255 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:06.646013 18255 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:06.649762 18255 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:06.651137  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:06.660687  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90c64867bae6493c846d941815aa1074" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:06.662672  5524 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:06.659207  5452 catalog_manager.cc:5697] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 38 to 40. New cstate: current_term: 40 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:06.665035 18255 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:30:06.673362  5452 catalog_manager.cc:5697] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 34 to 37, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 37 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:06.680275  5452 catalog_manager.cc:5697] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 29 to 35, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 35 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:06.691218  5456 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35586 (request call id 285) took 9590 ms. Trace:
W20260709 12:30:06.699348  5602 consensus_peers.cc:597] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:06.699987 18295 raft_consensus.cc:493] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:06.701566 18295 raft_consensus.cc:515] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:06.704862  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:06.705027  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2578c6c739884fad8fa75e16cfa14fec" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 41 candidate_status { last_received { term: 39 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:06.706486  5526 leader_election.cc:304] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:06.707357 18256 raft_consensus.cc:2749] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:30:06.708407 18295 leader_election.cc:290] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:06.698932  5452 catalog_manager.cc:5697] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 31 to 33. New cstate: current_term: 33 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:06.730768  5452 catalog_manager.cc:5697] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 34 to 36. New cstate: current_term: 36 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:30:06.731940  5602 consensus_peers.cc:597] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:06.740293  5452 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 32 to 33, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 33 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:06.736696  5456 rpcz_store.cc:276] 0709 12:29:57.101248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:30:00.572107 (+3470859us) service_pool.cc:224] Handling call
0709 12:30:06.691135 (+6119028us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:30:06.584343 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:30:06.585281 (+   938us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:30:06.585450 (+   169us) write_op.cc:435] Acquired schema lock
0709 12:30:06.585463 (+    13us) tablet.cc:662] Decoding operations
0709 12:30:06.585860 (+   397us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:30:06.585947 (+    87us) write_op.cc:641] Partition lock acquired for write op
0709 12:30:06.585977 (+    30us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:30:06.586318 (+   341us) tablet.cc:701] Row locks acquired
0709 12:30:06.586334 (+    16us) write_op.cc:260] PREPARE: finished
0709 12:30:06.586488 (+   154us) write_op.cc:270] Start()
0709 12:30:06.586707 (+   219us) write_op.cc:276] Timestamp: P: 1783600206586440 usec, L: 0
0709 12:30:06.586731 (+    24us) op_driver.cc:348] REPLICATION: starting
0709 12:30:06.587443 (+   712us) log.cc:844] Serialized 901 byte log entry
0709 12:30:06.593137 (+  5694us) op_driver.cc:464] REPLICATION: finished
0709 12:30:06.594267 (+  1130us) write_op.cc:301] APPLY: starting
0709 12:30:06.594358 (+    91us) tablet.cc:1367] starting BulkCheckPresence
0709 12:30:06.595820 (+  1462us) tablet.cc:1370] finished BulkCheckPresence
0709 12:30:06.595983 (+   163us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:30:06.597411 (+  1428us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:30:06.597592 (+   181us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 12:30:06.597637 (+    45us) write_op.cc:312] APPLY: finished
0709 12:30:06.597899 (+   262us) log.cc:844] Serialized 33 byte log entry
0709 12:30:06.598070 (+   171us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:06.598173 (+   103us) write_op.cc:454] Released schema lock
0709 12:30:06.598223 (+    50us) write_op.cc:341] FINISH: Updating metrics
0709 12:30:06.608885 (+ 10662us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:06.608932 (+    47us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4431,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":870,"apply.run_cpu_time_us":4362,"apply.run_wall_time_us":14586,"cfile_cache_hit":8,"cfile_cache_hit_bytes":8360,"num_ops":2,"prepare.queue_time_us":129,"prepare.run_cpu_time_us":3132,"prepare.run_wall_time_us":8498,"replication_time_us":6264}]]}
I20260709 12:30:06.754021  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:30:06.757477  4356 tablet_replica.cc:333] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:06.759222  4356 raft_consensus.cc:2243] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:06.760047  4356 raft_consensus.cc:2272] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:06.768683 18255 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:06.769251 18255 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:06.772115 18255 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:06.774899  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:06.775565  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52cee1d09af749489de4689a67047497" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 33 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:06.780102  5524 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:06.783223 18255 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:30:06.786456  4356 tablet_replica.cc:333] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:06.788784  4356 raft_consensus.cc:2243] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:06.813241  4356 pending_rounds.cc:70] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Trying to abort 1 pending ops.
I20260709 12:30:06.814086  4356 pending_rounds.cc:77] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f: Aborting op as it isn't in flight: id { term: 41 index: 25 } timestamp: 7305626326179303424 op_type: NO_OP noop_request { }
I20260709 12:30:06.790004  5452 catalog_manager.cc:5697] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 33 to 34, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 34 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:06.817764  5452 catalog_manager.cc:5697] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 35 to 36, leader changed from f613ba929b1046d3b5aed909414a95bf (127.4.65.3) to cea66aa15be3423880843974acdbf50f (127.4.65.1). New cstate: current_term: 36 leader_uuid: "cea66aa15be3423880843974acdbf50f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } } }
I20260709 12:30:06.821439  5452 catalog_manager.cc:5697] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 31 to 34, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 34 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:06.815749  4356 raft_consensus.cc:2889] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:30:06.822902  4356 raft_consensus.cc:2272] T 5d78f6c4aaa14250ba8528ff2fe6af34 P cea66aa15be3423880843974acdbf50f [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:06.838645  4356 tablet_replica.cc:333] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:06.842490  4356 raft_consensus.cc:2243] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:06.843302  4356 raft_consensus.cc:2272] T d816d235ce5a41c796c8949c48c0a53c P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:06.873513  4356 tablet_replica.cc:333] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:06.874982  4356 raft_consensus.cc:2243] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:06.875795  4356 raft_consensus.cc:2272] T edd23660363f48bf95cee8a70c5a2f26 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:06.889139  4356 tablet_replica.cc:333] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f: stopping tablet replica
W20260709 12:30:06.890893  5602 consensus_peers.cc:597] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:06.890442  4356 raft_consensus.cc:2243] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:06.911866  4356 raft_consensus.cc:2272] T fdcede9bc9104440a7ce1e3a4eb3a86f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:06.915752  4356 tablet_replica.cc:333] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:06.917071  4356 raft_consensus.cc:2243] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:06.917768  4356 raft_consensus.cc:2272] T 2578c6c739884fad8fa75e16cfa14fec P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:06.937069  4356 tablet_replica.cc:333] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:06.951673  4356 raft_consensus.cc:2243] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:30:06.953114  4356 raft_consensus.cc:2272] T 458143f1c77141febe7d9725625617c2 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:06.953830  5676 consensus_peers.cc:597] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:06.964811  4356 tablet_replica.cc:333] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:06.966087  4356 raft_consensus.cc:2243] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:06.966840  4356 raft_consensus.cc:2272] T 785e79f6fbd0463aafaca5498918a142 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:06.968173  6237 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:06.983229  4356 tablet_replica.cc:333] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:06.985608  4356 raft_consensus.cc:2243] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:06.986280  4356 pending_rounds.cc:70] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Trying to abort 1 pending ops.
I20260709 12:30:06.986552  4356 pending_rounds.cc:77] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f: Aborting op as it isn't in flight: id { term: 25 index: 19 } timestamp: 7305626369239621632 op_type: NO_OP noop_request { }
I20260709 12:30:06.986956  4356 raft_consensus.cc:2889] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:30:06.987552  4356 raft_consensus.cc:2272] T 9f7bab74248a4aeda157f7a50fa82233 P cea66aa15be3423880843974acdbf50f [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:07.012548  4356 tablet_replica.cc:333] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:07.013983  4356 raft_consensus.cc:2243] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:30:07.015444  4356 raft_consensus.cc:2272] T 64af413ffb404a58b5adea5cb742bf01 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:07.021288  6237 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:07.042349  4356 tablet_replica.cc:333] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f: stopping tablet replica
W20260709 12:30:07.044837  6237 consensus_peers.cc:597] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:07.056291  4356 raft_consensus.cc:2243] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:07.057335  4356 raft_consensus.cc:2272] T 832269e5980743c48521740b22eefa9f P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:07.057354  5452 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38466 (request call id 186) took 6802 ms. Trace:
I20260709 12:30:07.064877  5452 rpcz_store.cc:276] 0709 12:30:00.254875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:30:03.202533 (+2947658us) service_pool.cc:224] Handling call
0709 12:30:07.057249 (+3854716us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:30:06.861423 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:30:06.862056 (+   633us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:30:06.862101 (+    45us) write_op.cc:435] Acquired schema lock
0709 12:30:06.862131 (+    30us) tablet.cc:662] Decoding operations
0709 12:30:06.863129 (+   998us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:30:06.863191 (+    62us) write_op.cc:641] Partition lock acquired for write op
0709 12:30:06.863230 (+    39us) tablet.cc:685] Acquiring locks for 9 operations
0709 12:30:06.864035 (+   805us) tablet.cc:701] Row locks acquired
0709 12:30:06.864054 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:30:06.864210 (+   156us) write_op.cc:270] Start()
0709 12:30:06.864336 (+   126us) write_op.cc:276] Timestamp: P: 1783600206864158 usec, L: 0
0709 12:30:06.864385 (+    49us) op_driver.cc:348] REPLICATION: starting
0709 12:30:06.865442 (+  1057us) log.cc:844] Serialized 3516 byte log entry
0709 12:30:06.871509 (+  6067us) op_driver.cc:464] REPLICATION: finished
0709 12:30:06.872789 (+  1280us) write_op.cc:301] APPLY: starting
0709 12:30:06.872875 (+    86us) tablet.cc:1367] starting BulkCheckPresence
0709 12:30:06.947959 (+ 75084us) tablet.cc:1370] finished BulkCheckPresence
0709 12:30:06.948113 (+   154us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:30:07.020769 (+ 72656us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:30:07.022253 (+  1484us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=18,key_file_lookups=18,delta_file_lookups=0,mrs_lookups=0
0709 12:30:07.022303 (+    50us) write_op.cc:312] APPLY: finished
0709 12:30:07.022900 (+   597us) log.cc:844] Serialized 89 byte log entry
0709 12:30:07.023859 (+   959us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:07.024045 (+   186us) write_op.cc:454] Released schema lock
0709 12:30:07.024108 (+    63us) write_op.cc:341] FINISH: Updating metrics
0709 12:30:07.024618 (+   510us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:07.024652 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":374866,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":984,"apply.run_cpu_time_us":13755,"apply.run_wall_time_us":151763,"cfile_cache_hit":36,"cfile_cache_hit_bytes":37620,"num_ops":9,"prepare.queue_time_us":132,"prepare.run_cpu_time_us":4491,"prepare.run_wall_time_us":8617,"replication_time_us":6859,"spinlock_wait_cycles":106368}]]}
I20260709 12:30:07.121135  4356 tablet_replica.cc:333] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:07.135427  4356 raft_consensus.cc:2243] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:07.142879  5451 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:35602 (request call id 282) took 1214 ms. Trace:
I20260709 12:30:07.143340  5451 rpcz_store.cc:276] 0709 12:30:05.928609 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:30:05.928771 (+   162us) service_pool.cc:224] Handling call
0709 12:30:05.955834 (+ 27063us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b5400364f54. stack: 00007f4ff7553343 00007f4ff7553240 00007f4ff6ee21d4 00007f5000c20304 00007f5000c20299 00007f4ffa5dc090 00007f4ffa67af3a 00007f5000430e9c 00007f500052f04e 00007f4ffe9f02a4 00007f4ffe9f0232 00007f4ffe9f0161 00007f4ffe9f00dc 00007f4ffe9ef392 00007f4ff89ad48c 00007f4ff89ac8c6
0709 12:30:07.142811 (+1186977us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6000,"reported_tablets":42,"spinlock_wait_cycles":28653824}
I20260709 12:30:07.155821 18255 raft_consensus.cc:493] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:30:07.156571 18255 raft_consensus.cc:515] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:07.160256  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:07.170022 18295 raft_consensus.cc:493] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:07.170647 18295 raft_consensus.cc:515] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:07.174364 18295 leader_election.cc:290] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:07.178402  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:07.206040 18255 leader_election.cc:290] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:07.233408  4356 raft_consensus.cc:2272] T 3ca6496163ec40978a2dfd744edddb18 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:07.250119  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 77019.7us
I20260709 12:30:07.254956  4356 tablet_replica.cc:333] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:07.261845  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ccec0cc7269743d6ad581695ea601d1f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:30:07.263484  5526 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 58360.9us
I20260709 12:30:07.265893  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f938b4d80d34f89a34d90b3c31fa84a" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 40 candidate_status { last_received { term: 39 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:07.276546  5526 leader_election.cc:304] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:07.277969 18295 raft_consensus.cc:2749] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:30:07.290279  5602 consensus_peers.cc:597] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:07.256127  4356 raft_consensus.cc:2243] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:30:07.360193  5524 leader_election.cc:304] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:07.372123  4356 raft_consensus.cc:2272] T f4f86de9f88949219896e4c020485a11 P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:07.446169 18295 raft_consensus.cc:2749] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:30:07.451889  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 91833.1us
I20260709 12:30:07.468608  4356 tablet_replica.cc:333] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:07.470901 18295 raft_consensus.cc:493] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:07.472649 18295 raft_consensus.cc:515] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:07.475512 18295 leader_election.cc:290] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:07.477485  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:07.483116  4356 raft_consensus.cc:2243] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:07.527518  4356 raft_consensus.cc:2272] T 90c64867bae6493c846d941815aa1074 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:07.537002  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0a3a361d19b43b9a0a562f49f9b719e" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 38 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:30:07.562117  5602 consensus_peers.cc:597] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:07.563568  4356 tablet_replica.cc:333] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:07.582430  4356 raft_consensus.cc:2243] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:30:07.594381  4356 raft_consensus.cc:2272] T a3cceeae7fc046fb99b738e957155815 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:07.597920  4356 tablet_replica.cc:333] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:07.599565 18255 raft_consensus.cc:493] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:07.600291 18255 raft_consensus.cc:515] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:07.603124 18255 leader_election.cc:290] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:07.604928  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:07.593454  5524 leader_election.cc:304] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:07.634021 18295 raft_consensus.cc:2749] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:30:07.672214  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14483990837441a99fa83b4a981b4162" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:07.791178  4356 raft_consensus.cc:2243] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:30:07.794337  4356 raft_consensus.cc:2272] T 0b93fd2210654c988a29dff61aabacb6 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:07.824503  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 52011.4us
I20260709 12:30:07.799232  4356 tablet_replica.cc:333] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:08.027916  4356 raft_consensus.cc:2243] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:30:08.055742  5524 leader_election.cc:304] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:08.057333 18256 raft_consensus.cc:2749] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:30:08.068629  5599 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 91116.1us
I20260709 12:30:08.090595  4356 raft_consensus.cc:2272] T 52e5c61368994c6bbeee20e7007995e2 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:08.100495  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 118972us
I20260709 12:30:08.107630 18307 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:08.108268 18307 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:08.132663 18256 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:08.133307 18256 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:08.135639 18256 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:08.159307  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:08.189147  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:08.162181 18307 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:08.191190  5602 consensus_peers.cc:597] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:08.159446  4356 tablet_replica.cc:333] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:08.198822  4356 raft_consensus.cc:2243] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:08.202162  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ff38a877a89473392a3e27657516383" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:08.204586 18300 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:08.205562 18300 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:08.208021 18300 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:08.208832  5602 consensus_peers.cc:597] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:08.210685  4356 raft_consensus.cc:2272] T abd24f2e500a44729c9111fae83c25ed P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:08.211612  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:30:08.214062  6237 consensus_peers.cc:597] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:08.219231  5524 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:08.220147 18255 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:30:08.222791  6237 consensus_peers.cc:597] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:08.263306  4356 tablet_replica.cc:333] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:08.264588  4356 raft_consensus.cc:2243] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:30:08.276963 18297 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:08.277712 18297 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:08.280891  5602 leader_election.cc:336] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:08.281478  5602 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:08.283805 18305 raft_consensus.cc:2749] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:30:08.290877 18297 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:08.293387  4356 raft_consensus.cc:2272] T b3353dcd7bce4f9bbf059a7205c5bf3d P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:08.296442  4356 tablet_replica.cc:333] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:08.297343  4356 raft_consensus.cc:2243] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:30:08.305580  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:08.306278  5636 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 37.
I20260709 12:30:08.312912  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:08.314391 18297 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:30:08.314848 18297 raft_consensus.cc:493] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:08.315241 18297 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:30:08.345326  5446 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 59615.8us
I20260709 12:30:08.382798 18305 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:08.383401 18305 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:08.385707 18305 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:08.387692  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:08.388708  5703 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:30:08.390933  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:08.393702 18035 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:30:08.394079 18035 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:08.394389 18035 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:30:08.397262  4356 raft_consensus.cc:2272] T 607035a460944cc2b065c4d4cc59f813 P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:08.407053 18035 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:08.417017  5602 consensus_peers.cc:597] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:08.423843  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "64af413ffb404a58b5adea5cb742bf01" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:08.424630  5703 raft_consensus.cc:3060] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:30:08.430581 18035 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:08.447548  5602 leader_election.cc:336] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:30:08.447917  6237 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:08.435878  5703 raft_consensus.cc:2468] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
W20260709 12:30:08.450712  5602 leader_election.cc:336] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:08.452090  5599 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:08.453269 18035 raft_consensus.cc:2804] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:30:08.453761 18035 raft_consensus.cc:697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:30:08.454763 18035 consensus_queue.cc:237] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:08.445729  4356 tablet_replica.cc:333] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:08.480892 18297 raft_consensus.cc:515] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:08.482384  5446 catalog_manager.cc:4755] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 26edf338d41e42f1a5ce0d4416ec67e1: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer
I20260709 12:30:08.491671  4356 raft_consensus.cc:2243] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:08.492553  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "458143f1c77141febe7d9725625617c2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:08.493330  5635 raft_consensus.cc:3060] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:30:08.493397  4356 pending_rounds.cc:70] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Trying to abort 1 pending ops.
I20260709 12:30:08.494905  4356 pending_rounds.cc:77] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f: Aborting op as it isn't in flight: id { term: 38 index: 24 } timestamp: 7305626361663135744 op_type: NO_OP noop_request { }
I20260709 12:30:08.495378  4356 raft_consensus.cc:2889] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:30:08.495720  4356 raft_consensus.cc:2272] T 1b585af92ddc4f4fa5bada0f9b198495 P cea66aa15be3423880843974acdbf50f [term 40 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:08.504191  5676 leader_election.cc:336] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:08.504999  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:08.505999 18308 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:30:08.493782 18297 leader_election.cc:290] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:08.513489  5676 leader_election.cc:336] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:08.516059  5635 raft_consensus.cc:2468] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 38.
I20260709 12:30:08.521270  5675 leader_election.cc:304] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:08.522423 18297 raft_consensus.cc:2804] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:30:08.523079 18297 raft_consensus.cc:697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 38 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:30:08.529968 18297 consensus_queue.cc:237] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:08.534410  5451 catalog_manager.cc:5697] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 34 to 35, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 35 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:08.538280 18298 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:08.538904 18298 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:08.539840  5676 leader_election.cc:336] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:08.542968 18298 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:08.545584  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 28 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:08.543692  4356 tablet_replica.cc:333] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f: stopping tablet replica
W20260709 12:30:08.557374  6237 consensus_peers.cc:597] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:08.568688 18297 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:30:08.568915  5602 consensus_peers.cc:597] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:08.569355 18297 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:08.573073  4356 raft_consensus.cc:2243] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:30:08.574342  4356 raft_consensus.cc:2272] T 34a1556938b2414c9e883f1659d8fa0d P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:08.577477  4356 tablet_replica.cc:333] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:08.578436  4356 raft_consensus.cc:2243] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:08.579064  4356 pending_rounds.cc:70] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Trying to abort 1 pending ops.
I20260709 12:30:08.579305  4356 pending_rounds.cc:77] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f: Aborting op as it isn't in flight: id { term: 38 index: 23 } timestamp: 7305626363424497664 op_type: NO_OP noop_request { }
I20260709 12:30:08.579661  4356 raft_consensus.cc:2889] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:30:08.580000  4356 raft_consensus.cc:2272] T ccec0cc7269743d6ad581695ea601d1f P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:08.582948  4356 tablet_replica.cc:333] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
W20260709 12:30:08.583379  6237 leader_election.cc:336] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:08.583914  6237 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:08.588368 18082 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:30:08.588852  6237 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:08.583890  4356 raft_consensus.cc:2243] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:30:08.593326 18297 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:08.594537  4356 raft_consensus.cc:2272] T af208afb558f47dab74acae8d2b8da60 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:08.597666  5676 leader_election.cc:336] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:08.598013  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:08.599665  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:08.601565 18308 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:30:08.597710  4356 tablet_replica.cc:333] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:08.604317  4356 raft_consensus.cc:2243] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:08.605248  4356 pending_rounds.cc:70] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Trying to abort 1 pending ops.
I20260709 12:30:08.605533  4356 pending_rounds.cc:77] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f: Aborting op as it isn't in flight: id { term: 35 index: 24 } timestamp: 7305626373005180928 op_type: NO_OP noop_request { }
I20260709 12:30:08.605980  4356 raft_consensus.cc:2889] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:30:08.606344  4356 raft_consensus.cc:2272] T f0f7b2d2a7e34d629491c96dbfbc7fdc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:08.615916  4356 tablet_replica.cc:333] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:08.617445  4356 raft_consensus.cc:2243] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:08.619552 18297 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:08.620191 18297 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:08.621963  4356 raft_consensus.cc:2272] T e20e319f46504838a20c558a66692da6 P cea66aa15be3423880843974acdbf50f [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:08.622757 18297 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:08.626051  4356 tablet_replica.cc:333] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
W20260709 12:30:08.627139  5676 leader_election.cc:336] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:08.627640  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 28 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:08.628872 18035 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:08.640897  5452 catalog_manager.cc:5697] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 37 to 38, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 38 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:08.644757  6272 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 36.
I20260709 12:30:08.644793 18035 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:08.645735  4356 raft_consensus.cc:2243] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:30:08.647456  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:08.648401  4356 raft_consensus.cc:2272] T ac4fdba265724dea9f440080f8d803e2 P cea66aa15be3423880843974acdbf50f [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:08.648965 18035 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:08.649647 18297 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:30:08.650084 18297 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:08.650465 18297 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:30:08.652132  4356 tablet_replica.cc:333] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
W20260709 12:30:08.655009  5602 leader_election.cc:336] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:08.655206  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:08.655865  5703 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:30:08.658259  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:08.660293 18305 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:30:08.663295 18305 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:08.663712 18305 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:30:08.661067  6237 consensus_peers.cc:597] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:08.668455  4356 raft_consensus.cc:2243] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:08.669220  4356 raft_consensus.cc:2272] T 14483990837441a99fa83b4a981b4162 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:08.672339  4356 tablet_replica.cc:333] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:08.673362  4356 raft_consensus.cc:2243] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:08.674126  4356 raft_consensus.cc:2272] T 733a5b54e4dd4f86999821705033529f P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:08.677026  4356 tablet_replica.cc:333] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:08.680706 18297 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:08.694484 18297 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:08.695264  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f86de9f88949219896e4c020485a11" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 28 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:30:08.696060  6272 raft_consensus.cc:3060] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:30:08.696959  4356 raft_consensus.cc:2243] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 38 LEADER]: Raft consensus shutting down.
W20260709 12:30:08.697968  5676 leader_election.cc:336] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:30:08.708264  6237 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:08.708470  4356 raft_consensus.cc:2272] T df7bbda98d6948f9a851d59f4954b6bb P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:08.712298  4356 tablet_replica.cc:333] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:08.716956  6272 raft_consensus.cc:2468] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 37.
I20260709 12:30:08.723316  5675 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:08.724279 18297 raft_consensus.cc:2804] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:30:08.725086 18297 raft_consensus.cc:697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 37 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:30:08.726267 18297 consensus_queue.cc:237] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 36.28, Last appended by leader: 28, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:08.730883  5676 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 26: this message will repeat every 5th retry.
I20260709 12:30:08.730788 18305 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:08.734206  4356 raft_consensus.cc:2243] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:08.755090  4356 pending_rounds.cc:70] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Trying to abort 2 pending ops.
I20260709 12:30:08.755546  4356 pending_rounds.cc:77] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Aborting op as it isn't in flight: id { term: 33 index: 22 } timestamp: 7305626361638903808 op_type: NO_OP noop_request { }
I20260709 12:30:08.755944  4356 raft_consensus.cc:2889] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:30:08.756289  4356 pending_rounds.cc:77] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f: Aborting op as it isn't in flight: id { term: 32 index: 21 } timestamp: 7305626340130095104 op_type: NO_OP noop_request { }
I20260709 12:30:08.756656  4356 raft_consensus.cc:2889] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:30:08.757018  4356 raft_consensus.cc:2272] T 637a73babedf4b1aa47b3b50937db9ef P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:08.773003  5602 leader_election.cc:336] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:08.834869 18305 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:08.837059  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3cceeae7fc046fb99b738e957155815" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:08.837697  5707 raft_consensus.cc:3060] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:30:08.873871  4356 tablet_replica.cc:333] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:08.874912  4356 raft_consensus.cc:2243] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:08.875656  4356 raft_consensus.cc:2272] T 8b21feec8630479b94ed9384256483fc P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:08.880841  5451 catalog_manager.cc:5697] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 36 to 37, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 37 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:08.883185 18318 raft_consensus.cc:493] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 38736cc545eb4389b220a3bd21d3b466)
I20260709 12:30:08.883921 18318 raft_consensus.cc:515] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:08.886529  4356 tablet_replica.cc:333] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:08.888244  5707 raft_consensus.cc:2468] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:30:08.893217  5703 raft_consensus.cc:1275] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:30:08.900815  4356 raft_consensus.cc:2243] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:30:08.901477 18305 consensus_queue.cc:1048] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:30:08.901885  4356 raft_consensus.cc:2272] T 9db6962eb8f9432289d80decd309ff40 P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:08.905388  5599 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
W20260709 12:30:08.908505  5524 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 91048.5us
I20260709 12:30:08.908597 18307 raft_consensus.cc:2804] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:30:08.923171  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:08.927474 18307 raft_consensus.cc:697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:30:08.928586 18307 consensus_queue.cc:237] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:08.921684  6237 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:08.940505  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acd6f5df0204b2295547e021cd8cb9f" candidate_uuid: "cea66aa15be3423880843974acdbf50f" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:30:08.948411  5676 consensus_peers.cc:597] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:08.950850  5526 leader_election.cc:304] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cea66aa15be3423880843974acdbf50f; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
W20260709 12:30:08.951673  5602 consensus_peers.cc:597] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:08.951290 18318 leader_election.cc:290] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:08.959025 18320 raft_consensus.cc:2749] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:08.987090  4356 tablet_replica.cc:333] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:08.988197  4356 raft_consensus.cc:2243] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:08.988808  4356 raft_consensus.cc:2272] T b0a3a361d19b43b9a0a562f49f9b719e P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:09.001681 18298 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.003749 18298 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:09.005692  5451 catalog_manager.cc:5697] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 33 to 34, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 34 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:09.007099  5629 raft_consensus.cc:1275] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 38 index: 25. (index mismatch)
I20260709 12:30:09.012739  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
W20260709 12:30:09.014369  5676 consensus_peers.cc:597] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:09.015120  5676 consensus_peers.cc:597] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:09.017009  5602 consensus_peers.cc:597] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:09.017674 18297 consensus_queue.cc:1048] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:30:09.022243  4356 tablet_replica.cc:333] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:09.023368  4356 raft_consensus.cc:2243] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Raft consensus shutting down.
W20260709 12:30:09.024391  6237 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.034821  4356 pending_rounds.cc:70] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Trying to abort 1 pending ops.
I20260709 12:30:09.034544  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:09.035226  4356 pending_rounds.cc:77] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f: Aborting op as it isn't in flight: id { term: 34 index: 21 } timestamp: 7305626365941035008 op_type: NO_OP noop_request { }
I20260709 12:30:09.035768  4356 raft_consensus.cc:2889] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:30:09.035929 18082 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:30:09.036257  4356 raft_consensus.cc:2272] T 4d439f3f9c7c40288efc981cec05e7ab P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:09.040199  4356 tablet_replica.cc:333] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:09.041287  4356 raft_consensus.cc:2243] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:30:09.006949 18298 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:09.044406  4356 raft_consensus.cc:2272] T 423d6357e8364cb3b7725b1735b560ac P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:09.055204 18297 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.055996 18297 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:09.064535  5676 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.067088 18297 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:09.067764  4356 tablet_replica.cc:333] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:09.069156  4356 raft_consensus.cc:2243] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:09.069950  4356 pending_rounds.cc:70] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Trying to abort 1 pending ops.
I20260709 12:30:09.070255  4356 pending_rounds.cc:77] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f: Aborting op as it isn't in flight: id { term: 33 index: 25 } timestamp: 7305626380137861120 op_type: NO_OP noop_request { }
I20260709 12:30:09.070717  4356 raft_consensus.cc:2889] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:30:09.071151  4356 raft_consensus.cc:2272] T 35344119b8ff4cdebc4810661ff94f79 P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:09.078514  6237 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:09.082739  4356 tablet_replica.cc:333] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
W20260709 12:30:09.124703  6237 consensus_peers.cc:597] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:09.125685 18319 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.132476 18319 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:09.134913  6237 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:09.135089 18319 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:09.137637  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:09.138350  6272 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
W20260709 12:30:09.138767  6237 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:09.140254  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:09.140519  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:30:09.143079 18248 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:30:09.143508 18248 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.139760  4356 raft_consensus.cc:2243] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:09.143873 18248 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:30:09.144758  4356 raft_consensus.cc:2272] T 52cee1d09af749489de4689a67047497 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:09.151391  6272 raft_consensus.cc:1275] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 36 index: 28. Preceding OpId from leader: term: 37 index: 29. (index mismatch)
I20260709 12:30:09.153020 18297 consensus_queue.cc:1048] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20260709 12:30:09.156409  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:30:09.165799  5602 consensus_peers.cc:597] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:09.170677 18248 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:09.176851 18308 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.177453 18308 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:30:09.179739 18308 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:09.186870  5676 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.176841  4356 tablet_replica.cc:333] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:09.189385  4356 raft_consensus.cc:2243] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:30:09.190717  4356 raft_consensus.cc:2272] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:09.193930  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:09.160480 18297 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:30:09.204644  6237 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:09.204737  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:30:09.206269  5602 consensus_peers.cc:597] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:09.214273  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "52e5c61368994c6bbeee20e7007995e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:30:09.214965  6279 raft_consensus.cc:3060] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:30:09.216058 18308 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.216642  4356 tablet_replica.cc:333] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:09.216611 18308 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:09.217671  4356 raft_consensus.cc:2243] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:09.218288  4356 raft_consensus.cc:2272] T 9ff38a877a89473392a3e27657516383 P cea66aa15be3423880843974acdbf50f [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:09.218876 18308 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:09.220963  5602 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.224011  6279 raft_consensus.cc:2468] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
W20260709 12:30:09.223924  5676 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:30:09.223623  5602 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.226557  5601 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:09.224421  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:09.223766 18300 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:30:09.229131  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:09.227841 18035 raft_consensus.cc:2804] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:30:09.230540 18308 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:30:09.230950 18035 raft_consensus.cc:697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:30:09.231874 18035 consensus_queue.cc:237] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:09.233502 18248 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:09.245792  4356 tablet_replica.cc:333] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:09.247185  4356 raft_consensus.cc:2243] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:09.266829 18319 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.271263 18319 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:30:09.285372  5602 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.291428  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:09.292191  5707 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 38.
W20260709 12:30:09.292933  5676 consensus_peers.cc:597] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:09.294293  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:09.294508 18035 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.295029 18035 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:09.296644 18307 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:30:09.297739 18307 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.298167 18307 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:30:09.284075 18319 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:30:09.303133  5452 catalog_manager.cc:5697] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 34 to 35, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 35 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:09.304024 18307 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:30:09.303514  4356 raft_consensus.cc:2272] T addbcf2919194d2e866b5e8826422851 P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:09.313308  4356 tablet_replica.cc:333] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f: stopping tablet replica
W20260709 12:30:09.321478  5602 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.323377  4356 raft_consensus.cc:2243] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:09.324123  4356 raft_consensus.cc:2272] T 52ccafecca54436198d9d455d076ef8d P cea66aa15be3423880843974acdbf50f [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:09.325593 18035 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:09.328938  5602 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.330135 18307 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 election: Requested vote from peers f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979), cea66aa15be3423880843974acdbf50f (127.4.65.1:43285)
I20260709 12:30:09.344893  4356 tablet_replica.cc:333] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f: stopping tablet replica
W20260709 12:30:09.363854  5602 consensus_peers.cc:597] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:09.371253  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b93fd2210654c988a29dff61aabacb6" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 38 index: 21 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:09.371907  5707 raft_consensus.cc:3060] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:30:09.376123  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:09.376847  5701 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:30:09.384275  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:09.385577 18307 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:30:09.386042 18307 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.386454 18307 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:30:09.388715  4356 raft_consensus.cc:2243] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:30:09.390278  4356 raft_consensus.cc:2272] T 11fc89db1c9243b2af65c665c04ba84e P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:09.393332  5602 consensus_peers.cc:597] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:30:09.402158  4356 tablet_replica.cc:333] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:09.403261  4356 raft_consensus.cc:2243] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:09.406450  4356 raft_consensus.cc:2272] T fa0101c69f7746009c5d77c9616f3360 P cea66aa15be3423880843974acdbf50f [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:09.409605  4356 tablet_replica.cc:333] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:09.420979  4356 raft_consensus.cc:2243] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:09.421937  4356 pending_rounds.cc:70] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Trying to abort 1 pending ops.
I20260709 12:30:09.422248  4356 pending_rounds.cc:77] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f: Aborting op as it isn't in flight: id { term: 39 index: 23 } timestamp: 7305626358043013120 op_type: NO_OP noop_request { }
I20260709 12:30:09.422653  4356 raft_consensus.cc:2889] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:30:09.423063  4356 raft_consensus.cc:2272] T 6f938b4d80d34f89a34d90b3c31fa84a P cea66aa15be3423880843974acdbf50f [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:09.426841  4356 tablet_replica.cc:333] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:09.427863  4356 raft_consensus.cc:2243] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:09.428797  4356 raft_consensus.cc:2272] T e69273530292453d87614627debcc13e P cea66aa15be3423880843974acdbf50f [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:09.431800  4356 tablet_replica.cc:333] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:09.436980  4356 raft_consensus.cc:2243] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:09.439594  4356 raft_consensus.cc:2272] T 2acd6f5df0204b2295547e021cd8cb9f P cea66aa15be3423880843974acdbf50f [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:09.442554  4356 tablet_replica.cc:333] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f: stopping tablet replica
I20260709 12:30:09.448253  4356 raft_consensus.cc:2243] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:09.449100  4356 raft_consensus.cc:2272] T feb9fb12cc5a41e6958575c17961252f P cea66aa15be3423880843974acdbf50f [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:09.443538  5707 raft_consensus.cc:2468] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 39.
I20260709 12:30:09.462633  5599 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:09.466127 18035 raft_consensus.cc:2804] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:30:09.467115 18035 raft_consensus.cc:697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:30:09.467955 18035 consensus_queue.cc:237] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
W20260709 12:30:09.479615  6237 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:09.486070  5701 raft_consensus.cc:1275] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:30:09.488819 18319 consensus_queue.cc:1048] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:30:09.485755 18307 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:09.499992  5602 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.505494 18307 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:09.571254  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3353dcd7bce4f9bbf059a7205c5bf3d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:09.572203  5701 raft_consensus.cc:3060] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:30:09.589476 18330 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.590237 18330 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:09.593989 18330 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:09.598423  5701 raft_consensus.cc:2468] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
W20260709 12:30:09.601016  5676 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:30:09.605028  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 84041.7us
W20260709 12:30:09.612957  6237 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:09.650043  5599 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:09.662791  5452 catalog_manager.cc:5697] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 38 to 39, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 39 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:09.664644  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:09.666739 18337 raft_consensus.cc:2804] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:30:09.667337 18337 raft_consensus.cc:697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:30:09.668464 18337 consensus_queue.cc:237] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:09.675606  5676 consensus_peers.cc:597] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 26: this message will repeat every 5th retry.
I20260709 12:30:09.677932  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:09.681767 18297 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:30:09.688850 18319 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.689447 18319 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:09.691489 18319 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:09.693020  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:09.715210 18338 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.715929 18338 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:09.718019 18338 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:09.720081  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:09.721529 18338 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.722167 18338 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:09.724668 18338 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:09.743556  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:09.744473  5701 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 32.
I20260709 12:30:09.749570  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:09.753739 18337 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:30:09.754294 18337 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.754809 18337 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:30:09.752869  5602 consensus_peers.cc:597] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:09.763324  5602 consensus_peers.cc:597] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:09.775295  5452 catalog_manager.cc:5697] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 32 to 33, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 33 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:09.776258 18337 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:09.798120  5701 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ac4fdba265724dea9f440080f8d803e2" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:09.798810  5701 raft_consensus.cc:3060] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:30:09.776499 18082 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.800622 18082 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:09.803161 18330 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.803699 18330 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:09.804847 18082 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:09.805846 18330 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:09.806603  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:09.807247  5636 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 38.
W20260709 12:30:09.807580  6237 leader_election.cc:336] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:30:09.809783  5676 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.809907  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:09.815402 18340 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:30:09.815853 18340 raft_consensus.cc:493] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.816221 18340 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:30:09.819816  5701 raft_consensus.cc:2468] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 33.
I20260709 12:30:09.821033  5599 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:09.821839 18307 raft_consensus.cc:2804] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:30:09.823240 18307 raft_consensus.cc:697] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:30:09.824811 18307 consensus_queue.cc:237] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:09.794768  5602 leader_election.cc:336] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.834249  5602 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 38736cc545eb4389b220a3bd21d3b466, cea66aa15be3423880843974acdbf50f
I20260709 12:30:09.835083 18338 raft_consensus.cc:2749] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:30:09.836431  5602 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.843084  5602 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:09.827081 18340 raft_consensus.cc:515] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:09.845961 18340 leader_election.cc:290] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:09.851414  6237 leader_election.cc:336] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.846719  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607035a460944cc2b065c4d4cc59f813" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:09.854820  6272 raft_consensus.cc:1275] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:30:09.857744  5636 raft_consensus.cc:3060] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:30:09.843199 18337 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:09.873294  5602 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.873711 18307 raft_consensus.cc:2749] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:30:09.874428 18307 consensus_queue.cc:1048] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:30:09.877580  5635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:09.878201  5635 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 39.
I20260709 12:30:09.882525  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:09.883910 18330 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:30:09.884320 18330 raft_consensus.cc:493] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.885174 18330 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:30:09.890100  5602 consensus_peers.cc:597] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:09.891098  5602 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:09.892622  5452 catalog_manager.cc:5697] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 32 to 33, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 33 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:09.905493  5701 raft_consensus.cc:1275] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:30:09.907418 18337 consensus_queue.cc:1048] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:30:09.918952 18330 raft_consensus.cc:515] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:09.922586  5636 raft_consensus.cc:2468] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 39.
I20260709 12:30:09.924595  6236 leader_election.cc:304] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:09.925391 18082 raft_consensus.cc:2804] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:30:09.925923 18082 raft_consensus.cc:697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 39 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:30:09.926765 18082 consensus_queue.cc:237] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 38.23, Last appended by leader: 23, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:09.928130  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34a1556938b2414c9e883f1659d8fa0d" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 40 candidate_status { last_received { term: 39 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:09.928814  5629 raft_consensus.cc:3060] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:30:09.935668  5629 raft_consensus.cc:2468] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 40.
I20260709 12:30:09.929229 18330 leader_election.cc:290] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:09.936864  5675 leader_election.cc:304] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:30:09.937456  5676 leader_election.cc:336] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:30:09.966446  6237 consensus_peers.cc:597] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:09.967911  5452 catalog_manager.cc:5697] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 38 to 39, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 39 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:09.971192 18307 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.971774 18307 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:09.975112  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
W20260709 12:30:09.977533  5602 consensus_peers.cc:597] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:09.951375 18330 raft_consensus.cc:2804] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:30:09.985118 18336 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:09.985985 18307 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:10.013252  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 60156.8us
W20260709 12:30:10.018200  5602 leader_election.cc:336] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:10.018775  5602 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:10.020001 18338 raft_consensus.cc:2749] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:30:09.995836 18336 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:10.034065 18336 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:10.035907  5602 consensus_peers.cc:597] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:10.051759  5602 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:10.076120  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:10.077028  5707 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 34.
I20260709 12:30:10.068576 18330 raft_consensus.cc:697] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 40 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:30:10.079342 18330 consensus_queue.cc:237] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 39.25, Last appended by leader: 25, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:10.121479  5602 consensus_peers.cc:597] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:10.126192  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:10.127249 18350 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:10.136497 18248 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:30:10.137019 18248 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:10.137409 18248 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:30:10.133668 18350 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:10.143461 18350 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:10.147712  5452 catalog_manager.cc:5697] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 39 to 40, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 40 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:10.149519 18248 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:10.151522 18248 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:10.153988  5703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af208afb558f47dab74acae8d2b8da60" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:10.154623  5703 raft_consensus.cc:3060] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:30:10.159165  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:10.159873  5636 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 38.
I20260709 12:30:10.163335  5703 raft_consensus.cc:2468] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:30:10.162637  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:10.164857 18351 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:30:10.165484 18351 raft_consensus.cc:493] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:10.165910 18351 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:30:10.180490  5602 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:30:10.183629  5676 leader_election.cc:336] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:10.190114 18351 raft_consensus.cc:515] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:10.193886  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df7bbda98d6948f9a851d59f4954b6bb" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:10.194314  5599 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:10.194471  5636 raft_consensus.cc:3060] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:30:10.195530 18336 raft_consensus.cc:2804] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:30:10.196127 18336 raft_consensus.cc:697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:30:10.206044  5636 raft_consensus.cc:2468] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 39.
I20260709 12:30:10.201238 18351 leader_election.cc:290] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:10.207302  5675 leader_election.cc:304] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:10.207239 18336 consensus_queue.cc:237] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:10.219219  5676 leader_election.cc:336] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:10.220165 18330 raft_consensus.cc:2804] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:30:10.220795 18330 raft_consensus.cc:697] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 39 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:30:10.221266 18351 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:10.221787 18351 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:10.221660 18330 consensus_queue.cc:237] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.25, Last appended by leader: 25, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:10.235391  5676 connection.cc:573] client connection to 127.4.65.1:43285 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260709 12:30:10.252449 18351 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:10.253131  6237 connection.cc:769] client connection to 127.4.65.1:43285 send error: Network error: sendmsg error: Connection reset by peer (error 104)
I20260709 12:30:10.287765  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:10.292137  5701 raft_consensus.cc:1275] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:30:10.294859 18248 consensus_queue.cc:1048] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:30:10.310407  5452 catalog_manager.cc:5697] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 38 to 39, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 39 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:10.321607 18340 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:10.327116 18338 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:10.333204 18338 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:10.335837 18338 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:10.322453 18351 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:10.336688 18351 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:10.338753 18351 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:10.340032  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:10.340741  5702 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 35.
I20260709 12:30:10.340914  5636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 38 index: 18 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:10.331547 18340 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:10.351259  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:10.324890  5451 catalog_manager.cc:5697] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 34 to 35, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 35 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:10.361186  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:10.358611 18338 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:30:10.365578 18338 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:10.371269 18338 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:30:10.373739  5602 consensus_peers.cc:597] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:10.363045 18340 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:10.390527 18338 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:10.395278  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "10927a6912ad4d40b0d6c0d9e9cd2a1e" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:10.396025  5702 raft_consensus.cc:3060] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:30:10.417259  5701 raft_consensus.cc:1275] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:30:10.418293 18351 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:10.418809 18351 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:10.419147  5676 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:10.419611  5676 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:10.426662 18319 consensus_queue.cc:1048] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:30:10.430895 18351 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:10.433708  5702 raft_consensus.cc:2468] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 36.
I20260709 12:30:10.437722  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:10.438594  6279 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 33.
I20260709 12:30:10.440174  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:10.441546 18350 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:10.442708 18351 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:30:10.443123 18351 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:10.443542 18351 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:30:10.406220 18338 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:10.454285  5676 leader_election.cc:336] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:10.454877  5676 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:10.455687 18350 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:30:10.468686  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 105360us
W20260709 12:30:10.482856  6237 consensus_peers.cc:597] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:10.485721  6237 consensus_peers.cc:597] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:10.486220  6237 leader_election.cc:336] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:10.474298 18351 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:10.486646  6237 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:10.499085 18360 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:30:10.504606  5676 leader_election.cc:336] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:10.505402  5676 leader_election.cc:336] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:10.513950 18351 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:10.651752  5599 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:10.666476 18337 raft_consensus.cc:2804] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:30:10.667047 18337 raft_consensus.cc:697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:30:10.667994 18337 consensus_queue.cc:237] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:10.739310  5602 consensus_peers.cc:597] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:10.739943  5602 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:10.744604 18359 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:10.745481 18359 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:10.754528 18359 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:10.756119  5629 raft_consensus.cc:1275] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 39 index: 25. Preceding OpId from leader: term: 40 index: 26. (index mismatch)
W20260709 12:30:10.759186  5602 consensus_peers.cc:597] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:10.766067  5602 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:10.767136  5602 consensus_peers.cc:597] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:10.771564  5703 raft_consensus.cc:1275] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:30:10.787468  6279 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9db6962eb8f9432289d80decd309ff40" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:30:10.788152  6279 raft_consensus.cc:3060] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:30:10.789875 18351 consensus_queue.cc:1048] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:30:10.798681  6279 raft_consensus.cc:2468] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 34.
I20260709 12:30:10.812538 18307 consensus_queue.cc:1048] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:30:10.820051  5635 raft_consensus.cc:1275] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 38 index: 25. Preceding OpId from leader: term: 39 index: 26. (index mismatch)
I20260709 12:30:10.823778  5452 catalog_manager.cc:5697] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 35 to 36, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 36 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
W20260709 12:30:10.829244  5602 consensus_peers.cc:597] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:10.834517  5629 raft_consensus.cc:1275] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 38 index: 23. Preceding OpId from leader: term: 39 index: 24. (index mismatch)
I20260709 12:30:10.834322 18350 consensus_queue.cc:1048] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20260709 12:30:10.854861 18339 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election (detected failure of leader cea66aa15be3423880843974acdbf50f)
W20260709 12:30:10.848515  5676 consensus_peers.cc:597] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:10.855489 18339 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:10.857430  5676 consensus_peers.cc:597] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:10.857719 18339 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:10.860425  5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" is_pre_election: true
I20260709 12:30:10.861182  5629 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 35.
W20260709 12:30:10.864692  6237 consensus_peers.cc:597] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:10.869994  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 38 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:10.870713  5704 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 38.
I20260709 12:30:10.873216  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:10.874142 18319 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:30:10.874583 18319 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:10.875027 18319 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:30:10.882099 18340 consensus_queue.cc:1048] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
W20260709 12:30:10.883411  5602 leader_election.cc:336] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:10.885538  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:30:10.888182  5675 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:10.889060 18350 raft_consensus.cc:2804] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:30:10.889614 18350 raft_consensus.cc:697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:30:10.890590 18350 consensus_queue.cc:237] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:10.892796 18366 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:30:10.893253 18366 raft_consensus.cc:493] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting leader election (detected failure of leader cea66aa15be3423880843974acdbf50f)
I20260709 12:30:10.893656 18366 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:30:10.898509 18319 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:10.926887 18319 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:10.934965  6237 leader_election.cc:336] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:10.932408  5704 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d6357e8364cb3b7725b1735b560ac" candidate_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" candidate_term: 39 candidate_status { last_received { term: 38 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:10.935827  5704 raft_consensus.cc:3060] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:30:10.978744  5602 leader_election.cc:336] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:10.997762  6237 consensus_peers.cc:597] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:30:11.002453 18366 raft_consensus.cc:515] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:11.022210 18366 leader_election.cc:290] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 election: Requested vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:11.022399  5628 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11fc89db1c9243b2af65c665c04ba84e" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d"
I20260709 12:30:11.023046  5628 raft_consensus.cc:3060] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:30:11.030198  6237 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:30:11.030346  5628 raft_consensus.cc:2468] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 36.
I20260709 12:30:11.031483  6236 leader_election.cc:304] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d; no voters: 
I20260709 12:30:11.032612 18366 raft_consensus.cc:2804] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:30:11.036063  5704 raft_consensus.cc:2468] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a1c5e6dd8e249bf82d38f90fa37b43d in term 39.
I20260709 12:30:11.036585 18366 raft_consensus.cc:697] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 36 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:30:11.040300 18366 consensus_queue.cc:237] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.23, Last appended by leader: 23, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:11.042716  6237 leader_election.cc:336] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:11.055931  5599 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf; no voters: cea66aa15be3423880843974acdbf50f
I20260709 12:30:11.057397 18370 raft_consensus.cc:2804] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:30:11.058099 18370 raft_consensus.cc:697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 LEADER]: Becoming Leader. State: Replica: 4a1c5e6dd8e249bf82d38f90fa37b43d, State: Running, Role: LEADER
I20260709 12:30:11.057542  5451 catalog_manager.cc:5697] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 33 to 34, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 34 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:11.059270 18370 consensus_queue.cc:237] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 38.18, Last appended by leader: 18, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:11.112275  5456 catalog_manager.cc:5697] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d reported cstate change: term changed from 38 to 39, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2). New cstate: current_term: 39 leader_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:11.133112  5451 catalog_manager.cc:5697] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 35 to 36, leader changed from cea66aa15be3423880843974acdbf50f (127.4.65.1) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 36 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:11.141147  5704 raft_consensus.cc:1275] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:30:11.144682 18339 consensus_queue.cc:1048] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
W20260709 12:30:11.179755  5602 consensus_peers.cc:597] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:11.215034  5602 consensus_peers.cc:597] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:11.217640  5676 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:11.235864  5676 consensus_peers.cc:597] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:11.254266  6237 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:11.297911  5602 consensus_peers.cc:597] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:11.333743  4356 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 5582)
W20260709 12:30:11.346925  5676 consensus_peers.cc:597] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:11.367053  6237 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:11.380652  5676 consensus_peers.cc:597] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:11.395179  5676 consensus_peers.cc:597] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:11.398914  5602 consensus_peers.cc:597] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:11.401225  6237 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:11.431083  6237 consensus_peers.cc:597] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:11.480446  6237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111) [suppressed 402 similar messages]
W20260709 12:30:11.483858  6237 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:30:11.486344  5702 raft_consensus.cc:1275] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Refusing update from remote peer 4a1c5e6dd8e249bf82d38f90fa37b43d: Log matching property violated. Preceding OpId in replica: term: 38 index: 18. Preceding OpId from leader: term: 39 index: 19. (index mismatch)
I20260709 12:30:11.488065 18319 consensus_queue.cc:1048] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20260709 12:30:11.509876  5602 consensus_peers.cc:597] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:11.515413  6279 raft_consensus.cc:1275] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:30:11.516960 18347 consensus_queue.cc:1048] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:30:11.520539  5628 raft_consensus.cc:1275] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
I20260709 12:30:11.522058 18324 consensus_queue.cc:1048] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
W20260709 12:30:11.547760  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800330580 after lost signal to thread 4359
W20260709 12:30:11.549015  5490 debug-util.cc:398] Leaking SignalData structure 0x7b080033a380 after lost signal to thread 5493
W20260709 12:30:11.550372  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800038680 after lost signal to thread 5660
W20260709 12:30:11.551477  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800323560 after lost signal to thread 5734
W20260709 12:30:11.552834  5490 debug-util.cc:398] Leaking SignalData structure 0x7b08000ccfc0 after lost signal to thread 6304
W20260709 12:30:11.581476  6237 consensus_peers.cc:597] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:11.695195  6237 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:11.705673  5602 consensus_peers.cc:597] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:11.709787  6237 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:11.751201  5602 consensus_peers.cc:597] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:11.755340  5602 consensus_peers.cc:597] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:11.768092  6237 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:11.777220  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 78055.2us
W20260709 12:30:11.817255  5602 consensus_peers.cc:597] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:11.860064  5676 consensus_peers.cc:597] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:11.888285  6237 consensus_peers.cc:597] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:11.889021  6237 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:12.006121  5602 consensus_peers.cc:597] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:12.116231  5602 consensus_peers.cc:597] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:12.117611  5676 consensus_peers.cc:597] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:12.305764  6237 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:12.335402  4356 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 5582)
W20260709 12:30:12.429098  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 82490.2us
W20260709 12:30:12.525970  5602 consensus_peers.cc:597] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:12.542941  5602 consensus_peers.cc:597] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:12.543898  5602 consensus_peers.cc:597] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:12.559943  5602 consensus_peers.cc:597] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:12.560889  6237 consensus_peers.cc:597] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:12.646389  5602 consensus_peers.cc:597] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:12.775570  5446 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 46525us
W20260709 12:30:12.779456  5446 catalog_manager.cc:4755] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 26edf338d41e42f1a5ce0d4416ec67e1: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:12.880455  5602 consensus_peers.cc:597] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:12.913504  6237 consensus_peers.cc:597] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:12.936473  5602 consensus_peers.cc:597] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:13.011471  6237 consensus_peers.cc:597] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:13.029414  5602 consensus_peers.cc:597] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:13.037228  5602 consensus_peers.cc:597] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:13.198658  6237 consensus_peers.cc:597] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:13.297784  6237 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:13.335999  5676 consensus_peers.cc:597] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:13.336377  4356 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 5582)
W20260709 12:30:13.399819  6237 consensus_peers.cc:597] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:13.434540  5602 consensus_peers.cc:597] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:13.448184  6237 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:13.480602  5676 consensus_peers.cc:597] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:13.536841  6237 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:13.546303  5602 consensus_peers.cc:597] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:13.641520  5676 consensus_peers.cc:597] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:13.659512  6237 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:13.690172  5602 consensus_peers.cc:597] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:13.747790  6237 consensus_peers.cc:597] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:13.772177  5602 consensus_peers.cc:597] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:13.775393  5676 consensus_peers.cc:597] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:13.987571  5676 consensus_peers.cc:597] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:14.022243  6237 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:14.024305  5676 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:14.073181  6237 consensus_peers.cc:597] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:14.086679  6237 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:14.092398  5602 consensus_peers.cc:597] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:14.096114  5602 consensus_peers.cc:597] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:14.121887  6237 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:14.141727  5676 consensus_peers.cc:597] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:14.163757  5602 consensus_peers.cc:597] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:14.184867  5676 consensus_peers.cc:597] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:14.206627  5602 consensus_peers.cc:597] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:14.285706  6237 consensus_peers.cc:597] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:14.335083  6237 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:14.336853  4356 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 5582)
W20260709 12:30:14.336833  5602 consensus_peers.cc:597] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:14.382083  5602 consensus_peers.cc:597] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:14.395177  6237 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:14.419332  5602 consensus_peers.cc:597] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:14.426843  5676 consensus_peers.cc:597] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:14.485594  6237 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:14.821385  5602 consensus_peers.cc:597] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:14.874751  5602 consensus_peers.cc:597] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:14.924232  5602 consensus_peers.cc:597] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:14.924826  5602 consensus_peers.cc:597] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:15.006232  5602 consensus_peers.cc:597] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:15.194604  5602 consensus_peers.cc:597] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:15.263367  6237 consensus_peers.cc:597] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:15.310369  5602 consensus_peers.cc:597] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:15.340474  4356 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 5582)
W20260709 12:30:15.498940  5602 consensus_peers.cc:597] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:15.499600  5602 consensus_peers.cc:597] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:15.564729  6237 consensus_peers.cc:597] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:15.628400  6237 consensus_peers.cc:597] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:15.663913  5602 consensus_peers.cc:597] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:15.781849  6237 consensus_peers.cc:597] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:15.830374  6237 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:15.830925  6237 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:15.875308  6237 consensus_peers.cc:597] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:15.897886  5676 consensus_peers.cc:597] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:15.899811  5602 consensus_peers.cc:597] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:15.961104  6237 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:16.101109  6237 consensus_peers.cc:597] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:16.122046  5602 consensus_peers.cc:597] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:16.155581  5602 consensus_peers.cc:597] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:16.165272  5676 consensus_peers.cc:597] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:16.233327  5676 consensus_peers.cc:597] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:16.285409  6237 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:16.291615  5676 consensus_peers.cc:597] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:16.341516  4356 thread.cc:527] Waited for 6000ms trying to join with diag-logger (tid 5582)
W20260709 12:30:16.346848  5602 consensus_peers.cc:597] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:16.383814  6237 consensus_peers.cc:597] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:16.410655  5676 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:16.482208  5676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111) [suppressed 474 similar messages]
W20260709 12:30:16.496524  5602 consensus_peers.cc:597] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:16.516685  6237 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:16.579892  6237 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:16.582854  5676 consensus_peers.cc:597] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:16.624848  5676 consensus_peers.cc:597] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:16.634490  5602 consensus_peers.cc:597] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:16.643426  5602 consensus_peers.cc:597] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:16.698160  5602 consensus_peers.cc:597] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:16.728826  6237 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:16.770641  6237 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:16.777357  5602 consensus_peers.cc:597] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:16.804548  5602 consensus_peers.cc:597] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:16.807415  5676 consensus_peers.cc:597] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:16.902240  5602 consensus_peers.cc:597] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:16.920477  5676 consensus_peers.cc:597] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:17.026678  6237 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:17.034873  6237 consensus_peers.cc:597] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:17.048825  6237 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:17.216590  5602 consensus_peers.cc:597] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:17.280417  5602 consensus_peers.cc:597] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:17.342047  4356 thread.cc:527] Waited for 7000ms trying to join with diag-logger (tid 5582)
W20260709 12:30:17.411854  5602 consensus_peers.cc:597] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:17.417135  5601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 51767.9us
W20260709 12:30:17.457631  5602 consensus_peers.cc:597] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:17.578902  5602 consensus_peers.cc:597] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:17.639477  5602 consensus_peers.cc:597] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:17.703748  5602 consensus_peers.cc:597] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:17.835563  5602 consensus_peers.cc:597] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:17.835723  6237 consensus_peers.cc:597] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:17.862907  5602 consensus_peers.cc:597] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:18.025032  6237 consensus_peers.cc:597] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:18.248183  6237 consensus_peers.cc:597] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:18.249634  5602 consensus_peers.cc:597] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:18.284588  6237 consensus_peers.cc:597] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:18.290189  6237 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:18.335083  5602 consensus_peers.cc:597] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:18.342645  4356 thread.cc:527] Waited for 8000ms trying to join with diag-logger (tid 5582)
W20260709 12:30:18.375324  6237 consensus_peers.cc:597] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:18.395129  6237 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:18.459645  5676 consensus_peers.cc:597] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:18.641793  5676 consensus_peers.cc:597] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:18.648788  5602 consensus_peers.cc:597] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:18.665448  5676 consensus_peers.cc:597] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:18.769785  5602 consensus_peers.cc:597] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:18.781440  6237 consensus_peers.cc:597] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:18.821354  5602 consensus_peers.cc:597] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:18.822072  6237 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:18.839208  5602 consensus_peers.cc:597] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:18.865336  5676 consensus_peers.cc:597] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:18.873167  5676 consensus_peers.cc:597] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:18.933944  5676 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:18.935642  5602 consensus_peers.cc:597] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:18.955834  6237 consensus_peers.cc:597] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:19.051221  6237 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:19.051285  5657 debug-util.cc:398] Leaking SignalData structure 0x7b080035d220 after lost signal to thread 4359
W20260709 12:30:19.052804  5657 debug-util.cc:398] Leaking SignalData structure 0x7b08001bafc0 after lost signal to thread 5493
W20260709 12:30:19.054345  5657 debug-util.cc:398] Leaking SignalData structure 0x7b080030e840 after lost signal to thread 5660
W20260709 12:30:19.055619  5657 debug-util.cc:398] Leaking SignalData structure 0x7b080039dac0 after lost signal to thread 5734
W20260709 12:30:19.058013  5657 debug-util.cc:398] Leaking SignalData structure 0x7b0800210660 after lost signal to thread 6304
W20260709 12:30:19.079113  6237 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:19.136247  6237 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:19.144238  5602 consensus_peers.cc:597] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:19.151288  6237 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:19.221282  5676 consensus_peers.cc:597] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:19.242551  6237 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:19.273227  5676 consensus_peers.cc:597] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:19.297677  5602 consensus_peers.cc:597] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:30:19.298369  5602 consensus_peers.cc:597] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:19.343679  4356 thread.cc:527] Waited for 9000ms trying to join with diag-logger (tid 5582)
W20260709 12:30:19.397580  5676 consensus_peers.cc:597] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:19.421816  5602 consensus_peers.cc:597] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:19.559006  6237 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:19.629715  5602 consensus_peers.cc:597] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:19.671790  6237 consensus_peers.cc:597] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:19.805853  6237 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:19.809576  5602 consensus_peers.cc:597] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:19.921423  5602 consensus_peers.cc:597] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:19.977859  5602 consensus_peers.cc:597] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:20.034024  5602 consensus_peers.cc:597] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:20.175660  6237 consensus_peers.cc:597] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:20.183151  5602 consensus_peers.cc:597] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:20.201941  5602 consensus_peers.cc:597] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:20.318843  5602 consensus_peers.cc:597] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:20.447855  5602 consensus_peers.cc:597] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:20.449579  5602 consensus_peers.cc:597] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:20.528823  6237 consensus_peers.cc:597] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:20.658874  6237 consensus_peers.cc:597] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:20.766980  6237 consensus_peers.cc:597] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:20.834589  5602 consensus_peers.cc:597] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:20.835223  6237 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:20.919813  5676 consensus_peers.cc:597] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:21.018024  5602 consensus_peers.cc:597] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:21.019127  5676 consensus_peers.cc:597] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:21.025034  5602 consensus_peers.cc:597] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:21.071039  5446 catalog_manager.cc:4755] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 26edf338d41e42f1a5ce0d4416ec67e1: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:21.086023  6237 consensus_peers.cc:597] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:21.115396  5602 consensus_peers.cc:597] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:21.123538  6237 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:21.170253  5602 consensus_peers.cc:597] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:21.257627  5602 consensus_peers.cc:597] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:21.339212  5676 consensus_peers.cc:597] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:21.373057  5602 consensus_peers.cc:597] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:21.434953  5676 consensus_peers.cc:597] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:30:21.485133  6237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111) [suppressed 471 similar messages]
W20260709 12:30:21.489187  6237 consensus_peers.cc:597] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:21.497658  6237 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:21.537675  5676 consensus_peers.cc:597] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:21.553587  5676 consensus_peers.cc:597] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:21.615345  5602 consensus_peers.cc:597] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:30:21.712291  6237 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:21.713085  6237 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:21.713654  6237 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:21.714182  6237 consensus_peers.cc:597] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:21.737987  5676 consensus_peers.cc:597] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:30:21.753554  6237 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:21.786084  5676 consensus_peers.cc:597] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:21.786722  5602 consensus_peers.cc:597] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:21.909328  5676 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:30:22.050844  5602 consensus_peers.cc:597] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:22.068665  6237 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:22.070319  5602 consensus_peers.cc:597] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:22.072623  6237 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:22.082767  6237 consensus_peers.cc:597] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:22.161150  5602 consensus_peers.cc:597] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:22.174412  5602 consensus_peers.cc:597] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:30:22.281702  4356 tablet_server.cc:196] TabletServer@127.4.65.1:0 shutdown complete.
W20260709 12:30:22.310283  6237 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:22.504573  5602 consensus_peers.cc:597] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:22.591913  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 50598.2us
W20260709 12:30:22.603382  5602 consensus_peers.cc:597] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:22.643457  5602 consensus_peers.cc:597] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:22.663125  5602 consensus_peers.cc:597] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:22.738763  5602 consensus_peers.cc:597] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:22.787159  6237 consensus_peers.cc:597] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:22.830853  5602 consensus_peers.cc:597] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:30:22.854841  4356 tablet_server.cc:179] TabletServer@127.4.65.2:0 shutting down...
W20260709 12:30:22.884750  5675 consensus_peers.cc:597] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:22.889550  5602 consensus_peers.cc:597] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:22.899966  6236 consensus_peers.cc:597] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:22.942375  6236 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:22.957366  5602 consensus_peers.cc:597] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:30:22.996856  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:30:22.999759  4356 tablet_replica.cc:333] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.000944  4356 raft_consensus.cc:2243] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.007946  4356 raft_consensus.cc:2272] T d816d235ce5a41c796c8949c48c0a53c P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.008338  5675 consensus_peers.cc:597] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:23.009203  5675 consensus_peers.cc:597] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:23.011484  4356 tablet_replica.cc:333] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.012507  4356 raft_consensus.cc:2243] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.013826  4356 raft_consensus.cc:2272] T edd23660363f48bf95cee8a70c5a2f26 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.018410  4356 tablet_replica.cc:333] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.019645  4356 raft_consensus.cc:2243] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.021184  4356 raft_consensus.cc:2272] T 8c6474db7a70405694509f6976d850ce P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.024302  4356 tablet_replica.cc:333] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.025805  4356 raft_consensus.cc:2243] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.027323  4356 raft_consensus.cc:2272] T fdcede9bc9104440a7ce1e3a4eb3a86f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.030514  4356 tablet_replica.cc:333] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.031615  4356 raft_consensus.cc:2243] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.032405  4356 raft_consensus.cc:2272] T 458143f1c77141febe7d9725625617c2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.035568  4356 tablet_replica.cc:333] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.036556  4356 raft_consensus.cc:2243] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.037375  4356 raft_consensus.cc:2272] T fa0101c69f7746009c5d77c9616f3360 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.038192  6236 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:23.040469  4356 tablet_replica.cc:333] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.041546  4356 raft_consensus.cc:2243] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 43 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.042891  4356 raft_consensus.cc:2272] T 5d78f6c4aaa14250ba8528ff2fe6af34 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.056124  4356 tablet_replica.cc:333] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.061625  4356 raft_consensus.cc:2243] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.065905  4356 raft_consensus.cc:2272] T 9f7bab74248a4aeda157f7a50fa82233 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.048317  6236 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:23.073859  6236 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:23.077152  4356 tablet_replica.cc:333] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.086302  4356 raft_consensus.cc:2243] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.088788  4356 raft_consensus.cc:2272] T 64af413ffb404a58b5adea5cb742bf01 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.092257  4356 tablet_replica.cc:333] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.097793  4356 raft_consensus.cc:2243] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.115384  4356 raft_consensus.cc:2272] T 785e79f6fbd0463aafaca5498918a142 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.098495  5675 consensus_peers.cc:597] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:23.112788  6236 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:23.118877  4356 tablet_replica.cc:333] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.128075  4356 raft_consensus.cc:2243] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.129181  4356 raft_consensus.cc:2272] T 832269e5980743c48521740b22eefa9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.132894  4356 tablet_replica.cc:333] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.135842  4356 raft_consensus.cc:2243] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.136857  4356 raft_consensus.cc:2272] T 3ca6496163ec40978a2dfd744edddb18 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.143471  5675 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:23.145088  6236 consensus_peers.cc:597] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:23.146684  4356 tablet_replica.cc:333] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.155470  4356 raft_consensus.cc:2243] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.157595  4356 raft_consensus.cc:2272] T 90c64867bae6493c846d941815aa1074 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.173271  4356 tablet_replica.cc:333] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.174331  4356 raft_consensus.cc:2243] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.175832  4356 raft_consensus.cc:2272] T a3cceeae7fc046fb99b738e957155815 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.181470  4356 tablet_replica.cc:333] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.182471  4356 raft_consensus.cc:2243] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.183465  4356 raft_consensus.cc:2272] T e20e319f46504838a20c558a66692da6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 30 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.194334  5675 consensus_peers.cc:597] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:23.195029  4356 tablet_replica.cc:333] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.196096  4356 raft_consensus.cc:2243] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.196895  4356 raft_consensus.cc:2272] T 378854044c0e42f4b0a737cdbdbcc464 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.197321  6236 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:23.200146  4356 tablet_replica.cc:333] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.213161  4356 raft_consensus.cc:2243] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.214613  4356 raft_consensus.cc:2272] T ac4fdba265724dea9f440080f8d803e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.216956  6237 consensus_peers.cc:597] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:30:23.217638  4356 tablet_replica.cc:333] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.218541  4356 raft_consensus.cc:2243] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.220198  4356 raft_consensus.cc:2272] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 32 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.222144  6237 consensus_peers.cc:597] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:30:23.223472  4356 tablet_replica.cc:333] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.248613  4356 raft_consensus.cc:2243] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.250267  4356 raft_consensus.cc:2272] T 0b93fd2210654c988a29dff61aabacb6 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.253720  4356 tablet_replica.cc:333] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.254603  4356 raft_consensus.cc:2243] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.256217  4356 raft_consensus.cc:2272] T 52e5c61368994c6bbeee20e7007995e2 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.259343  4356 tablet_replica.cc:333] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
W20260709 12:30:23.233443  6236 consensus_peers.cc:597] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:23.274540  6236 consensus_peers.cc:597] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:23.275532  4356 raft_consensus.cc:2243] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.278226  4356 raft_consensus.cc:2272] T bf758b480ecf41619471236195bf1838 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.279832  6236 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:23.284674  4356 tablet_replica.cc:333] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.301095  4356 raft_consensus.cc:2243] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.302632  4356 raft_consensus.cc:2272] T abd24f2e500a44729c9111fae83c25ed P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.305863  4356 tablet_replica.cc:333] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
W20260709 12:30:23.296563  6237 consensus_peers.cc:597] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:30:23.315137  4356 raft_consensus.cc:2243] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.316601  4356 raft_consensus.cc:2272] T af208afb558f47dab74acae8d2b8da60 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.319805  4356 tablet_replica.cc:333] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.320676  4356 raft_consensus.cc:2243] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.321362  4356 raft_consensus.cc:2272] T 11fc89db1c9243b2af65c665c04ba84e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.324466  4356 tablet_replica.cc:333] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.325307  4356 raft_consensus.cc:2243] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.326694  4356 raft_consensus.cc:2272] T 52ccafecca54436198d9d455d076ef8d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.328233  6237 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:30:23.330264  4356 tablet_replica.cc:333] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.331169  4356 raft_consensus.cc:2243] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.332218  4356 raft_consensus.cc:2272] T 8b21feec8630479b94ed9384256483fc P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.339740  5676 consensus_peers.cc:597] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:30:23.341650  4356 tablet_replica.cc:333] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.342724  4356 raft_consensus.cc:2243] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.343899  4356 raft_consensus.cc:2272] T 637a73babedf4b1aa47b3b50937db9ef P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.345703  5675 consensus_peers.cc:597] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:23.347682  5675 consensus_peers.cc:597] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:23.348681  5675 consensus_peers.cc:597] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:23.347692  4356 tablet_replica.cc:333] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.382506  4356 raft_consensus.cc:2243] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.385561  4356 raft_consensus.cc:2272] T b3353dcd7bce4f9bbf059a7205c5bf3d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.395007  4356 tablet_replica.cc:333] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
W20260709 12:30:23.421573  5675 consensus_peers.cc:597] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:23.422848  4356 raft_consensus.cc:2243] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.426646  4356 raft_consensus.cc:2272] T 607035a460944cc2b065c4d4cc59f813 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.432284  4356 tablet_replica.cc:333] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.444409  4356 raft_consensus.cc:2243] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.445297  4356 raft_consensus.cc:2272] T 9efccdcce1164ab5b57aa40caea54717 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.442981  5602 consensus_peers.cc:597] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:30:23.454782  4356 tablet_replica.cc:333] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.455802  4356 raft_consensus.cc:2243] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.456820  4356 raft_consensus.cc:2272] T 34a1556938b2414c9e883f1659d8fa0d P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.460177  4356 tablet_replica.cc:333] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.461674  4356 raft_consensus.cc:2243] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.462450  4356 raft_consensus.cc:2272] T addbcf2919194d2e866b5e8826422851 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.466229  4356 tablet_replica.cc:333] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.467509  4356 raft_consensus.cc:2243] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 43 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.469265  4356 raft_consensus.cc:2272] T fa294357a99443e8bd25587ba629d3de P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.473352  4356 tablet_replica.cc:333] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.484071  4356 raft_consensus.cc:2243] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.507336  4356 raft_consensus.cc:2272] T 733a5b54e4dd4f86999821705033529f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.524547  4356 tablet_replica.cc:333] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.529167  4356 raft_consensus.cc:2243] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.529961  4356 raft_consensus.cc:2272] T df7bbda98d6948f9a851d59f4954b6bb P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.556025  4356 tablet_replica.cc:333] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.557134  4356 raft_consensus.cc:2243] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.558521  4356 raft_consensus.cc:2272] T 26edf338d41e42f1a5ce0d4416ec67e1 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.565330  4356 tablet_replica.cc:333] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.566124  4356 raft_consensus.cc:2243] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.567893  4356 raft_consensus.cc:2272] T 9ff38a877a89473392a3e27657516383 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.571135  4356 tablet_replica.cc:333] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.572027  4356 raft_consensus.cc:2243] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.573442  4356 raft_consensus.cc:2272] T 44c764c1f49048f28e1caefabe1cc376 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.576498  4356 tablet_replica.cc:333] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.577313  4356 raft_consensus.cc:2243] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.578680  4356 raft_consensus.cc:2272] T 423d6357e8364cb3b7725b1735b560ac P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.581571  4356 tablet_replica.cc:333] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.582391  4356 raft_consensus.cc:2243] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.583874  4356 raft_consensus.cc:2272] T 52cee1d09af749489de4689a67047497 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.587280  4356 tablet_replica.cc:333] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.588091  4356 raft_consensus.cc:2243] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.594568  4356 raft_consensus.cc:2272] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.605191  4356 tablet_replica.cc:333] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.606099  4356 raft_consensus.cc:2243] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.607465  4356 raft_consensus.cc:2272] T 786a49bad9be400ea9ce584030aad446 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.610316  4356 tablet_replica.cc:333] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.611130  4356 raft_consensus.cc:2243] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.612643  4356 raft_consensus.cc:2272] T feb9fb12cc5a41e6958575c17961252f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.615686  4356 tablet_replica.cc:333] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.616515  4356 raft_consensus.cc:2243] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.617187  4356 raft_consensus.cc:2272] T 1af549c25a724a2f9d4f65cc1c59f22a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.656174  4356 tablet_replica.cc:333] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.656978  4356 raft_consensus.cc:2243] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.657619  4356 raft_consensus.cc:2272] T 6f938b4d80d34f89a34d90b3c31fa84a P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.660925  4356 tablet_replica.cc:333] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.661633  4356 raft_consensus.cc:2243] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:23.662808  4356 raft_consensus.cc:2272] T e69273530292453d87614627debcc13e P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.665766  4356 tablet_replica.cc:333] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.666406  4356 raft_consensus.cc:2243] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:30:23.667908  4356 raft_consensus.cc:2272] T 4ad0bd0e70f549b7bdcfb140817535f7 P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:23.671485  4356 tablet_replica.cc:333] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d: stopping tablet replica
I20260709 12:30:23.674171  4356 raft_consensus.cc:2243] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Raft consensus shutting down.
W20260709 12:30:23.687465  5676 consensus_peers.cc:597] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:23.731848  5676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 44448.7us
I20260709 12:30:23.743767  4356 raft_consensus.cc:2272] T 2acd6f5df0204b2295547e021cd8cb9f P 4a1c5e6dd8e249bf82d38f90fa37b43d [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:23.794104  6237 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:30:23.795480  5676 consensus_peers.cc:597] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:23.816461  6237 consensus_peers.cc:597] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:23.957839  5676 consensus_peers.cc:597] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:30:23.970700  5676 consensus_peers.cc:597] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:23.999486  6237 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:30:24.013408  6237 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:30:24.039294  5676 consensus_peers.cc:597] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:24.104426  6237 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:30:24.107056 18431 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.107729 18431 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.112514 18431 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:24.115985  5675 leader_election.cc:336] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
W20260709 12:30:24.139348  6237 consensus_peers.cc:597] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:30:24.139740  5676 leader_election.cc:336] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:24.140246  5676 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:24.146355 18431 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:30:24.160673 18431 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.161217 18431 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.166230 18431 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:24.168818 18431 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.170439 18431 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:24.172879  5675 leader_election.cc:336] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:24.173007 18431 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:24.173792  5676 leader_election.cc:336] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:24.174237  5676 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:24.176159 18431 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:30:24.184027  5676 leader_election.cc:336] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:24.186928  5675 leader_election.cc:336] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:24.187561  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:24.189093 18431 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:30:24.238698 18431 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.239310 18431 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.241624 18431 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:24.249717  5676 leader_election.cc:336] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:24.250514  5675 leader_election.cc:336] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:24.250941  5675 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:24.251883 18431 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:24.263880 18431 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.266382 18431 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:24.273280  5676 consensus_peers.cc:597] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:30:24.275940 18428 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.276579 18428 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:24.278990  5676 leader_election.cc:336] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:24.280270 18428 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:24.284065  5675 leader_election.cc:336] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
W20260709 12:30:24.283429  6236 leader_election.cc:336] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:24.281442 18431 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:24.286609  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:24.287885  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:24.291941  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:24.293637 18428 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:30:24.296454 18431 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:30:24.315557  5676 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:30:24.332659  6237 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:30:24.340971  6237 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:30:24.349573  5676 consensus_peers.cc:597] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:30:24.353617 18431 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.354311 18431 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.357214 18431 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:24.361981  6237 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:30:24.367010  5675 leader_election.cc:336] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
W20260709 12:30:24.368431  5676 leader_election.cc:336] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:24.368986  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:24.369859 18431 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:30:24.389178  6237 consensus_peers.cc:597] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:30:24.439949 18429 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.440626 18429 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:24.444034 18429 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:24.450826  6237 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:24.454640 18431 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.455457 18431 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:24.474274 18431 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:24.476192 18432 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.497483 18432 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:24.499928 18432 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:24.481933  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 33 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
W20260709 12:30:24.488449  5676 leader_election.cc:336] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:24.506377  5675 leader_election.cc:336] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:24.507301  5675 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:24.508277 18439 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:30:24.510326 18439 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.510957 18439 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.513469  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:24.513906 18439 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:24.514496  6274 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 31.
W20260709 12:30:24.522573  5676 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:24.534240  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:24.537683 18441 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:30:24.538568 18441 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.538918 18441 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:30:24.544883  5675 leader_election.cc:336] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:24.554963 18441 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:24.564126 18441 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:24.570083 18441 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.570699 18441 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.573063 18441 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:24.573920 18428 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.574641 18428 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:24.582785  5676 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:24.587087 18428 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:24.597764  6237 leader_election.cc:336] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:24.564047  5675 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:24.606745  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:24.608142 18440 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:30:24.609337  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 45325.2us
W20260709 12:30:24.617832  6236 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:24.618513  6236 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:24.619877 18429 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:24.634476  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c6474db7a70405694509f6976d850ce" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:30:24.635490  6274 raft_consensus.cc:3060] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:30:24.651363  6274 raft_consensus.cc:2468] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 32.
I20260709 12:30:24.655220 18440 raft_consensus.cc:493] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.701864 18440 raft_consensus.cc:515] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.713173 18440 leader_election.cc:290] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:24.716501  5676 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:24.658882  6234 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:45979, user_credentials={real_user=slave}} blocked reactor thread for 80205.9us
I20260709 12:30:24.663621 18441 raft_consensus.cc:493] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:30:24.673893  6237 consensus_peers.cc:597] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:30:24.721385  6237 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:43285, user_credentials={real_user=slave}} blocked reactor thread for 47594.9us
I20260709 12:30:24.676520 18438 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.680755 18439 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.722848 18438 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:24.725327 18438 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:24.688282 18431 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.720283 18441 raft_consensus.cc:515] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.726135 18431 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.727675 18441 leader_election.cc:290] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:24.722900 18439 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:24.728274 18431 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:24.729974 18439 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:24.681802 18432 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.730957 18432 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:30:24.733095 18432 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:24.740744  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "785e79f6fbd0463aafaca5498918a142" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 33 candidate_status { last_received { term: 32 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:24.741541  5702 raft_consensus.cc:2468] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 32.
I20260709 12:30:24.747455 18443 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.748726 18443 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:24.752600 18443 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:24.758303  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:24.770394  5707 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 29.
W20260709 12:30:24.772456  6237 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:30:24.774627 18432 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.775151 18432 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:24.777194 18432 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:30:24.783069  5675 leader_election.cc:336] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
W20260709 12:30:24.783670  5675 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:24.784013  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:24.788944 18444 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:30:24.792302  5676 leader_election.cc:336] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:24.794872  5676 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:24.795298  5676 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:24.796200  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:24.797042 18443 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:30:24.797420 18443 raft_consensus.cc:493] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.797793 18443 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:30:24.809243  6236 leader_election.cc:336] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:24.809777  6236 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:24.813125 18428 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:30:24.816916  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:24.823135  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:24.833331  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:24.845896 18442 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.846519 18442 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 43 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:24.849210 18442 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:24.856776 18443 raft_consensus.cc:515] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:24.858142  6274 raft_consensus.cc:2393] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f613ba929b1046d3b5aed909414a95bf in current term 30: Already voted for candidate 38736cc545eb4389b220a3bd21d3b466 in this term.
I20260709 12:30:24.859091 18443 leader_election.cc:290] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:24.862118  5702 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:24.862815  5702 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 43.
I20260709 12:30:24.863359  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26edf338d41e42f1a5ce0d4416ec67e1" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:24.863935  5709 raft_consensus.cc:3060] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:30:24.881354  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:24.882897 18429 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 43 FOLLOWER]: Leader pre-election won for term 44
I20260709 12:30:24.883606 18429 raft_consensus.cc:493] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 43 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.884040 18429 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 43 FOLLOWER]: Advancing to term 44
I20260709 12:30:24.898600 18428 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.899291 18428 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:24.905092  5709 raft_consensus.cc:2468] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 30.
I20260709 12:30:24.906854 18428 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:24.909024  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:24.925276  6234 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:24.926623 18428 raft_consensus.cc:2804] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:30:24.927407 18428 raft_consensus.cc:697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:30:24.928426 18428 consensus_queue.cc:237] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:24.914345 18444 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.936270 18444 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.939579 18444 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:24.920680 18443 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
W20260709 12:30:24.942298  5676 leader_election.cc:336] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:24.941944 18443 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:24.922739  6236 leader_election.cc:336] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:24.923770 18446 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.945935  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:24.946034 18446 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.950390 18446 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:24.912726 18429 raft_consensus.cc:515] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 44 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:24.930706 18432 raft_consensus.cc:493] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.952008 18432 raft_consensus.cc:515] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.954092 18432 leader_election.cc:290] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:24.931283 18439 raft_consensus.cc:493] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.954779 18439 raft_consensus.cc:515] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.931888 18431 raft_consensus.cc:493] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.956611 18431 raft_consensus.cc:515] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.958535 18431 leader_election.cc:290] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:24.932562 18441 raft_consensus.cc:493] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:24.959280 18441 raft_consensus.cc:515] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:24.961334 18441 leader_election.cc:290] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:24.962566 18439 leader_election.cc:290] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:24.967074  5676 leader_election.cc:336] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:24.969354  5709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa294357a99443e8bd25587ba629d3de" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 44 candidate_status { last_received { term: 43 index: 27 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:24.969951  5709 raft_consensus.cc:3060] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Advancing to term 44
W20260709 12:30:24.970883  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 48168.6us
I20260709 12:30:24.976926  5709 raft_consensus.cc:2468] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 44.
I20260709 12:30:24.980268 18443 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:24.933216  5675 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:24.981554 18439 raft_consensus.cc:2804] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:30:24.982267 18439 raft_consensus.cc:697] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:30:24.983773 18439 consensus_queue.cc:237] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:24.988592  6237 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.4.65.62:38921, user_credentials={real_user=slave}} blocked reactor thread for 53533.9us
W20260709 12:30:24.994822  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 61687.1us
I20260709 12:30:24.998687 18429 leader_election.cc:290] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 44 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:24.999732  5676 leader_election.cc:336] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:25.000385  5676 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:25.000933  5676 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:25.001490  5676 leader_election.cc:336] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:25.004230  6234 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:25.005743 18429 raft_consensus.cc:2804] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 44 FOLLOWER]: Leader election won for term 44
I20260709 12:30:25.018201 18429 raft_consensus.cc:697] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 44 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:30:25.019253 18429 consensus_queue.cc:237] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 43.27, Last appended by leader: 27, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:25.030228  6236 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 54219.1us
I20260709 12:30:25.037549 18439 raft_consensus.cc:493] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:25.038334 18439 raft_consensus.cc:515] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:25.041011 18439 leader_election.cc:290] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:25.044525  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:38587, user_credentials={real_user=slave}} blocked reactor thread for 46085us
I20260709 12:30:25.052162  5456 catalog_manager.cc:5697] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 31 to 32, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 32 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:30:25.054255  5676 leader_election.cc:336] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:25.057555 18428 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:25.068823  5451 catalog_manager.cc:5697] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 29 to 30, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 30 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:25.069677 18428 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:25.073444 18428 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:25.076387  6237 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:30:25.083180  6236 leader_election.cc:336] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
W20260709 12:30:25.084026  6236 leader_election.cc:336] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.088073  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf" is_pre_election: true
I20260709 12:30:25.088958  5707 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 38.
I20260709 12:30:25.093070  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:25.094436 18442 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:30:25.094925 18442 raft_consensus.cc:493] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:25.095379 18442 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:30:25.106024 18442 raft_consensus.cc:515] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:25.122139  6236 leader_election.cc:336] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.122663  6236 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:30:25.124629  5675 leader_election.cc:336] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.125123  5675 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:25.127990 18428 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:30:25.132027  6236 leader_election.cc:336] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.132637  6236 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:25.134671 18439 raft_consensus.cc:2749] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:30:25.135964  5675 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.136435  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:25.137367 18439 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:30:25.141058  5675 leader_election.cc:336] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.143532 18428 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:30:25.144816  6236 leader_election.cc:336] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 44 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.141499  5675 leader_election.cc:304] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:25.148131 18439 raft_consensus.cc:2749] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 44 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:30:25.157019  5707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "44c764c1f49048f28e1caefabe1cc376" candidate_uuid: "38736cc545eb4389b220a3bd21d3b466" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "f613ba929b1046d3b5aed909414a95bf"
I20260709 12:30:25.157672  5707 raft_consensus.cc:3060] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:30:25.167248  5707 raft_consensus.cc:2468] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38736cc545eb4389b220a3bd21d3b466 in term 39.
I20260709 12:30:25.168393  6234 leader_election.cc:304] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:25.169235 18429 raft_consensus.cc:2804] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:30:25.169708 18429 raft_consensus.cc:697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 39 LEADER]: Becoming Leader. State: Replica: 38736cc545eb4389b220a3bd21d3b466, State: Running, Role: LEADER
I20260709 12:30:25.170727 18429 consensus_queue.cc:237] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 38.25, Last appended by leader: 25, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:25.184620 18442 leader_election.cc:290] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:25.228588  6236 leader_election.cc:336] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.233258 18439 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:25.233978 18439 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:25.235901 18439 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:25.239272 18439 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:25.239804 18439 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:25.242216 18439 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:30:25.267205  6237 consensus_peers.cc:597] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:30:25.272337  5675 leader_election.cc:336] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.284384  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:30:25.290565  6236 leader_election.cc:336] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.295995 18441 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:30:25.297262  5675 leader_election.cc:336] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.297683  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:25.298236  5451 catalog_manager.cc:5697] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 43 to 44, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 44 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:30:25.300230 18441 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:30:25.299909  5451 catalog_manager.cc:5697] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 reported cstate change: term changed from 38 to 39, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4). New cstate: current_term: 39 leader_uuid: "38736cc545eb4389b220a3bd21d3b466" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:30:25.302645  5675 leader_election.cc:336] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.303089  5675 leader_election.cc:304] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:25.312392 18441 raft_consensus.cc:2749] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:30:25.313717  5675 leader_election.cc:336] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.314145  5675 leader_election.cc:304] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:25.322069 18441 raft_consensus.cc:2749] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:30:25.331810  5675 leader_election.cc:336] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.332297  5675 leader_election.cc:304] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:30:25.335662  6236 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:25.338470 18441 raft_consensus.cc:2749] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:30:25.339493  5675 leader_election.cc:336] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.339946  5675 leader_election.cc:304] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:25.342873 18441 raft_consensus.cc:2749] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:30:25.350633  5675 leader_election.cc:336] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.351336  5675 leader_election.cc:304] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:25.375528 18441 raft_consensus.cc:2749] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:30:25.378750  5675 leader_election.cc:336] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.379284  5675 leader_election.cc:304] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:25.380412 18441 raft_consensus.cc:2749] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:30:25.381770  5675 leader_election.cc:336] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.382254  5675 leader_election.cc:304] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 38736cc545eb4389b220a3bd21d3b466, 4a1c5e6dd8e249bf82d38f90fa37b43d
I20260709 12:30:25.383028 18441 raft_consensus.cc:2749] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:30:25.385030  5675 leader_election.cc:336] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.385604  5675 leader_election.cc:304] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:25.386457 18441 raft_consensus.cc:2749] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:30:25.387809  5675 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.388374  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:25.389262 18441 raft_consensus.cc:2749] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:30:25.396538  6236 consensus_peers.cc:597] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:25.403591  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:25.404383  6274 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 37.
I20260709 12:30:25.406340  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466" is_pre_election: true
I20260709 12:30:25.406930  6274 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 36.
I20260709 12:30:25.454030  5707 raft_consensus.cc:1275] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 44 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 43 index: 27. Preceding OpId from leader: term: 44 index: 28. (index mismatch)
I20260709 12:30:25.458884 18442 consensus_queue.cc:1048] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20260709 12:30:25.472384  6236 consensus_peers.cc:597] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:25.478127  5709 raft_consensus.cc:1275] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:30:25.480645 18429 consensus_queue.cc:1048] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20260709 12:30:25.533865  6236 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:25.544440  6236 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:25.560624  6274 raft_consensus.cc:1275] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:30:25.607808  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
W20260709 12:30:25.622753  6236 consensus_peers.cc:597] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:25.639042  6236 consensus_peers.cc:597] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:25.641310 18441 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:30:25.641779 18441 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:25.642128 18441 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:30:25.653494  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 45761.5us
W20260709 12:30:25.666553  6236 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:25.669382  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:25.675829 18439 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:30:25.676262 18439 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting leader election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:25.676656 18439 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:30:25.671133 18441 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:25.697288 18441 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
I20260709 12:30:25.685979 18439 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:25.710814  6236 consensus_peers.cc:597] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:25.714314 18439 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 election: Requested vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), 38736cc545eb4389b220a3bd21d3b466 (127.4.65.4:46059)
W20260709 12:30:25.707609  6237 consensus_peers.cc:597] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:30:25.719367 18459 consensus_queue.cc:1048] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:30:25.722833  5675 leader_election.cc:336] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
W20260709 12:30:25.725579  5675 leader_election.cc:336] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
W20260709 12:30:25.741925  6236 consensus_peers.cc:597] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:25.752746  6237 consensus_peers.cc:597] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:30:25.754405  6274 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "786a49bad9be400ea9ce584030aad446" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:30:25.761524  6274 raft_consensus.cc:3060] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:30:25.774361  6274 raft_consensus.cc:2468] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 38.
I20260709 12:30:25.759335  6272 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ad0bd0e70f549b7bdcfb140817535f7" candidate_uuid: "f613ba929b1046d3b5aed909414a95bf" candidate_term: 37 candidate_status { last_received { term: 36 index: 25 } } ignore_live_leader: false dest_uuid: "38736cc545eb4389b220a3bd21d3b466"
I20260709 12:30:25.775272  6272 raft_consensus.cc:3060] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:30:25.760876  5675 consensus_peers.cc:597] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:25.759116  5676 consensus_peers.cc:597] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:25.792076  6236 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:25.798328  6272 raft_consensus.cc:2468] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f613ba929b1046d3b5aed909414a95bf in term 37.
I20260709 12:30:25.800194  5675 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:25.808704  5709 raft_consensus.cc:1275] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Refusing update from remote peer 38736cc545eb4389b220a3bd21d3b466: Log matching property violated. Preceding OpId in replica: term: 38 index: 25. Preceding OpId from leader: term: 39 index: 26. (index mismatch)
I20260709 12:30:25.811293 18459 raft_consensus.cc:2804] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:30:25.812974 18442 consensus_queue.cc:1048] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:30:25.814070 18459 raft_consensus.cc:697] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 38 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:30:25.816149 18459 consensus_queue.cc:237] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:25.825431  6236 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:25.836619  5675 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466, f613ba929b1046d3b5aed909414a95bf; no voters: 
I20260709 12:30:25.845431 18459 raft_consensus.cc:2804] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:30:25.846199 18459 raft_consensus.cc:697] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 37 LEADER]: Becoming Leader. State: Replica: f613ba929b1046d3b5aed909414a95bf, State: Running, Role: LEADER
I20260709 12:30:25.850518 18459 consensus_queue.cc:237] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 36.25, Last appended by leader: 25, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
W20260709 12:30:25.853780  6236 consensus_peers.cc:597] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:25.861929  5451 catalog_manager.cc:5697] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 37 to 38, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 38 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:30:25.894670  6237 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:30:25.903630  5675 consensus_peers.cc:597] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:25.916379  5675 consensus_peers.cc:597] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:25.917348  5675 leader_election.cc:336] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
W20260709 12:30:25.925218  5675 consensus_peers.cc:597] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:25.925938  5675 leader_election.cc:336] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.921250 18459 raft_consensus.cc:493] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:25.927893 18459 raft_consensus.cc:515] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:25.930194 18459 leader_election.cc:290] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:25.942514  5676 leader_election.cc:336] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:25.944018  5675 consensus_peers.cc:597] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:25.945398  6237 consensus_peers.cc:597] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:25.952752  5675 leader_election.cc:336] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:25.953321  5675 leader_election.cc:304] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:25.954109 18441 raft_consensus.cc:2749] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:30:25.998469  5456 catalog_manager.cc:5697] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf reported cstate change: term changed from 36 to 37, leader changed from 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2) to f613ba929b1046d3b5aed909414a95bf (127.4.65.3). New cstate: current_term: 37 leader_uuid: "f613ba929b1046d3b5aed909414a95bf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:30:26.001021  6236 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:26.011286  4356 tablet_server.cc:196] TabletServer@127.4.65.2:0 shutdown complete.
I20260709 12:30:26.056017 18441 raft_consensus.cc:493] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:26.056834 18441 raft_consensus.cc:515] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:26.062657 18441 leader_election.cc:290] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:26.094508  5675 leader_election.cc:336] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
W20260709 12:30:26.099334  5676 leader_election.cc:336] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:26.100096  5676 leader_election.cc:304] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:26.101202 18441 raft_consensus.cc:2749] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:30:26.102464  5676 consensus_peers.cc:597] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:30:26.103303 18441 raft_consensus.cc:493] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:26.103884 18441 raft_consensus.cc:515] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:26.106289 18441 leader_election.cc:290] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:26.126592  5675 consensus_peers.cc:597] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:26.127664  5675 leader_election.cc:336] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
W20260709 12:30:26.140308  5675 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:26.170272 18441 raft_consensus.cc:493] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:26.170964 18441 raft_consensus.cc:515] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
W20260709 12:30:26.173316  5676 leader_election.cc:336] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:26.173869  5676 leader_election.cc:304] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:26.176128 18459 raft_consensus.cc:2749] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:26.173329 18441 leader_election.cc:290] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 44 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:26.186790  5676 leader_election.cc:336] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:26.188647  6237 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:30:26.499886 18459 raft_consensus.cc:493] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:26.500669 18459 raft_consensus.cc:515] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:26.503480 18459 leader_election.cc:290] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:26.567446  5675 consensus_peers.cc:597] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:26.568594  5675 leader_election.cc:336] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 44 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:26.569211  5675 leader_election.cc:304] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 44 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:26.573191 18470 raft_consensus.cc:2749] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Leader pre-election lost for term 44. Reason: could not achieve majority
I20260709 12:30:26.576895 18471 raft_consensus.cc:493] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:26.577590 18471 raft_consensus.cc:515] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:26.579934 18471 leader_election.cc:290] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:26.590502 18470 raft_consensus.cc:493] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:26.593662 18470 raft_consensus.cc:515] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:26.598621 18470 leader_election.cc:290] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:26.611451  6274 raft_consensus.cc:1275] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 38 index: 25. (index mismatch)
I20260709 12:30:26.620998  6274 raft_consensus.cc:1275] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Refusing update from remote peer f613ba929b1046d3b5aed909414a95bf: Log matching property violated. Preceding OpId in replica: term: 36 index: 25. Preceding OpId from leader: term: 37 index: 26. (index mismatch)
W20260709 12:30:26.639750  6237 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111) [suppressed 535 similar messages]
W20260709 12:30:26.646723  6237 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:30:26.655687 18441 raft_consensus.cc:493] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:26.660837 18441 raft_consensus.cc:515] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:26.663530 18441 leader_election.cc:290] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:26.664630  6237 consensus_peers.cc:597] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:30:26.668951  6237 consensus_peers.cc:597] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:30:26.657377  6236 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
W20260709 12:30:26.670993  6237 leader_election.cc:336] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:26.671619  6237 leader_election.cc:304] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:26.672628 18443 raft_consensus.cc:2749] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:30:26.673080  6237 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:30:26.677215 18474 consensus_queue.cc:1048] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:30:26.689004 18474 consensus_queue.cc:1048] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20260709 12:30:26.708971  4356 tablet_server.cc:179] TabletServer@127.4.65.3:0 shutting down...
I20260709 12:30:26.732571 18474 raft_consensus.cc:493] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:26.733403 18474 raft_consensus.cc:515] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:26.735775 18474 leader_election.cc:290] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:26.757117  5676 leader_election.cc:336] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:26.757934  5676 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:26.758881  5676 consensus_peers.cc:597] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:30:26.766803 18474 raft_consensus.cc:493] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:26.767452 18474 raft_consensus.cc:515] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } }
I20260709 12:30:26.770068 18474 leader_election.cc:290] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:26.760507  5676 leader_election.cc:336] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:26.777825  5676 consensus_peers.cc:597] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:26.788465  6234 consensus_peers.cc:597] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:26.789094  6234 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:26.789729  6234 consensus_peers.cc:597] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:26.790318  6234 consensus_peers.cc:597] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:26.790927  6234 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:26.794641  5676 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:26.806447  5676 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:26.810858  6237 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:30:26.824931  5676 consensus_peers.cc:597] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:30:26.824846  5675 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:46059, user_credentials={real_user=slave}} blocked reactor thread for 46420.3us
W20260709 12:30:26.837121  5675 leader_election.cc:336] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:26.837776  5675 leader_election.cc:304] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:26.838641 18441 raft_consensus.cc:2749] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:30:26.839918  5675 consensus_peers.cc:597] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:26.840535  5675 leader_election.cc:336] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:26.840873  5675 leader_election.cc:304] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:26.841552 18441 raft_consensus.cc:2749] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:30:26.841774  5675 consensus_peers.cc:597] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:26.842319  5675 consensus_peers.cc:597] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:26.843129  5675 consensus_peers.cc:597] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:26.845780  5675 leader_election.cc:336] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:26.846292  5675 leader_election.cc:304] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:26.847162 18441 raft_consensus.cc:2749] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:30:26.847433  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 12:30:26.848297  5675 consensus_peers.cc:597] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:26.848627  4356 tablet_replica.cc:333] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.849566  4356 raft_consensus.cc:2243] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:26.850488  4356 raft_consensus.cc:2272] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:26.854532  4356 tablet_replica.cc:333] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.855873  4356 raft_consensus.cc:2243] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:26.857152  4356 raft_consensus.cc:2272] T d816d235ce5a41c796c8949c48c0a53c P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:26.861495  4356 tablet_replica.cc:333] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.864012  4356 raft_consensus.cc:2243] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:26.864826  4356 raft_consensus.cc:2272] T fdcede9bc9104440a7ce1e3a4eb3a86f P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:26.868453  4356 tablet_replica.cc:333] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.878245 18441 raft_consensus.cc:493] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:26.878947 18441 raft_consensus.cc:515] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:26.883253 18441 leader_election.cc:290] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
W20260709 12:30:26.886632  6234 consensus_peers.cc:597] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:26.886960  5676 leader_election.cc:336] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:26.888758  5675 leader_election.cc:336] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:26.889204  5675 leader_election.cc:304] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:30:26.891883  6237 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:30:26.891283 18441 raft_consensus.cc:2749] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:30:26.902426  4356 raft_consensus.cc:2243] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:30:26.903617  4356 raft_consensus.cc:2272] T edd23660363f48bf95cee8a70c5a2f26 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:26.893128  5675 leader_election.cc:336] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:26.906737  5675 leader_election.cc:304] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:26.907521 18441 raft_consensus.cc:2749] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:30:26.910619  4356 tablet_replica.cc:333] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.915617  4356 raft_consensus.cc:2243] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:26.920508  4356 raft_consensus.cc:2272] T 2578c6c739884fad8fa75e16cfa14fec P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:26.921639  6234 consensus_peers.cc:597] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:26.925571  4356 tablet_replica.cc:333] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
W20260709 12:30:26.926537  5675 leader_election.cc:336] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:26.926935  4356 raft_consensus.cc:2243] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:26.927143  5675 leader_election.cc:304] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:26.928172 18441 raft_consensus.cc:2749] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:30:26.929210  4356 raft_consensus.cc:2272] T 5d78f6c4aaa14250ba8528ff2fe6af34 P f613ba929b1046d3b5aed909414a95bf [term 43 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:26.933140  4356 tablet_replica.cc:333] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.934089  4356 raft_consensus.cc:2243] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:30:26.935874  4356 raft_consensus.cc:2272] T 8c6474db7a70405694509f6976d850ce P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:26.938730  4356 tablet_replica.cc:333] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.939635  4356 raft_consensus.cc:2243] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:30:26.940804  4356 raft_consensus.cc:2272] T 458143f1c77141febe7d9725625617c2 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:26.943475  4356 tablet_replica.cc:333] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.944507  4356 raft_consensus.cc:2243] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:26.945597  4356 raft_consensus.cc:2272] T 64af413ffb404a58b5adea5cb742bf01 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:26.948588  4356 tablet_replica.cc:333] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.949431  4356 raft_consensus.cc:2243] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:26.951857  4356 raft_consensus.cc:2272] T 785e79f6fbd0463aafaca5498918a142 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:26.954870  4356 tablet_replica.cc:333] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.955725  4356 raft_consensus.cc:2243] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:30:26.957059  4356 raft_consensus.cc:2272] T 832269e5980743c48521740b22eefa9f P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:26.957331  5676 consensus_peers.cc:597] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:30:26.959935  4356 tablet_replica.cc:333] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.960796  4356 raft_consensus.cc:2243] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:26.961474  4356 raft_consensus.cc:2272] T 0b93fd2210654c988a29dff61aabacb6 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:26.964989  4356 tablet_replica.cc:333] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.966187  4356 raft_consensus.cc:2243] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:30:26.967778 18441 raft_consensus.cc:493] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:26.968045  4356 raft_consensus.cc:2272] T f4f86de9f88949219896e4c020485a11 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:26.968369 18441 raft_consensus.cc:515] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:26.970641 18441 leader_election.cc:290] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:26.971433  4356 tablet_replica.cc:333] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.972761  4356 raft_consensus.cc:2243] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:26.973625  4356 raft_consensus.cc:2272] T 90c64867bae6493c846d941815aa1074 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:26.976459  4356 tablet_replica.cc:333] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
W20260709 12:30:26.976759  5676 leader_election.cc:336] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:26.977267  4356 raft_consensus.cc:2243] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:26.978091  4356 raft_consensus.cc:2272] T a3cceeae7fc046fb99b738e957155815 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:26.981844  4356 tablet_replica.cc:333] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.982792  4356 raft_consensus.cc:2243] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:30:26.983909  4356 raft_consensus.cc:2272] T 6adc8ae91dfe46039ab5bd2bc4c69708 P f613ba929b1046d3b5aed909414a95bf [term 32 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:26.986779  5675 leader_election.cc:336] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:26.987299  5675 leader_election.cc:304] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
I20260709 12:30:26.988119 18441 raft_consensus.cc:2749] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:30:26.989095  4356 tablet_replica.cc:333] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.990075  4356 raft_consensus.cc:2243] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:26.990792  4356 raft_consensus.cc:2272] T 1b585af92ddc4f4fa5bada0f9b198495 P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:26.995419  4356 tablet_replica.cc:333] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:26.996309  4356 raft_consensus.cc:2243] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 40 LEADER]: Raft consensus shutting down.
W20260709 12:30:26.996378  6234 consensus_peers.cc:597] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:26.998235  4356 raft_consensus.cc:2272] T 34a1556938b2414c9e883f1659d8fa0d P f613ba929b1046d3b5aed909414a95bf [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:26.999105 18441 raft_consensus.cc:493] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:26.999598 18441 raft_consensus.cc:515] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } }
I20260709 12:30:27.000825  4356 tablet_replica.cc:333] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.001722  4356 raft_consensus.cc:2243] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.001772 18441 leader_election.cc:290] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587)
I20260709 12:30:27.002481  4356 raft_consensus.cc:2272] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.006142  4356 tablet_replica.cc:333] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.007071  4356 raft_consensus.cc:2243] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.007802  4356 raft_consensus.cc:2272] T 9efccdcce1164ab5b57aa40caea54717 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.011337  4356 tablet_replica.cc:333] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.012277  4356 raft_consensus.cc:2243] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.013085  4356 raft_consensus.cc:2272] T af208afb558f47dab74acae8d2b8da60 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.015700  4356 tablet_replica.cc:333] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.016659  4356 raft_consensus.cc:2243] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.017454  4356 raft_consensus.cc:2272] T ccec0cc7269743d6ad581695ea601d1f P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.020267  4356 tablet_replica.cc:333] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.021140  4356 raft_consensus.cc:2243] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.022596  4356 raft_consensus.cc:2272] T bf758b480ecf41619471236195bf1838 P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.025898  4356 tablet_replica.cc:333] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.026785  4356 raft_consensus.cc:2243] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.027442  4356 raft_consensus.cc:2272] T abd24f2e500a44729c9111fae83c25ed P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.030282  4356 tablet_replica.cc:333] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.031103  4356 raft_consensus.cc:2243] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:30:27.034215  4356 raft_consensus.cc:2272] T 378854044c0e42f4b0a737cdbdbcc464 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:27.035626  6234 consensus_peers.cc:597] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:27.034205 18443 raft_consensus.cc:493] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4a1c5e6dd8e249bf82d38f90fa37b43d)
I20260709 12:30:27.036708 18443 raft_consensus.cc:515] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } attrs { replace: true } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:27.037775  4356 tablet_replica.cc:333] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.038635  4356 raft_consensus.cc:2243] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.039345  4356 raft_consensus.cc:2272] T ac4fdba265724dea9f440080f8d803e2 P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.042178  4356 tablet_replica.cc:333] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.043118  4356 raft_consensus.cc:2243] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:30:27.044654  4356 raft_consensus.cc:2272] T 637a73babedf4b1aa47b3b50937db9ef P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.048005  4356 tablet_replica.cc:333] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.049230  4356 raft_consensus.cc:2243] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.050148  4356 raft_consensus.cc:2272] T 8b21feec8630479b94ed9384256483fc P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.053093  4356 tablet_replica.cc:333] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.053951  4356 raft_consensus.cc:2243] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:30:27.055145  4356 raft_consensus.cc:2272] T 9db6962eb8f9432289d80decd309ff40 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.057952  4356 tablet_replica.cc:333] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.058822  4356 raft_consensus.cc:2243] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.059880  4356 raft_consensus.cc:2272] T b0a3a361d19b43b9a0a562f49f9b719e P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:27.062817  6237 consensus_peers.cc:597] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:30:27.063683  4356 tablet_replica.cc:333] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.064662  4356 raft_consensus.cc:2243] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.065488  4356 raft_consensus.cc:2272] T 4d439f3f9c7c40288efc981cec05e7ab P f613ba929b1046d3b5aed909414a95bf [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.067543 18443 leader_election.cc:290] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:27.068252  6236 leader_election.cc:336] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:27.068616  4356 tablet_replica.cc:333] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.070039  4356 raft_consensus.cc:2243] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.070955  4356 raft_consensus.cc:2272] T 14483990837441a99fa83b4a981b4162 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.074129  4356 tablet_replica.cc:333] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.075040  4356 raft_consensus.cc:2243] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.075773  4356 raft_consensus.cc:2272] T 733a5b54e4dd4f86999821705033529f P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.078485  4356 tablet_replica.cc:333] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.079443  4356 raft_consensus.cc:2243] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:30:27.080780  4356 raft_consensus.cc:2272] T df7bbda98d6948f9a851d59f4954b6bb P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:27.083271  5676 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:27.083876  4356 tablet_replica.cc:333] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.084929  4356 raft_consensus.cc:2243] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.092164  4356 raft_consensus.cc:2272] T 52ccafecca54436198d9d455d076ef8d P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.095148  4356 tablet_replica.cc:333] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
W20260709 12:30:27.095257  5675 leader_election.cc:336] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:27.095917  5675 leader_election.cc:304] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f613ba929b1046d3b5aed909414a95bf; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:30:27.096673  5675 raft_consensus.cc:2686] T b3353dcd7bce4f9bbf059a7205c5bf3d P f613ba929b1046d3b5aed909414a95bf: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:30:27.096077  4356 raft_consensus.cc:2243] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:30:27.098461  4356 raft_consensus.cc:2272] T addbcf2919194d2e866b5e8826422851 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.102895  4356 tablet_replica.cc:333] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
W20260709 12:30:27.103037  6237 leader_election.cc:336] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:27.105679  4356 raft_consensus.cc:2243] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 44 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.105571  6237 leader_election.cc:304] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, cea66aa15be3423880843974acdbf50f
W20260709 12:30:27.106897  6234 leader_election.cc:336] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:30:27.107307  4356 raft_consensus.cc:2272] T fa294357a99443e8bd25587ba629d3de P f613ba929b1046d3b5aed909414a95bf [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.110131  4356 tablet_replica.cc:333] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.111025  4356 raft_consensus.cc:2243] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.110747 18443 raft_consensus.cc:2749] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:30:27.111819  4356 raft_consensus.cc:2272] T f0f7b2d2a7e34d629491c96dbfbc7fdc P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.114692  4356 tablet_replica.cc:333] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.115471  4356 raft_consensus.cc:2243] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.116015  4356 raft_consensus.cc:2272] T 44c764c1f49048f28e1caefabe1cc376 P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.118871  4356 tablet_replica.cc:333] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.119616  4356 raft_consensus.cc:2243] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.120244  4356 raft_consensus.cc:2272] T 52cee1d09af749489de4689a67047497 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:27.110050  6234 consensus_peers.cc:597] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:27.125243  4356 tablet_replica.cc:333] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.126060  4356 raft_consensus.cc:2243] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.126667  4356 raft_consensus.cc:2272] T 35344119b8ff4cdebc4810661ff94f79 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.129338  4356 tablet_replica.cc:333] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.130262  4356 raft_consensus.cc:2243] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.131146  4356 raft_consensus.cc:2272] T 423d6357e8364cb3b7725b1735b560ac P f613ba929b1046d3b5aed909414a95bf [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.133912  4356 tablet_replica.cc:333] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.135449  4356 raft_consensus.cc:2243] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.136062  4356 raft_consensus.cc:2272] T 26edf338d41e42f1a5ce0d4416ec67e1 P f613ba929b1046d3b5aed909414a95bf [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.138918  4356 tablet_replica.cc:333] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.139603  4356 raft_consensus.cc:2243] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.140292  4356 raft_consensus.cc:2272] T 9ff38a877a89473392a3e27657516383 P f613ba929b1046d3b5aed909414a95bf [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.143227  4356 tablet_replica.cc:333] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.144289  4356 raft_consensus.cc:2243] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:30:27.145804  4356 raft_consensus.cc:2272] T 4ad0bd0e70f549b7bdcfb140817535f7 P f613ba929b1046d3b5aed909414a95bf [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.150432  4356 tablet_replica.cc:333] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.151302  4356 raft_consensus.cc:2243] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:30:27.152801  4356 raft_consensus.cc:2272] T 786a49bad9be400ea9ce584030aad446 P f613ba929b1046d3b5aed909414a95bf [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.173426  4356 tablet_replica.cc:333] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.174222  4356 raft_consensus.cc:2243] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:30:27.175493  4356 raft_consensus.cc:2272] T 1af549c25a724a2f9d4f65cc1c59f22a P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:27.180496  4356 tablet_replica.cc:333] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.181182  4356 raft_consensus.cc:2243] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.181999  4356 raft_consensus.cc:2272] T feb9fb12cc5a41e6958575c17961252f P f613ba929b1046d3b5aed909414a95bf [term 33 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:27.183985  6237 consensus_peers.cc:597] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:30:27.184686  6237 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:30:27.189258  4356 tablet_replica.cc:333] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf: stopping tablet replica
I20260709 12:30:27.190147  4356 raft_consensus.cc:2243] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:27.190927  4356 raft_consensus.cc:2272] T 10927a6912ad4d40b0d6c0d9e9cd2a1e P f613ba929b1046d3b5aed909414a95bf [term 36 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:27.198455  6237 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:30:27.237938  6234 consensus_peers.cc:597] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:27.240319  6234 consensus_peers.cc:597] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:27.241180  6234 consensus_peers.cc:597] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:27.241796  6234 consensus_peers.cc:597] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 -> Peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Couldn't send request to peer f613ba929b1046d3b5aed909414a95bf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:27.441074  6237 consensus_peers.cc:597] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:30:27.517288  6237 consensus_peers.cc:597] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:30:27.570839  6301 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5900 after lost signal to thread 4359
W20260709 12:30:27.572134  6301 debug-util.cc:398] Leaking SignalData structure 0x7b0800351b20 after lost signal to thread 5493
W20260709 12:30:27.573571  6301 debug-util.cc:398] Leaking SignalData structure 0x7b08002cd160 after lost signal to thread 5734
W20260709 12:30:27.576395  6301 debug-util.cc:398] Leaking SignalData structure 0x7b08003f5c60 after lost signal to thread 6304
W20260709 12:30:27.804863  6236 consensus_peers.cc:597] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:27.882963  6236 consensus_peers.cc:597] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:27.895654  6237 consensus_peers.cc:597] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:30:27.902472  6237 consensus_peers.cc:597] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:30:28.108520 18472 raft_consensus.cc:493] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:30:28.109100 18472 raft_consensus.cc:515] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:28.112619 18472 leader_election.cc:290] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:28.123452  6236 leader_election.cc:336] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
W20260709 12:30:28.135830  6236 consensus_peers.cc:597] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:28.138641  6234 leader_election.cc:336] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Network error: Client connection negotiation failed: client connection to 127.4.65.3:45979: connect: Connection refused (error 111)
I20260709 12:30:28.139245  6234 leader_election.cc:304] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:28.139973 18472 raft_consensus.cc:2749] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:30:28.195940 18472 raft_consensus.cc:493] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:30:28.196523 18472 raft_consensus.cc:515] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:28.198694 18472 leader_election.cc:290] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:28.204427  6237 leader_election.cc:336] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:28.216332  6234 leader_election.cc:336] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Network error: Client connection negotiation failed: client connection to 127.4.65.3:45979: connect: Connection refused (error 111)
I20260709 12:30:28.216837  6234 leader_election.cc:304] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:28.217579 18472 raft_consensus.cc:2749] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:30:28.227677  6236 consensus_peers.cc:597] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:28.242906  6236 consensus_peers.cc:597] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:28.286042  6236 consensus_peers.cc:597] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:28.353288 18472 raft_consensus.cc:493] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:30:28.353832 18472 raft_consensus.cc:515] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:28.355835 18472 leader_election.cc:290] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
I20260709 12:30:28.358892 18472 raft_consensus.cc:493] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:30:28.359378 18472 raft_consensus.cc:515] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:28.361603 18472 leader_election.cc:290] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:28.375698  6236 leader_election.cc:336] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
W20260709 12:30:28.376230  6236 leader_election.cc:336] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
W20260709 12:30:28.378219  6234 leader_election.cc:336] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Network error: Client connection negotiation failed: client connection to 127.4.65.3:45979: connect: Connection refused (error 111)
I20260709 12:30:28.379197  6234 leader_election.cc:304] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
W20260709 12:30:28.380096  6234 leader_election.cc:336] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Network error: Client connection negotiation failed: client connection to 127.4.65.3:45979: connect: Connection refused (error 111)
I20260709 12:30:28.382992  6234 leader_election.cc:304] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:28.380577 18472 raft_consensus.cc:2749] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:30:28.386531 18475 raft_consensus.cc:2749] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:30:28.387918  6236 consensus_peers.cc:597] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:28.416206 18472 raft_consensus.cc:493] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:30:28.416767 18472 raft_consensus.cc:515] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:28.419006 18472 leader_election.cc:290] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:28.423835  6236 leader_election.cc:336] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:28.425185 18472 raft_consensus.cc:493] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:30:28.425917 18472 raft_consensus.cc:515] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cea66aa15be3423880843974acdbf50f" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43285 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:28.451961 18472 leader_election.cc:290] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers cea66aa15be3423880843974acdbf50f (127.4.65.1:43285), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:28.452279  6236 consensus_peers.cc:597] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:28.452806  6236 consensus_peers.cc:597] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:30:28.457967  6234 leader_election.cc:336] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Network error: Client connection negotiation failed: client connection to 127.4.65.3:45979: connect: Connection refused (error 111)
I20260709 12:30:28.458436  6234 leader_election.cc:304] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:28.459120 18472 raft_consensus.cc:2749] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:30:28.467602  6237 leader_election.cc:336] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
W20260709 12:30:28.477051  6234 leader_election.cc:336] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Network error: Client connection negotiation failed: client connection to 127.4.65.3:45979: connect: Connection refused (error 111)
I20260709 12:30:28.477509  6234 leader_election.cc:304] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: cea66aa15be3423880843974acdbf50f, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:28.488251 18484 raft_consensus.cc:2749] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:30:28.492432 18484 raft_consensus.cc:493] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:30:28.493177 18484 raft_consensus.cc:515] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:28.509397  4356 tablet_server.cc:196] TabletServer@127.4.65.3:0 shutdown complete.
W20260709 12:30:28.512405  6236 leader_election.cc:336] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:28.514495 18484 leader_election.cc:290] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:28.522219  6234 leader_election.cc:336] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Network error: Client connection negotiation failed: client connection to 127.4.65.3:45979: connect: Connection refused (error 111)
I20260709 12:30:28.522716  6234 leader_election.cc:304] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:28.526192 18484 raft_consensus.cc:2749] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:30:28.606523  6236 consensus_peers.cc:597] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:28.625299  4356 tablet_server.cc:179] TabletServer@127.4.65.4:0 shutting down...
I20260709 12:30:28.637243 18484 raft_consensus.cc:493] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader f613ba929b1046d3b5aed909414a95bf)
I20260709 12:30:28.639142 18484 raft_consensus.cc:515] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4a1c5e6dd8e249bf82d38f90fa37b43d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38587 } } peers { permanent_uuid: "f613ba929b1046d3b5aed909414a95bf" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 45979 } } peers { permanent_uuid: "38736cc545eb4389b220a3bd21d3b466" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 46059 } attrs { promote: false } }
I20260709 12:30:28.643793 18484 leader_election.cc:290] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587), f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979)
W20260709 12:30:28.652038  6234 leader_election.cc:336] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer f613ba929b1046d3b5aed909414a95bf (127.4.65.3:45979): Network error: Client connection negotiation failed: client connection to 127.4.65.3:45979: connect: Connection refused (error 111)
W20260709 12:30:28.652849  6236 leader_election.cc:336] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111)
I20260709 12:30:28.653456  6236 leader_election.cc:304] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 38736cc545eb4389b220a3bd21d3b466; no voters: 4a1c5e6dd8e249bf82d38f90fa37b43d, f613ba929b1046d3b5aed909414a95bf
I20260709 12:30:28.654306 18484 raft_consensus.cc:2749] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:30:28.654546  6236 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:28.664263  6236 consensus_peers.cc:597] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:30:28.677874  6237 consensus_peers.cc:597] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:30:28.690043  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:30:28.691591  4356 tablet_replica.cc:333] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.693063  4356 raft_consensus.cc:2243] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:28.693970  4356 raft_consensus.cc:2272] T 785e79f6fbd0463aafaca5498918a142 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.697542  4356 tablet_replica.cc:333] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.698617  4356 raft_consensus.cc:2243] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.700031  4356 raft_consensus.cc:2272] T f0f7b2d2a7e34d629491c96dbfbc7fdc P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.703159  4356 tablet_replica.cc:333] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.704330  4356 raft_consensus.cc:2243] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.706328  4356 raft_consensus.cc:2272] T 26edf338d41e42f1a5ce0d4416ec67e1 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.709079  4356 tablet_replica.cc:333] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.709892  4356 raft_consensus.cc:2243] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.710973  4356 raft_consensus.cc:2272] T 2acd6f5df0204b2295547e021cd8cb9f P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.714198  4356 tablet_replica.cc:333] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.715096  4356 raft_consensus.cc:2243] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.716511  4356 raft_consensus.cc:2272] T ccec0cc7269743d6ad581695ea601d1f P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.719447  4356 tablet_replica.cc:333] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.720323  4356 raft_consensus.cc:2243] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:28.721045  4356 raft_consensus.cc:2272] T 1af549c25a724a2f9d4f65cc1c59f22a P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.724403  4356 tablet_replica.cc:333] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.725540  4356 raft_consensus.cc:2243] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:28.726291  4356 raft_consensus.cc:2272] T 6adc8ae91dfe46039ab5bd2bc4c69708 P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:28.730109  6237 consensus_peers.cc:597] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:30:28.733626  4356 tablet_replica.cc:333] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.734649  4356 raft_consensus.cc:2243] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.736057  4356 raft_consensus.cc:2272] T 3ca6496163ec40978a2dfd744edddb18 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:28.736558  6237 consensus_peers.cc:597] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 -> Peer cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): Couldn't send request to peer cea66aa15be3423880843974acdbf50f. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:30:28.739066  4356 tablet_replica.cc:333] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.740404  4356 raft_consensus.cc:2243] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.741549  4356 raft_consensus.cc:2272] T fa0101c69f7746009c5d77c9616f3360 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.744557  4356 tablet_replica.cc:333] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.745448  4356 raft_consensus.cc:2243] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:28.746150  4356 raft_consensus.cc:2272] T 378854044c0e42f4b0a737cdbdbcc464 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.749023  4356 tablet_replica.cc:333] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.749861  4356 raft_consensus.cc:2243] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.751113  4356 raft_consensus.cc:2272] T 35344119b8ff4cdebc4810661ff94f79 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.754498  4356 tablet_replica.cc:333] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.755349  4356 raft_consensus.cc:2243] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.756701  4356 raft_consensus.cc:2272] T 4d439f3f9c7c40288efc981cec05e7ab P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.759531  4356 tablet_replica.cc:333] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.760407  4356 raft_consensus.cc:2243] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 40 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.761721  4356 raft_consensus.cc:2272] T 1b585af92ddc4f4fa5bada0f9b198495 P 38736cc545eb4389b220a3bd21d3b466 [term 40 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:28.762215  6236 consensus_peers.cc:597] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 -> Peer 4a1c5e6dd8e249bf82d38f90fa37b43d (127.4.65.2:38587): Couldn't send request to peer 4a1c5e6dd8e249bf82d38f90fa37b43d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38587: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:30:28.764585  4356 tablet_replica.cc:333] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.765547  4356 raft_consensus.cc:2243] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.766815  4356 raft_consensus.cc:2272] T b0a3a361d19b43b9a0a562f49f9b719e P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.769903  4356 tablet_replica.cc:333] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.771221  4356 raft_consensus.cc:2243] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.772307  4356 raft_consensus.cc:2272] T 9efccdcce1164ab5b57aa40caea54717 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.775588  4356 tablet_replica.cc:333] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.776664  4356 raft_consensus.cc:2243] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:28.777251  4356 raft_consensus.cc:2272] T 8c6474db7a70405694509f6976d850ce P 38736cc545eb4389b220a3bd21d3b466 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.779882  4356 tablet_replica.cc:333] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.780711  4356 raft_consensus.cc:2243] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:28.781363  4356 raft_consensus.cc:2272] T 9db6962eb8f9432289d80decd309ff40 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.784168  4356 tablet_replica.cc:333] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.784981  4356 raft_consensus.cc:2243] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.786137  4356 raft_consensus.cc:2272] T 021ceb67cc2f4876b5dbdf1ca21c5f07 P 38736cc545eb4389b220a3bd21d3b466 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.788885  4356 tablet_replica.cc:333] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.789672  4356 raft_consensus.cc:2243] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.790623  4356 raft_consensus.cc:2272] T 44c764c1f49048f28e1caefabe1cc376 P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.793236  4356 tablet_replica.cc:333] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.793831  4356 raft_consensus.cc:2243] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.794663  4356 raft_consensus.cc:2272] T bf758b480ecf41619471236195bf1838 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.796815  4356 tablet_replica.cc:333] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.797418  4356 raft_consensus.cc:2243] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:28.797967  4356 raft_consensus.cc:2272] T 786a49bad9be400ea9ce584030aad446 P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.800087  4356 tablet_replica.cc:333] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.800714  4356 raft_consensus.cc:2243] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 26 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.801575  4356 raft_consensus.cc:2272] T 9f7bab74248a4aeda157f7a50fa82233 P 38736cc545eb4389b220a3bd21d3b466 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.803694  4356 tablet_replica.cc:333] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.804297  4356 raft_consensus.cc:2243] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.805307  4356 raft_consensus.cc:2272] T e20e319f46504838a20c558a66692da6 P 38736cc545eb4389b220a3bd21d3b466 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.807618  4356 tablet_replica.cc:333] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.808228  4356 raft_consensus.cc:2243] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:28.808807  4356 raft_consensus.cc:2272] T 52e5c61368994c6bbeee20e7007995e2 P 38736cc545eb4389b220a3bd21d3b466 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.811002  4356 tablet_replica.cc:333] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.811537  4356 raft_consensus.cc:2243] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.812490  4356 raft_consensus.cc:2272] T 11fc89db1c9243b2af65c665c04ba84e P 38736cc545eb4389b220a3bd21d3b466 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.814538  4356 tablet_replica.cc:333] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.815097  4356 raft_consensus.cc:2243] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 44 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.816007  4356 raft_consensus.cc:2272] T fa294357a99443e8bd25587ba629d3de P 38736cc545eb4389b220a3bd21d3b466 [term 44 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.817942  4356 tablet_replica.cc:333] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.818487  4356 raft_consensus.cc:2243] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.819360  4356 raft_consensus.cc:2272] T e69273530292453d87614627debcc13e P 38736cc545eb4389b220a3bd21d3b466 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.821444  4356 tablet_replica.cc:333] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.821977  4356 raft_consensus.cc:2243] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:28.822444  4356 raft_consensus.cc:2272] T 4ad0bd0e70f549b7bdcfb140817535f7 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.824790  4356 tablet_replica.cc:333] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.825320  4356 raft_consensus.cc:2243] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.826277  4356 raft_consensus.cc:2272] T 6f938b4d80d34f89a34d90b3c31fa84a P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.828332  4356 tablet_replica.cc:333] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.828907  4356 raft_consensus.cc:2243] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.829903  4356 raft_consensus.cc:2272] T 14483990837441a99fa83b4a981b4162 P 38736cc545eb4389b220a3bd21d3b466 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.831920  4356 tablet_replica.cc:333] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.832499  4356 raft_consensus.cc:2243] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:28.833060  4356 raft_consensus.cc:2272] T f4f86de9f88949219896e4c020485a11 P 38736cc545eb4389b220a3bd21d3b466 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.835150  4356 tablet_replica.cc:333] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.835635  4356 raft_consensus.cc:2243] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.836591  4356 raft_consensus.cc:2272] T 607035a460944cc2b065c4d4cc59f813 P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:28.838492  4356 tablet_replica.cc:333] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466: stopping tablet replica
I20260709 12:30:28.838969  4356 raft_consensus.cc:2243] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:30:28.839882  4356 raft_consensus.cc:2272] T 2578c6c739884fad8fa75e16cfa14fec P 38736cc545eb4389b220a3bd21d3b466 [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:29.185602  5448 maintenance_manager.cc:419] P b3df7d767dc641b7bb01c3435d842dd5: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033349
I20260709 12:30:29.234777  5447 maintenance_manager.cc:643] P b3df7d767dc641b7bb01c3435d842dd5: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.048s	user 0.034s	sys 0.011s Metrics: {"bytes_written":6633,"delete_count":0,"lbm_write_time_us":1420,"lbm_writes_lt_1ms":6,"reinsert_count":0,"thread_start_us":647,"threads_started":1,"update_count":271,"wal-append.queue_time_us":1089}
I20260709 12:30:29.237123  5448 maintenance_manager.cc:419] P b3df7d767dc641b7bb01c3435d842dd5: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.296473
I20260709 12:30:29.335883  5447 maintenance_manager.cc:643] P b3df7d767dc641b7bb01c3435d842dd5: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.098s	user 0.091s	sys 0.007s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":26448,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":2047,"dirs.run_cpu_time_us":560,"dirs.run_wall_time_us":1238,"lbm_read_time_us":2374,"lbm_reads_lt_1ms":27,"lbm_write_time_us":1530,"lbm_writes_lt_1ms":11,"mutex_wait_us":62,"peak_mem_usage":293603,"reinsert_count":0,"thread_start_us":2083,"threads_started":2,"update_count":271}
I20260709 12:30:29.681866  4356 tablet_server.cc:196] TabletServer@127.4.65.4:0 shutdown complete.
I20260709 12:30:29.742805  4356 master.cc:562] Master@127.4.65.62:38921 shutting down...
W20260709 12:30:29.778469  5507 auto_rebalancer.cc:873] Removing replace marker failed: Catalog manager is not initialized. State: Closing
W20260709 12:30:29.778899  5507 auto_rebalancer.cc:901] Could not move replica: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 12:30:29.779171  5507 auto_rebalancer.cc:330] scheduled replica move failed to complete: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 12:30:37.488444  5446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111) [suppressed 276 similar messages]
W20260709 12:30:37.492970  5446 catalog_manager.cc:4755] TS cea66aa15be3423880843974acdbf50f (127.4.65.1:43285): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 26edf338d41e42f1a5ce0d4416ec67e1: Network error: Client connection negotiation failed: client connection to 127.4.65.1:43285: connect: Connection refused (error 111)
I20260709 12:30:38.511274  4356 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:30:38.512651  4356 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:38.513123  4356 tablet_replica.cc:333] T 00000000000000000000000000000000 P b3df7d767dc641b7bb01c3435d842dd5: stopping tablet replica
W20260709 12:30:39.568271  4356 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 5490)
W20260709 12:30:39.569145  5490 debug-util.cc:398] Leaking SignalData structure 0x7b0800404600 after lost signal to thread 4359
I20260709 12:30:39.857497  4356 master.cc:584] Master@127.4.65.62:38921 shutdown complete.
I20260709 12:30:39.915490  4356 test_util.cc:182] -----------------------------------------------
I20260709 12:30:39.915740  4356 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599783304823-4356-0
[  FAILED  ] LeaderRebalancerTest.AddTserver (395261 ms)
[ RUN      ] LeaderRebalancerTest.RestartTserver
I20260709 12:30:39.919816  4356 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.65.62:41379
I20260709 12:30:39.921129  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:30:39.927690 18491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:30:39.927806 18492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:30:39.928378 18494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:30:39.929275  4356 server_base.cc:1061] running on GCE node
I20260709 12:30:39.934319  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:30:39.934563  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:30:39.934795  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600239934741 us; error 0 us; skew 500 ppm
I20260709 12:30:39.935386  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:30:39.938822  4356 webserver.cc:533] Webserver started at http://127.4.65.62:45353/ using document root <none> and password file <none>
I20260709 12:30:39.939582  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:30:39.939805  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:30:39.940152  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:30:39.941797  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/master-0-root/instance:
uuid: "f41933e050fe40c89e214498078505f5"
format_stamp: "Formatted at 2026-07-09 12:30:39 on dist-test-slave-l1c9"
I20260709 12:30:39.947431  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.004s
I20260709 12:30:39.951912 18499 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:30:39.953228  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:30:39.953680  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/master-0-root
uuid: "f41933e050fe40c89e214498078505f5"
format_stamp: "Formatted at 2026-07-09 12:30:39 on dist-test-slave-l1c9"
I20260709 12:30:39.954104  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:30:39.980235  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:30:39.981570  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:30:40.025482  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.62:41379
I20260709 12:30:40.025604 18550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.62:41379 every 8 connection(s)
I20260709 12:30:40.029834 18551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.042814 18551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5: Bootstrap starting.
I20260709 12:30:40.047838 18551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:40.052527 18551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5: No bootstrap required, opened a new log
I20260709 12:30:40.055104 18551 raft_consensus.cc:359] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41933e050fe40c89e214498078505f5" member_type: VOTER }
I20260709 12:30:40.055681 18551 raft_consensus.cc:385] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:40.055945 18551 raft_consensus.cc:740] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41933e050fe40c89e214498078505f5, State: Initialized, Role: FOLLOWER
I20260709 12:30:40.056764 18551 consensus_queue.cc:260] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41933e050fe40c89e214498078505f5" member_type: VOTER }
I20260709 12:30:40.057261 18551 raft_consensus.cc:399] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:30:40.057509 18551 raft_consensus.cc:493] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:30:40.058048 18551 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:40.064692 18551 raft_consensus.cc:515] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41933e050fe40c89e214498078505f5" member_type: VOTER }
I20260709 12:30:40.065448 18551 leader_election.cc:304] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f41933e050fe40c89e214498078505f5; no voters: 
I20260709 12:30:40.066916 18551 leader_election.cc:290] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:30:40.067471 18554 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:40.069325 18554 raft_consensus.cc:697] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [term 1 LEADER]: Becoming Leader. State: Replica: f41933e050fe40c89e214498078505f5, State: Running, Role: LEADER
I20260709 12:30:40.070113 18554 consensus_queue.cc:237] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41933e050fe40c89e214498078505f5" member_type: VOTER }
I20260709 12:30:40.070554 18551 sys_catalog.cc:565] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:30:40.079233 18555 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f41933e050fe40c89e214498078505f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41933e050fe40c89e214498078505f5" member_type: VOTER } }
I20260709 12:30:40.079118 18556 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f41933e050fe40c89e214498078505f5. Latest consensus state: current_term: 1 leader_uuid: "f41933e050fe40c89e214498078505f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41933e050fe40c89e214498078505f5" member_type: VOTER } }
I20260709 12:30:40.080396 18555 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [sys.catalog]: This master's current role is: LEADER
I20260709 12:30:40.080657 18556 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [sys.catalog]: This master's current role is: LEADER
I20260709 12:30:40.082943 18564 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:30:40.092072 18564 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:30:40.093194  4356 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:30:40.102816 18564 catalog_manager.cc:1383] Generated new cluster ID: cb846af5278644088e5164457f479b4c
I20260709 12:30:40.103157 18564 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:30:40.117615 18564 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:30:40.119000 18564 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:30:40.139155 18564 catalog_manager.cc:6070] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5: Generated new TSK 0
I20260709 12:30:40.139933 18564 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:30:40.160718  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:30:40.168092 18572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:30:40.169456 18573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:30:40.171307 18575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:30:40.172279  4356 server_base.cc:1061] running on GCE node
I20260709 12:30:40.175484  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:30:40.175786  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:30:40.175969  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600240175954 us; error 0 us; skew 500 ppm
I20260709 12:30:40.176642  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:30:40.179383  4356 webserver.cc:533] Webserver started at http://127.4.65.1:45235/ using document root <none> and password file <none>
I20260709 12:30:40.179929  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:30:40.180152  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:30:40.180508  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:30:40.181756  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/instance:
uuid: "84812365c65f43f388836c087640adbb"
format_stamp: "Formatted at 2026-07-09 12:30:40 on dist-test-slave-l1c9"
I20260709 12:30:40.186970  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:30:40.191351 18580 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:30:40.192602  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20260709 12:30:40.192934  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-0-root
uuid: "84812365c65f43f388836c087640adbb"
format_stamp: "Formatted at 2026-07-09 12:30:40 on dist-test-slave-l1c9"
I20260709 12:30:40.193272  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:30:40.219558  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:30:40.220904  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:30:40.222556  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:30:40.225162  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:30:40.225411  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:30:40.225657  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:30:40.225857  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:30:40.272915  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.1:36569
I20260709 12:30:40.273003 18642 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.1:36569 every 8 connection(s)
I20260709 12:30:40.278892  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:30:40.288371 18647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:30:40.290258 18648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:30:40.292160 18650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:30:40.292659  4356 server_base.cc:1061] running on GCE node
I20260709 12:30:40.295981  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:30:40.296198  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:30:40.296482  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600240296459 us; error 0 us; skew 500 ppm
I20260709 12:30:40.297083  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:30:40.300185 18643 heartbeater.cc:344] Connected to a master server at 127.4.65.62:41379
I20260709 12:30:40.300184  4356 webserver.cc:533] Webserver started at http://127.4.65.2:39551/ using document root <none> and password file <none>
I20260709 12:30:40.300822 18643 heartbeater.cc:461] Registering TS with master...
I20260709 12:30:40.301128  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:30:40.301410  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:30:40.301692  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:30:40.301942 18643 heartbeater.cc:507] Master 127.4.65.62:41379 requested a full tablet report, sending...
I20260709 12:30:40.303112  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/instance:
uuid: "60d61e0800194dd89be34d3aff3f374d"
format_stamp: "Formatted at 2026-07-09 12:30:40 on dist-test-slave-l1c9"
I20260709 12:30:40.304788 18516 ts_manager.cc:194] Registered new tserver with Master: 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:40.307521 18516 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43910
I20260709 12:30:40.308646  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:30:40.312533 18655 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:30:40.313381  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20260709 12:30:40.313828  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-1-root
uuid: "60d61e0800194dd89be34d3aff3f374d"
format_stamp: "Formatted at 2026-07-09 12:30:40 on dist-test-slave-l1c9"
I20260709 12:30:40.314162  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:30:40.329457  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:30:40.330744  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:30:40.332278  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:30:40.334800  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:30:40.335052  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:30:40.335301  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:30:40.335475  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:30:40.385188  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.2:44143
I20260709 12:30:40.385313 18717 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.2:44143 every 8 connection(s)
I20260709 12:30:40.390822  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:30:40.399986 18722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:30:40.400024 18721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:30:40.403800 18724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:30:40.404342  4356 server_base.cc:1061] running on GCE node
I20260709 12:30:40.405700 18718 heartbeater.cc:344] Connected to a master server at 127.4.65.62:41379
I20260709 12:30:40.405738  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:30:40.406293  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:30:40.406302 18718 heartbeater.cc:461] Registering TS with master...
I20260709 12:30:40.406564  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600240406548 us; error 0 us; skew 500 ppm
I20260709 12:30:40.407275 18718 heartbeater.cc:507] Master 127.4.65.62:41379 requested a full tablet report, sending...
I20260709 12:30:40.407315  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:30:40.410521 18516 ts_manager.cc:194] Registered new tserver with Master: 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:40.411113  4356 webserver.cc:533] Webserver started at http://127.4.65.3:44333/ using document root <none> and password file <none>
I20260709 12:30:40.411804  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:30:40.412050  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:30:40.412397  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:30:40.412506 18516 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43926
I20260709 12:30:40.414073  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-2-root/instance:
uuid: "29f82a8594d741bc92dd24ee4a368e01"
format_stamp: "Formatted at 2026-07-09 12:30:40 on dist-test-slave-l1c9"
I20260709 12:30:40.419529  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:30:40.423046 18729 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:30:40.424103  4356 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:30:40.424575  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-2-root
uuid: "29f82a8594d741bc92dd24ee4a368e01"
format_stamp: "Formatted at 2026-07-09 12:30:40 on dist-test-slave-l1c9"
I20260709 12:30:40.425040  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:30:40.436980  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:30:40.438237  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:30:40.439862  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:30:40.442337  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:30:40.442582  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:30:40.442835  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:30:40.443018  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:30:40.490386  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.3:40335
I20260709 12:30:40.490482 18791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.3:40335 every 8 connection(s)
I20260709 12:30:40.495749  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:30:40.502602 18795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:30:40.503687 18796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:30:40.507642 18792 heartbeater.cc:344] Connected to a master server at 127.4.65.62:41379
I20260709 12:30:40.508040 18792 heartbeater.cc:461] Registering TS with master...
I20260709 12:30:40.509213 18792 heartbeater.cc:507] Master 127.4.65.62:41379 requested a full tablet report, sending...
I20260709 12:30:40.510617  4356 server_base.cc:1061] running on GCE node
I20260709 12:30:40.511493 18516 ts_manager.cc:194] Registered new tserver with Master: 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
W20260709 12:30:40.511621 18798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:30:40.512753  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:30:40.512987  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:30:40.513070 18516 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43928
I20260709 12:30:40.513212  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600240513191 us; error 0 us; skew 500 ppm
I20260709 12:30:40.514041  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:30:40.517194  4356 webserver.cc:533] Webserver started at http://127.4.65.4:35723/ using document root <none> and password file <none>
I20260709 12:30:40.517827  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:30:40.518060  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:30:40.518383  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:30:40.519716  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-3-root/instance:
uuid: "df352ed4600b4ddb83ef57ecd2944fce"
format_stamp: "Formatted at 2026-07-09 12:30:40 on dist-test-slave-l1c9"
I20260709 12:30:40.524952  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:30:40.529495 18803 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:30:40.530504  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:30:40.530854  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-3-root
uuid: "df352ed4600b4ddb83ef57ecd2944fce"
format_stamp: "Formatted at 2026-07-09 12:30:40 on dist-test-slave-l1c9"
I20260709 12:30:40.531167  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:30:40.548563  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:30:40.549841  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:30:40.551379  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:30:40.553895  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:30:40.554323  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:30:40.554694  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:30:40.554944  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:30:40.609074  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.4:44517
I20260709 12:30:40.609167 18865 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.4:44517 every 8 connection(s)
I20260709 12:30:40.625906 18866 heartbeater.cc:344] Connected to a master server at 127.4.65.62:41379
I20260709 12:30:40.626318 18866 heartbeater.cc:461] Registering TS with master...
I20260709 12:30:40.627204 18866 heartbeater.cc:507] Master 127.4.65.62:41379 requested a full tablet report, sending...
I20260709 12:30:40.629103 18516 ts_manager.cc:194] Registered new tserver with Master: df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:40.629454  4356 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.016468391s
I20260709 12:30:40.630492 18516 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43936
I20260709 12:30:40.658874 18516 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43940:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260709 12:30:40.919703 18831 tablet_service.cc:1511] Processing CreateTablet for tablet 14206c16a2154538ad2a781e824a65dd (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:30:40.920409 18683 tablet_service.cc:1511] Processing CreateTablet for tablet 14206c16a2154538ad2a781e824a65dd (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:30:40.921571 18831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14206c16a2154538ad2a781e824a65dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.921260 18682 tablet_service.cc:1511] Processing CreateTablet for tablet d8bf5c53527542a6a9069536baff450f (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:30:40.922056 18683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14206c16a2154538ad2a781e824a65dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.922864 18682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bf5c53527542a6a9069536baff450f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.925829 18830 tablet_service.cc:1511] Processing CreateTablet for tablet 9219e0ff53f348ff98bdc3422e5fb74c (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:30:40.927438 18830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9219e0ff53f348ff98bdc3422e5fb74c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.928000 18681 tablet_service.cc:1511] Processing CreateTablet for tablet 9219e0ff53f348ff98bdc3422e5fb74c (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:30:40.929606 18681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9219e0ff53f348ff98bdc3422e5fb74c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.929553 18756 tablet_service.cc:1511] Processing CreateTablet for tablet d8bf5c53527542a6a9069536baff450f (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:30:40.931111 18756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bf5c53527542a6a9069536baff450f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.933853 18757 tablet_service.cc:1511] Processing CreateTablet for tablet 14206c16a2154538ad2a781e824a65dd (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:30:40.935506 18757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14206c16a2154538ad2a781e824a65dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.950393 18608 tablet_service.cc:1511] Processing CreateTablet for tablet d8bf5c53527542a6a9069536baff450f (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:30:40.951953 18608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8bf5c53527542a6a9069536baff450f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.954035 18829 tablet_service.cc:1511] Processing CreateTablet for tablet d659dc42667d4202a29b529fe3388c8e (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:30:40.955528 18829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d659dc42667d4202a29b529fe3388c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.957149 18607 tablet_service.cc:1511] Processing CreateTablet for tablet 9219e0ff53f348ff98bdc3422e5fb74c (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:30:40.958700 18607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9219e0ff53f348ff98bdc3422e5fb74c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.961150 18606 tablet_service.cc:1511] Processing CreateTablet for tablet d659dc42667d4202a29b529fe3388c8e (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:30:40.963102 18606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d659dc42667d4202a29b529fe3388c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.986938 18681 tablet_service.cc:1511] Processing CreateTablet for tablet f651b7eda3134c7a897855ac4300de2f (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:30:40.988426 18681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f651b7eda3134c7a897855ac4300de2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.990082 18828 tablet_service.cc:1511] Processing CreateTablet for tablet d6e49ecce4f040e99a419db81f1953af (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:30:40.991575 18828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6e49ecce4f040e99a419db81f1953af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.993940 18755 tablet_service.cc:1511] Processing CreateTablet for tablet d659dc42667d4202a29b529fe3388c8e (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:30:40.995561 18755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d659dc42667d4202a29b529fe3388c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:40.997635 18754 tablet_service.cc:1511] Processing CreateTablet for tablet f651b7eda3134c7a897855ac4300de2f (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:30:40.999116 18754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f651b7eda3134c7a897855ac4300de2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.005371 18680 tablet_service.cc:1511] Processing CreateTablet for tablet d6e49ecce4f040e99a419db81f1953af (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:30:41.007426 18680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6e49ecce4f040e99a419db81f1953af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.011936 18605 tablet_service.cc:1511] Processing CreateTablet for tablet f651b7eda3134c7a897855ac4300de2f (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:30:41.014045 18605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f651b7eda3134c7a897855ac4300de2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.024156 18604 tablet_service.cc:1511] Processing CreateTablet for tablet d6e49ecce4f040e99a419db81f1953af (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:30:41.026312 18604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6e49ecce4f040e99a419db81f1953af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.055677 18753 tablet_service.cc:1511] Processing CreateTablet for tablet 2dfef0fa153945568323437021ce2921 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:30:41.057621 18753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dfef0fa153945568323437021ce2921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.059532 18827 tablet_service.cc:1511] Processing CreateTablet for tablet 2dfef0fa153945568323437021ce2921 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:30:41.061437 18827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dfef0fa153945568323437021ce2921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.067992 18887 tablet_bootstrap.cc:492] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:41.071171 18603 tablet_service.cc:1511] Processing CreateTablet for tablet 2dfef0fa153945568323437021ce2921 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:30:41.072763 18603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dfef0fa153945568323437021ce2921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.083341 18886 tablet_bootstrap.cc:492] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:41.084628 18683 tablet_service.cc:1511] Processing CreateTablet for tablet 2acf997d23744fe697db1fe947dca883 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:30:41.086165 18683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2acf997d23744fe697db1fe947dca883. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:30:41.087838 18563 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:30:41.090503 18886 tablet_bootstrap.cc:654] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:41.102206 18888 tablet_bootstrap.cc:492] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:41.099992 18831 tablet_service.cc:1511] Processing CreateTablet for tablet 2acf997d23744fe697db1fe947dca883 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:30:41.109429 18831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2acf997d23744fe697db1fe947dca883. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.111151 18887 tablet_bootstrap.cc:654] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:41.096522 18756 tablet_service.cc:1511] Processing CreateTablet for tablet 2acf997d23744fe697db1fe947dca883 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:30:41.114624 18682 tablet_service.cc:1511] Processing CreateTablet for tablet c7f9d9bb9b8742ac930302b42228aad9 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:30:41.116060 18682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7f9d9bb9b8742ac930302b42228aad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.116160 18602 tablet_service.cc:1511] Processing CreateTablet for tablet c7f9d9bb9b8742ac930302b42228aad9 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:30:41.117694 18602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7f9d9bb9b8742ac930302b42228aad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.119122 18757 tablet_service.cc:1511] Processing CreateTablet for tablet c7f9d9bb9b8742ac930302b42228aad9 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:30:41.114619 18756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2acf997d23744fe697db1fe947dca883. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.126502 18757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7f9d9bb9b8742ac930302b42228aad9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.140137 18830 tablet_service.cc:1511] Processing CreateTablet for tablet 671e989970204ce48b700c86d5faf91d (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:30:41.147591 18830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 671e989970204ce48b700c86d5faf91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.146026 18601 tablet_service.cc:1511] Processing CreateTablet for tablet 671e989970204ce48b700c86d5faf91d (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:30:41.144971 18888 tablet_bootstrap.cc:654] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:41.155946 18601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 671e989970204ce48b700c86d5faf91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.190883 18828 tablet_service.cc:1511] Processing CreateTablet for tablet 73eeb703f09a4330ad726c1d5e42eee3 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:30:41.192634 18828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73eeb703f09a4330ad726c1d5e42eee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.209052 18682 tablet_service.cc:1511] Processing CreateTablet for tablet 671e989970204ce48b700c86d5faf91d (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:30:41.210327 18682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 671e989970204ce48b700c86d5faf91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.213601 18887 tablet_bootstrap.cc:492] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:41.214151 18887 ts_tablet_manager.cc:1403] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.147s	user 0.016s	sys 0.014s
I20260709 12:30:41.217409 18887 raft_consensus.cc:359] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:41.218163 18887 raft_consensus.cc:385] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:41.218466 18887 raft_consensus.cc:740] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:41.219540 18887 consensus_queue.cc:260] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:41.221714 18754 tablet_service.cc:1511] Processing CreateTablet for tablet 73eeb703f09a4330ad726c1d5e42eee3 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:30:41.225517 18888 tablet_bootstrap.cc:492] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:41.225664 18754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73eeb703f09a4330ad726c1d5e42eee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.228012 18604 tablet_service.cc:1511] Processing CreateTablet for tablet 73eeb703f09a4330ad726c1d5e42eee3 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:30:41.229498 18604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73eeb703f09a4330ad726c1d5e42eee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.233898 18888 ts_tablet_manager.cc:1403] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.132s	user 0.018s	sys 0.015s
I20260709 12:30:41.237859 18886 tablet_bootstrap.cc:492] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:41.237232 18680 tablet_service.cc:1511] Processing CreateTablet for tablet 8d163b2d60ca4572bca7a1252c65277a (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:30:41.238453 18886 ts_tablet_manager.cc:1403] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.156s	user 0.010s	sys 0.026s
I20260709 12:30:41.239240 18680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d163b2d60ca4572bca7a1252c65277a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.237274 18888 raft_consensus.cc:359] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:41.242924 18888 raft_consensus.cc:385] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:41.243391 18888 raft_consensus.cc:740] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:41.245097 18888 consensus_queue.cc:260] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:41.252833 18892 tablet_bootstrap.cc:492] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:41.257513 18886 raft_consensus.cc:359] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:41.258137 18866 heartbeater.cc:499] Master 127.4.65.62:41379 was elected leader, sending a full tablet report...
I20260709 12:30:41.258397 18886 raft_consensus.cc:385] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:41.258798 18886 raft_consensus.cc:740] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:41.259583 18886 consensus_queue.cc:260] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:41.273833 18681 tablet_service.cc:1511] Processing CreateTablet for tablet f71d9e90994b4dcfa86f9bb9e3b45b51 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:30:41.275326 18681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71d9e90994b4dcfa86f9bb9e3b45b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.276439 18757 tablet_service.cc:1511] Processing CreateTablet for tablet 8d163b2d60ca4572bca7a1252c65277a (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:30:41.277977 18757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d163b2d60ca4572bca7a1252c65277a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.280670 18887 ts_tablet_manager.cc:1434] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.066s	user 0.011s	sys 0.005s
I20260709 12:30:41.280565 18831 tablet_service.cc:1511] Processing CreateTablet for tablet 8d163b2d60ca4572bca7a1252c65277a (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:30:41.289978 18831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d163b2d60ca4572bca7a1252c65277a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.289870 18753 tablet_service.cc:1511] Processing CreateTablet for tablet f71d9e90994b4dcfa86f9bb9e3b45b51 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:30:41.291342 18753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71d9e90994b4dcfa86f9bb9e3b45b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.284245 18601 tablet_service.cc:1511] Processing CreateTablet for tablet f71d9e90994b4dcfa86f9bb9e3b45b51 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:30:41.293776 18601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71d9e90994b4dcfa86f9bb9e3b45b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.306506 18718 heartbeater.cc:499] Master 127.4.65.62:41379 was elected leader, sending a full tablet report...
I20260709 12:30:41.316660 18643 heartbeater.cc:499] Master 127.4.65.62:41379 was elected leader, sending a full tablet report...
I20260709 12:30:41.322742 18830 tablet_service.cc:1511] Processing CreateTablet for tablet 872ec24b41d24573a9a8cc1d484fd6f9 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:30:41.324220 18830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 872ec24b41d24573a9a8cc1d484fd6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.324991 18887 tablet_bootstrap.cc:492] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:41.304715 18892 tablet_bootstrap.cc:654] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:41.331969 18602 tablet_service.cc:1511] Processing CreateTablet for tablet 872ec24b41d24573a9a8cc1d484fd6f9 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:30:41.333621 18602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 872ec24b41d24573a9a8cc1d484fd6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.339556 18886 ts_tablet_manager.cc:1434] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.101s	user 0.017s	sys 0.033s
I20260709 12:30:41.341200 18886 tablet_bootstrap.cc:492] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:41.358913 18682 tablet_service.cc:1511] Processing CreateTablet for tablet 872ec24b41d24573a9a8cc1d484fd6f9 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:30:41.360448 18682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 872ec24b41d24573a9a8cc1d484fd6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.361897 18828 tablet_service.cc:1511] Processing CreateTablet for tablet 089c667ac1ae4a22b9b2e484f978c5d4 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:30:41.363312 18828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089c667ac1ae4a22b9b2e484f978c5d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:30:41.364434 18867 tablet.cc:2406] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:30:41.365862 18683 tablet_service.cc:1511] Processing CreateTablet for tablet 79ade62c22cf46b3bf97770f5bc7d597 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:30:41.367169 18683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79ade62c22cf46b3bf97770f5bc7d597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.364991 18602 tablet_service.cc:1511] Processing CreateTablet for tablet 089c667ac1ae4a22b9b2e484f978c5d4 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:30:41.368925 18602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089c667ac1ae4a22b9b2e484f978c5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.369657 18827 tablet_service.cc:1511] Processing CreateTablet for tablet 79ade62c22cf46b3bf97770f5bc7d597 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:30:41.370980 18827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79ade62c22cf46b3bf97770f5bc7d597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.361753 18753 tablet_service.cc:1511] Processing CreateTablet for tablet 089c667ac1ae4a22b9b2e484f978c5d4 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:30:41.373395 18887 tablet_bootstrap.cc:654] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:41.386373 18606 tablet_service.cc:1511] Processing CreateTablet for tablet c180f8ddcbe14366a60537e1bad39332 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:30:41.388228 18606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c180f8ddcbe14366a60537e1bad39332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.395382 18754 tablet_service.cc:1511] Processing CreateTablet for tablet 79ade62c22cf46b3bf97770f5bc7d597 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:30:41.396960 18754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79ade62c22cf46b3bf97770f5bc7d597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.398119 18888 ts_tablet_manager.cc:1434] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.164s	user 0.011s	sys 0.015s
I20260709 12:30:41.413755 18888 tablet_bootstrap.cc:492] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:41.417785 18792 heartbeater.cc:499] Master 127.4.65.62:41379 was elected leader, sending a full tablet report...
I20260709 12:30:41.421191 18892 tablet_bootstrap.cc:492] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:41.421774 18892 ts_tablet_manager.cc:1403] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.170s	user 0.016s	sys 0.013s
I20260709 12:30:41.427892 18892 raft_consensus.cc:359] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:41.428910 18892 raft_consensus.cc:385] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:41.429203 18892 raft_consensus.cc:740] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:41.430008 18892 consensus_queue.cc:260] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:41.444547 18753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 089c667ac1ae4a22b9b2e484f978c5d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.448295 18893 raft_consensus.cc:493] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:41.448958 18893 raft_consensus.cc:515] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:41.471041 18886 tablet_bootstrap.cc:654] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:41.476590 18888 tablet_bootstrap.cc:654] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:41.486089 18887 tablet_bootstrap.cc:492] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:41.486965 18887 ts_tablet_manager.cc:1403] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.162s	user 0.040s	sys 0.026s
I20260709 12:30:41.494158 18757 tablet_service.cc:1511] Processing CreateTablet for tablet c180f8ddcbe14366a60537e1bad39332 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:30:41.498111 18757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c180f8ddcbe14366a60537e1bad39332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.501410 18828 tablet_service.cc:1511] Processing CreateTablet for tablet f7b389579627488bb406ec0d112f6938 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:30:41.502671 18892 ts_tablet_manager.cc:1434] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.081s	user 0.010s	sys 0.006s
I20260709 12:30:41.502058 18602 tablet_service.cc:1511] Processing CreateTablet for tablet f7b389579627488bb406ec0d112f6938 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:30:41.510207 18681 tablet_service.cc:1511] Processing CreateTablet for tablet c180f8ddcbe14366a60537e1bad39332 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:30:41.512300 18681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c180f8ddcbe14366a60537e1bad39332. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.512058 18683 tablet_service.cc:1511] Processing CreateTablet for tablet f7b389579627488bb406ec0d112f6938 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:30:41.513593 18828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b389579627488bb406ec0d112f6938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.494460 18893 leader_election.cc:290] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:41.513842 18602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b389579627488bb406ec0d112f6938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.518360 18887 raft_consensus.cc:359] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:41.519141 18887 raft_consensus.cc:385] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:41.519438 18887 raft_consensus.cc:740] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:41.523869 18683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7b389579627488bb406ec0d112f6938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.533182 18895 raft_consensus.cc:493] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:41.540055 18886 tablet_bootstrap.cc:492] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:41.543951 18886 ts_tablet_manager.cc:1403] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.203s	user 0.036s	sys 0.040s
I20260709 12:30:41.520258 18887 consensus_queue.cc:260] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:41.553128 18895 raft_consensus.cc:515] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:41.527367 18756 tablet_service.cc:1511] Processing CreateTablet for tablet 03deb6b823ad408c8cdecc1fbd819940 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:30:41.554529 18830 tablet_service.cc:1511] Processing CreateTablet for tablet 03deb6b823ad408c8cdecc1fbd819940 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:30:41.564723 18680 tablet_service.cc:1511] Processing CreateTablet for tablet 31e16042d49945ea80b31f9eee56f2cf (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:30:41.565876 18755 tablet_service.cc:1511] Processing CreateTablet for tablet 31e16042d49945ea80b31f9eee56f2cf (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:30:41.566056 18606 tablet_service.cc:1511] Processing CreateTablet for tablet 03deb6b823ad408c8cdecc1fbd819940 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:30:41.582559 18893 raft_consensus.cc:493] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:41.582742 18902 raft_consensus.cc:493] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:41.584662 18895 leader_election.cc:290] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:41.588583 18893 raft_consensus.cc:515] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:41.589986 18606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03deb6b823ad408c8cdecc1fbd819940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.588508 18680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31e16042d49945ea80b31f9eee56f2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.584254 18887 ts_tablet_manager.cc:1434] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.097s	user 0.011s	sys 0.000s
I20260709 12:30:41.588508 18830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03deb6b823ad408c8cdecc1fbd819940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.590863 18902 raft_consensus.cc:515] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:41.608127 18894 raft_consensus.cc:493] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:41.621280 18756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03deb6b823ad408c8cdecc1fbd819940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.629302 18604 tablet_service.cc:1511] Processing CreateTablet for tablet 3fb81949f42946cf893026acc49357bc (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:30:41.630228 18831 tablet_service.cc:1511] Processing CreateTablet for tablet 31e16042d49945ea80b31f9eee56f2cf (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:30:41.631382 18604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fb81949f42946cf893026acc49357bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.632263 18831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31e16042d49945ea80b31f9eee56f2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.636185 18892 tablet_bootstrap.cc:492] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:41.642071 18887 tablet_bootstrap.cc:492] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:41.608896 18894 raft_consensus.cc:515] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:41.655007 18894 leader_election.cc:290] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:41.654891 18682 tablet_service.cc:1511] Processing CreateTablet for tablet 3fb81949f42946cf893026acc49357bc (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:30:41.644164 18902 leader_election.cc:290] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:41.656617 18682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fb81949f42946cf893026acc49357bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.657727 18829 tablet_service.cc:1511] Processing CreateTablet for tablet 2ee74ad2fe774e0aa780b1fdf18b3003 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:30:41.659530 18829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ee74ad2fe774e0aa780b1fdf18b3003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.667482 18888 tablet_bootstrap.cc:492] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:41.667797 18892 tablet_bootstrap.cc:654] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:41.669342 18893 leader_election.cc:290] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:41.683533 18755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31e16042d49945ea80b31f9eee56f2cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.688905 18888 ts_tablet_manager.cc:1403] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.275s	user 0.028s	sys 0.036s
I20260709 12:30:41.717274 18887 tablet_bootstrap.cc:654] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:41.726572 18754 tablet_service.cc:1511] Processing CreateTablet for tablet 3fb81949f42946cf893026acc49357bc (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:30:41.728044 18754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fb81949f42946cf893026acc49357bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.728585 18679 tablet_service.cc:1511] Processing CreateTablet for tablet 2ee74ad2fe774e0aa780b1fdf18b3003 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:30:41.730084 18679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ee74ad2fe774e0aa780b1fdf18b3003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.734196 18888 raft_consensus.cc:359] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:41.735031 18888 raft_consensus.cc:385] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:41.735345 18888 raft_consensus.cc:740] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:41.736020 18601 tablet_service.cc:1511] Processing CreateTablet for tablet 2ee74ad2fe774e0aa780b1fdf18b3003 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:30:41.737715 18601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ee74ad2fe774e0aa780b1fdf18b3003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.740993 18886 raft_consensus.cc:359] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:41.741791 18886 raft_consensus.cc:385] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:41.742113 18886 raft_consensus.cc:740] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:41.742906 18886 consensus_queue.cc:260] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:41.762140 18827 tablet_service.cc:1511] Processing CreateTablet for tablet 6f16818c4dfd4e9ab187ffb53d8fc3ce (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:30:41.763588 18827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f16818c4dfd4e9ab187ffb53d8fc3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.773054 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14206c16a2154538ad2a781e824a65dd" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:41.774755 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d659dc42667d4202a29b529fe3388c8e" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:41.777223 18886 ts_tablet_manager.cc:1434] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.196s	user 0.006s	sys 0.003s
I20260709 12:30:41.776949 18757 tablet_service.cc:1511] Processing CreateTablet for tablet 6f16818c4dfd4e9ab187ffb53d8fc3ce (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:30:41.778283 18757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f16818c4dfd4e9ab187ffb53d8fc3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.736133 18888 consensus_queue.cc:260] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
W20260709 12:30:41.781275 18805 leader_election.cc:343] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:41.783706 18888 ts_tablet_manager.cc:1434] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.075s	user 0.007s	sys 0.000s
I20260709 12:30:41.826362 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f651b7eda3134c7a897855ac4300de2f" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:41.828222 18582 leader_election.cc:343] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:41.788548 18767 raft_consensus.cc:2468] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:41.795370 18679 tablet_service.cc:1511] Processing CreateTablet for tablet 9c5cc1ca1ea8468199e1eeddb8c10756 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:30:41.791690 18602 tablet_service.cc:1511] Processing CreateTablet for tablet 6f16818c4dfd4e9ab187ffb53d8fc3ce (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:30:41.831172 18679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c5cc1ca1ea8468199e1eeddb8c10756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.831918 18602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f16818c4dfd4e9ab187ffb53d8fc3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.784818 18888 tablet_bootstrap.cc:492] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:41.841387 18753 tablet_service.cc:1511] Processing CreateTablet for tablet 9c5cc1ca1ea8468199e1eeddb8c10756 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:30:41.842907 18753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c5cc1ca1ea8468199e1eeddb8c10756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.846256 18680 tablet_service.cc:1511] Processing CreateTablet for tablet d4f9d88127114fda91668ab529abd23c (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:30:41.781294 18886 tablet_bootstrap.cc:492] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:41.847667 18680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f9d88127114fda91668ab529abd23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.846462 18828 tablet_service.cc:1511] Processing CreateTablet for tablet 9c5cc1ca1ea8468199e1eeddb8c10756 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:30:41.847936 18605 tablet_service.cc:1511] Processing CreateTablet for tablet d4f9d88127114fda91668ab529abd23c (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:30:41.849202 18605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f9d88127114fda91668ab529abd23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.856535 18805 leader_election.cc:304] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:41.857514 18893 raft_consensus.cc:2804] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:41.857940 18893 raft_consensus.cc:493] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:41.858294 18893 raft_consensus.cc:3060] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:41.849202 18828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c5cc1ca1ea8468199e1eeddb8c10756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.875340 18752 tablet_service.cc:1511] Processing CreateTablet for tablet d4f9d88127114fda91668ab529abd23c (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:30:41.878024 18752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4f9d88127114fda91668ab529abd23c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.918350 18829 tablet_service.cc:1511] Processing CreateTablet for tablet df295fba24764e2ab46a8985e569bad8 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:30:41.920106 18829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df295fba24764e2ab46a8985e569bad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.934231 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bf5c53527542a6a9069536baff450f" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:41.937284 18888 tablet_bootstrap.cc:654] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:41.944550 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14206c16a2154538ad2a781e824a65dd" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:41.946882 18886 tablet_bootstrap.cc:654] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:41.946748 18805 leader_election.cc:343] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:41.936689 18892 tablet_bootstrap.cc:492] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:41.948122 18892 ts_tablet_manager.cc:1403] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.312s	user 0.010s	sys 0.015s
I20260709 12:30:41.951776 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f651b7eda3134c7a897855ac4300de2f" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:41.952926 18892 raft_consensus.cc:359] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
W20260709 12:30:41.953348 18582 leader_election.cc:343] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:41.954598 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d659dc42667d4202a29b529fe3388c8e" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:41.958791 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bf5c53527542a6a9069536baff450f" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:41.969003 18582 leader_election.cc:304] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:41.969590 18767 raft_consensus.cc:2468] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 0.
I20260709 12:30:41.959558 18887 tablet_bootstrap.cc:492] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
W20260709 12:30:41.961163 18730 leader_election.cc:343] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:41.970382 18902 raft_consensus.cc:2749] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:41.974562 18657 leader_election.cc:304] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:41.975827 18887 ts_tablet_manager.cc:1403] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.356s	user 0.018s	sys 0.000s
I20260709 12:30:41.979272 18894 raft_consensus.cc:2804] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:41.981215 18894 raft_consensus.cc:493] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:30:41.984658 18804 leader_election.cc:343] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:41.985325 18804 leader_election.cc:304] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:41.985595 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bf5c53527542a6a9069536baff450f" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:41.984802 18894 raft_consensus.cc:3060] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:41.990718 18606 tablet_service.cc:1511] Processing CreateTablet for tablet df295fba24764e2ab46a8985e569bad8 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:30:41.968104 18892 raft_consensus.cc:385] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:41.995236 18906 raft_consensus.cc:2749] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:41.995651 18887 raft_consensus.cc:359] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:41.996492 18887 raft_consensus.cc:385] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:41.996814 18887 raft_consensus.cc:740] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:41.997886 18606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df295fba24764e2ab46a8985e569bad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:41.997915 18887 consensus_queue.cc:260] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:42.002032 18893 raft_consensus.cc:515] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:42.008244 18683 tablet_service.cc:1511] Processing CreateTablet for tablet df295fba24764e2ab46a8985e569bad8 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:30:42.010874 18887 ts_tablet_manager.cc:1434] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.000s
I20260709 12:30:42.021449 18683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df295fba24764e2ab46a8985e569bad8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.021667 18887 tablet_bootstrap.cc:492] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:41.994509 18892 raft_consensus.cc:740] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:42.023484 18892 consensus_queue.cc:260] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:42.026775 18892 ts_tablet_manager.cc:1434] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.078s	user 0.008s	sys 0.000s
I20260709 12:30:42.028123 18892 tablet_bootstrap.cc:492] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:42.035404 18892 tablet_bootstrap.cc:654] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:42.040215 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bf5c53527542a6a9069536baff450f" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:42.041123 18618 raft_consensus.cc:2468] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 0.
I20260709 12:30:42.047623 18756 tablet_service.cc:1511] Processing CreateTablet for tablet efa9444e110e433aa97b25a05d5769a9 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:30:42.048085 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14206c16a2154538ad2a781e824a65dd" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:42.048802 18767 raft_consensus.cc:3060] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:42.049249 18756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa9444e110e433aa97b25a05d5769a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.058197 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14206c16a2154538ad2a781e824a65dd" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:42.060026 18893 leader_election.cc:290] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:42.059904 18601 tablet_service.cc:1511] Processing CreateTablet for tablet efa9444e110e433aa97b25a05d5769a9 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:30:42.061630 18601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa9444e110e433aa97b25a05d5769a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.063138 18894 raft_consensus.cc:515] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
W20260709 12:30:42.059931 18805 leader_election.cc:343] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:42.068420 18827 tablet_service.cc:1511] Processing CreateTablet for tablet efa9444e110e433aa97b25a05d5769a9 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:30:42.070291 18827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efa9444e110e433aa97b25a05d5769a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.076553 18693 raft_consensus.cc:2393] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 29f82a8594d741bc92dd24ee4a368e01 in current term 1: Already voted for candidate 60d61e0800194dd89be34d3aff3f374d in this term.
I20260709 12:30:42.077669 18767 raft_consensus.cc:2468] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:42.078547 18888 tablet_bootstrap.cc:492] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:42.079154 18888 ts_tablet_manager.cc:1403] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.295s	user 0.015s	sys 0.004s
I20260709 12:30:42.079859 18805 leader_election.cc:304] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:42.081125 18731 leader_election.cc:304] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb
I20260709 12:30:42.081749 18893 raft_consensus.cc:2804] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:42.082598 18893 raft_consensus.cc:697] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:42.082643 18895 raft_consensus.cc:3060] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:42.083123 18886 tablet_bootstrap.cc:492] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:42.082827 18888 raft_consensus.cc:359] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:42.083741 18886 ts_tablet_manager.cc:1403] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.303s	user 0.043s	sys 0.034s
I20260709 12:30:42.083798 18888 raft_consensus.cc:385] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:42.084205 18888 raft_consensus.cc:740] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:42.085052 18888 consensus_queue.cc:260] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
W20260709 12:30:42.090690 18563 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:30:42.091771 18893 consensus_queue.cc:237] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:42.093986 18895 raft_consensus.cc:2749] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:42.097818 18680 tablet_service.cc:1511] Processing CreateTablet for tablet 7089bc11f6424c178808e4579d967e14 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:30:42.100863 18680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7089bc11f6424c178808e4579d967e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.103868 18681 tablet_service.cc:1511] Processing CreateTablet for tablet 9d43c01f2ed54eec9d5a7b6b75cd6eeb (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:30:42.087533 18886 raft_consensus.cc:359] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:42.105849 18886 raft_consensus.cc:385] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:42.113037 18681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d43c01f2ed54eec9d5a7b6b75cd6eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.117475 18605 tablet_service.cc:1511] Processing CreateTablet for tablet 3fddb762772a40acb39a71286c8d4f41 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:30:42.118551 18602 tablet_service.cc:1511] Processing CreateTablet for tablet 9d43c01f2ed54eec9d5a7b6b75cd6eeb (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:30:42.120679 18605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fddb762772a40acb39a71286c8d4f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.121757 18828 tablet_service.cc:1511] Processing CreateTablet for tablet 7089bc11f6424c178808e4579d967e14 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:30:42.117543 18892 tablet_bootstrap.cc:492] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:42.123250 18892 ts_tablet_manager.cc:1403] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.095s	user 0.015s	sys 0.000s
I20260709 12:30:42.123525 18828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7089bc11f6424c178808e4579d967e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.127003 18753 tablet_service.cc:1511] Processing CreateTablet for tablet 9d43c01f2ed54eec9d5a7b6b75cd6eeb (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:30:42.128722 18602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d43c01f2ed54eec9d5a7b6b75cd6eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.133939 18829 tablet_service.cc:1511] Processing CreateTablet for tablet 3fddb762772a40acb39a71286c8d4f41 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:30:42.134403 18756 tablet_service.cc:1511] Processing CreateTablet for tablet 7089bc11f6424c178808e4579d967e14 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:30:42.135946 18829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fddb762772a40acb39a71286c8d4f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.136504 18892 raft_consensus.cc:359] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:42.137423 18892 raft_consensus.cc:385] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:42.137806 18892 raft_consensus.cc:740] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:42.138273 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bf5c53527542a6a9069536baff450f" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:42.138954 18618 raft_consensus.cc:3060] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:42.146914 18753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d43c01f2ed54eec9d5a7b6b75cd6eeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.165910 18682 tablet_service.cc:1511] Processing CreateTablet for tablet 3fddb762772a40acb39a71286c8d4f41 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:30:42.167454 18682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fddb762772a40acb39a71286c8d4f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.167747 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8bf5c53527542a6a9069536baff450f" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:42.168217 18887 tablet_bootstrap.cc:654] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:42.112231 18886 raft_consensus.cc:740] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:42.172596 18756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7089bc11f6424c178808e4579d967e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.173238 18894 leader_election.cc:290] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:42.183182 18767 raft_consensus.cc:2468] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:42.185122 18657 leader_election.cc:304] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:42.186810 18618 raft_consensus.cc:2468] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:42.192653 18894 raft_consensus.cc:2804] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:42.194204 18888 ts_tablet_manager.cc:1434] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.115s	user 0.038s	sys 0.037s
I20260709 12:30:42.195518 18888 tablet_bootstrap.cc:492] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:42.161664 18892 consensus_queue.cc:260] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:42.187295 18514 catalog_manager.cc:5697] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:42.207173 18757 tablet_service.cc:1511] Processing CreateTablet for tablet 79f6d22c56d84a6cb05eb4469f831e6e (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:30:42.211444 18757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79f6d22c56d84a6cb05eb4469f831e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.211716 18831 tablet_service.cc:1511] Processing CreateTablet for tablet 79f6d22c56d84a6cb05eb4469f831e6e (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:30:42.213140 18831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79f6d22c56d84a6cb05eb4469f831e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.226786 18604 tablet_service.cc:1511] Processing CreateTablet for tablet 79f6d22c56d84a6cb05eb4469f831e6e (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:30:42.228283 18604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79f6d22c56d84a6cb05eb4469f831e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.230087 18887 tablet_bootstrap.cc:492] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:42.230671 18887 ts_tablet_manager.cc:1403] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.209s	user 0.018s	sys 0.025s
I20260709 12:30:42.237416 18679 tablet_service.cc:1511] Processing CreateTablet for tablet c2abc15bf7b445a4b4269e1d93cce949 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:30:42.238929 18679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2abc15bf7b445a4b4269e1d93cce949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.240305 18888 tablet_bootstrap.cc:654] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:42.172449 18886 consensus_queue.cc:260] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:42.245721 18893 raft_consensus.cc:493] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:42.246308 18893 raft_consensus.cc:515] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:42.247906 18892 ts_tablet_manager.cc:1434] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.124s	user 0.008s	sys 0.003s
I20260709 12:30:42.248706 18893 leader_election.cc:290] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:42.249102 18892 tablet_bootstrap.cc:492] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:42.254308 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9219e0ff53f348ff98bdc3422e5fb74c" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:42.255548 18887 raft_consensus.cc:359] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
W20260709 12:30:42.255964 18804 leader_election.cc:343] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:42.256410 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9219e0ff53f348ff98bdc3422e5fb74c" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:42.257109 18693 raft_consensus.cc:2468] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:42.258437 18805 leader_election.cc:304] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:42.259186 18830 tablet_service.cc:1511] Processing CreateTablet for tablet c2abc15bf7b445a4b4269e1d93cce949 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:30:42.261180 18893 raft_consensus.cc:2804] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:42.262908 18893 raft_consensus.cc:493] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:42.263284 18893 raft_consensus.cc:3060] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:42.267633 18892 tablet_bootstrap.cc:654] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:42.252951 18894 raft_consensus.cc:697] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:30:42.269142 18894 consensus_queue.cc:237] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:42.270054 18830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2abc15bf7b445a4b4269e1d93cce949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.283674 18606 tablet_service.cc:1511] Processing CreateTablet for tablet 9e3d64db91924be490d4ed1f797d338f (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:30:42.286377 18606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e3d64db91924be490d4ed1f797d338f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.289433 18752 tablet_service.cc:1511] Processing CreateTablet for tablet c2abc15bf7b445a4b4269e1d93cce949 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:30:42.291229 18752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2abc15bf7b445a4b4269e1d93cce949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.295884 18893 raft_consensus.cc:515] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:42.298321 18893 leader_election.cc:290] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:42.300771 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9219e0ff53f348ff98bdc3422e5fb74c" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
W20260709 12:30:42.302709 18804 leader_election.cc:343] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:42.303763 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9219e0ff53f348ff98bdc3422e5fb74c" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:42.304709 18693 raft_consensus.cc:3060] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:42.261575 18887 raft_consensus.cc:385] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:42.319690 18886 ts_tablet_manager.cc:1434] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.236s	user 0.032s	sys 0.037s
I20260709 12:30:42.321414 18886 tablet_bootstrap.cc:492] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:42.312219 18887 raft_consensus.cc:740] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:42.327559 18887 consensus_queue.cc:260] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:42.341841 18926 raft_consensus.cc:493] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:42.342526 18886 tablet_bootstrap.cc:654] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:42.342482 18926 raft_consensus.cc:515] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:42.343093 18683 tablet_service.cc:1511] Processing CreateTablet for tablet 9e3d64db91924be490d4ed1f797d338f (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:30:42.343767 18887 ts_tablet_manager.cc:1434] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.113s	user 0.007s	sys 0.004s
I20260709 12:30:42.344594 18683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e3d64db91924be490d4ed1f797d338f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.345396 18887 tablet_bootstrap.cc:492] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:42.348953 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d659dc42667d4202a29b529fe3388c8e" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
W20260709 12:30:42.350364 18730 leader_election.cc:343] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:42.351233 18753 tablet_service.cc:1511] Processing CreateTablet for tablet 9e3d64db91924be490d4ed1f797d338f (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:30:42.349107 18888 tablet_bootstrap.cc:492] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:42.360113 18888 ts_tablet_manager.cc:1403] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.165s	user 0.017s	sys 0.002s
I20260709 12:30:42.360608 18753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e3d64db91924be490d4ed1f797d338f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.362378 18893 raft_consensus.cc:493] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:42.363027 18893 raft_consensus.cc:515] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:42.357360 18693 raft_consensus.cc:2468] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:42.352761 18926 leader_election.cc:290] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:42.398746 18805 leader_election.cc:304] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:42.400038 18906 raft_consensus.cc:2804] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:42.400663 18893 leader_election.cc:290] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:42.402242 18888 raft_consensus.cc:359] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:42.404502 18892 tablet_bootstrap.cc:492] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:42.406602 18604 tablet_service.cc:1511] Processing CreateTablet for tablet d9e24af510de430aabe2a8c0c1010629 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:30:42.407970 18680 tablet_service.cc:1511] Processing CreateTablet for tablet d9e24af510de430aabe2a8c0c1010629 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:30:42.410807 18888 raft_consensus.cc:385] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:42.424520 18604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9e24af510de430aabe2a8c0c1010629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.415319 18906 raft_consensus.cc:697] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:42.416651 18892 ts_tablet_manager.cc:1403] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.167s	user 0.022s	sys 0.017s
I20260709 12:30:42.422430 18888 raft_consensus.cc:740] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:42.441870 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6e49ecce4f040e99a419db81f1953af" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:42.432854 18906 consensus_queue.cc:237] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:42.435847 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6e49ecce4f040e99a419db81f1953af" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:42.433089 18829 tablet_service.cc:1511] Processing CreateTablet for tablet d9e24af510de430aabe2a8c0c1010629 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:30:42.445531 18680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9e24af510de430aabe2a8c0c1010629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.448278 18887 tablet_bootstrap.cc:654] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:42.445561 18831 tablet_service.cc:1511] Processing CreateTablet for tablet 9c4804407f8b43bd868a716c04e955e0 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:30:42.449139 18683 tablet_service.cc:1511] Processing CreateTablet for tablet a147020c5d724ef2a43e17ee907fe200 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:30:42.454259 18888 consensus_queue.cc:260] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:42.468765 18757 tablet_service.cc:1511] Processing CreateTablet for tablet a147020c5d724ef2a43e17ee907fe200 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:30:42.469357 18606 tablet_service.cc:1511] Processing CreateTablet for tablet 3c6af67983744957af1aee8d948169c2 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:30:42.470254 18757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a147020c5d724ef2a43e17ee907fe200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.470672 18606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c6af67983744957af1aee8d948169c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.472702 18682 tablet_service.cc:1511] Processing CreateTablet for tablet 3c6af67983744957af1aee8d948169c2 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:30:42.473985 18827 tablet_service.cc:1511] Processing CreateTablet for tablet a147020c5d724ef2a43e17ee907fe200 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:30:42.475725 18756 tablet_service.cc:1511] Processing CreateTablet for tablet 9c4804407f8b43bd868a716c04e955e0 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:30:42.477172 18756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c4804407f8b43bd868a716c04e955e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.477555 18755 tablet_service.cc:1511] Processing CreateTablet for tablet 3c6af67983744957af1aee8d948169c2 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:30:42.479074 18755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c6af67983744957af1aee8d948169c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.465595 18829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9e24af510de430aabe2a8c0c1010629. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:30:42.462970 18805 leader_election.cc:343] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:42.493557 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d659dc42667d4202a29b529fe3388c8e" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:42.497890 18841 raft_consensus.cc:2468] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:42.489446 18681 tablet_service.cc:1511] Processing CreateTablet for tablet f4eead98f0854adfaf72daa76fdf9c9b (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:30:42.500821 18827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a147020c5d724ef2a43e17ee907fe200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.502158 18602 tablet_service.cc:1511] Processing CreateTablet for tablet 9c4804407f8b43bd868a716c04e955e0 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:30:42.512701 18902 raft_consensus.cc:493] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:42.513785 18886 tablet_bootstrap.cc:492] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:42.513695 18902 raft_consensus.cc:515] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:42.517711 18730 leader_election.cc:304] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:42.518734 18926 raft_consensus.cc:2804] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:42.519167 18926 raft_consensus.cc:493] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:42.519532 18926 raft_consensus.cc:3060] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:30:42.520121 18804 leader_election.cc:343] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:42.520717 18804 leader_election.cc:304] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb
I20260709 12:30:42.521062 18893 consensus_queue.cc:1048] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:42.523409 18893 raft_consensus.cc:2749] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:42.514591 18886 ts_tablet_manager.cc:1403] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.193s	user 0.021s	sys 0.023s
I20260709 12:30:42.526697 18831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c4804407f8b43bd868a716c04e955e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.531388 18693 raft_consensus.cc:3060] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:42.532832 18602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c4804407f8b43bd868a716c04e955e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.532708 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dfef0fa153945568323437021ce2921" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:42.546541 18582 leader_election.cc:343] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:42.529035 18683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a147020c5d724ef2a43e17ee907fe200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.548010 18681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4eead98f0854adfaf72daa76fdf9c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.550781 18895 raft_consensus.cc:493] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:42.551343 18895 raft_consensus.cc:515] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:42.535777 18888 ts_tablet_manager.cc:1434] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.175s	user 0.008s	sys 0.001s
I20260709 12:30:42.496073 18830 tablet_service.cc:1511] Processing CreateTablet for tablet f4eead98f0854adfaf72daa76fdf9c9b (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:30:42.555572 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f651b7eda3134c7a897855ac4300de2f" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
W20260709 12:30:42.557782 18731 leader_election.cc:343] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:42.557848 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f651b7eda3134c7a897855ac4300de2f" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:42.558668 18618 raft_consensus.cc:2468] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:42.561628 18753 tablet_service.cc:1511] Processing CreateTablet for tablet 16afc1d74df849c48d10f5da0a8f80c4 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:30:42.562475 18730 leader_election.cc:304] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:42.563400 18753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16afc1d74df849c48d10f5da0a8f80c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.563812 18830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4eead98f0854adfaf72daa76fdf9c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.537168 18892 raft_consensus.cc:359] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:42.543745 18828 tablet_service.cc:1511] Processing CreateTablet for tablet 16afc1d74df849c48d10f5da0a8f80c4 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:30:42.553637 18895 leader_election.cc:290] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:42.540762 18902 leader_election.cc:290] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:42.566586 18682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c6af67983744957af1aee8d948169c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.589331 18828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16afc1d74df849c48d10f5da0a8f80c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.566921 18892 raft_consensus.cc:385] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:42.581637 18888 tablet_bootstrap.cc:492] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:42.585814 18895 raft_consensus.cc:2804] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:42.596094 18892 raft_consensus.cc:740] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:42.605412 18895 raft_consensus.cc:493] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:42.610157 18887 tablet_bootstrap.cc:492] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:42.618184 18895 raft_consensus.cc:3060] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:42.618588 18887 ts_tablet_manager.cc:1403] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.274s	user 0.012s	sys 0.016s
I20260709 12:30:42.621466 18894 raft_consensus.cc:493] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:42.622301 18887 raft_consensus.cc:359] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:42.622248 18894 raft_consensus.cc:515] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:42.623370 18887 raft_consensus.cc:385] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:42.623790 18887 raft_consensus.cc:740] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:42.599598 18886 raft_consensus.cc:359] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:42.624557 18886 raft_consensus.cc:385] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:42.624941 18886 raft_consensus.cc:740] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:42.626113 18894 leader_election.cc:290] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:42.625826 18886 consensus_queue.cc:260] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:42.629407 18886 ts_tablet_manager.cc:1434] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.100s	user 0.004s	sys 0.005s
I20260709 12:30:42.629922 18926 raft_consensus.cc:515] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:42.630818 18886 tablet_bootstrap.cc:492] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:42.632645 18926 leader_election.cc:290] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:42.633456 18887 consensus_queue.cc:260] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:42.635957 18887 ts_tablet_manager.cc:1434] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 12:30:42.636775 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d659dc42667d4202a29b529fe3388c8e" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:42.637104 18887 tablet_bootstrap.cc:492] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:42.637468 18841 raft_consensus.cc:3060] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:42.676669 18886 tablet_bootstrap.cc:654] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:42.677570 18895 raft_consensus.cc:515] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:42.682993 18947 raft_consensus.cc:493] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:42.670092 18892 consensus_queue.cc:260] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:42.683571 18947 raft_consensus.cc:515] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:42.685195 18892 ts_tablet_manager.cc:1434] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.249s	user 0.013s	sys 0.000s
I20260709 12:30:42.685382 18895 leader_election.cc:290] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:42.685889 18947 leader_election.cc:290] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:42.716575 18604 tablet_service.cc:1511] Processing CreateTablet for tablet f4eead98f0854adfaf72daa76fdf9c9b (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:30:42.721748 18604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4eead98f0854adfaf72daa76fdf9c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.728060 18515 catalog_manager.cc:5697] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:42.732960 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d659dc42667d4202a29b529fe3388c8e" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:42.730465 18606 tablet_service.cc:1511] Processing CreateTablet for tablet 16afc1d74df849c48d10f5da0a8f80c4 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:30:42.753823 18679 tablet_service.cc:1511] Processing CreateTablet for tablet 3eed637e5890430db2e802ef6d4362a7 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:30:42.761641 18752 tablet_service.cc:1511] Processing CreateTablet for tablet 3eed637e5890430db2e802ef6d4362a7 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:30:42.756176 18679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eed637e5890430db2e802ef6d4362a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.769167 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dfef0fa153945568323437021ce2921" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:42.771111 18616 raft_consensus.cc:2468] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:42.761740 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dfef0fa153945568323437021ce2921" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:42.776971 18829 tablet_service.cc:1511] Processing CreateTablet for tablet 3eed637e5890430db2e802ef6d4362a7 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:30:42.779609 18829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eed637e5890430db2e802ef6d4362a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.762692 18949 consensus_queue.cc:1048] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260709 12:30:42.782505 18605 tablet_service.cc:1511] Processing CreateTablet for tablet c11b1d6d5a5c4991bd34743fff5ca507 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:30:42.789340 18752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3eed637e5890430db2e802ef6d4362a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.792840 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14206c16a2154538ad2a781e824a65dd" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:42.798775 18894 consensus_queue.cc:1048] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:42.801158 18606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16afc1d74df849c48d10f5da0a8f80c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.806506 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f651b7eda3134c7a897855ac4300de2f" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:42.808125 18617 raft_consensus.cc:3060] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:42.808305 18894 raft_consensus.cc:493] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:42.809386 18894 raft_consensus.cc:515] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:42.817405 18804 leader_election.cc:304] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:42.818991 18949 raft_consensus.cc:2804] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:42.819482 18949 raft_consensus.cc:493] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:42.822947 18678 tablet_service.cc:1511] Processing CreateTablet for tablet c11b1d6d5a5c4991bd34743fff5ca507 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:30:42.819964 18949 raft_consensus.cc:3060] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:42.825117 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f651b7eda3134c7a897855ac4300de2f" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:42.807220 18605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11b1d6d5a5c4991bd34743fff5ca507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.826177 18693 raft_consensus.cc:3060] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:42.826249 18678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11b1d6d5a5c4991bd34743fff5ca507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.831430 18892 tablet_bootstrap.cc:492] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:30:42.836145 18730 leader_election.cc:343] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:42.851939 18902 raft_consensus.cc:493] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:42.853865 18902 raft_consensus.cc:515] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:42.870018 18513 catalog_manager.cc:5697] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:42.872172 18754 tablet_service.cc:1511] Processing CreateTablet for tablet c11b1d6d5a5c4991bd34743fff5ca507 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:30:42.874032 18831 tablet_service.cc:1511] Processing CreateTablet for tablet 4794853627e84898986305827ec4ea46 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:30:42.874632 18754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c11b1d6d5a5c4991bd34743fff5ca507. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.877377 18902 leader_election.cc:290] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:42.886351 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dfef0fa153945568323437021ce2921" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:42.889297 18839 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6e49ecce4f040e99a419db81f1953af" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:42.890599 18839 raft_consensus.cc:2468] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 0.
I20260709 12:30:42.890746 18888 tablet_bootstrap.cc:654] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:42.892501 18581 leader_election.cc:304] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:42.897049 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f651b7eda3134c7a897855ac4300de2f" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:42.898542 18831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4794853627e84898986305827ec4ea46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.907112 18902 raft_consensus.cc:2804] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:42.907724 18902 raft_consensus.cc:493] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:42.907538 18601 tablet_service.cc:1511] Processing CreateTablet for tablet 4794853627e84898986305827ec4ea46 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:30:42.908216 18902 raft_consensus.cc:3060] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:42.909860 18601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4794853627e84898986305827ec4ea46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.919426 18887 tablet_bootstrap.cc:654] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:42.926718 18894 leader_election.cc:290] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:42.930756 18886 tablet_bootstrap.cc:492] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:42.931967 18886 ts_tablet_manager.cc:1403] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.301s	user 0.012s	sys 0.017s
I20260709 12:30:42.932725 18894 consensus_queue.cc:1048] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:42.934943 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f651b7eda3134c7a897855ac4300de2f" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:42.944605 18765 raft_consensus.cc:2393] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d61e0800194dd89be34d3aff3f374d in current term 1: Already voted for candidate 29f82a8594d741bc92dd24ee4a368e01 in this term.
I20260709 12:30:42.937424 18841 raft_consensus.cc:2468] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:42.955878 18730 leader_election.cc:304] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:42.958442 18895 raft_consensus.cc:2804] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:42.959091 18895 raft_consensus.cc:697] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:42.960414 18895 consensus_queue.cc:237] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:42.961294 18830 tablet_service.cc:1511] Processing CreateTablet for tablet 7eb0c96994864b03a8ec7cda3cc8d85b (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:30:42.963080 18830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb0c96994864b03a8ec7cda3cc8d85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.963856 18751 tablet_service.cc:1511] Processing CreateTablet for tablet 7eb0c96994864b03a8ec7cda3cc8d85b (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:30:42.974874 18892 tablet_bootstrap.cc:654] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:42.976943 18751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb0c96994864b03a8ec7cda3cc8d85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.976863 18607 tablet_service.cc:1511] Processing CreateTablet for tablet 7eb0c96994864b03a8ec7cda3cc8d85b (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:30:42.980188 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6e49ecce4f040e99a419db81f1953af" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:42.981848 18607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb0c96994864b03a8ec7cda3cc8d85b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:30:42.985502 18805 leader_election.cc:343] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:42.992499 18657 leader_election.cc:304] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:42.975673 18683 tablet_service.cc:1511] Processing CreateTablet for tablet 4794853627e84898986305827ec4ea46 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:30:42.995272 18683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4794853627e84898986305827ec4ea46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:42.997114 18886 raft_consensus.cc:359] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:42.998140 18886 raft_consensus.cc:385] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:42.998551 18886 raft_consensus.cc:740] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:42.999416 18886 consensus_queue.cc:260] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
W20260709 12:30:43.001609 18804 consensus_peers.cc:597] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:43.003371 18680 tablet_service.cc:1511] Processing CreateTablet for tablet ba28ff472b0f46e8884f70afb21857ae (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:30:43.004701 18680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba28ff472b0f46e8884f70afb21857ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.004550 18756 tablet_service.cc:1511] Processing CreateTablet for tablet ba28ff472b0f46e8884f70afb21857ae (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:30:43.005951 18756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba28ff472b0f46e8884f70afb21857ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.033090 18949 raft_consensus.cc:515] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:43.036564 18840 raft_consensus.cc:2393] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84812365c65f43f388836c087640adbb in current term 1: Already voted for candidate df352ed4600b4ddb83ef57ecd2944fce in this term.
I20260709 12:30:43.037541 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dfef0fa153945568323437021ce2921" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:43.038435 18616 raft_consensus.cc:3060] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:43.040400 18617 raft_consensus.cc:2468] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:43.041808 18949 leader_election.cc:290] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:43.042660 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dfef0fa153945568323437021ce2921" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:43.045301 18730 leader_election.cc:304] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:30:43.066181 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14206c16a2154538ad2a781e824a65dd" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:43.070338 18888 tablet_bootstrap.cc:492] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:43.071470 18888 ts_tablet_manager.cc:1403] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.505s	user 0.016s	sys 0.008s
I20260709 12:30:43.073580 18656 leader_election.cc:304] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:43.073951 18827 tablet_service.cc:1511] Processing CreateTablet for tablet ba28ff472b0f46e8884f70afb21857ae (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:30:43.076092 18827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba28ff472b0f46e8884f70afb21857ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.079860 18581 leader_election.cc:304] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:43.081827 18895 raft_consensus.cc:2804] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:30:43.108551 18805 leader_election.cc:343] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:43.109799 18886 ts_tablet_manager.cc:1434] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.160s	user 0.011s	sys 0.002s
I20260709 12:30:43.133956 18681 tablet_service.cc:1511] Processing CreateTablet for tablet 23ca4b75eb0e49dca52e1b9fb4f8b356 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:30:43.135627 18681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23ca4b75eb0e49dca52e1b9fb4f8b356. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:30:43.136929 18581 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:44517, user_credentials={real_user=slave}} blocked reactor thread for 57318.1us
I20260709 12:30:43.141059 18616 raft_consensus.cc:2468] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:43.117970 18895 raft_consensus.cc:697] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:43.144915 18949 consensus_queue.cc:1048] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:30:43.150138 18895 consensus_queue.cc:237] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:43.152994 18693 raft_consensus.cc:2468] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:43.155220 18513 catalog_manager.cc:5697] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:43.158212 18603 tablet_service.cc:1511] Processing CreateTablet for tablet 23ca4b75eb0e49dca52e1b9fb4f8b356 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:30:43.161936 18603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23ca4b75eb0e49dca52e1b9fb4f8b356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.161783 18902 raft_consensus.cc:515] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:43.166920 18887 tablet_bootstrap.cc:492] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:43.158918 18894 raft_consensus.cc:2749] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:43.167788 18887 ts_tablet_manager.cc:1403] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.531s	user 0.020s	sys 0.019s
I20260709 12:30:43.169013 18957 raft_consensus.cc:2749] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:43.172029 18804 leader_election.cc:304] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:30:43.172520 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6e49ecce4f040e99a419db81f1953af" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:43.173234 18906 raft_consensus.cc:2804] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:43.173508 18840 raft_consensus.cc:3060] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:43.173832 18906 raft_consensus.cc:697] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:43.174865 18906 consensus_queue.cc:237] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
W20260709 12:30:43.105250 18657 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:40335, user_credentials={real_user=slave}} blocked reactor thread for 111202us
W20260709 12:30:43.187340 18656 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:44517, user_credentials={real_user=slave}} blocked reactor thread for 114128us
I20260709 12:30:43.191291 18894 raft_consensus.cc:2749] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:30:43.204876 18563 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 12:30:43.213912 18582 leader_election.cc:343] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:43.233363 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6e49ecce4f040e99a419db81f1953af" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:43.233453 18754 tablet_service.cc:1511] Processing CreateTablet for tablet 23ca4b75eb0e49dca52e1b9fb4f8b356 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
W20260709 12:30:43.235157 18582 leader_election.cc:343] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:43.235507 18754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23ca4b75eb0e49dca52e1b9fb4f8b356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.227092 18902 leader_election.cc:290] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:43.248775 18840 raft_consensus.cc:2468] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 1.
I20260709 12:30:43.250259 18581 leader_election.cc:304] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:43.262171 18892 tablet_bootstrap.cc:492] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:43.262288 18828 tablet_service.cc:1511] Processing CreateTablet for tablet 6256d41aab434828a66720ab61fa9ba5 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:30:43.262966 18892 ts_tablet_manager.cc:1403] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.577s	user 0.033s	sys 0.023s
I20260709 12:30:43.269853 18828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6256d41aab434828a66720ab61fa9ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.269595 18607 tablet_service.cc:1511] Processing CreateTablet for tablet 6256d41aab434828a66720ab61fa9ba5 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:30:43.270936 18607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6256d41aab434828a66720ab61fa9ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.274042 18682 tablet_service.cc:1511] Processing CreateTablet for tablet 6256d41aab434828a66720ab61fa9ba5 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:30:43.282959 18682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6256d41aab434828a66720ab61fa9ba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.284300 18514 catalog_manager.cc:5697] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:43.295984 18513 catalog_manager.cc:5697] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:43.301674 18826 tablet_service.cc:1511] Processing CreateTablet for tablet fc988d3d39094696b3b4516b33a9c37d (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:30:43.303048 18826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc988d3d39094696b3b4516b33a9c37d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.301508 18753 tablet_service.cc:1511] Processing CreateTablet for tablet fc988d3d39094696b3b4516b33a9c37d (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:30:43.304232 18753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc988d3d39094696b3b4516b33a9c37d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.306659 18888 raft_consensus.cc:359] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:43.307394 18888 raft_consensus.cc:385] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:43.307653 18888 raft_consensus.cc:740] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:43.308494 18888 consensus_queue.cc:260] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:43.309973 18886 tablet_bootstrap.cc:492] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:43.313922 18892 raft_consensus.cc:359] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:43.314688 18892 raft_consensus.cc:385] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:43.315008 18892 raft_consensus.cc:740] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:43.316329 18602 tablet_service.cc:1511] Processing CreateTablet for tablet fc988d3d39094696b3b4516b33a9c37d (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:30:43.323801 18602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc988d3d39094696b3b4516b33a9c37d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.318480 18887 raft_consensus.cc:359] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:43.328992 18887 raft_consensus.cc:385] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:43.329324 18887 raft_consensus.cc:740] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:43.315810 18892 consensus_queue.cc:260] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:43.256685 18902 raft_consensus.cc:2804] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:30:43.321381 18755 consensus_meta.cc:287] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01: Time spent flushing consensus metadata: real 0.614s	user 0.007s	sys 0.012s
I20260709 12:30:43.327942 18886 tablet_bootstrap.cc:654] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:43.318120 18888 ts_tablet_manager.cc:1434] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.246s	user 0.012s	sys 0.000s
I20260709 12:30:43.339685 18682 tablet_service.cc:1511] Processing CreateTablet for tablet 98dfe85df2e240449ec5f18cb3280c9b (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:30:43.330101 18887 consensus_queue.cc:260] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:43.354386 18682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98dfe85df2e240449ec5f18cb3280c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.356551 18960 raft_consensus.cc:493] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:43.357129 18960 raft_consensus.cc:515] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:43.358435 18892 ts_tablet_manager.cc:1434] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.095s	user 0.002s	sys 0.007s
I20260709 12:30:43.359709 18887 ts_tablet_manager.cc:1434] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.191s	user 0.007s	sys 0.006s
I20260709 12:30:43.360025 18892 tablet_bootstrap.cc:492] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:43.361390 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:43.362139 18888 tablet_bootstrap.cc:492] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:43.362283 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
W20260709 12:30:43.363162 18657 leader_election.cc:343] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:43.361420 18752 tablet_service.cc:1511] Processing CreateTablet for tablet 98dfe85df2e240449ec5f18cb3280c9b (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
W20260709 12:30:43.364190 18656 leader_election.cc:343] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:43.364993 18656 leader_election.cc:304] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:43.365278 18752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98dfe85df2e240449ec5f18cb3280c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.365379 18960 leader_election.cc:290] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:43.366454 18955 raft_consensus.cc:2749] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:43.381016 18887 tablet_bootstrap.cc:492] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:43.404917 18892 tablet_bootstrap.cc:654] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:43.410259 18756 tablet_service.cc:1511] Processing CreateTablet for tablet 1a66b75f31e14b98902006a8f9d53a95 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:30:43.411863 18756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a66b75f31e14b98902006a8f9d53a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.412972 18830 tablet_service.cc:1511] Processing CreateTablet for tablet 98dfe85df2e240449ec5f18cb3280c9b (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:30:43.406132 18602 tablet_service.cc:1511] Processing CreateTablet for tablet 1a66b75f31e14b98902006a8f9d53a95 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:30:43.414842 18830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98dfe85df2e240449ec5f18cb3280c9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.415299 18602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a66b75f31e14b98902006a8f9d53a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.420200 18678 tablet_service.cc:1511] Processing CreateTablet for tablet d216c51d889b417bb2e59f20b6fd6d96 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:30:43.421728 18678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d216c51d889b417bb2e59f20b6fd6d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.421126 18755 tablet_service.cc:1511] Processing CreateTablet for tablet 1ab29498797f49d198e49d29f9800da2 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:30:43.425256 18755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ab29498797f49d198e49d29f9800da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.430604 18680 tablet_service.cc:1511] Processing CreateTablet for tablet ef4e065748614b3c87fd0a834017c455 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:30:43.432126 18680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef4e065748614b3c87fd0a834017c455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.433167 18754 tablet_service.cc:1511] Processing CreateTablet for tablet ef4e065748614b3c87fd0a834017c455 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:30:43.434630 18754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef4e065748614b3c87fd0a834017c455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.441969 18888 tablet_bootstrap.cc:654] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:43.446482 18827 tablet_service.cc:1511] Processing CreateTablet for tablet d216c51d889b417bb2e59f20b6fd6d96 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:30:43.449174 18827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d216c51d889b417bb2e59f20b6fd6d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.450506 18607 tablet_service.cc:1511] Processing CreateTablet for tablet d216c51d889b417bb2e59f20b6fd6d96 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:30:43.451403 18828 tablet_service.cc:1511] Processing CreateTablet for tablet 1ab29498797f49d198e49d29f9800da2 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:30:43.454389 18607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d216c51d889b417bb2e59f20b6fd6d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.454389 18828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ab29498797f49d198e49d29f9800da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.452632 18603 tablet_service.cc:1511] Processing CreateTablet for tablet 1ab29498797f49d198e49d29f9800da2 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:30:43.459161 18902 raft_consensus.cc:697] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:30:43.466440 18902 consensus_queue.cc:237] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:43.468528 18603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ab29498797f49d198e49d29f9800da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.478104 18683 tablet_service.cc:1511] Processing CreateTablet for tablet a6fc5ff8d0014a9f9c1dadcc157e1d9a (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:30:43.479382 18683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6fc5ff8d0014a9f9c1dadcc157e1d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.478104 18681 tablet_service.cc:1511] Processing CreateTablet for tablet 1a66b75f31e14b98902006a8f9d53a95 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:30:43.485767 18895 consensus_queue.cc:1048] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:43.491840 18975 raft_consensus.cc:493] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:43.492596 18975 raft_consensus.cc:515] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:43.496289 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d659dc42667d4202a29b529fe3388c8e" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:43.496459 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d659dc42667d4202a29b529fe3388c8e" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:43.498488 18604 tablet_service.cc:1511] Processing CreateTablet for tablet a6fc5ff8d0014a9f9c1dadcc157e1d9a (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:30:43.498885 18581 leader_election.cc:304] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:43.499845 18957 raft_consensus.cc:2749] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:43.500051 18604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6fc5ff8d0014a9f9c1dadcc157e1d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.503518 18679 tablet_service.cc:1511] Processing CreateTablet for tablet e06e3fb1322840a6b22c5da53900345b (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:30:43.509133 18753 tablet_service.cc:1511] Processing CreateTablet for tablet a6fc5ff8d0014a9f9c1dadcc157e1d9a (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:30:43.511543 18753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6fc5ff8d0014a9f9c1dadcc157e1d9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.482102 18681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a66b75f31e14b98902006a8f9d53a95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.528995 18829 tablet_service.cc:1511] Processing CreateTablet for tablet e06e3fb1322840a6b22c5da53900345b (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:30:43.529492 18754 tablet_service.cc:1511] Processing CreateTablet for tablet e4c43366d9fb416e8649aba9676aed59 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:30:43.530607 18829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06e3fb1322840a6b22c5da53900345b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.534389 18895 raft_consensus.cc:493] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:43.513381 18975 leader_election.cc:290] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:43.534967 18895 raft_consensus.cc:515] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:43.535230 18606 tablet_service.cc:1511] Processing CreateTablet for tablet e06e3fb1322840a6b22c5da53900345b (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:30:43.536567 18606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06e3fb1322840a6b22c5da53900345b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.536198 18826 tablet_service.cc:1511] Processing CreateTablet for tablet ef4e065748614b3c87fd0a834017c455 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:30:43.537566 18826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef4e065748614b3c87fd0a834017c455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.539033 18887 tablet_bootstrap.cc:654] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:43.545145 18886 tablet_bootstrap.cc:492] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:43.545897 18886 ts_tablet_manager.cc:1403] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.269s	user 0.025s	sys 0.009s
I20260709 12:30:43.546695 18892 tablet_bootstrap.cc:492] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:43.547219 18892 ts_tablet_manager.cc:1403] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.188s	user 0.020s	sys 0.004s
I20260709 12:30:43.549113 18754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4c43366d9fb416e8649aba9676aed59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.559587 18892 raft_consensus.cc:359] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:43.560508 18892 raft_consensus.cc:385] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:43.560833 18892 raft_consensus.cc:740] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:43.561619 18602 tablet_service.cc:1511] Processing CreateTablet for tablet e4c43366d9fb416e8649aba9676aed59 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:30:43.562984 18602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4c43366d9fb416e8649aba9676aed59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.566685 18828 tablet_service.cc:1511] Processing CreateTablet for tablet e4c43366d9fb416e8649aba9676aed59 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:30:43.567991 18828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4c43366d9fb416e8649aba9676aed59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.561580 18892 consensus_queue.cc:260] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:43.568862 18679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06e3fb1322840a6b22c5da53900345b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.579046 18682 tablet_service.cc:1511] Processing CreateTablet for tablet 6a46101454ef46beb00f1652fabfa153 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:30:43.585788 18892 ts_tablet_manager.cc:1434] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.038s	user 0.004s	sys 0.004s
I20260709 12:30:43.587203 18892 tablet_bootstrap.cc:492] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:43.590466 18949 raft_consensus.cc:493] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:43.591132 18949 raft_consensus.cc:515] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:43.595152 18616 raft_consensus.cc:3060] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:43.597657 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acf997d23744fe697db1fe947dca883" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
W20260709 12:30:43.599556 18805 leader_election.cc:343] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:43.601049 18895 leader_election.cc:290] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:43.600600 18886 raft_consensus.cc:359] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:43.605268 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dfef0fa153945568323437021ce2921" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:43.620481 18618 raft_consensus.cc:2393] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 29f82a8594d741bc92dd24ee4a368e01 in current term 1: Already voted for candidate df352ed4600b4ddb83ef57ecd2944fce in this term.
I20260709 12:30:43.595620 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dfef0fa153945568323437021ce2921" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:43.621516 18829 tablet_service.cc:1511] Processing CreateTablet for tablet 6a46101454ef46beb00f1652fabfa153 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:30:43.597273 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acf997d23744fe697db1fe947dca883" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:43.622915 18829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a46101454ef46beb00f1652fabfa153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.624109 18892 tablet_bootstrap.cc:654] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:43.614168 18888 tablet_bootstrap.cc:492] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:43.619024 18753 tablet_service.cc:1511] Processing CreateTablet for tablet 6a46101454ef46beb00f1652fabfa153 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:30:43.624261 18730 leader_election.cc:304] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:43.636524 18895 raft_consensus.cc:3060] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:30:43.624970 18805 leader_election.cc:343] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:43.638741 18805 leader_election.cc:304] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:43.640260 18972 raft_consensus.cc:2749] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:43.593672 18949 leader_election.cc:290] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:43.625741 18886 raft_consensus.cc:385] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:43.629571 18682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a46101454ef46beb00f1652fabfa153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.653524 18981 consensus_queue.cc:1048] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:43.657756 18602 tablet_service.cc:1511] Processing CreateTablet for tablet 0840843cb482476a8e18344ff8f1e315 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:30:43.631794 18753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a46101454ef46beb00f1652fabfa153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.661255 18678 tablet_service.cc:1511] Processing CreateTablet for tablet 0840843cb482476a8e18344ff8f1e315 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:30:43.650399 18886 raft_consensus.cc:740] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:43.659318 18888 ts_tablet_manager.cc:1403] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.290s	user 0.020s	sys 0.018s
I20260709 12:30:43.693722 18678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0840843cb482476a8e18344ff8f1e315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.695557 18602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0840843cb482476a8e18344ff8f1e315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.699564 18887 tablet_bootstrap.cc:492] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:43.700409 18828 tablet_service.cc:1511] Processing CreateTablet for tablet 8c4a560e00884f1cb6f8720dfee7d27c (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:30:43.701735 18828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c4a560e00884f1cb6f8720dfee7d27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.705410 18681 tablet_service.cc:1511] Processing CreateTablet for tablet 8c4a560e00884f1cb6f8720dfee7d27c (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:30:43.706615 18681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c4a560e00884f1cb6f8720dfee7d27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.707039 18607 tablet_service.cc:1511] Processing CreateTablet for tablet 8c4a560e00884f1cb6f8720dfee7d27c (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:30:43.700278 18887 ts_tablet_manager.cc:1403] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.320s	user 0.016s	sys 0.005s
I20260709 12:30:43.712006 18949 consensus_queue.cc:1048] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:43.715744 18887 raft_consensus.cc:359] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:43.729645 18515 catalog_manager.cc:5697] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 1 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:43.737520 18607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c4a560e00884f1cb6f8720dfee7d27c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.719168 18886 consensus_queue.cc:260] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:43.746593 18755 tablet_service.cc:1511] Processing CreateTablet for tablet 0840843cb482476a8e18344ff8f1e315 (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:30:43.749064 18755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0840843cb482476a8e18344ff8f1e315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.746038 18827 tablet_service.cc:1511] Processing CreateTablet for tablet 4d64e328a205406bb8de3be26e2a70bf (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:30:43.748984 18886 ts_tablet_manager.cc:1434] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.203s	user 0.004s	sys 0.014s
I20260709 12:30:43.744915 18752 tablet_service.cc:1511] Processing CreateTablet for tablet 4d64e328a205406bb8de3be26e2a70bf (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:30:43.723388 18981 consensus_queue.cc:1048] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260709 12:30:43.765393 18886 tablet_bootstrap.cc:492] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:43.766067 18888 raft_consensus.cc:359] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:43.750572 18887 raft_consensus.cc:385] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:43.775748 18888 raft_consensus.cc:385] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:43.781477 18888 raft_consensus.cc:740] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:43.777429 18887 raft_consensus.cc:740] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:43.782368 18888 consensus_queue.cc:260] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:43.782518 18887 consensus_queue.cc:260] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:43.776187 18902 raft_consensus.cc:493] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:43.785084 18902 raft_consensus.cc:515] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:43.789892 18902 leader_election.cc:290] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:43.791644 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9219e0ff53f348ff98bdc3422e5fb74c" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:43.792655 18926 consensus_queue.cc:1048] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:43.794595 18827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d64e328a205406bb8de3be26e2a70bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.797525 18892 tablet_bootstrap.cc:492] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:43.798169 18892 ts_tablet_manager.cc:1403] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.211s	user 0.023s	sys 0.030s
I20260709 12:30:43.801254 18887 ts_tablet_manager.cc:1434] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.089s	user 0.004s	sys 0.004s
I20260709 12:30:43.801960 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9219e0ff53f348ff98bdc3422e5fb74c" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:43.801827 18892 raft_consensus.cc:359] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:43.802649 18892 raft_consensus.cc:385] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:43.802964 18892 raft_consensus.cc:740] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:43.803922 18892 consensus_queue.cc:260] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:43.804435 18581 leader_election.cc:304] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:43.805042 18752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d64e328a205406bb8de3be26e2a70bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.806232 18957 raft_consensus.cc:2749] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:43.806648 18892 ts_tablet_manager.cc:1434] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.003s
I20260709 12:30:43.807803 18892 tablet_bootstrap.cc:492] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:43.810102 18887 tablet_bootstrap.cc:492] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:43.810035 18895 raft_consensus.cc:2749] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:43.810911 18895 raft_consensus.cc:493] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:43.811340 18895 raft_consensus.cc:515] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:43.814713 18892 tablet_bootstrap.cc:654] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:43.817963 18888 ts_tablet_manager.cc:1434] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.127s	user 0.005s	sys 0.002s
I20260709 12:30:43.819540 18888 tablet_bootstrap.cc:492] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:43.822113 18949 consensus_queue.cc:1048] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:43.823727 18887 tablet_bootstrap.cc:654] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:43.826902 18886 tablet_bootstrap.cc:654] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:43.844326 18602 tablet_service.cc:1511] Processing CreateTablet for tablet 4d64e328a205406bb8de3be26e2a70bf (DEFAULT_TABLE table=test-workload [id=2613363c4984450cba4062fe5a58f2c1]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:30:43.846256 18602 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d64e328a205406bb8de3be26e2a70bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:30:43.853147 18892 tablet_bootstrap.cc:492] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:43.853722 18892 ts_tablet_manager.cc:1403] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.046s	user 0.016s	sys 0.004s
I20260709 12:30:43.856751 18892 raft_consensus.cc:359] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:43.857451 18892 raft_consensus.cc:385] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:43.857641 18895 leader_election.cc:290] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:43.857736 18892 raft_consensus.cc:740] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:43.884903 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acf997d23744fe697db1fe947dca883" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:43.885677 18840 raft_consensus.cc:2468] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:43.890704 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acf997d23744fe697db1fe947dca883" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
W20260709 12:30:43.893399 18731 leader_election.cc:343] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:43.899539 18902 raft_consensus.cc:493] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:43.900118 18902 raft_consensus.cc:515] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:43.903465 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e989970204ce48b700c86d5faf91d" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:43.904138 18840 raft_consensus.cc:2468] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 0.
I20260709 12:30:43.905436 18581 leader_election.cc:304] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:43.906296 18975 raft_consensus.cc:2804] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:43.906694 18975 raft_consensus.cc:493] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:43.907037 18975 raft_consensus.cc:3060] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:43.910318 18730 leader_election.cc:304] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:43.912081 18926 raft_consensus.cc:2804] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:43.912565 18926 raft_consensus.cc:493] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:43.912918 18926 raft_consensus.cc:3060] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:43.915452 18975 raft_consensus.cc:515] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:43.918515 18888 tablet_bootstrap.cc:654] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:43.919219 18892 consensus_queue.cc:260] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:43.922372 18892 ts_tablet_manager.cc:1434] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.068s	user 0.007s	sys 0.000s
I20260709 12:30:43.922526 18902 leader_election.cc:290] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:43.923838 18975 leader_election.cc:290] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:43.924103 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e989970204ce48b700c86d5faf91d" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:43.924526 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e989970204ce48b700c86d5faf91d" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:43.933262 18840 raft_consensus.cc:3060] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:43.925743 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e989970204ce48b700c86d5faf91d" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:43.932317 18892 tablet_bootstrap.cc:492] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:30:43.941246 18582 leader_election.cc:343] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:30:43.947729 18582 leader_election.cc:343] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:43.948302 18892 tablet_bootstrap.cc:654] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:43.957612 18840 raft_consensus.cc:2468] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 1.
I20260709 12:30:43.958463 18887 tablet_bootstrap.cc:492] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:43.959561 18581 leader_election.cc:304] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:43.960289 18887 ts_tablet_manager.cc:1403] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.150s	user 0.015s	sys 0.013s
I20260709 12:30:43.963567 18886 tablet_bootstrap.cc:492] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:43.960670 18975 raft_consensus.cc:2804] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:43.964191 18886 ts_tablet_manager.cc:1403] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.199s	user 0.013s	sys 0.021s
I20260709 12:30:43.964255 18975 raft_consensus.cc:697] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:30:43.966104 18975 consensus_queue.cc:237] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:43.968165 18886 raft_consensus.cc:359] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:43.969763 18886 raft_consensus.cc:385] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:43.970121 18886 raft_consensus.cc:740] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:43.971104 18902 consensus_queue.cc:1048] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:43.975692 18926 raft_consensus.cc:515] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:43.978817 18926 leader_election.cc:290] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:43.979095 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acf997d23744fe697db1fe947dca883" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:43.971169 18886 consensus_queue.cc:260] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:43.981560 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acf997d23744fe697db1fe947dca883" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:43.982030 18691 raft_consensus.cc:3060] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:43.982297 18841 raft_consensus.cc:3060] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:43.984431 18892 tablet_bootstrap.cc:492] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:43.985111 18892 ts_tablet_manager.cc:1403] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.053s	user 0.015s	sys 0.006s
I20260709 12:30:43.963701 18887 raft_consensus.cc:359] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:43.991830 18887 raft_consensus.cc:385] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:43.992216 18887 raft_consensus.cc:740] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:43.993213 18887 consensus_queue.cc:260] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:43.995335 18691 raft_consensus.cc:2468] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:43.998298 18731 leader_election.cc:304] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:43.999329 18926 raft_consensus.cc:2804] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:43.999847 18926 raft_consensus.cc:697] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:44.000702 18926 consensus_queue.cc:237] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.001374 18886 ts_tablet_manager.cc:1434] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.037s	user 0.008s	sys 0.002s
I20260709 12:30:44.002615 18886 tablet_bootstrap.cc:492] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:44.003563 18892 raft_consensus.cc:359] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.004323 18892 raft_consensus.cc:385] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.004873 18892 raft_consensus.cc:740] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:43.994025 18841 raft_consensus.cc:2468] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:44.010442 18887 ts_tablet_manager.cc:1434] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.049s	user 0.006s	sys 0.000s
I20260709 12:30:44.011571 18691 raft_consensus.cc:3060] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.012084 18887 tablet_bootstrap.cc:492] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:44.018556 18888 tablet_bootstrap.cc:492] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:44.018793 18887 tablet_bootstrap.cc:654] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.019158 18888 ts_tablet_manager.cc:1403] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.200s	user 0.023s	sys 0.005s
I20260709 12:30:44.005642 18892 consensus_queue.cc:260] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.021575 18886 tablet_bootstrap.cc:654] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.023931 18892 ts_tablet_manager.cc:1434] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.038s	user 0.004s	sys 0.003s
I20260709 12:30:44.026182 18892 tablet_bootstrap.cc:492] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:44.029213 18888 raft_consensus.cc:359] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.037884 18957 consensus_queue.cc:1048] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:44.037604 18513 catalog_manager.cc:5697] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:44.030038 18888 raft_consensus.cc:385] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.041119 18888 raft_consensus.cc:740] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.043084 18888 consensus_queue.cc:260] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.047612 18888 ts_tablet_manager.cc:1434] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.004s
I20260709 12:30:44.049196 18888 tablet_bootstrap.cc:492] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:44.056600 18514 catalog_manager.cc:5697] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 1 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:44.061494 18888 tablet_bootstrap.cc:654] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.076647 18887 tablet_bootstrap.cc:492] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:44.077214 18887 ts_tablet_manager.cc:1403] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.065s	user 0.015s	sys 0.018s
I20260709 12:30:44.077989 18886 tablet_bootstrap.cc:492] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:44.079051 18886 ts_tablet_manager.cc:1403] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.077s	user 0.024s	sys 0.012s
I20260709 12:30:44.080428 18887 raft_consensus.cc:359] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.080956 18617 raft_consensus.cc:3060] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.081236 18887 raft_consensus.cc:385] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.081617 18887 raft_consensus.cc:740] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.082137 18886 raft_consensus.cc:359] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.082409 18887 consensus_queue.cc:260] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.083081 18886 raft_consensus.cc:385] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.083420 18886 raft_consensus.cc:740] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.084229 18886 consensus_queue.cc:260] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.086798 18886 ts_tablet_manager.cc:1434] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:30:44.087901 18886 tablet_bootstrap.cc:492] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:44.094431 18887 ts_tablet_manager.cc:1434] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 12:30:44.095638 18886 tablet_bootstrap.cc:654] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.102689 18892 tablet_bootstrap.cc:654] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.112010 18887 tablet_bootstrap.cc:492] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:44.117100 18887 tablet_bootstrap.cc:654] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.117733 18886 tablet_bootstrap.cc:492] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:44.118124 18888 tablet_bootstrap.cc:492] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:44.118343 18886 ts_tablet_manager.cc:1403] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.031s	user 0.007s	sys 0.017s
I20260709 12:30:44.118661 18888 ts_tablet_manager.cc:1403] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.070s	user 0.025s	sys 0.004s
I20260709 12:30:44.121809 18888 raft_consensus.cc:359] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.122453 18888 raft_consensus.cc:385] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.122726 18888 raft_consensus.cc:740] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.123407 18888 consensus_queue.cc:260] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.125777 18888 ts_tablet_manager.cc:1434] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:30:44.127748 18886 raft_consensus.cc:359] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.128473 18886 raft_consensus.cc:385] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.128760 18886 raft_consensus.cc:740] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.129558 18886 consensus_queue.cc:260] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.134407 18888 tablet_bootstrap.cc:492] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:44.139864 18892 tablet_bootstrap.cc:492] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:44.140590 18892 ts_tablet_manager.cc:1403] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.115s	user 0.017s	sys 0.006s
I20260709 12:30:44.143633 18888 tablet_bootstrap.cc:654] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.147116 18892 raft_consensus.cc:359] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.147977 18892 raft_consensus.cc:385] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.148283 18892 raft_consensus.cc:740] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.149194 18892 consensus_queue.cc:260] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.151676 18892 ts_tablet_manager.cc:1434] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.001s
I20260709 12:30:44.151927 18887 tablet_bootstrap.cc:492] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:44.153899 18887 ts_tablet_manager.cc:1403] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.045s	user 0.024s	sys 0.011s
I20260709 12:30:44.156675 18892 tablet_bootstrap.cc:492] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:44.153241 18886 ts_tablet_manager.cc:1434] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.034s	user 0.004s	sys 0.003s
I20260709 12:30:44.158584 18886 tablet_bootstrap.cc:492] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:44.159889 18888 tablet_bootstrap.cc:492] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:44.160495 18888 ts_tablet_manager.cc:1403] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.034s	user 0.015s	sys 0.004s
I20260709 12:30:44.163503 18888 raft_consensus.cc:359] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.164163 18888 raft_consensus.cc:385] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.164539 18886 tablet_bootstrap.cc:654] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.164485 18888 raft_consensus.cc:740] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.165526 18888 consensus_queue.cc:260] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.167685 18887 raft_consensus.cc:359] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.168406 18887 raft_consensus.cc:385] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.168689 18887 raft_consensus.cc:740] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.169039 18892 tablet_bootstrap.cc:654] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.169400 18887 consensus_queue.cc:260] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.171911 18887 ts_tablet_manager.cc:1434] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.000s
I20260709 12:30:44.173197 18887 tablet_bootstrap.cc:492] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:44.180171 18887 tablet_bootstrap.cc:654] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.183313 18886 tablet_bootstrap.cc:492] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:44.184085 18886 ts_tablet_manager.cc:1403] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.026s	user 0.006s	sys 0.006s
I20260709 12:30:44.188091 18886 raft_consensus.cc:359] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.189167 18886 raft_consensus.cc:385] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.189594 18886 raft_consensus.cc:740] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.190745 18886 consensus_queue.cc:260] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.191519 18887 tablet_bootstrap.cc:492] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:44.192454 18887 ts_tablet_manager.cc:1403] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.020s	user 0.004s	sys 0.013s
I20260709 12:30:44.195382 18887 raft_consensus.cc:359] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.196029 18887 raft_consensus.cc:385] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.196280 18887 raft_consensus.cc:740] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.197021 18887 consensus_queue.cc:260] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.202899 18887 ts_tablet_manager.cc:1434] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260709 12:30:44.203940 18887 tablet_bootstrap.cc:492] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:44.204228 18888 ts_tablet_manager.cc:1434] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.043s	user 0.007s	sys 0.000s
I20260709 12:30:44.204790 18886 ts_tablet_manager.cc:1434] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.020s	user 0.009s	sys 0.000s
I20260709 12:30:44.205322 18888 tablet_bootstrap.cc:492] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:44.206171 18886 tablet_bootstrap.cc:492] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:44.210189 18887 tablet_bootstrap.cc:654] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.213045 18886 tablet_bootstrap.cc:654] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:44.215693 18563 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:30:44.226541 18888 tablet_bootstrap.cc:654] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.226742 18887 tablet_bootstrap.cc:492] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:44.228395 18887 ts_tablet_manager.cc:1403] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.025s	user 0.010s	sys 0.008s
I20260709 12:30:44.229514 18892 tablet_bootstrap.cc:492] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:44.231297 18949 raft_consensus.cc:493] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:44.235062 18892 ts_tablet_manager.cc:1403] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.082s	user 0.030s	sys 0.023s
I20260709 12:30:44.235232 18886 tablet_bootstrap.cc:492] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:44.235088 18949 raft_consensus.cc:515] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.235922 18888 tablet_bootstrap.cc:492] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:44.232208 18887 raft_consensus.cc:359] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.236130 18886 ts_tablet_manager.cc:1403] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.030s	user 0.014s	sys 0.004s
I20260709 12:30:44.236451 18888 ts_tablet_manager.cc:1403] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.031s	user 0.009s	sys 0.005s
I20260709 12:30:44.238060 18949 leader_election.cc:290] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:44.236537 18887 raft_consensus.cc:385] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.239660 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73eeb703f09a4330ad726c1d5e42eee3" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:44.239784 18887 raft_consensus.cc:740] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.240127 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73eeb703f09a4330ad726c1d5e42eee3" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:44.240413 18764 raft_consensus.cc:2468] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:44.240589 18886 raft_consensus.cc:359] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.240803 18887 consensus_queue.cc:260] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.241384 18888 raft_consensus.cc:359] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.242287 18888 raft_consensus.cc:385] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.242548 18888 raft_consensus.cc:740] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.242591 18805 leader_election.cc:304] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:44.241369 18886 raft_consensus.cc:385] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.238866 18892 raft_consensus.cc:359] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.243278 18888 consensus_queue.cc:260] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.243960 18892 raft_consensus.cc:385] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.243783 18949 raft_consensus.cc:2804] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:44.243794 18886 raft_consensus.cc:740] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.245078 18887 ts_tablet_manager.cc:1434] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 12:30:44.247120 18887 tablet_bootstrap.cc:492] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:44.246709 18886 consensus_queue.cc:260] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.250103 18886 ts_tablet_manager.cc:1434] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 12:30:44.251820 18886 tablet_bootstrap.cc:492] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:44.255566 18888 ts_tablet_manager.cc:1434] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.002s
I20260709 12:30:44.261094 18886 tablet_bootstrap.cc:654] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.244822 18949 raft_consensus.cc:493] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:44.264253 18949 raft_consensus.cc:3060] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.244556 18892 raft_consensus.cc:740] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.266300 18892 consensus_queue.cc:260] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.267349 18617 raft_consensus.cc:2468] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:44.269614 18888 tablet_bootstrap.cc:492] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:44.272758 18949 raft_consensus.cc:515] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.276129 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73eeb703f09a4330ad726c1d5e42eee3" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:44.276753 18764 raft_consensus.cc:3060] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.277896 18888 tablet_bootstrap.cc:654] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.279476 18892 ts_tablet_manager.cc:1434] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.044s	user 0.007s	sys 0.000s
I20260709 12:30:44.291227 18764 raft_consensus.cc:2468] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:44.286983 18887 tablet_bootstrap.cc:654] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.289589 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73eeb703f09a4330ad726c1d5e42eee3" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:44.293109 18805 leader_election.cc:304] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:44.296514 18949 leader_election.cc:290] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:44.297024 18906 raft_consensus.cc:2804] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:44.297843 18906 raft_consensus.cc:697] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:44.299247 18906 consensus_queue.cc:237] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.293545 18617 raft_consensus.cc:3060] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.303967 18887 tablet_bootstrap.cc:492] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:44.304517 18888 tablet_bootstrap.cc:492] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:44.304525 18887 ts_tablet_manager.cc:1403] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.058s	user 0.008s	sys 0.008s
I20260709 12:30:44.305235 18888 ts_tablet_manager.cc:1403] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.036s	user 0.013s	sys 0.019s
I20260709 12:30:44.307768 18887 raft_consensus.cc:359] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.307935 18888 raft_consensus.cc:359] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.308539 18887 raft_consensus.cc:385] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.308696 18888 raft_consensus.cc:385] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.309063 18888 raft_consensus.cc:740] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.308924 18887 raft_consensus.cc:740] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.309930 18888 consensus_queue.cc:260] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.291647 18892 tablet_bootstrap.cc:492] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:44.312160 18888 ts_tablet_manager.cc:1434] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:30:44.313722 18888 tablet_bootstrap.cc:492] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:44.319756 18888 tablet_bootstrap.cc:654] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.319794 18886 tablet_bootstrap.cc:492] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:44.320667 18886 ts_tablet_manager.cc:1403] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.069s	user 0.027s	sys 0.026s
I20260709 12:30:44.323736 18886 raft_consensus.cc:359] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.324450 18886 raft_consensus.cc:385] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.324659 18975 raft_consensus.cc:493] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:44.325314 18975 raft_consensus.cc:515] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.327575 18975 leader_election.cc:290] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:44.328581 18617 raft_consensus.cc:2468] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:44.330085 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:44.330509 18892 tablet_bootstrap.cc:654] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.330946 18906 raft_consensus.cc:493] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:44.331583 18906 raft_consensus.cc:515] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.330888 18764 raft_consensus.cc:2468] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 0.
I20260709 12:30:44.332762 18516 catalog_manager.cc:5697] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:44.334059 18582 leader_election.cc:304] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:30:44.335206 18957 raft_consensus.cc:493] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:44.335721 18957 raft_consensus.cc:515] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.336808 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d163b2d60ca4572bca7a1252c65277a" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:44.337962 18957 leader_election.cc:290] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:44.337841 18691 raft_consensus.cc:2468] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:44.338620 18957 raft_consensus.cc:2804] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:44.339003 18957 raft_consensus.cc:493] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:44.339341 18957 raft_consensus.cc:3060] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.340811 18887 consensus_queue.cc:260] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.341619 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "872ec24b41d24573a9a8cc1d484fd6f9" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:44.342247 18840 raft_consensus.cc:2468] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 0.
I20260709 12:30:44.343039 18887 ts_tablet_manager.cc:1434] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.038s	user 0.000s	sys 0.004s
I20260709 12:30:44.343926 18581 leader_election.cc:304] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:44.344548 18887 tablet_bootstrap.cc:492] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:44.344853 18902 raft_consensus.cc:2804] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:44.345217 18902 raft_consensus.cc:493] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:44.345553 18902 raft_consensus.cc:3060] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.349740 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "872ec24b41d24573a9a8cc1d484fd6f9" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:44.351540 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d163b2d60ca4572bca7a1252c65277a" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:44.352402 18764 raft_consensus.cc:2468] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:44.353597 18805 leader_election.cc:304] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:44.332123 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:44.355317 18693 raft_consensus.cc:2468] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 0.
I20260709 12:30:44.356585 18949 raft_consensus.cc:2804] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:44.357160 18949 raft_consensus.cc:493] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:44.357275 18902 raft_consensus.cc:515] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.357697 18949 raft_consensus.cc:3060] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.360052 18887 tablet_bootstrap.cc:654] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.360209 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "872ec24b41d24573a9a8cc1d484fd6f9" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:44.360819 18840 raft_consensus.cc:3060] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.324734 18886 raft_consensus.cc:740] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.372964 18975 raft_consensus.cc:493] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:44.334051 18906 leader_election.cc:290] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:44.373879 18886 consensus_queue.cc:260] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.373919 18975 raft_consensus.cc:515] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.378131 18975 leader_election.cc:290] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:44.378916 18691 raft_consensus.cc:2468] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 0.
I20260709 12:30:44.382068 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71d9e90994b4dcfa86f9bb9e3b45b51" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:44.382772 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71d9e90994b4dcfa86f9bb9e3b45b51" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:44.382953 18693 raft_consensus.cc:2468] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 0.
I20260709 12:30:44.383473 18764 raft_consensus.cc:2468] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 0.
I20260709 12:30:44.385008 18582 leader_election.cc:304] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:30:44.388778 18975 raft_consensus.cc:2804] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:44.389309 18975 raft_consensus.cc:493] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:44.389757 18975 raft_consensus.cc:3060] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.397210 18902 leader_election.cc:290] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:44.376322 18949 raft_consensus.cc:515] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.400884 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d163b2d60ca4572bca7a1252c65277a" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:44.401748 18691 raft_consensus.cc:3060] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.403302 18886 ts_tablet_manager.cc:1434] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.082s	user 0.007s	sys 0.001s
I20260709 12:30:44.407280 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d163b2d60ca4572bca7a1252c65277a" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:44.408176 18764 raft_consensus.cc:3060] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.411679 18888 tablet_bootstrap.cc:492] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:44.412846 18888 ts_tablet_manager.cc:1403] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.099s	user 0.015s	sys 0.004s
I20260709 12:30:44.423930 18981 consensus_queue.cc:1048] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:44.417248 18949 leader_election.cc:290] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:44.419318 18957 raft_consensus.cc:515] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
W20260709 12:30:44.428597 18582 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:40335, user_credentials={real_user=slave}} blocked reactor thread for 43668.2us
I20260709 12:30:44.440915 18886 tablet_bootstrap.cc:492] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:44.452395 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:44.453100 18692 raft_consensus.cc:3060] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.451083 18892 tablet_bootstrap.cc:492] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:44.455161 18892 ts_tablet_manager.cc:1403] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.164s	user 0.045s	sys 0.056s
I20260709 12:30:44.458534 18982 consensus_queue.cc:1048] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:44.451539 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "872ec24b41d24573a9a8cc1d484fd6f9" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:44.451650 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:44.459581 18693 raft_consensus.cc:3060] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.459649 18765 raft_consensus.cc:3060] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.466145 18692 raft_consensus.cc:2468] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 1.
I20260709 12:30:44.468123 18765 raft_consensus.cc:2468] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 1.
I20260709 12:30:44.471626 18693 raft_consensus.cc:2468] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 1.
I20260709 12:30:44.472975 18582 leader_election.cc:304] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:30:44.480116 18975 raft_consensus.cc:515] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.481040 19021 raft_consensus.cc:2804] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:44.481621 19021 raft_consensus.cc:697] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:30:44.475315 18892 raft_consensus.cc:359] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.420495 18840 raft_consensus.cc:2468] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 1.
I20260709 12:30:44.437989 18691 raft_consensus.cc:2468] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:44.478595 18764 raft_consensus.cc:2468] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:44.479132 18887 tablet_bootstrap.cc:492] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:44.445844 18957 leader_election.cc:290] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:44.482679 18892 raft_consensus.cc:385] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.483968 18887 ts_tablet_manager.cc:1403] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.140s	user 0.043s	sys 0.044s
I20260709 12:30:44.486617 18892 raft_consensus.cc:740] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.480628 18582 leader_election.cc:304] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:30:44.485087 18805 leader_election.cc:304] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:44.488852 18906 raft_consensus.cc:2804] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:44.488708 18892 consensus_queue.cc:260] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.489638 18906 raft_consensus.cc:697] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:44.490732 18906 consensus_queue.cc:237] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.491514 18892 ts_tablet_manager.cc:1434] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.036s	user 0.005s	sys 0.003s
I20260709 12:30:44.486441 18888 raft_consensus.cc:359] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.492226 18888 raft_consensus.cc:385] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.492605 18888 raft_consensus.cc:740] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.482359 18975 leader_election.cc:290] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:44.493512 19021 consensus_queue.cc:237] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.493743 18888 consensus_queue.cc:260] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.495309 18887 raft_consensus.cc:359] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.501014 18887 raft_consensus.cc:385] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.496501 18886 tablet_bootstrap.cc:654] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.501348 18887 raft_consensus.cc:740] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.503650 18888 ts_tablet_manager.cc:1434] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.062s	user 0.008s	sys 0.000s
I20260709 12:30:44.505172 18892 tablet_bootstrap.cc:492] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:44.505035 18887 consensus_queue.cc:260] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.504838 18972 raft_consensus.cc:493] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:44.507036 18972 raft_consensus.cc:515] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.502053 18902 raft_consensus.cc:2804] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:44.509099 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71d9e90994b4dcfa86f9bb9e3b45b51" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:44.509819 18691 raft_consensus.cc:3060] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.510551 18888 tablet_bootstrap.cc:492] T 089c667ac1ae4a22b9b2e484f978c5d4 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:44.512679 18887 ts_tablet_manager.cc:1434] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.000s
I20260709 12:30:44.509218 18902 raft_consensus.cc:697] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:30:44.517557 18972 leader_election.cc:290] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:44.518534 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "089c667ac1ae4a22b9b2e484f978c5d4" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:44.520174 18805 leader_election.cc:343] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:44.521292 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "089c667ac1ae4a22b9b2e484f978c5d4" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:44.522015 18617 raft_consensus.cc:2468] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:44.524327 18888 tablet_bootstrap.cc:654] T 089c667ac1ae4a22b9b2e484f978c5d4 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.525956 18887 tablet_bootstrap.cc:492] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:44.528997 18804 leader_election.cc:304] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:30:44.531298 18906 raft_consensus.cc:2804] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:44.531855 18906 raft_consensus.cc:493] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:44.532230 18906 raft_consensus.cc:3060] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.534005 18965 raft_consensus.cc:493] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:44.534535 18965 raft_consensus.cc:515] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.537206 18965 leader_election.cc:290] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:44.537731 18886 tablet_bootstrap.cc:492] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:44.537680 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ade62c22cf46b3bf97770f5bc7d597" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:44.538379 18886 ts_tablet_manager.cc:1403] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.098s	user 0.012s	sys 0.001s
I20260709 12:30:44.538532 18765 raft_consensus.cc:2468] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 0.
I20260709 12:30:44.539184 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ade62c22cf46b3bf97770f5bc7d597" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:44.539697 18657 leader_election.cc:304] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:44.539873 18840 raft_consensus.cc:2468] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 0.
I20260709 12:30:44.541597 18965 raft_consensus.cc:2804] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:44.541985 18965 raft_consensus.cc:493] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:44.541285 18515 catalog_manager.cc:5697] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:44.542331 18965 raft_consensus.cc:3060] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.545863 18693 raft_consensus.cc:3060] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.548226 18902 consensus_queue.cc:237] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.546837 18886 raft_consensus.cc:359] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.553161 18886 raft_consensus.cc:385] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.553572 18886 raft_consensus.cc:740] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.528445 18691 raft_consensus.cc:2468] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 1.
I20260709 12:30:44.560783 18582 leader_election.cc:304] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:30:44.561802 18957 raft_consensus.cc:2804] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:44.561908 18886 consensus_queue.cc:260] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.562295 18906 raft_consensus.cc:515] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.562610 18957 raft_consensus.cc:697] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:30:44.563723 18957 consensus_queue.cc:237] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.569902 18887 tablet_bootstrap.cc:654] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.582641 18886 ts_tablet_manager.cc:1434] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.044s	user 0.007s	sys 0.004s
I20260709 12:30:44.586722 18892 tablet_bootstrap.cc:654] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.590570 18965 raft_consensus.cc:515] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.592530 18949 raft_consensus.cc:493] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:44.593107 18949 raft_consensus.cc:515] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.593709 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ade62c22cf46b3bf97770f5bc7d597" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:44.594290 18765 raft_consensus.cc:3060] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.595309 18949 leader_election.cc:290] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:44.597747 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7b389579627488bb406ec0d112f6938" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:44.582864 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71d9e90994b4dcfa86f9bb9e3b45b51" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:44.599838 18764 raft_consensus.cc:3060] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.601140 18981 raft_consensus.cc:493] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:44.601632 18981 raft_consensus.cc:515] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.604777 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c180f8ddcbe14366a60537e1bad39332" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:44.604872 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c180f8ddcbe14366a60537e1bad39332" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
W20260709 12:30:44.606613 18731 leader_election.cc:343] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:30:44.607240 18730 leader_election.cc:343] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:44.607679 18730 leader_election.cc:304] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb
I20260709 12:30:44.608176 18957 consensus_queue.cc:1048] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:44.608584 18982 raft_consensus.cc:2749] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:30:44.607730 18804 leader_election.cc:343] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:44.619731 18886 tablet_bootstrap.cc:492] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:44.613469 18981 leader_election.cc:290] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:44.619637 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7b389579627488bb406ec0d112f6938" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:44.619422 18957 consensus_queue.cc:1048] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:30:44.636699 18906 leader_election.cc:290] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:44.633821 18965 leader_election.cc:290] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:44.637621 18765 raft_consensus.cc:2468] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
W20260709 12:30:44.640758 18805 leader_election.cc:343] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:44.640900 18764 raft_consensus.cc:2468] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 1.
I20260709 12:30:44.641292 18805 leader_election.cc:304] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb
I20260709 12:30:44.642585 18906 raft_consensus.cc:2749] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:44.642800 18965 raft_consensus.cc:493] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:44.651543 18515 catalog_manager.cc:5697] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 1 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:44.643909 18965 raft_consensus.cc:515] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.662963 18888 tablet_bootstrap.cc:492] T 089c667ac1ae4a22b9b2e484f978c5d4 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:44.663823 18888 ts_tablet_manager.cc:1403] T 089c667ac1ae4a22b9b2e484f978c5d4 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.154s	user 0.028s	sys 0.024s
I20260709 12:30:44.651047 18657 leader_election.cc:304] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:44.670311 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ade62c22cf46b3bf97770f5bc7d597" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:44.670678 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "089c667ac1ae4a22b9b2e484f978c5d4" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:44.671157 18840 raft_consensus.cc:3060] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.671753 18617 raft_consensus.cc:3060] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.675494 18888 raft_consensus.cc:359] T 089c667ac1ae4a22b9b2e484f978c5d4 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.676196 18888 raft_consensus.cc:385] T 089c667ac1ae4a22b9b2e484f978c5d4 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.676477 18888 raft_consensus.cc:740] T 089c667ac1ae4a22b9b2e484f978c5d4 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.668061 19030 raft_consensus.cc:2804] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:44.680224 19030 raft_consensus.cc:697] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:30:44.681497 19030 consensus_queue.cc:237] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.685372 18888 consensus_queue.cc:260] T 089c667ac1ae4a22b9b2e484f978c5d4 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.687003 19028 raft_consensus.cc:493] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:44.692179 19028 raft_consensus.cc:515] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.692636 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "089c667ac1ae4a22b9b2e484f978c5d4" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:44.719791 18887 tablet_bootstrap.cc:492] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:44.697399 18839 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e989970204ce48b700c86d5faf91d" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:44.720625 18887 ts_tablet_manager.cc:1403] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.195s	user 0.026s	sys 0.009s
I20260709 12:30:44.723093 18888 ts_tablet_manager.cc:1434] T 089c667ac1ae4a22b9b2e484f978c5d4 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.059s	user 0.006s	sys 0.006s
I20260709 12:30:44.723554 18764 raft_consensus.cc:3060] T 089c667ac1ae4a22b9b2e484f978c5d4 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.726015 18892 tablet_bootstrap.cc:492] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:44.726745 18892 ts_tablet_manager.cc:1403] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.224s	user 0.018s	sys 0.046s
I20260709 12:30:44.707123 18886 tablet_bootstrap.cc:654] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.730748 18892 raft_consensus.cc:359] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.731925 18892 raft_consensus.cc:385] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.735605 18965 leader_election.cc:290] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:44.737859 18888 tablet_bootstrap.cc:492] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:44.732285 18892 raft_consensus.cc:740] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.739779 19028 leader_election.cc:290] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:44.756445 18840 raft_consensus.cc:2468] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:44.749963 18892 consensus_queue.cc:260] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:44.761336 18906 consensus_queue.cc:1048] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:44.739976 18764 raft_consensus.cc:2468] T 089c667ac1ae4a22b9b2e484f978c5d4 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:44.768028 18892 ts_tablet_manager.cc:1434] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.041s	user 0.011s	sys 0.000s
I20260709 12:30:44.769259 18892 tablet_bootstrap.cc:492] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:44.772912 18887 raft_consensus.cc:359] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.773744 18887 raft_consensus.cc:385] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.774009 18887 raft_consensus.cc:740] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.775022 18887 consensus_queue.cc:260] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.780469 18805 leader_election.cc:304] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:44.781096 18887 ts_tablet_manager.cc:1434] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.060s	user 0.006s	sys 0.003s
I20260709 12:30:44.781773 18906 raft_consensus.cc:2804] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:44.782368 18906 raft_consensus.cc:697] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:44.789558 18886 tablet_bootstrap.cc:492] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:44.789505 18906 consensus_queue.cc:237] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.790194 18886 ts_tablet_manager.cc:1403] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.179s	user 0.011s	sys 0.005s
I20260709 12:30:44.793501 18886 raft_consensus.cc:359] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.794224 18886 raft_consensus.cc:385] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.794512 18886 raft_consensus.cc:740] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.795150 18886 consensus_queue.cc:260] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.797546 18886 ts_tablet_manager.cc:1434] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:30:44.798465 18886 tablet_bootstrap.cc:492] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:44.803138 18617 raft_consensus.cc:2468] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:44.805723 18886 tablet_bootstrap.cc:654] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.808851 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee74ad2fe774e0aa780b1fdf18b3003" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:44.809607 18841 raft_consensus.cc:2468] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 0.
I20260709 12:30:44.813361 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e989970204ce48b700c86d5faf91d" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:44.813926 18516 catalog_manager.cc:5697] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 1 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:44.817584 18656 leader_election.cc:304] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:44.815595 18516 catalog_manager.cc:5697] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 1 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:44.818902 19030 raft_consensus.cc:2749] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:44.819326 18656 leader_election.cc:304] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:44.820199 19030 raft_consensus.cc:2804] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:44.820724 19030 raft_consensus.cc:493] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:44.821146 19030 raft_consensus.cc:3060] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.825052 18888 tablet_bootstrap.cc:654] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.830283 18887 tablet_bootstrap.cc:492] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:44.829470 18515 catalog_manager.cc:5697] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:44.841866 18892 tablet_bootstrap.cc:654] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.842664 19030 raft_consensus.cc:515] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.838212 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee74ad2fe774e0aa780b1fdf18b3003" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:44.847594 19030 leader_election.cc:290] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:44.850919 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee74ad2fe774e0aa780b1fdf18b3003" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:44.851634 18841 raft_consensus.cc:3060] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.853211 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee74ad2fe774e0aa780b1fdf18b3003" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
W20260709 12:30:44.855397 18656 leader_election.cc:343] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:44.856002 18886 tablet_bootstrap.cc:492] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
W20260709 12:30:44.856503 18656 leader_election.cc:343] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:44.857340 18887 tablet_bootstrap.cc:654] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.856534 18886 ts_tablet_manager.cc:1403] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.058s	user 0.019s	sys 0.003s
I20260709 12:30:44.871352 18841 raft_consensus.cc:2468] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:44.871717 18972 consensus_queue.cc:1048] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:44.872579 18892 tablet_bootstrap.cc:492] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:44.881301 18892 ts_tablet_manager.cc:1403] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.112s	user 0.012s	sys 0.003s
I20260709 12:30:44.875649 18888 tablet_bootstrap.cc:492] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:44.878029 18887 tablet_bootstrap.cc:492] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:44.885941 18892 raft_consensus.cc:359] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.887971 18892 raft_consensus.cc:385] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.888373 18892 raft_consensus.cc:740] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.889597 18892 consensus_queue.cc:260] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.886444 18888 ts_tablet_manager.cc:1403] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.149s	user 0.013s	sys 0.007s
I20260709 12:30:44.886915 18887 ts_tablet_manager.cc:1403] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.100s	user 0.009s	sys 0.007s
I20260709 12:30:44.897186 18887 raft_consensus.cc:359] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.898172 18887 raft_consensus.cc:385] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.898526 18887 raft_consensus.cc:740] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.903010 18888 raft_consensus.cc:359] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.905234 18887 consensus_queue.cc:260] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:44.908291 18887 ts_tablet_manager.cc:1434] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.001s
I20260709 12:30:44.909768 18887 tablet_bootstrap.cc:492] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:44.910243 18656 leader_election.cc:304] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:44.911425 19030 raft_consensus.cc:2804] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:44.911947 19030 raft_consensus.cc:697] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:30:44.904107 18888 raft_consensus.cc:385] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:44.912621 18888 raft_consensus.cc:740] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:44.912827 19030 consensus_queue.cc:237] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.913825 18888 consensus_queue.cc:260] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.916868 18888 ts_tablet_manager.cc:1434] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.017s	user 0.009s	sys 0.000s
I20260709 12:30:44.918260 18888 tablet_bootstrap.cc:492] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:44.920634 18892 ts_tablet_manager.cc:1434] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.039s	user 0.010s	sys 0.000s
I20260709 12:30:44.920945 18972 consensus_queue.cc:1048] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:44.922236 18892 tablet_bootstrap.cc:492] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:44.931949 18892 tablet_bootstrap.cc:654] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.936084 18902 consensus_queue.cc:1048] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:44.936888 18887 tablet_bootstrap.cc:654] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.941545 18902 raft_consensus.cc:493] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:44.942214 18902 raft_consensus.cc:515] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:44.946599 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7b389579627488bb406ec0d112f6938" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:44.946338 18514 catalog_manager.cc:5697] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:44.947587 18840 raft_consensus.cc:2468] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 0.
I20260709 12:30:44.948531 18972 raft_consensus.cc:493] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:44.949177 18581 leader_election.cc:304] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:44.949487 18972 raft_consensus.cc:515] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:44.950384 18957 raft_consensus.cc:2804] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:44.950839 18957 raft_consensus.cc:493] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:44.951207 18957 raft_consensus.cc:3060] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:44.952589 18888 tablet_bootstrap.cc:654] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:44.970161 19021 consensus_queue.cc:1048] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:44.984488 19021 consensus_queue.cc:1048] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:44.985427 18975 consensus_queue.cc:1048] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:44.994625 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03deb6b823ad408c8cdecc1fbd819940" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:44.995483 18766 raft_consensus.cc:2468] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:44.999248 18805 leader_election.cc:304] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:45.002995 18949 raft_consensus.cc:2804] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:45.002841 19023 consensus_queue.cc:1048] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:45.003469 18949 raft_consensus.cc:493] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:45.003876 18949 raft_consensus.cc:3060] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.010686 18902 leader_election.cc:290] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:45.011484 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7b389579627488bb406ec0d112f6938" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
W20260709 12:30:45.013260 18582 leader_election.cc:343] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:45.015455 19060 consensus_queue.cc:1048] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:45.021864 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03deb6b823ad408c8cdecc1fbd819940" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:45.025339 19038 raft_consensus.cc:493] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:45.026021 19038 raft_consensus.cc:515] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.027410 18949 raft_consensus.cc:515] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.030256 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e16042d49945ea80b31f9eee56f2cf" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:45.032299 19038 leader_election.cc:290] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
W20260709 12:30:45.033326 18804 leader_election.cc:343] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:44.878818 18886 raft_consensus.cc:359] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.075001 18886 raft_consensus.cc:385] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.020291 18957 raft_consensus.cc:515] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.038264 18972 leader_election.cc:290] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:45.042992 18887 tablet_bootstrap.cc:492] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:45.045343 18892 tablet_bootstrap.cc:492] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:45.052191 19023 consensus_queue.cc:1048] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:45.071888 18888 tablet_bootstrap.cc:492] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:45.078389 18892 ts_tablet_manager.cc:1403] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.156s	user 0.022s	sys 0.009s
I20260709 12:30:45.078469 18887 ts_tablet_manager.cc:1403] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.169s	user 0.012s	sys 0.020s
I20260709 12:30:45.078734 18888 ts_tablet_manager.cc:1403] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.161s	user 0.014s	sys 0.027s
I20260709 12:30:45.075382 18886 raft_consensus.cc:740] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.081152 18886 consensus_queue.cc:260] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
W20260709 12:30:45.083122 18804 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 49851.5us
I20260709 12:30:45.087522 18887 raft_consensus.cc:359] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.088307 18887 raft_consensus.cc:385] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.088606 18887 raft_consensus.cc:740] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.089370 18887 consensus_queue.cc:260] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.089672 18892 raft_consensus.cc:359] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.090416 18892 raft_consensus.cc:385] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:30:45.089673 18657 leader_election.cc:343] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:45.091434 18887 ts_tablet_manager.cc:1434] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260709 12:30:45.092502 18887 tablet_bootstrap.cc:492] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:45.090696 18892 raft_consensus.cc:740] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.093508 18892 consensus_queue.cc:260] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
W20260709 12:30:45.076941 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 68122.4us
I20260709 12:30:45.102278 18886 ts_tablet_manager.cc:1434] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.244s	user 0.007s	sys 0.001s
I20260709 12:30:45.105543 18892 ts_tablet_manager.cc:1434] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.027s	user 0.008s	sys 0.000s
I20260709 12:30:45.113862 18957 leader_election.cc:290] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:45.115144 18892 tablet_bootstrap.cc:492] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:45.120676 18886 tablet_bootstrap.cc:492] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:45.120781 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7b389579627488bb406ec0d112f6938" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:45.123725 18841 raft_consensus.cc:3060] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.121562 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7b389579627488bb406ec0d112f6938" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:45.122766 18888 raft_consensus.cc:359] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.124199 18892 tablet_bootstrap.cc:654] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.125059 18888 raft_consensus.cc:385] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.126053 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e16042d49945ea80b31f9eee56f2cf" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:45.126700 18840 raft_consensus.cc:2468] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 0.
W20260709 12:30:45.126875 18582 leader_election.cc:343] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:45.125648 18888 raft_consensus.cc:740] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.129330 18888 consensus_queue.cc:260] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.131819 18888 ts_tablet_manager.cc:1434] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.053s	user 0.007s	sys 0.000s
I20260709 12:30:45.131623 18515 catalog_manager.cc:5697] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:45.133147 18888 tablet_bootstrap.cc:492] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:45.133960 18656 leader_election.cc:304] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:30:45.134387 18887 tablet_bootstrap.cc:654] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.135403 18841 raft_consensus.cc:2468] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 1.
I20260709 12:30:45.135594 19030 raft_consensus.cc:2804] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:45.136005 19030 raft_consensus.cc:493] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:45.136369 19030 raft_consensus.cc:3060] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.136579 18581 leader_election.cc:304] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:45.139329 18957 raft_consensus.cc:2804] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:45.141212 18957 raft_consensus.cc:697] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:30:45.142480 18949 leader_election.cc:290] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:45.142302 18957 consensus_queue.cc:237] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.137918 19038 consensus_queue.cc:1048] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:30:45.145957 18886 tablet_bootstrap.cc:654] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.147547 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03deb6b823ad408c8cdecc1fbd819940" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:45.151935 18616 raft_consensus.cc:3060] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.147890 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03deb6b823ad408c8cdecc1fbd819940" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:45.185477 18764 raft_consensus.cc:3060] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.195071 19030 raft_consensus.cc:515] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.203703 18764 raft_consensus.cc:2468] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:45.204965 19038 consensus_queue.cc:1048] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:45.210371 19030 leader_election.cc:290] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:45.214300 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e16042d49945ea80b31f9eee56f2cf" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:45.216915 18886 tablet_bootstrap.cc:492] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:45.228099 18886 ts_tablet_manager.cc:1403] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.121s	user 0.012s	sys 0.005s
I20260709 12:30:45.200065 18616 raft_consensus.cc:2468] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:45.219360 18887 tablet_bootstrap.cc:492] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:45.231012 18886 raft_consensus.cc:359] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.231736 18886 raft_consensus.cc:385] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.231994 18886 raft_consensus.cc:740] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.217214 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e16042d49945ea80b31f9eee56f2cf" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:45.233227 18840 raft_consensus.cc:3060] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.232685 18887 ts_tablet_manager.cc:1403] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.140s	user 0.043s	sys 0.025s
I20260709 12:30:45.234067 18804 leader_election.cc:304] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
W20260709 12:30:45.233438 18657 leader_election.cc:343] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:45.236873 18886 consensus_queue.cc:260] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.237309 18887 raft_consensus.cc:359] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.238369 18887 raft_consensus.cc:385] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.238744 18887 raft_consensus.cc:740] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.240113 18972 raft_consensus.cc:2804] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:45.240828 18972 raft_consensus.cc:697] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:45.240020 18887 consensus_queue.cc:260] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.241778 18972 consensus_queue.cc:237] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.243616 18887 ts_tablet_manager.cc:1434] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:30:45.244830 18887 tablet_bootstrap.cc:492] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:45.245225 18957 raft_consensus.cc:493] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:45.245877 18957 raft_consensus.cc:515] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.248701 18957 leader_election.cc:290] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:45.240116 18886 ts_tablet_manager.cc:1434] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:30:45.250633 18886 tablet_bootstrap.cc:492] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:45.250820 18887 tablet_bootstrap.cc:654] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.254134 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c180f8ddcbe14366a60537e1bad39332" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:45.255126 18690 raft_consensus.cc:2468] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 0.
I20260709 12:30:45.257458 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c180f8ddcbe14366a60537e1bad39332" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:45.258440 18765 raft_consensus.cc:2468] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 0.
I20260709 12:30:45.259783 18840 raft_consensus.cc:2468] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:45.228124 18892 tablet_bootstrap.cc:492] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:45.261466 18582 leader_election.cc:304] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:30:45.271673 18656 leader_election.cc:304] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:30:45.271996 18957 raft_consensus.cc:2804] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:45.273154 19030 raft_consensus.cc:2804] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:45.273880 19030 raft_consensus.cc:697] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:30:45.274780 19030 consensus_queue.cc:237] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.275154 18892 ts_tablet_manager.cc:1403] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.160s	user 0.009s	sys 0.006s
I20260709 12:30:45.262038 18888 tablet_bootstrap.cc:654] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.267576 18887 tablet_bootstrap.cc:492] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:45.278199 18887 ts_tablet_manager.cc:1403] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.034s	user 0.016s	sys 0.001s
I20260709 12:30:45.269061 18515 catalog_manager.cc:5697] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 1 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:45.265611 18972 raft_consensus.cc:493] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:45.280555 18972 raft_consensus.cc:515] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.281509 18887 raft_consensus.cc:359] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.282392 18887 raft_consensus.cc:385] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.282686 18887 raft_consensus.cc:740] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.283479 18887 consensus_queue.cc:260] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.286439 18887 ts_tablet_manager.cc:1434] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:30:45.287680 18972 leader_election.cc:290] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:45.272749 18957 raft_consensus.cc:493] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:45.290370 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c5cc1ca1ea8468199e1eeddb8c10756" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
W20260709 12:30:45.291988 18563 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:30:45.275233 18886 tablet_bootstrap.cc:654] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:45.295717 18805 leader_election.cc:343] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:45.296870 18888 tablet_bootstrap.cc:492] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:45.297390 18888 ts_tablet_manager.cc:1403] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.164s	user 0.013s	sys 0.000s
I20260709 12:30:45.297427 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c5cc1ca1ea8468199e1eeddb8c10756" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:45.292189 18887 tablet_bootstrap.cc:492] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:45.293891 18892 raft_consensus.cc:359] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.298799 18892 raft_consensus.cc:385] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.299095 18892 raft_consensus.cc:740] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.299814 18892 consensus_queue.cc:260] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.302006 18957 raft_consensus.cc:3060] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.302281 18892 ts_tablet_manager.cc:1434] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.026s	user 0.005s	sys 0.000s
I20260709 12:30:45.303449 18892 tablet_bootstrap.cc:492] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:45.304703 19060 consensus_queue.cc:1048] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:45.305148 18887 tablet_bootstrap.cc:654] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.309453 18892 tablet_bootstrap.cc:654] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.312736 19028 consensus_queue.cc:1048] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:45.314153 18888 raft_consensus.cc:359] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.314868 18888 raft_consensus.cc:385] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.315135 18888 raft_consensus.cc:740] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.315936 18888 consensus_queue.cc:260] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.318657 18888 ts_tablet_manager.cc:1434] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 12:30:45.319664 18888 tablet_bootstrap.cc:492] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:45.321997 18957 raft_consensus.cc:515] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.321215 18513 catalog_manager.cc:5697] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:45.325654 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c180f8ddcbe14366a60537e1bad39332" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:45.326457 18690 raft_consensus.cc:3060] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.328490 18765 raft_consensus.cc:2468] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:45.330217 18892 tablet_bootstrap.cc:492] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:45.352849 18892 ts_tablet_manager.cc:1403] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.050s	user 0.009s	sys 0.002s
I20260709 12:30:45.326699 18888 tablet_bootstrap.cc:654] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.338394 18690 raft_consensus.cc:2468] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 1.
I20260709 12:30:45.358031 18582 leader_election.cc:304] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:30:45.355734 18892 raft_consensus.cc:359] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.359541 18892 raft_consensus.cc:385] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.357612 19059 consensus_queue.cc:1048] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:30:45.359874 18892 raft_consensus.cc:740] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.338848 18957 leader_election.cc:290] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:45.339927 18886 tablet_bootstrap.cc:492] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:45.340382 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c180f8ddcbe14366a60537e1bad39332" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:45.361618 18892 consensus_queue.cc:260] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.357342 18805 leader_election.cc:304] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:45.362183 18764 raft_consensus.cc:3060] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.341480 18972 raft_consensus.cc:493] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:45.363489 18972 raft_consensus.cc:515] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.363981 18892 ts_tablet_manager.cc:1434] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.002s
I20260709 12:30:45.365024 18892 tablet_bootstrap.cc:492] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:45.365773 18972 leader_election.cc:290] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:45.366338 18972 raft_consensus.cc:2804] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:45.366668 18972 raft_consensus.cc:493] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:45.366990 18972 raft_consensus.cc:3060] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.361914 18886 ts_tablet_manager.cc:1403] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.112s	user 0.015s	sys 0.002s
I20260709 12:30:45.350894 18616 raft_consensus.cc:3060] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.359774 18975 raft_consensus.cc:2804] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:45.374581 18886 raft_consensus.cc:359] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.376662 18886 raft_consensus.cc:385] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.377043 18886 raft_consensus.cc:740] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.376960 18975 raft_consensus.cc:697] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:30:45.379544 18886 consensus_queue.cc:260] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.379662 18975 consensus_queue.cc:237] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.381578 18886 ts_tablet_manager.cc:1434] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 12:30:45.382766 18886 tablet_bootstrap.cc:492] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:45.381628 18892 tablet_bootstrap.cc:654] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.389413 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df295fba24764e2ab46a8985e569bad8" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:45.390089 18618 raft_consensus.cc:2468] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:45.391983 18887 tablet_bootstrap.cc:492] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:45.392735 18887 ts_tablet_manager.cc:1403] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.101s	user 0.032s	sys 0.026s
I20260709 12:30:45.392803 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df295fba24764e2ab46a8985e569bad8" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:45.396867 18887 raft_consensus.cc:359] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.397230 18804 leader_election.cc:304] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
W20260709 12:30:45.398198 18805 leader_election.cc:343] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:45.398365 19062 raft_consensus.cc:2804] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:45.398926 19062 raft_consensus.cc:493] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:45.399358 19062 raft_consensus.cc:3060] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.402624 18764 raft_consensus.cc:2468] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 1.
I20260709 12:30:45.405993 18888 tablet_bootstrap.cc:492] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:45.406495 18888 ts_tablet_manager.cc:1403] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.087s	user 0.021s	sys 0.000s
I20260709 12:30:45.411803 18888 raft_consensus.cc:359] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.412511 18888 raft_consensus.cc:385] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.412837 18888 raft_consensus.cc:740] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.414067 18888 consensus_queue.cc:260] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.417711 18888 ts_tablet_manager.cc:1434] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.005s
I20260709 12:30:45.418911 18888 tablet_bootstrap.cc:492] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:45.421754 18887 raft_consensus.cc:385] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.427625 18957 raft_consensus.cc:493] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:45.428334 18957 raft_consensus.cc:515] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.422197 18887 raft_consensus.cc:740] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.433044 18887 consensus_queue.cc:260] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.435992 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee74ad2fe774e0aa780b1fdf18b3003" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:45.437991 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee74ad2fe774e0aa780b1fdf18b3003" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:45.438877 18957 leader_election.cc:290] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:45.417362 18972 raft_consensus.cc:515] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.439554 18581 leader_election.cc:304] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:45.440840 19062 raft_consensus.cc:515] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.441879 18888 tablet_bootstrap.cc:654] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.446326 18887 ts_tablet_manager.cc:1434] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.053s	user 0.008s	sys 0.001s
I20260709 12:30:45.449353 18926 raft_consensus.cc:493] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:45.449855 18926 raft_consensus.cc:515] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.443418 18957 raft_consensus.cc:2749] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:45.451980 19062 leader_election.cc:290] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:45.452984 18887 tablet_bootstrap.cc:492] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:45.453595 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df295fba24764e2ab46a8985e569bad8" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:45.451980 18926 leader_election.cc:290] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:45.455729 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fb81949f42946cf893026acc49357bc" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:45.465869 18690 raft_consensus.cc:2468] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
W20260709 12:30:45.466866 18805 leader_election.cc:343] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:45.467382 18731 leader_election.cc:304] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:45.468391 18926 raft_consensus.cc:2804] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:45.469007 18926 raft_consensus.cc:493] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:45.466408 18892 tablet_bootstrap.cc:492] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:45.469399 18886 tablet_bootstrap.cc:654] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.469890 18892 ts_tablet_manager.cc:1403] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.105s	user 0.008s	sys 0.015s
I20260709 12:30:45.470216 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fb81949f42946cf893026acc49357bc" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:45.468624 18516 catalog_manager.cc:5697] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:45.470400 19028 raft_consensus.cc:493] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:45.473489 19028 raft_consensus.cc:515] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.475626 19028 leader_election.cc:290] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:45.477855 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7b389579627488bb406ec0d112f6938" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:45.469413 18926 raft_consensus.cc:3060] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.491303 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7b389579627488bb406ec0d112f6938" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:45.492233 18840 raft_consensus.cc:2393] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d61e0800194dd89be34d3aff3f374d in current term 1: Already voted for candidate 84812365c65f43f388836c087640adbb in this term.
I20260709 12:30:45.456171 19030 consensus_queue.cc:1048] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20260709 12:30:45.494964 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c5cc1ca1ea8468199e1eeddb8c10756" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:45.496906 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c5cc1ca1ea8468199e1eeddb8c10756" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:45.498983 18765 raft_consensus.cc:3060] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.497395 18887 tablet_bootstrap.cc:654] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:45.503414 18730 leader_election.cc:343] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:30:45.505062 18805 leader_election.cc:343] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:45.509168 18892 raft_consensus.cc:359] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.509703 18892 raft_consensus.cc:385] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.509994 18892 raft_consensus.cc:740] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.505414 18656 leader_election.cc:304] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:45.504006 18972 leader_election.cc:290] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:45.512323 18888 tablet_bootstrap.cc:492] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:45.515365 18888 ts_tablet_manager.cc:1403] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.097s	user 0.016s	sys 0.003s
I20260709 12:30:45.510574 18892 consensus_queue.cc:260] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.500785 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df295fba24764e2ab46a8985e569bad8" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:45.518081 18616 raft_consensus.cc:3060] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.522516 18926 raft_consensus.cc:515] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.512333 19030 raft_consensus.cc:3060] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.525748 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fb81949f42946cf893026acc49357bc" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:45.526417 18618 raft_consensus.cc:3060] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.529609 18765 raft_consensus.cc:2468] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:45.529208 18888 raft_consensus.cc:359] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.530615 18888 raft_consensus.cc:385] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.531104 18888 raft_consensus.cc:740] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.533703 18805 leader_election.cc:304] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:45.534379 18892 ts_tablet_manager.cc:1434] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.062s	user 0.006s	sys 0.001s
I20260709 12:30:45.534915 18972 raft_consensus.cc:2804] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:45.535490 18972 raft_consensus.cc:697] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:45.535986 18892 tablet_bootstrap.cc:492] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:45.536870 18886 tablet_bootstrap.cc:492] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:45.536381 18972 consensus_queue.cc:237] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.537395 18886 ts_tablet_manager.cc:1403] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.155s	user 0.012s	sys 0.003s
I20260709 12:30:45.550376 18888 consensus_queue.cc:260] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.557591 18886 raft_consensus.cc:359] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.558913 18886 raft_consensus.cc:385] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.559340 18886 raft_consensus.cc:740] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.563235 18926 leader_election.cc:290] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:45.566718 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fb81949f42946cf893026acc49357bc" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:45.573920 18690 raft_consensus.cc:3060] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.574371 18616 raft_consensus.cc:2468] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:45.577999 18804 leader_election.cc:304] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:45.579196 18906 raft_consensus.cc:2804] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:45.579704 18906 raft_consensus.cc:697] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:45.580575 18906 consensus_queue.cc:237] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.580453 18886 consensus_queue.cc:260] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.583187 18690 raft_consensus.cc:2468] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:45.583856 18618 raft_consensus.cc:2468] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:45.584558 18731 leader_election.cc:304] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:45.585556 18926 raft_consensus.cc:2804] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:45.583240 19062 raft_consensus.cc:493] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:45.586253 18926 raft_consensus.cc:697] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:45.589314 18886 ts_tablet_manager.cc:1434] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.052s	user 0.009s	sys 0.001s
I20260709 12:30:45.590143 18888 ts_tablet_manager.cc:1434] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.074s	user 0.005s	sys 0.005s
I20260709 12:30:45.587474 19062 raft_consensus.cc:515] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.594933 18926 consensus_queue.cc:237] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.596026 18516 catalog_manager.cc:5697] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:45.599362 18887 tablet_bootstrap.cc:492] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:45.599990 18887 ts_tablet_manager.cc:1403] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.147s	user 0.031s	sys 0.029s
I20260709 12:30:45.600862 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f16818c4dfd4e9ab187ffb53d8fc3ce" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:45.601606 18765 raft_consensus.cc:2468] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:45.603250 18886 tablet_bootstrap.cc:492] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:45.604285 18887 raft_consensus.cc:359] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.605070 18887 raft_consensus.cc:385] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.605386 18887 raft_consensus.cc:740] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.606247 18887 consensus_queue.cc:260] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.608422 18887 ts_tablet_manager.cc:1434] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:30:45.609696 18887 tablet_bootstrap.cc:492] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:45.610352 18886 tablet_bootstrap.cc:654] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.615545 19062 leader_election.cc:290] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:45.615459 18805 leader_election.cc:304] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:45.618541 18515 catalog_manager.cc:5697] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 1 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:45.624580 18888 tablet_bootstrap.cc:492] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:45.627185 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f16818c4dfd4e9ab187ffb53d8fc3ce" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:45.618995 19062 raft_consensus.cc:2804] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:45.629124 18887 tablet_bootstrap.cc:654] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:45.629207 18804 leader_election.cc:343] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:45.628650 19062 raft_consensus.cc:493] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:45.630604 19062 raft_consensus.cc:3060] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.635181 18513 catalog_manager.cc:5697] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:45.636080 19021 consensus_queue.cc:1048] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:45.638023 19030 raft_consensus.cc:2749] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:45.646018 18892 tablet_bootstrap.cc:654] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.669786 18888 tablet_bootstrap.cc:654] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.679183 18887 tablet_bootstrap.cc:492] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:45.679816 18887 ts_tablet_manager.cc:1403] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.070s	user 0.016s	sys 0.033s
I20260709 12:30:45.684670 18887 raft_consensus.cc:359] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.685511 18887 raft_consensus.cc:385] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.685825 18887 raft_consensus.cc:740] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.686616 18887 consensus_queue.cc:260] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.688040 18886 tablet_bootstrap.cc:492] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:45.688665 18886 ts_tablet_manager.cc:1403] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.086s	user 0.007s	sys 0.007s
I20260709 12:30:45.689734 18887 ts_tablet_manager.cc:1434] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.005s
I20260709 12:30:45.691112 18887 tablet_bootstrap.cc:492] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:45.692018 18886 raft_consensus.cc:359] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.692951 18886 raft_consensus.cc:385] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.693212 18886 raft_consensus.cc:740] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.693959 18886 consensus_queue.cc:260] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.698124 18887 tablet_bootstrap.cc:654] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.701634 18886 ts_tablet_manager.cc:1434] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:30:45.702998 18886 tablet_bootstrap.cc:492] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:45.707633 18888 tablet_bootstrap.cc:492] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:45.708356 18888 ts_tablet_manager.cc:1403] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.106s	user 0.016s	sys 0.013s
I20260709 12:30:45.711439 18888 raft_consensus.cc:359] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.715602 19062 raft_consensus.cc:515] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.717754 18886 tablet_bootstrap.cc:654] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.712193 18888 raft_consensus.cc:385] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.720276 18972 consensus_queue.cc:1048] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:45.720448 19030 raft_consensus.cc:493] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:45.721311 19030 raft_consensus.cc:515] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.723474 19030 leader_election.cc:290] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:45.720700 18888 raft_consensus.cc:740] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.725015 18888 consensus_queue.cc:260] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.727108 18888 ts_tablet_manager.cc:1434] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.018s	user 0.003s	sys 0.001s
I20260709 12:30:45.731405 18887 tablet_bootstrap.cc:492] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:45.732124 18887 ts_tablet_manager.cc:1403] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.041s	user 0.010s	sys 0.011s
I20260709 12:30:45.734700 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c5cc1ca1ea8468199e1eeddb8c10756" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:45.735582 18765 raft_consensus.cc:2393] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d61e0800194dd89be34d3aff3f374d in current term 1: Already voted for candidate df352ed4600b4ddb83ef57ecd2944fce in this term.
I20260709 12:30:45.735589 19062 leader_election.cc:290] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:45.729053 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c5cc1ca1ea8468199e1eeddb8c10756" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:45.738153 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f16818c4dfd4e9ab187ffb53d8fc3ce" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:45.738754 18764 raft_consensus.cc:3060] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.742929 18656 leader_election.cc:304] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:45.744136 19030 raft_consensus.cc:3060] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.745854 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f16818c4dfd4e9ab187ffb53d8fc3ce" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
W20260709 12:30:45.748404 18804 leader_election.cc:343] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:45.749878 18957 consensus_queue.cc:1048] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:45.751700 19030 raft_consensus.cc:2749] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:45.768311 18888 tablet_bootstrap.cc:492] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:45.779558 19062 consensus_queue.cc:1048] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:45.783097 18764 raft_consensus.cc:2468] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:45.785796 18805 leader_election.cc:304] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:45.787285 19062 raft_consensus.cc:2804] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:45.787810 19062 raft_consensus.cc:697] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:45.789356 18892 tablet_bootstrap.cc:492] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:45.789222 19062 consensus_queue.cc:237] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.790197 18892 ts_tablet_manager.cc:1403] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.255s	user 0.012s	sys 0.006s
I20260709 12:30:45.791393 18886 tablet_bootstrap.cc:492] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:45.791901 18886 ts_tablet_manager.cc:1403] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.089s	user 0.026s	sys 0.026s
I20260709 12:30:45.795686 18888 tablet_bootstrap.cc:654] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.801699 18892 raft_consensus.cc:359] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.802536 18887 raft_consensus.cc:359] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.802990 18892 raft_consensus.cc:385] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.803233 18887 raft_consensus.cc:385] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.803293 18892 raft_consensus.cc:740] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.803604 18887 raft_consensus.cc:740] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.807111 18926 raft_consensus.cc:493] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:45.807896 18926 raft_consensus.cc:515] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.810194 18926 leader_election.cc:290] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:45.811519 19028 consensus_queue.cc:1048] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:45.811676 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e16042d49945ea80b31f9eee56f2cf" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:45.814872 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e16042d49945ea80b31f9eee56f2cf" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:45.815707 18840 raft_consensus.cc:2393] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 29f82a8594d741bc92dd24ee4a368e01 in current term 1: Already voted for candidate 60d61e0800194dd89be34d3aff3f374d in this term.
I20260709 12:30:45.807654 18887 consensus_queue.cc:260] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.823724 18730 leader_election.cc:304] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:45.818693 18887 ts_tablet_manager.cc:1434] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.086s	user 0.004s	sys 0.001s
I20260709 12:30:45.827248 18887 tablet_bootstrap.cc:492] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:45.827215 18926 raft_consensus.cc:3060] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.831142 18886 raft_consensus.cc:359] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.832063 18886 raft_consensus.cc:385] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.832609 18886 raft_consensus.cc:740] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.834017 18886 consensus_queue.cc:260] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.834455 18957 consensus_queue.cc:1048] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:45.836578 18886 ts_tablet_manager.cc:1434] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.044s	user 0.008s	sys 0.001s
I20260709 12:30:45.836755 18887 tablet_bootstrap.cc:654] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.842388 18886 tablet_bootstrap.cc:492] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:45.844110 18972 raft_consensus.cc:493] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:45.844729 18972 raft_consensus.cc:515] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:45.847119 18972 leader_election.cc:290] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:45.847409 18516 catalog_manager.cc:5697] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:45.849191 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7089bc11f6424c178808e4579d967e14" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
W20260709 12:30:45.851240 18805 leader_election.cc:343] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:45.852722 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7089bc11f6424c178808e4579d967e14" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:45.857380 18805 leader_election.cc:343] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:45.857898 18805 leader_election.cc:304] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:45.858650 18906 raft_consensus.cc:2749] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:45.861819 18886 tablet_bootstrap.cc:654] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.804670 18892 consensus_queue.cc:260] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.870301 18957 raft_consensus.cc:493] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:45.870992 18957 raft_consensus.cc:515] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.873772 18886 tablet_bootstrap.cc:492] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:45.874336 18886 ts_tablet_manager.cc:1403] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.037s	user 0.015s	sys 0.002s
I20260709 12:30:45.877054 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f16818c4dfd4e9ab187ffb53d8fc3ce" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:45.877971 18764 raft_consensus.cc:2393] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84812365c65f43f388836c087640adbb in current term 1: Already voted for candidate df352ed4600b4ddb83ef57ecd2944fce in this term.
I20260709 12:30:45.878794 18892 ts_tablet_manager.cc:1434] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.087s	user 0.006s	sys 0.000s
I20260709 12:30:45.880007 18886 raft_consensus.cc:359] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.880791 18886 raft_consensus.cc:385] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.881064 18886 raft_consensus.cc:740] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.886248 18886 consensus_queue.cc:260] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.888098 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f16818c4dfd4e9ab187ffb53d8fc3ce" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:45.889006 18886 ts_tablet_manager.cc:1434] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:30:45.890129 18886 tablet_bootstrap.cc:492] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:45.893379 18581 leader_election.cc:304] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:45.894330 19021 raft_consensus.cc:3060] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.896292 18886 tablet_bootstrap.cc:654] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.905973 18957 leader_election.cc:290] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:45.907275 19021 raft_consensus.cc:2749] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:45.909072 18926 raft_consensus.cc:2749] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:45.880177 18892 tablet_bootstrap.cc:492] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:45.911051 18886 tablet_bootstrap.cc:492] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:45.911680 18886 ts_tablet_manager.cc:1403] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.022s	user 0.011s	sys 0.006s
I20260709 12:30:45.914814 18886 raft_consensus.cc:359] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.915596 18886 raft_consensus.cc:385] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.915942 18886 raft_consensus.cc:740] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.916834 18886 consensus_queue.cc:260] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.919497 18888 tablet_bootstrap.cc:492] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:45.919535 18886 ts_tablet_manager.cc:1434] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:30:45.920257 18888 ts_tablet_manager.cc:1403] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.180s	user 0.026s	sys 0.019s
I20260709 12:30:45.921036 18886 tablet_bootstrap.cc:492] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:45.923148 18888 raft_consensus.cc:359] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.923835 18888 raft_consensus.cc:385] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:45.924129 18888 raft_consensus.cc:740] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:45.925048 18888 consensus_queue.cc:260] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.927565 18886 tablet_bootstrap.cc:654] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:45.919430 18965 consensus_queue.cc:1048] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:30:45.938448 18975 consensus_queue.cc:1048] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:45.944056 19062 raft_consensus.cc:493] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:45.944908 19062 raft_consensus.cc:515] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:45.948897 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fddb762772a40acb39a71286c8d4f41" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
W20260709 12:30:45.951587 18804 leader_election.cc:343] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:45.960668 19062 leader_election.cc:290] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:45.969128 18887 tablet_bootstrap.cc:492] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:45.969695 18887 ts_tablet_manager.cc:1403] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.143s	user 0.016s	sys 0.022s
I20260709 12:30:45.960752 19086 raft_consensus.cc:493] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:45.973026 19086 raft_consensus.cc:515] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:45.974550 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fddb762772a40acb39a71286c8d4f41" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:45.975315 18692 raft_consensus.cc:2468] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:45.964200 18516 catalog_manager.cc:5697] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:45.976208 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f9d88127114fda91668ab529abd23c" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:45.976616 18805 leader_election.cc:304] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
W20260709 12:30:45.977783 18730 leader_election.cc:343] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:45.977857 18906 consensus_queue.cc:1048] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:45.979321 19086 leader_election.cc:290] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:45.979377 18906 raft_consensus.cc:2804] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:45.979878 18906 raft_consensus.cc:493] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:45.981909 18888 ts_tablet_manager.cc:1434] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.061s	user 0.006s	sys 0.000s
I20260709 12:30:45.983018 18888 tablet_bootstrap.cc:492] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:45.985886 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f9d88127114fda91668ab529abd23c" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:45.986692 18690 raft_consensus.cc:2468] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:45.980240 18906 raft_consensus.cc:3060] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:45.991091 18731 leader_election.cc:304] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:45.997895 18906 raft_consensus.cc:515] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.001017 18926 raft_consensus.cc:2804] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:46.004386 18926 raft_consensus.cc:493] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:46.004868 18926 raft_consensus.cc:3060] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.001003 18892 tablet_bootstrap.cc:654] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.020613 18972 raft_consensus.cc:493] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.021202 18972 raft_consensus.cc:515] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:45.988735 18887 raft_consensus.cc:359] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.023296 18972 leader_election.cc:290] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:46.023522 18887 raft_consensus.cc:385] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.023900 18887 raft_consensus.cc:740] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.024585 18887 consensus_queue.cc:260] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.027554 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efa9444e110e433aa97b25a05d5769a9" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:46.030932 18888 tablet_bootstrap.cc:654] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.031055 19086 consensus_queue.cc:1048] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:46.034013 18887 ts_tablet_manager.cc:1434] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.064s	user 0.008s	sys 0.000s
I20260709 12:30:46.035084 18887 tablet_bootstrap.cc:492] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:46.041435 18690 raft_consensus.cc:3060] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.042394 18886 tablet_bootstrap.cc:492] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:46.042572 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efa9444e110e433aa97b25a05d5769a9" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:46.042925 18886 ts_tablet_manager.cc:1403] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.122s	user 0.034s	sys 0.050s
I20260709 12:30:46.043170 18887 tablet_bootstrap.cc:654] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.043231 18766 raft_consensus.cc:2468] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:46.044492 18805 leader_election.cc:304] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
W20260709 12:30:46.045184 18804 leader_election.cc:343] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.045420 19059 raft_consensus.cc:2804] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:46.045828 19059 raft_consensus.cc:493] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:46.046159 19059 raft_consensus.cc:3060] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.053493 19059 raft_consensus.cc:515] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.059878 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efa9444e110e433aa97b25a05d5769a9" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:46.061906 18766 raft_consensus.cc:3060] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.088573 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fddb762772a40acb39a71286c8d4f41" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:46.092679 18692 raft_consensus.cc:3060] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.094920 18886 raft_consensus.cc:359] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.095685 18886 raft_consensus.cc:385] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.095978 18886 raft_consensus.cc:740] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.097501 18888 tablet_bootstrap.cc:492] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:46.098017 18888 ts_tablet_manager.cc:1403] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.115s	user 0.013s	sys 0.004s
I20260709 12:30:46.100960 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fddb762772a40acb39a71286c8d4f41" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:46.101516 18886 consensus_queue.cc:260] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.102401 18888 raft_consensus.cc:359] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.104056 18886 ts_tablet_manager.cc:1434] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.061s	user 0.009s	sys 0.005s
I20260709 12:30:46.104107 18888 raft_consensus.cc:385] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.105095 18888 raft_consensus.cc:740] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.105391 18886 tablet_bootstrap.cc:492] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:46.106887 18888 consensus_queue.cc:260] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.107513 19086 consensus_queue.cc:1048] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:30:46.108740 18804 leader_election.cc:343] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.109650 19059 leader_election.cc:290] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:46.014314 18926 raft_consensus.cc:515] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.114289 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efa9444e110e433aa97b25a05d5769a9" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
W20260709 12:30:46.116389 18804 leader_election.cc:343] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.117079 19059 raft_consensus.cc:493] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.117220 18888 ts_tablet_manager.cc:1434] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.002s
I20260709 12:30:46.117672 19059 raft_consensus.cc:515] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.119210 18888 tablet_bootstrap.cc:492] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:46.125041 19059 leader_election.cc:290] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:46.126169 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2abc15bf7b445a4b4269e1d93cce949" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:46.112509 18906 leader_election.cc:290] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:46.145681 18888 tablet_bootstrap.cc:654] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:46.146463 18805 leader_election.cc:343] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.149600 18886 tablet_bootstrap.cc:654] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.150625 19059 raft_consensus.cc:493] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:30:46.150820 18730 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 50490.7us
I20260709 12:30:46.155773 19059 raft_consensus.cc:515] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.158525 19059 leader_election.cc:290] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:46.160046 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:46.163106 19059 consensus_queue.cc:1048] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:30:46.165861 18804 leader_election.cc:343] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.170001 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f9d88127114fda91668ab529abd23c" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:46.189670 19028 raft_consensus.cc:493] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.190222 19028 raft_consensus.cc:515] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.194231 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f9d88127114fda91668ab529abd23c" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:46.195293 18691 raft_consensus.cc:3060] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.180989 18692 raft_consensus.cc:2468] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:46.198284 18805 leader_election.cc:304] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:46.199645 18906 raft_consensus.cc:2804] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:46.198031 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df295fba24764e2ab46a8985e569bad8" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:46.200165 18906 raft_consensus.cc:697] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:46.202762 18906 consensus_queue.cc:237] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.182515 18926 leader_election.cc:290] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:46.208406 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df295fba24764e2ab46a8985e569bad8" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:46.198284 19028 leader_election.cc:290] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:46.210978 19059 consensus_queue.cc:1048] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:46.211947 18656 leader_election.cc:304] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:46.213785 19030 raft_consensus.cc:2749] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:46.216233 19030 raft_consensus.cc:493] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.216975 19030 raft_consensus.cc:515] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.221895 19030 leader_election.cc:290] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:46.222915 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d43c01f2ed54eec9d5a7b6b75cd6eeb" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:46.225492 18657 leader_election.cc:343] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.228303 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d43c01f2ed54eec9d5a7b6b75cd6eeb" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:46.229676 18892 tablet_bootstrap.cc:492] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:46.230222 18892 ts_tablet_manager.cc:1403] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.350s	user 0.012s	sys 0.017s
W20260709 12:30:46.230391 18656 leader_election.cc:343] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.231011 18656 leader_election.cc:304] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:46.231869 19028 raft_consensus.cc:2749] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:46.233193 18892 raft_consensus.cc:359] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.234373 18892 raft_consensus.cc:385] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.234576 19118 raft_consensus.cc:493] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.234668 18892 raft_consensus.cc:740] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.235044 18887 tablet_bootstrap.cc:492] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:46.235163 19118 raft_consensus.cc:515] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.235678 18892 consensus_queue.cc:260] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.236063 18766 raft_consensus.cc:2468] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:46.235603 18887 ts_tablet_manager.cc:1403] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.201s	user 0.021s	sys 0.022s
I20260709 12:30:46.251750 19118 leader_election.cc:290] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:46.253945 18892 ts_tablet_manager.cc:1434] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.001s
I20260709 12:30:46.255288 18892 tablet_bootstrap.cc:492] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:46.258787 18691 raft_consensus.cc:2468] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:46.278578 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:46.279212 19062 consensus_queue.cc:1048] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:46.279464 18765 raft_consensus.cc:2468] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:46.283397 18805 leader_election.cc:304] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:46.287572 18892 tablet_bootstrap.cc:654] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.293274 18515 catalog_manager.cc:5697] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:46.286720 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2abc15bf7b445a4b4269e1d93cce949" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:46.301347 18731 leader_election.cc:304] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:46.298273 19062 raft_consensus.cc:2804] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:46.311551 18888 tablet_bootstrap.cc:492] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:46.312151 18888 ts_tablet_manager.cc:1403] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.193s	user 0.017s	sys 0.016s
I20260709 12:30:46.312005 19062 raft_consensus.cc:697] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:46.313781 19062 consensus_queue.cc:237] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.315092 18888 raft_consensus.cc:359] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.315491 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:46.315811 18888 raft_consensus.cc:385] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.316128 18888 raft_consensus.cc:740] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.316170 18841 raft_consensus.cc:2468] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:46.317195 18888 consensus_queue.cc:260] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.306638 19117 raft_consensus.cc:2804] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:46.319808 19117 raft_consensus.cc:697] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:46.320689 19117 consensus_queue.cc:237] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.303886 19030 raft_consensus.cc:493] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.326618 19030 raft_consensus.cc:515] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.329779 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7089bc11f6424c178808e4579d967e14" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:46.331535 18657 leader_election.cc:343] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.331838 18888 ts_tablet_manager.cc:1434] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:30:46.332892 19030 leader_election.cc:290] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:46.333245 18888 tablet_bootstrap.cc:492] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:46.356824 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:46.357214 18888 tablet_bootstrap.cc:654] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.358700 19021 raft_consensus.cc:493] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.359207 19021 raft_consensus.cc:515] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.361348 19021 leader_election.cc:290] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:46.363253 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f9d88127114fda91668ab529abd23c" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:46.364184 18693 raft_consensus.cc:2393] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84812365c65f43f388836c087640adbb in current term 1: Already voted for candidate 29f82a8594d741bc92dd24ee4a368e01 in this term.
I20260709 12:30:46.370965 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f9d88127114fda91668ab529abd23c" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:46.372859 18582 leader_election.cc:304] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:46.377355 19021 raft_consensus.cc:3060] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.376056 19060 consensus_queue.cc:1048] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:30:46.380718 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7089bc11f6424c178808e4579d967e14" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:46.381428 18840 raft_consensus.cc:2468] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 0.
I20260709 12:30:46.382998 18887 raft_consensus.cc:359] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.383749 18887 raft_consensus.cc:385] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.384066 18887 raft_consensus.cc:740] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.385106 18887 consensus_queue.cc:260] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.387456 18886 tablet_bootstrap.cc:492] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:46.388041 18886 ts_tablet_manager.cc:1403] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.283s	user 0.029s	sys 0.012s
I20260709 12:30:46.389065 18887 ts_tablet_manager.cc:1434] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.144s	user 0.004s	sys 0.002s
I20260709 12:30:46.399432 19021 raft_consensus.cc:2749] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:30:46.403501 18730 leader_election.cc:343] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:30:46.405790 18730 leader_election.cc:343] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:30:46.408172 18563 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:30:46.402226 18516 catalog_manager.cc:5697] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:46.408994 18730 leader_election.cc:304] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:46.409770 19124 raft_consensus.cc:2804] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:46.410136 19124 raft_consensus.cc:493] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:46.410450 19124 raft_consensus.cc:3060] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.410761 18886 raft_consensus.cc:359] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.412933 18886 raft_consensus.cc:385] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.413187 18886 raft_consensus.cc:740] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.413930 18515 catalog_manager.cc:5697] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:46.414186 18886 consensus_queue.cc:260] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.418048 18656 leader_election.cc:304] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:30:46.418848 18892 tablet_bootstrap.cc:492] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:46.419299 18892 ts_tablet_manager.cc:1403] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.164s	user 0.016s	sys 0.008s
I20260709 12:30:46.425330 18887 tablet_bootstrap.cc:492] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:46.428400 19028 raft_consensus.cc:2804] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:46.428900 19028 raft_consensus.cc:493] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:46.429265 19028 raft_consensus.cc:3060] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.436793 18892 raft_consensus.cc:359] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.437934 18892 raft_consensus.cc:385] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.438295 18892 raft_consensus.cc:740] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.439226 18892 consensus_queue.cc:260] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.441340 18805 leader_election.cc:304] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:46.442247 19062 raft_consensus.cc:2804] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:46.442247 19028 raft_consensus.cc:515] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.442883 19062 raft_consensus.cc:493] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:46.443282 19062 raft_consensus.cc:3060] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.444097 18892 ts_tablet_manager.cc:1434] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.002s
I20260709 12:30:46.445503 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7089bc11f6424c178808e4579d967e14" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:46.445914 18892 tablet_bootstrap.cc:492] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:30:46.447063 18657 leader_election.cc:343] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.448678 18886 ts_tablet_manager.cc:1434] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.060s	user 0.005s	sys 0.001s
I20260709 12:30:46.450419 18886 tablet_bootstrap.cc:492] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:46.453758 18888 tablet_bootstrap.cc:492] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:46.454268 18888 ts_tablet_manager.cc:1403] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.121s	user 0.012s	sys 0.008s
W20260709 12:30:46.448941 18805 leader_election.cc:343] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.459780 18892 tablet_bootstrap.cc:654] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.470175 18887 tablet_bootstrap.cc:654] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.472424 18886 tablet_bootstrap.cc:654] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.473671 18888 raft_consensus.cc:359] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.474655 18888 raft_consensus.cc:385] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.464576 18805 leader_election.cc:304] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:46.478468 18972 raft_consensus.cc:2749] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:46.472455 19028 leader_election.cc:290] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:46.478461 19021 raft_consensus.cc:493] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.480087 19021 raft_consensus.cc:515] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.482362 19021 leader_election.cc:290] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:46.477031 19124 raft_consensus.cc:515] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.483875 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efa9444e110e433aa97b25a05d5769a9" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:46.484259 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efa9444e110e433aa97b25a05d5769a9" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:46.484807 18764 raft_consensus.cc:2393] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84812365c65f43f388836c087640adbb in current term 1: Already voted for candidate df352ed4600b4ddb83ef57ecd2944fce in this term.
I20260709 12:30:46.485766 19124 leader_election.cc:290] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:46.486656 18582 leader_election.cc:304] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:46.486861 19062 raft_consensus.cc:515] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.488653 19021 raft_consensus.cc:3060] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.489303 19062 leader_election.cc:290] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:46.475113 18888 raft_consensus.cc:740] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.490383 18888 consensus_queue.cc:260] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.492640 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:46.493263 18888 ts_tablet_manager.cc:1434] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.001s
W20260709 12:30:46.494225 18804 leader_election.cc:343] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.494690 18888 tablet_bootstrap.cc:492] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:46.499785 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:46.500880 18840 raft_consensus.cc:2393] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 29f82a8594d741bc92dd24ee4a368e01 in current term 1: Already voted for candidate df352ed4600b4ddb83ef57ecd2944fce in this term.
W20260709 12:30:46.509053 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:40335, user_credentials={real_user=slave}} blocked reactor thread for 60148us
I20260709 12:30:46.513008 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:46.518867 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7089bc11f6424c178808e4579d967e14" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:46.519611 18840 raft_consensus.cc:3060] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:30:46.520531 18730 leader_election.cc:343] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.521086 18730 leader_election.cc:304] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:46.521904 19124 raft_consensus.cc:2749] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:30:46.522927 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:46.523768 18765 raft_consensus.cc:2393] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df352ed4600b4ddb83ef57ecd2944fce in current term 1: Already voted for candidate 29f82a8594d741bc92dd24ee4a368e01 in this term.
I20260709 12:30:46.529314 18972 consensus_queue.cc:1048] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:46.533016 18888 tablet_bootstrap.cc:654] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.530350 19021 raft_consensus.cc:2749] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:46.528111 18840 raft_consensus.cc:2468] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:46.535084 18892 tablet_bootstrap.cc:492] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:46.538856 19124 raft_consensus.cc:493] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.540987 18892 ts_tablet_manager.cc:1403] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.095s	user 0.009s	sys 0.005s
I20260709 12:30:46.541713 19124 raft_consensus.cc:515] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.544638 18892 raft_consensus.cc:359] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.545141 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d43c01f2ed54eec9d5a7b6b75cd6eeb" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:46.545584 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d43c01f2ed54eec9d5a7b6b75cd6eeb" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:46.545845 18888 tablet_bootstrap.cc:492] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:46.546418 18888 ts_tablet_manager.cc:1403] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.052s	user 0.012s	sys 0.008s
I20260709 12:30:46.546293 18693 raft_consensus.cc:2468] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:46.538439 18887 tablet_bootstrap.cc:492] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
W20260709 12:30:46.546825 18730 leader_election.cc:343] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.547103 18887 ts_tablet_manager.cc:1403] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.122s	user 0.018s	sys 0.003s
I20260709 12:30:46.547742 18731 leader_election.cc:304] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:46.548182 18805 leader_election.cc:304] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:46.548686 19117 raft_consensus.cc:2804] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:46.549139 19117 raft_consensus.cc:493] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:46.549484 19117 raft_consensus.cc:3060] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.550590 18887 raft_consensus.cc:359] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.551306 18887 raft_consensus.cc:385] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.551576 18887 raft_consensus.cc:740] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.545380 18892 raft_consensus.cc:385] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.552515 18972 raft_consensus.cc:2749] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:30:46.553401 19124 leader_election.cc:290] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:46.556529 18656 leader_election.cc:304] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:30:46.557843 19028 raft_consensus.cc:2804] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:46.558319 19028 raft_consensus.cc:697] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:30:46.552276 18887 consensus_queue.cc:260] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.559150 19028 consensus_queue.cc:237] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.561278 18887 ts_tablet_manager.cc:1434] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:30:46.552657 18892 raft_consensus.cc:740] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.566097 18887 tablet_bootstrap.cc:492] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:46.566890 18892 consensus_queue.cc:260] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.570746 18888 raft_consensus.cc:359] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.571547 18888 raft_consensus.cc:385] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.573554 18887 tablet_bootstrap.cc:654] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.573385 19117 raft_consensus.cc:515] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.573700 18892 ts_tablet_manager.cc:1434] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.000s
I20260709 12:30:46.577148 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d43c01f2ed54eec9d5a7b6b75cd6eeb" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:46.577625 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d43c01f2ed54eec9d5a7b6b75cd6eeb" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:46.584887 19117 leader_election.cc:290] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:46.571817 18888 raft_consensus.cc:740] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.589670 18886 tablet_bootstrap.cc:492] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:46.596004 18886 ts_tablet_manager.cc:1403] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.146s	user 0.025s	sys 0.023s
I20260709 12:30:46.587419 18887 tablet_bootstrap.cc:492] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:46.599656 18887 ts_tablet_manager.cc:1403] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.034s	user 0.014s	sys 0.005s
I20260709 12:30:46.590130 18693 raft_consensus.cc:3060] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:30:46.591004 18730 leader_election.cc:343] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.602398 18513 catalog_manager.cc:5697] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:46.598830 18886 raft_consensus.cc:359] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.603789 18886 raft_consensus.cc:385] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.604197 18886 raft_consensus.cc:740] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.605262 18886 consensus_queue.cc:260] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.607513 18886 ts_tablet_manager.cc:1434] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.000s
I20260709 12:30:46.608522 18886 tablet_bootstrap.cc:492] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:46.610517 18693 raft_consensus.cc:2468] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:46.612180 18731 leader_election.cc:304] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:46.613205 19117 raft_consensus.cc:2804] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:46.614075 19117 raft_consensus.cc:697] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:46.615483 19117 consensus_queue.cc:237] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.591274 18892 tablet_bootstrap.cc:492] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:46.622916 18886 tablet_bootstrap.cc:654] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.623575 19028 raft_consensus.cc:493] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.624114 19028 raft_consensus.cc:515] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.594324 18888 consensus_queue.cc:260] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.626293 19028 leader_election.cc:290] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:46.627990 18888 ts_tablet_manager.cc:1434] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.065s	user 0.002s	sys 0.005s
I20260709 12:30:46.629429 18888 tablet_bootstrap.cc:492] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:46.629530 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e3d64db91924be490d4ed1f797d338f" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
W20260709 12:30:46.631646 18656 leader_election.cc:343] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.632781 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e3d64db91924be490d4ed1f797d338f" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:46.602594 18887 raft_consensus.cc:359] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.633919 18887 raft_consensus.cc:385] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.634338 18887 raft_consensus.cc:740] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.633486 18765 raft_consensus.cc:2468] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 0.
I20260709 12:30:46.641862 18657 leader_election.cc:304] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:46.643923 19028 raft_consensus.cc:2804] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:46.644623 19028 raft_consensus.cc:493] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:46.645157 19028 raft_consensus.cc:3060] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.650909 18886 tablet_bootstrap.cc:492] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:46.651494 18886 ts_tablet_manager.cc:1403] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.043s	user 0.014s	sys 0.012s
I20260709 12:30:46.654322 18886 raft_consensus.cc:359] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.655019 18886 raft_consensus.cc:385] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.655315 18886 raft_consensus.cc:740] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.656632 18887 consensus_queue.cc:260] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.659221 18887 ts_tablet_manager.cc:1434] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.059s	user 0.001s	sys 0.005s
I20260709 12:30:46.658833 18886 consensus_queue.cc:260] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.664093 18888 tablet_bootstrap.cc:654] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.664547 18886 ts_tablet_manager.cc:1434] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.001s
I20260709 12:30:46.665581 18886 tablet_bootstrap.cc:492] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:46.667822 18516 catalog_manager.cc:5697] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:46.671475 18886 tablet_bootstrap.cc:654] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.679646 18972 raft_consensus.cc:493] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.680001 18887 tablet_bootstrap.cc:492] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:46.680544 18972 raft_consensus.cc:515] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.685263 18892 tablet_bootstrap.cc:654] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.685576 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4eead98f0854adfaf72daa76fdf9c9b" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
W20260709 12:30:46.688134 18804 leader_election.cc:343] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.691131 18887 tablet_bootstrap.cc:654] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.692669 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4eead98f0854adfaf72daa76fdf9c9b" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:46.693677 18693 raft_consensus.cc:2468] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:46.688665 19028 raft_consensus.cc:515] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.696194 19028 leader_election.cc:290] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:46.697309 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e3d64db91924be490d4ed1f797d338f" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:46.700057 18765 raft_consensus.cc:3060] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.702250 18805 leader_election.cc:304] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:46.703539 19062 raft_consensus.cc:2804] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:46.704052 19062 raft_consensus.cc:493] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:46.704561 19062 raft_consensus.cc:3060] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.704335 18888 tablet_bootstrap.cc:492] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:46.706004 18888 ts_tablet_manager.cc:1403] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.077s	user 0.012s	sys 0.002s
I20260709 12:30:46.706048 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e3d64db91924be490d4ed1f797d338f" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
W20260709 12:30:46.707610 18656 leader_election.cc:343] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.709568 18765 raft_consensus.cc:2468] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:46.710201 18886 tablet_bootstrap.cc:492] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:46.711403 18886 ts_tablet_manager.cc:1403] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.046s	user 0.013s	sys 0.028s
I20260709 12:30:46.711565 18657 leader_election.cc:304] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:46.712117 18616 raft_consensus.cc:3060] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.716266 18886 raft_consensus.cc:359] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.717072 18886 raft_consensus.cc:385] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.717406 18886 raft_consensus.cc:740] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.718776 19028 raft_consensus.cc:2804] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:46.718568 18886 consensus_queue.cc:260] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.720925 18886 ts_tablet_manager.cc:1434] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:30:46.722018 18886 tablet_bootstrap.cc:492] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:46.722653 19117 consensus_queue.cc:1048] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:46.713232 18887 tablet_bootstrap.cc:492] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:46.719321 19028 raft_consensus.cc:697] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:30:46.724028 18887 ts_tablet_manager.cc:1403] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.044s	user 0.012s	sys 0.000s
I20260709 12:30:46.724664 19028 consensus_queue.cc:237] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:46.712996 18972 leader_election.cc:290] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:46.728667 18886 tablet_bootstrap.cc:654] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.728966 18892 tablet_bootstrap.cc:492] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:46.729586 18892 ts_tablet_manager.cc:1403] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.139s	user 0.017s	sys 0.000s
I20260709 12:30:46.735143 19059 consensus_queue.cc:1048] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:46.735263 18888 raft_consensus.cc:359] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.736423 18888 raft_consensus.cc:385] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.737576 18892 raft_consensus.cc:359] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.736836 18888 raft_consensus.cc:740] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.738574 19060 consensus_queue.cc:1048] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:46.739465 18888 consensus_queue.cc:260] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.742195 18888 ts_tablet_manager.cc:1434] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.000s
I20260709 12:30:46.738610 18892 raft_consensus.cc:385] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.747694 18892 raft_consensus.cc:740] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.728902 18887 raft_consensus.cc:359] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.748840 18887 raft_consensus.cc:385] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.749095 18887 raft_consensus.cc:740] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.749011 18892 consensus_queue.cc:260] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.752992 18887 consensus_queue.cc:260] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.757355 19062 raft_consensus.cc:515] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.759792 19117 consensus_queue.cc:1048] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:46.761744 18887 ts_tablet_manager.cc:1434] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.037s	user 0.008s	sys 0.000s
I20260709 12:30:46.762789 18887 tablet_bootstrap.cc:492] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:46.763625 18886 tablet_bootstrap.cc:492] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:46.764139 18886 ts_tablet_manager.cc:1403] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.042s	user 0.005s	sys 0.030s
I20260709 12:30:46.769860 18887 tablet_bootstrap.cc:654] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.774914 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4eead98f0854adfaf72daa76fdf9c9b" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:46.776598 19117 raft_consensus.cc:493] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.777170 19117 raft_consensus.cc:515] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.779268 19117 leader_election.cc:290] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
W20260709 12:30:46.779497 18804 leader_election.cc:343] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.784922 18886 raft_consensus.cc:359] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.785812 18886 raft_consensus.cc:385] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:46.786150 18886 raft_consensus.cc:740] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:46.786705 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7089bc11f6424c178808e4579d967e14" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:46.787537 18840 raft_consensus.cc:2393] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 29f82a8594d741bc92dd24ee4a368e01 in current term 1: Already voted for candidate 60d61e0800194dd89be34d3aff3f374d in this term.
I20260709 12:30:46.787243 18886 consensus_queue.cc:260] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.789324 18886 ts_tablet_manager.cc:1434] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.000s
I20260709 12:30:46.790315 18886 tablet_bootstrap.cc:492] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:46.790853 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7089bc11f6424c178808e4579d967e14" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:46.793207 18731 leader_election.cc:304] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:46.794749 19118 raft_consensus.cc:3060] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.797407 18886 tablet_bootstrap.cc:654] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.800961 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4eead98f0854adfaf72daa76fdf9c9b" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:46.801630 18693 raft_consensus.cc:3060] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.804178 19059 consensus_queue.cc:1048] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:46.806129 18514 catalog_manager.cc:5697] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:46.808691 18926 raft_consensus.cc:493] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.809490 18926 raft_consensus.cc:515] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.812026 18926 leader_election.cc:290] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:46.819069 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2abc15bf7b445a4b4269e1d93cce949" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:46.819875 18840 raft_consensus.cc:2468] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:46.821588 18730 leader_election.cc:304] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:46.823009 19117 raft_consensus.cc:2804] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:46.823416 19117 raft_consensus.cc:493] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:46.823774 19117 raft_consensus.cc:3060] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.826258 18892 ts_tablet_manager.cc:1434] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.096s	user 0.009s	sys 0.001s
I20260709 12:30:46.827852 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2abc15bf7b445a4b4269e1d93cce949" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:46.835719 18888 tablet_bootstrap.cc:492] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:46.841360 18972 consensus_queue.cc:1048] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:46.836221 18690 raft_consensus.cc:2468] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:46.836971 19060 raft_consensus.cc:493] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.842374 19060 raft_consensus.cc:515] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:46.844542 19060 leader_election.cc:290] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:46.846179 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c4804407f8b43bd868a716c04e955e0" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:46.848343 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c4804407f8b43bd868a716c04e955e0" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:46.850116 18804 leader_election.cc:343] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:30:46.854208 18805 leader_election.cc:343] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.854779 18805 leader_election.cc:304] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:46.855273 18892 tablet_bootstrap.cc:492] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:46.856235 19060 raft_consensus.cc:2749] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:46.858675 19118 raft_consensus.cc:2749] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:46.862442 18693 raft_consensus.cc:2468] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:46.772773 19062 leader_election.cc:290] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:46.864503 18805 leader_election.cc:304] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:46.872079 18957 raft_consensus.cc:493] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df352ed4600b4ddb83ef57ecd2944fce)
I20260709 12:30:46.880246 18957 raft_consensus.cc:515] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.883896 18957 leader_election.cc:290] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:46.886430 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fddb762772a40acb39a71286c8d4f41" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:46.887274 19062 raft_consensus.cc:2804] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:46.888159 19062 raft_consensus.cc:697] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:46.889123 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fddb762772a40acb39a71286c8d4f41" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:46.891134 18581 leader_election.cc:304] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:46.892740 18957 raft_consensus.cc:2749] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:46.889235 19062 consensus_queue.cc:237] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.882155 19117 raft_consensus.cc:515] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.904824 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2abc15bf7b445a4b4269e1d93cce949" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:46.905524 18841 raft_consensus.cc:3060] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.907498 19117 leader_election.cc:290] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:46.916983 18693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2abc15bf7b445a4b4269e1d93cce949" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:46.917706 18693 raft_consensus.cc:3060] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.925402 18892 tablet_bootstrap.cc:654] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.925660 18693 raft_consensus.cc:2468] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:46.927964 18888 tablet_bootstrap.cc:654] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:46.930127 18731 leader_election.cc:304] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:46.931358 19118 raft_consensus.cc:2804] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:46.932133 19118 raft_consensus.cc:697] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:46.933071 19118 consensus_queue.cc:237] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.935734 19028 raft_consensus.cc:493] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.936424 19028 raft_consensus.cc:515] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:46.940790 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a147020c5d724ef2a43e17ee907fe200" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:46.943008 18657 leader_election.cc:343] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:46.945076 19028 leader_election.cc:290] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:46.947043 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a147020c5d724ef2a43e17ee907fe200" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:46.947945 18840 raft_consensus.cc:2468] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 0.
I20260709 12:30:46.949494 18656 leader_election.cc:304] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:30:46.951092 19028 raft_consensus.cc:2804] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:46.951661 19028 raft_consensus.cc:493] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:46.952150 19028 raft_consensus.cc:3060] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:46.916040 18841 raft_consensus.cc:2468] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:46.970458 19062 raft_consensus.cc:493] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.971468 19062 raft_consensus.cc:515] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.987491 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9e24af510de430aabe2a8c0c1010629" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
W20260709 12:30:46.989164 18804 leader_election.cc:343] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:30:46.972710 19154 log.cc:927] Time spent T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:30:46.980610 18965 raft_consensus.cc:493] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:46.990168 18965 raft_consensus.cc:515] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:46.995926 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9e24af510de430aabe2a8c0c1010629" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:46.993543 18965 leader_election.cc:290] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
W20260709 12:30:46.998665 18656 leader_election.cc:343] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:47.000958 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9e24af510de430aabe2a8c0c1010629" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:47.002043 18840 raft_consensus.cc:2468] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 0.
I20260709 12:30:47.003856 19062 leader_election.cc:290] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:46.990072 18888 tablet_bootstrap.cc:492] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:47.012075 18656 leader_election.cc:304] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:47.016372 18965 raft_consensus.cc:2804] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:47.016913 18965 raft_consensus.cc:493] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:47.017354 18965 raft_consensus.cc:3060] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:47.014374 18892 tablet_bootstrap.cc:492] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:47.014724 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9e24af510de430aabe2a8c0c1010629" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:47.023315 18892 ts_tablet_manager.cc:1403] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.186s	user 0.011s	sys 0.002s
I20260709 12:30:47.024035 18888 ts_tablet_manager.cc:1403] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.189s	user 0.013s	sys 0.001s
I20260709 12:30:47.020769 18512 catalog_manager.cc:5697] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:47.012166 19028 raft_consensus.cc:515] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:47.051196 18892 raft_consensus.cc:359] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:47.051859 18892 raft_consensus.cc:385] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:47.052053 18892 raft_consensus.cc:740] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.054850 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a147020c5d724ef2a43e17ee907fe200" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:47.055060 19028 leader_election.cc:290] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:47.055780 18888 raft_consensus.cc:359] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
W20260709 12:30:47.056461 18657 leader_election.cc:343] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:47.056770 18888 raft_consensus.cc:385] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:47.057206 18888 raft_consensus.cc:740] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.058017 18888 consensus_queue.cc:260] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:47.063884 18965 raft_consensus.cc:515] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:47.065438 18690 raft_consensus.cc:2393] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df352ed4600b4ddb83ef57ecd2944fce in current term 1: Already voted for candidate 60d61e0800194dd89be34d3aff3f374d in this term.
I20260709 12:30:47.063691 18515 catalog_manager.cc:5697] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:47.067880 18886 tablet_bootstrap.cc:492] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:47.068568 18887 tablet_bootstrap.cc:492] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:47.069185 18887 ts_tablet_manager.cc:1403] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.307s	user 0.034s	sys 0.028s
I20260709 12:30:47.060273 18888 ts_tablet_manager.cc:1434] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.035s	user 0.007s	sys 0.000s
I20260709 12:30:47.071781 18888 tablet_bootstrap.cc:492] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:47.074097 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9e24af510de430aabe2a8c0c1010629" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:47.068567 18886 ts_tablet_manager.cc:1403] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.278s	user 0.091s	sys 0.119s
I20260709 12:30:47.081518 18886 raft_consensus.cc:359] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:47.082058 18888 tablet_bootstrap.cc:654] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:47.082222 18656 leader_election.cc:343] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:47.082544 18886 raft_consensus.cc:385] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:30:47.083041 18730 consensus_peers.cc:597] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:47.083173 18886 raft_consensus.cc:740] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.084265 18886 consensus_queue.cc:260] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:47.085522 19030 consensus_queue.cc:1048] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:47.089609 18886 ts_tablet_manager.cc:1434] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:30:47.090893 18886 tablet_bootstrap.cc:492] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:47.072459 18887 raft_consensus.cc:359] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:47.092916 18887 raft_consensus.cc:385] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:47.093240 18887 raft_consensus.cc:740] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.096422 18805 leader_election.cc:304] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb
I20260709 12:30:47.100586 19062 raft_consensus.cc:3060] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:47.094705 18887 consensus_queue.cc:260] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:47.074133 18965 leader_election.cc:290] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:47.103142 18892 consensus_queue.cc:260] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:47.105435 18887 ts_tablet_manager.cc:1434] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.000s
I20260709 12:30:47.106817 18887 tablet_bootstrap.cc:492] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:47.116779 18892 ts_tablet_manager.cc:1434] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.092s	user 0.008s	sys 0.000s
I20260709 12:30:47.121048 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a147020c5d724ef2a43e17ee907fe200" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:47.121776 18841 raft_consensus.cc:3060] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:47.122392 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9e24af510de430aabe2a8c0c1010629" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:47.127126 19062 raft_consensus.cc:2749] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:47.130226 18888 tablet_bootstrap.cc:492] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:47.130520 19118 raft_consensus.cc:493] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:47.131264 19118 raft_consensus.cc:515] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:47.134438 19118 leader_election.cc:290] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:47.136092 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c4804407f8b43bd868a716c04e955e0" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:47.136695 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c4804407f8b43bd868a716c04e955e0" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:47.137354 18838 raft_consensus.cc:2468] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
W20260709 12:30:47.137820 18730 leader_election.cc:343] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:47.138619 18730 leader_election.cc:304] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:47.139921 19118 raft_consensus.cc:2804] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:47.140334 19118 raft_consensus.cc:493] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:47.140712 19118 raft_consensus.cc:3060] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:47.142958 18841 raft_consensus.cc:2468] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:47.136560 18888 ts_tablet_manager.cc:1403] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.065s	user 0.015s	sys 0.012s
I20260709 12:30:47.144574 19030 raft_consensus.cc:493] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:47.145407 19030 raft_consensus.cc:515] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:47.148275 18888 raft_consensus.cc:359] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:47.157648 18888 raft_consensus.cc:385] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:47.158068 18888 raft_consensus.cc:740] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.151324 18887 tablet_bootstrap.cc:654] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:47.162940 18892 tablet_bootstrap.cc:492] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:47.156409 18886 tablet_bootstrap.cc:654] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:47.159067 18888 consensus_queue.cc:260] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:47.150144 18656 leader_election.cc:304] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:30:47.177408 18888 ts_tablet_manager.cc:1434] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.033s	user 0.004s	sys 0.004s
I20260709 12:30:47.148774 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c6af67983744957af1aee8d948169c2" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:47.186581 18888 tablet_bootstrap.cc:492] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:47.155684 19030 leader_election.cc:290] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:47.193079 19030 raft_consensus.cc:2804] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:47.193780 19030 raft_consensus.cc:697] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:30:47.189904 18892 tablet_bootstrap.cc:654] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:47.194900 19030 consensus_queue.cc:237] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:47.215575 19118 raft_consensus.cc:515] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:47.229776 18840 raft_consensus.cc:2468] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:47.238186 19023 raft_consensus.cc:493] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:30:47.230178 18657 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:40335, user_credentials={real_user=slave}} blocked reactor thread for 75391.8us
W20260709 12:30:47.248306 18656 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:44517, user_credentials={real_user=slave}} blocked reactor thread for 98284.3us
I20260709 12:30:47.253188 19118 leader_election.cc:290] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:47.255201 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c4804407f8b43bd868a716c04e955e0" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:47.255280 19023 raft_consensus.cc:515] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:47.257300 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c6af67983744957af1aee8d948169c2" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:47.257829 18656 leader_election.cc:343] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:47.259610 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:47.260622 18765 raft_consensus.cc:2393] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84812365c65f43f388836c087640adbb in current term 1: Already voted for candidate 29f82a8594d741bc92dd24ee4a368e01 in this term.
W20260709 12:30:47.261482 18730 leader_election.cc:343] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:47.262025 19117 consensus_queue.cc:1048] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:47.257702 19023 leader_election.cc:290] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:47.266019 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c4804407f8b43bd868a716c04e955e0" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:47.266741 18839 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
W20260709 12:30:47.272213 18657 leader_election.cc:343] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:47.272948 18657 leader_election.cc:304] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:47.274665 19028 raft_consensus.cc:2749] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:47.266800 18841 raft_consensus.cc:3060] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:47.275758 19028 consensus_queue.cc:1048] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:47.277846 18839 raft_consensus.cc:2393] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84812365c65f43f388836c087640adbb in current term 1: Already voted for candidate df352ed4600b4ddb83ef57ecd2944fce in this term.
I20260709 12:30:47.279733 18888 tablet_bootstrap.cc:654] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:47.280936 18581 leader_election.cc:304] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:47.286425 19023 raft_consensus.cc:3060] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:47.277910 19028 consensus_queue.cc:1048] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260709 12:30:47.300311 19023 raft_consensus.cc:2749] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:47.320274 19059 raft_consensus.cc:493] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:47.321071 19059 raft_consensus.cc:515] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:47.327155 19023 raft_consensus.cc:493] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:47.327291 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16afc1d74df849c48d10f5da0a8f80c4" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:47.327903 19023 raft_consensus.cc:515] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:47.329901 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16afc1d74df849c48d10f5da0a8f80c4" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
W20260709 12:30:47.332909 18804 leader_election.cc:343] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:47.334441 19023 leader_election.cc:290] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
W20260709 12:30:47.335716 18805 leader_election.cc:343] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:47.336498 18805 leader_election.cc:304] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:47.337909 19062 raft_consensus.cc:2749] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:47.339251 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e3d64db91924be490d4ed1f797d338f" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:47.358747 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e3d64db91924be490d4ed1f797d338f" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:47.360955 18887 tablet_bootstrap.cc:492] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:47.361248 18886 tablet_bootstrap.cc:492] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:47.361595 18887 ts_tablet_manager.cc:1403] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.255s	user 0.015s	sys 0.018s
I20260709 12:30:47.361725 18886 ts_tablet_manager.cc:1403] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.271s	user 0.019s	sys 0.026s
I20260709 12:30:47.329705 19059 leader_election.cc:290] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:47.363682 18656 leader_election.cc:304] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:47.368494 18886 raft_consensus.cc:359] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:47.372699 18886 raft_consensus.cc:385] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:47.373440 18886 raft_consensus.cc:740] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.374859 18886 consensus_queue.cc:260] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:47.372128 18965 raft_consensus.cc:2804] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:47.382233 18618 raft_consensus.cc:3060] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:47.382514 18965 raft_consensus.cc:697] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:30:47.385087 18887 raft_consensus.cc:359] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:47.388101 18887 raft_consensus.cc:385] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:47.388756 18887 raft_consensus.cc:740] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.389854 18887 consensus_queue.cc:260] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:47.397379 18886 ts_tablet_manager.cc:1434] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.035s	user 0.009s	sys 0.000s
I20260709 12:30:47.398319 18886 tablet_bootstrap.cc:492] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:47.410349 18887 ts_tablet_manager.cc:1434] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.046s	user 0.008s	sys 0.001s
I20260709 12:30:47.411520 18887 tablet_bootstrap.cc:492] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:47.387117 19059 consensus_queue.cc:1048] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
W20260709 12:30:47.432468 18804 consensus_peers.cc:597] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:47.418962 18965 consensus_queue.cc:237] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:47.416289 19118 raft_consensus.cc:493] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:47.444936 19118 raft_consensus.cc:515] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:47.447461 18512 catalog_manager.cc:5697] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:47.449378 19118 leader_election.cc:290] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:47.451176 18839 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a147020c5d724ef2a43e17ee907fe200" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:47.452034 18839 raft_consensus.cc:2393] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 29f82a8594d741bc92dd24ee4a368e01 in current term 1: Already voted for candidate 60d61e0800194dd89be34d3aff3f374d in this term.
I20260709 12:30:47.414430 18582 leader_election.cc:304] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
W20260709 12:30:47.453078 18582 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 45882.3us
I20260709 12:30:47.456718 18892 tablet_bootstrap.cc:492] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:47.457206 18892 ts_tablet_manager.cc:1403] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.295s	user 0.013s	sys 0.017s
I20260709 12:30:47.457408 18841 raft_consensus.cc:2468] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:47.459646 19023 raft_consensus.cc:2749] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:47.462090 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a147020c5d724ef2a43e17ee907fe200" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:47.460044 18892 raft_consensus.cc:359] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:47.463613 18892 raft_consensus.cc:385] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:47.463967 18892 raft_consensus.cc:740] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.464865 18892 consensus_queue.cc:260] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:47.465252 18730 leader_election.cc:304] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
W20260709 12:30:47.480827 19169 log.cc:927] Time spent T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:30:47.482613 19030 consensus_queue.cc:1048] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:47.482856 18887 tablet_bootstrap.cc:654] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:47.495693 18892 ts_tablet_manager.cc:1434] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.038s	user 0.008s	sys 0.000s
I20260709 12:30:47.496795 18888 tablet_bootstrap.cc:492] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:47.497290 18888 ts_tablet_manager.cc:1403] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.319s	user 0.027s	sys 0.020s
I20260709 12:30:47.498283 18926 raft_consensus.cc:2804] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:47.498860 18926 raft_consensus.cc:697] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:47.499745 18926 consensus_queue.cc:237] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:47.509558 18886 tablet_bootstrap.cc:654] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:47.493983 18731 leader_election.cc:304] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
W20260709 12:30:47.519884 18730 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:44517, user_credentials={real_user=slave}} blocked reactor thread for 54757.1us
I20260709 12:30:47.533483 19178 raft_consensus.cc:3060] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:47.536639 18892 tablet_bootstrap.cc:492] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:47.539149 19028 raft_consensus.cc:493] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:47.539728 19028 raft_consensus.cc:515] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:47.541834 19028 leader_election.cc:290] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:47.547379 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eed637e5890430db2e802ef6d4362a7" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:47.549295 18657 leader_election.cc:343] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:47.557771 18892 tablet_bootstrap.cc:654] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:47.558385 19178 raft_consensus.cc:2749] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:47.568224 18618 raft_consensus.cc:3060] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:47.570082 18892 tablet_bootstrap.cc:492] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:47.571856 18892 ts_tablet_manager.cc:1403] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.042s	user 0.010s	sys 0.006s
I20260709 12:30:47.577922 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eed637e5890430db2e802ef6d4362a7" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:47.580108 18892 raft_consensus.cc:359] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:47.581149 18892 raft_consensus.cc:385] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:47.581714 18892 raft_consensus.cc:740] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.582820 18892 consensus_queue.cc:260] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:47.581580 18512 catalog_manager.cc:5697] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:47.603241 18888 raft_consensus.cc:359] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:47.604032 18888 raft_consensus.cc:385] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:47.604305 18888 raft_consensus.cc:740] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.606127 18888 consensus_queue.cc:260] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
W20260709 12:30:47.607551 18656 leader_election.cc:343] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:47.608031 18656 leader_election.cc:304] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:47.611356 19030 raft_consensus.cc:2749] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:47.621165 18892 ts_tablet_manager.cc:1434] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.049s	user 0.014s	sys 0.007s
I20260709 12:30:47.612144 18926 consensus_queue.cc:1048] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 12:30:47.627661 18888 ts_tablet_manager.cc:1434] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.130s	user 0.004s	sys 0.004s
I20260709 12:30:47.628942 18888 tablet_bootstrap.cc:492] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:47.635483 18888 tablet_bootstrap.cc:654] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:47.636209 18563 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:30:47.646143 18892 tablet_bootstrap.cc:492] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:47.655158 18886 tablet_bootstrap.cc:492] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:47.655838 18886 ts_tablet_manager.cc:1403] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.258s	user 0.021s	sys 0.003s
I20260709 12:30:47.659600 18926 consensus_queue.cc:1048] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:47.662822 18886 raft_consensus.cc:359] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:47.663579 18886 raft_consensus.cc:385] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:47.663857 18886 raft_consensus.cc:740] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.664026 18892 tablet_bootstrap.cc:654] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:47.664589 18886 consensus_queue.cc:260] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:47.669209 18887 tablet_bootstrap.cc:492] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:47.671101 18887 ts_tablet_manager.cc:1403] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.260s	user 0.019s	sys 0.013s
I20260709 12:30:47.684798 18886 ts_tablet_manager.cc:1434] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.000s
I20260709 12:30:47.685911 18886 tablet_bootstrap.cc:492] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:47.692312 18887 raft_consensus.cc:359] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:47.696615 18887 raft_consensus.cc:385] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:47.696965 18887 raft_consensus.cc:740] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.697777 18887 consensus_queue.cc:260] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:47.699769 18886 tablet_bootstrap.cc:654] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:47.702821 18887 ts_tablet_manager.cc:1434] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.031s	user 0.008s	sys 0.000s
I20260709 12:30:47.708247 18887 tablet_bootstrap.cc:492] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:47.714844 18887 tablet_bootstrap.cc:654] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:47.715121 19059 raft_consensus.cc:493] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:47.722389 19059 raft_consensus.cc:515] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:47.738410 19028 consensus_queue.cc:1048] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:47.750519 19059 leader_election.cc:290] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:47.754022 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4794853627e84898986305827ec4ea46" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:47.765192 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4794853627e84898986305827ec4ea46" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:47.766187 18691 raft_consensus.cc:2468] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:47.767005 18888 tablet_bootstrap.cc:492] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:47.767534 18888 ts_tablet_manager.cc:1403] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.139s	user 0.016s	sys 0.000s
I20260709 12:30:47.768528 18805 leader_election.cc:304] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:47.769593 19060 raft_consensus.cc:2804] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:47.770012 19060 raft_consensus.cc:493] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:47.770474 19060 raft_consensus.cc:3060] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:47.756924 18512 catalog_manager.cc:5697] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:47.770138 18892 tablet_bootstrap.cc:492] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
W20260709 12:30:47.772115 18804 leader_election.cc:343] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:47.772878 18892 ts_tablet_manager.cc:1403] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.128s	user 0.023s	sys 0.010s
I20260709 12:30:47.783391 18888 raft_consensus.cc:359] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:47.784112 18888 raft_consensus.cc:385] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:47.789875 18892 raft_consensus.cc:359] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:47.790818 18892 raft_consensus.cc:385] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:47.791082 18892 raft_consensus.cc:740] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.791390 18886 tablet_bootstrap.cc:492] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:47.791955 18886 ts_tablet_manager.cc:1403] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.106s	user 0.017s	sys 0.003s
I20260709 12:30:47.794938 18886 raft_consensus.cc:359] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:47.795691 18886 raft_consensus.cc:385] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:47.795969 18886 raft_consensus.cc:740] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.796877 18886 consensus_queue.cc:260] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:47.799255 18886 ts_tablet_manager.cc:1434] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:30:47.800467 18886 tablet_bootstrap.cc:492] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:47.791939 18892 consensus_queue.cc:260] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:47.807729 18892 ts_tablet_manager.cc:1434] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.035s	user 0.003s	sys 0.002s
I20260709 12:30:47.808923 18892 tablet_bootstrap.cc:492] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:47.810675 19062 raft_consensus.cc:493] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:47.811206 19062 raft_consensus.cc:515] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:47.813268 18886 tablet_bootstrap.cc:654] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:47.815356 19062 leader_election.cc:290] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:47.787469 18888 raft_consensus.cc:740] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:47.818393 18888 consensus_queue.cc:260] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:47.849982 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eed637e5890430db2e802ef6d4362a7" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:47.851660 18691 raft_consensus.cc:2468] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:47.851748 18892 tablet_bootstrap.cc:654] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:47.858789 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eed637e5890430db2e802ef6d4362a7" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:47.860906 18805 leader_election.cc:343] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:47.862336 18888 ts_tablet_manager.cc:1434] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.094s	user 0.008s	sys 0.000s
I20260709 12:30:47.868605 18888 tablet_bootstrap.cc:492] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:47.872462 19060 raft_consensus.cc:515] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:47.891458 18888 tablet_bootstrap.cc:654] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:47.896938 18805 leader_election.cc:304] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:30:47.898108 19062 raft_consensus.cc:2804] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:47.898519 19062 raft_consensus.cc:493] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:47.898847 19062 raft_consensus.cc:3060] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:47.921816 18887 tablet_bootstrap.cc:492] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:47.922493 18887 ts_tablet_manager.cc:1403] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.219s	user 0.034s	sys 0.014s
I20260709 12:30:47.931805 18892 tablet_bootstrap.cc:492] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:47.974069 18892 ts_tablet_manager.cc:1403] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.165s	user 0.006s	sys 0.021s
I20260709 12:30:48.102751 19062 raft_consensus.cc:515] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:48.128136 19062 leader_election.cc:290] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:48.160271 18886 tablet_bootstrap.cc:492] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:48.161067 18886 ts_tablet_manager.cc:1403] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.361s	user 0.021s	sys 0.032s
I20260709 12:30:48.162418 18887 raft_consensus.cc:359] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:48.163331 18887 raft_consensus.cc:385] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:48.163697 18887 raft_consensus.cc:740] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:48.164525 18887 consensus_queue.cc:260] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:48.166407 18887 ts_tablet_manager.cc:1434] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.244s	user 0.013s	sys 0.000s
I20260709 12:30:48.162963 18892 raft_consensus.cc:359] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:48.169653 18888 tablet_bootstrap.cc:492] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:48.170251 18888 ts_tablet_manager.cc:1403] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.307s	user 0.012s	sys 0.009s
I20260709 12:30:48.173631 18888 raft_consensus.cc:359] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:48.243429 18892 raft_consensus.cc:385] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:48.243968 18892 raft_consensus.cc:740] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:48.244791 18892 consensus_queue.cc:260] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:48.174525 18888 raft_consensus.cc:385] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:48.246430 18888 raft_consensus.cc:740] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:48.246860 18892 ts_tablet_manager.cc:1434] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.272s	user 0.009s	sys 0.010s
I20260709 12:30:48.247187 18888 consensus_queue.cc:260] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:48.249465 18888 ts_tablet_manager.cc:1434] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.079s	user 0.009s	sys 0.000s
I20260709 12:30:48.250522 18888 tablet_bootstrap.cc:492] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:48.257522 18888 tablet_bootstrap.cc:654] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:48.262943 18886 raft_consensus.cc:359] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:48.382530 18886 raft_consensus.cc:385] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:48.382882 18886 raft_consensus.cc:740] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:48.385830 18886 consensus_queue.cc:260] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:48.389664 18887 tablet_bootstrap.cc:492] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:48.391952 18886 ts_tablet_manager.cc:1434] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.230s	user 0.006s	sys 0.001s
I20260709 12:30:48.393230 18886 tablet_bootstrap.cc:492] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:48.406388 18888 tablet_bootstrap.cc:492] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:48.418044 18887 tablet_bootstrap.cc:654] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:48.436308 18730 consensus_peers.cc:597] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:48.438711 18886 tablet_bootstrap.cc:654] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:48.417729 19182 raft_consensus.cc:493] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:48.439685 19182 raft_consensus.cc:515] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:48.441298 18888 ts_tablet_manager.cc:1403] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.191s	user 0.018s	sys 0.007s
I20260709 12:30:48.441955 19182 leader_election.cc:290] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:48.444417 18615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:48.445277 18615 raft_consensus.cc:2468] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:48.446939 18730 leader_election.cc:304] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:30:48.447881 19198 raft_consensus.cc:2804] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:30:48.448374 19198 raft_consensus.cc:493] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:48.444907 18888 raft_consensus.cc:359] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:48.448900 19198 raft_consensus.cc:3060] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:48.449266 18888 raft_consensus.cc:385] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:48.449719 18888 raft_consensus.cc:740] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:48.450932 18957 raft_consensus.cc:493] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:48.451473 18957 raft_consensus.cc:515] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:48.453601 18957 leader_election.cc:290] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:48.453020 18888 consensus_queue.cc:260] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:48.457719 18888 ts_tablet_manager.cc:1434] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 12:30:48.459124 18888 tablet_bootstrap.cc:492] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:48.463070 19198 raft_consensus.cc:515] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:48.465481 19198 leader_election.cc:290] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:48.466034 19021 raft_consensus.cc:493] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:48.466501 19021 raft_consensus.cc:515] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:48.466799 18615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:48.467372 18615 raft_consensus.cc:3060] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:48.482520 18888 tablet_bootstrap.cc:654] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:48.485275 19198 raft_consensus.cc:493] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:48.490947 19198 raft_consensus.cc:515] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:48.493453 19198 leader_election.cc:290] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:48.487773 18615 raft_consensus.cc:2468] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 2.
I20260709 12:30:48.494726 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16afc1d74df849c48d10f5da0a8f80c4" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:48.486843 19021 leader_election.cc:290] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:48.497067 18730 leader_election.cc:304] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 
W20260709 12:30:48.498159 18730 leader_election.cc:343] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:48.505429 19205 raft_consensus.cc:493] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:48.506019 19205 raft_consensus.cc:515] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:48.510456 19198 raft_consensus.cc:2804] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:30:48.511243 19198 raft_consensus.cc:697] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:48.512382 19198 consensus_queue.cc:237] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:48.515269 19205 leader_election.cc:290] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:48.521391 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba28ff472b0f46e8884f70afb21857ae" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:48.555742 19060 leader_election.cc:290] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:48.564759 18887 tablet_bootstrap.cc:492] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:48.588634 18887 ts_tablet_manager.cc:1403] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.199s	user 0.026s	sys 0.017s
W20260709 12:30:48.565229 18657 leader_election.cc:343] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:48.570102 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4794853627e84898986305827ec4ea46" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:48.576781 18839 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:48.595558 18888 tablet_bootstrap.cc:492] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:48.596342 18888 ts_tablet_manager.cc:1403] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.138s	user 0.045s	sys 0.023s
I20260709 12:30:48.598317 18839 raft_consensus.cc:2468] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:48.603397 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eed637e5890430db2e802ef6d4362a7" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:48.604223 18691 raft_consensus.cc:3060] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:48.598662 19178 raft_consensus.cc:493] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:48.582850 18764 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eed637e5890430db2e802ef6d4362a7" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:48.583632 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:48.611599 18840 raft_consensus.cc:3060] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:48.584407 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16afc1d74df849c48d10f5da0a8f80c4" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:48.617038 18841 raft_consensus.cc:2468] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:48.586746 18886 tablet_bootstrap.cc:492] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:48.618932 18886 ts_tablet_manager.cc:1403] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.226s	user 0.044s	sys 0.052s
I20260709 12:30:48.606961 19178 raft_consensus.cc:515] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:48.594868 19182 consensus_queue.cc:1048] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:30:48.600203 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4794853627e84898986305827ec4ea46" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:48.623337 18886 raft_consensus.cc:359] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:48.626704 18886 raft_consensus.cc:385] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:48.627172 18886 raft_consensus.cc:740] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:48.630234 18886 consensus_queue.cc:260] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:48.633720 18886 ts_tablet_manager.cc:1434] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.001s
I20260709 12:30:48.638016 18888 raft_consensus.cc:359] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:48.639284 18888 raft_consensus.cc:385] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:48.639608 18888 raft_consensus.cc:740] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:48.630477 18692 raft_consensus.cc:3060] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:48.640538 18888 consensus_queue.cc:260] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:48.637981 18886 tablet_bootstrap.cc:492] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:48.643083 18764 raft_consensus.cc:3060] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:30:48.758246 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 135282us
I20260709 12:30:48.777526 18888 ts_tablet_manager.cc:1434] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.181s	user 0.010s	sys 0.005s
I20260709 12:30:48.773589 19221 raft_consensus.cc:493] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:48.773825 18691 raft_consensus.cc:2468] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:48.780427 18839 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9e24af510de430aabe2a8c0c1010629" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:48.782521 19204 raft_consensus.cc:493] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29f82a8594d741bc92dd24ee4a368e01)
I20260709 12:30:48.772261 19207 raft_consensus.cc:493] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:30:48.772423 18804 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 185548us
I20260709 12:30:48.793467 18764 raft_consensus.cc:2468] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:48.801349 18839 raft_consensus.cc:2393] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84812365c65f43f388836c087640adbb in current term 1: Already voted for candidate 60d61e0800194dd89be34d3aff3f374d in this term.
I20260709 12:30:48.809954 18887 raft_consensus.cc:359] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:48.810926 18887 raft_consensus.cc:385] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:48.811240 18887 raft_consensus.cc:740] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:48.812029 18887 consensus_queue.cc:260] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:48.801558 19204 raft_consensus.cc:515] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:48.800679 18692 raft_consensus.cc:2468] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:48.821175 18887 ts_tablet_manager.cc:1434] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.232s	user 0.006s	sys 0.001s
I20260709 12:30:48.822266 18887 tablet_bootstrap.cc:492] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:48.815464 18840 raft_consensus.cc:2468] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 2.
I20260709 12:30:48.797679 19221 raft_consensus.cc:515] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:48.821106 19062 raft_consensus.cc:493] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d61e0800194dd89be34d3aff3f374d)
I20260709 12:30:48.824209 19215 raft_consensus.cc:493] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:48.813514 19207 raft_consensus.cc:515] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:48.836864 19062 raft_consensus.cc:515] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:48.829671 19178 leader_election.cc:290] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
W20260709 12:30:48.823175 18804 leader_election.cc:343] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:30:48.936714 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:40335, user_credentials={real_user=slave}} blocked reactor thread for 171202us
I20260709 12:30:48.938225 18892 tablet_bootstrap.cc:492] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:48.943203 18886 tablet_bootstrap.cc:654] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:48.940054 18957 raft_consensus.cc:493] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df352ed4600b4ddb83ef57ecd2944fce)
I20260709 12:30:48.964785 18957 raft_consensus.cc:515] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:48.968726 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9e24af510de430aabe2a8c0c1010629" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:48.939800 19167 raft_consensus.cc:493] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29f82a8594d741bc92dd24ee4a368e01)
I20260709 12:30:49.091709 19167 raft_consensus.cc:515] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:48.970388 19221 leader_election.cc:290] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:48.939657 19215 raft_consensus.cc:515] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:48.971283 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11b1d6d5a5c4991bd34743fff5ca507" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
W20260709 12:30:48.942057 18581 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:44517, user_credentials={real_user=slave}} blocked reactor thread for 110566us
I20260709 12:30:48.973752 19204 leader_election.cc:290] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:48.980715 19207 leader_election.cc:290] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
W20260709 12:30:48.941713 18804 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 118440us
I20260709 12:30:49.095392 19215 leader_election.cc:290] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:48.984138 18805 leader_election.cc:304] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
W20260709 12:30:49.096185 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 112191us
I20260709 12:30:49.096824 18805 leader_election.cc:304] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:49.098024 19210 raft_consensus.cc:2804] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:48.997995 19205 raft_consensus.cc:493] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29f82a8594d741bc92dd24ee4a368e01)
I20260709 12:30:49.098958 19205 raft_consensus.cc:515] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:49.100114 19217 raft_consensus.cc:2804] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:49.101415 19205 leader_election.cc:290] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:49.102550 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f16818c4dfd4e9ab187ffb53d8fc3ce" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:49.011943 18618 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:53592 (request call id 46) took 1153 ms. Trace:
I20260709 12:30:49.104416 18618 rpcz_store.cc:276] 0709 12:30:47.858154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:30:49.010954 (+1152800us) service_pool.cc:224] Handling call
0709 12:30:49.011369 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:30:49.011573 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:30:49.011603 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:30:49.011650 (+    47us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:30:49.011688 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:30:49.011757 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:30:49.011877 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:30:49.106014 18609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee74ad2fe774e0aa780b1fdf18b3003" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:49.109479 19167 leader_election.cc:290] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:49.015712 19227 raft_consensus.cc:493] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df352ed4600b4ddb83ef57ecd2944fce)
I20260709 12:30:49.112644 19227 raft_consensus.cc:515] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:49.114991 19227 leader_election.cc:290] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:49.115075 18609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f9d88127114fda91668ab529abd23c" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:49.093590 19198 raft_consensus.cc:493] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:49.116289 19198 raft_consensus.cc:515] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:49.118350 19198 leader_election.cc:290] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:49.124147 19210 raft_consensus.cc:697] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:49.124855 19062 leader_election.cc:290] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:49.124790 18615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb0c96994864b03a8ec7cda3cc8d85b" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:49.125303 19210 consensus_queue.cc:237] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:49.127521 18615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ca4b75eb0e49dca52e1b9fb4f8b356" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:49.128796 18609 raft_consensus.cc:1275] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Refusing update from remote peer 29f82a8594d741bc92dd24ee4a368e01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:30:49.129022 18611 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fddb762772a40acb39a71286c8d4f41" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:49.129628 18514 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43926 (request call id 64) took 1320 ms. Trace:
I20260709 12:30:49.129994 18514 rpcz_store.cc:276] 0709 12:30:47.809168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:30:48.993323 (+1184155us) service_pool.cc:224] Handling call
0709 12:30:49.129563 (+136240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":76191,"reported_tablets":1}
I20260709 12:30:49.023356 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f651b7eda3134c7a897855ac4300de2f" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
W20260709 12:30:49.137763 18563 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:30:49.021286 18892 tablet_bootstrap.cc:654] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:49.028065 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11b1d6d5a5c4991bd34743fff5ca507" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:49.029277 18582 leader_election.cc:304] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:49.032078 19228 raft_consensus.cc:493] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29f82a8594d741bc92dd24ee4a368e01)
I20260709 12:30:49.036603 19231 consensus_queue.cc:1048] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:49.040452 18613 raft_consensus.cc:3060] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.043247 19232 raft_consensus.cc:493] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29f82a8594d741bc92dd24ee4a368e01)
I20260709 12:30:49.045874 18887 tablet_bootstrap.cc:654] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:49.054502 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c6af67983744957af1aee8d948169c2" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:49.061375 19060 raft_consensus.cc:493] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d61e0800194dd89be34d3aff3f374d)
I20260709 12:30:49.067232 18957 leader_election.cc:290] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:49.086414 18685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eed637e5890430db2e802ef6d4362a7" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:49.086953 18512 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43928 (request call id 70) took 1419 ms. Trace:
I20260709 12:30:49.087412 18684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11b1d6d5a5c4991bd34743fff5ca507" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
W20260709 12:30:49.088382 19212 log.cc:927] Time spent T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01: Append to log took a long time: real 0.319s	user 0.000s	sys 0.004s
W20260709 12:30:49.158102 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 59749.3us
I20260709 12:30:49.094257 18617 raft_consensus.cc:2468] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:49.127591 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f651b7eda3134c7a897855ac4300de2f" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:49.122735 18730 leader_election.cc:304] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:49.024679 19229 raft_consensus.cc:493] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d61e0800194dd89be34d3aff3f374d)
I20260709 12:30:49.139659 19217 raft_consensus.cc:697] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:49.141911 19215 raft_consensus.cc:493] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d61e0800194dd89be34d3aff3f374d)
W20260709 12:30:49.166759 18730 leader_election.cc:343] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:49.093588 19226 raft_consensus.cc:493] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:49.152285 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb0c96994864b03a8ec7cda3cc8d85b" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:49.164611 18886 tablet_bootstrap.cc:492] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:49.169749 19215 raft_consensus.cc:515] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:49.169930 19217 consensus_queue.cc:237] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:49.156735 18689 raft_consensus.cc:2468] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 0.
I20260709 12:30:49.170576 18685 raft_consensus.cc:2393] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 29f82a8594d741bc92dd24ee4a368e01 in current term 1: Already voted for candidate df352ed4600b4ddb83ef57ecd2944fce in this term.
I20260709 12:30:49.156461 19060 raft_consensus.cc:515] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:49.146169 18690 raft_consensus.cc:2468] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 0.
W20260709 12:30:49.178155 18730 leader_election.cc:343] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:49.177688 18886 ts_tablet_manager.cc:1403] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.541s	user 0.034s	sys 0.038s
I20260709 12:30:49.180006 19239 raft_consensus.cc:493] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:49.168661 18839 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3eed637e5890430db2e802ef6d4362a7" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:49.190462 18730 leader_election.cc:304] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:30:49.201479 19060 leader_election.cc:290] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:49.186937 18838 raft_consensus.cc:2468] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:49.189404 19200 raft_consensus.cc:2804] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:49.146445 19228 raft_consensus.cc:515] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:49.209300 19200 raft_consensus.cc:493] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:49.209721 19200 raft_consensus.cc:3060] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.210945 19228 leader_election.cc:290] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:49.167908 19229 raft_consensus.cc:515] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:49.205668 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee74ad2fe774e0aa780b1fdf18b3003" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:49.208638 19222 raft_consensus.cc:2804] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:49.148918 19232 raft_consensus.cc:515] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:49.178417 19226 raft_consensus.cc:515] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:49.223323 19232 leader_election.cc:290] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:49.224803 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ca4b75eb0e49dca52e1b9fb4f8b356" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
W20260709 12:30:49.227105 18582 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 197982us
I20260709 12:30:49.227824 19226 leader_election.cc:290] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:49.216868 19222 raft_consensus.cc:493] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:49.230901 19222 raft_consensus.cc:3060] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.235280 19223 consensus_queue.cc:1048] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.042s
I20260709 12:30:49.235419 19200 raft_consensus.cc:515] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:49.237077 19021 raft_consensus.cc:2749] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:49.238325 19200 leader_election.cc:290] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:49.223814 18886 raft_consensus.cc:359] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:49.201581 19215 leader_election.cc:290] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:49.246301 18886 raft_consensus.cc:385] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:49.246693 18886 raft_consensus.cc:740] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:49.248827 18886 consensus_queue.cc:260] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:49.252029 18688 raft_consensus.cc:2468] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:49.254930 19229 leader_election.cc:290] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:49.256781 18886 ts_tablet_manager.cc:1434] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.066s	user 0.007s	sys 0.000s
I20260709 12:30:49.257900 18886 tablet_bootstrap.cc:492] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:49.197459 19239 raft_consensus.cc:515] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:49.269164 18613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16afc1d74df849c48d10f5da0a8f80c4" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:49.269727 18839 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba28ff472b0f46e8884f70afb21857ae" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
W20260709 12:30:49.270880 18730 leader_election.cc:343] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:49.276073 18730 leader_election.cc:304] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:49.281147 19178 raft_consensus.cc:2749] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:49.290310 19239 leader_election.cc:290] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:49.290832 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7089bc11f6424c178808e4579d967e14" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:49.296478 18888 tablet_bootstrap.cc:492] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:49.300673 18762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c6af67983744957af1aee8d948169c2" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.301800 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6256d41aab434828a66720ab61fa9ba5" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:49.302515 19232 consensus_queue.cc:1048] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:49.306645 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f9d88127114fda91668ab529abd23c" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.307965 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f651b7eda3134c7a897855ac4300de2f" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.309206 18763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f16818c4dfd4e9ab187ffb53d8fc3ce" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.310796 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f651b7eda3134c7a897855ac4300de2f" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.314158 18886 tablet_bootstrap.cc:654] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:49.302644 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11b1d6d5a5c4991bd34743fff5ca507" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.316247 18730 leader_election.cc:304] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:49.158629 18512 rpcz_store.cc:276] 0709 12:30:47.667466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:30:47.673292 (+  5826us) service_pool.cc:224] Handling call
0709 12:30:48.398713 (+725421us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:48.398745 (+    32us) write_op.cc:454] Released schema lock
0709 12:30:49.086869 (+688124us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:30:47.784467 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:30:47.785070 (+   603us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:30:47.785105 (+    35us) write_op.cc:435] Acquired schema lock
0709 12:30:47.785118 (+    13us) tablet.cc:662] Decoding operations
0709 12:30:47.785439 (+   321us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:30:47.785496 (+    57us) write_op.cc:641] Partition lock acquired for write op
0709 12:30:47.785524 (+    28us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:30:47.785885 (+   361us) tablet.cc:701] Row locks acquired
0709 12:30:47.785904 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:30:47.786156 (+   252us) write_op.cc:270] Start()
0709 12:30:47.786261 (+   105us) write_op.cc:276] Timestamp: P: 1783600247786079 usec, L: 0
0709 12:30:47.786281 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 12:30:47.792565 (+  6284us) log.cc:844] Serialized 531 byte log entry
0709 12:30:47.794618 (+  2053us) op_driver.cc:464] REPLICATION: finished
0709 12:30:47.795074 (+   456us) write_op.cc:301] APPLY: starting
0709 12:30:47.795169 (+    95us) tablet.cc:1367] starting BulkCheckPresence
0709 12:30:47.795268 (+    99us) tablet.cc:1370] finished BulkCheckPresence
0709 12:30:47.795370 (+   102us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:30:47.795563 (+   193us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:30:47.795740 (+   177us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:30:47.795783 (+    43us) write_op.cc:312] APPLY: finished
0709 12:30:47.795963 (+   180us) log.cc:844] Serialized 22 byte log entry
0709 12:30:47.796114 (+   151us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:47.796204 (+    90us) write_op.cc:454] Released schema lock
0709 12:30:47.796244 (+    40us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":5434,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":239,"apply.run_cpu_time_us":1463,"apply.run_wall_time_us":1463,"num_ops":1,"prepare.queue_time_us":129,"prepare.run_cpu_time_us":2796,"prepare.run_wall_time_us":8409,"replication_time_us":8223}]]}
I20260709 12:30:49.325805 19234 raft_consensus.cc:493] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29f82a8594d741bc92dd24ee4a368e01)
I20260709 12:30:49.328847 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fddb762772a40acb39a71286c8d4f41" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:49.321627 19223 raft_consensus.cc:2804] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:30:49.322715 18656 leader_election.cc:343] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:49.350854 19223 raft_consensus.cc:493] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:49.352953 18839 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee74ad2fe774e0aa780b1fdf18b3003" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:49.340729 19234 raft_consensus.cc:515] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:49.353283 18836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acf997d23744fe697db1fe947dca883" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
W20260709 12:30:49.326349 18582 leader_election.cc:343] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:49.354373 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6256d41aab434828a66720ab61fa9ba5" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:49.355643 18887 tablet_bootstrap.cc:492] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:49.359922 18838 raft_consensus.cc:2468] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 0.
I20260709 12:30:49.356927 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7089bc11f6424c178808e4579d967e14" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:49.360059 18887 ts_tablet_manager.cc:1403] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.538s	user 0.026s	sys 0.011s
I20260709 12:30:49.360008 18582 leader_election.cc:304] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:49.361348 19222 raft_consensus.cc:515] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:49.361812 19229 raft_consensus.cc:2749] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:49.355948 18892 tablet_bootstrap.cc:492] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:49.362615 18892 ts_tablet_manager.cc:1403] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.425s	user 0.028s	sys 0.010s
I20260709 12:30:49.362666 18767 raft_consensus.cc:2393] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84812365c65f43f388836c087640adbb in current term 1: Already voted for candidate 29f82a8594d741bc92dd24ee4a368e01 in this term.
I20260709 12:30:49.360328 19234 leader_election.cc:290] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:49.364481 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11b1d6d5a5c4991bd34743fff5ca507" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:49.364725 19222 leader_election.cc:290] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:49.365211 18617 raft_consensus.cc:3060] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.366291 18892 raft_consensus.cc:359] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:49.367286 18892 raft_consensus.cc:385] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:49.368515 18582 leader_election.cc:304] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:49.370484 19021 raft_consensus.cc:2749] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:49.371299 18582 leader_election.cc:304] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:30:49.372040 18888 tablet_bootstrap.cc:654] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:49.372210 19021 raft_consensus.cc:2804] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:49.368516 19223 raft_consensus.cc:3060] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.372696 19021 raft_consensus.cc:493] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:30:49.372869 18656 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 51770.5us
I20260709 12:30:49.373152 19021 raft_consensus.cc:3060] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.373872 18731 leader_election.cc:304] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:49.377696 19178 raft_consensus.cc:2804] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:49.378161 19178 raft_consensus.cc:493] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:49.378085 18835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16afc1d74df849c48d10f5da0a8f80c4" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:49.378700 19178 raft_consensus.cc:3060] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.378933 18835 raft_consensus.cc:3060] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.380398 18762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2acf997d23744fe697db1fe947dca883" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.381841 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fb81949f42946cf893026acc49357bc" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.383632 18731 leader_election.cc:304] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:49.385066 19215 raft_consensus.cc:2749] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:49.377959 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fb81949f42946cf893026acc49357bc" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:49.388427 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11b1d6d5a5c4991bd34743fff5ca507" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:49.390749 18690 raft_consensus.cc:3060] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.395252 18687 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e16042d49945ea80b31f9eee56f2cf" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:49.367640 18892 raft_consensus.cc:740] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:49.401225 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31e16042d49945ea80b31f9eee56f2cf" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.398644 18657 leader_election.cc:304] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:49.404661 19178 raft_consensus.cc:515] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:49.406913 19178 leader_election.cc:290] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:49.400738 18892 consensus_queue.cc:260] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:49.417718 18805 leader_election.cc:304] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb
I20260709 12:30:49.424328 18886 tablet_bootstrap.cc:492] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:49.425148 18886 ts_tablet_manager.cc:1403] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.167s	user 0.017s	sys 0.000s
I20260709 12:30:49.408607 18582 leader_election.cc:304] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:30:49.433420 18762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2abc15bf7b445a4b4269e1d93cce949" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.454962 18617 raft_consensus.cc:2468] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:49.449141 18686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a147020c5d724ef2a43e17ee907fe200" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
W20260709 12:30:49.457255 18656 leader_election.cc:343] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:49.412025 19226 raft_consensus.cc:2749] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:49.457744 18656 leader_election.cc:304] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:49.459002 19226 raft_consensus.cc:2749] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:49.459196 19227 consensus_queue.cc:1048] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:49.459800 18887 raft_consensus.cc:359] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:49.460532 18887 raft_consensus.cc:385] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:49.460831 18887 raft_consensus.cc:740] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:49.420007 19210 raft_consensus.cc:493] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:49.452014 19223 raft_consensus.cc:515] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:49.461457 19210 raft_consensus.cc:515] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:49.461601 18887 consensus_queue.cc:260] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:49.462632 19206 raft_consensus.cc:2764] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:30:49.463600 19210 leader_election.cc:290] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:49.464653 18730 leader_election.cc:304] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:30:49.465431 19222 raft_consensus.cc:2804] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:49.465931 19222 raft_consensus.cc:697] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:49.467175 19222 consensus_queue.cc:237] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:49.444806 18685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2abc15bf7b445a4b4269e1d93cce949" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:49.468453 18892 ts_tablet_manager.cc:1434] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.105s	user 0.009s	sys 0.000s
I20260709 12:30:49.469599 18892 tablet_bootstrap.cc:492] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:49.453954 18690 raft_consensus.cc:2468] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:49.453693 18657 leader_election.cc:304] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:49.454463 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a147020c5d724ef2a43e17ee907fe200" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.444864 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ca4b75eb0e49dca52e1b9fb4f8b356" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:49.478581 18886 raft_consensus.cc:359] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
W20260709 12:30:49.479252 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 61698.7us
I20260709 12:30:49.479563 18886 raft_consensus.cc:385] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:49.479923 18886 raft_consensus.cc:740] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:49.482894 18514 catalog_manager.cc:5697] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:49.484797 18514 catalog_manager.cc:5697] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:49.489867 18657 leader_election.cc:304] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
W20260709 12:30:49.480718 18804 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 49234.4us
I20260709 12:30:49.491272 18887 ts_tablet_manager.cc:1434] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.131s	user 0.006s	sys 0.000s
I20260709 12:30:49.494808 18887 tablet_bootstrap.cc:492] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:49.491659 19226 raft_consensus.cc:2749] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:49.491979 19224 raft_consensus.cc:2749] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:49.481498 18886 consensus_queue.cc:260] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:49.471839 19223 leader_election.cc:290] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
W20260709 12:30:49.504627 18730 leader_election.cc:343] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:49.505234 18886 ts_tablet_manager.cc:1434] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.080s	user 0.008s	sys 0.001s
I20260709 12:30:49.506021 18886 tablet_bootstrap.cc:492] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:49.499541 18656 leader_election.cc:304] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:49.510666 19205 raft_consensus.cc:2749] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:49.506779 19021 raft_consensus.cc:515] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:49.503975 19228 raft_consensus.cc:2749] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:49.515447 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb0c96994864b03a8ec7cda3cc8d85b" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:49.537837 18656 leader_election.cc:304] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:49.516036 18835 raft_consensus.cc:2468] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:49.541410 18656 leader_election.cc:304] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
W20260709 12:30:49.538899 18730 leader_election.cc:343] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:49.544173 18805 leader_election.cc:304] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:49.543417 19205 raft_consensus.cc:2749] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:49.545055 19228 raft_consensus.cc:2804] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:49.555154 18887 tablet_bootstrap.cc:654] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:49.557531 18805 leader_election.cc:304] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:49.552433 19210 raft_consensus.cc:2749] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:30:49.561198 18804 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 50808.1us
I20260709 12:30:49.563140 18685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9e24af510de430aabe2a8c0c1010629" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:49.565455 18892 tablet_bootstrap.cc:654] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:49.568261 18686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ca4b75eb0e49dca52e1b9fb4f8b356" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:49.583832 18762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c6af67983744957af1aee8d948169c2" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:49.572892 19021 leader_election.cc:290] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:49.578871 19228 raft_consensus.cc:493] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:49.593703 18886 tablet_bootstrap.cc:654] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:49.596850 18888 tablet_bootstrap.cc:492] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:49.606932 18888 ts_tablet_manager.cc:1403] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.311s	user 0.023s	sys 0.015s
W20260709 12:30:49.600544 18582 leader_election.cc:343] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:49.604859 18730 leader_election.cc:304] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
W20260709 12:30:49.607352 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 49889.6us
I20260709 12:30:49.609465 18805 leader_election.cc:304] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:49.604468 18840 raft_consensus.cc:1275] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Refusing update from remote peer 29f82a8594d741bc92dd24ee4a368e01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:30:49.611168 18611 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9e24af510de430aabe2a8c0c1010629" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:49.612394 19211 raft_consensus.cc:2749] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:49.614197 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ee74ad2fe774e0aa780b1fdf18b3003" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:49.615834 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb0c96994864b03a8ec7cda3cc8d85b" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:49.616494 18840 raft_consensus.cc:3060] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.583047 18686 raft_consensus.cc:3060] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.620491 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c6af67983744957af1aee8d948169c2" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:49.621219 18692 raft_consensus.cc:3060] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.621940 18804 leader_election.cc:304] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb
I20260709 12:30:49.627019 18840 raft_consensus.cc:2468] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:49.609731 19228 raft_consensus.cc:3060] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.648051 19210 raft_consensus.cc:2749] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:49.676925 18888 raft_consensus.cc:359] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:49.678121 18888 raft_consensus.cc:385] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:49.678501 18888 raft_consensus.cc:740] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:49.666702 19222 consensus_queue.cc:1048] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260709 12:30:49.664160 19200 raft_consensus.cc:2804] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:49.679502 18888 consensus_queue.cc:260] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:49.684188 19243 raft_consensus.cc:2749] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:49.702521 18686 raft_consensus.cc:2468] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
W20260709 12:30:49.710678 18730 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:44517, user_credentials={real_user=slave}} blocked reactor thread for 49007.6us
I20260709 12:30:49.714332 18730 leader_election.cc:304] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:49.727000 18892 tablet_bootstrap.cc:492] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:49.727596 18892 ts_tablet_manager.cc:1403] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.258s	user 0.011s	sys 0.026s
I20260709 12:30:49.733481 19213 raft_consensus.cc:493] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:49.737236 18888 ts_tablet_manager.cc:1434] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.130s	user 0.008s	sys 0.001s
I20260709 12:30:49.738574 18888 tablet_bootstrap.cc:492] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:49.749156 18692 raft_consensus.cc:2468] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 1.
I20260709 12:30:49.750188 18887 tablet_bootstrap.cc:492] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:49.739566 19200 raft_consensus.cc:697] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:49.750752 18887 ts_tablet_manager.cc:1403] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.256s	user 0.019s	sys 0.028s
I20260709 12:30:49.751327 19200 consensus_queue.cc:237] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:49.755424 18582 leader_election.cc:304] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:49.745600 18892 raft_consensus.cc:359] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:49.757351 18892 raft_consensus.cc:385] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:49.757597 18892 raft_consensus.cc:740] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:49.760772 18957 raft_consensus.cc:2749] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:49.781394 18886 tablet_bootstrap.cc:492] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:49.782192 18886 ts_tablet_manager.cc:1403] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.276s	user 0.014s	sys 0.024s
I20260709 12:30:49.738770 19213 raft_consensus.cc:515] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:49.786562 19213 leader_election.cc:290] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:49.791363 19228 raft_consensus.cc:515] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:49.795771 18892 consensus_queue.cc:260] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:49.806329 18887 raft_consensus.cc:359] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:49.807837 18887 raft_consensus.cc:385] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:49.808245 18887 raft_consensus.cc:740] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
W20260709 12:30:49.809401 18730 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 47039.1us
I20260709 12:30:49.813388 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba28ff472b0f46e8884f70afb21857ae" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.818192 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba28ff472b0f46e8884f70afb21857ae" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:49.818918 18690 raft_consensus.cc:2468] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:49.823483 19266 raft_consensus.cc:2804] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:49.823592 19226 raft_consensus.cc:493] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:49.824723 19226 raft_consensus.cc:515] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:49.827756 19226 leader_election.cc:290] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:49.832530 19206 raft_consensus.cc:493] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:49.833261 18892 ts_tablet_manager.cc:1434] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.105s	user 0.010s	sys 0.000s
I20260709 12:30:49.833606 19206 raft_consensus.cc:515] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:49.834769 18892 tablet_bootstrap.cc:492] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:49.836014 19206 leader_election.cc:290] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:49.842840 18888 tablet_bootstrap.cc:654] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:49.838310 18582 leader_election.cc:304] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:30:49.839238 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef4e065748614b3c87fd0a834017c455" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.831759 18886 raft_consensus.cc:359] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:49.849613 18886 raft_consensus.cc:385] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:49.849910 18886 raft_consensus.cc:740] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:49.846442 19206 raft_consensus.cc:493] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:49.850999 18886 consensus_queue.cc:260] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:49.852912 18887 consensus_queue.cc:260] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:49.856326 18886 ts_tablet_manager.cc:1434] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.074s	user 0.010s	sys 0.000s
I20260709 12:30:49.857328 18886 tablet_bootstrap.cc:492] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:49.849177 19266 raft_consensus.cc:697] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:49.851876 19206 raft_consensus.cc:515] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:49.860890 18957 raft_consensus.cc:2804] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:49.878885 18887 ts_tablet_manager.cc:1434] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.128s	user 0.011s	sys 0.000s
I20260709 12:30:49.881814 18887 tablet_bootstrap.cc:492] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:49.883397 18805 leader_election.cc:304] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:49.888406 19227 raft_consensus.cc:493] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:49.888625 18731 leader_election.cc:304] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:49.888985 19227 raft_consensus.cc:515] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
W20260709 12:30:49.875630 18657 leader_election.cc:343] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:49.891126 19227 leader_election.cc:290] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:49.891885 19182 raft_consensus.cc:2804] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:49.892442 19182 raft_consensus.cc:697] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:49.892326 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98dfe85df2e240449ec5f18cb3280c9b" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.893385 19182 consensus_queue.cc:237] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:49.893769 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef4e065748614b3c87fd0a834017c455" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:49.896579 19228 leader_election.cc:290] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:49.884218 18957 raft_consensus.cc:697] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:30:49.896942 19214 raft_consensus.cc:2804] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:49.897851 18957 consensus_queue.cc:237] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:49.897845 19214 raft_consensus.cc:493] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:49.899184 19214 raft_consensus.cc:3060] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.898345 19266 consensus_queue.cc:237] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
W20260709 12:30:49.894459 18657 leader_election.cc:343] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:49.915771 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16afc1d74df849c48d10f5da0a8f80c4" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:49.916072 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16afc1d74df849c48d10f5da0a8f80c4" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:49.919358 18892 tablet_bootstrap.cc:654] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:49.919883 18611 raft_consensus.cc:3060] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:49.927775 18841 raft_consensus.cc:2393] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84812365c65f43f388836c087640adbb in current term 1: Already voted for candidate 29f82a8594d741bc92dd24ee4a368e01 in this term.
I20260709 12:30:49.932881 19206 leader_election.cc:290] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:49.936375 18692 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4794853627e84898986305827ec4ea46" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:49.937233 18692 raft_consensus.cc:2393] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84812365c65f43f388836c087640adbb in current term 1: Already voted for candidate df352ed4600b4ddb83ef57ecd2944fce in this term.
I20260709 12:30:49.939769 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6256d41aab434828a66720ab61fa9ba5" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:49.945618 18886 tablet_bootstrap.cc:654] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:49.946158 18656 leader_election.cc:343] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:49.942070 18887 tablet_bootstrap.cc:654] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:49.939713 19214 raft_consensus.cc:515] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:49.962441 18892 tablet_bootstrap.cc:492] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:49.963165 18892 ts_tablet_manager.cc:1403] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.129s	user 0.020s	sys 0.002s
I20260709 12:30:49.967519 18887 tablet_bootstrap.cc:492] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:49.973284 18887 ts_tablet_manager.cc:1403] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.093s	user 0.017s	sys 0.000s
I20260709 12:30:49.975641 18581 leader_election.cc:304] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
W20260709 12:30:49.968258 18862 debug-util.cc:398] Leaking SignalData structure 0x7b08002c6480 after lost signal to thread 4359
I20260709 12:30:49.977468 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4794853627e84898986305827ec4ea46" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
W20260709 12:30:49.977901 18862 debug-util.cc:398] Leaking SignalData structure 0x7b0800217260 after lost signal to thread 18550
I20260709 12:30:49.978794 18892 raft_consensus.cc:359] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
W20260709 12:30:49.969605 18804 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 81462.4us
W20260709 12:30:49.971565 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 71381.9us
I20260709 12:30:49.979537 18892 raft_consensus.cc:385] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:30:49.990834 18805 leader_election.cc:343] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:30:49.979498 18862 debug-util.cc:398] Leaking SignalData structure 0x7b08001bb7c0 after lost signal to thread 18642
I20260709 12:30:49.979007 18887 raft_consensus.cc:359] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:50.007682 18887 raft_consensus.cc:385] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:50.008124 18887 raft_consensus.cc:740] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:50.009645 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba28ff472b0f46e8884f70afb21857ae" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:50.013296 18512 catalog_manager.cc:5697] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 2, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 2 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } }
W20260709 12:30:50.014816 18656 leader_election.cc:343] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:50.009111 18887 consensus_queue.cc:260] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:50.015317 18656 leader_election.cc:304] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:50.017362 18887 ts_tablet_manager.cc:1434] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.043s	user 0.007s	sys 0.000s
I20260709 12:30:49.970966 19214 leader_election.cc:290] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:50.004626 18892 raft_consensus.cc:740] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:50.002463 19206 raft_consensus.cc:3060] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:50.021147 18685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba28ff472b0f46e8884f70afb21857ae" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:50.042109 18685 raft_consensus.cc:3060] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:50.044987 18837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98dfe85df2e240449ec5f18cb3280c9b" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:50.046310 18892 consensus_queue.cc:260] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
W20260709 12:30:50.018735 18805 leader_election.cc:343] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:50.050630 18887 tablet_bootstrap.cc:492] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:50.027460 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6256d41aab434828a66720ab61fa9ba5" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:50.051585 18841 raft_consensus.cc:3060] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:50.051447 19241 consensus_queue.cc:1048] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:50.053102 19213 consensus_queue.cc:1048] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:30:50.054795 18656 leader_election.cc:343] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:50.055343 18656 leader_election.cc:304] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:50.029518 19235 consensus_queue.cc:1048] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:50.056180 19226 raft_consensus.cc:2749] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:50.054265 19206 raft_consensus.cc:2749] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:30:50.034144 18862 debug-util.cc:398] Leaking SignalData structure 0x7b08001f60e0 after lost signal to thread 18717
I20260709 12:30:50.020182 19232 raft_consensus.cc:2749] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:49.979765 18581 leader_election.cc:304] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
W20260709 12:30:50.058600 18862 debug-util.cc:398] Leaking SignalData structure 0x7b08003cc5e0 after lost signal to thread 18791
W20260709 12:30:50.058678 18581 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:44517, user_credentials={real_user=slave}} blocked reactor thread for 79018.9us
I20260709 12:30:50.060343 19204 raft_consensus.cc:2749] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:30:50.060261 18862 debug-util.cc:398] Leaking SignalData structure 0x7b08002582c0 after lost signal to thread 18865
I20260709 12:30:50.088207 18892 ts_tablet_manager.cc:1434] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.125s	user 0.010s	sys 0.001s
I20260709 12:30:50.089442 18892 tablet_bootstrap.cc:492] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:50.092070 18886 tablet_bootstrap.cc:492] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:50.092770 18886 ts_tablet_manager.cc:1403] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.236s	user 0.031s	sys 0.011s
I20260709 12:30:50.100672 18892 tablet_bootstrap.cc:654] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:50.099861 18513 catalog_manager.cc:5697] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:50.102056 18513 catalog_manager.cc:5697] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:50.107710 18513 catalog_manager.cc:5697] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:50.109747 18513 catalog_manager.cc:5697] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:50.081851 19214 consensus_queue.cc:1048] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260709 12:30:50.104833 19182 consensus_queue.cc:1048] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260709 12:30:50.122222 19198 consensus_queue.cc:1048] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:50.112197 18886 raft_consensus.cc:359] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:50.132061 18886 raft_consensus.cc:385] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:50.132754 18886 raft_consensus.cc:740] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:50.101909 18841 raft_consensus.cc:2468] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:50.135785 18886 consensus_queue.cc:260] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:50.162761 18886 ts_tablet_manager.cc:1434] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.070s	user 0.010s	sys 0.000s
I20260709 12:30:50.134506 19214 raft_consensus.cc:493] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:50.166100 18888 tablet_bootstrap.cc:492] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:50.166656 18888 ts_tablet_manager.cc:1403] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.429s	user 0.021s	sys 0.034s
I20260709 12:30:50.169893 18886 tablet_bootstrap.cc:492] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:50.150628 19266 raft_consensus.cc:493] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:50.153213 18685 raft_consensus.cc:2468] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:50.195210 18656 leader_election.cc:304] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:50.206422 18887 tablet_bootstrap.cc:654] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:50.170660 18582 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:40335, user_credentials={real_user=slave}} blocked reactor thread for 53685.1us
I20260709 12:30:50.167369 19214 raft_consensus.cc:515] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:50.201275 19226 raft_consensus.cc:2804] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:50.222692 18805 leader_election.cc:304] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:30:50.229027 19226 raft_consensus.cc:697] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:30:50.240430 19021 raft_consensus.cc:493] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:50.201538 19266 raft_consensus.cc:515] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:50.236899 19211 raft_consensus.cc:2804] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:50.295466 19211 raft_consensus.cc:697] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:50.296516 19211 consensus_queue.cc:237] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:50.303774 19214 leader_election.cc:290] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:50.305477 19226 consensus_queue.cc:237] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:50.305557 19204 consensus_queue.cc:1048] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:30:50.282948 18888 raft_consensus.cc:359] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:50.327940 18888 raft_consensus.cc:385] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:50.284880 19021 raft_consensus.cc:515] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:50.294152 19266 leader_election.cc:290] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:50.324725 19223 consensus_queue.cc:1048] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:50.325459 18685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c6af67983744957af1aee8d948169c2" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:50.328425 18888 raft_consensus.cc:740] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
W20260709 12:30:50.333561 18563 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:30:50.349285 18886 tablet_bootstrap.cc:654] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:50.350831 19214 raft_consensus.cc:493] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:50.351401 19214 raft_consensus.cc:515] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:50.353500 19214 leader_election.cc:290] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:50.339780 18888 consensus_queue.cc:260] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:50.353755 19021 leader_election.cc:290] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:50.371342 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ab29498797f49d198e49d29f9800da2" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:50.373966 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c6af67983744957af1aee8d948169c2" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:50.374568 18837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb0c96994864b03a8ec7cda3cc8d85b" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:50.375391 18837 raft_consensus.cc:2393] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84812365c65f43f388836c087640adbb in current term 1: Already voted for candidate 29f82a8594d741bc92dd24ee4a368e01 in this term.
I20260709 12:30:50.377018 18888 ts_tablet_manager.cc:1434] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.210s	user 0.011s	sys 0.000s
I20260709 12:30:50.393631 18516 catalog_manager.cc:5697] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 1 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:50.372081 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ab29498797f49d198e49d29f9800da2" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:50.404156 18760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc988d3d39094696b3b4516b33a9c37d" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:50.428642 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb0c96994864b03a8ec7cda3cc8d85b" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:50.440728 18582 leader_election.cc:304] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:50.445613 19286 raft_consensus.cc:3060] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:50.450979 18888 tablet_bootstrap.cc:492] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:50.492776 18892 tablet_bootstrap.cc:492] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:50.493577 18892 ts_tablet_manager.cc:1403] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.404s	user 0.026s	sys 0.022s
I20260709 12:30:50.498768 18892 raft_consensus.cc:359] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:50.499521 18892 raft_consensus.cc:385] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:50.499800 18892 raft_consensus.cc:740] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:50.500053 18761 raft_consensus.cc:3060] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:50.500605 18892 consensus_queue.cc:260] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:50.502964 18892 ts_tablet_manager.cc:1434] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:30:50.503949 18892 tablet_bootstrap.cc:492] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:30:50.506043 18730 consensus_peers.cc:597] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:50.535296 18804 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 177816us
I20260709 12:30:50.527799 19271 consensus_queue.cc:1048] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
W20260709 12:30:50.560845 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:40335, user_credentials={real_user=slave}} blocked reactor thread for 77844.6us
I20260709 12:30:50.562593 19205 raft_consensus.cc:493] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:50.577204 19205 raft_consensus.cc:515] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:50.581674 19205 leader_election.cc:290] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:50.585439 18731 leader_election.cc:304] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb
I20260709 12:30:50.586532 18762 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6fc5ff8d0014a9f9c1dadcc157e1d9a" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:50.583217 18887 tablet_bootstrap.cc:492] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:50.583591 19286 raft_consensus.cc:2749] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:50.592028 18887 ts_tablet_manager.cc:1403] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.542s	user 0.034s	sys 0.006s
W20260709 12:30:50.610479 18657 leader_election.cc:343] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:50.612973 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6fc5ff8d0014a9f9c1dadcc157e1d9a" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:50.620750 18888 tablet_bootstrap.cc:654] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:50.621299 18887 raft_consensus.cc:359] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:50.622040 18887 raft_consensus.cc:385] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:50.622294 18887 raft_consensus.cc:740] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:50.623040 18887 consensus_queue.cc:260] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
W20260709 12:30:50.626259 18805 leader_election.cc:343] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:50.626699 19222 consensus_queue.cc:1048] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:30:50.631392 18805 leader_election.cc:343] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:50.627839 18887 ts_tablet_manager.cc:1434] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.035s	user 0.006s	sys 0.000s
I20260709 12:30:50.636240 18513 catalog_manager.cc:5697] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:50.633407 18887 tablet_bootstrap.cc:492] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
W20260709 12:30:50.644095 18804 leader_election.cc:343] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:50.648772 18892 tablet_bootstrap.cc:654] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:50.648568 18804 leader_election.cc:304] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:50.652824 19298 raft_consensus.cc:2749] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:50.655678 18613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc988d3d39094696b3b4516b33a9c37d" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:50.659070 18613 raft_consensus.cc:3060] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:50.667438 19221 consensus_queue.cc:1048] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:50.670224 19270 raft_consensus.cc:2749] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:30:50.694517 18656 leader_election.cc:343] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:50.695813 18656 leader_election.cc:304] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:50.704268 19226 raft_consensus.cc:2749] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:30:50.724233 18804 leader_election.cc:343] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:50.725095 18804 leader_election.cc:304] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:50.766472 18886 tablet_bootstrap.cc:492] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:50.767408 18886 ts_tablet_manager.cc:1403] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.598s	user 0.021s	sys 0.023s
I20260709 12:30:50.768816 19299 raft_consensus.cc:2749] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:50.788902 18887 tablet_bootstrap.cc:654] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:50.799108 18886 raft_consensus.cc:359] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:50.799986 18886 raft_consensus.cc:385] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:50.809593 19286 consensus_queue.cc:1048] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:50.814864 18888 tablet_bootstrap.cc:492] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:50.815433 18888 ts_tablet_manager.cc:1403] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.365s	user 0.015s	sys 0.014s
I20260709 12:30:50.800278 18886 raft_consensus.cc:740] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:50.820120 18886 consensus_queue.cc:260] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
W20260709 12:30:50.833809 18656 consensus_peers.cc:597] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:50.835846 18886 ts_tablet_manager.cc:1434] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.068s	user 0.011s	sys 0.000s
I20260709 12:30:50.854538 18888 raft_consensus.cc:359] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:50.855432 18888 raft_consensus.cc:385] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:50.855731 18888 raft_consensus.cc:740] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:50.856483 18888 consensus_queue.cc:260] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:50.830011 19221 consensus_queue.cc:1048] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260709 12:30:50.905484 18892 tablet_bootstrap.cc:492] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:50.906256 18892 ts_tablet_manager.cc:1403] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.403s	user 0.016s	sys 0.020s
I20260709 12:30:50.916262 19198 raft_consensus.cc:493] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:50.917145 19198 raft_consensus.cc:515] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:50.923379 18886 tablet_bootstrap.cc:492] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:50.927807 19232 consensus_queue.cc:1048] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:50.937065 18888 ts_tablet_manager.cc:1434] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.121s	user 0.008s	sys 0.000s
I20260709 12:30:50.938431 18888 tablet_bootstrap.cc:492] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:50.901764 19226 raft_consensus.cc:493] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:50.962260 18888 tablet_bootstrap.cc:654] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:50.963521 18892 raft_consensus.cc:359] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:50.964422 18892 raft_consensus.cc:385] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:50.964725 18892 raft_consensus.cc:740] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:50.967751 18892 consensus_queue.cc:260] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:50.977130 18892 ts_tablet_manager.cc:1434] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.070s	user 0.009s	sys 0.001s
I20260709 12:30:50.978276 18892 tablet_bootstrap.cc:492] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:50.989559 19311 raft_consensus.cc:493] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:50.990217 19311 raft_consensus.cc:515] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:50.950623 19226 raft_consensus.cc:515] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:50.994638 18837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba28ff472b0f46e8884f70afb21857ae" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:50.996541 19226 leader_election.cc:290] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:50.996826 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba28ff472b0f46e8884f70afb21857ae" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:50.987186 19198 leader_election.cc:290] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:51.000432 18760 raft_consensus.cc:3060] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:51.006340 19311 leader_election.cc:290] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:51.006352 19243 consensus_queue.cc:1048] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:51.011445 18609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d216c51d889b417bb2e59f20b6fd6d96" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
W20260709 12:30:51.013202 18656 leader_election.cc:343] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:51.014197 18837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d216c51d889b417bb2e59f20b6fd6d96" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:51.018880 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98dfe85df2e240449ec5f18cb3280c9b" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:51.019784 18689 raft_consensus.cc:2468] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:51.021590 18887 tablet_bootstrap.cc:492] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:51.022509 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98dfe85df2e240449ec5f18cb3280c9b" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:51.023979 18886 tablet_bootstrap.cc:654] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:51.023900 18656 leader_election.cc:343] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:51.026705 18887 ts_tablet_manager.cc:1403] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.394s	user 0.016s	sys 0.013s
I20260709 12:30:51.026957 18656 leader_election.cc:304] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:51.025337 18730 leader_election.cc:304] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:51.028473 19200 raft_consensus.cc:2749] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:51.032896 19226 raft_consensus.cc:2749] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:51.040051 18892 tablet_bootstrap.cc:654] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:51.046236 18887 raft_consensus.cc:359] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:51.049443 18887 raft_consensus.cc:385] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:51.049825 18887 raft_consensus.cc:740] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:51.050630 18887 consensus_queue.cc:260] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
W20260709 12:30:51.053269 18805 leader_election.cc:343] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:51.055428 19243 consensus_queue.cc:1048] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:51.060470 18887 ts_tablet_manager.cc:1434] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.033s	user 0.008s	sys 0.001s
I20260709 12:30:51.105242 18805 leader_election.cc:304] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:30:51.107020 19243 raft_consensus.cc:2804] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:51.107437 19243 raft_consensus.cc:493] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:51.107788 19243 raft_consensus.cc:3060] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:51.111212 18886 tablet_bootstrap.cc:492] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:51.112464 18886 ts_tablet_manager.cc:1403] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.189s	user 0.036s	sys 0.012s
I20260709 12:30:51.119606 18888 tablet_bootstrap.cc:492] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:51.120441 18888 ts_tablet_manager.cc:1403] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.182s	user 0.021s	sys 0.001s
I20260709 12:30:51.123668 18888 raft_consensus.cc:359] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:51.126987 18886 raft_consensus.cc:359] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:51.128104 18892 tablet_bootstrap.cc:492] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:51.128688 18886 raft_consensus.cc:385] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:51.128888 18892 ts_tablet_manager.cc:1403] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.151s	user 0.015s	sys 0.006s
I20260709 12:30:51.129063 18886 raft_consensus.cc:740] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:51.129604 18887 tablet_bootstrap.cc:492] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:51.130102 18886 consensus_queue.cc:260] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:51.131835 18886 ts_tablet_manager.cc:1434] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.019s	user 0.004s	sys 0.002s
I20260709 12:30:51.132500 18892 raft_consensus.cc:359] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:51.128454 18888 raft_consensus.cc:385] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:51.133481 18886 tablet_bootstrap.cc:492] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:51.133515 18892 raft_consensus.cc:385] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:51.134634 18892 raft_consensus.cc:740] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:51.134964 18888 raft_consensus.cc:740] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:51.119788 18514 catalog_manager.cc:5697] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:51.146642 18888 consensus_queue.cc:260] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:51.161079 19226 raft_consensus.cc:493] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:51.161939 19226 raft_consensus.cc:515] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:51.168220 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a66b75f31e14b98902006a8f9d53a95" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:51.169250 18761 raft_consensus.cc:2468] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 0.
I20260709 12:30:51.171305 18657 leader_election.cc:304] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:51.172463 19228 raft_consensus.cc:2804] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:51.172981 19228 raft_consensus.cc:493] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:51.173406 19228 raft_consensus.cc:3060] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:51.178318 18886 tablet_bootstrap.cc:654] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:51.179020 19226 leader_election.cc:290] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:51.179360 18887 tablet_bootstrap.cc:654] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:51.192150 18609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a66b75f31e14b98902006a8f9d53a95" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:51.193622 19234 raft_consensus.cc:493] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:51.194262 19234 raft_consensus.cc:515] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:51.188366 18892 consensus_queue.cc:260] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:51.196134 18609 raft_consensus.cc:3060] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:51.196749 19234 leader_election.cc:290] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
W20260709 12:30:51.204963 18656 leader_election.cc:343] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:51.208662 18887 tablet_bootstrap.cc:492] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:51.209897 18887 ts_tablet_manager.cc:1403] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.081s	user 0.011s	sys 0.005s
I20260709 12:30:51.214512 18888 ts_tablet_manager.cc:1434] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.094s	user 0.011s	sys 0.000s
I20260709 12:30:51.214993 18613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:51.222231 18888 tablet_bootstrap.cc:492] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:51.215843 18892 ts_tablet_manager.cc:1434] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.087s	user 0.007s	sys 0.006s
I20260709 12:30:51.223580 18892 tablet_bootstrap.cc:492] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:51.186514 19228 raft_consensus.cc:515] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:51.228471 19228 leader_election.cc:290] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:51.228611 18613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a66b75f31e14b98902006a8f9d53a95" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:51.229611 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a66b75f31e14b98902006a8f9d53a95" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
W20260709 12:30:51.230196 18656 leader_election.cc:343] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:51.230221 18761 raft_consensus.cc:3060] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:30:51.232757 18804 leader_election.cc:343] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:51.233973 18886 tablet_bootstrap.cc:492] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:51.234515 18886 ts_tablet_manager.cc:1403] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.101s	user 0.022s	sys 0.024s
I20260709 12:30:51.237596 18886 raft_consensus.cc:359] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:51.238310 18886 raft_consensus.cc:385] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:51.238588 18886 raft_consensus.cc:740] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:51.184063 19243 raft_consensus.cc:515] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:51.239725 18761 raft_consensus.cc:2468] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:51.241966 18887 raft_consensus.cc:359] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:51.242832 18887 raft_consensus.cc:385] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:51.243132 18887 raft_consensus.cc:740] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:51.243456 18657 leader_election.cc:304] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:51.245070 19227 raft_consensus.cc:2804] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:51.245532 19227 raft_consensus.cc:697] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:30:51.246117 18888 tablet_bootstrap.cc:654] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:51.246404 19227 consensus_queue.cc:237] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:51.254498 18886 consensus_queue.cc:260] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:51.259315 18887 consensus_queue.cc:260] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:51.263778 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:51.267289 18886 ts_tablet_manager.cc:1434] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.002s
I20260709 12:30:51.269137 18886 tablet_bootstrap.cc:492] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:51.272300 18892 tablet_bootstrap.cc:654] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:51.269819 18887 ts_tablet_manager.cc:1434] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.038s	user 0.004s	sys 0.003s
I20260709 12:30:51.273729 18887 tablet_bootstrap.cc:492] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:51.278992 18888 tablet_bootstrap.cc:492] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:51.279598 18888 ts_tablet_manager.cc:1403] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.058s	user 0.012s	sys 0.006s
I20260709 12:30:51.280954 19243 leader_election.cc:290] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:51.282845 18888 raft_consensus.cc:359] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:51.283789 18888 raft_consensus.cc:385] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:51.284070 18888 raft_consensus.cc:740] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:51.285005 18888 consensus_queue.cc:260] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:51.290958 18888 ts_tablet_manager.cc:1434] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:30:51.291818 18887 tablet_bootstrap.cc:654] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:51.292778 18888 tablet_bootstrap.cc:492] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:51.292392 18886 tablet_bootstrap.cc:654] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:51.304605 18805 leader_election.cc:343] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:51.305204 18805 leader_election.cc:304] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb
I20260709 12:30:51.306502 19303 raft_consensus.cc:2749] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:51.307693 18888 tablet_bootstrap.cc:654] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:51.311913 18887 tablet_bootstrap.cc:492] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:51.312484 18887 ts_tablet_manager.cc:1403] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.039s	user 0.008s	sys 0.010s
I20260709 12:30:51.316807 18887 raft_consensus.cc:359] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:51.317782 18887 raft_consensus.cc:385] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:51.318127 18887 raft_consensus.cc:740] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:51.319154 18887 consensus_queue.cc:260] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:51.321619 18887 ts_tablet_manager.cc:1434] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260709 12:30:51.322582 18887 tablet_bootstrap.cc:492] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:51.324469 18888 tablet_bootstrap.cc:492] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:51.325191 18888 ts_tablet_manager.cc:1403] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.033s	user 0.016s	sys 0.004s
I20260709 12:30:51.329416 18887 tablet_bootstrap.cc:654] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:51.330192 18888 raft_consensus.cc:359] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:51.331488 18888 raft_consensus.cc:385] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:51.331866 18888 raft_consensus.cc:740] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:51.333019 18888 consensus_queue.cc:260] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:51.342886 18892 tablet_bootstrap.cc:492] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:51.344321 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98dfe85df2e240449ec5f18cb3280c9b" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:51.344791 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98dfe85df2e240449ec5f18cb3280c9b" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:51.345162 18761 raft_consensus.cc:3060] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:51.345427 18689 raft_consensus.cc:3060] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:51.349359 19270 raft_consensus.cc:493] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:51.349530 18892 ts_tablet_manager.cc:1403] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.126s	user 0.009s	sys 0.005s
I20260709 12:30:51.350008 19270 raft_consensus.cc:515] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:51.353735 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc988d3d39094696b3b4516b33a9c37d" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:51.354521 18841 raft_consensus.cc:2468] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:51.360882 19270 leader_election.cc:290] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:51.363202 18892 raft_consensus.cc:359] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:51.364009 18892 raft_consensus.cc:385] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:51.364290 18892 raft_consensus.cc:740] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:51.365015 18892 consensus_queue.cc:260] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:51.366745 18730 leader_election.cc:304] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:51.367632 18892 ts_tablet_manager.cc:1434] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.000s
I20260709 12:30:51.368578 18613 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc988d3d39094696b3b4516b33a9c37d" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:51.369077 18892 tablet_bootstrap.cc:492] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:51.369253 18613 raft_consensus.cc:2468] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:51.371795 19200 raft_consensus.cc:2804] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:51.372643 19200 raft_consensus.cc:493] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:51.373040 19200 raft_consensus.cc:3060] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:51.381098 18689 raft_consensus.cc:2468] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:51.386157 18761 raft_consensus.cc:2468] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:51.388594 18805 leader_election.cc:304] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:51.389453 19234 raft_consensus.cc:2804] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:51.390015 19234 raft_consensus.cc:697] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:51.390978 19234 consensus_queue.cc:237] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:51.338655 18888 ts_tablet_manager.cc:1434] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.001s
I20260709 12:30:51.407018 18888 tablet_bootstrap.cc:492] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:51.381125 19200 raft_consensus.cc:515] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:51.428494 19200 leader_election.cc:290] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:51.438541 18888 tablet_bootstrap.cc:654] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:51.449942 19270 raft_consensus.cc:493] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:51.450996 19270 raft_consensus.cc:515] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:51.611274 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc988d3d39094696b3b4516b33a9c37d" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:51.618501 18887 tablet_bootstrap.cc:492] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:51.619225 18887 ts_tablet_manager.cc:1403] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.297s	user 0.030s	sys 0.035s
I20260709 12:30:51.622236 18886 tablet_bootstrap.cc:492] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:51.622911 18886 ts_tablet_manager.cc:1403] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.354s	user 0.027s	sys 0.027s
I20260709 12:30:51.625145 19270 leader_election.cc:290] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:51.628082 18609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc988d3d39094696b3b4516b33a9c37d" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:51.628994 18609 raft_consensus.cc:3060] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:51.635224 18888 tablet_bootstrap.cc:492] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:51.641587 18888 ts_tablet_manager.cc:1403] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.235s	user 0.018s	sys 0.007s
I20260709 12:30:51.639823 18609 raft_consensus.cc:2468] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:51.612198 18841 raft_consensus.cc:3060] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:51.663089 18892 tablet_bootstrap.cc:654] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:51.677757 18835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef4e065748614b3c87fd0a834017c455" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:51.678174 18613 raft_consensus.cc:3060] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:51.678956 18887 raft_consensus.cc:359] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:51.679668 18887 raft_consensus.cc:385] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:51.679936 18887 raft_consensus.cc:740] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:51.680627 18887 consensus_queue.cc:260] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:51.682204 18887 ts_tablet_manager.cc:1434] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.063s	user 0.004s	sys 0.001s
I20260709 12:30:51.683363 18887 tablet_bootstrap.cc:492] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:51.708321 18841 raft_consensus.cc:2468] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
W20260709 12:30:51.718526 18730 leader_election.cc:343] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:51.719269 18730 leader_election.cc:304] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:51.722852 18886 raft_consensus.cc:359] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:51.723948 18886 raft_consensus.cc:385] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:51.724280 18886 raft_consensus.cc:740] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:51.729590 18886 consensus_queue.cc:260] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:51.736477 19270 raft_consensus.cc:2804] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:51.721259 19344 raft_consensus.cc:493] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:51.762519 18886 ts_tablet_manager.cc:1434] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.139s	user 0.010s	sys 0.000s
I20260709 12:30:51.769629 18886 tablet_bootstrap.cc:492] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:51.740752 19270 raft_consensus.cc:697] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:51.777300 19270 consensus_queue.cc:237] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:51.758550 19228 raft_consensus.cc:493] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:51.780793 19228 raft_consensus.cc:515] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
W20260709 12:30:51.781580 18563 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:30:51.783056 19228 leader_election.cc:290] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:51.787240 19214 raft_consensus.cc:493] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:30:51.794227 18657 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:40335, user_credentials={real_user=slave}} blocked reactor thread for 117428us
I20260709 12:30:51.787976 19214 raft_consensus.cc:515] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:51.803241 18888 raft_consensus.cc:359] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:51.809334 18888 raft_consensus.cc:385] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:51.809736 18888 raft_consensus.cc:740] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
W20260709 12:30:51.811790 18804 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 46936.3us
I20260709 12:30:51.813055 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef4e065748614b3c87fd0a834017c455" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:51.813745 18690 raft_consensus.cc:2468] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:51.795533 19344 raft_consensus.cc:515] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:51.824714 18888 consensus_queue.cc:260] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:51.832700 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4a560e00884f1cb6f8720dfee7d27c" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:51.833583 18841 raft_consensus.cc:2468] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 0.
I20260709 12:30:51.854969 18887 tablet_bootstrap.cc:654] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:51.858455 18892 tablet_bootstrap.cc:492] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:51.859135 18892 ts_tablet_manager.cc:1403] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.490s	user 0.014s	sys 0.019s
I20260709 12:30:51.863940 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d216c51d889b417bb2e59f20b6fd6d96" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
W20260709 12:30:51.867925 18804 leader_election.cc:343] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:51.873194 18892 raft_consensus.cc:359] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:51.874150 18892 raft_consensus.cc:385] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:51.874449 18892 raft_consensus.cc:740] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:51.875505 18656 leader_election.cc:304] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:51.876868 19228 raft_consensus.cc:2804] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:51.877245 19228 raft_consensus.cc:493] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:51.877580 19228 raft_consensus.cc:3060] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:51.881064 18892 consensus_queue.cc:260] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:51.891871 18892 ts_tablet_manager.cc:1434] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.032s	user 0.010s	sys 0.000s
I20260709 12:30:51.882340 19214 leader_election.cc:290] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:51.918696 19344 leader_election.cc:290] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:51.920768 19221 raft_consensus.cc:493] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:51.921825 19210 raft_consensus.cc:493] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:51.927608 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a46101454ef46beb00f1652fabfa153" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
W20260709 12:30:51.972623 18656 leader_election.cc:343] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:51.987401 18886 tablet_bootstrap.cc:654] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:51.989038 18582 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 173909us
I20260709 12:30:51.924654 19232 consensus_queue.cc:1048] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.060s
W20260709 12:30:51.999646 18730 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 178886us
I20260709 12:30:52.005244 18892 tablet_bootstrap.cc:492] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:51.976658 19221 raft_consensus.cc:515] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:51.978109 19210 raft_consensus.cc:515] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:52.005005 18763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a46101454ef46beb00f1652fabfa153" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:52.011281 18731 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 154488us
I20260709 12:30:52.055773 18887 tablet_bootstrap.cc:492] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:52.060822 19210 leader_election.cc:290] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:52.062247 18887 ts_tablet_manager.cc:1403] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.379s	user 0.019s	sys 0.002s
I20260709 12:30:52.063254 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4c43366d9fb416e8649aba9676aed59" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
W20260709 12:30:52.069314 18804 leader_election.cc:343] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:52.071456 19221 leader_election.cc:290] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:52.070631 18887 raft_consensus.cc:359] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:52.072142 18887 raft_consensus.cc:385] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:52.072649 18887 raft_consensus.cc:740] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:52.073621 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d216c51d889b417bb2e59f20b6fd6d96" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:52.074607 18690 raft_consensus.cc:2468] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:52.073585 18887 consensus_queue.cc:260] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:52.103493 18887 ts_tablet_manager.cc:1434] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.041s	user 0.009s	sys 0.000s
I20260709 12:30:52.022922 19228 raft_consensus.cc:515] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:52.104959 18887 tablet_bootstrap.cc:492] T ef4e065748614b3c87fd0a834017c455 P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:52.123003 19228 leader_election.cc:290] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:52.105595 18731 leader_election.cc:304] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:52.103492 18888 ts_tablet_manager.cc:1434] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.462s	user 0.015s	sys 0.009s
I20260709 12:30:52.135830 18888 tablet_bootstrap.cc:492] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
W20260709 12:30:52.150080 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:40335, user_credentials={real_user=slave}} blocked reactor thread for 141944us
I20260709 12:30:52.158886 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4a560e00884f1cb6f8720dfee7d27c" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:52.159569 18841 raft_consensus.cc:3060] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:52.162099 18805 leader_election.cc:304] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:52.131916 19360 raft_consensus.cc:2804] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:52.172710 18841 raft_consensus.cc:2468] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:52.157485 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ab29498797f49d198e49d29f9800da2" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:52.166500 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4c43366d9fb416e8649aba9676aed59" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:52.166775 19360 raft_consensus.cc:493] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:52.193399 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4a560e00884f1cb6f8720dfee7d27c" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:52.200021 19234 raft_consensus.cc:2804] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:52.200539 19234 raft_consensus.cc:493] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:52.200892 19234 raft_consensus.cc:3060] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:52.201572 18888 tablet_bootstrap.cc:654] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:52.203691 18656 leader_election.cc:343] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:52.212852 18615 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c4a560e00884f1cb6f8720dfee7d27c" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:52.216475 18886 tablet_bootstrap.cc:492] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:52.217085 18886 ts_tablet_manager.cc:1403] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.448s	user 0.009s	sys 0.026s
I20260709 12:30:52.187399 18838 raft_consensus.cc:2468] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:52.234012 18611 raft_consensus.cc:3060] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:52.235083 18886 raft_consensus.cc:359] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:52.235828 18886 raft_consensus.cc:385] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:52.236093 18886 raft_consensus.cc:740] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:52.236873 18886 consensus_queue.cc:260] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:52.239346 18886 ts_tablet_manager.cc:1434] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.000s
I20260709 12:30:52.240373 18886 tablet_bootstrap.cc:492] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d: Bootstrap starting.
I20260709 12:30:52.247040 18730 leader_election.cc:304] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:52.243610 18516 catalog_manager.cc:5697] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } }
W20260709 12:30:52.248548 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 86573.3us
I20260709 12:30:52.196025 19360 raft_consensus.cc:3060] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:52.255852 19357 raft_consensus.cc:2804] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:30:52.269799 18657 leader_election.cc:343] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:52.271371 18657 leader_election.cc:304] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:52.263947 19357 raft_consensus.cc:493] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:52.283082 18513 catalog_manager.cc:5697] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:52.273684 19232 raft_consensus.cc:2749] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:30:52.286821 18805 leader_election.cc:343] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:52.288951 18805 leader_election.cc:304] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:52.297004 18887 tablet_bootstrap.cc:654] T ef4e065748614b3c87fd0a834017c455 P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:52.299832 18892 tablet_bootstrap.cc:654] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:52.280840 19357 raft_consensus.cc:3060] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:52.278472 19298 consensus_queue.cc:1048] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20260709 12:30:52.336606 18957 raft_consensus.cc:493] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 29f82a8594d741bc92dd24ee4a368e01)
I20260709 12:30:52.337195 18957 raft_consensus.cc:515] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
W20260709 12:30:52.338011 18581 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:44517, user_credentials={real_user=slave}} blocked reactor thread for 143895us
I20260709 12:30:52.339540 19363 consensus_queue.cc:1048] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:30:52.340493 18582 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 135909us
I20260709 12:30:52.341445 18957 leader_election.cc:290] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
W20260709 12:30:52.346676 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:40335, user_credentials={real_user=slave}} blocked reactor thread for 49363.7us
I20260709 12:30:52.349056 18656 leader_election.cc:304] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:52.305236 19368 raft_consensus.cc:2749] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:52.366993 19344 raft_consensus.cc:2804] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:52.370591 19226 raft_consensus.cc:493] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:52.395718 18886 tablet_bootstrap.cc:654] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:52.409859 18582 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 68730.6us
I20260709 12:30:52.410548 18888 tablet_bootstrap.cc:492] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:52.411051 18888 ts_tablet_manager.cc:1403] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.276s	user 0.024s	sys 0.019s
I20260709 12:30:52.419178 19360 raft_consensus.cc:515] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:52.422044 19345 raft_consensus.cc:493] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:52.429324 19345 raft_consensus.cc:515] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:52.431571 19345 leader_election.cc:290] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:52.392092 19226 raft_consensus.cc:515] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:52.433845 19226 leader_election.cc:290] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:52.396982 19344 raft_consensus.cc:697] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:30:52.435109 19344 consensus_queue.cc:237] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:52.444773 19234 raft_consensus.cc:515] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:52.449810 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0840843cb482476a8e18344ff8f1e315" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:52.451356 18763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6fc5ff8d0014a9f9c1dadcc157e1d9a" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:52.459975 18657 leader_election.cc:343] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:30:52.460933 18657 leader_election.cc:343] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:52.448190 19234 leader_election.cc:290] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:52.465160 18892 tablet_bootstrap.cc:492] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:52.464742 18888 raft_consensus.cc:359] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:52.465660 18888 raft_consensus.cc:385] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:52.465821 18892 ts_tablet_manager.cc:1403] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.468s	user 0.028s	sys 0.019s
I20260709 12:30:52.465972 18888 raft_consensus.cc:740] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:52.467026 18888 consensus_queue.cc:260] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:52.476864 18616 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d216c51d889b417bb2e59f20b6fd6d96" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:52.478996 18888 ts_tablet_manager.cc:1434] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.068s	user 0.008s	sys 0.000s
I20260709 12:30:52.481462 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ca4b75eb0e49dca52e1b9fb4f8b356" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
W20260709 12:30:52.483825 18804 leader_election.cc:343] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:52.487761 18887 tablet_bootstrap.cc:492] T ef4e065748614b3c87fd0a834017c455 P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:52.489413 18887 ts_tablet_manager.cc:1403] T ef4e065748614b3c87fd0a834017c455 P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.385s	user 0.020s	sys 0.002s
I20260709 12:30:52.505065 19298 raft_consensus.cc:493] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:52.540915 18888 tablet_bootstrap.cc:492] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:52.457158 19357 raft_consensus.cc:515] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:52.561533 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ca4b75eb0e49dca52e1b9fb4f8b356" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:52.579948 18656 consensus_peers.cc:597] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:52.580691 18656 leader_election.cc:343] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:52.581866 19353 raft_consensus.cc:493] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60d61e0800194dd89be34d3aff3f374d)
I20260709 12:30:52.587451 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ab29498797f49d198e49d29f9800da2" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:52.588142 18841 raft_consensus.cc:3060] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:52.590870 18887 raft_consensus.cc:359] T ef4e065748614b3c87fd0a834017c455 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:52.591624 18887 raft_consensus.cc:385] T ef4e065748614b3c87fd0a834017c455 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:52.591878 18887 raft_consensus.cc:740] T ef4e065748614b3c87fd0a834017c455 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:52.592521 18835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef4e065748614b3c87fd0a834017c455" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:52.592440 19350 consensus_queue.cc:1048] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:52.561797 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ab29498797f49d198e49d29f9800da2" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
W20260709 12:30:52.577709 19381 log.cc:927] Time spent T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:30:52.586261 19353 raft_consensus.cc:515] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:52.606731 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d216c51d889b417bb2e59f20b6fd6d96" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:52.505684 19298 raft_consensus.cc:515] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:52.595289 18887 consensus_queue.cc:260] T ef4e065748614b3c87fd0a834017c455 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:52.623486 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef4e065748614b3c87fd0a834017c455" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:52.614477 18841 raft_consensus.cc:2468] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:52.624392 18690 raft_consensus.cc:3060] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:52.624204 18691 raft_consensus.cc:3060] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:52.626056 18892 raft_consensus.cc:359] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:52.627104 18892 raft_consensus.cc:385] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:52.627818 18892 raft_consensus.cc:740] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:52.584949 19360 leader_election.cc:290] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:52.636045 18582 leader_election.cc:304] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:52.639151 18690 raft_consensus.cc:2468] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:52.642375 18957 raft_consensus.cc:2749] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:52.603097 19357 leader_election.cc:290] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:52.640443 18835 raft_consensus.cc:3060] T ef4e065748614b3c87fd0a834017c455 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:52.638149 18892 consensus_queue.cc:260] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:52.653337 18730 leader_election.cc:304] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:52.683149 19298 leader_election.cc:290] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:52.688387 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a46101454ef46beb00f1652fabfa153" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:52.689188 18688 raft_consensus.cc:2468] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:52.673576 19360 raft_consensus.cc:2804] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:52.696704 19360 raft_consensus.cc:697] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:52.703502 18805 leader_election.cc:304] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:52.704747 18892 ts_tablet_manager.cc:1434] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.239s	user 0.009s	sys 0.000s
I20260709 12:30:52.706130 18892 tablet_bootstrap.cc:492] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:52.701630 18691 raft_consensus.cc:2468] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:52.719175 18805 leader_election.cc:304] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:52.721822 18886 tablet_bootstrap.cc:492] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d: No bootstrap required, opened a new log
I20260709 12:30:52.722558 18886 ts_tablet_manager.cc:1403] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d: Time spent bootstrapping tablet: real 0.482s	user 0.019s	sys 0.003s
I20260709 12:30:52.723582 19348 raft_consensus.cc:2804] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:52.723745 19298 raft_consensus.cc:2804] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:52.745878 19298 raft_consensus.cc:493] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:52.746256 19298 raft_consensus.cc:3060] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:52.747325 18835 raft_consensus.cc:2468] T ef4e065748614b3c87fd0a834017c455 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:52.735193 18731 leader_election.cc:304] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:52.744197 18886 raft_consensus.cc:359] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:52.745432 19348 raft_consensus.cc:697] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:52.753976 18886 raft_consensus.cc:385] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:52.755126 18886 raft_consensus.cc:740] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Initialized, Role: FOLLOWER
I20260709 12:30:52.751705 18888 tablet_bootstrap.cc:654] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:52.738240 19360 consensus_queue.cc:237] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:52.757105 19366 raft_consensus.cc:493] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:52.762212 19366 raft_consensus.cc:515] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:52.762521 19348 consensus_queue.cc:237] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:52.758203 18886 consensus_queue.cc:260] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:52.766418 18887 ts_tablet_manager.cc:1434] T ef4e065748614b3c87fd0a834017c455 P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.277s	user 0.000s	sys 0.011s
I20260709 12:30:52.767203 18887 tablet_bootstrap.cc:492] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce: Bootstrap starting.
I20260709 12:30:52.772148 19270 raft_consensus.cc:2804] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:52.772782 19270 raft_consensus.cc:697] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:52.773623 19270 consensus_queue.cc:237] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:52.781500 18886 ts_tablet_manager.cc:1434] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d: Time spent starting tablet: real 0.059s	user 0.007s	sys 0.002s
I20260709 12:30:52.787972 18887 tablet_bootstrap.cc:654] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:52.795445 19298 raft_consensus.cc:515] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:52.797673 19298 leader_election.cc:290] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:52.801399 19366 leader_election.cc:290] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:52.802796 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6fc5ff8d0014a9f9c1dadcc157e1d9a" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:52.803285 18765 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a46101454ef46beb00f1652fabfa153" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:52.803812 18691 raft_consensus.cc:2468] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:52.804883 18731 leader_election.cc:304] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:52.805730 19366 raft_consensus.cc:2804] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:52.806087 19353 leader_election.cc:290] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:52.806089 19366 raft_consensus.cc:493] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:52.806813 19366 raft_consensus.cc:3060] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:52.844422 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a46101454ef46beb00f1652fabfa153" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:52.845099 18688 raft_consensus.cc:3060] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:30:52.842664 18805 leader_election.cc:343] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:52.825069 19226 raft_consensus.cc:493] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:52.875649 18688 raft_consensus.cc:2468] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:52.889079 18610 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0840843cb482476a8e18344ff8f1e315" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:52.890431 18837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6256d41aab434828a66720ab61fa9ba5" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
W20260709 12:30:52.890625 18563 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:30:52.890728 18609 raft_consensus.cc:3060] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:52.893013 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6fc5ff8d0014a9f9c1dadcc157e1d9a" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:52.893996 18611 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6fc5ff8d0014a9f9c1dadcc157e1d9a" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:52.878026 18514 catalog_manager.cc:5697] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:52.861541 18763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a46101454ef46beb00f1652fabfa153" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:52.895684 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6256d41aab434828a66720ab61fa9ba5" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:52.888520 19366 raft_consensus.cc:515] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:52.894851 19226 raft_consensus.cc:515] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
W20260709 12:30:52.938373 18730 leader_election.cc:343] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:52.960304 19366 leader_election.cc:290] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:52.962515 18892 tablet_bootstrap.cc:654] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:52.970520 19368 raft_consensus.cc:493] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:52.971252 19368 raft_consensus.cc:515] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
W20260709 12:30:52.974023 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:40335, user_credentials={real_user=slave}} blocked reactor thread for 120218us
I20260709 12:30:52.980533 18805 leader_election.cc:304] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:52.972950 19354 consensus_queue.cc:1048] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20260709 12:30:53.067078 18610 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ab29498797f49d198e49d29f9800da2" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:53.086527 18582 leader_election.cc:304] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
W20260709 12:30:53.091329 18731 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 176911us
I20260709 12:30:53.115486 19368 leader_election.cc:290] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
W20260709 12:30:53.129823 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 149160us
I20260709 12:30:53.133273 19395 raft_consensus.cc:2804] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:53.134382 19395 raft_consensus.cc:697] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:53.153784 18887 tablet_bootstrap.cc:492] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce: No bootstrap required, opened a new log
I20260709 12:30:53.136308 19395 consensus_queue.cc:237] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:53.156181 19226 leader_election.cc:290] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:53.155359 18614 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:53.158542 18887 ts_tablet_manager.cc:1403] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce: Time spent bootstrapping tablet: real 0.391s	user 0.010s	sys 0.032s
I20260709 12:30:53.182758 18888 tablet_bootstrap.cc:492] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:53.190531 19376 raft_consensus.cc:2749] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:53.184139 18888 ts_tablet_manager.cc:1403] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.644s	user 0.014s	sys 0.013s
I20260709 12:30:53.199301 18888 raft_consensus.cc:359] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:53.207628 18888 raft_consensus.cc:385] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:53.211593 18888 raft_consensus.cc:740] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
W20260709 12:30:53.214007 18582 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 128064us
I20260709 12:30:53.212797 18888 consensus_queue.cc:260] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:53.237985 18888 ts_tablet_manager.cc:1434] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.038s	user 0.010s	sys 0.000s
W20260709 12:30:53.227303 18804 leader_election.cc:343] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:53.240324 18888 tablet_bootstrap.cc:492] T 6a46101454ef46beb00f1652fabfa153 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:53.258136 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:53.259179 18690 raft_consensus.cc:2468] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
W20260709 12:30:53.251760 18656 leader_election.cc:343] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:53.285914 18656 leader_election.cc:304] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:53.298972 19347 raft_consensus.cc:2749] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:30:53.299412 18656 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 129507us
I20260709 12:30:53.310274 18892 tablet_bootstrap.cc:492] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:53.311496 18892 ts_tablet_manager.cc:1403] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.606s	user 0.032s	sys 0.020s
I20260709 12:30:53.311957 18888 tablet_bootstrap.cc:654] T 6a46101454ef46beb00f1652fabfa153 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:53.331179 18730 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 350040us
I20260709 12:30:53.332962 18805 leader_election.cc:304] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:53.335747 19234 raft_consensus.cc:2804] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:53.336323 19234 raft_consensus.cc:493] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:30:53.336484 18656 leader_election.cc:343] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:53.340029 18514 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:43926 (request call id 72) took 1303 ms. Trace:
I20260709 12:30:53.305178 18887 raft_consensus.cc:359] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:53.340420 18656 leader_election.cc:304] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:53.341075 18887 raft_consensus.cc:385] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:53.341439 18887 raft_consensus.cc:740] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Initialized, Role: FOLLOWER
I20260709 12:30:53.340134 19234 raft_consensus.cc:3060] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:53.346254 19347 raft_consensus.cc:2749] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:53.342697 18887 consensus_queue.cc:260] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:53.356807 18887 ts_tablet_manager.cc:1434] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce: Time spent starting tablet: real 0.149s	user 0.008s	sys 0.002s
I20260709 12:30:53.356319 18892 raft_consensus.cc:359] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:53.357594 18892 raft_consensus.cc:385] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:53.358003 18892 raft_consensus.cc:740] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:53.359086 18892 consensus_queue.cc:260] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:53.390908 19360 raft_consensus.cc:493] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df352ed4600b4ddb83ef57ecd2944fce)
I20260709 12:30:53.391914 19360 raft_consensus.cc:515] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:53.389446 19226 consensus_queue.cc:1048] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:30:53.395607 19360 leader_election.cc:290] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:53.404845 18690 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6fc5ff8d0014a9f9c1dadcc157e1d9a" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:53.405793 18690 raft_consensus.cc:3060] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:53.409934 18614 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:53.411226 18837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:53.340828 18514 rpcz_store.cc:276] 0709 12:30:52.069814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:30:52.070135 (+   321us) service_pool.cc:224] Handling call
0709 12:30:53.339839 (+1269704us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:30:53.125345 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:30:53.126715 (+  1370us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:30:53.126829 (+   114us) write_op.cc:435] Acquired schema lock
0709 12:30:53.126875 (+    46us) tablet.cc:662] Decoding operations
0709 12:30:53.127748 (+   873us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:30:53.127883 (+   135us) write_op.cc:641] Partition lock acquired for write op
0709 12:30:53.127923 (+    40us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:30:53.128313 (+   390us) tablet.cc:701] Row locks acquired
0709 12:30:53.143918 (+ 15605us) write_op.cc:260] PREPARE: finished
0709 12:30:53.160242 (+ 16324us) write_op.cc:270] Start()
0709 12:30:53.160512 (+   270us) write_op.cc:276] Timestamp: P: 1783600253160160 usec, L: 0
0709 12:30:53.160556 (+    44us) op_driver.cc:348] REPLICATION: starting
0709 12:30:53.161710 (+  1154us) log.cc:844] Serialized 531 byte log entry
0709 12:30:53.205159 (+ 43449us) op_driver.cc:464] REPLICATION: finished
0709 12:30:53.209466 (+  4307us) write_op.cc:301] APPLY: starting
0709 12:30:53.210130 (+   664us) tablet.cc:1367] starting BulkCheckPresence
0709 12:30:53.210330 (+   200us) tablet.cc:1370] finished BulkCheckPresence
0709 12:30:53.210580 (+   250us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:30:53.211078 (+   498us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:30:53.220042 (+  8964us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:30:53.220174 (+   132us) write_op.cc:312] APPLY: finished
0709 12:30:53.220700 (+   526us) log.cc:844] Serialized 22 byte log entry
0709 12:30:53.220974 (+   274us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:53.224779 (+  3805us) write_op.cc:454] Released schema lock
0709 12:30:53.231310 (+  6531us) write_op.cc:341] FINISH: Updating metrics
0709 12:30:53.234316 (+  3006us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:30:53.234360 (+    44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":40578,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":3877,"apply.run_cpu_time_us":3659,"apply.run_wall_time_us":24783,"num_ops":1,"prepare.queue_time_us":18903,"prepare.run_cpu_time_us":5590,"prepare.run_wall_time_us":37302,"replication_time_us":44380}]]}
W20260709 12:30:53.424561 18804 consensus_peers.cc:597] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:53.424973 18888 tablet_bootstrap.cc:492] T 6a46101454ef46beb00f1652fabfa153 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:53.425372 18892 ts_tablet_manager.cc:1434] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.114s	user 0.010s	sys 0.001s
I20260709 12:30:53.425590 18888 ts_tablet_manager.cc:1403] T 6a46101454ef46beb00f1652fabfa153 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.186s	user 0.013s	sys 0.021s
I20260709 12:30:53.427894 18892 tablet_bootstrap.cc:492] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:53.405522 18512 catalog_manager.cc:5697] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:53.437366 18888 raft_consensus.cc:359] T 6a46101454ef46beb00f1652fabfa153 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:53.440709 18614 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6fc5ff8d0014a9f9c1dadcc157e1d9a" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
W20260709 12:30:53.444082 18805 leader_election.cc:343] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:53.446869 18690 raft_consensus.cc:2468] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:53.447492 18888 raft_consensus.cc:385] T 6a46101454ef46beb00f1652fabfa153 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:53.448607 18888 raft_consensus.cc:740] T 6a46101454ef46beb00f1652fabfa153 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:53.450076 18888 consensus_queue.cc:260] T 6a46101454ef46beb00f1652fabfa153 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:53.450325 18512 catalog_manager.cc:5697] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:53.451133 18614 raft_consensus.cc:3060] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:53.475634 18888 ts_tablet_manager.cc:1434] T 6a46101454ef46beb00f1652fabfa153 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.050s	user 0.008s	sys 0.002s
I20260709 12:30:53.467231 19234 raft_consensus.cc:515] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:53.481508 18837 raft_consensus.cc:2393] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d61e0800194dd89be34d3aff3f374d in current term 1: Already voted for candidate df352ed4600b4ddb83ef57ecd2944fce in this term.
I20260709 12:30:53.486874 19270 raft_consensus.cc:493] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df352ed4600b4ddb83ef57ecd2944fce)
I20260709 12:30:53.487402 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14206c16a2154538ad2a781e824a65dd" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:53.487633 19270 raft_consensus.cc:515] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:30:53.490464 19270 leader_election.cc:290] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:53.493465 18731 leader_election.cc:304] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:53.496454 19405 raft_consensus.cc:2804] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:53.497051 19405 raft_consensus.cc:697] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:53.498282 19360 consensus_queue.cc:1048] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:53.502205 19405 consensus_queue.cc:237] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:53.518031 19234 leader_election.cc:290] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:53.528637 18888 tablet_bootstrap.cc:492] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:53.529417 18609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:53.522640 18614 raft_consensus.cc:2468] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
W20260709 12:30:53.532197 18804 leader_election.cc:343] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:30:53.533995 18656 consensus_peers.cc:597] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:53.534807 18656 leader_election.cc:343] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:53.542001 18892 tablet_bootstrap.cc:654] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
W20260709 12:30:53.544904 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:40335, user_credentials={real_user=slave}} blocked reactor thread for 100894us
I20260709 12:30:53.545646 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d163b2d60ca4572bca7a1252c65277a" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:53.551138 18685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:53.551903 18685 raft_consensus.cc:3060] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:53.563555 18888 tablet_bootstrap.cc:654] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:53.581787 18656 leader_election.cc:304] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:53.599261 18516 catalog_manager.cc:5697] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 0 to 1, leader changed from <none> to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 1 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:53.591490 18514 catalog_manager.cc:5697] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:53.602793 19405 consensus_queue.cc:1048] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:53.604060 18514 catalog_manager.cc:5697] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
W20260709 12:30:53.608165 18730 leader_election.cc:343] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:53.609578 18685 raft_consensus.cc:2468] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:53.627322 19226 raft_consensus.cc:2749] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:30:53.621759 18730 leader_election.cc:343] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:53.627545 18888 tablet_bootstrap.cc:492] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:53.629693 18888 ts_tablet_manager.cc:1403] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.149s	user 0.018s	sys 0.004s
I20260709 12:30:53.634156 18892 tablet_bootstrap.cc:492] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:53.634725 18892 ts_tablet_manager.cc:1403] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.207s	user 0.009s	sys 0.009s
I20260709 12:30:53.635815 19397 consensus_queue.cc:1048] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260709 12:30:53.642874 18805 leader_election.cc:304] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:53.646847 18888 raft_consensus.cc:359] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:53.648685 18888 raft_consensus.cc:385] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:53.648990 18888 raft_consensus.cc:740] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:53.647359 19234 raft_consensus.cc:2804] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:53.654768 19234 raft_consensus.cc:697] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:53.664836 19355 consensus_queue.cc:1048] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:30:53.665400 18892 raft_consensus.cc:359] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:53.666316 18892 raft_consensus.cc:385] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:53.666796 18892 raft_consensus.cc:740] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:53.669059 18888 consensus_queue.cc:260] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
W20260709 12:30:53.675279 18730 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 53629.6us
I20260709 12:30:53.670950 18892 consensus_queue.cc:260] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
W20260709 12:30:53.684734 18730 consensus_peers.cc:597] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:53.674479 19234 consensus_queue.cc:237] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:53.719972 19270 raft_consensus.cc:493] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:53.720527 19270 raft_consensus.cc:515] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:53.722671 19270 leader_election.cc:290] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:53.726028 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d163b2d60ca4572bca7a1252c65277a" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:53.726956 18888 ts_tablet_manager.cc:1434] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.097s	user 0.007s	sys 0.001s
I20260709 12:30:53.728470 18888 tablet_bootstrap.cc:492] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01: Bootstrap starting.
I20260709 12:30:53.728569 19395 raft_consensus.cc:493] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:53.729173 19395 raft_consensus.cc:515] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:53.740865 18837 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14206c16a2154538ad2a781e824a65dd" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:53.741716 19395 leader_election.cc:290] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:53.744729 18617 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4c43366d9fb416e8649aba9676aed59" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:53.745414 18617 raft_consensus.cc:2468] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:53.747170 18839 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4c43366d9fb416e8649aba9676aed59" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:53.747903 18839 raft_consensus.cc:2468] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:53.763731 18611 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:53.764165 18730 leader_election.cc:304] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 
W20260709 12:30:53.767252 18804 leader_election.cc:343] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:53.765727 18730 leader_election.cc:304] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:53.771656 18892 ts_tablet_manager.cc:1434] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.137s	user 0.007s	sys 0.002s
I20260709 12:30:53.785270 18730 leader_election.cc:304] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:53.777799 19350 raft_consensus.cc:2804] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:53.791167 19350 raft_consensus.cc:493] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:53.791689 19350 raft_consensus.cc:3060] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:53.793025 18892 tablet_bootstrap.cc:492] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:53.794121 19405 raft_consensus.cc:2749] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:53.806839 19200 consensus_queue.cc:1048] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 12:30:53.808277 19346 raft_consensus.cc:2749] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:53.808890 18892 tablet_bootstrap.cc:654] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:53.814559 19354 raft_consensus.cc:493] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:53.815454 19354 raft_consensus.cc:515] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:53.859952 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:53.860661 18841 raft_consensus.cc:2468] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
I20260709 12:30:53.862176 18730 leader_election.cc:304] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:53.877318 18763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:53.878084 18763 raft_consensus.cc:2468] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 0.
I20260709 12:30:53.880578 18761 raft_consensus.cc:3060] T 6a46101454ef46beb00f1652fabfa153 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:53.881666 19405 raft_consensus.cc:2804] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:53.882126 19405 raft_consensus.cc:493] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:53.882500 19405 raft_consensus.cc:3060] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:53.886627 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:53.886888 19354 leader_election.cc:290] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
W20260709 12:30:53.890476 18730 leader_election.cc:343] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:53.903314 18513 catalog_manager.cc:5697] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:53.909577 18888 tablet_bootstrap.cc:654] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:53.929760 18805 leader_election.cc:304] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:53.948436 19210 raft_consensus.cc:2804] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:53.948870 19210 raft_consensus.cc:493] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:53.949211 19210 raft_consensus.cc:3060] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:53.962956 19350 raft_consensus.cc:515] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:54.004105 18892 tablet_bootstrap.cc:492] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:54.004837 18892 ts_tablet_manager.cc:1403] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.212s	user 0.029s	sys 0.011s
I20260709 12:30:54.010314 19350 leader_election.cc:290] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:54.017778 19405 raft_consensus.cc:515] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:54.022502 18611 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4c43366d9fb416e8649aba9676aed59" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:54.029942 18611 raft_consensus.cc:3060] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:54.027926 18835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4c43366d9fb416e8649aba9676aed59" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:54.032913 18835 raft_consensus.cc:3060] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:54.047333 18888 tablet_bootstrap.cc:492] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01: No bootstrap required, opened a new log
I20260709 12:30:54.048086 18888 ts_tablet_manager.cc:1403] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01: Time spent bootstrapping tablet: real 0.320s	user 0.021s	sys 0.008s
I20260709 12:30:54.060091 18892 raft_consensus.cc:359] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:54.060887 18892 raft_consensus.cc:385] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:54.061173 18892 raft_consensus.cc:740] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:54.061971 18892 consensus_queue.cc:260] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:54.079720 19405 leader_election.cc:290] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:54.111802 18611 raft_consensus.cc:2468] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:54.105448 19346 consensus_queue.cc:1048] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260709 12:30:54.083420 18892 ts_tablet_manager.cc:1434] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.075s	user 0.005s	sys 0.000s
I20260709 12:30:54.081941 19210 raft_consensus.cc:515] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:54.121863 19270 consensus_queue.cc:1048] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:54.122824 18892 tablet_bootstrap.cc:492] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:54.081135 18618 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:54.124785 18609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:54.072170 18835 raft_consensus.cc:2468] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
W20260709 12:30:54.126466 18804 leader_election.cc:343] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:30:54.128531 18730 leader_election.cc:343] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:54.129287 18730 leader_election.cc:304] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:30:54.071568 18513 catalog_manager.cc:5697] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 1, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 1 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:54.119036 18888 raft_consensus.cc:359] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:54.133967 18888 raft_consensus.cc:385] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:54.134373 18888 raft_consensus.cc:740] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Initialized, Role: FOLLOWER
I20260709 12:30:54.081871 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:54.138185 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:54.138495 18838 raft_consensus.cc:2393] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 29f82a8594d741bc92dd24ee4a368e01 in current term 1: Already voted for candidate df352ed4600b4ddb83ef57ecd2944fce in this term.
I20260709 12:30:54.139003 18761 raft_consensus.cc:2393] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df352ed4600b4ddb83ef57ecd2944fce in current term 1: Already voted for candidate 29f82a8594d741bc92dd24ee4a368e01 in this term.
I20260709 12:30:54.137074 19270 raft_consensus.cc:2804] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:54.141232 19270 raft_consensus.cc:697] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:54.142122 19270 consensus_queue.cc:237] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:54.114360 19433 consensus_queue.cc:1048] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20260709 12:30:54.149080 18730 leader_election.cc:304] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:54.149890 18805 leader_election.cc:304] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:54.156853 19433 raft_consensus.cc:2749] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:30:54.149889 19210 leader_election.cc:290] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 1 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:54.160531 18888 consensus_queue.cc:260] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:54.173445 19346 raft_consensus.cc:2749] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:30:54.176190 18892 tablet_bootstrap.cc:654] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:54.180738 18888 ts_tablet_manager.cc:1434] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01: Time spent starting tablet: real 0.132s	user 0.007s	sys 0.001s
W20260709 12:30:54.233598 18804 consensus_peers.cc:597] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:54.237365 19210 consensus_queue.cc:1048] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:54.266142 18515 catalog_manager.cc:5697] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:54.290122 18892 tablet_bootstrap.cc:492] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:54.290721 18892 ts_tablet_manager.cc:1403] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.168s	user 0.026s	sys 0.025s
I20260709 12:30:54.295535 18892 raft_consensus.cc:359] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:54.296422 18892 raft_consensus.cc:385] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:54.296716 18892 raft_consensus.cc:740] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:54.297515 18892 consensus_queue.cc:260] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:54.299660 18892 ts_tablet_manager.cc:1434] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:30:54.300745 18892 tablet_bootstrap.cc:492] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:54.301414 19346 raft_consensus.cc:493] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:54.301955 19346 raft_consensus.cc:515] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:54.309257 19346 leader_election.cc:290] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:54.309494 18609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0840843cb482476a8e18344ff8f1e315" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:54.309476 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0840843cb482476a8e18344ff8f1e315" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:54.311002 18688 raft_consensus.cc:2468] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 0.
W20260709 12:30:54.311473 18730 leader_election.cc:343] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:54.314342 18731 leader_election.cc:304] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:54.316176 19405 raft_consensus.cc:2804] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:30:54.317513 19405 raft_consensus.cc:493] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:54.317880 19405 raft_consensus.cc:3060] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:54.319567 18892 tablet_bootstrap.cc:654] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:54.358963 19405 raft_consensus.cc:515] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:54.387405 18892 tablet_bootstrap.cc:492] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:54.388073 18892 ts_tablet_manager.cc:1403] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.088s	user 0.026s	sys 0.013s
I20260709 12:30:54.391163 18892 raft_consensus.cc:359] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:54.393270 18892 raft_consensus.cc:385] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:54.393548 18892 raft_consensus.cc:740] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
W20260709 12:30:54.393680 18563 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:30:54.394268 18892 consensus_queue.cc:260] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:54.398379 18892 ts_tablet_manager.cc:1434] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.002s
I20260709 12:30:54.400058 18892 tablet_bootstrap.cc:492] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:54.402666 18611 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0840843cb482476a8e18344ff8f1e315" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
W20260709 12:30:54.404740 18730 leader_election.cc:343] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:30:54.407938 18892 tablet_bootstrap.cc:654] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:54.408996 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0840843cb482476a8e18344ff8f1e315" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:54.411963 18691 raft_consensus.cc:3060] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:54.427927 19405 leader_election.cc:290] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:54.490157 18892 tablet_bootstrap.cc:492] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:54.490787 18892 ts_tablet_manager.cc:1403] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.091s	user 0.022s	sys 0.027s
I20260709 12:30:54.447773 18691 raft_consensus.cc:2468] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:54.495162 18731 leader_election.cc:304] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:54.495265 18892 raft_consensus.cc:359] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:54.496389 18892 raft_consensus.cc:385] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:54.496735 18892 raft_consensus.cc:740] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:54.497905 18892 consensus_queue.cc:260] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:54.498658 19412 raft_consensus.cc:493] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:54.499171 19377 raft_consensus.cc:493] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:54.499312 19412 raft_consensus.cc:515] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:54.499847 19377 raft_consensus.cc:515] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:54.500846 19446 raft_consensus.cc:493] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:54.501453 19446 raft_consensus.cc:515] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:54.504031 19446 leader_election.cc:290] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:54.505779 18685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d216c51d889b417bb2e59f20b6fd6d96" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:54.507619 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0840843cb482476a8e18344ff8f1e315" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:54.507433 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d216c51d889b417bb2e59f20b6fd6d96" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:54.504978 19377 leader_election.cc:290] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:54.511333 18582 leader_election.cc:304] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:54.513195 18892 ts_tablet_manager.cc:1434] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.022s	user 0.010s	sys 0.001s
I20260709 12:30:54.513307 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ab29498797f49d198e49d29f9800da2" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:54.518289 19377 raft_consensus.cc:2749] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:54.496599 19405 raft_consensus.cc:2804] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:30:54.521334 19405 raft_consensus.cc:697] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:54.522330 19405 consensus_queue.cc:237] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:54.519320 19412 leader_election.cc:290] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:54.523490 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0840843cb482476a8e18344ff8f1e315" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:54.524008 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ab29498797f49d198e49d29f9800da2" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:54.524735 18691 raft_consensus.cc:2393] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84812365c65f43f388836c087640adbb in current term 1: Already voted for candidate 29f82a8594d741bc92dd24ee4a368e01 in this term.
I20260709 12:30:54.525959 18581 leader_election.cc:304] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:54.528244 19412 raft_consensus.cc:2749] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:54.514676 18892 tablet_bootstrap.cc:492] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:54.532949 18611 raft_consensus.cc:3060] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:54.540828 19350 consensus_queue.cc:1048] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:54.576891 19412 raft_consensus.cc:493] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:54.577543 19412 raft_consensus.cc:515] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:54.580942 18835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:54.585121 19412 leader_election.cc:290] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:54.586634 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:54.620796 19200 consensus_queue.cc:1048] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:54.639163 18582 leader_election.cc:304] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:54.647542 19412 raft_consensus.cc:2749] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:54.667287 18892 tablet_bootstrap.cc:654] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:54.701562 18582 leader_election.cc:304] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:30:54.702930 19412 raft_consensus.cc:3060] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:54.714568 18892 tablet_bootstrap.cc:492] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:54.715170 18892 ts_tablet_manager.cc:1403] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.201s	user 0.012s	sys 0.006s
I20260709 12:30:54.717958 18892 raft_consensus.cc:359] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:54.718655 18892 raft_consensus.cc:385] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:54.718912 18892 raft_consensus.cc:740] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:54.719576 18892 consensus_queue.cc:260] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:54.733987 18892 ts_tablet_manager.cc:1434] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.019s	user 0.004s	sys 0.004s
I20260709 12:30:54.734005 19412 raft_consensus.cc:2749] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:30:54.736073 18892 tablet_bootstrap.cc:492] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:54.736938 18611 raft_consensus.cc:3060] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:54.743731 18892 tablet_bootstrap.cc:654] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb: Neither blocks nor log segments found. Creating new log.
I20260709 12:30:54.754047 18515 catalog_manager.cc:5697] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 1 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:54.795359 18892 tablet_bootstrap.cc:492] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb: No bootstrap required, opened a new log
I20260709 12:30:54.796077 18892 ts_tablet_manager.cc:1403] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.060s	user 0.016s	sys 0.012s
I20260709 12:30:54.798879 18892 raft_consensus.cc:359] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:54.806356 18892 raft_consensus.cc:385] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:30:54.806771 18892 raft_consensus.cc:740] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:54.807523 18892 consensus_queue.cc:260] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:54.809952 18892 ts_tablet_manager.cc:1434] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.000s
I20260709 12:30:54.838330 18611 raft_consensus.cc:3060] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:54.931805 19405 consensus_queue.cc:1048] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:54.951376 18611 raft_consensus.cc:3060] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:54.956405 19405 consensus_queue.cc:1048] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:30:55.180716 19377 raft_consensus.cc:493] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:55.181342 19377 raft_consensus.cc:515] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:55.183825 19377 leader_election.cc:290] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:30:55.184682 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:55.185149 18835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:55.185940 18767 raft_consensus.cc:2393] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84812365c65f43f388836c087640adbb in current term 1: Already voted for candidate 29f82a8594d741bc92dd24ee4a368e01 in this term.
I20260709 12:30:55.186283 18835 raft_consensus.cc:2393] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84812365c65f43f388836c087640adbb in current term 1: Already voted for candidate df352ed4600b4ddb83ef57ecd2944fce in this term.
I20260709 12:30:55.187722 18581 leader_election.cc:304] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:55.188802 19377 raft_consensus.cc:3060] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:30:55.200542 19377 raft_consensus.cc:2749] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:30:55.456204 18563 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:30:55.731123 19210 raft_consensus.cc:493] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:30:55.731704 19210 raft_consensus.cc:515] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:55.733953 19210 leader_election.cc:290] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:55.735234 18609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb" is_pre_election: true
I20260709 12:30:55.735461 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:55.735967 18609 raft_consensus.cc:2468] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:55.736205 18767 raft_consensus.cc:2468] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:55.737125 18804 leader_election.cc:304] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:55.737821 19210 raft_consensus.cc:2804] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:30:55.738253 19210 raft_consensus.cc:493] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:30:55.738623 19210 raft_consensus.cc:3060] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:55.747517 19210 raft_consensus.cc:515] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:55.749934 19210 leader_election.cc:290] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:55.750947 18609 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84812365c65f43f388836c087640adbb"
I20260709 12:30:55.750967 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:55.751885 18609 raft_consensus.cc:3060] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:55.751967 18767 raft_consensus.cc:3060] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:55.757740 18609 raft_consensus.cc:2468] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 2.
I20260709 12:30:55.759630 18804 leader_election.cc:304] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:55.761248 18767 raft_consensus.cc:2468] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 2.
I20260709 12:30:55.762068 19210 raft_consensus.cc:2804] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:30:55.764139 19210 raft_consensus.cc:697] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 2 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:55.765029 19210 consensus_queue.cc:237] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:55.796007 18515 catalog_manager.cc:5697] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 0 to 2, leader changed from <none> to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 2 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:56.274163 18609 raft_consensus.cc:1275] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Refusing update from remote peer df352ed4600b4ddb83ef57ecd2944fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:30:56.275349 19210 consensus_queue.cc:1048] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:56.288679 18767 raft_consensus.cc:1275] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Refusing update from remote peer df352ed4600b4ddb83ef57ecd2944fce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:30:56.290340 19210 consensus_queue.cc:1048] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:30:57.650239  4356 tablet_server.cc:179] TabletServer@127.4.65.1:0 shutting down...
W20260709 12:30:57.667680 18804 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:30:57.675720 18804 consensus_peers.cc:597] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:57.679961 18804 consensus_peers.cc:597] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:57.705896 18730 consensus_peers.cc:597] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:57.716883 18804 consensus_peers.cc:597] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:57.718538 18730 consensus_peers.cc:597] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:57.719242 18730 consensus_peers.cc:597] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:57.721515 18730 consensus_peers.cc:597] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:57.722147 18730 consensus_peers.cc:597] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:57.735313 18804 consensus_peers.cc:597] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:57.743698 18804 consensus_peers.cc:597] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:57.759271 18730 consensus_peers.cc:597] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:57.760026 18730 consensus_peers.cc:597] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.766397  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:30:57.767633  4356 tablet_replica.cc:333] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.768687  4356 raft_consensus.cc:2243] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.769326  4356 raft_consensus.cc:2272] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.772039  4356 tablet_replica.cc:333] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.773082  4356 raft_consensus.cc:2243] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.773622  4356 raft_consensus.cc:2272] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.776257  4356 tablet_replica.cc:333] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.777173  4356 raft_consensus.cc:2243] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.777729  4356 raft_consensus.cc:2272] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.780308  4356 tablet_replica.cc:333] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.781176  4356 raft_consensus.cc:2243] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.781754  4356 raft_consensus.cc:2272] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.784413  4356 tablet_replica.cc:333] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.789546  4356 raft_consensus.cc:2243] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.790132  4356 raft_consensus.cc:2272] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.792613  4356 tablet_replica.cc:333] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.793292  4356 raft_consensus.cc:2243] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.793866  4356 raft_consensus.cc:2272] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.796814  4356 tablet_replica.cc:333] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.797837  4356 raft_consensus.cc:2243] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.798574  4356 raft_consensus.cc:2272] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.801909  4356 tablet_replica.cc:333] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.802980  4356 raft_consensus.cc:2243] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.803527  4356 raft_consensus.cc:2272] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.806063  4356 tablet_replica.cc:333] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.806762  4356 raft_consensus.cc:2243] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.807271  4356 raft_consensus.cc:2272] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.810659  4356 tablet_replica.cc:333] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.811829  4356 raft_consensus.cc:2243] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.812628  4356 raft_consensus.cc:2272] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.816594  4356 tablet_replica.cc:333] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.817579  4356 raft_consensus.cc:2243] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.818159  4356 raft_consensus.cc:2272] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:57.819299 18804 consensus_peers.cc:597] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.820552  4356 tablet_replica.cc:333] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.821260  4356 raft_consensus.cc:2243] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.821720  4356 raft_consensus.cc:2272] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.823961  4356 tablet_replica.cc:333] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.824913  4356 raft_consensus.cc:2243] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.825485  4356 raft_consensus.cc:2272] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.828612  4356 tablet_replica.cc:333] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.829478  4356 raft_consensus.cc:2243] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.830055  4356 raft_consensus.cc:2272] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.833320  4356 tablet_replica.cc:333] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.834268  4356 raft_consensus.cc:2243] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:30:57.834177 18656 consensus_peers.cc:597] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.835026  4356 raft_consensus.cc:2272] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.837297  4356 tablet_replica.cc:333] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.837909  4356 raft_consensus.cc:2243] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.838462  4356 raft_consensus.cc:2272] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.841673  4356 tablet_replica.cc:333] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.842552  4356 raft_consensus.cc:2243] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.843348  4356 raft_consensus.cc:2272] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.847232  4356 tablet_replica.cc:333] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb: stopping tablet replica
W20260709 12:30:57.847998 18730 consensus_peers.cc:597] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.848093  4356 raft_consensus.cc:2243] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.849429  4356 raft_consensus.cc:2272] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.852298  4356 tablet_replica.cc:333] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.853222  4356 raft_consensus.cc:2243] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.853835  4356 raft_consensus.cc:2272] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.857010  4356 tablet_replica.cc:333] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.857870  4356 raft_consensus.cc:2243] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:30:57.858222 18804 consensus_peers.cc:597] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:57.858624 18656 consensus_peers.cc:597] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.858423  4356 raft_consensus.cc:2272] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.862561  4356 tablet_replica.cc:333] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.863595  4356 raft_consensus.cc:2243] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.864284  4356 raft_consensus.cc:2272] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:57.866007 18730 consensus_peers.cc:597] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.867830  4356 tablet_replica.cc:333] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb: stopping tablet replica
W20260709 12:30:57.868950 18730 consensus_peers.cc:597] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.869004  4356 raft_consensus.cc:2243] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.869972  4356 raft_consensus.cc:2272] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.873003  4356 tablet_replica.cc:333] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.873867  4356 raft_consensus.cc:2243] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.874506  4356 raft_consensus.cc:2272] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:57.877995 18730 consensus_peers.cc:597] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.878435  4356 tablet_replica.cc:333] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.879458  4356 raft_consensus.cc:2243] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.880427  4356 raft_consensus.cc:2272] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.884600  4356 tablet_replica.cc:333] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.885751  4356 raft_consensus.cc:2243] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:30:57.886739  4356 raft_consensus.cc:2272] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.889891  4356 tablet_replica.cc:333] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.890785  4356 raft_consensus.cc:2243] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.891338  4356 raft_consensus.cc:2272] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:57.894162 18804 consensus_peers.cc:597] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.894604  4356 tablet_replica.cc:333] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.895897  4356 raft_consensus.cc:2243] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:30:57.897142  4356 raft_consensus.cc:2272] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:57.898257 18656 consensus_peers.cc:597] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.900233  4356 tablet_replica.cc:333] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.901227  4356 raft_consensus.cc:2243] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.901839  4356 raft_consensus.cc:2272] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:57.902547 18656 consensus_peers.cc:597] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.905488  4356 tablet_replica.cc:333] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.906410  4356 raft_consensus.cc:2243] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.906967  4356 raft_consensus.cc:2272] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.910334  4356 tablet_replica.cc:333] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.911181  4356 raft_consensus.cc:2243] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.911744  4356 raft_consensus.cc:2272] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.915025  4356 tablet_replica.cc:333] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.915939  4356 raft_consensus.cc:2243] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:30:57.917222  4356 raft_consensus.cc:2272] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.920423  4356 tablet_replica.cc:333] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.921257  4356 raft_consensus.cc:2243] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.921825  4356 raft_consensus.cc:2272] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.925215  4356 tablet_replica.cc:333] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.926065  4356 raft_consensus.cc:2243] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Raft consensus shutting down.
W20260709 12:30:57.926971 18804 consensus_peers.cc:597] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.927207  4356 raft_consensus.cc:2272] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:57.930032 18730 consensus_peers.cc:597] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.930440  4356 tablet_replica.cc:333] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.931339  4356 raft_consensus.cc:2243] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:30:57.932682  4356 raft_consensus.cc:2272] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.935734  4356 tablet_replica.cc:333] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.936553  4356 raft_consensus.cc:2243] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.937134  4356 raft_consensus.cc:2272] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.940034  4356 tablet_replica.cc:333] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.940871  4356 raft_consensus.cc:2243] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:30:57.941252 18730 consensus_peers.cc:597] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.941490  4356 raft_consensus.cc:2272] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.944732  4356 tablet_replica.cc:333] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.945596  4356 raft_consensus.cc:2243] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.946149  4356 raft_consensus.cc:2272] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.949208  4356 tablet_replica.cc:333] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.950102  4356 raft_consensus.cc:2243] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:30:57.951236  4356 raft_consensus.cc:2272] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.954447  4356 tablet_replica.cc:333] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.955250  4356 raft_consensus.cc:2243] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.955850  4356 raft_consensus.cc:2272] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.958909  4356 tablet_replica.cc:333] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.959759  4356 raft_consensus.cc:2243] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.960340  4356 raft_consensus.cc:2272] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.963449  4356 tablet_replica.cc:333] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.964210  4356 raft_consensus.cc:2243] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:30:57.965431  4356 raft_consensus.cc:2272] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.968554  4356 tablet_replica.cc:333] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.969358  4356 raft_consensus.cc:2243] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:30:57.969511 18804 consensus_peers.cc:597] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.969996  4356 raft_consensus.cc:2272] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:57.972085 18730 consensus_peers.cc:597] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.973155  4356 tablet_replica.cc:333] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.974020  4356 raft_consensus.cc:2243] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.974663  4356 raft_consensus.cc:2272] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:57.977025 18804 consensus_peers.cc:597] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:57.977926  4356 tablet_replica.cc:333] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.978744  4356 raft_consensus.cc:2243] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:30:57.979362  4356 raft_consensus.cc:2272] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:30:57.982236  4356 tablet_replica.cc:333] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb: stopping tablet replica
I20260709 12:30:57.982991  4356 raft_consensus.cc:2243] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:30:57.984112  4356 raft_consensus.cc:2272] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:30:57.994593 18804 consensus_peers.cc:597] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:57.996711 18656 consensus_peers.cc:597] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:58.054066 18804 consensus_peers.cc:597] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:58.069335 18656 consensus_peers.cc:597] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:58.071460 18656 consensus_peers.cc:597] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:58.079154  4356 tablet_server.cc:196] TabletServer@127.4.65.1:0 shutdown complete.
W20260709 12:30:58.133388 18730 consensus_peers.cc:597] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:58.150501 18730 consensus_peers.cc:597] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:30:58.158757 18804 consensus_peers.cc:597] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:58.230707  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:30:58.253072 19484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:30:58.261761  4356 server_base.cc:1061] running on GCE node
W20260709 12:30:58.266232 19487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:30:58.279261 19483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:30:58.286453  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:30:58.286818  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:30:58.287081  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600258287062 us; error 0 us; skew 500 ppm
I20260709 12:30:58.287923  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:30:58.312275  4356 webserver.cc:533] Webserver started at http://127.4.65.1:45235/ using document root <none> and password file <none>
I20260709 12:30:58.313169  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:30:58.313434  4356 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 12:30:58.337324  4356 fs_manager.cc:714] Time spent opening directory manager: real 0.022s	user 0.006s	sys 0.004s
I20260709 12:30:58.375584 19495 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:30:58.377810  4356 fs_manager.cc:730] Time spent opening block manager: real 0.030s	user 0.005s	sys 0.008s
I20260709 12:30:58.378175  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-0-root
uuid: "84812365c65f43f388836c087640adbb"
format_stamp: "Formatted at 2026-07-09 12:30:40 on dist-test-slave-l1c9"
I20260709 12:30:58.378628  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599783304823-4356-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:30:58.491891  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:30:58.493659  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:30:58.558323  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:30:58.604578 19505 ts_tablet_manager.cc:548] Loading tablet metadata (0/45 complete)
I20260709 12:30:58.902272 19499 raft_consensus.cc:493] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:58.902873 19499 raft_consensus.cc:515] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:58.913169 19499 leader_election.cc:290] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:58.913847 18760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c180f8ddcbe14366a60537e1bad39332" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
W20260709 12:30:58.915107 18656 leader_election.cc:336] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:58.915820 18657 leader_election.cc:304] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:58.917659 19499 raft_consensus.cc:2749] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:58.966166  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (45 total tablets, 45 live tablets)
I20260709 12:30:58.966738  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.371s	user 0.006s	sys 0.006s
I20260709 12:30:58.967202  4356 ts_tablet_manager.cc:600] Registering tablets (0/45 complete)
I20260709 12:30:58.970108 19496 raft_consensus.cc:493] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:58.970845 19496 raft_consensus.cc:515] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:58.973289 19496 leader_election.cc:290] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:58.975337 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e989970204ce48b700c86d5faf91d" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:58.975950 19505 tablet_bootstrap.cc:492] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:30:58.992166 18804 leader_election.cc:336] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:58.996258 19496 raft_consensus.cc:493] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:58.996905 19496 raft_consensus.cc:515] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:58.999173 19496 leader_election.cc:290] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:59.000684 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7b389579627488bb406ec0d112f6938" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:58.992846 18804 leader_election.cc:304] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb
I20260709 12:30:59.010891 19496 raft_consensus.cc:2749] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:59.017921 19505 tablet_bootstrap.cc:492] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.019157 19505 tablet_bootstrap.cc:492] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.019927 19505 ts_tablet_manager.cc:1403] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.044s	user 0.019s	sys 0.016s
I20260709 12:30:59.023533 19505 raft_consensus.cc:359] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:59.024372 19505 raft_consensus.cc:740] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.025349 19505 consensus_queue.cc:260] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
W20260709 12:30:59.034387 18804 leader_election.cc:336] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.034974 18804 leader_election.cc:304] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb
I20260709 12:30:59.035693 19210 raft_consensus.cc:2749] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:59.037395 19505 ts_tablet_manager.cc:1434] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.017s	user 0.001s	sys 0.010s
I20260709 12:30:59.038444 19505 tablet_bootstrap.cc:492] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:59.040776 19498 raft_consensus.cc:493] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.041319 19498 raft_consensus.cc:515] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.043637 19498 leader_election.cc:290] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:59.045141 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "872ec24b41d24573a9a8cc1d484fd6f9" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
W20260709 12:30:59.050637 18656 leader_election.cc:336] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.051100 18656 leader_election.cc:304] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:30:59.052708 19498 raft_consensus.cc:2749] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:59.060945 19505 tablet_bootstrap.cc:492] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.062337 19505 tablet_bootstrap.cc:492] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.063143 19505 ts_tablet_manager.cc:1403] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.025s	user 0.015s	sys 0.008s
I20260709 12:30:59.065984 19505 raft_consensus.cc:359] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.066780 19505 raft_consensus.cc:740] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.067685 19505 consensus_queue.cc:260] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.069679 19505 ts_tablet_manager.cc:1434] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:30:59.070519 19505 tablet_bootstrap.cc:492] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:59.073196 19498 raft_consensus.cc:493] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.073743 19498 raft_consensus.cc:515] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.075868 19498 leader_election.cc:290] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:59.078474 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e989970204ce48b700c86d5faf91d" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:59.079188 18838 raft_consensus.cc:2468] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:59.082856 18656 leader_election.cc:304] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:59.085008 19499 raft_consensus.cc:493] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.085506 19499 raft_consensus.cc:515] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.087478 19499 leader_election.cc:290] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:59.087990 19499 raft_consensus.cc:2804] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:30:59.089471 18760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71d9e90994b4dcfa86f9bb9e3b45b51" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:59.088292 19499 raft_consensus.cc:493] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.091679 19499 raft_consensus.cc:3060] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:30:59.096608 18656 leader_election.cc:336] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.104875 19505 tablet_bootstrap.cc:492] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.106051 19505 tablet_bootstrap.cc:492] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.106810 19505 ts_tablet_manager.cc:1403] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.036s	user 0.015s	sys 0.020s
I20260709 12:30:59.109344 19499 raft_consensus.cc:515] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.109685 19505 raft_consensus.cc:359] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
W20260709 12:30:59.111573 18656 leader_election.cc:336] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.120430 19505 raft_consensus.cc:740] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.120569 18656 leader_election.cc:304] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:59.121557 19498 raft_consensus.cc:2749] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:59.122246 19350 raft_consensus.cc:493] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.121774 19505 consensus_queue.cc:260] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.122699 19350 raft_consensus.cc:515] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.124106 19505 ts_tablet_manager.cc:1434] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.000s
I20260709 12:30:59.124857 19350 leader_election.cc:290] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:59.127880 19350 raft_consensus.cc:493] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.129714 19350 raft_consensus.cc:515] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.132921 19498 raft_consensus.cc:493] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.133569 19498 raft_consensus.cc:515] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.137367 19350 leader_election.cc:290] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
W20260709 12:30:59.138049 18730 leader_election.cc:336] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.139124 19498 leader_election.cc:290] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:59.139292 18760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:59.141695 19505 tablet_bootstrap.cc:492] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:59.144601 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71d9e90994b4dcfa86f9bb9e3b45b51" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:59.145359 18688 raft_consensus.cc:2468] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:59.148509 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c180f8ddcbe14366a60537e1bad39332" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:59.149161 18688 raft_consensus.cc:2468] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
I20260709 12:30:59.155967 19499 leader_election.cc:290] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
W20260709 12:30:59.157838 18730 leader_election.cc:336] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.160091 18731 leader_election.cc:304] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:59.164130 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e989970204ce48b700c86d5faf91d" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:59.165010 18838 raft_consensus.cc:3060] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:30:59.167696 18656 leader_election.cc:336] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.168191 18656 leader_election.cc:304] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:30:59.169456 19499 raft_consensus.cc:2749] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:59.172000 18731 leader_election.cc:304] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:59.173022 19479 raft_consensus.cc:2804] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:30:59.173388 19479 raft_consensus.cc:493] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.165117 19350 raft_consensus.cc:2804] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:30:59.173729 19479 raft_consensus.cc:3060] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:59.173987 19350 raft_consensus.cc:493] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.174428 19350 raft_consensus.cc:3060] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:59.181048 19505 tablet_bootstrap.cc:492] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.182410 19505 tablet_bootstrap.cc:492] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.183447 19346 raft_consensus.cc:493] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.183954 19346 raft_consensus.cc:515] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.185196 18838 raft_consensus.cc:2468] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 2.
I20260709 12:30:59.186116 19346 leader_election.cc:290] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:59.188665 19505 ts_tablet_manager.cc:1403] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.064s	user 0.018s	sys 0.007s
I20260709 12:30:59.191236 19350 raft_consensus.cc:515] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.193567 19350 leader_election.cc:290] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:59.182287 19479 raft_consensus.cc:515] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.201103 19505 raft_consensus.cc:359] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.206131 19505 raft_consensus.cc:740] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.205474 19479 leader_election.cc:290] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:59.208220 19505 consensus_queue.cc:260] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.211760 19505 ts_tablet_manager.cc:1434] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.000s
I20260709 12:30:59.215806 19505 tablet_bootstrap.cc:492] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:59.216687 19496 raft_consensus.cc:493] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.217672 19496 raft_consensus.cc:515] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.220858 19496 leader_election.cc:290] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:59.222067 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "872ec24b41d24573a9a8cc1d484fd6f9" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:59.222841 18688 raft_consensus.cc:2468] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:59.225692 18805 leader_election.cc:304] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:59.226637 19475 raft_consensus.cc:2804] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:30:59.227025 19475 raft_consensus.cc:493] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.227511 19475 raft_consensus.cc:3060] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:59.233825 19512 raft_consensus.cc:493] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.240605 19512 raft_consensus.cc:515] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.243511 19512 leader_election.cc:290] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:59.248811 18656 leader_election.cc:304] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:59.248870 19498 raft_consensus.cc:493] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.249526 19498 raft_consensus.cc:515] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.250083 19474 raft_consensus.cc:2804] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:30:59.250609 19474 raft_consensus.cc:697] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 2 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:30:59.242329 19475 raft_consensus.cc:515] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.251579 19474 consensus_queue.cc:237] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.253607 18685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "872ec24b41d24573a9a8cc1d484fd6f9" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:59.254040 18767 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c6af67983744957af1aee8d948169c2" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:30:59.254678 18767 raft_consensus.cc:2468] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:59.254199 18685 raft_consensus.cc:3060] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:59.261703 18657 leader_election.cc:304] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:59.262964 19514 raft_consensus.cc:2804] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:30:59.263374 19514 raft_consensus.cc:493] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.263712 19514 raft_consensus.cc:3060] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:59.264891 19498 leader_election.cc:290] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:59.269237 18685 raft_consensus.cc:2468] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 2.
W20260709 12:30:59.270977 18804 leader_election.cc:336] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.274513 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c6af67983744957af1aee8d948169c2" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:59.279891 18805 leader_election.cc:304] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:59.281129 19210 raft_consensus.cc:2804] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:30:59.281584 19210 raft_consensus.cc:697] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 2 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:59.282368 19210 consensus_queue.cc:237] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
W20260709 12:30:59.285573 18804 leader_election.cc:336] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.287564 19514 raft_consensus.cc:515] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.288816 18688 raft_consensus.cc:2393] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 29f82a8594d741bc92dd24ee4a368e01 in current term 2: Already voted for candidate 60d61e0800194dd89be34d3aff3f374d in this term.
I20260709 12:30:59.289821 18515 catalog_manager.cc:5697] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 1 to 2, leader changed from 84812365c65f43f388836c087640adbb (127.4.65.1) to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:59.292622 19475 leader_election.cc:290] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:59.293903 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71d9e90994b4dcfa86f9bb9e3b45b51" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:59.294529 18691 raft_consensus.cc:3060] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:59.298159 19514 leader_election.cc:290] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:30:59.302340 18691 raft_consensus.cc:2468] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 2.
I20260709 12:30:59.298173 18760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c6af67983744957af1aee8d948169c2" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:30:59.303843 18760 raft_consensus.cc:3060] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:59.308014 19505 tablet_bootstrap.cc:492] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.309350 19505 tablet_bootstrap.cc:492] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.310117 19505 ts_tablet_manager.cc:1403] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.095s	user 0.019s	sys 0.003s
I20260709 12:30:59.311163 18760 raft_consensus.cc:2468] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 2.
I20260709 12:30:59.312793 18657 leader_election.cc:304] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:59.313542 19514 raft_consensus.cc:2804] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:30:59.314045 19514 raft_consensus.cc:697] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 2 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:30:59.313740 19505 raft_consensus.cc:359] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.314813 19505 raft_consensus.cc:740] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.314998 19514 consensus_queue.cc:237] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.315904 19505 consensus_queue.cc:260] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.321033 19505 ts_tablet_manager.cc:1434] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 12:30:59.318078 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c180f8ddcbe14366a60537e1bad39332" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:59.322177 19505 tablet_bootstrap.cc:492] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:59.322201 18688 raft_consensus.cc:3060] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:59.328181 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:59.328997 18689 raft_consensus.cc:2468] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 1.
W20260709 12:30:59.343892 18656 leader_election.cc:336] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.348620 18688 raft_consensus.cc:2468] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 2.
I20260709 12:30:59.354704  4356 ts_tablet_manager.cc:616] Registered 45 tablets
I20260709 12:30:59.355127  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.388s	user 0.285s	sys 0.007s
W20260709 12:30:59.357327 18730 leader_election.cc:336] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.358055 18730 leader_election.cc:304] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb
I20260709 12:30:59.359882 18731 leader_election.cc:304] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:59.366087 19505 tablet_bootstrap.cc:492] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.366662 19512 raft_consensus.cc:2749] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:59.367449 19505 tablet_bootstrap.cc:492] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.368010 19479 raft_consensus.cc:2804] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:30:59.368587 19479 raft_consensus.cc:697] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:59.369508 19479 consensus_queue.cc:237] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.369731 18513 catalog_manager.cc:5697] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 1 to 2, leader changed from 84812365c65f43f388836c087640adbb (127.4.65.1) to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 2 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:59.372745 18731 leader_election.cc:304] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:59.373603 19512 raft_consensus.cc:2804] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:30:59.373988 19512 raft_consensus.cc:493] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.374339 19512 raft_consensus.cc:3060] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:59.379020 19514 raft_consensus.cc:493] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.379874 19514 raft_consensus.cc:515] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.383576 18731 leader_election.cc:304] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:59.384471 19346 raft_consensus.cc:2804] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:30:59.384949 19346 raft_consensus.cc:697] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
W20260709 12:30:59.385594 18730 leader_election.cc:336] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.385833 19346 consensus_queue.cc:237] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.386153 19499 raft_consensus.cc:493] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.386885 19499 raft_consensus.cc:515] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
W20260709 12:30:59.394392 18730 leader_election.cc:336] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.396721 19499 leader_election.cc:290] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:59.395275 19514 leader_election.cc:290] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:59.416457 19505 ts_tablet_manager.cc:1403] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.095s	user 0.025s	sys 0.002s
I20260709 12:30:59.417608 18835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7b389579627488bb406ec0d112f6938" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:59.420825 18835 raft_consensus.cc:2468] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 1.
I20260709 12:30:59.423260 19505 raft_consensus.cc:359] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.424021 19505 raft_consensus.cc:740] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.426318 19496 raft_consensus.cc:493] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.427069 19496 raft_consensus.cc:515] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.431185 19496 leader_election.cc:290] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:59.427668 19512 raft_consensus.cc:515] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.432328 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6e49ecce4f040e99a419db81f1953af" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:30:59.433048 18688 raft_consensus.cc:2468] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 1.
I20260709 12:30:59.433904 19512 leader_election.cc:290] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:30:59.434422 18805 leader_election.cc:304] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
W20260709 12:30:59.435303 18804 leader_election.cc:336] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.435730 19496 raft_consensus.cc:2804] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:30:59.436134 19496 raft_consensus.cc:493] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Starting leader election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.436528 19496 raft_consensus.cc:3060] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:59.434981 19505 consensus_queue.cc:260] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.440297 19505 ts_tablet_manager.cc:1434] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.019s	user 0.001s	sys 0.004s
I20260709 12:30:59.441617 19505 tablet_bootstrap.cc:492] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:59.450031 18835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6e49ecce4f040e99a419db81f1953af" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:30:59.457353 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:59.458154 18689 raft_consensus.cc:3060] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:59.469992 18689 raft_consensus.cc:2468] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 2.
W20260709 12:30:59.471829 18656 leader_election.cc:336] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.477945 19505 tablet_bootstrap.cc:492] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.479060 19505 tablet_bootstrap.cc:492] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.479815 19505 ts_tablet_manager.cc:1403] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.038s	user 0.014s	sys 0.011s
I20260709 12:30:59.481035 19496 raft_consensus.cc:515] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.482259 18835 raft_consensus.cc:2393] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 60d61e0800194dd89be34d3aff3f374d in current term 2: Already voted for candidate df352ed4600b4ddb83ef57ecd2944fce in this term.
I20260709 12:30:59.483060 19505 raft_consensus.cc:359] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:59.483846 19505 raft_consensus.cc:740] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
W20260709 12:30:59.484658 18656 leader_election.cc:336] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.484783 19505 consensus_queue.cc:260] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:59.487460 19505 ts_tablet_manager.cc:1434] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.007s	user 0.009s	sys 0.000s
I20260709 12:30:59.488372 19505 tablet_bootstrap.cc:492] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:59.490396 18516 catalog_manager.cc:5697] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 1 to 2, leader changed from 84812365c65f43f388836c087640adbb (127.4.65.1) to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 2 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:59.493271 18515 catalog_manager.cc:5697] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 1 to 2, leader changed from 84812365c65f43f388836c087640adbb (127.4.65.1) to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:59.500247 18731 leader_election.cc:304] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:30:59.501189 19512 raft_consensus.cc:2804] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:30:59.502187 19512 raft_consensus.cc:697] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:30:59.503099 19512 consensus_queue.cc:237] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
W20260709 12:30:59.509851 18730 leader_election.cc:336] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.512658 18656 leader_election.cc:304] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:30:59.514303 19496 leader_election.cc:290] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 election: Requested vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
W20260709 12:30:59.514773 18804 leader_election.cc:336] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.515815 19499 raft_consensus.cc:2804] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:30:59.516227 19499 raft_consensus.cc:493] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:30:59.516615 19499 raft_consensus.cc:3060] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:59.517966 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6e49ecce4f040e99a419db81f1953af" candidate_uuid: "df352ed4600b4ddb83ef57ecd2944fce" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:30:59.518666 18689 raft_consensus.cc:3060] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:30:59.535881 18656 leader_election.cc:336] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.536496 18656 leader_election.cc:304] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d; no voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce
W20260709 12:30:59.545622 18656 leader_election.cc:336] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
W20260709 12:30:59.549793 18730 leader_election.cc:336] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.549566 18689 raft_consensus.cc:2468] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df352ed4600b4ddb83ef57ecd2944fce in term 2.
I20260709 12:30:59.556468 19498 raft_consensus.cc:2749] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:30:59.562085 18805 leader_election.cc:304] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:59.562958 19496 raft_consensus.cc:2804] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:30:59.564153 19496 raft_consensus.cc:697] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 2 LEADER]: Becoming Leader. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:30:59.564916 19505 tablet_bootstrap.cc:492] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.565062 19496 consensus_queue.cc:237] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.566123 19505 tablet_bootstrap.cc:492] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.566964 19505 ts_tablet_manager.cc:1403] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.079s	user 0.036s	sys 0.011s
I20260709 12:30:59.570202 18515 catalog_manager.cc:5697] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 1 to 2, leader changed from 84812365c65f43f388836c087640adbb (127.4.65.1) to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 2 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:59.571097 19505 raft_consensus.cc:359] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.575650 19505 raft_consensus.cc:740] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.576597 19505 consensus_queue.cc:260] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.578377 18515 catalog_manager.cc:5697] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 1 to 2, leader changed from 84812365c65f43f388836c087640adbb (127.4.65.1) to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 2 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:59.578909 19505 ts_tablet_manager.cc:1434] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:30:59.580333 19505 tablet_bootstrap.cc:492] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:59.581683 18516 catalog_manager.cc:5697] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce reported cstate change: term changed from 1 to 2, leader changed from 84812365c65f43f388836c087640adbb (127.4.65.1) to df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4). New cstate: current_term: 2 leader_uuid: "df352ed4600b4ddb83ef57ecd2944fce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260709 12:30:59.584537 19499 raft_consensus.cc:515] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.587675 19499 leader_election.cc:290] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:30:59.607769 19505 tablet_bootstrap.cc:492] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.608879 19505 tablet_bootstrap.cc:492] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.609586 19505 ts_tablet_manager.cc:1403] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.029s	user 0.011s	sys 0.010s
I20260709 12:30:59.612293 19505 raft_consensus.cc:359] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:59.613085 19505 raft_consensus.cc:740] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.613926 19505 consensus_queue.cc:260] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:59.616266 19505 ts_tablet_manager.cc:1434] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:30:59.617117 19505 tablet_bootstrap.cc:492] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:30:59.629164 18730 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 78379.9us
I20260709 12:30:59.648936 18835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7b389579627488bb406ec0d112f6938" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:30:59.649582 18835 raft_consensus.cc:3060] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:30:59.657153 18835 raft_consensus.cc:2468] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 2.
I20260709 12:30:59.674875 19505 tablet_bootstrap.cc:492] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.677124 19505 tablet_bootstrap.cc:492] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.677860 19505 ts_tablet_manager.cc:1403] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.061s	user 0.007s	sys 0.021s
I20260709 12:30:59.683172 19505 raft_consensus.cc:359] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.683982 19505 raft_consensus.cc:740] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.684916 19505 consensus_queue.cc:260] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.687609 19505 ts_tablet_manager.cc:1434] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:30:59.688493 19505 tablet_bootstrap.cc:492] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:59.697681 18688 raft_consensus.cc:1275] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Refusing update from remote peer df352ed4600b4ddb83ef57ecd2944fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:30:59.699599 19475 consensus_queue.cc:1048] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:30:59.700448 18804 consensus_peers.cc:597] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:59.710249 18767 raft_consensus.cc:1275] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Refusing update from remote peer 60d61e0800194dd89be34d3aff3f374d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:30:59.712684 19499 consensus_queue.cc:1048] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:30:59.749241 19505 tablet_bootstrap.cc:492] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.750295 19505 tablet_bootstrap.cc:492] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.751009 19505 ts_tablet_manager.cc:1403] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.063s	user 0.016s	sys 0.023s
I20260709 12:30:59.753782 19505 raft_consensus.cc:359] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.754524 19505 raft_consensus.cc:740] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.755420 19505 consensus_queue.cc:260] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:30:59.758224 19505 ts_tablet_manager.cc:1434] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:30:59.759327 19505 tablet_bootstrap.cc:492] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:30:59.786863 18656 leader_election.cc:336] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:30:59.801540 19505 tablet_bootstrap.cc:492] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.802673 19505 tablet_bootstrap.cc:492] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.803428 19505 ts_tablet_manager.cc:1403] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.044s	user 0.019s	sys 0.012s
I20260709 12:30:59.806490 18656 leader_election.cc:304] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce; no voters: 84812365c65f43f388836c087640adbb
I20260709 12:30:59.806385 19505 raft_consensus.cc:359] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.807403 19499 raft_consensus.cc:2804] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:30:59.807425 19505 raft_consensus.cc:740] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.808382 19499 raft_consensus.cc:697] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 2 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:30:59.809037 19505 consensus_queue.cc:260] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.809477 19499 consensus_queue.cc:237] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.814138 19505 ts_tablet_manager.cc:1434] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.000s
I20260709 12:30:59.815090 19505 tablet_bootstrap.cc:492] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:59.849810 19505 tablet_bootstrap.cc:492] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.850869 19505 tablet_bootstrap.cc:492] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.851876 19505 ts_tablet_manager.cc:1403] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.037s	user 0.015s	sys 0.014s
W20260709 12:30:59.855036 18656 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.4:44517, user_credentials={real_user=slave}} blocked reactor thread for 48628.5us
I20260709 12:30:59.857524 19505 raft_consensus.cc:359] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:59.858925 19505 raft_consensus.cc:740] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.859800 19505 consensus_queue.cc:260] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:59.862360 19505 ts_tablet_manager.cc:1434] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.003s
I20260709 12:30:59.863211 19505 tablet_bootstrap.cc:492] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:59.883957 18838 raft_consensus.cc:1275] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Refusing update from remote peer 60d61e0800194dd89be34d3aff3f374d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:30:59.901908 18688 raft_consensus.cc:1275] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Refusing update from remote peer 29f82a8594d741bc92dd24ee4a368e01: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:30:59.901870 19505 tablet_bootstrap.cc:492] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.903182 18689 raft_consensus.cc:1275] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Refusing update from remote peer 29f82a8594d741bc92dd24ee4a368e01: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:30:59.905790 19505 tablet_bootstrap.cc:492] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.906608 19505 ts_tablet_manager.cc:1403] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.044s	user 0.015s	sys 0.018s
W20260709 12:30:59.907867 18804 consensus_peers.cc:597] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:30:59.910735 19505 raft_consensus.cc:359] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.911545 19505 raft_consensus.cc:740] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.912422 19505 consensus_queue.cc:260] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.916390 19505 ts_tablet_manager.cc:1434] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:30:59.917270 19505 tablet_bootstrap.cc:492] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:30:59.917738 18730 consensus_peers.cc:597] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:30:59.925801 18516 catalog_manager.cc:5697] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 1 to 2, leader changed from 84812365c65f43f388836c087640adbb (127.4.65.1) to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } }
I20260709 12:30:59.940201 19505 tablet_bootstrap.cc:492] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.941397 19505 tablet_bootstrap.cc:492] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.942149 19505 ts_tablet_manager.cc:1403] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.025s	user 0.012s	sys 0.010s
I20260709 12:30:59.945077 19505 raft_consensus.cc:359] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:59.945840 19505 raft_consensus.cc:740] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.947440 19505 consensus_queue.cc:260] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:30:59.949329 19479 consensus_queue.cc:1048] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:30:59.949899 19505 ts_tablet_manager.cc:1434] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.003s
I20260709 12:30:59.950845 19505 tablet_bootstrap.cc:492] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:30:59.957619 19479 consensus_queue.cc:1048] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:30:59.973428 19514 consensus_queue.cc:1048] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:30:59.974922 19505 tablet_bootstrap.cc:492] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:30:59.976037 19505 tablet_bootstrap.cc:492] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:30:59.976779 19505 ts_tablet_manager.cc:1403] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.026s	user 0.019s	sys 0.005s
I20260709 12:30:59.980049 19505 raft_consensus.cc:359] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.981007 19505 raft_consensus.cc:740] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:30:59.981976 19505 consensus_queue.cc:260] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:30:59.984453 19505 ts_tablet_manager.cc:1434] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:30:59.985335 19505 tablet_bootstrap.cc:492] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:30:59.993047 18730 consensus_peers.cc:597] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:00.007743 19505 tablet_bootstrap.cc:492] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:00.008781 19505 tablet_bootstrap.cc:492] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb: Bootstrap complete.
W20260709 12:31:00.009155 18804 consensus_peers.cc:597] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.009423 19505 ts_tablet_manager.cc:1403] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.011s
I20260709 12:31:00.012455 19505 raft_consensus.cc:359] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:00.013187 19505 raft_consensus.cc:740] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:00.013964 19505 consensus_queue.cc:260] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:00.016549 19505 ts_tablet_manager.cc:1434] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:31:00.017354 19505 tablet_bootstrap.cc:492] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:00.017403 18691 raft_consensus.cc:1275] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Refusing update from remote peer 29f82a8594d741bc92dd24ee4a368e01: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:00.043071 19505 tablet_bootstrap.cc:492] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:00.044098 19505 tablet_bootstrap.cc:492] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:00.044896 19505 ts_tablet_manager.cc:1403] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.028s	user 0.018s	sys 0.004s
I20260709 12:31:00.047871 19505 raft_consensus.cc:359] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:00.049803 18688 raft_consensus.cc:1275] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Refusing update from remote peer df352ed4600b4ddb83ef57ecd2944fce: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:00.051743 19496 consensus_queue.cc:1048] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:00.048777 19505 raft_consensus.cc:740] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:00.055241 19505 consensus_queue.cc:260] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:00.058563 19505 ts_tablet_manager.cc:1434] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:31:00.059623 19505 tablet_bootstrap.cc:492] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:31:00.083091 18730 consensus_peers.cc:597] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:00.090849 18730 consensus_peers.cc:597] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.091297 19505 tablet_bootstrap.cc:492] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:00.092672 19505 tablet_bootstrap.cc:492] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:00.093442 19505 ts_tablet_manager.cc:1403] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.034s	user 0.021s	sys 0.008s
I20260709 12:31:00.096586 19505 raft_consensus.cc:359] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:00.097321 19505 raft_consensus.cc:740] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:00.098160 19505 consensus_queue.cc:260] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:00.100761 19505 ts_tablet_manager.cc:1434] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.001s
I20260709 12:31:00.101636 19505 tablet_bootstrap.cc:492] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:31:00.113634 18656 consensus_peers.cc:597] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:00.114276 18656 consensus_peers.cc:597] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:00.127688 19505 tablet_bootstrap.cc:492] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:00.128975 19505 tablet_bootstrap.cc:492] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:00.129712 19505 ts_tablet_manager.cc:1403] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.028s	user 0.017s	sys 0.006s
I20260709 12:31:00.132612 19505 raft_consensus.cc:359] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:00.133394 19505 raft_consensus.cc:740] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:00.134434 19505 consensus_queue.cc:260] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:00.137279 19505 ts_tablet_manager.cc:1434] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.001s
I20260709 12:31:00.138221 19505 tablet_bootstrap.cc:492] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:31:00.139518 18804 consensus_peers.cc:597] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:00.142781 19479 consensus_queue.cc:1048] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:00.162847 19505 tablet_bootstrap.cc:492] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:00.163920 19505 tablet_bootstrap.cc:492] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:00.164786 19505 ts_tablet_manager.cc:1403] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.027s	user 0.021s	sys 0.003s
W20260709 12:31:00.163736 18730 consensus_peers.cc:597] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.168179 19505 raft_consensus.cc:359] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:00.169071 19505 raft_consensus.cc:740] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:00.170109 19505 consensus_queue.cc:260] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:00.173066 19505 ts_tablet_manager.cc:1434] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:31:00.174226 19505 tablet_bootstrap.cc:492] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:31:00.174846 18804 consensus_peers.cc:597] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.184660 18804 consensus_peers.cc:597] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.217922 18730 consensus_peers.cc:597] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.220315 19505 tablet_bootstrap.cc:492] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:00.221518 19505 tablet_bootstrap.cc:492] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:00.222498 19505 ts_tablet_manager.cc:1403] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.049s	user 0.032s	sys 0.008s
I20260709 12:31:00.225472 19505 raft_consensus.cc:359] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:00.230480 19505 raft_consensus.cc:740] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:00.231365 19505 consensus_queue.cc:260] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:00.233954 19505 ts_tablet_manager.cc:1434] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
W20260709 12:31:00.234102 18804 consensus_peers.cc:597] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.234817 19505 tablet_bootstrap.cc:492] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:00.268644 19505 tablet_bootstrap.cc:492] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:00.270277 19505 tablet_bootstrap.cc:492] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:00.270983 19505 ts_tablet_manager.cc:1403] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.036s	user 0.020s	sys 0.011s
I20260709 12:31:00.275233 19505 raft_consensus.cc:359] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:00.275904 19505 raft_consensus.cc:740] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:00.276669 19505 consensus_queue.cc:260] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:00.278795 19505 ts_tablet_manager.cc:1434] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:31:00.279542 19505 tablet_bootstrap.cc:492] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:31:00.291512 18804 consensus_peers.cc:597] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.310997 19505 tablet_bootstrap.cc:492] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:00.312459 19505 tablet_bootstrap.cc:492] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:00.313143 19505 ts_tablet_manager.cc:1403] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.034s	user 0.016s	sys 0.006s
I20260709 12:31:00.315732 19505 raft_consensus.cc:359] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:00.316493 19505 raft_consensus.cc:740] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:00.318096 19505 consensus_queue.cc:260] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:00.321434 19505 ts_tablet_manager.cc:1434] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.002s
I20260709 12:31:00.322464 19505 tablet_bootstrap.cc:492] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:31:00.336949 18804 consensus_peers.cc:597] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.379700 18841 raft_consensus.cc:1275] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Refusing update from remote peer 60d61e0800194dd89be34d3aff3f374d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:31:00.390197 18804 consensus_peers.cc:597] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.394131 18730 consensus_peers.cc:597] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.394991 18730 consensus_peers.cc:597] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.395421 18730 consensus_peers.cc:597] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.447077 18730 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 51305.3us
W20260709 12:31:00.459826 18804 consensus_peers.cc:597] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.461367 19499 consensus_queue.cc:1048] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:31:00.480604 19505 tablet_bootstrap.cc:492] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:00.481843 19505 tablet_bootstrap.cc:492] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:00.482532 19505 ts_tablet_manager.cc:1403] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.160s	user 0.076s	sys 0.075s
W20260709 12:31:00.484948 18730 consensus_peers.cc:597] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.486016 19505 raft_consensus.cc:359] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:00.490644 19505 raft_consensus.cc:740] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:00.491678 19505 consensus_queue.cc:260] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:00.494777 19505 ts_tablet_manager.cc:1434] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 12:31:00.495823 19505 tablet_bootstrap.cc:492] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:31:00.502385 18804 consensus_peers.cc:597] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.531526 18730 consensus_peers.cc:597] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.532071 18730 consensus_peers.cc:597] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.534077 18804 consensus_peers.cc:597] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.546511 18804 consensus_peers.cc:597] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.550606 18731 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 68127.3us
W20260709 12:31:00.566752 18730 consensus_peers.cc:597] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.567323 18730 consensus_peers.cc:597] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.567821 18730 consensus_peers.cc:597] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.587929 18730 consensus_peers.cc:597] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.588863 18656 consensus_peers.cc:597] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:00.596858 18656 consensus_peers.cc:597] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.612937 18730 consensus_peers.cc:597] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.618232 19505 tablet_bootstrap.cc:492] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:00.627658 19508 raft_consensus.cc:493] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:00.629763 19508 raft_consensus.cc:515] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:00.630369 19505 tablet_bootstrap.cc:492] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:00.631157 19505 ts_tablet_manager.cc:1403] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.136s	user 0.025s	sys 0.070s
I20260709 12:31:00.634359 19505 raft_consensus.cc:359] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:00.635088 19505 raft_consensus.cc:740] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:00.636101 19505 consensus_queue.cc:260] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:00.641371 19508 leader_election.cc:290] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:31:00.648240 19609 raft_consensus.cc:493] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:31:00.647245 18730 consensus_peers.cc:597] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.650027 19609 raft_consensus.cc:515] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:00.657513 19609 leader_election.cc:290] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
W20260709 12:31:00.684672 18804 consensus_peers.cc:597] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.690517 19505 ts_tablet_manager.cc:1434] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.059s	user 0.008s	sys 0.004s
I20260709 12:31:00.691608 19505 tablet_bootstrap.cc:492] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:00.699311 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4eead98f0854adfaf72daa76fdf9c9b" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:31:00.717087 19612 raft_consensus.cc:493] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:00.717729 19612 raft_consensus.cc:515] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
W20260709 12:31:00.718245 18656 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:44517, user_credentials={real_user=slave}} blocked reactor thread for 44528.7us
I20260709 12:31:00.722005 19612 leader_election.cc:290] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:31:00.723460 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:31:00.728003 19505 tablet_bootstrap.cc:492] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:00.729312 19505 tablet_bootstrap.cc:492] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:00.731341 19505 ts_tablet_manager.cc:1403] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.040s	user 0.014s	sys 0.009s
I20260709 12:31:00.735140 19505 raft_consensus.cc:359] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:00.736908 19505 raft_consensus.cc:740] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:00.737712 19505 consensus_queue.cc:260] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
W20260709 12:31:00.738667 18656 consensus_peers.cc:597] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.739411 18656 consensus_peers.cc:597] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.740301 19505 ts_tablet_manager.cc:1434] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.001s
I20260709 12:31:00.741195 19505 tablet_bootstrap.cc:492] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:00.744385 19612 raft_consensus.cc:493] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:00.744962 19612 raft_consensus.cc:515] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:00.747186 19612 leader_election.cc:290] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
W20260709 12:31:00.747874 18730 consensus_peers.cc:597] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.750454 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:31:00.765892 19612 raft_consensus.cc:493] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:00.766345 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4eead98f0854adfaf72daa76fdf9c9b" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:31:00.766523 19612 raft_consensus.cc:515] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:00.768831 19612 leader_election.cc:290] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:31:00.768240 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71d9e90994b4dcfa86f9bb9e3b45b51" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:31:00.770448 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df295fba24764e2ab46a8985e569bad8" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:31:00.772285 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:31:00.775244 19500 leader_election.cc:304] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:31:00.776643 19612 raft_consensus.cc:2749] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:00.777661 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df295fba24764e2ab46a8985e569bad8" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:31:00.780042 19500 leader_election.cc:304] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:31:00.781723 19612 raft_consensus.cc:2749] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:00.796429 19500 leader_election.cc:304] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:31:00.798370 19612 raft_consensus.cc:2749] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:00.801064 19612 raft_consensus.cc:493] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:00.801841 19612 raft_consensus.cc:515] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:00.825415 19612 leader_election.cc:290] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:31:00.832055 18760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71d9e90994b4dcfa86f9bb9e3b45b51" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:31:00.827067 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0840843cb482476a8e18344ff8f1e315" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:31:00.838521 18760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f6d22c56d84a6cb05eb4469f831e6e" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:31:00.845317 19500 leader_election.cc:304] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:31:00.848551 19612 raft_consensus.cc:2749] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:00.846899 18760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0840843cb482476a8e18344ff8f1e315" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:31:00.855865 19500 leader_election.cc:304] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:31:00.857647 19612 raft_consensus.cc:2749] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:31:00.861706 18656 consensus_peers.cc:597] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.862524 18656 consensus_peers.cc:597] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:00.863143 18804 consensus_peers.cc:597] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.864871 19612 raft_consensus.cc:493] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:00.867146 19612 raft_consensus.cc:515] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:00.871366 19612 leader_election.cc:290] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:31:00.868556 19500 leader_election.cc:304] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:31:00.878511 19612 raft_consensus.cc:2749] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:00.890152 19505 tablet_bootstrap.cc:492] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:00.891084 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fb81949f42946cf893026acc49357bc" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:31:00.891614 19505 tablet_bootstrap.cc:492] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:00.892710 18760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fb81949f42946cf893026acc49357bc" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:31:00.894115 19505 ts_tablet_manager.cc:1403] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.153s	user 0.030s	sys 0.043s
I20260709 12:31:00.896991 19500 leader_election.cc:304] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:31:00.897892 19609 raft_consensus.cc:2749] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:00.899595 19505 raft_consensus.cc:359] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:00.900372 19505 raft_consensus.cc:740] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:00.901190 19505 consensus_queue.cc:260] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:00.904544 19505 ts_tablet_manager.cc:1434] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:31:00.906309 19505 tablet_bootstrap.cc:492] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb: Bootstrap starting.
W20260709 12:31:00.921182 18656 consensus_peers.cc:597] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:31:00.959671 19633 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.1:36569 every 8 connection(s)
I20260709 12:31:00.960062  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.1:36569
I20260709 12:31:00.964331 19505 tablet_bootstrap.cc:492] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:00.965221 19505 tablet_bootstrap.cc:492] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:00.965869 19505 ts_tablet_manager.cc:1403] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.060s	user 0.018s	sys 0.026s
I20260709 12:31:00.974030 19505 raft_consensus.cc:359] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:00.974879 19505 raft_consensus.cc:740] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:00.975592 19505 consensus_queue.cc:260] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:00.978855 19505 ts_tablet_manager.cc:1434] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:31:00.979807 19505 tablet_bootstrap.cc:492] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:01.014348 19612 raft_consensus.cc:493] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:01.014968 19612 raft_consensus.cc:515] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:01.024060 19634 heartbeater.cc:344] Connected to a master server at 127.4.65.62:41379
I20260709 12:31:01.031004 18835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73eeb703f09a4330ad726c1d5e42eee3" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:31:01.031004 18760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "73eeb703f09a4330ad726c1d5e42eee3" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:31:01.045264 19500 leader_election.cc:304] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:31:01.046269 19609 raft_consensus.cc:2749] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:01.031140 19612 leader_election.cc:290] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:31:01.034253 19505 tablet_bootstrap.cc:492] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:01.048259 19505 tablet_bootstrap.cc:492] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:01.049044 19505 ts_tablet_manager.cc:1403] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.069s	user 0.023s	sys 0.007s
I20260709 12:31:01.051599 19505 raft_consensus.cc:359] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:01.052137 19505 raft_consensus.cc:740] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:01.052842 19634 heartbeater.cc:461] Registering TS with master...
I20260709 12:31:01.052932 19505 consensus_queue.cc:260] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:01.053962 19634 heartbeater.cc:507] Master 127.4.65.62:41379 requested a full tablet report, sending...
I20260709 12:31:01.054782 19505 ts_tablet_manager.cc:1434] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:31:01.055930 19505 tablet_bootstrap.cc:492] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:01.068784 19612 raft_consensus.cc:493] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:01.069410 19612 raft_consensus.cc:515] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:01.071489 19612 leader_election.cc:290] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:31:01.076146 18760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:31:01.079121 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:31:01.081156 19500 leader_election.cc:304] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d
I20260709 12:31:01.081322 18516 ts_manager.cc:194] Re-registered known tserver with Master: 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:31:01.082465 19612 raft_consensus.cc:2749] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:01.106448 19505 tablet_bootstrap.cc:492] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:01.107719 19505 tablet_bootstrap.cc:492] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:01.108567 19505 ts_tablet_manager.cc:1403] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.053s	user 0.032s	sys 0.013s
I20260709 12:31:01.112254 19505 raft_consensus.cc:359] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:01.113804 19505 raft_consensus.cc:740] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:01.114534 19505 consensus_queue.cc:260] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:01.116986 19505 ts_tablet_manager.cc:1434] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:31:01.118983 19505 tablet_bootstrap.cc:492] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:01.146348 19583 raft_consensus.cc:3060] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:01.153632 19505 tablet_bootstrap.cc:492] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:01.155148 19505 tablet_bootstrap.cc:492] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:01.158658 19505 ts_tablet_manager.cc:1403] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.041s	user 0.027s	sys 0.007s
I20260709 12:31:01.160665 18516 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48954
I20260709 12:31:01.162137 19505 raft_consensus.cc:359] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:01.162978 19505 raft_consensus.cc:740] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:01.164201 19505 consensus_queue.cc:260] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:01.167846 19505 ts_tablet_manager.cc:1434] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:31:01.167929 19634 heartbeater.cc:499] Master 127.4.65.62:41379 was elected leader, sending a full tablet report...
I20260709 12:31:01.172071 19505 tablet_bootstrap.cc:492] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:01.188730 19587 raft_consensus.cc:3060] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:01.206342 19505 tablet_bootstrap.cc:492] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:01.207914 19505 tablet_bootstrap.cc:492] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:01.209089 19505 ts_tablet_manager.cc:1403] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.037s	user 0.014s	sys 0.021s
I20260709 12:31:01.214046 19505 raft_consensus.cc:359] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:31:01.214818 19505 raft_consensus.cc:740] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:01.215569 19505 consensus_queue.cc:260] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:31:01.217804 19505 ts_tablet_manager.cc:1434] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:31:01.218660 19505 tablet_bootstrap.cc:492] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:01.243192 19586 raft_consensus.cc:3060] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:01.257121 19609 raft_consensus.cc:493] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:01.257771 19609 raft_consensus.cc:515] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:01.260968 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4c43366d9fb416e8649aba9676aed59" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:31:01.263854 19585 raft_consensus.cc:3060] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:01.279026 18835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4c43366d9fb416e8649aba9676aed59" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:31:01.281114 19497 leader_election.cc:304] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:31:01.282092 19612 raft_consensus.cc:2749] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:01.286103 19612 raft_consensus.cc:493] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:01.286684 19612 raft_consensus.cc:515] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:01.303575 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16afc1d74df849c48d10f5da0a8f80c4" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:31:01.307013 19609 leader_election.cc:290] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:31:01.384531 19612 leader_election.cc:290] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:31:01.408694 19647 raft_consensus.cc:493] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:01.405004 19609 raft_consensus.cc:493] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:01.414211 19609 raft_consensus.cc:515] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:01.416317 19609 leader_election.cc:290] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:31:01.417245 18763 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f16818c4dfd4e9ab187ffb53d8fc3ce" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "29f82a8594d741bc92dd24ee4a368e01" is_pre_election: true
I20260709 12:31:01.436725 18835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16afc1d74df849c48d10f5da0a8f80c4" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:31:01.436906 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f16818c4dfd4e9ab187ffb53d8fc3ce" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:31:01.409457 19647 raft_consensus.cc:515] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:01.440701 19505 tablet_bootstrap.cc:492] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:01.442142 19505 tablet_bootstrap.cc:492] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:01.442519 19647 leader_election.cc:290] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:31:01.442833 19505 ts_tablet_manager.cc:1403] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.224s	user 0.055s	sys 0.071s
I20260709 12:31:01.442950 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4794853627e84898986305827ec4ea46" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:31:01.445972 19505 raft_consensus.cc:359] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:01.446722 19505 raft_consensus.cc:740] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:01.447422 19505 consensus_queue.cc:260] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:01.449385 19505 ts_tablet_manager.cc:1434] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:31:01.450171 19505 tablet_bootstrap.cc:492] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:01.451457 19497 leader_election.cc:304] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:31:01.452843 19648 raft_consensus.cc:2749] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:01.453094 19497 leader_election.cc:304] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:31:01.454506 19609 raft_consensus.cc:493] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:01.455202 19609 raft_consensus.cc:515] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:01.456604 19647 raft_consensus.cc:2749] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:01.459230 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e989970204ce48b700c86d5faf91d" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "60d61e0800194dd89be34d3aff3f374d" is_pre_election: true
I20260709 12:31:01.461374 19609 leader_election.cc:290] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:31:01.466050 18841 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4794853627e84898986305827ec4ea46" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:31:01.466298 18835 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "671e989970204ce48b700c86d5faf91d" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce" is_pre_election: true
I20260709 12:31:01.469734 19497 leader_election.cc:304] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:31:01.476972 19609 raft_consensus.cc:2749] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:01.478330 19497 leader_election.cc:304] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84812365c65f43f388836c087640adbb; no voters: 60d61e0800194dd89be34d3aff3f374d, df352ed4600b4ddb83ef57ecd2944fce
I20260709 12:31:01.479192 19648 raft_consensus.cc:2749] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:01.480772 19505 tablet_bootstrap.cc:492] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:01.482002 19505 tablet_bootstrap.cc:492] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:01.482682 19505 ts_tablet_manager.cc:1403] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.033s	user 0.012s	sys 0.005s
I20260709 12:31:01.486320 19505 raft_consensus.cc:359] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:01.487174 19505 raft_consensus.cc:740] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:01.487862 19505 consensus_queue.cc:260] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:01.489655 19505 ts_tablet_manager.cc:1434] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:31:01.490474 19505 tablet_bootstrap.cc:492] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:01.490761 19582 raft_consensus.cc:3060] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:01.521576 19505 tablet_bootstrap.cc:492] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:01.522678 19505 tablet_bootstrap.cc:492] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:01.523372 19505 ts_tablet_manager.cc:1403] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.033s	user 0.014s	sys 0.006s
I20260709 12:31:01.526185 19505 raft_consensus.cc:359] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:01.526861 19505 raft_consensus.cc:740] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:01.527541 19505 consensus_queue.cc:260] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:01.529551 19505 ts_tablet_manager.cc:1434] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:31:01.531312 19505 tablet_bootstrap.cc:492] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:01.571305 19505 tablet_bootstrap.cc:492] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:01.572463 19505 tablet_bootstrap.cc:492] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:01.573395 19505 ts_tablet_manager.cc:1403] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.043s	user 0.019s	sys 0.004s
I20260709 12:31:01.599562 19505 raft_consensus.cc:359] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:01.600445 19505 raft_consensus.cc:740] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:01.601251 19505 consensus_queue.cc:260] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:01.607383 19505 ts_tablet_manager.cc:1434] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:31:01.608498 19505 tablet_bootstrap.cc:492] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:01.667280 19505 tablet_bootstrap.cc:492] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:01.668326 19505 tablet_bootstrap.cc:492] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:01.669445 19505 ts_tablet_manager.cc:1403] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.061s	user 0.027s	sys 0.013s
I20260709 12:31:01.673138 19505 raft_consensus.cc:359] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:01.673856 19505 raft_consensus.cc:740] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:01.674667 19505 consensus_queue.cc:260] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:01.676862 19505 ts_tablet_manager.cc:1434] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.001s
I20260709 12:31:01.677655 19505 tablet_bootstrap.cc:492] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:01.702029 19505 tablet_bootstrap.cc:492] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:01.703163 19505 tablet_bootstrap.cc:492] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:01.703887 19505 ts_tablet_manager.cc:1403] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.026s	user 0.011s	sys 0.008s
I20260709 12:31:01.706704 19505 raft_consensus.cc:359] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:01.707513 19505 raft_consensus.cc:740] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:01.708266 19505 consensus_queue.cc:260] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:01.712072 19505 ts_tablet_manager.cc:1434] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:31:01.712947 19505 tablet_bootstrap.cc:492] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:01.744675 19505 tablet_bootstrap.cc:492] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:01.745669 19505 tablet_bootstrap.cc:492] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:01.746385 19505 ts_tablet_manager.cc:1403] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.034s	user 0.020s	sys 0.000s
I20260709 12:31:01.749079 19505 raft_consensus.cc:359] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:01.749888 19505 raft_consensus.cc:740] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:01.750764 19505 consensus_queue.cc:260] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:01.754025 19505 ts_tablet_manager.cc:1434] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:31:01.755002 19505 tablet_bootstrap.cc:492] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:01.768873 19582 raft_consensus.cc:3060] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:01.947642 19505 tablet_bootstrap.cc:492] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:01.949028 19505 tablet_bootstrap.cc:492] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:01.949836 19505 ts_tablet_manager.cc:1403] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.195s	user 0.056s	sys 0.039s
I20260709 12:31:01.974135 19505 raft_consensus.cc:359] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:01.975252 19505 raft_consensus.cc:740] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:01.975966 19505 consensus_queue.cc:260] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:01.979926 19505 ts_tablet_manager.cc:1434] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.004s
I20260709 12:31:01.980903 19505 tablet_bootstrap.cc:492] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:02.005111 19505 tablet_bootstrap.cc:492] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:02.006108 19505 tablet_bootstrap.cc:492] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:02.007812 19505 ts_tablet_manager.cc:1403] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.026s	user 0.014s	sys 0.007s
I20260709 12:31:02.010277 19505 raft_consensus.cc:359] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:02.011101 19505 raft_consensus.cc:740] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:02.014190 19505 consensus_queue.cc:260] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:02.017210 19505 ts_tablet_manager.cc:1434] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260709 12:31:02.018249 19505 tablet_bootstrap.cc:492] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:02.061450 19505 tablet_bootstrap.cc:492] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:02.062464 19505 tablet_bootstrap.cc:492] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:02.063181 19505 ts_tablet_manager.cc:1403] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.045s	user 0.028s	sys 0.009s
I20260709 12:31:02.065889 19505 raft_consensus.cc:359] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:02.066941 19505 raft_consensus.cc:740] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:02.067770 19505 consensus_queue.cc:260] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:02.070214 19505 ts_tablet_manager.cc:1434] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:31:02.071107 19505 tablet_bootstrap.cc:492] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:02.147230 19588 raft_consensus.cc:3060] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:02.240638 19505 tablet_bootstrap.cc:492] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:02.242281 19505 tablet_bootstrap.cc:492] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:02.243278 19505 ts_tablet_manager.cc:1403] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.172s	user 0.052s	sys 0.066s
I20260709 12:31:02.246647 19505 raft_consensus.cc:359] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:02.247603 19505 raft_consensus.cc:740] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:02.248515 19505 consensus_queue.cc:260] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:02.253317 19505 ts_tablet_manager.cc:1434] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260709 12:31:02.254681 19505 tablet_bootstrap.cc:492] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb: Bootstrap starting.
I20260709 12:31:02.322803 19505 tablet_bootstrap.cc:492] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:02.323635 19505 tablet_bootstrap.cc:492] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb: Bootstrap complete.
I20260709 12:31:02.324298 19505 ts_tablet_manager.cc:1403] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb: Time spent bootstrapping tablet: real 0.070s	user 0.025s	sys 0.034s
I20260709 12:31:02.327554 19505 raft_consensus.cc:359] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:02.328279 19505 raft_consensus.cc:740] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84812365c65f43f388836c087640adbb, State: Initialized, Role: FOLLOWER
I20260709 12:31:02.328975 19505 consensus_queue.cc:260] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:02.330960 19505 ts_tablet_manager.cc:1434] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 12:31:02.346618 19582 raft_consensus.cc:3060] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:31:03.296690 18805 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 54565.4us
W20260709 12:31:03.371799 18731 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 60553.5us
W20260709 12:31:03.452920 18731 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 80223us
W20260709 12:31:05.127931 18788 debug-util.cc:398] Leaking SignalData structure 0x7b0800164ce0 after lost signal to thread 4359
W20260709 12:31:05.129050 18788 debug-util.cc:398] Leaking SignalData structure 0x7b080026eb00 after lost signal to thread 18550
W20260709 12:31:05.130187 18788 debug-util.cc:398] Leaking SignalData structure 0x7b08001303a0 after lost signal to thread 18717
W20260709 12:31:05.131275 18788 debug-util.cc:398] Leaking SignalData structure 0x7b08001027e0 after lost signal to thread 18791
W20260709 12:31:05.132510 18788 debug-util.cc:398] Leaking SignalData structure 0x7b08003a1f60 after lost signal to thread 18865
W20260709 12:31:05.133805 18788 debug-util.cc:398] Leaking SignalData structure 0x7b0800054fe0 after lost signal to thread 19633
I20260709 12:31:10.999508  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:11.068627 18766 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "23ca4b75eb0e49dca52e1b9fb4f8b356"
dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
mode: GRACEFUL
new_leader_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:34360
I20260709 12:31:11.069478 18766 raft_consensus.cc:606] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Received request to transfer leadership to TS 84812365c65f43f388836c087640adbb
I20260709 12:31:11.084452 18839 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9219e0ff53f348ff98bdc3422e5fb74c"
dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
mode: GRACEFUL
new_leader_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:39040
I20260709 12:31:11.085287 18839 raft_consensus.cc:606] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Received request to transfer leadership to TS 84812365c65f43f388836c087640adbb
I20260709 12:31:11.089658 18766 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a6fc5ff8d0014a9f9c1dadcc157e1d9a"
dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
mode: GRACEFUL
new_leader_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:34360
I20260709 12:31:11.090454 18766 raft_consensus.cc:606] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Received request to transfer leadership to TS 84812365c65f43f388836c087640adbb
I20260709 12:31:11.095538 18766 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c11b1d6d5a5c4991bd34743fff5ca507"
dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
mode: GRACEFUL
new_leader_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:34360
I20260709 12:31:11.096138 18766 raft_consensus.cc:606] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Received request to transfer leadership to TS 84812365c65f43f388836c087640adbb
I20260709 12:31:11.099457 18766 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c180f8ddcbe14366a60537e1bad39332"
dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
mode: GRACEFUL
new_leader_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:34360
I20260709 12:31:11.100159 18766 raft_consensus.cc:606] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Received request to transfer leadership to TS 84812365c65f43f388836c087640adbb
I20260709 12:31:11.105746 18766 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d4f9d88127114fda91668ab529abd23c"
dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
mode: GRACEFUL
new_leader_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:34360
I20260709 12:31:11.106495 18766 raft_consensus.cc:606] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Received request to transfer leadership to TS 84812365c65f43f388836c087640adbb
I20260709 12:31:11.108749  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:31:11.110648  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:11.113481  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:11.118120 19711 raft_consensus.cc:993] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01: : Instructing follower 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:11.119303 19692 raft_consensus.cc:1081] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Signalling peer 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:11.122094 19583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a6fc5ff8d0014a9f9c1dadcc157e1d9a"
dest_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:51180
I20260709 12:31:11.122709 19583 raft_consensus.cc:493] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:11.123093 19583 raft_consensus.cc:3060] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.145926 19583 raft_consensus.cc:515] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.148898 19583 leader_election.cc:290] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:31:11.150300 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6fc5ff8d0014a9f9c1dadcc157e1d9a" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:31:11.151242 18766 raft_consensus.cc:3055] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:11.151553 18766 raft_consensus.cc:740] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:31:11.152375 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a6fc5ff8d0014a9f9c1dadcc157e1d9a" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:31:11.153169 18689 raft_consensus.cc:3060] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.152621 18766 consensus_queue.cc:260] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.155818 18766 raft_consensus.cc:3060] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.162006 18689 raft_consensus.cc:2468] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:11.164618 18766 raft_consensus.cc:2468] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:11.164837 19500 leader_election.cc:304] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:31:11.167209 19716 raft_consensus.cc:2804] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:11.169027 19716 raft_consensus.cc:697] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:31:11.170198 19716 consensus_queue.cc:237] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.193733 18515 catalog_manager.cc:5697] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 1 to 2, leader changed from 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3) to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 2 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:11.285321 19711 raft_consensus.cc:993] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01: : Instructing follower 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:11.286055 19708 raft_consensus.cc:1081] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Signalling peer 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:11.288146 19583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d4f9d88127114fda91668ab529abd23c"
dest_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:51180
I20260709 12:31:11.288914 19583 raft_consensus.cc:493] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:11.289247 19583 raft_consensus.cc:3060] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.295838 19583 raft_consensus.cc:515] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.298517 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f9d88127114fda91668ab529abd23c" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:31:11.298667 19583 leader_election.cc:290] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:31:11.299135 18688 raft_consensus.cc:3060] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.301144 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4f9d88127114fda91668ab529abd23c" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:31:11.301780 18766 raft_consensus.cc:3055] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:11.302073 18766 raft_consensus.cc:740] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:31:11.302875 18766 consensus_queue.cc:260] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.304155 18766 raft_consensus.cc:3060] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.306684 18688 raft_consensus.cc:2468] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:11.307916 19500 leader_election.cc:304] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:31:11.308789 19716 raft_consensus.cc:2804] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:11.309299 19716 raft_consensus.cc:697] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:31:11.310168 19716 consensus_queue.cc:237] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.311749 18766 raft_consensus.cc:2468] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:11.321868 18515 catalog_manager.cc:5697] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 1 to 2, leader changed from 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3) to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 2 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:11.335951 19711 raft_consensus.cc:993] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01: : Instructing follower 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:11.336736 19708 raft_consensus.cc:1081] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Signalling peer 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:11.338691 19586 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c11b1d6d5a5c4991bd34743fff5ca507"
dest_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:51180
I20260709 12:31:11.339200 19586 raft_consensus.cc:493] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:11.339506 19586 raft_consensus.cc:3060] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.346112 19586 raft_consensus.cc:515] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.348235 19586 leader_election.cc:290] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:31:11.349416 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11b1d6d5a5c4991bd34743fff5ca507" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:31:11.350208 18766 raft_consensus.cc:3055] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:11.350528 18766 raft_consensus.cc:740] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:31:11.350927 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c11b1d6d5a5c4991bd34743fff5ca507" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:31:11.351629 18688 raft_consensus.cc:3060] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.351513 18766 consensus_queue.cc:260] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.353034 18766 raft_consensus.cc:3060] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.359692 18688 raft_consensus.cc:2468] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:11.360327 18766 raft_consensus.cc:2468] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:11.361198 19500 leader_election.cc:304] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:31:11.362483 19716 raft_consensus.cc:2804] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:11.363011 19716 raft_consensus.cc:697] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:31:11.363999 19716 consensus_queue.cc:237] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.375025 19708 raft_consensus.cc:993] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01: : Instructing follower 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:11.375489 19708 raft_consensus.cc:1081] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Signalling peer 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:11.377477 19582 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "23ca4b75eb0e49dca52e1b9fb4f8b356"
dest_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:51180
I20260709 12:31:11.378086 19582 raft_consensus.cc:493] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:11.378515 19582 raft_consensus.cc:3060] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.382783 19711 raft_consensus.cc:993] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01: : Instructing follower 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:11.384434 19711 raft_consensus.cc:1081] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Signalling peer 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:11.390930 19582 raft_consensus.cc:515] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.394913 18515 catalog_manager.cc:5697] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 1 to 2, leader changed from 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3) to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 2 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:11.397058 19583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c180f8ddcbe14366a60537e1bad39332"
dest_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:51180
I20260709 12:31:11.397675 19583 raft_consensus.cc:493] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:11.398049 19583 raft_consensus.cc:3060] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:11.402602 18688 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ca4b75eb0e49dca52e1b9fb4f8b356" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:31:11.403302 18688 raft_consensus.cc:3060] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.405965 19583 raft_consensus.cc:515] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.409171 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c180f8ddcbe14366a60537e1bad39332" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:31:11.409718 18689 raft_consensus.cc:3060] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:11.409611 19582 leader_election.cc:290] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:31:11.411331 19583 leader_election.cc:290] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 3 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:31:11.414542 18688 raft_consensus.cc:2468] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:11.415568 19500 leader_election.cc:304] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:31:11.415584 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23ca4b75eb0e49dca52e1b9fb4f8b356" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:31:11.416383 18766 raft_consensus.cc:3055] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:11.416527 19716 raft_consensus.cc:2804] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:11.416713 18766 raft_consensus.cc:740] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:31:11.417120 19716 raft_consensus.cc:697] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:31:11.417346 18689 raft_consensus.cc:2468] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 3.
I20260709 12:31:11.417706 18766 consensus_queue.cc:260] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.418190 19716 consensus_queue.cc:237] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.418737 19500 leader_election.cc:304] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:31:11.415825 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c180f8ddcbe14366a60537e1bad39332" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:31:11.419056 18766 raft_consensus.cc:3060] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.419530 18761 raft_consensus.cc:3055] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:31:11.419852 18761 raft_consensus.cc:740] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:31:11.420325 19717 raft_consensus.cc:2804] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:31:11.420888 19717 raft_consensus.cc:697] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 3 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:31:11.421023 18761 consensus_queue.cc:260] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.421933 19717 consensus_queue.cc:237] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:11.422494 18761 raft_consensus.cc:3060] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:11.426790 18766 raft_consensus.cc:2468] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:11.432503 18761 raft_consensus.cc:2468] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 3.
I20260709 12:31:11.442408 18515 catalog_manager.cc:5697] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 1 to 2, leader changed from 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3) to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 2 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:11.458007 18515 catalog_manager.cc:5697] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 2 to 3, leader changed from 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3) to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 3 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:11.527469 19709 raft_consensus.cc:993] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce: : Instructing follower 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:11.528126 19699 raft_consensus.cc:1081] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Signalling peer 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:11.529844 19582 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9219e0ff53f348ff98bdc3422e5fb74c"
dest_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:51200
I20260709 12:31:11.530485 19582 raft_consensus.cc:493] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:11.530865 19582 raft_consensus.cc:3060] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.536228 19582 raft_consensus.cc:515] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:11.537808 19582 leader_election.cc:290] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:31:11.539149 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9219e0ff53f348ff98bdc3422e5fb74c" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:31:11.539149 18839 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9219e0ff53f348ff98bdc3422e5fb74c" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:31:11.540230 18689 raft_consensus.cc:3060] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.540419 18839 raft_consensus.cc:3055] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:11.540968 18839 raft_consensus.cc:740] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:31:11.541896 18839 consensus_queue.cc:260] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:11.543962 18839 raft_consensus.cc:3060] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:11.548995 18689 raft_consensus.cc:2468] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:11.550469 19500 leader_election.cc:304] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:31:11.551388 18839 raft_consensus.cc:2468] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:11.551407 19716 raft_consensus.cc:2804] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:11.552210 19716 raft_consensus.cc:697] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:31:11.553153 19716 consensus_queue.cc:237] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:11.566047 18515 catalog_manager.cc:5697] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 1 to 2, leader changed from df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4) to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 2 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:11.708110 18691 raft_consensus.cc:1275] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:11.710007 19716 consensus_queue.cc:1048] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:11.729256 18766 raft_consensus.cc:1275] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:11.730607 19725 consensus_queue.cc:1048] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:11.761603 18766 raft_consensus.cc:1275] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:11.763172 19717 consensus_queue.cc:1048] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:11.775782 18689 raft_consensus.cc:1275] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:11.778228 19725 consensus_queue.cc:1048] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:31:11.833079 18766 raft_consensus.cc:1275] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:11.834311 19725 consensus_queue.cc:1048] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:11.847296 18689 raft_consensus.cc:1275] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:11.849403 19716 consensus_queue.cc:1048] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:11.858402 18691 raft_consensus.cc:1275] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:11.860479 19716 consensus_queue.cc:1048] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:11.881003 18766 raft_consensus.cc:1275] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:11.883031 19716 consensus_queue.cc:1048] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:11.977281 18761 raft_consensus.cc:1275] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:31:11.978830 19717 consensus_queue.cc:1048] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:31:11.989845 18689 raft_consensus.cc:1275] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 3 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:31:11.991983 19716 consensus_queue.cc:1048] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:31:12.067447 18691 raft_consensus.cc:1275] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:12.069110 19716 consensus_queue.cc:1048] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:12.080621 18838 raft_consensus.cc:1275] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:12.082476 19716 consensus_queue.cc:1048] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:12.196777  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:12.244273 18838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "03deb6b823ad408c8cdecc1fbd819940"
dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
mode: GRACEFUL
new_leader_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:39040
I20260709 12:31:12.244992 18838 raft_consensus.cc:606] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Received request to transfer leadership to TS 84812365c65f43f388836c087640adbb
I20260709 12:31:12.247830 18838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "14206c16a2154538ad2a781e824a65dd"
dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
mode: GRACEFUL
new_leader_uuid: "60d61e0800194dd89be34d3aff3f374d"
 from {username='slave'} at 127.0.0.1:39040
I20260709 12:31:12.248409 18838 raft_consensus.cc:606] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Received request to transfer leadership to TS 60d61e0800194dd89be34d3aff3f374d
I20260709 12:31:12.251941 18761 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1ab29498797f49d198e49d29f9800da2"
dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
mode: GRACEFUL
new_leader_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:34360
I20260709 12:31:12.252744 18761 raft_consensus.cc:606] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Received request to transfer leadership to TS 84812365c65f43f388836c087640adbb
I20260709 12:31:12.255576 18838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf"
dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
mode: GRACEFUL
new_leader_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:39040
I20260709 12:31:12.256122 18838 raft_consensus.cc:606] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 2 LEADER]: Received request to transfer leadership to TS 84812365c65f43f388836c087640adbb
I20260709 12:31:12.258735 18761 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9"
dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
mode: GRACEFUL
new_leader_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:34360
I20260709 12:31:12.259240 18761 raft_consensus.cc:606] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Received request to transfer leadership to TS 84812365c65f43f388836c087640adbb
I20260709 12:31:12.260496  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:31:12.262236  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:12.262981  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:12.336514 19709 raft_consensus.cc:993] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce: : Instructing follower 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:12.337057 19744 raft_consensus.cc:1081] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 2 LEADER]: Signalling peer 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:12.338843 19582 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf"
dest_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:51200
I20260709 12:31:12.339506 19582 raft_consensus.cc:493] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:12.339880 19582 raft_consensus.cc:3060] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:12.346232 19582 raft_consensus.cc:515] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:12.349324 19582 leader_election.cc:290] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 3 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:31:12.349653 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:31:12.350392 18761 raft_consensus.cc:3060] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:12.350579 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d64e328a205406bb8de3be26e2a70bf" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:31:12.351166 18838 raft_consensus.cc:3055] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:31:12.351424 18838 raft_consensus.cc:740] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:31:12.352232 18838 consensus_queue.cc:260] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:12.353506 18838 raft_consensus.cc:3060] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:12.358752 18761 raft_consensus.cc:2468] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 3.
I20260709 12:31:12.360267 19500 leader_election.cc:304] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:31:12.361240 19716 raft_consensus.cc:2804] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:31:12.361764 19716 raft_consensus.cc:697] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 3 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:31:12.362815 18838 raft_consensus.cc:2468] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 3.
I20260709 12:31:12.362703 19716 consensus_queue.cc:237] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:12.373844 18515 catalog_manager.cc:5697] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 2 to 3, leader changed from df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4) to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 3 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:12.512251 19709 raft_consensus.cc:993] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce: : Instructing follower 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:12.512835 19744 raft_consensus.cc:1081] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Signalling peer 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:12.514223 19583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "03deb6b823ad408c8cdecc1fbd819940"
dest_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:51200
I20260709 12:31:12.514799 19583 raft_consensus.cc:493] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:12.515069 19583 raft_consensus.cc:3060] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:12.519731 19583 raft_consensus.cc:515] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:12.522571 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03deb6b823ad408c8cdecc1fbd819940" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:31:12.522651 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "03deb6b823ad408c8cdecc1fbd819940" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:31:12.523177 18838 raft_consensus.cc:3055] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:12.523531 18838 raft_consensus.cc:740] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:31:12.524621 19583 leader_election.cc:290] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:31:12.524307 18838 consensus_queue.cc:260] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:12.525992 18838 raft_consensus.cc:3060] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:12.531301 18838 raft_consensus.cc:2468] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:12.532739 19497 leader_election.cc:304] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:31:12.533623 19716 raft_consensus.cc:2804] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:12.534066 19716 raft_consensus.cc:697] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:31:12.534973 19716 consensus_queue.cc:237] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:12.544529 18515 catalog_manager.cc:5697] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 1 to 2, leader changed from df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4) to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 2 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:12.650215 19744 raft_consensus.cc:993] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce: : Instructing follower 60d61e0800194dd89be34d3aff3f374d to start an election
I20260709 12:31:12.650708 19744 raft_consensus.cc:1081] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Signalling peer 60d61e0800194dd89be34d3aff3f374d to start an election
I20260709 12:31:12.652500 18691 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "14206c16a2154538ad2a781e824a65dd"
dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
 from {username='slave'} at 127.0.0.1:47548
I20260709 12:31:12.653007 18691 raft_consensus.cc:493] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:12.653251 18691 raft_consensus.cc:3060] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:12.660396 18691 raft_consensus.cc:515] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:31:12.662505 18691 leader_election.cc:290] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:31:12.663208 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14206c16a2154538ad2a781e824a65dd" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:31:12.663208 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "14206c16a2154538ad2a781e824a65dd" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:31:12.664288 18761 raft_consensus.cc:3060] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:12.664309 18838 raft_consensus.cc:3055] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:12.665031 18838 raft_consensus.cc:740] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:31:12.666049 18838 consensus_queue.cc:260] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:31:12.667776 18838 raft_consensus.cc:3060] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:12.673015 18761 raft_consensus.cc:2468] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 2.
I20260709 12:31:12.674546 18657 leader_election.cc:304] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:31:12.675472 18838 raft_consensus.cc:2468] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 2.
I20260709 12:31:12.675693 19705 raft_consensus.cc:2804] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:12.678297 19705 raft_consensus.cc:697] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [term 2 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:31:12.679920 19705 consensus_queue.cc:237] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:31:12.695621 18515 catalog_manager.cc:5697] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 1 to 2, leader changed from df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4) to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:12.715613 19711 raft_consensus.cc:993] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01: : Instructing follower 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:12.716225 19708 raft_consensus.cc:1081] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Signalling peer 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:12.733443 19583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1ab29498797f49d198e49d29f9800da2"
dest_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:51180
I20260709 12:31:12.734170 19583 raft_consensus.cc:493] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:12.737269 19583 raft_consensus.cc:3060] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:12.740165 19711 raft_consensus.cc:993] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01: : Instructing follower 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:12.740870 19711 raft_consensus.cc:1081] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Signalling peer 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:12.744892 19582 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9"
dest_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:51180
I20260709 12:31:12.745781 19582 raft_consensus.cc:493] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:12.746228 19582 raft_consensus.cc:3060] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:12.748034 19583 raft_consensus.cc:515] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:12.750427 19583 leader_election.cc:290] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:31:12.751106 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ab29498797f49d198e49d29f9800da2" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:31:12.751876 18761 raft_consensus.cc:3055] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:12.751963 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ab29498797f49d198e49d29f9800da2" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:31:12.752322 18761 raft_consensus.cc:740] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:31:12.752691 18838 raft_consensus.cc:3060] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:12.753360 18761 consensus_queue.cc:260] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:12.754426 18761 raft_consensus.cc:3060] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:12.757504 19582 raft_consensus.cc:515] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:12.759860 18761 raft_consensus.cc:2468] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:12.760267 18838 raft_consensus.cc:2468] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:12.763168 19497 leader_election.cc:304] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:31:12.764405 19716 raft_consensus.cc:2804] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:12.764989 19716 raft_consensus.cc:697] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:31:12.765946 19716 consensus_queue.cc:237] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:12.767580 19582 leader_election.cc:290] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 3 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:31:12.770555 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:31:12.771358 18761 raft_consensus.cc:3055] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:31:12.772203 18761 raft_consensus.cc:740] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:31:12.771348 18691 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7f9d9bb9b8742ac930302b42228aad9" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:31:12.773432 18761 consensus_queue.cc:260] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:12.774303 18691 raft_consensus.cc:3060] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:12.774986 18761 raft_consensus.cc:3060] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:12.788285 18761 raft_consensus.cc:2468] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 3.
I20260709 12:31:12.790949 19500 leader_election.cc:304] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:31:12.792619 19725 raft_consensus.cc:2804] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:31:12.793128 19725 raft_consensus.cc:697] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 3 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:31:12.794013 19725 consensus_queue.cc:237] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:12.795090 18691 raft_consensus.cc:2468] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 3.
I20260709 12:31:12.803197 18515 catalog_manager.cc:5697] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 1 to 2, leader changed from 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3) to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 2 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:12.825354 18515 catalog_manager.cc:5697] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 2 to 3, leader changed from 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3) to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 3 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:12.936275 18838 raft_consensus.cc:1275] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 3 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:31:12.938304 19716 consensus_queue.cc:1048] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:12.951645 18761 raft_consensus.cc:1275] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:31:12.953338 19716 consensus_queue.cc:1048] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:31:13.023679 18761 raft_consensus.cc:3060] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:13.031458 18761 raft_consensus.cc:1275] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:13.033264 19717 consensus_queue.cc:1048] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:31:13.046916 18838 raft_consensus.cc:1275] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:13.048833 19725 consensus_queue.cc:1048] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:13.171478 18761 raft_consensus.cc:1275] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Refusing update from remote peer 60d61e0800194dd89be34d3aff3f374d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:13.172968 19705 consensus_queue.cc:1048] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:13.188773 18838 raft_consensus.cc:1275] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Refusing update from remote peer 60d61e0800194dd89be34d3aff3f374d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:13.190369 19755 consensus_queue.cc:1048] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:13.198704 18761 raft_consensus.cc:1275] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:13.200695 19725 consensus_queue.cc:1048] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:13.219996 18839 raft_consensus.cc:1275] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:13.222502 19717 consensus_queue.cc:1048] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:13.256986 18761 raft_consensus.cc:1275] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:31:13.258689 19725 consensus_queue.cc:1048] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:31:13.273214 18689 raft_consensus.cc:1275] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 3 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:31:13.280753 19716 consensus_queue.cc:1048] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:31:13.329214  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:31:13.470162 18730 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.1:36569, user_credentials={real_user=slave}} blocked reactor thread for 53903.4us
W20260709 12:31:13.544538 19500 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:40335, user_credentials={real_user=slave}} blocked reactor thread for 62750us
I20260709 12:31:13.678679 18838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dfef0fa153945568323437021ce2921"
dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
mode: GRACEFUL
new_leader_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:39040
I20260709 12:31:13.679549 18838 raft_consensus.cc:606] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Received request to transfer leadership to TS 84812365c65f43f388836c087640adbb
I20260709 12:31:13.683816 18838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "98dfe85df2e240449ec5f18cb3280c9b"
dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
mode: GRACEFUL
new_leader_uuid: "29f82a8594d741bc92dd24ee4a368e01"
 from {username='slave'} at 127.0.0.1:39040
I20260709 12:31:13.684520 18838 raft_consensus.cc:606] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Received request to transfer leadership to TS 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:31:13.687515 18838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ba28ff472b0f46e8884f70afb21857ae"
dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
mode: GRACEFUL
new_leader_uuid: "60d61e0800194dd89be34d3aff3f374d"
 from {username='slave'} at 127.0.0.1:39040
I20260709 12:31:13.688062 18838 raft_consensus.cc:606] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Received request to transfer leadership to TS 60d61e0800194dd89be34d3aff3f374d
I20260709 12:31:13.691233 18838 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b"
dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
mode: GRACEFUL
new_leader_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:39040
I20260709 12:31:13.693030 18838 raft_consensus.cc:606] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Received request to transfer leadership to TS 84812365c65f43f388836c087640adbb
I20260709 12:31:13.695487  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:31:13.697425  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:13.698424  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:13.829030 19776 raft_consensus.cc:993] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce: : Instructing follower 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:13.829743 19709 raft_consensus.cc:1081] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Signalling peer 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:13.831600 19583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2dfef0fa153945568323437021ce2921"
dest_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:51200
I20260709 12:31:13.832288 19583 raft_consensus.cc:493] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:13.832778 19583 raft_consensus.cc:3060] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:13.841686 19583 raft_consensus.cc:515] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:13.845046 19583 leader_election.cc:290] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:31:13.845527 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dfef0fa153945568323437021ce2921" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:31:13.845656 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dfef0fa153945568323437021ce2921" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:31:13.846544 18766 raft_consensus.cc:3060] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:13.846828 18838 raft_consensus.cc:3055] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:13.847220 18838 raft_consensus.cc:740] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:31:13.848593 18838 consensus_queue.cc:260] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:13.850069 18838 raft_consensus.cc:3060] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:13.853140 18766 raft_consensus.cc:2468] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:13.854616 19500 leader_election.cc:304] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:31:13.855717 19717 raft_consensus.cc:2804] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:13.856336 19717 raft_consensus.cc:697] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:31:13.857378 18838 raft_consensus.cc:2468] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:13.857334 19717 consensus_queue.cc:237] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:13.870422 18513 catalog_manager.cc:5697] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 1 to 2, leader changed from df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4) to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 2 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:13.898201 19709 raft_consensus.cc:993] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce: : Instructing follower 29f82a8594d741bc92dd24ee4a368e01 to start an election
I20260709 12:31:13.898909 19777 raft_consensus.cc:1081] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Signalling peer 29f82a8594d741bc92dd24ee4a368e01 to start an election
I20260709 12:31:13.901723 18766 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "98dfe85df2e240449ec5f18cb3280c9b"
dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
 from {username='slave'} at 127.0.0.1:56490
I20260709 12:31:13.902498 18766 raft_consensus.cc:493] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:13.902872 18766 raft_consensus.cc:3060] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:13.909224 18766 raft_consensus.cc:515] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:31:13.912567 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98dfe85df2e240449ec5f18cb3280c9b" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:31:13.913491 18689 raft_consensus.cc:3060] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:13.913163 18766 leader_election.cc:290] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:31:13.912701 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98dfe85df2e240449ec5f18cb3280c9b" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:31:13.918708 18838 raft_consensus.cc:3055] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:13.919143 18838 raft_consensus.cc:740] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:31:13.920125 18838 consensus_queue.cc:260] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:31:13.921778 18838 raft_consensus.cc:3060] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:13.928375 18689 raft_consensus.cc:2468] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 2.
I20260709 12:31:13.930220 18731 leader_election.cc:304] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:31:13.931437 19779 raft_consensus.cc:2804] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:13.931964 19779 raft_consensus.cc:697] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:31:13.932477 19709 raft_consensus.cc:993] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce: : Instructing follower 60d61e0800194dd89be34d3aff3f374d to start an election
I20260709 12:31:13.933410 19775 raft_consensus.cc:1081] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Signalling peer 60d61e0800194dd89be34d3aff3f374d to start an election
I20260709 12:31:13.933022 19779 consensus_queue.cc:237] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:31:13.937520 18689 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ba28ff472b0f46e8884f70afb21857ae"
dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
 from {username='slave'} at 127.0.0.1:47548
I20260709 12:31:13.938127 18689 raft_consensus.cc:493] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:13.938472 18689 raft_consensus.cc:3060] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:13.944988 18838 raft_consensus.cc:2468] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 2.
I20260709 12:31:13.948668 18689 raft_consensus.cc:515] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:31:13.951947 18689 leader_election.cc:290] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:31:13.953279 18766 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba28ff472b0f46e8884f70afb21857ae" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:31:13.954299 18766 raft_consensus.cc:3060] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:13.954914 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ba28ff472b0f46e8884f70afb21857ae" candidate_uuid: "60d61e0800194dd89be34d3aff3f374d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:31:13.955703 18840 raft_consensus.cc:3055] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:13.956090 18840 raft_consensus.cc:740] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:31:13.957446 18840 consensus_queue.cc:260] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:31:13.959369 18840 raft_consensus.cc:3060] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:13.965698 18766 raft_consensus.cc:2468] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 2.
I20260709 12:31:13.967733 18657 leader_election.cc:304] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, 60d61e0800194dd89be34d3aff3f374d; no voters: 
I20260709 12:31:13.969048 19755 raft_consensus.cc:2804] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:13.969707 19755 raft_consensus.cc:697] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 2 LEADER]: Becoming Leader. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:31:13.969835 18840 raft_consensus.cc:2468] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60d61e0800194dd89be34d3aff3f374d in term 2.
I20260709 12:31:13.970854 19755 consensus_queue.cc:237] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:31:13.976261 18513 catalog_manager.cc:5697] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 1 to 2, leader changed from df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4) to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 2 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:13.988059 18515 catalog_manager.cc:5697] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d reported cstate change: term changed from 1 to 2, leader changed from df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4) to 60d61e0800194dd89be34d3aff3f374d (127.4.65.2). New cstate: current_term: 2 leader_uuid: "60d61e0800194dd89be34d3aff3f374d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:14.078722 19709 raft_consensus.cc:993] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce: : Instructing follower 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:14.079272 19709 raft_consensus.cc:1081] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Signalling peer 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:14.081533 19582 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b"
dest_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:51200
I20260709 12:31:14.082196 19582 raft_consensus.cc:493] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:14.082525 19582 raft_consensus.cc:3060] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:14.089053 19582 raft_consensus.cc:515] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:14.090725 19582 leader_election.cc:290] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Requested vote from peers df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
I20260709 12:31:14.091406 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:31:14.091794 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06e3fb1322840a6b22c5da53900345b" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:31:14.092067 18840 raft_consensus.cc:3055] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:14.092375 18689 raft_consensus.cc:3060] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:14.092468 18840 raft_consensus.cc:740] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df352ed4600b4ddb83ef57ecd2944fce, State: Running, Role: LEADER
I20260709 12:31:14.093405 18840 consensus_queue.cc:260] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:14.094703 18840 raft_consensus.cc:3060] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:14.102475 18840 raft_consensus.cc:2468] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:14.100215 18689 raft_consensus.cc:2468] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:14.104264 19500 leader_election.cc:304] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:31:14.105873 19716 raft_consensus.cc:2804] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:14.106313 19716 raft_consensus.cc:697] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:31:14.107076 19716 consensus_queue.cc:237] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:14.117128 18515 catalog_manager.cc:5697] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 1 to 2, leader changed from df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4) to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 2 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:14.376049 18838 raft_consensus.cc:1275] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:14.377466 19716 consensus_queue.cc:1048] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:14.389175 18766 raft_consensus.cc:1275] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:14.390429 19716 consensus_queue.cc:1048] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:14.431227 18838 raft_consensus.cc:1275] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Refusing update from remote peer 60d61e0800194dd89be34d3aff3f374d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:14.432520 19755 consensus_queue.cc:1048] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:14.450865 18761 raft_consensus.cc:1275] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Refusing update from remote peer 60d61e0800194dd89be34d3aff3f374d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:14.453236 19705 consensus_queue.cc:1048] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:31:14.455691 18691 raft_consensus.cc:1275] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Refusing update from remote peer 29f82a8594d741bc92dd24ee4a368e01: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:14.457373 19779 consensus_queue.cc:1048] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:14.486300 18838 raft_consensus.cc:1275] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Refusing update from remote peer 29f82a8594d741bc92dd24ee4a368e01: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:14.488080 19711 consensus_queue.cc:1048] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:31:14.533795 18689 raft_consensus.cc:1275] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:14.538215 19716 consensus_queue.cc:1048] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:14.559386 18838 raft_consensus.cc:1275] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:14.563148 19782 consensus_queue.cc:1048] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:31:14.739836  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:14.792515 18766 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3fb81949f42946cf893026acc49357bc"
dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
mode: GRACEFUL
new_leader_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:34360
I20260709 12:31:14.793149 18766 raft_consensus.cc:606] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Received request to transfer leadership to TS 84812365c65f43f388836c087640adbb
I20260709 12:31:14.810824 18689 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "79ade62c22cf46b3bf97770f5bc7d597"
dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
mode: GRACEFUL
new_leader_uuid: "29f82a8594d741bc92dd24ee4a368e01"
 from {username='slave'} at 127.0.0.1:34466
I20260709 12:31:14.811364 18689 raft_consensus.cc:606] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Received request to transfer leadership to TS 29f82a8594d741bc92dd24ee4a368e01
I20260709 12:31:14.812727  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:31:14.813937  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:14.814628  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:15.142460 19805 raft_consensus.cc:993] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01: : Instructing follower 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:15.143564 19711 raft_consensus.cc:1081] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Signalling peer 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:15.146267 19583 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3fb81949f42946cf893026acc49357bc"
dest_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:51180
I20260709 12:31:15.147146 19583 raft_consensus.cc:493] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:15.147774 19583 raft_consensus.cc:3060] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:15.154220 19583 raft_consensus.cc:515] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:15.156189 19583 leader_election.cc:290] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335)
I20260709 12:31:15.157140 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fb81949f42946cf893026acc49357bc" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:31:15.157222 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3fb81949f42946cf893026acc49357bc" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:31:15.157891 18689 raft_consensus.cc:3060] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:15.157969 18761 raft_consensus.cc:3055] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:15.158413 18761 raft_consensus.cc:740] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:31:15.159051 18761 consensus_queue.cc:260] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:15.160045 18761 raft_consensus.cc:3060] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:15.163431 18689 raft_consensus.cc:2468] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:15.164844 19500 leader_election.cc:304] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb; no voters: 
I20260709 12:31:15.166776 19716 raft_consensus.cc:2804] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:15.167302 19716 raft_consensus.cc:697] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:31:15.167447 18761 raft_consensus.cc:2468] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:15.168279 19716 consensus_queue.cc:237] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:15.182626 18515 catalog_manager.cc:5697] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 1 to 2, leader changed from 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3) to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 2 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:15.344293 19705 raft_consensus.cc:993] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d: : Instructing follower 29f82a8594d741bc92dd24ee4a368e01 to start an election
I20260709 12:31:15.345141 19767 raft_consensus.cc:1081] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Signalling peer 29f82a8594d741bc92dd24ee4a368e01 to start an election
I20260709 12:31:15.346814 18761 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "79ade62c22cf46b3bf97770f5bc7d597"
dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
 from {username='slave'} at 127.0.0.1:56498
I20260709 12:31:15.347283 18761 raft_consensus.cc:493] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:15.347515 18761 raft_consensus.cc:3060] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:15.352947 18761 raft_consensus.cc:515] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:31:15.355954 18689 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ade62c22cf46b3bf97770f5bc7d597" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "60d61e0800194dd89be34d3aff3f374d"
I20260709 12:31:15.356411 18761 leader_election.cc:290] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: Requested vote from peers 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:31:15.356652 18689 raft_consensus.cc:3055] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:15.356834 18840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79ade62c22cf46b3bf97770f5bc7d597" candidate_uuid: "29f82a8594d741bc92dd24ee4a368e01" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:31:15.357131 18689 raft_consensus.cc:740] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 60d61e0800194dd89be34d3aff3f374d, State: Running, Role: LEADER
I20260709 12:31:15.357409 18840 raft_consensus.cc:3060] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:15.358069 18689 consensus_queue.cc:260] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:31:15.359505 18689 raft_consensus.cc:3060] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:15.365027 18840 raft_consensus.cc:2468] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 2.
I20260709 12:31:15.366441 18689 raft_consensus.cc:2468] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29f82a8594d741bc92dd24ee4a368e01 in term 2.
I20260709 12:31:15.366459 18730 leader_election.cc:304] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:31:15.367776 19779 raft_consensus.cc:2804] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:15.368305 19779 raft_consensus.cc:697] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Becoming Leader. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:31:15.369243 19779 consensus_queue.cc:237] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } }
I20260709 12:31:15.381747 18515 catalog_manager.cc:5697] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 reported cstate change: term changed from 1 to 2, leader changed from 60d61e0800194dd89be34d3aff3f374d (127.4.65.2) to 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3). New cstate: current_term: 2 leader_uuid: "29f82a8594d741bc92dd24ee4a368e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:15.618297 18689 raft_consensus.cc:1275] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:15.619635 19717 consensus_queue.cc:1048] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:15.631194 18761 raft_consensus.cc:1275] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:15.634589 19806 consensus_queue.cc:1048] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:15.853684  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:15.894506 18761 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9c4804407f8b43bd868a716c04e955e0"
dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
mode: GRACEFUL
new_leader_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:34360
I20260709 12:31:15.895227 18761 raft_consensus.cc:606] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Received request to transfer leadership to TS 84812365c65f43f388836c087640adbb
I20260709 12:31:15.896940  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260709 12:31:15.898703  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:15.899478  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:15.944005 18838 raft_consensus.cc:1275] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Refusing update from remote peer 29f82a8594d741bc92dd24ee4a368e01: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:15.944969 18691 raft_consensus.cc:1275] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Refusing update from remote peer 29f82a8594d741bc92dd24ee4a368e01: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:15.947883 19779 consensus_queue.cc:1048] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:31:15.950320 19805 consensus_queue.cc:1048] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:31:16.610198 19500 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 122216us
W20260709 12:31:16.682708 18730 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:44517, user_credentials={real_user=slave}} blocked reactor thread for 60906.6us
W20260709 12:31:16.700472 18731 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 77117.9us
W20260709 12:31:16.733688 18730 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:44517, user_credentials={real_user=slave}} blocked reactor thread for 44180.6us
I20260709 12:31:16.736481 19828 raft_consensus.cc:993] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01: : Instructing follower 84812365c65f43f388836c087640adbb to start an election
I20260709 12:31:16.737154 19828 raft_consensus.cc:1081] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Signalling peer 84812365c65f43f388836c087640adbb to start an election
W20260709 12:31:16.737775 19497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:44517, user_credentials={real_user=slave}} blocked reactor thread for 45510.7us
I20260709 12:31:16.744745 19581 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9c4804407f8b43bd868a716c04e955e0"
dest_uuid: "84812365c65f43f388836c087640adbb"
 from {username='slave'} at 127.0.0.1:51180
I20260709 12:31:16.745579 19581 raft_consensus.cc:493] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:16.747526 19581 raft_consensus.cc:3060] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:31:16.757376 18731 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 55361.1us
I20260709 12:31:16.772151 19581 raft_consensus.cc:515] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:16.775694 19581 leader_election.cc:290] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Requested vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), df352ed4600b4ddb83ef57ecd2944fce (127.4.65.4:44517)
I20260709 12:31:16.777151 18838 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c4804407f8b43bd868a716c04e955e0" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df352ed4600b4ddb83ef57ecd2944fce"
I20260709 12:31:16.786198 18838 raft_consensus.cc:3060] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:31:16.797988 18656 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.4:44517, user_credentials={real_user=slave}} blocked reactor thread for 74285us
W20260709 12:31:16.802367 19500 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:44143, user_credentials={real_user=slave}} blocked reactor thread for 106671us
I20260709 12:31:16.803993 18838 raft_consensus.cc:2468] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:16.805800 19497 leader_election.cc:304] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84812365c65f43f388836c087640adbb, df352ed4600b4ddb83ef57ecd2944fce; no voters: 
I20260709 12:31:16.806617 19806 raft_consensus.cc:2804] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:16.807080 19806 raft_consensus.cc:697] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Becoming Leader. State: Replica: 84812365c65f43f388836c087640adbb, State: Running, Role: LEADER
I20260709 12:31:16.807977 19806 consensus_queue.cc:237] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:16.826783 18761 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c4804407f8b43bd868a716c04e955e0" candidate_uuid: "84812365c65f43f388836c087640adbb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "29f82a8594d741bc92dd24ee4a368e01"
I20260709 12:31:16.827724 18761 raft_consensus.cc:3055] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:16.828151 18761 raft_consensus.cc:740] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 29f82a8594d741bc92dd24ee4a368e01, State: Running, Role: LEADER
I20260709 12:31:16.830457 18761 consensus_queue.cc:260] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:16.832338 18761 raft_consensus.cc:3060] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:16.845610 18761 raft_consensus.cc:2468] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84812365c65f43f388836c087640adbb in term 2.
I20260709 12:31:16.866356 18516 catalog_manager.cc:5697] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb reported cstate change: term changed from 1 to 2, leader changed from 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3) to 84812365c65f43f388836c087640adbb (127.4.65.1). New cstate: current_term: 2 leader_uuid: "84812365c65f43f388836c087640adbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:17.325301 18841 raft_consensus.cc:1275] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:17.327014 19806 consensus_queue.cc:1048] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:31:17.367707 18763 raft_consensus.cc:1275] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Refusing update from remote peer 84812365c65f43f388836c087640adbb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:17.374655 19851 consensus_queue.cc:1048] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:31:17.503850 18862 debug-util.cc:398] Leaking SignalData structure 0x7b080039c380 after lost signal to thread 4359
W20260709 12:31:17.505122 18862 debug-util.cc:398] Leaking SignalData structure 0x7b080031ec60 after lost signal to thread 18550
W20260709 12:31:17.506513 18862 debug-util.cc:398] Leaking SignalData structure 0x7b08002bed40 after lost signal to thread 18717
W20260709 12:31:17.507715 18862 debug-util.cc:398] Leaking SignalData structure 0x7b08001a69e0 after lost signal to thread 18791
W20260709 12:31:17.508863 18862 debug-util.cc:398] Leaking SignalData structure 0x7b08002352a0 after lost signal to thread 18865
W20260709 12:31:17.510180 18862 debug-util.cc:398] Leaking SignalData structure 0x7b080020e1c0 after lost signal to thread 19633
I20260709 12:31:17.513640  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:17.557711  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:31:17.559219  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:17.559836  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:17.603116  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:17.661659  4356 tablet_server.cc:179] TabletServer@127.4.65.1:36569 shutting down...
W20260709 12:31:17.668939 18656 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 285 similar messages]
W20260709 12:31:17.677035 18656 consensus_peers.cc:597] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:17.692060 18656 consensus_peers.cc:597] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:17.711434  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:31:17.712522  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.713675  4356 raft_consensus.cc:2243] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.714301  4356 raft_consensus.cc:2272] T f7b389579627488bb406ec0d112f6938 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.718029  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.719146  4356 raft_consensus.cc:2243] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.719753  4356 raft_consensus.cc:2272] T 7eb0c96994864b03a8ec7cda3cc8d85b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:17.721525 18656 consensus_peers.cc:597] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:17.723398  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.724212  4356 raft_consensus.cc:2243] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.724716  4356 raft_consensus.cc:2272] T d8bf5c53527542a6a9069536baff450f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.727382  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.728039  4356 raft_consensus.cc:2243] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.728523  4356 raft_consensus.cc:2272] T d659dc42667d4202a29b529fe3388c8e P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.731176  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.731860  4356 raft_consensus.cc:2243] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:17.732963  4356 raft_consensus.cc:2272] T 1ab29498797f49d198e49d29f9800da2 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.735944  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.736688  4356 raft_consensus.cc:2243] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.737182  4356 raft_consensus.cc:2272] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.739728  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.740432  4356 raft_consensus.cc:2243] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.740888  4356 raft_consensus.cc:2272] T d216c51d889b417bb2e59f20b6fd6d96 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.743566  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.744515  4356 raft_consensus.cc:2243] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:17.745712  4356 raft_consensus.cc:2272] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.749102  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.750088  4356 raft_consensus.cc:2243] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:31:17.751240  4356 raft_consensus.cc:2272] T c180f8ddcbe14366a60537e1bad39332 P 84812365c65f43f388836c087640adbb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.755512  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.757365  4356 raft_consensus.cc:2243] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.757995  4356 raft_consensus.cc:2272] T efa9444e110e433aa97b25a05d5769a9 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.762552  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.763525  4356 raft_consensus.cc:2243] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.764252  4356 raft_consensus.cc:2272] T fc988d3d39094696b3b4516b33a9c37d P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:17.765895 18804 consensus_peers.cc:597] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:17.768390  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.769711  4356 raft_consensus.cc:2243] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.770527  4356 raft_consensus.cc:2272] T f651b7eda3134c7a897855ac4300de2f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.775594  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.776639  4356 raft_consensus.cc:2243] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 12:31:17.777141 18730 consensus_peers.cc:597] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:17.777320  4356 raft_consensus.cc:2272] T d6e49ecce4f040e99a419db81f1953af P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.780436  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.781140  4356 raft_consensus.cc:2243] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:31:17.782212  4356 raft_consensus.cc:2272] T 4d64e328a205406bb8de3be26e2a70bf P 84812365c65f43f388836c087640adbb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.785286  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.786078  4356 raft_consensus.cc:2243] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.786513  4356 raft_consensus.cc:2272] T 4794853627e84898986305827ec4ea46 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.789172  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.789740  4356 raft_consensus.cc:2243] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.790123  4356 raft_consensus.cc:2272] T e4c43366d9fb416e8649aba9676aed59 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.792586  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.793177  4356 raft_consensus.cc:2243] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:17.794063  4356 raft_consensus.cc:2272] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.796718  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.797694  4356 raft_consensus.cc:2243] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:17.798808  4356 raft_consensus.cc:2272] T d4f9d88127114fda91668ab529abd23c P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.801645  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.802893  4356 raft_consensus.cc:2243] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:17.804275  4356 raft_consensus.cc:2272] T c11b1d6d5a5c4991bd34743fff5ca507 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.807380  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.808046  4356 raft_consensus.cc:2243] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.808526  4356 raft_consensus.cc:2272] T 0840843cb482476a8e18344ff8f1e315 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.810979  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.811563  4356 raft_consensus.cc:2243] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.811968  4356 raft_consensus.cc:2272] T 8c4a560e00884f1cb6f8720dfee7d27c P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.814268  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.814827  4356 raft_consensus.cc:2243] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:17.815653  4356 raft_consensus.cc:2272] T 9c4804407f8b43bd868a716c04e955e0 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.818821  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.819468  4356 raft_consensus.cc:2243] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:17.820431  4356 raft_consensus.cc:2272] T 2dfef0fa153945568323437021ce2921 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.822788  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.823402  4356 raft_consensus.cc:2243] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.823828  4356 raft_consensus.cc:2272] T 6256d41aab434828a66720ab61fa9ba5 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.826581  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.827277  4356 raft_consensus.cc:2243] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:31:17.828125  4356 raft_consensus.cc:2272] T c7f9d9bb9b8742ac930302b42228aad9 P 84812365c65f43f388836c087640adbb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.831207  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.831812  4356 raft_consensus.cc:2243] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:17.832686  4356 raft_consensus.cc:2272] T 03deb6b823ad408c8cdecc1fbd819940 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.835170  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.835747  4356 raft_consensus.cc:2243] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.836179  4356 raft_consensus.cc:2272] T 1a66b75f31e14b98902006a8f9d53a95 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.838506  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.839085  4356 raft_consensus.cc:2243] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.839498  4356 raft_consensus.cc:2272] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.842154  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.842808  4356 raft_consensus.cc:2243] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.843227  4356 raft_consensus.cc:2272] T 73eeb703f09a4330ad726c1d5e42eee3 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.845906  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.846477  4356 raft_consensus.cc:2243] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.846890  4356 raft_consensus.cc:2272] T 16afc1d74df849c48d10f5da0a8f80c4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.849216  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.850028  4356 raft_consensus.cc:2243] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.850677  4356 raft_consensus.cc:2272] T 872ec24b41d24573a9a8cc1d484fd6f9 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.854386  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.855355  4356 raft_consensus.cc:2243] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.855906  4356 raft_consensus.cc:2272] T 3fddb762772a40acb39a71286c8d4f41 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.858918  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.859599  4356 raft_consensus.cc:2243] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:31:17.859808 18730 consensus_peers.cc:597] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:17.860113  4356 raft_consensus.cc:2272] T d9e24af510de430aabe2a8c0c1010629 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.862710  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.863343  4356 raft_consensus.cc:2243] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.863845  4356 raft_consensus.cc:2272] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.866503  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.867343  4356 raft_consensus.cc:2243] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.867908  4356 raft_consensus.cc:2272] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.870949  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.871765  4356 raft_consensus.cc:2243] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.872332  4356 raft_consensus.cc:2272] T 3c6af67983744957af1aee8d948169c2 P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:17.873795 18804 consensus_peers.cc:597] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:17.875663  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.876814  4356 raft_consensus.cc:2243] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.877408  4356 raft_consensus.cc:2272] T 089c667ac1ae4a22b9b2e484f978c5d4 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.881285  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.882244  4356 raft_consensus.cc:2243] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.882920  4356 raft_consensus.cc:2272] T 671e989970204ce48b700c86d5faf91d P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:17.885252 18804 consensus_peers.cc:597] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:17.886300  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.887718  4356 raft_consensus.cc:2243] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:17.888922  4356 raft_consensus.cc:2272] T 9219e0ff53f348ff98bdc3422e5fb74c P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.892227  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.893121  4356 raft_consensus.cc:2243] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:17.894224  4356 raft_consensus.cc:2272] T 3fb81949f42946cf893026acc49357bc P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:17.895879 18804 consensus_peers.cc:597] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:17.897331  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.898293  4356 raft_consensus.cc:2243] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:31:17.898674 18656 consensus_peers.cc:597] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:17.898840  4356 raft_consensus.cc:2272] T 9e3d64db91924be490d4ed1f797d338f P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.902439  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.903332  4356 raft_consensus.cc:2243] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.903926  4356 raft_consensus.cc:2272] T df295fba24764e2ab46a8985e569bad8 P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:17.904645 18804 consensus_peers.cc:597] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:17.908070  4356 tablet_replica.cc:333] stopping tablet replica
W20260709 12:31:17.908865 18730 consensus_peers.cc:597] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:17.908921  4356 raft_consensus.cc:2243] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.909822  4356 raft_consensus.cc:2272] T f4eead98f0854adfaf72daa76fdf9c9b P 84812365c65f43f388836c087640adbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.912963  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.913678  4356 raft_consensus.cc:2243] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:17.914230  4356 raft_consensus.cc:2272] T 79f6d22c56d84a6cb05eb4469f831e6e P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:17.917233  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:17.917912  4356 raft_consensus.cc:2243] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:17.918813  4356 raft_consensus.cc:2272] T e06e3fb1322840a6b22c5da53900345b P 84812365c65f43f388836c087640adbb [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:17.925369 18804 consensus_peers.cc:597] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:17.933524 18730 consensus_peers.cc:597] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:17.945169 18730 consensus_peers.cc:597] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:17.976817 18656 consensus_peers.cc:597] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:17.986740 18804 consensus_peers.cc:597] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.013406 18656 consensus_peers.cc:597] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.014158 18656 consensus_peers.cc:597] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.023236  4356 tablet_server.cc:196] TabletServer@127.4.65.1:36569 shutdown complete.
W20260709 12:31:18.043519 18730 consensus_peers.cc:597] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.046097 18730 consensus_peers.cc:597] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.052986 18804 consensus_peers.cc:597] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.061446 18804 consensus_peers.cc:597] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.064965 18804 consensus_peers.cc:597] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.083068 18656 consensus_peers.cc:597] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.088513 18656 consensus_peers.cc:597] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.088925 18804 consensus_peers.cc:597] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.114241 18730 consensus_peers.cc:597] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.157743  4356 tablet_server.cc:179] TabletServer@127.4.65.2:0 shutting down...
W20260709 12:31:18.168619 18730 consensus_peers.cc:597] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.185194 18805 consensus_peers.cc:597] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.201479 18730 consensus_peers.cc:597] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.213282 18656 consensus_peers.cc:597] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d -> Peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Couldn't send request to peer 84812365c65f43f388836c087640adbb. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.218236 18805 consensus_peers.cc:597] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.224030 18805 consensus_peers.cc:597] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.228403  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:31:18.229875  4356 tablet_replica.cc:333] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.231210  4356 raft_consensus.cc:2243] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 12:31:18.231475 18805 consensus_peers.cc:597] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.231876  4356 raft_consensus.cc:2272] T e06e3fb1322840a6b22c5da53900345b P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.235306  4356 tablet_replica.cc:333] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.236342  4356 raft_consensus.cc:2243] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.236996  4356 raft_consensus.cc:2272] T 6a46101454ef46beb00f1652fabfa153 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.240028  4356 tablet_replica.cc:333] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.241196  4356 raft_consensus.cc:2243] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.241878  4356 raft_consensus.cc:2272] T d216c51d889b417bb2e59f20b6fd6d96 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.245640  4356 tablet_replica.cc:333] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.246654  4356 raft_consensus.cc:2243] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.247876  4356 raft_consensus.cc:2272] T 6256d41aab434828a66720ab61fa9ba5 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.251209  4356 tablet_replica.cc:333] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.252141  4356 raft_consensus.cc:2243] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.252838  4356 raft_consensus.cc:2272] T 0840843cb482476a8e18344ff8f1e315 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.256201  4356 tablet_replica.cc:333] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.257190  4356 raft_consensus.cc:2243] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.258374  4356 raft_consensus.cc:2272] T 8c4a560e00884f1cb6f8720dfee7d27c P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.261389  4356 tablet_replica.cc:333] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.262313  4356 raft_consensus.cc:2243] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.263293  4356 raft_consensus.cc:2272] T c11b1d6d5a5c4991bd34743fff5ca507 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.266254  4356 tablet_replica.cc:333] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.267016  4356 raft_consensus.cc:2243] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.267894  4356 raft_consensus.cc:2272] T ba28ff472b0f46e8884f70afb21857ae P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.270526  4356 tablet_replica.cc:333] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.271392  4356 raft_consensus.cc:2243] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.271976  4356 raft_consensus.cc:2272] T 4794853627e84898986305827ec4ea46 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.275063  4356 tablet_replica.cc:333] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.275939  4356 raft_consensus.cc:2243] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.276526  4356 raft_consensus.cc:2272] T 3eed637e5890430db2e802ef6d4362a7 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.279496  4356 tablet_replica.cc:333] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.280381  4356 raft_consensus.cc:2243] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.280962  4356 raft_consensus.cc:2272] T f4eead98f0854adfaf72daa76fdf9c9b P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.286082  4356 tablet_replica.cc:333] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.286947  4356 raft_consensus.cc:2243] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.287514  4356 raft_consensus.cc:2272] T c2abc15bf7b445a4b4269e1d93cce949 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.291057  4356 tablet_replica.cc:333] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.291850  4356 raft_consensus.cc:2243] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.293033  4356 raft_consensus.cc:2272] T 9e3d64db91924be490d4ed1f797d338f P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.296090  4356 tablet_replica.cc:333] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.297001  4356 raft_consensus.cc:2243] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.297610  4356 raft_consensus.cc:2272] T df295fba24764e2ab46a8985e569bad8 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:18.297792 18731 consensus_peers.cc:597] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.300753  4356 tablet_replica.cc:333] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.302663  4356 raft_consensus.cc:2243] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.304733  4356 raft_consensus.cc:2272] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.307653  4356 tablet_replica.cc:333] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.308526  4356 raft_consensus.cc:2243] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.309309  4356 raft_consensus.cc:2272] T d4f9d88127114fda91668ab529abd23c P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.317215  4356 tablet_replica.cc:333] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.318156  4356 raft_consensus.cc:2243] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.319406  4356 raft_consensus.cc:2272] T f7b389579627488bb406ec0d112f6938 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.322316  4356 tablet_replica.cc:333] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.323226  4356 raft_consensus.cc:2243] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.323939  4356 raft_consensus.cc:2272] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.327220  4356 tablet_replica.cc:333] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.328254  4356 raft_consensus.cc:2243] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.328980  4356 raft_consensus.cc:2272] T c180f8ddcbe14366a60537e1bad39332 P 60d61e0800194dd89be34d3aff3f374d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.339430  4356 tablet_replica.cc:333] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.340343  4356 raft_consensus.cc:2243] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.340972  4356 raft_consensus.cc:2272] T 3fddb762772a40acb39a71286c8d4f41 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.344102  4356 tablet_replica.cc:333] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.345014  4356 raft_consensus.cc:2243] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.345578  4356 raft_consensus.cc:2272] T 872ec24b41d24573a9a8cc1d484fd6f9 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.348817  4356 tablet_replica.cc:333] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.349673  4356 raft_consensus.cc:2243] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.350257  4356 raft_consensus.cc:2272] T 79ade62c22cf46b3bf97770f5bc7d597 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.353148  4356 tablet_replica.cc:333] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.354070  4356 raft_consensus.cc:2243] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.354647  4356 raft_consensus.cc:2272] T ef4e065748614b3c87fd0a834017c455 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.357712  4356 tablet_replica.cc:333] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.358533  4356 raft_consensus.cc:2243] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.359596  4356 raft_consensus.cc:2272] T d9e24af510de430aabe2a8c0c1010629 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.362020  4356 tablet_replica.cc:333] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.362593  4356 raft_consensus.cc:2243] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.363099  4356 raft_consensus.cc:2272] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.366135  4356 tablet_replica.cc:333] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.367012  4356 raft_consensus.cc:2243] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.367628  4356 raft_consensus.cc:2272] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:18.370210 18805 consensus_peers.cc:597] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.370513  4356 tablet_replica.cc:333] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.371366  4356 raft_consensus.cc:2243] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.372005  4356 raft_consensus.cc:2272] T 9219e0ff53f348ff98bdc3422e5fb74c P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:18.373551 18805 consensus_peers.cc:597] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.374647 18805 consensus_peers.cc:597] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.375188  4356 tablet_replica.cc:333] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.376145  4356 raft_consensus.cc:2243] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.376775  4356 raft_consensus.cc:2272] T 3fb81949f42946cf893026acc49357bc P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.379573  4356 tablet_replica.cc:333] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.380414  4356 raft_consensus.cc:2243] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.380995  4356 raft_consensus.cc:2272] T 8d163b2d60ca4572bca7a1252c65277a P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.383801  4356 tablet_replica.cc:333] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.384601  4356 raft_consensus.cc:2243] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.385632  4356 raft_consensus.cc:2272] T 3c6af67983744957af1aee8d948169c2 P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:18.388139 18731 consensus_peers.cc:597] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.388871 18731 consensus_peers.cc:597] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.388872  4356 tablet_replica.cc:333] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.390108  4356 raft_consensus.cc:2243] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.391237  4356 raft_consensus.cc:2272] T 671e989970204ce48b700c86d5faf91d P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.394063  4356 tablet_replica.cc:333] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.394903  4356 raft_consensus.cc:2243] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.395964  4356 raft_consensus.cc:2272] T a147020c5d724ef2a43e17ee907fe200 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:18.396443 18805 consensus_peers.cc:597] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.399370  4356 tablet_replica.cc:333] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.400490  4356 raft_consensus.cc:2243] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.401247  4356 raft_consensus.cc:2272] T 2acf997d23744fe697db1fe947dca883 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:18.403815 18731 consensus_peers.cc:597] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.404811  4356 tablet_replica.cc:333] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.405543  4356 raft_consensus.cc:2243] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.406147  4356 raft_consensus.cc:2272] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:18.407927 18731 consensus_peers.cc:597] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.409078  4356 tablet_replica.cc:333] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.410045  4356 raft_consensus.cc:2243] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.410647  4356 raft_consensus.cc:2272] T f651b7eda3134c7a897855ac4300de2f P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.413868  4356 tablet_replica.cc:333] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.414749  4356 raft_consensus.cc:2243] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.415339  4356 raft_consensus.cc:2272] T d6e49ecce4f040e99a419db81f1953af P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.418892  4356 tablet_replica.cc:333] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.419984  4356 raft_consensus.cc:2243] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.420885  4356 raft_consensus.cc:2272] T c7f9d9bb9b8742ac930302b42228aad9 P 60d61e0800194dd89be34d3aff3f374d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.424654  4356 tablet_replica.cc:333] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.425523  4356 raft_consensus.cc:2243] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.426767  4356 raft_consensus.cc:2272] T 14206c16a2154538ad2a781e824a65dd P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:18.430132 18805 consensus_peers.cc:597] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.430481  4356 tablet_replica.cc:333] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.431483  4356 raft_consensus.cc:2243] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.432765  4356 raft_consensus.cc:2272] T 7089bc11f6424c178808e4579d967e14 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.437748  4356 tablet_replica.cc:333] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.438571  4356 raft_consensus.cc:2243] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.439630  4356 raft_consensus.cc:2272] T 31e16042d49945ea80b31f9eee56f2cf P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.442565  4356 tablet_replica.cc:333] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.443392  4356 raft_consensus.cc:2243] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.444530  4356 raft_consensus.cc:2272] T 1a66b75f31e14b98902006a8f9d53a95 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.447479  4356 tablet_replica.cc:333] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.448259  4356 raft_consensus.cc:2243] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.449930  4356 raft_consensus.cc:2272] T 2ee74ad2fe774e0aa780b1fdf18b3003 P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.452972  4356 tablet_replica.cc:333] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.453747  4356 raft_consensus.cc:2243] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.454351  4356 raft_consensus.cc:2272] T 98dfe85df2e240449ec5f18cb3280c9b P 60d61e0800194dd89be34d3aff3f374d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.460556  4356 tablet_replica.cc:333] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d: stopping tablet replica
I20260709 12:31:18.461489  4356 raft_consensus.cc:2243] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.463652  4356 raft_consensus.cc:2272] T d8bf5c53527542a6a9069536baff450f P 60d61e0800194dd89be34d3aff3f374d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:18.471663 18805 consensus_peers.cc:597] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.476142 18731 consensus_peers.cc:597] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.484251 18731 consensus_peers.cc:597] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.486043 18805 consensus_peers.cc:597] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.518401 18731 consensus_peers.cc:597] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.565918  4356 tablet_server.cc:196] TabletServer@127.4.65.2:0 shutdown complete.
W20260709 12:31:18.600819 18731 consensus_peers.cc:597] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 -> Peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Couldn't send request to peer 60d61e0800194dd89be34d3aff3f374d. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:44143: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.660899  4356 tablet_server.cc:179] TabletServer@127.4.65.3:0 shutting down...
I20260709 12:31:18.699874 19876 raft_consensus.cc:493] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:31:18.700678 19876 raft_consensus.cc:515] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } } peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } }
I20260709 12:31:18.703410 19876 leader_election.cc:290] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
W20260709 12:31:18.709338 18730 leader_election.cc:336] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:31:18.709972  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 12:31:18.710582 18731 leader_election.cc:336] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Network error: Client connection negotiation failed: client connection to 127.4.65.2:44143: connect: Connection refused (error 111)
I20260709 12:31:18.711035  4356 tablet_replica.cc:333] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.711068 18731 leader_election.cc:304] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 29f82a8594d741bc92dd24ee4a368e01; no voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb
I20260709 12:31:18.712173 19876 raft_consensus.cc:2749] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:31:18.712380  4356 raft_consensus.cc:2243] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.713268  4356 raft_consensus.cc:2272] T 4d64e328a205406bb8de3be26e2a70bf P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.715781  4356 tablet_replica.cc:333] T 6a46101454ef46beb00f1652fabfa153 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.716626  4356 raft_consensus.cc:2243] T 6a46101454ef46beb00f1652fabfa153 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.717473  4356 raft_consensus.cc:2272] T 6a46101454ef46beb00f1652fabfa153 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.720427  4356 tablet_replica.cc:333] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.721747  4356 raft_consensus.cc:2243] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.722981  4356 raft_consensus.cc:2272] T 7eb0c96994864b03a8ec7cda3cc8d85b P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.725736  4356 tablet_replica.cc:333] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.726428  4356 raft_consensus.cc:2243] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.726894  4356 raft_consensus.cc:2272] T 3eed637e5890430db2e802ef6d4362a7 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.729523  4356 tablet_replica.cc:333] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.730304  4356 raft_consensus.cc:2243] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.731356  4356 raft_consensus.cc:2272] T 16afc1d74df849c48d10f5da0a8f80c4 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.734128  4356 tablet_replica.cc:333] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.734799  4356 raft_consensus.cc:2243] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.735613  4356 raft_consensus.cc:2272] T 0840843cb482476a8e18344ff8f1e315 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.737767  4356 tablet_replica.cc:333] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.738324  4356 raft_consensus.cc:2243] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.738734  4356 raft_consensus.cc:2272] T c11b1d6d5a5c4991bd34743fff5ca507 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.740906  4356 tablet_replica.cc:333] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.741499  4356 raft_consensus.cc:2243] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.741948  4356 raft_consensus.cc:2272] T ba28ff472b0f46e8884f70afb21857ae P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.744009  4356 tablet_replica.cc:333] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.744657  4356 raft_consensus.cc:2243] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.745092  4356 raft_consensus.cc:2272] T 9c4804407f8b43bd868a716c04e955e0 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.747771  4356 tablet_replica.cc:333] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.748656  4356 raft_consensus.cc:2243] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.749204  4356 raft_consensus.cc:2272] T 1ab29498797f49d198e49d29f9800da2 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.751981  4356 tablet_replica.cc:333] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.752770  4356 raft_consensus.cc:2243] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.753800  4356 raft_consensus.cc:2272] T c2abc15bf7b445a4b4269e1d93cce949 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.756526  4356 tablet_replica.cc:333] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.757295  4356 raft_consensus.cc:2243] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.757848  4356 raft_consensus.cc:2272] T 9e3d64db91924be490d4ed1f797d338f P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.760600  4356 tablet_replica.cc:333] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.760491 19877 raft_consensus.cc:493] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:31:18.761191 19877 raft_consensus.cc:515] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:18.761634  4356 raft_consensus.cc:2243] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.763224  4356 raft_consensus.cc:2272] T 79f6d22c56d84a6cb05eb4469f831e6e P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.764421 19877 leader_election.cc:290] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:31:18.766855  4356 tablet_replica.cc:333] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.768186  4356 raft_consensus.cc:2243] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.769016  4356 raft_consensus.cc:2272] T 23ca4b75eb0e49dca52e1b9fb4f8b356 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:18.769681 18805 leader_election.cc:336] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:31:18.770989 18804 leader_election.cc:336] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:31:18.771440 18804 leader_election.cc:304] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:31:18.772192  4356 tablet_replica.cc:333] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.772174 19877 raft_consensus.cc:2749] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:31:18.773099  4356 raft_consensus.cc:2243] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.773581  4356 raft_consensus.cc:2272] T d4f9d88127114fda91668ab529abd23c P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.775596  4356 tablet_replica.cc:333] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.776279  4356 raft_consensus.cc:2243] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.777148  4356 raft_consensus.cc:2272] T fc988d3d39094696b3b4516b33a9c37d P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.779153  4356 tablet_replica.cc:333] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.779758  4356 raft_consensus.cc:2243] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.780207  4356 raft_consensus.cc:2272] T 8d163b2d60ca4572bca7a1252c65277a P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.783004  4356 tablet_replica.cc:333] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.783916  4356 raft_consensus.cc:2243] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.784525  4356 raft_consensus.cc:2272] T efa9444e110e433aa97b25a05d5769a9 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.787623  4356 tablet_replica.cc:333] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.788511  4356 raft_consensus.cc:2243] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.789152  4356 raft_consensus.cc:2272] T a6fc5ff8d0014a9f9c1dadcc157e1d9a P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.792588  4356 tablet_replica.cc:333] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.793391  4356 raft_consensus.cc:2243] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.793973  4356 raft_consensus.cc:2272] T c180f8ddcbe14366a60537e1bad39332 P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.796780  4356 tablet_replica.cc:333] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.797405  4356 raft_consensus.cc:2243] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.798208  4356 raft_consensus.cc:2272] T 79ade62c22cf46b3bf97770f5bc7d597 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.800459  4356 tablet_replica.cc:333] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.801057  4356 raft_consensus.cc:2243] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.802098  4356 raft_consensus.cc:2272] T ef4e065748614b3c87fd0a834017c455 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.804513  4356 tablet_replica.cc:333] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.805078  4356 raft_consensus.cc:2243] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.805908  4356 raft_consensus.cc:2272] T f71d9e90994b4dcfa86f9bb9e3b45b51 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.807940  4356 tablet_replica.cc:333] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.808519  4356 raft_consensus.cc:2243] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.808969  4356 raft_consensus.cc:2272] T 9c5cc1ca1ea8468199e1eeddb8c10756 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.810725  4356 tablet_replica.cc:333] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.811276  4356 raft_consensus.cc:2243] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.811688  4356 raft_consensus.cc:2272] T 3fb81949f42946cf893026acc49357bc P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.813795  4356 tablet_replica.cc:333] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.814352  4356 raft_consensus.cc:2243] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.814822  4356 raft_consensus.cc:2272] T 73eeb703f09a4330ad726c1d5e42eee3 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.816941  4356 tablet_replica.cc:333] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.817507  4356 raft_consensus.cc:2243] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.817989  4356 raft_consensus.cc:2272] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.819957  4356 tablet_replica.cc:333] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.820560  4356 raft_consensus.cc:2243] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.820993  4356 raft_consensus.cc:2272] T 31e16042d49945ea80b31f9eee56f2cf P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.822940  4356 tablet_replica.cc:333] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.823513  4356 raft_consensus.cc:2243] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.823961  4356 raft_consensus.cc:2272] T 1a66b75f31e14b98902006a8f9d53a95 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.826195  4356 tablet_replica.cc:333] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.826885  4356 raft_consensus.cc:2243] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.827452  4356 raft_consensus.cc:2272] T 03deb6b823ad408c8cdecc1fbd819940 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.830066  4356 tablet_replica.cc:333] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
W20260709 12:31:18.830412 18805 consensus_peers.cc:597] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Couldn't send request to peer 29f82a8594d741bc92dd24ee4a368e01. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.830860  4356 raft_consensus.cc:2243] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.831844  4356 raft_consensus.cc:2272] T e4c43366d9fb416e8649aba9676aed59 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.834889  4356 tablet_replica.cc:333] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.835739  4356 raft_consensus.cc:2243] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.836189  4356 raft_consensus.cc:2272] T 7089bc11f6424c178808e4579d967e14 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.838291  4356 tablet_replica.cc:333] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.838919  4356 raft_consensus.cc:2243] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.839432  4356 raft_consensus.cc:2272] T 14206c16a2154538ad2a781e824a65dd P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.839632 19877 raft_consensus.cc:493] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:31:18.840198 19877 raft_consensus.cc:515] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:18.842340  4356 tablet_replica.cc:333] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.842520 19877 leader_election.cc:290] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
I20260709 12:31:18.843266  4356 raft_consensus.cc:2243] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.844425  4356 raft_consensus.cc:2272] T 9d43c01f2ed54eec9d5a7b6b75cd6eeb P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:18.846215 18805 leader_election.cc:336] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:31:18.847280  4356 tablet_replica.cc:333] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
W20260709 12:31:18.847615 18804 leader_election.cc:336] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:31:18.848049 18804 leader_election.cc:304] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:31:18.848127  4356 raft_consensus.cc:2243] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.848726 19877 raft_consensus.cc:2749] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:31:18.848794  4356 raft_consensus.cc:2272] T 2dfef0fa153945568323437021ce2921 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.851222  4356 tablet_replica.cc:333] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.851764  4356 raft_consensus.cc:2243] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.852144  4356 raft_consensus.cc:2272] T c7f9d9bb9b8742ac930302b42228aad9 P 29f82a8594d741bc92dd24ee4a368e01 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.854243  4356 tablet_replica.cc:333] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.854805  4356 raft_consensus.cc:2243] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.855316 19877 raft_consensus.cc:493] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:31:18.855719  4356 raft_consensus.cc:2272] T f651b7eda3134c7a897855ac4300de2f P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.855890 19877 raft_consensus.cc:515] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:18.858444  4356 tablet_replica.cc:333] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.859301  4356 raft_consensus.cc:2243] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.859984  4356 raft_consensus.cc:2272] T 3c6af67983744957af1aee8d948169c2 P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.861732 19877 leader_election.cc:290] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
W20260709 12:31:18.862985 18805 leader_election.cc:336] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:31:18.863152  4356 tablet_replica.cc:333] T 089c667ac1ae4a22b9b2e484f978c5d4 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.864217  4356 raft_consensus.cc:2243] T 089c667ac1ae4a22b9b2e484f978c5d4 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:31:18.864662 18804 leader_election.cc:336] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:31:18.864908  4356 raft_consensus.cc:2272] T 089c667ac1ae4a22b9b2e484f978c5d4 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.865198 18804 leader_election.cc:304] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:31:18.866078 19877 raft_consensus.cc:2749] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:31:18.867583  4356 tablet_replica.cc:333] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.868274  4356 raft_consensus.cc:2243] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.868815  4356 raft_consensus.cc:2272] T a147020c5d724ef2a43e17ee907fe200 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.871372  4356 tablet_replica.cc:333] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.871942  4356 raft_consensus.cc:2243] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.873065  4356 raft_consensus.cc:2272] T 2acf997d23744fe697db1fe947dca883 P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:18.874621 18805 consensus_peers.cc:597] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Couldn't send request to peer 29f82a8594d741bc92dd24ee4a368e01. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.875837  4356 tablet_replica.cc:333] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.876588  4356 raft_consensus.cc:2243] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.877643  4356 raft_consensus.cc:2272] T d659dc42667d4202a29b529fe3388c8e P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.879674  4356 tablet_replica.cc:333] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.880198  4356 raft_consensus.cc:2243] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:18.881026  4356 raft_consensus.cc:2272] T 98dfe85df2e240449ec5f18cb3280c9b P 29f82a8594d741bc92dd24ee4a368e01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:18.882885  4356 tablet_replica.cc:333] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01: stopping tablet replica
I20260709 12:31:18.883355  4356 raft_consensus.cc:2243] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:18.883770  4356 raft_consensus.cc:2272] T d8bf5c53527542a6a9069536baff450f P 29f82a8594d741bc92dd24ee4a368e01 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:18.905768 18805 consensus_peers.cc:597] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Couldn't send request to peer 29f82a8594d741bc92dd24ee4a368e01. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:18.967099 18805 consensus_peers.cc:597] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Couldn't send request to peer 29f82a8594d741bc92dd24ee4a368e01. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.3:40335: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:18.987040 19813 raft_consensus.cc:493] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:31:18.987704 19813 raft_consensus.cc:515] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:18.990329 19813 leader_election.cc:290] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
W20260709 12:31:18.995297 18805 leader_election.cc:336] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Network error: Client connection negotiation failed: client connection to 127.4.65.3:40335: connect: Connection refused (error 111)
W20260709 12:31:18.997949 18804 leader_election.cc:336] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
I20260709 12:31:18.998469 18804 leader_election.cc:304] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:31:18.999183 19823 raft_consensus.cc:2749] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:31:19.002450  4356 tablet_server.cc:196] TabletServer@127.4.65.3:0 shutdown complete.
W20260709 12:31:19.007670 18805 consensus_peers.cc:597] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Couldn't send request to peer 29f82a8594d741bc92dd24ee4a368e01. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.3:40335: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:19.017796 18805 consensus_peers.cc:597] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce -> Peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Couldn't send request to peer 29f82a8594d741bc92dd24ee4a368e01. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.3:40335: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:19.023183 19823 raft_consensus.cc:493] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:31:19.023831 19823 raft_consensus.cc:515] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "60d61e0800194dd89be34d3aff3f374d" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 44143 } }
I20260709 12:31:19.025961 19823 leader_election.cc:290] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 84812365c65f43f388836c087640adbb (127.4.65.1:36569), 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143)
W20260709 12:31:19.032027 18804 leader_election.cc:336] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
W20260709 12:31:19.032773 18805 leader_election.cc:336] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 60d61e0800194dd89be34d3aff3f374d (127.4.65.2:44143): Network error: Client connection negotiation failed: client connection to 127.4.65.2:44143: connect: Connection refused (error 111)
I20260709 12:31:19.033200 18805 leader_election.cc:304] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 60d61e0800194dd89be34d3aff3f374d, 84812365c65f43f388836c087640adbb
I20260709 12:31:19.033852 19823 raft_consensus.cc:2749] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:31:19.041152 18805 consensus_peers.cc:597] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce -> Peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Couldn't send request to peer 29f82a8594d741bc92dd24ee4a368e01. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.3:40335: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:19.105530  4356 tablet_server.cc:179] TabletServer@127.4.65.4:0 shutting down...
I20260709 12:31:19.146202 19823 raft_consensus.cc:493] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 84812365c65f43f388836c087640adbb)
I20260709 12:31:19.146762 19823 raft_consensus.cc:515] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29f82a8594d741bc92dd24ee4a368e01" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 40335 } } peers { permanent_uuid: "df352ed4600b4ddb83ef57ecd2944fce" member_type: VOTER last_known_addr { host: "127.4.65.4" port: 44517 } } peers { permanent_uuid: "84812365c65f43f388836c087640adbb" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 36569 } }
I20260709 12:31:19.148757 19823 leader_election.cc:290] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335), 84812365c65f43f388836c087640adbb (127.4.65.1:36569)
W20260709 12:31:19.152961 18804 leader_election.cc:336] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 84812365c65f43f388836c087640adbb (127.4.65.1:36569): Network error: Client connection negotiation failed: client connection to 127.4.65.1:36569: connect: Connection refused (error 111)
W20260709 12:31:19.153963 18805 leader_election.cc:336] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Network error: Client connection negotiation failed: client connection to 127.4.65.3:40335: connect: Connection refused (error 111)
I20260709 12:31:19.154683 18805 leader_election.cc:304] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df352ed4600b4ddb83ef57ecd2944fce; no voters: 29f82a8594d741bc92dd24ee4a368e01, 84812365c65f43f388836c087640adbb
I20260709 12:31:19.155328 19823 raft_consensus.cc:2749] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:31:19.157030  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 12:31:19.157078 18805 consensus_peers.cc:597] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce -> Peer 29f82a8594d741bc92dd24ee4a368e01 (127.4.65.3:40335): Couldn't send request to peer 29f82a8594d741bc92dd24ee4a368e01. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.3:40335: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:19.157902  4356 tablet_replica.cc:333] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.158617  4356 raft_consensus.cc:2243] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.159111  4356 raft_consensus.cc:2272] T 4d64e328a205406bb8de3be26e2a70bf P df352ed4600b4ddb83ef57ecd2944fce [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.161438  4356 tablet_replica.cc:333] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.162004  4356 raft_consensus.cc:2243] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.162781  4356 raft_consensus.cc:2272] T 6a46101454ef46beb00f1652fabfa153 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.165870  4356 tablet_replica.cc:333] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.166702  4356 raft_consensus.cc:2243] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.167261  4356 raft_consensus.cc:2272] T 8c4a560e00884f1cb6f8720dfee7d27c P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.169757  4356 tablet_replica.cc:333] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.170321  4356 raft_consensus.cc:2243] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.171041  4356 raft_consensus.cc:2272] T d216c51d889b417bb2e59f20b6fd6d96 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.173023  4356 tablet_replica.cc:333] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.173559  4356 raft_consensus.cc:2243] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.173933  4356 raft_consensus.cc:2272] T e06e3fb1322840a6b22c5da53900345b P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.175712  4356 tablet_replica.cc:333] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.176296  4356 raft_consensus.cc:2243] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.176754  4356 raft_consensus.cc:2272] T 98dfe85df2e240449ec5f18cb3280c9b P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.178634  4356 tablet_replica.cc:333] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.179217  4356 raft_consensus.cc:2243] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.179647  4356 raft_consensus.cc:2272] T 7eb0c96994864b03a8ec7cda3cc8d85b P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.181754  4356 tablet_replica.cc:333] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.182344  4356 raft_consensus.cc:2243] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.183104  4356 raft_consensus.cc:2272] T 3eed637e5890430db2e802ef6d4362a7 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.185145  4356 tablet_replica.cc:333] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.185709  4356 raft_consensus.cc:2243] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.186496  4356 raft_consensus.cc:2272] T 4794853627e84898986305827ec4ea46 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.188485  4356 tablet_replica.cc:333] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.189029  4356 raft_consensus.cc:2243] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.189422  4356 raft_consensus.cc:2272] T 16afc1d74df849c48d10f5da0a8f80c4 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.191345  4356 tablet_replica.cc:333] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.191902  4356 raft_consensus.cc:2243] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.192751  4356 raft_consensus.cc:2272] T f4eead98f0854adfaf72daa76fdf9c9b P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.194710  4356 tablet_replica.cc:333] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.195312  4356 raft_consensus.cc:2243] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.195716  4356 raft_consensus.cc:2272] T ba28ff472b0f46e8884f70afb21857ae P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.197672  4356 tablet_replica.cc:333] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.198246  4356 raft_consensus.cc:2243] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.198657  4356 raft_consensus.cc:2272] T 9c4804407f8b43bd868a716c04e955e0 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.200575  4356 tablet_replica.cc:333] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.201249  4356 raft_consensus.cc:2243] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.201656  4356 raft_consensus.cc:2272] T 1ab29498797f49d198e49d29f9800da2 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.203475  4356 tablet_replica.cc:333] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.203990  4356 raft_consensus.cc:2243] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.204385  4356 raft_consensus.cc:2272] T c2abc15bf7b445a4b4269e1d93cce949 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.206317  4356 tablet_replica.cc:333] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.206861  4356 raft_consensus.cc:2243] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.207224  4356 raft_consensus.cc:2272] T fc988d3d39094696b3b4516b33a9c37d P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.209080  4356 tablet_replica.cc:333] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.209570  4356 raft_consensus.cc:2243] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.210306  4356 raft_consensus.cc:2272] T 8d163b2d60ca4572bca7a1252c65277a P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.212182  4356 tablet_replica.cc:333] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.212827  4356 raft_consensus.cc:2243] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.213630  4356 raft_consensus.cc:2272] T efa9444e110e433aa97b25a05d5769a9 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.215440  4356 tablet_replica.cc:333] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.215934  4356 raft_consensus.cc:2243] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.216300  4356 raft_consensus.cc:2272] T e4c43366d9fb416e8649aba9676aed59 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.218175  4356 tablet_replica.cc:333] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.218703  4356 raft_consensus.cc:2243] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.219096  4356 raft_consensus.cc:2272] T 14206c16a2154538ad2a781e824a65dd P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.220966  4356 tablet_replica.cc:333] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.221452  4356 raft_consensus.cc:2243] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.221818  4356 raft_consensus.cc:2272] T 7089bc11f6424c178808e4579d967e14 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.223623  4356 tablet_replica.cc:333] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.224129  4356 raft_consensus.cc:2243] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.224599  4356 raft_consensus.cc:2272] T 79f6d22c56d84a6cb05eb4469f831e6e P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.226476  4356 tablet_replica.cc:333] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.227059  4356 raft_consensus.cc:2243] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.227852  4356 raft_consensus.cc:2272] T df295fba24764e2ab46a8985e569bad8 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.229868  4356 tablet_replica.cc:333] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.230461  4356 raft_consensus.cc:2243] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.231284  4356 raft_consensus.cc:2272] T 872ec24b41d24573a9a8cc1d484fd6f9 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.233320  4356 tablet_replica.cc:333] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.233896  4356 raft_consensus.cc:2243] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.234643  4356 raft_consensus.cc:2272] T 3fddb762772a40acb39a71286c8d4f41 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.236527  4356 tablet_replica.cc:333] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.237072  4356 raft_consensus.cc:2243] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.237855  4356 raft_consensus.cc:2272] T 6f16818c4dfd4e9ab187ffb53d8fc3ce P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.239687  4356 tablet_replica.cc:333] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.240166  4356 raft_consensus.cc:2243] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.240626  4356 raft_consensus.cc:2272] T 03deb6b823ad408c8cdecc1fbd819940 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.242491  4356 tablet_replica.cc:333] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.243070  4356 raft_consensus.cc:2243] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.243501  4356 raft_consensus.cc:2272] T 2ee74ad2fe774e0aa780b1fdf18b3003 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.245347  4356 tablet_replica.cc:333] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.245930  4356 raft_consensus.cc:2243] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.246373  4356 raft_consensus.cc:2272] T f7b389579627488bb406ec0d112f6938 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.248270  4356 tablet_replica.cc:333] T ef4e065748614b3c87fd0a834017c455 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.248850  4356 raft_consensus.cc:2243] T ef4e065748614b3c87fd0a834017c455 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.249293  4356 raft_consensus.cc:2272] T ef4e065748614b3c87fd0a834017c455 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.251302  4356 tablet_replica.cc:333] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.251832  4356 raft_consensus.cc:2243] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.252199  4356 raft_consensus.cc:2272] T d9e24af510de430aabe2a8c0c1010629 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.254103  4356 tablet_replica.cc:333] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.254559  4356 raft_consensus.cc:2243] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.255259  4356 raft_consensus.cc:2272] T 9c5cc1ca1ea8468199e1eeddb8c10756 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.257236  4356 tablet_replica.cc:333] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.257731  4356 raft_consensus.cc:2243] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.258443  4356 raft_consensus.cc:2272] T 089c667ac1ae4a22b9b2e484f978c5d4 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.260449  4356 tablet_replica.cc:333] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.261016  4356 raft_consensus.cc:2243] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.261458  4356 raft_consensus.cc:2272] T 671e989970204ce48b700c86d5faf91d P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.263605  4356 tablet_replica.cc:333] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.264153  4356 raft_consensus.cc:2243] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.264595  4356 raft_consensus.cc:2272] T d659dc42667d4202a29b529fe3388c8e P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.266494  4356 tablet_replica.cc:333] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.267020  4356 raft_consensus.cc:2243] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.267419  4356 raft_consensus.cc:2272] T a147020c5d724ef2a43e17ee907fe200 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.269356  4356 tablet_replica.cc:333] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.269946  4356 raft_consensus.cc:2243] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.270370  4356 raft_consensus.cc:2272] T 2acf997d23744fe697db1fe947dca883 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.272249  4356 tablet_replica.cc:333] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.272806  4356 raft_consensus.cc:2243] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.273586  4356 raft_consensus.cc:2272] T 73eeb703f09a4330ad726c1d5e42eee3 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.275491  4356 tablet_replica.cc:333] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.275996  4356 raft_consensus.cc:2243] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.276432  4356 raft_consensus.cc:2272] T 31e16042d49945ea80b31f9eee56f2cf P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.278275  4356 tablet_replica.cc:333] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.278762  4356 raft_consensus.cc:2243] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.279158  4356 raft_consensus.cc:2272] T 6256d41aab434828a66720ab61fa9ba5 P df352ed4600b4ddb83ef57ecd2944fce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.281504  4356 tablet_replica.cc:333] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.282052  4356 raft_consensus.cc:2243] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.282421  4356 raft_consensus.cc:2272] T 2dfef0fa153945568323437021ce2921 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.284190  4356 tablet_replica.cc:333] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.284680  4356 raft_consensus.cc:2243] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.285477  4356 raft_consensus.cc:2272] T d6e49ecce4f040e99a419db81f1953af P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.287493  4356 tablet_replica.cc:333] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.287979  4356 raft_consensus.cc:2243] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.288448  4356 raft_consensus.cc:2272] T 9219e0ff53f348ff98bdc3422e5fb74c P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.290378  4356 tablet_replica.cc:333] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce: stopping tablet replica
I20260709 12:31:19.290863  4356 raft_consensus.cc:2243] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:19.291292  4356 raft_consensus.cc:2272] T 79ade62c22cf46b3bf97770f5bc7d597 P df352ed4600b4ddb83ef57ecd2944fce [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.404938  4356 tablet_server.cc:196] TabletServer@127.4.65.4:0 shutdown complete.
I20260709 12:31:19.470283  4356 master.cc:562] Master@127.4.65.62:41379 shutting down...
I20260709 12:31:19.503542  4356 raft_consensus.cc:2243] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:19.504237  4356 raft_consensus.cc:2272] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:19.504666  4356 tablet_replica.cc:333] T 00000000000000000000000000000000 P f41933e050fe40c89e214498078505f5: stopping tablet replica
I20260709 12:31:19.528499  4356 master.cc:584] Master@127.4.65.62:41379 shutdown complete.
[       OK ] LeaderRebalancerTest.RestartTserver (39707 ms)
[ RUN      ] LeaderRebalancerTest.TestMaintenanceMode
I20260709 12:31:19.627076  4356 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.65.62:44901
I20260709 12:31:19.628296  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:31:19.634917 19881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:19.635092 19882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:19.636193 19884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:31:19.639597  4356 server_base.cc:1061] running on GCE node
I20260709 12:31:19.640997  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:31:19.641199  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:31:19.641390  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600279641374 us; error 0 us; skew 500 ppm
I20260709 12:31:19.641989  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:31:19.645183  4356 webserver.cc:533] Webserver started at http://127.4.65.62:32925/ using document root <none> and password file <none>
I20260709 12:31:19.645709  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:31:19.645891  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:31:19.646236  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:31:19.647430  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/master-0-root/instance:
uuid: "275b9d09083d49e69693e575775f2008"
format_stamp: "Formatted at 2026-07-09 12:31:19 on dist-test-slave-l1c9"
I20260709 12:31:19.652243  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.006s
I20260709 12:31:19.655908 19889 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:19.657119  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:31:19.657469  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/master-0-root
uuid: "275b9d09083d49e69693e575775f2008"
format_stamp: "Formatted at 2026-07-09 12:31:19 on dist-test-slave-l1c9"
I20260709 12:31:19.657795  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:31:19.669819  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:31:19.671092  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:31:19.717990  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.62:44901
I20260709 12:31:19.718115 19940 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.62:44901 every 8 connection(s)
I20260709 12:31:19.722033 19941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:19.734252 19941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008: Bootstrap starting.
I20260709 12:31:19.739588 19941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:19.744135 19941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008: No bootstrap required, opened a new log
I20260709 12:31:19.746558 19941 raft_consensus.cc:359] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275b9d09083d49e69693e575775f2008" member_type: VOTER }
I20260709 12:31:19.746969 19941 raft_consensus.cc:385] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:19.747251 19941 raft_consensus.cc:740] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 275b9d09083d49e69693e575775f2008, State: Initialized, Role: FOLLOWER
I20260709 12:31:19.747995 19941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275b9d09083d49e69693e575775f2008" member_type: VOTER }
I20260709 12:31:19.748487 19941 raft_consensus.cc:399] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:31:19.748718 19941 raft_consensus.cc:493] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:31:19.749012 19941 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:19.754072 19941 raft_consensus.cc:515] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275b9d09083d49e69693e575775f2008" member_type: VOTER }
I20260709 12:31:19.754659 19941 leader_election.cc:304] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 275b9d09083d49e69693e575775f2008; no voters: 
I20260709 12:31:19.755853 19941 leader_election.cc:290] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:31:19.756227 19944 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:19.757656 19944 raft_consensus.cc:697] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [term 1 LEADER]: Becoming Leader. State: Replica: 275b9d09083d49e69693e575775f2008, State: Running, Role: LEADER
I20260709 12:31:19.758396 19944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275b9d09083d49e69693e575775f2008" member_type: VOTER }
I20260709 12:31:19.759011 19941 sys_catalog.cc:565] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:31:19.763021 19945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "275b9d09083d49e69693e575775f2008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275b9d09083d49e69693e575775f2008" member_type: VOTER } }
I20260709 12:31:19.763860 19945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [sys.catalog]: This master's current role is: LEADER
I20260709 12:31:19.763919 19946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 275b9d09083d49e69693e575775f2008. Latest consensus state: current_term: 1 leader_uuid: "275b9d09083d49e69693e575775f2008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "275b9d09083d49e69693e575775f2008" member_type: VOTER } }
I20260709 12:31:19.764647 19946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [sys.catalog]: This master's current role is: LEADER
I20260709 12:31:19.767315 19951 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:31:19.775599 19951 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:31:19.776782  4356 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:31:19.785624 19951 catalog_manager.cc:1383] Generated new cluster ID: d9ef9c73799b46b68d5472d9c9ce9af6
I20260709 12:31:19.786026 19951 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:31:19.816270 19951 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:31:19.817629 19951 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:31:19.831089 19951 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008: Generated new TSK 0
I20260709 12:31:19.831815 19951 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:31:19.844682  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:31:19.850996 19962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:19.852008 19963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:19.855253 19965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:31:19.855432  4356 server_base.cc:1061] running on GCE node
I20260709 12:31:19.856959  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:31:19.857241  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:31:19.857610  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600279857572 us; error 0 us; skew 500 ppm
I20260709 12:31:19.858317  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:31:19.861347  4356 webserver.cc:533] Webserver started at http://127.4.65.1:36645/ using document root <none> and password file <none>
I20260709 12:31:19.861891  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:31:19.862113  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:31:19.862406  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:31:19.863718  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-0-root/instance:
uuid: "971f449846cf4a269f817059278b86ea"
format_stamp: "Formatted at 2026-07-09 12:31:19 on dist-test-slave-l1c9"
I20260709 12:31:19.869019  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:31:19.873739 19970 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:19.874637  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.001s
I20260709 12:31:19.874976  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-0-root
uuid: "971f449846cf4a269f817059278b86ea"
format_stamp: "Formatted at 2026-07-09 12:31:19 on dist-test-slave-l1c9"
I20260709 12:31:19.875324  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:31:19.904053  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:31:19.905449  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:31:19.906941  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:31:19.910298  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:31:19.910598  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:19.910885  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:31:19.911093  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:19.959581  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.1:44731
I20260709 12:31:19.959913 20032 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.1:44731 every 8 connection(s)
I20260709 12:31:19.965183  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:31:19.973587 20037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:19.974540 20038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:31:19.977628  4356 server_base.cc:1061] running on GCE node
W20260709 12:31:19.978454 20040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:31:19.979586  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:31:19.979882  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:31:19.980091  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600279980076 us; error 0 us; skew 500 ppm
I20260709 12:31:19.980854  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:31:19.983011 20033 heartbeater.cc:344] Connected to a master server at 127.4.65.62:44901
I20260709 12:31:19.983399 20033 heartbeater.cc:461] Registering TS with master...
I20260709 12:31:19.983750  4356 webserver.cc:533] Webserver started at http://127.4.65.2:36663/ using document root <none> and password file <none>
I20260709 12:31:19.984254 20033 heartbeater.cc:507] Master 127.4.65.62:44901 requested a full tablet report, sending...
I20260709 12:31:19.984411  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:31:19.984724  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:31:19.985010  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:31:19.986384  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-1-root/instance:
uuid: "4ab3d29ae436426ab6d05fef08b4641b"
format_stamp: "Formatted at 2026-07-09 12:31:19 on dist-test-slave-l1c9"
I20260709 12:31:19.986526 19906 ts_manager.cc:194] Registered new tserver with Master: 971f449846cf4a269f817059278b86ea (127.4.65.1:44731)
I20260709 12:31:19.988339 19906 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51404
I20260709 12:31:19.991783  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:31:19.995630 20045 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:19.996512  4356 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:31:19.996836  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-1-root
uuid: "4ab3d29ae436426ab6d05fef08b4641b"
format_stamp: "Formatted at 2026-07-09 12:31:19 on dist-test-slave-l1c9"
I20260709 12:31:19.997153  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:31:20.012332  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:31:20.013711  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:31:20.015250  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:31:20.017522  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:31:20.017766  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:20.018019  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:31:20.018217  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:20.069257  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.2:38481
I20260709 12:31:20.069427 20107 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.2:38481 every 8 connection(s)
I20260709 12:31:20.075232  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:31:20.083529 20111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:20.085032 20112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:31:20.089025 20108 heartbeater.cc:344] Connected to a master server at 127.4.65.62:44901
I20260709 12:31:20.089479  4356 server_base.cc:1061] running on GCE node
I20260709 12:31:20.089505 20108 heartbeater.cc:461] Registering TS with master...
W20260709 12:31:20.090405 20114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:31:20.090785 20108 heartbeater.cc:507] Master 127.4.65.62:44901 requested a full tablet report, sending...
I20260709 12:31:20.091485  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:31:20.091768  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:31:20.091957  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600280091943 us; error 0 us; skew 500 ppm
I20260709 12:31:20.092758  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:31:20.093464 19906 ts_manager.cc:194] Registered new tserver with Master: 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:20.095633  4356 webserver.cc:533] Webserver started at http://127.4.65.3:40689/ using document root <none> and password file <none>
I20260709 12:31:20.095777 19906 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51406
I20260709 12:31:20.096442  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:31:20.096719  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:31:20.097002  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:31:20.098217  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-2-root/instance:
uuid: "44efc7828ed44d76bb553655566fd86e"
format_stamp: "Formatted at 2026-07-09 12:31:20 on dist-test-slave-l1c9"
I20260709 12:31:20.103051  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 12:31:20.106487 20119 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:20.107281  4356 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:31:20.107589  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-2-root
uuid: "44efc7828ed44d76bb553655566fd86e"
format_stamp: "Formatted at 2026-07-09 12:31:20 on dist-test-slave-l1c9"
I20260709 12:31:20.107908  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:31:20.128129  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:31:20.129364  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:31:20.130975  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:31:20.133471  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:31:20.133736  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:20.133992  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:31:20.134173  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:20.182732  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.3:44645
I20260709 12:31:20.182864 20181 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.3:44645 every 8 connection(s)
I20260709 12:31:20.198482 20182 heartbeater.cc:344] Connected to a master server at 127.4.65.62:44901
I20260709 12:31:20.198942 20182 heartbeater.cc:461] Registering TS with master...
I20260709 12:31:20.199669 20182 heartbeater.cc:507] Master 127.4.65.62:44901 requested a full tablet report, sending...
I20260709 12:31:20.201720 19906 ts_manager.cc:194] Registered new tserver with Master: 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:20.203002  4356 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016873413s
I20260709 12:31:20.203202 19906 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51418
I20260709 12:31:20.232589 19906 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51426:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:31:20.235769 19906 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:31:20.324817 20147 tablet_service.cc:1511] Processing CreateTablet for tablet c8b4cb1a18524c3d8189e535e8666620 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:31:20.326386 20147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8b4cb1a18524c3d8189e535e8666620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.327477 20145 tablet_service.cc:1511] Processing CreateTablet for tablet d224f01fe0364162a1f99b2863aab517 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:31:20.329027 20145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d224f01fe0364162a1f99b2863aab517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.326289 20146 tablet_service.cc:1511] Processing CreateTablet for tablet 6d74a1b173ce4594b4b9090f9b497f87 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:31:20.330605 20146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d74a1b173ce4594b4b9090f9b497f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.330687 20144 tablet_service.cc:1511] Processing CreateTablet for tablet daaf356d34ab43e399517f8b355739a9 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:31:20.332139 20144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daaf356d34ab43e399517f8b355739a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.344386 20143 tablet_service.cc:1511] Processing CreateTablet for tablet b3177f3e632242e5ba149e10e70ab098 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:31:20.345943 20143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3177f3e632242e5ba149e10e70ab098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.346705 19998 tablet_service.cc:1511] Processing CreateTablet for tablet c8b4cb1a18524c3d8189e535e8666620 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:31:20.349766 19997 tablet_service.cc:1511] Processing CreateTablet for tablet 6d74a1b173ce4594b4b9090f9b497f87 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:31:20.350606 20142 tablet_service.cc:1511] Processing CreateTablet for tablet 594477803faa4705a2b7bd7eb621295d (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:31:20.352459 20142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 594477803faa4705a2b7bd7eb621295d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.352252 19996 tablet_service.cc:1511] Processing CreateTablet for tablet d224f01fe0364162a1f99b2863aab517 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:31:20.354868 19995 tablet_service.cc:1511] Processing CreateTablet for tablet daaf356d34ab43e399517f8b355739a9 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:31:20.356190 20141 tablet_service.cc:1511] Processing CreateTablet for tablet a1aa8c0c52f34a8c8b4a2aa87fa57d6e (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:31:20.357718 19998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8b4cb1a18524c3d8189e535e8666620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.357787 20141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1aa8c0c52f34a8c8b4a2aa87fa57d6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.359925 19993 tablet_service.cc:1511] Processing CreateTablet for tablet 594477803faa4705a2b7bd7eb621295d (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:31:20.360692 19996 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d224f01fe0364162a1f99b2863aab517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.361481 19993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 594477803faa4705a2b7bd7eb621295d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.362949 20140 tablet_service.cc:1511] Processing CreateTablet for tablet aa280773cc224cea8253ae8fa2e32e27 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:31:20.363400 19997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d74a1b173ce4594b4b9090f9b497f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.364446 20140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa280773cc224cea8253ae8fa2e32e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.364826 19995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daaf356d34ab43e399517f8b355739a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.367707 19992 tablet_service.cc:1511] Processing CreateTablet for tablet a1aa8c0c52f34a8c8b4a2aa87fa57d6e (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:31:20.370149 19992 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1aa8c0c52f34a8c8b4a2aa87fa57d6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.371196 20139 tablet_service.cc:1511] Processing CreateTablet for tablet 043dc64f584a4a35b66b8c7382e92af4 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:31:20.377326 20139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 043dc64f584a4a35b66b8c7382e92af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.372089 19991 tablet_service.cc:1511] Processing CreateTablet for tablet aa280773cc224cea8253ae8fa2e32e27 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:31:20.381779 19991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa280773cc224cea8253ae8fa2e32e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.386492 20073 tablet_service.cc:1511] Processing CreateTablet for tablet c8b4cb1a18524c3d8189e535e8666620 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:31:20.387193 20138 tablet_service.cc:1511] Processing CreateTablet for tablet 5e2f9bc36a87410f9f14a842ef3785ba (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:31:20.388868 19990 tablet_service.cc:1511] Processing CreateTablet for tablet 043dc64f584a4a35b66b8c7382e92af4 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:31:20.390648 19990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 043dc64f584a4a35b66b8c7382e92af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.390241 20072 tablet_service.cc:1511] Processing CreateTablet for tablet 6d74a1b173ce4594b4b9090f9b497f87 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:31:20.388259 20073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8b4cb1a18524c3d8189e535e8666620. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.388854 20138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2f9bc36a87410f9f14a842ef3785ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.395673 20072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d74a1b173ce4594b4b9090f9b497f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.357319 19994 tablet_service.cc:1511] Processing CreateTablet for tablet b3177f3e632242e5ba149e10e70ab098 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:31:20.397313 19994 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3177f3e632242e5ba149e10e70ab098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.401355 19989 tablet_service.cc:1511] Processing CreateTablet for tablet 5e2f9bc36a87410f9f14a842ef3785ba (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:31:20.402802 19989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2f9bc36a87410f9f14a842ef3785ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.405439 20071 tablet_service.cc:1511] Processing CreateTablet for tablet d224f01fe0364162a1f99b2863aab517 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:31:20.406899 20071 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d224f01fe0364162a1f99b2863aab517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.410133 20070 tablet_service.cc:1511] Processing CreateTablet for tablet daaf356d34ab43e399517f8b355739a9 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:31:20.411558 20070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daaf356d34ab43e399517f8b355739a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.422313 20069 tablet_service.cc:1511] Processing CreateTablet for tablet b3177f3e632242e5ba149e10e70ab098 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:31:20.424412 20069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3177f3e632242e5ba149e10e70ab098. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.430203 20068 tablet_service.cc:1511] Processing CreateTablet for tablet 594477803faa4705a2b7bd7eb621295d (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:31:20.431916 20068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 594477803faa4705a2b7bd7eb621295d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.438306 20067 tablet_service.cc:1511] Processing CreateTablet for tablet a1aa8c0c52f34a8c8b4a2aa87fa57d6e (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:31:20.439896 20067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1aa8c0c52f34a8c8b4a2aa87fa57d6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.489699 20066 tablet_service.cc:1511] Processing CreateTablet for tablet aa280773cc224cea8253ae8fa2e32e27 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:31:20.491294 20066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa280773cc224cea8253ae8fa2e32e27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.492537 20064 tablet_service.cc:1511] Processing CreateTablet for tablet 5e2f9bc36a87410f9f14a842ef3785ba (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:31:20.493993 20064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e2f9bc36a87410f9f14a842ef3785ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.497659 20065 tablet_service.cc:1511] Processing CreateTablet for tablet 043dc64f584a4a35b66b8c7382e92af4 (DEFAULT_TABLE table=test-workload [id=67b817616df14b6b83a9a287f6726cf0]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:31:20.499151 20065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 043dc64f584a4a35b66b8c7382e92af4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:20.520186 20198 tablet_bootstrap.cc:492] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b: Bootstrap starting.
I20260709 12:31:20.527086 20198 tablet_bootstrap.cc:654] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.540486 20198 tablet_bootstrap.cc:492] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b: No bootstrap required, opened a new log
I20260709 12:31:20.541061 20198 ts_tablet_manager.cc:1403] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b: Time spent bootstrapping tablet: real 0.021s	user 0.011s	sys 0.007s
I20260709 12:31:20.541743 20199 tablet_bootstrap.cc:492] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:20.544298 20198 raft_consensus.cc:359] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.545220 20198 raft_consensus.cc:385] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.549072 20198 raft_consensus.cc:740] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.550814 20198 consensus_queue.cc:260] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.551930 20199 tablet_bootstrap.cc:654] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.553675 20197 tablet_bootstrap.cc:492] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e: Bootstrap starting.
I20260709 12:31:20.560477 20198 ts_tablet_manager.cc:1434] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b: Time spent starting tablet: real 0.019s	user 0.011s	sys 0.004s
I20260709 12:31:20.561410 20108 heartbeater.cc:499] Master 127.4.65.62:44901 was elected leader, sending a full tablet report...
I20260709 12:31:20.564901 20198 tablet_bootstrap.cc:492] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b: Bootstrap starting.
I20260709 12:31:20.566085 20197 tablet_bootstrap.cc:654] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.572203 20198 tablet_bootstrap.cc:654] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b: Neither blocks nor log segments found. Creating new log.
W20260709 12:31:20.574275 20109 tablet.cc:2406] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:31:20.585017 20198 tablet_bootstrap.cc:492] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b: No bootstrap required, opened a new log
I20260709 12:31:20.585726 20198 ts_tablet_manager.cc:1403] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b: Time spent bootstrapping tablet: real 0.021s	user 0.011s	sys 0.007s
I20260709 12:31:20.588833 20197 tablet_bootstrap.cc:492] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e: No bootstrap required, opened a new log
I20260709 12:31:20.589439 20197 ts_tablet_manager.cc:1403] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e: Time spent bootstrapping tablet: real 0.036s	user 0.013s	sys 0.008s
I20260709 12:31:20.592432 20197 raft_consensus.cc:359] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.593195 20197 raft_consensus.cc:385] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.593490 20197 raft_consensus.cc:740] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.594446 20197 consensus_queue.cc:260] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.595269 20198 raft_consensus.cc:359] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.596773 20198 raft_consensus.cc:385] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.597126 20198 raft_consensus.cc:740] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.598171 20198 consensus_queue.cc:260] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.600827 20199 tablet_bootstrap.cc:492] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea: No bootstrap required, opened a new log
I20260709 12:31:20.601576 20199 ts_tablet_manager.cc:1403] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.060s	user 0.014s	sys 0.000s
I20260709 12:31:20.602195 20197 ts_tablet_manager.cc:1434] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.001s
I20260709 12:31:20.603075 20182 heartbeater.cc:499] Master 127.4.65.62:44901 was elected leader, sending a full tablet report...
I20260709 12:31:20.605552 20199 raft_consensus.cc:359] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.606472 20199 raft_consensus.cc:385] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.608234 20199 raft_consensus.cc:740] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.609133 20199 consensus_queue.cc:260] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.609964 20198 ts_tablet_manager.cc:1434] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.002s
I20260709 12:31:20.617203 20198 tablet_bootstrap.cc:492] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b: Bootstrap starting.
I20260709 12:31:20.619237 20197 tablet_bootstrap.cc:492] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e: Bootstrap starting.
I20260709 12:31:20.626801 20198 tablet_bootstrap.cc:654] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.642550 20197 tablet_bootstrap.cc:654] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.647909 20033 heartbeater.cc:499] Master 127.4.65.62:44901 was elected leader, sending a full tablet report...
I20260709 12:31:20.654992 20199 ts_tablet_manager.cc:1434] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.053s	user 0.012s	sys 0.014s
I20260709 12:31:20.656589 20199 tablet_bootstrap.cc:492] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:20.661458 20197 tablet_bootstrap.cc:492] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e: No bootstrap required, opened a new log
I20260709 12:31:20.662062 20197 ts_tablet_manager.cc:1403] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e: Time spent bootstrapping tablet: real 0.043s	user 0.009s	sys 0.003s
I20260709 12:31:20.663595 20198 tablet_bootstrap.cc:492] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b: No bootstrap required, opened a new log
I20260709 12:31:20.664444 20198 ts_tablet_manager.cc:1403] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b: Time spent bootstrapping tablet: real 0.047s	user 0.027s	sys 0.012s
I20260709 12:31:20.665244 20197 raft_consensus.cc:359] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.666090 20197 raft_consensus.cc:385] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.666395 20197 raft_consensus.cc:740] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.667335 20197 consensus_queue.cc:260] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.668499 20198 raft_consensus.cc:359] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.669744 20198 raft_consensus.cc:385] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.670135 20198 raft_consensus.cc:740] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.670607 20197 ts_tablet_manager.cc:1434] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.007s
I20260709 12:31:20.671295 20198 consensus_queue.cc:260] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.671978 20197 tablet_bootstrap.cc:492] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e: Bootstrap starting.
I20260709 12:31:20.672523 20199 tablet_bootstrap.cc:654] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.674753 20198 ts_tablet_manager.cc:1434] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.005s
I20260709 12:31:20.676129 20198 tablet_bootstrap.cc:492] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b: Bootstrap starting.
I20260709 12:31:20.680995 20197 tablet_bootstrap.cc:654] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.689869 20198 tablet_bootstrap.cc:654] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.704528 20197 tablet_bootstrap.cc:492] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e: No bootstrap required, opened a new log
I20260709 12:31:20.705322 20197 ts_tablet_manager.cc:1403] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e: Time spent bootstrapping tablet: real 0.034s	user 0.019s	sys 0.013s
I20260709 12:31:20.705600 20198 tablet_bootstrap.cc:492] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b: No bootstrap required, opened a new log
I20260709 12:31:20.706358 20198 ts_tablet_manager.cc:1403] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b: Time spent bootstrapping tablet: real 0.030s	user 0.008s	sys 0.003s
I20260709 12:31:20.708691 20197 raft_consensus.cc:359] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.709637 20197 raft_consensus.cc:385] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.709990 20197 raft_consensus.cc:740] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.710381 20198 raft_consensus.cc:359] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.710961 20197 consensus_queue.cc:260] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.711392 20198 raft_consensus.cc:385] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.712003 20198 raft_consensus.cc:740] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.713569 20199 tablet_bootstrap.cc:492] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea: No bootstrap required, opened a new log
I20260709 12:31:20.713217 20198 consensus_queue.cc:260] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.713588 20197 ts_tablet_manager.cc:1434] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.005s
I20260709 12:31:20.714262 20199 ts_tablet_manager.cc:1403] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.058s	user 0.013s	sys 0.033s
I20260709 12:31:20.715818 20197 tablet_bootstrap.cc:492] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e: Bootstrap starting.
I20260709 12:31:20.717741 20199 raft_consensus.cc:359] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.718540 20199 raft_consensus.cc:385] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.718865 20199 raft_consensus.cc:740] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.719810 20199 consensus_queue.cc:260] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.723646 20198 ts_tablet_manager.cc:1434] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b: Time spent starting tablet: real 0.017s	user 0.010s	sys 0.000s
I20260709 12:31:20.724016 20199 ts_tablet_manager.cc:1434] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:31:20.725064 20199 tablet_bootstrap.cc:492] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:20.725356 20197 tablet_bootstrap.cc:654] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.727546 20198 tablet_bootstrap.cc:492] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b: Bootstrap starting.
I20260709 12:31:20.731850 20199 tablet_bootstrap.cc:654] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.736224 20198 tablet_bootstrap.cc:654] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.741850 20197 tablet_bootstrap.cc:492] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e: No bootstrap required, opened a new log
I20260709 12:31:20.742419 20197 ts_tablet_manager.cc:1403] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e: Time spent bootstrapping tablet: real 0.027s	user 0.014s	sys 0.003s
I20260709 12:31:20.742817 20198 tablet_bootstrap.cc:492] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b: No bootstrap required, opened a new log
I20260709 12:31:20.743379 20198 ts_tablet_manager.cc:1403] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b: Time spent bootstrapping tablet: real 0.016s	user 0.008s	sys 0.004s
I20260709 12:31:20.745424 20197 raft_consensus.cc:359] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.746240 20197 raft_consensus.cc:385] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.746567 20197 raft_consensus.cc:740] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.746851 20199 tablet_bootstrap.cc:492] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea: No bootstrap required, opened a new log
I20260709 12:31:20.746421 20198 raft_consensus.cc:359] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.747339 20198 raft_consensus.cc:385] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.747442 20199 ts_tablet_manager.cc:1403] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.023s	user 0.009s	sys 0.010s
I20260709 12:31:20.747682 20198 raft_consensus.cc:740] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.747603 20197 consensus_queue.cc:260] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.748653 20198 consensus_queue.cc:260] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.750128 20197 ts_tablet_manager.cc:1434] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.003s
I20260709 12:31:20.750617 20199 raft_consensus.cc:359] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.751216 20197 tablet_bootstrap.cc:492] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e: Bootstrap starting.
I20260709 12:31:20.751402 20199 raft_consensus.cc:385] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.751780 20199 raft_consensus.cc:740] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.752646 20199 consensus_queue.cc:260] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.754927 20199 ts_tablet_manager.cc:1434] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:31:20.755939 20199 tablet_bootstrap.cc:492] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:20.756613 20198 ts_tablet_manager.cc:1434] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.004s
I20260709 12:31:20.757793 20198 tablet_bootstrap.cc:492] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b: Bootstrap starting.
I20260709 12:31:20.758663 20197 tablet_bootstrap.cc:654] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.762631 20199 tablet_bootstrap.cc:654] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.769905 20198 tablet_bootstrap.cc:654] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b: Neither blocks nor log segments found. Creating new log.
W20260709 12:31:20.769829 19954 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:31:20.778167 20197 tablet_bootstrap.cc:492] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e: No bootstrap required, opened a new log
I20260709 12:31:20.778748 20198 tablet_bootstrap.cc:492] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b: No bootstrap required, opened a new log
I20260709 12:31:20.778827 20197 ts_tablet_manager.cc:1403] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e: Time spent bootstrapping tablet: real 0.028s	user 0.018s	sys 0.005s
I20260709 12:31:20.779572 20198 ts_tablet_manager.cc:1403] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.004s
I20260709 12:31:20.783095 20197 raft_consensus.cc:359] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.783283 20198 raft_consensus.cc:359] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.784034 20197 raft_consensus.cc:385] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.784092 20198 raft_consensus.cc:385] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.784549 20197 raft_consensus.cc:740] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.784613 20198 raft_consensus.cc:740] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.785709 20198 consensus_queue.cc:260] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.786033 20197 consensus_queue.cc:260] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.789250 20198 ts_tablet_manager.cc:1434] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
I20260709 12:31:20.790617 20198 tablet_bootstrap.cc:492] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b: Bootstrap starting.
I20260709 12:31:20.790660 20197 ts_tablet_manager.cc:1434] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:31:20.792140 20197 tablet_bootstrap.cc:492] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e: Bootstrap starting.
I20260709 12:31:20.799734 20198 tablet_bootstrap.cc:654] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.799734 20197 tablet_bootstrap.cc:654] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.812502 20197 tablet_bootstrap.cc:492] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e: No bootstrap required, opened a new log
I20260709 12:31:20.812505 20198 tablet_bootstrap.cc:492] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b: No bootstrap required, opened a new log
I20260709 12:31:20.816040 20199 tablet_bootstrap.cc:492] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: No bootstrap required, opened a new log
I20260709 12:31:20.821095 20197 ts_tablet_manager.cc:1403] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e: Time spent bootstrapping tablet: real 0.029s	user 0.013s	sys 0.006s
I20260709 12:31:20.821208 20198 ts_tablet_manager.cc:1403] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b: Time spent bootstrapping tablet: real 0.031s	user 0.022s	sys 0.007s
I20260709 12:31:20.821379 20199 ts_tablet_manager.cc:1403] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.066s	user 0.034s	sys 0.031s
I20260709 12:31:20.824632 20199 raft_consensus.cc:359] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.825224 20199 raft_consensus.cc:385] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.825203 20198 raft_consensus.cc:359] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.825874 20199 raft_consensus.cc:740] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.825488 20197 raft_consensus.cc:359] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.826083 20198 raft_consensus.cc:385] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.826542 20197 raft_consensus.cc:385] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.826565 20198 raft_consensus.cc:740] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.827059 20197 raft_consensus.cc:740] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.826938 20199 consensus_queue.cc:260] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.827778 20198 consensus_queue.cc:260] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.828244 20197 consensus_queue.cc:260] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.830687 20198 ts_tablet_manager.cc:1434] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:31:20.830832 20199 ts_tablet_manager.cc:1434] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260709 12:31:20.831275 20197 ts_tablet_manager.cc:1434] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:31:20.831997 20199 tablet_bootstrap.cc:492] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:20.832103 20198 tablet_bootstrap.cc:492] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b: Bootstrap starting.
I20260709 12:31:20.832597 20197 tablet_bootstrap.cc:492] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e: Bootstrap starting.
I20260709 12:31:20.839605 20197 tablet_bootstrap.cc:654] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.839793 20199 tablet_bootstrap.cc:654] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.839612 20198 tablet_bootstrap.cc:654] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.842911 20201 raft_consensus.cc:493] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:20.843562 20201 raft_consensus.cc:515] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.862336 20201 leader_election.cc:290] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:20.863171 20201 raft_consensus.cc:493] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:20.863853 20201 raft_consensus.cc:515] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.865125 20205 raft_consensus.cc:493] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:20.865913 20205 raft_consensus.cc:515] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.867053 20201 leader_election.cc:290] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:20.876495 20205 leader_election.cc:290] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:20.907370 20198 tablet_bootstrap.cc:492] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b: No bootstrap required, opened a new log
I20260709 12:31:20.908164 20198 ts_tablet_manager.cc:1403] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b: Time spent bootstrapping tablet: real 0.076s	user 0.011s	sys 0.001s
I20260709 12:31:20.916163 20198 raft_consensus.cc:359] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:20.917316 20198 raft_consensus.cc:385] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.917682 20198 raft_consensus.cc:740] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.918629 20198 consensus_queue.cc:260] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:20.923002 20199 tablet_bootstrap.cc:492] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea: No bootstrap required, opened a new log
I20260709 12:31:20.923720 20199 ts_tablet_manager.cc:1403] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.092s	user 0.025s	sys 0.063s
I20260709 12:31:20.924773 20226 raft_consensus.cc:493] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:20.925448 20226 raft_consensus.cc:515] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.926391 20198 ts_tablet_manager.cc:1434] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.000s
I20260709 12:31:20.926947 20199 raft_consensus.cc:359] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.927557 20198 tablet_bootstrap.cc:492] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b: Bootstrap starting.
I20260709 12:31:20.927714 20199 raft_consensus.cc:385] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.928048 20199 raft_consensus.cc:740] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.928467 20226 leader_election.cc:290] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:20.929167 20206 raft_consensus.cc:493] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:20.929720 20206 raft_consensus.cc:515] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.935563 20206 leader_election.cc:290] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:20.940798 20198 tablet_bootstrap.cc:654] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:20.942523 20199 consensus_queue.cc:260] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.952848 20198 tablet_bootstrap.cc:492] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b: No bootstrap required, opened a new log
I20260709 12:31:20.953478 20198 ts_tablet_manager.cc:1403] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b: Time spent bootstrapping tablet: real 0.026s	user 0.011s	sys 0.007s
I20260709 12:31:20.956372 20199 ts_tablet_manager.cc:1434] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.032s	user 0.009s	sys 0.000s
I20260709 12:31:20.956382 20198 raft_consensus.cc:359] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.958029 20206 raft_consensus.cc:493] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:20.958710 20206 raft_consensus.cc:515] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.957406 20198 raft_consensus.cc:385] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:20.960996 20198 raft_consensus.cc:740] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Initialized, Role: FOLLOWER
I20260709 12:31:20.962116 20198 consensus_queue.cc:260] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.962669 20199 tablet_bootstrap.cc:492] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:20.963578 20205 raft_consensus.cc:493] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:20.964282 20205 raft_consensus.cc:515] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:20.965066 20206 leader_election.cc:290] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:20.968029 20205 leader_election.cc:290] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:20.974324 20198 ts_tablet_manager.cc:1434] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
I20260709 12:31:20.975881 20198 tablet_bootstrap.cc:492] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b: Bootstrap starting.
I20260709 12:31:20.986670 20198 tablet_bootstrap.cc:654] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:21.001075 20008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea" is_pre_election: true
I20260709 12:31:21.002741 20206 raft_consensus.cc:493] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:21.002641 20007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3177f3e632242e5ba149e10e70ab098" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea" is_pre_election: true
I20260709 12:31:21.003553 20007 raft_consensus.cc:2468] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 0.
I20260709 12:31:21.004856 20199 tablet_bootstrap.cc:654] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea: Neither blocks nor log segments found. Creating new log.
W20260709 12:31:21.005167 20122 leader_election.cc:343] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:21.007099 20122 leader_election.cc:304] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:21.007308 20157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:21.003392 20206 raft_consensus.cc:515] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.008950 20156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "043dc64f584a4a35b66b8c7382e92af4" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:21.009816 20206 leader_election.cc:290] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.010409 20156 raft_consensus.cc:2468] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 0.
I20260709 12:31:21.006273 20197 tablet_bootstrap.cc:492] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e: No bootstrap required, opened a new log
I20260709 12:31:21.012153 20155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa280773cc224cea8253ae8fa2e32e27" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:21.012727 20197 ts_tablet_manager.cc:1403] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e: Time spent bootstrapping tablet: real 0.180s	user 0.061s	sys 0.075s
I20260709 12:31:21.013214 20155 raft_consensus.cc:2468] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 0.
I20260709 12:31:21.014801 20205 raft_consensus.cc:2804] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:21.015432 20205 raft_consensus.cc:493] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:21.015868 20205 raft_consensus.cc:3060] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.016914 20197 raft_consensus.cc:359] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.017876 20197 raft_consensus.cc:385] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:21.018369 20197 raft_consensus.cc:740] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Initialized, Role: FOLLOWER
I20260709 12:31:21.019279 20197 consensus_queue.cc:260] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.020754 20198 tablet_bootstrap.cc:492] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b: No bootstrap required, opened a new log
I20260709 12:31:21.021363 20198 ts_tablet_manager.cc:1403] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b: Time spent bootstrapping tablet: real 0.046s	user 0.018s	sys 0.000s
I20260709 12:31:21.021622 20155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
W20260709 12:31:21.023557 20047 leader_election.cc:343] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:21.024331 20047 leader_election.cc:304] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 
I20260709 12:31:21.024488 20198 raft_consensus.cc:359] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.025311 20198 raft_consensus.cc:385] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:21.025673 20198 raft_consensus.cc:740] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Initialized, Role: FOLLOWER
W20260709 12:31:21.026562 19972 leader_election.cc:343] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:21.027558 20157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "594477803faa4705a2b7bd7eb621295d" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:21.027714 20201 raft_consensus.cc:493] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:21.025481 20226 raft_consensus.cc:2804] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:21.028441 20226 raft_consensus.cc:493] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:21.028228 20201 raft_consensus.cc:515] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.028806 20226 raft_consensus.cc:3060] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.030439 20201 leader_election.cc:290] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.028239 20157 raft_consensus.cc:2468] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 0.
I20260709 12:31:21.032980 20156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3177f3e632242e5ba149e10e70ab098" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:21.033908 19972 leader_election.cc:304] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:21.025311 20047 leader_election.cc:304] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 
I20260709 12:31:21.035235 20206 raft_consensus.cc:2804] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:21.035794 20206 raft_consensus.cc:493] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:21.036263 20206 raft_consensus.cc:3060] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.036417 20201 raft_consensus.cc:2804] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:21.040777 20201 raft_consensus.cc:493] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:21.041448 20201 raft_consensus.cc:3060] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.026835 20198 consensus_queue.cc:260] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.065968 20242 raft_consensus.cc:493] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:21.066589 20242 raft_consensus.cc:515] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.068709 20206 raft_consensus.cc:515] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.071425 20206 leader_election.cc:290] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 election: Requested vote from peers 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.071971 20206 raft_consensus.cc:493] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:21.075599 20242 leader_election.cc:290] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.077207 20206 raft_consensus.cc:515] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.078379 20205 raft_consensus.cc:515] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.081261 20156 raft_consensus.cc:2393] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 971f449846cf4a269f817059278b86ea in current term 1: Already voted for candidate 44efc7828ed44d76bb553655566fd86e in this term.
I20260709 12:31:21.079382 20206 leader_election.cc:290] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.080039 20201 raft_consensus.cc:515] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.085026 20226 raft_consensus.cc:515] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.087105 20201 leader_election.cc:290] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.087388 20226 leader_election.cc:290] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.090958 20157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa280773cc224cea8253ae8fa2e32e27" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:21.091105 20155 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1aa8c0c52f34a8c8b4a2aa87fa57d6e" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:21.091375 20007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3177f3e632242e5ba149e10e70ab098" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:21.092170 20157 raft_consensus.cc:2468] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 0.
I20260709 12:31:21.092790 20007 raft_consensus.cc:3060] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.092922 20205 leader_election.cc:290] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:21.093758 20205 raft_consensus.cc:493] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:21.094251 20156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "594477803faa4705a2b7bd7eb621295d" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:21.096859 20155 raft_consensus.cc:2468] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 0.
I20260709 12:31:21.102851 19972 leader_election.cc:304] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:21.103094 20007 raft_consensus.cc:2468] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 1.
I20260709 12:31:21.104786 20157 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daaf356d34ab43e399517f8b355739a9" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:21.106544 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa280773cc224cea8253ae8fa2e32e27" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:21.113381 20083 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:21.108011 20153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "043dc64f584a4a35b66b8c7382e92af4" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:21.121531 20157 raft_consensus.cc:2468] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 0.
I20260709 12:31:21.121835 20243 raft_consensus.cc:2804] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:21.122138 20153 raft_consensus.cc:3060] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.122648 20243 raft_consensus.cc:493] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:21.123041 20243 raft_consensus.cc:3060] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.123456 20154 raft_consensus.cc:3060] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.124441 20122 leader_election.cc:304] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:21.123174 20081 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "594477803faa4705a2b7bd7eb621295d" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:21.125550 20081 raft_consensus.cc:2468] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 0.
I20260709 12:31:21.109750 20205 raft_consensus.cc:515] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.127216 20156 raft_consensus.cc:3060] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.114638 20082 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3177f3e632242e5ba149e10e70ab098" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:21.128938 20082 raft_consensus.cc:2468] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 0.
I20260709 12:31:21.130978 20243 raft_consensus.cc:515] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.132112 20153 raft_consensus.cc:2468] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 1.
I20260709 12:31:21.132884 20047 leader_election.cc:304] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 
I20260709 12:31:21.133898 20047 leader_election.cc:304] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 
I20260709 12:31:21.134099 19972 leader_election.cc:304] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:21.113327 20244 raft_consensus.cc:493] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:21.107131 20226 raft_consensus.cc:493] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:21.135661 20206 raft_consensus.cc:2804] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:21.136529 20206 raft_consensus.cc:493] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:21.112442 20198 ts_tablet_manager.cc:1434] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b: Time spent starting tablet: real 0.091s	user 0.009s	sys 0.000s
I20260709 12:31:21.137320 20206 raft_consensus.cc:3060] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.140789 20241 raft_consensus.cc:2804] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:21.141590 20241 raft_consensus.cc:697] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Becoming Leader. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:21.142756 20153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa280773cc224cea8253ae8fa2e32e27" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:21.141108 20243 leader_election.cc:290] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 election: Requested vote from peers 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.146059 20199 tablet_bootstrap.cc:492] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea: No bootstrap required, opened a new log
I20260709 12:31:21.136567 20226 raft_consensus.cc:515] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.110626 20240 raft_consensus.cc:493] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:21.136312 20201 raft_consensus.cc:2804] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:21.148270 20081 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "594477803faa4705a2b7bd7eb621295d" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:21.152380 20201 raft_consensus.cc:493] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:21.153425 20201 raft_consensus.cc:3060] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.152794 20081 raft_consensus.cc:3060] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.151171 20240 raft_consensus.cc:515] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.131232 20154 raft_consensus.cc:2468] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 1.
I20260709 12:31:21.158016 20199 ts_tablet_manager.cc:1403] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.196s	user 0.019s	sys 0.038s
I20260709 12:31:21.159488 20226 leader_election.cc:290] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.166702 20240 leader_election.cc:290] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:21.166915 20007 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1aa8c0c52f34a8c8b4a2aa87fa57d6e" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea" is_pre_election: true
I20260709 12:31:21.158649 20156 raft_consensus.cc:2468] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 1.
I20260709 12:31:21.170034 19972 leader_election.cc:304] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:21.170385 20082 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa280773cc224cea8253ae8fa2e32e27" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:21.170959 20243 raft_consensus.cc:2804] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:21.136550 20244 raft_consensus.cc:515] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.171633 20082 raft_consensus.cc:2393] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 971f449846cf4a269f817059278b86ea in current term 1: Already voted for candidate 4ab3d29ae436426ab6d05fef08b4641b in this term.
I20260709 12:31:21.171600 20243 raft_consensus.cc:697] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 1 LEADER]: Becoming Leader. State: Replica: 971f449846cf4a269f817059278b86ea, State: Running, Role: LEADER
I20260709 12:31:21.172490 20083 raft_consensus.cc:2468] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 0.
I20260709 12:31:21.173278 20243 consensus_queue.cc:237] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.170959 20199 raft_consensus.cc:359] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.177831 20199 raft_consensus.cc:385] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:21.178179 20199 raft_consensus.cc:740] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:21.179142 20244 leader_election.cc:290] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.186465 20245 raft_consensus.cc:2804] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:21.186803 20205 leader_election.cc:290] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:21.189101 20002 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa280773cc224cea8253ae8fa2e32e27" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:21.189929 20002 raft_consensus.cc:2393] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4ab3d29ae436426ab6d05fef08b4641b in current term 1: Already voted for candidate 971f449846cf4a269f817059278b86ea in this term.
I20260709 12:31:21.190877 20080 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:21.191044 20003 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "594477803faa4705a2b7bd7eb621295d" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea" is_pre_election: true
I20260709 12:31:21.191660 20080 raft_consensus.cc:2468] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 0.
I20260709 12:31:21.192987 20076 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "594477803faa4705a2b7bd7eb621295d" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:21.181241 20082 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1aa8c0c52f34a8c8b4a2aa87fa57d6e" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:21.181617 20206 raft_consensus.cc:515] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.182121 20197 ts_tablet_manager.cc:1434] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e: Time spent starting tablet: real 0.169s	user 0.038s	sys 0.049s
I20260709 12:31:21.194675 20077 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1aa8c0c52f34a8c8b4a2aa87fa57d6e" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:21.195037 20007 raft_consensus.cc:2393] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44efc7828ed44d76bb553655566fd86e in current term 1: Already voted for candidate 971f449846cf4a269f817059278b86ea in this term.
I20260709 12:31:21.195353 20197 tablet_bootstrap.cc:492] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e: Bootstrap starting.
I20260709 12:31:21.195958 20206 leader_election.cc:290] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 election: Requested vote from peers 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.183187 20083 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa280773cc224cea8253ae8fa2e32e27" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:21.196866 20083 raft_consensus.cc:2393] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 971f449846cf4a269f817059278b86ea in current term 1: Already voted for candidate 4ab3d29ae436426ab6d05fef08b4641b in this term.
I20260709 12:31:21.184950 19971 leader_election.cc:304] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b, 971f449846cf4a269f817059278b86ea; no voters: 44efc7828ed44d76bb553655566fd86e
I20260709 12:31:21.197875 20079 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3177f3e632242e5ba149e10e70ab098" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:21.198463 20079 raft_consensus.cc:2468] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 0.
I20260709 12:31:21.196874 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3177f3e632242e5ba149e10e70ab098" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:21.199440 20078 raft_consensus.cc:3060] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.199558 20001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "043dc64f584a4a35b66b8c7382e92af4" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:21.186501 20008 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea" is_pre_election: true
I20260709 12:31:21.200649 20201 raft_consensus.cc:515] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.186874 20006 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "043dc64f584a4a35b66b8c7382e92af4" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea" is_pre_election: true
I20260709 12:31:21.202955 20201 leader_election.cc:290] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.196240 19999 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea" is_pre_election: true
I20260709 12:31:21.187255 20005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aa280773cc224cea8253ae8fa2e32e27" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea" is_pre_election: true
I20260709 12:31:21.198395 19971 leader_election.cc:304] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b, 971f449846cf4a269f817059278b86ea; no voters: 44efc7828ed44d76bb553655566fd86e
I20260709 12:31:21.205030 20005 raft_consensus.cc:2393] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4ab3d29ae436426ab6d05fef08b4641b in current term 1: Already voted for candidate 971f449846cf4a269f817059278b86ea in this term.
I20260709 12:31:21.191612 20081 raft_consensus.cc:2468] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 1.
I20260709 12:31:21.205863 20156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1aa8c0c52f34a8c8b4a2aa87fa57d6e" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:21.142632 20241 consensus_queue.cc:237] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.206284 19971 leader_election.cc:304] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 971f449846cf4a269f817059278b86ea; no voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b
I20260709 12:31:21.206543 20156 raft_consensus.cc:3060] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.207507 20243 raft_consensus.cc:2764] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:31:21.190912 19904 catalog_manager.cc:5697] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 971f449846cf4a269f817059278b86ea (127.4.65.1). New cstate: current_term: 1 leader_uuid: "971f449846cf4a269f817059278b86ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:21.193085 20120 leader_election.cc:304] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 971f449846cf4a269f817059278b86ea
I20260709 12:31:21.187096 20245 raft_consensus.cc:697] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 LEADER]: Becoming Leader. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
I20260709 12:31:21.194020 20082 raft_consensus.cc:2468] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 0.
I20260709 12:31:21.210621 20120 leader_election.cc:304] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e; no voters: 4ab3d29ae436426ab6d05fef08b4641b, 971f449846cf4a269f817059278b86ea
I20260709 12:31:21.204124 19999 raft_consensus.cc:2468] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 0.
I20260709 12:31:21.197135 20000 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea" is_pre_election: true
I20260709 12:31:21.212656 20243 raft_consensus.cc:2804] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:21.188112 20004 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daaf356d34ab43e399517f8b355739a9" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea" is_pre_election: true
I20260709 12:31:21.212638 20242 raft_consensus.cc:2749] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:31:21.215222 20243 raft_consensus.cc:493] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:21.215739 20243 raft_consensus.cc:3060] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.215842 20122 leader_election.cc:304] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e; no voters: 4ab3d29ae436426ab6d05fef08b4641b, 971f449846cf4a269f817059278b86ea
I20260709 12:31:21.217686 20197 tablet_bootstrap.cc:654] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:21.210389 20245 consensus_queue.cc:237] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.222472 20005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daaf356d34ab43e399517f8b355739a9" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:21.220045 20239 raft_consensus.cc:2804] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:21.225385 20239 raft_consensus.cc:493] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:21.225822 20239 raft_consensus.cc:3060] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:31:21.180338 20047 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.65.3:44645, user_credentials={real_user=slave}} blocked reactor thread for 46512.3us
I20260709 12:31:21.229259 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:21.229426 20082 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1aa8c0c52f34a8c8b4a2aa87fa57d6e" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:21.230012 20082 raft_consensus.cc:3060] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.219831 20078 raft_consensus.cc:2468] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 1.
I20260709 12:31:21.223055 20156 raft_consensus.cc:2468] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 1.
I20260709 12:31:21.234263 20205 raft_consensus.cc:2749] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:21.234779 20153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:21.223263 20240 raft_consensus.cc:2749] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:21.238168 20153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daaf356d34ab43e399517f8b355739a9" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:21.238351 19972 leader_election.cc:304] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:21.238839 20153 raft_consensus.cc:3060] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.239923 20047 leader_election.cc:304] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 971f449846cf4a269f817059278b86ea
I20260709 12:31:21.242697 20201 raft_consensus.cc:2804] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:21.243176 20201 raft_consensus.cc:697] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 LEADER]: Becoming Leader. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
I20260709 12:31:21.244045 20201 consensus_queue.cc:237] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.246795 20239 raft_consensus.cc:515] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.247629 20153 raft_consensus.cc:2468] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 1.
I20260709 12:31:21.179080 20199 consensus_queue.cc:260] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.240563 20206 raft_consensus.cc:2804] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:21.249537 20206 raft_consensus.cc:697] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 1 LEADER]: Becoming Leader. State: Replica: 971f449846cf4a269f817059278b86ea, State: Running, Role: LEADER
I20260709 12:31:21.250267 20197 tablet_bootstrap.cc:492] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e: No bootstrap required, opened a new log
W20260709 12:31:21.249080 20048 leader_election.cc:343] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:21.250900 20048 leader_election.cc:304] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b; no voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea
I20260709 12:31:21.250798 20206 consensus_queue.cc:237] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.250844 20197 ts_tablet_manager.cc:1403] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e: Time spent bootstrapping tablet: real 0.056s	user 0.013s	sys 0.021s
I20260709 12:31:21.252260 20005 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea"
W20260709 12:31:21.253932 20122 leader_election.cc:343] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:21.248047 20154 raft_consensus.cc:2393] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4ab3d29ae436426ab6d05fef08b4641b in current term 1: Already voted for candidate 44efc7828ed44d76bb553655566fd86e in this term.
I20260709 12:31:21.254897 20197 raft_consensus.cc:359] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:21.255582 20197 raft_consensus.cc:385] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:21.255926 20197 raft_consensus.cc:740] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Initialized, Role: FOLLOWER
I20260709 12:31:21.257704 20239 leader_election.cc:290] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:21.257905 20001 raft_consensus.cc:3060] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.258563 20197 consensus_queue.cc:260] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:21.261020 20197 ts_tablet_manager.cc:1434] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:31:21.260885 20201 raft_consensus.cc:2749] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:21.262255 20197 tablet_bootstrap.cc:492] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e: Bootstrap starting.
I20260709 12:31:21.261118 19903 catalog_manager.cc:5697] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e reported cstate change: term changed from 0 to 1, leader changed from <none> to 44efc7828ed44d76bb553655566fd86e (127.4.65.3). New cstate: current_term: 1 leader_uuid: "44efc7828ed44d76bb553655566fd86e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:21.264048 20048 leader_election.cc:304] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:21.238835 20082 raft_consensus.cc:2468] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 1.
I20260709 12:31:21.268049 20197 tablet_bootstrap.cc:654] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:21.268138 20001 raft_consensus.cc:2468] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 1.
I20260709 12:31:21.271023 20199 ts_tablet_manager.cc:1434] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.109s	user 0.007s	sys 0.003s
I20260709 12:31:21.272686 20199 tablet_bootstrap.cc:492] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:21.278002 20201 raft_consensus.cc:2804] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:21.278656 20201 raft_consensus.cc:493] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:21.279110 20201 raft_consensus.cc:3060] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.278967 19906 catalog_manager.cc:5697] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:31:21.281148 20047 leader_election.cc:343] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:21.282090 20047 leader_election.cc:304] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 
I20260709 12:31:21.284578 20245 raft_consensus.cc:2804] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:21.285094 20245 raft_consensus.cc:697] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 LEADER]: Becoming Leader. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
W20260709 12:31:21.285590 20048 leader_election.cc:343] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:21.285975 20245 consensus_queue.cc:237] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.286311 20048 leader_election.cc:304] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b; no voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea
I20260709 12:31:21.287392 20244 raft_consensus.cc:3060] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:31:21.287992 20048 leader_election.cc:343] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:31:21.289352 20048 leader_election.cc:343] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:21.294670 20243 raft_consensus.cc:515] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.295674 20082 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:21.297994 20244 raft_consensus.cc:2749] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:21.299441 20199 tablet_bootstrap.cc:654] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:21.302444 20197 tablet_bootstrap.cc:492] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e: No bootstrap required, opened a new log
I20260709 12:31:21.302492 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:21.303093 20197 ts_tablet_manager.cc:1403] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e: Time spent bootstrapping tablet: real 0.041s	user 0.009s	sys 0.007s
I20260709 12:31:21.303747 20243 leader_election.cc:290] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 election: Requested vote from peers 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.305186 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:21.305876 20197 raft_consensus.cc:359] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.306589 20197 raft_consensus.cc:385] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:21.306852 20197 raft_consensus.cc:740] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Initialized, Role: FOLLOWER
W20260709 12:31:21.307144 19972 leader_election.cc:343] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:21.307540 20197 consensus_queue.cc:260] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.310111 20197 ts_tablet_manager.cc:1434] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 12:31:21.310454 20082 raft_consensus.cc:2468] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 1.
I20260709 12:31:21.311292 20197 tablet_bootstrap.cc:492] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e: Bootstrap starting.
I20260709 12:31:21.312623 20120 leader_election.cc:304] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 971f449846cf4a269f817059278b86ea
I20260709 12:31:21.308739 20201 raft_consensus.cc:515] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.317925 20241 raft_consensus.cc:2804] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:21.318624 20078 raft_consensus.cc:2393] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 971f449846cf4a269f817059278b86ea in current term 1: Already voted for candidate 4ab3d29ae436426ab6d05fef08b4641b in this term.
I20260709 12:31:21.318773 20241 raft_consensus.cc:697] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Becoming Leader. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:21.319825 20241 consensus_queue.cc:237] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.321396 19971 leader_election.cc:304] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 971f449846cf4a269f817059278b86ea; no voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b
I20260709 12:31:21.321575 20001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:21.322057 20201 leader_election.cc:290] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.322386 20001 raft_consensus.cc:2393] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4ab3d29ae436426ab6d05fef08b4641b in current term 1: Already voted for candidate 971f449846cf4a269f817059278b86ea in this term.
I20260709 12:31:21.322922 20243 raft_consensus.cc:2749] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:31:21.323695 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e"
W20260709 12:31:21.325389 20047 leader_election.cc:343] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:21.325937 20047 leader_election.cc:304] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b; no voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea
I20260709 12:31:21.328928 20197 tablet_bootstrap.cc:654] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:21.328877 20201 raft_consensus.cc:2749] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:31:21.330618 20239 raft_consensus.cc:493] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:21.331184 20239 raft_consensus.cc:515] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:21.349051 20197 tablet_bootstrap.cc:492] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e: No bootstrap required, opened a new log
I20260709 12:31:21.349615 20197 ts_tablet_manager.cc:1403] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e: Time spent bootstrapping tablet: real 0.039s	user 0.018s	sys 0.005s
I20260709 12:31:21.350481 20001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8b4cb1a18524c3d8189e535e8666620" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea" is_pre_election: true
W20260709 12:31:21.351927 20122 leader_election.cc:343] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:21.352267 20239 leader_election.cc:290] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:21.352784 20197 raft_consensus.cc:359] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.353494 20197 raft_consensus.cc:385] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:21.353349 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8b4cb1a18524c3d8189e535e8666620" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:21.353833 20197 raft_consensus.cc:740] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Initialized, Role: FOLLOWER
I20260709 12:31:21.354128 20078 raft_consensus.cc:2468] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 0.
I20260709 12:31:21.354547 20197 consensus_queue.cc:260] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.355700 20120 leader_election.cc:304] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 971f449846cf4a269f817059278b86ea
I20260709 12:31:21.356076 20199 tablet_bootstrap.cc:492] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea: No bootstrap required, opened a new log
I20260709 12:31:21.356674 20199 ts_tablet_manager.cc:1403] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.084s	user 0.005s	sys 0.025s
I20260709 12:31:21.357007 20197 ts_tablet_manager.cc:1434] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.002s
I20260709 12:31:21.357028 20239 raft_consensus.cc:2804] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:21.357601 20239 raft_consensus.cc:493] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:21.357942 20239 raft_consensus.cc:3060] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.359278 20199 raft_consensus.cc:359] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:21.359973 20199 raft_consensus.cc:385] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:21.360231 20199 raft_consensus.cc:740] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:21.361209 20199 consensus_queue.cc:260] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:21.363557 20199 ts_tablet_manager.cc:1434] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.001s
I20260709 12:31:21.364709 20199 tablet_bootstrap.cc:492] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:21.364598 20239 raft_consensus.cc:515] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:21.367395 20001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8b4cb1a18524c3d8189e535e8666620" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:21.367601 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8b4cb1a18524c3d8189e535e8666620" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:21.368044 20001 raft_consensus.cc:3060] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.368155 20078 raft_consensus.cc:3060] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.369446 19903 catalog_manager.cc:5697] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e reported cstate change: term changed from 0 to 1, leader changed from <none> to 44efc7828ed44d76bb553655566fd86e (127.4.65.3). New cstate: current_term: 1 leader_uuid: "44efc7828ed44d76bb553655566fd86e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:21.372248 20199 tablet_bootstrap.cc:654] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:21.376605 20239 leader_election.cc:290] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:21.378233 20078 raft_consensus.cc:2468] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 1.
I20260709 12:31:21.378995 20001 raft_consensus.cc:2468] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 1.
I20260709 12:31:21.379835 20120 leader_election.cc:304] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 
I20260709 12:31:21.380884 20239 raft_consensus.cc:2804] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:21.381637 20239 raft_consensus.cc:697] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Becoming Leader. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:21.382572 20239 consensus_queue.cc:237] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:21.384469 20199 tablet_bootstrap.cc:492] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea: No bootstrap required, opened a new log
I20260709 12:31:21.383885 19904 catalog_manager.cc:5697] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 971f449846cf4a269f817059278b86ea (127.4.65.1). New cstate: current_term: 1 leader_uuid: "971f449846cf4a269f817059278b86ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:21.385079 20199 ts_tablet_manager.cc:1403] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.021s	user 0.010s	sys 0.009s
I20260709 12:31:21.388046 20199 raft_consensus.cc:359] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.388777 20199 raft_consensus.cc:385] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:21.389039 20199 raft_consensus.cc:740] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:21.389813 20199 consensus_queue.cc:260] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.392122 20199 ts_tablet_manager.cc:1434] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:31:21.393253 20199 tablet_bootstrap.cc:492] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:21.399587 20199 tablet_bootstrap.cc:654] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:21.405223 20199 tablet_bootstrap.cc:492] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea: No bootstrap required, opened a new log
I20260709 12:31:21.404506 19906 catalog_manager.cc:5697] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:21.405984 20199 ts_tablet_manager.cc:1403] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.013s	user 0.010s	sys 0.000s
I20260709 12:31:21.406838 19906 catalog_manager.cc:5697] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2). New cstate: current_term: 1 leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:21.408913 20199 raft_consensus.cc:359] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.410122 20199 raft_consensus.cc:385] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:21.410631 20199 raft_consensus.cc:740] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:21.411469 20199 consensus_queue.cc:260] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.413282 20199 ts_tablet_manager.cc:1434] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:31:21.414280 20199 tablet_bootstrap.cc:492] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:21.420526 20199 tablet_bootstrap.cc:654] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:21.423290 19903 catalog_manager.cc:5697] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e reported cstate change: term changed from 0 to 1, leader changed from <none> to 44efc7828ed44d76bb553655566fd86e (127.4.65.3). New cstate: current_term: 1 leader_uuid: "44efc7828ed44d76bb553655566fd86e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:21.438105 20199 tablet_bootstrap.cc:492] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea: No bootstrap required, opened a new log
I20260709 12:31:21.438719 20199 ts_tablet_manager.cc:1403] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.025s	user 0.013s	sys 0.009s
I20260709 12:31:21.440852 20199 raft_consensus.cc:359] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.441322 20199 raft_consensus.cc:385] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:21.441507 20199 raft_consensus.cc:740] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:21.442034 20199 consensus_queue.cc:260] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.443789 20199 ts_tablet_manager.cc:1434] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.002s
I20260709 12:31:21.486732 20243 raft_consensus.cc:493] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:21.487242 20243 raft_consensus.cc:515] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.489581 20243 leader_election.cc:290] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.490278 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:21.490767 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:21.491009 20078 raft_consensus.cc:2468] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 0.
I20260709 12:31:21.491458 20154 raft_consensus.cc:2468] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 0.
I20260709 12:31:21.492167 19971 leader_election.cc:304] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:21.493046 20243 raft_consensus.cc:2804] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:21.493475 20243 raft_consensus.cc:493] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:21.493759 20243 raft_consensus.cc:3060] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.498871 20243 raft_consensus.cc:515] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.500792 20243 leader_election.cc:290] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 election: Requested vote from peers 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.501605 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:21.501855 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:21.502286 20078 raft_consensus.cc:3060] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.502380 20154 raft_consensus.cc:3060] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.509995 20154 raft_consensus.cc:2468] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 1.
I20260709 12:31:21.509995 20078 raft_consensus.cc:2468] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 1.
I20260709 12:31:21.511544 19971 leader_election.cc:304] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:21.512323 20243 raft_consensus.cc:2804] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:21.512888 20243 raft_consensus.cc:697] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 1 LEADER]: Becoming Leader. State: Replica: 971f449846cf4a269f817059278b86ea, State: Running, Role: LEADER
I20260709 12:31:21.513696 20243 consensus_queue.cc:237] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.519371 20243 raft_consensus.cc:493] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:21.519878 20243 raft_consensus.cc:515] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.522197 20243 leader_election.cc:290] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:21.521945 19903 catalog_manager.cc:5697] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 971f449846cf4a269f817059278b86ea (127.4.65.1). New cstate: current_term: 1 leader_uuid: "971f449846cf4a269f817059278b86ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:21.523540 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daaf356d34ab43e399517f8b355739a9" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:21.523752 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daaf356d34ab43e399517f8b355739a9" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:21.525059 20154 raft_consensus.cc:2393] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 971f449846cf4a269f817059278b86ea in current term 1: Already voted for candidate 4ab3d29ae436426ab6d05fef08b4641b in this term.
I20260709 12:31:21.526329 19972 leader_election.cc:304] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 971f449846cf4a269f817059278b86ea; no voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b
I20260709 12:31:21.527180 20243 raft_consensus.cc:3060] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.534294 20243 raft_consensus.cc:2749] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:21.597553 20239 consensus_queue.cc:1048] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:21.610601 20239 consensus_queue.cc:1048] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:21.615537 20243 consensus_queue.cc:1048] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:21.632084 20206 consensus_queue.cc:1048] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:21.681380 20206 consensus_queue.cc:1048] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:21.696632 20242 consensus_queue.cc:1048] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:21.721961 20239 raft_consensus.cc:493] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:21.722455 20239 raft_consensus.cc:515] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:21.724756 20239 leader_election.cc:290] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:21.725718 20001 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "971f449846cf4a269f817059278b86ea" is_pre_election: true
I20260709 12:31:21.725754 20201 consensus_queue.cc:1048] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:21.726300 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:21.726606 20001 raft_consensus.cc:2393] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44efc7828ed44d76bb553655566fd86e in current term 1: Already voted for candidate 971f449846cf4a269f817059278b86ea in this term.
I20260709 12:31:21.727164 20078 raft_consensus.cc:2393] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44efc7828ed44d76bb553655566fd86e in current term 1: Already voted for candidate 4ab3d29ae436426ab6d05fef08b4641b in this term.
I20260709 12:31:21.728585 20120 leader_election.cc:304] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e; no voters: 4ab3d29ae436426ab6d05fef08b4641b, 971f449846cf4a269f817059278b86ea
I20260709 12:31:21.729512 20240 raft_consensus.cc:3060] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.734544 20239 consensus_queue.cc:1048] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:21.742466 20201 consensus_queue.cc:1048] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:21.751730 20240 raft_consensus.cc:2749] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:21.765359 20001 raft_consensus.cc:3060] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:21.769796 20244 consensus_queue.cc:1048] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:21.775703 20239 consensus_queue.cc:1048] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:21.787010 20245 consensus_queue.cc:1048] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:31:21.795006 19954 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:31:21.878225 20245 consensus_queue.cc:1048] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:21.890726 20239 consensus_queue.cc:1048] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:21.892271 20282 consensus_queue.cc:1048] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:21.910770 20240 consensus_queue.cc:1048] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:22.031102 20242 consensus_queue.cc:1048] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:22.042165 20243 consensus_queue.cc:1048] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:31:22.802449 19954 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:31:23.435776 20242 raft_consensus.cc:493] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:23.437863 20242 raft_consensus.cc:515] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:23.444516 20242 leader_election.cc:290] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:23.446465 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:23.446171 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:23.447870 20154 raft_consensus.cc:2468] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 1.
I20260709 12:31:23.447994 20078 raft_consensus.cc:2468] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 1.
I20260709 12:31:23.449888 19972 leader_election.cc:304] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:23.450958 20242 raft_consensus.cc:2804] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:31:23.451368 20242 raft_consensus.cc:493] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:23.451715 20242 raft_consensus.cc:3060] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:23.461608 20242 raft_consensus.cc:515] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:23.463658 20242 leader_election.cc:290] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 2 election: Requested vote from peers 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:23.464854 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:23.464808 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "971f449846cf4a269f817059278b86ea" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:23.465685 20154 raft_consensus.cc:3060] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:23.465728 20078 raft_consensus.cc:3060] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:23.475016 20078 raft_consensus.cc:2468] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 2.
I20260709 12:31:23.475232 20154 raft_consensus.cc:2468] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 971f449846cf4a269f817059278b86ea in term 2.
I20260709 12:31:23.476974 19971 leader_election.cc:304] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:23.478238 20242 raft_consensus.cc:2804] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:23.478935 20242 raft_consensus.cc:697] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 2 LEADER]: Becoming Leader. State: Replica: 971f449846cf4a269f817059278b86ea, State: Running, Role: LEADER
I20260709 12:31:23.480974 20242 consensus_queue.cc:237] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:23.491310 19906 catalog_manager.cc:5697] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea reported cstate change: term changed from 0 to 2, leader changed from <none> to 971f449846cf4a269f817059278b86ea (127.4.65.1). New cstate: current_term: 2 leader_uuid: "971f449846cf4a269f817059278b86ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:23.626921  4356 ts_manager.cc:295] Set tserver state for 971f449846cf4a269f817059278b86ea to MAINTENANCE_MODE
I20260709 12:31:23.627882  4356 tablet_server.cc:179] TabletServer@127.4.65.1:0 shutting down...
I20260709 12:31:23.662748  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:31:23.663858  4356 tablet_replica.cc:333] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea: stopping tablet replica
I20260709 12:31:23.664937  4356 raft_consensus.cc:2243] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:23.665688  4356 raft_consensus.cc:2272] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:23.670177  4356 tablet_replica.cc:333] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea: stopping tablet replica
I20260709 12:31:23.671025  4356 raft_consensus.cc:2243] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:23.671594  4356 raft_consensus.cc:2272] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:23.674302  4356 tablet_replica.cc:333] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea: stopping tablet replica
I20260709 12:31:23.675004  4356 raft_consensus.cc:2243] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:23.676124  4356 raft_consensus.cc:2272] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:23.678830  4356 tablet_replica.cc:333] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea: stopping tablet replica
I20260709 12:31:23.679445  4356 raft_consensus.cc:2243] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:23.680032  4356 raft_consensus.cc:2272] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:23.682760  4356 tablet_replica.cc:333] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea: stopping tablet replica
I20260709 12:31:23.683465  4356 raft_consensus.cc:2243] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:23.684970  4356 raft_consensus.cc:2272] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:23.687739  4356 tablet_replica.cc:333] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea: stopping tablet replica
I20260709 12:31:23.688467  4356 raft_consensus.cc:2243] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:23.689150  4356 raft_consensus.cc:2272] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:23.691717  4356 tablet_replica.cc:333] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea: stopping tablet replica
I20260709 12:31:23.692293  4356 raft_consensus.cc:2243] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:23.693547  4356 raft_consensus.cc:2272] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:23.696242  4356 tablet_replica.cc:333] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea: stopping tablet replica
I20260709 12:31:23.696880  4356 raft_consensus.cc:2243] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:23.697511  4356 raft_consensus.cc:2272] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:23.700439  4356 tablet_replica.cc:333] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea: stopping tablet replica
I20260709 12:31:23.700994  4356 raft_consensus.cc:2243] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:23.701819  4356 raft_consensus.cc:2272] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:23.705080  4356 tablet_replica.cc:333] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: stopping tablet replica
I20260709 12:31:23.705597  4356 raft_consensus.cc:2243] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:23.706171  4356 pending_rounds.cc:70] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: Trying to abort 1 pending ops.
I20260709 12:31:23.706338  4356 pending_rounds.cc:77] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7305626761148325888 op_type: NO_OP noop_request { }
I20260709 12:31:23.706600  4356 raft_consensus.cc:2889] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:31:23.706863  4356 raft_consensus.cc:2272] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:23.738610 20048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20260709 12:31:23.738611 20122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111)
W20260709 12:31:23.743908 20122 consensus_peers.cc:597] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:23.743938 20048 consensus_peers.cc:597] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:23.745869  4356 tablet_server.cc:196] TabletServer@127.4.65.1:0 shutdown complete.
W20260709 12:31:23.803894 19954 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:31:23.838254 20048 consensus_peers.cc:597] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:23.840309 20048 consensus_peers.cc:597] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:24.027894 20122 consensus_peers.cc:597] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:24.054170 20122 consensus_peers.cc:597] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:24.594385 20303 raft_consensus.cc:493] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 971f449846cf4a269f817059278b86ea)
I20260709 12:31:24.595182 20303 raft_consensus.cc:515] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:24.598049 20303 leader_election.cc:290] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:24.599356 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
W20260709 12:31:24.601954 20048 leader_election.cc:336] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111)
I20260709 12:31:24.602485 20048 leader_election.cc:304] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b; no voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea
I20260709 12:31:24.603312 20303 raft_consensus.cc:2749] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:24.702966 20240 raft_consensus.cc:493] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 971f449846cf4a269f817059278b86ea)
I20260709 12:31:24.703635 20240 raft_consensus.cc:515] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:24.705507 20240 leader_election.cc:290] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:24.706676 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "594477803faa4705a2b7bd7eb621295d" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
W20260709 12:31:24.709019 20122 leader_election.cc:336] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111)
I20260709 12:31:24.709519 20122 leader_election.cc:304] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e; no voters: 4ab3d29ae436426ab6d05fef08b4641b, 971f449846cf4a269f817059278b86ea
I20260709 12:31:24.710135 20240 raft_consensus.cc:2749] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:31:24.771899 20303 raft_consensus.cc:493] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 971f449846cf4a269f817059278b86ea)
I20260709 12:31:24.772580 20303 raft_consensus.cc:515] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:24.774247 20303 leader_election.cc:290] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:24.775389 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1aa8c0c52f34a8c8b4a2aa87fa57d6e" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
W20260709 12:31:24.778049 20048 leader_election.cc:336] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111)
I20260709 12:31:24.778435 20048 leader_election.cc:304] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b; no voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea
I20260709 12:31:24.779078 20303 raft_consensus.cc:2749] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:31:24.805341 19954 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:31:24.827203 20303 raft_consensus.cc:493] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 971f449846cf4a269f817059278b86ea)
I20260709 12:31:24.827798 20303 raft_consensus.cc:515] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:24.829586 20303 leader_election.cc:290] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:24.830765 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "594477803faa4705a2b7bd7eb621295d" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:24.831370 20154 raft_consensus.cc:2468] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 1.
I20260709 12:31:24.832618 20047 leader_election.cc:304] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 
I20260709 12:31:24.833418 20303 raft_consensus.cc:2804] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:31:24.833834 20303 raft_consensus.cc:493] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 971f449846cf4a269f817059278b86ea)
W20260709 12:31:24.834147 20048 leader_election.cc:336] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111)
I20260709 12:31:24.834308 20303 raft_consensus.cc:3060] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:24.842267 20303 raft_consensus.cc:515] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:24.844075 20303 leader_election.cc:290] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:24.845602 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "594477803faa4705a2b7bd7eb621295d" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:24.846503 20154 raft_consensus.cc:3060] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:31:24.849457 20048 leader_election.cc:336] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111)
I20260709 12:31:24.855036 20154 raft_consensus.cc:2468] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 2.
I20260709 12:31:24.856832 20047 leader_election.cc:304] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 971f449846cf4a269f817059278b86ea
I20260709 12:31:24.857695 20303 raft_consensus.cc:2804] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:24.858258 20303 raft_consensus.cc:697] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Becoming Leader. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
I20260709 12:31:24.859436 20303 consensus_queue.cc:237] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:24.869117 19906 catalog_manager.cc:5697] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b reported cstate change: term changed from 1 to 2, leader changed from 971f449846cf4a269f817059278b86ea (127.4.65.1) to 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:24.945035 20240 raft_consensus.cc:493] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 971f449846cf4a269f817059278b86ea)
I20260709 12:31:24.945969 20240 raft_consensus.cc:515] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:24.948062 20240 leader_election.cc:290] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:24.949579 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:24.950418 20078 raft_consensus.cc:2468] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 1.
I20260709 12:31:24.951568 20120 leader_election.cc:304] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 
I20260709 12:31:24.952399 20240 raft_consensus.cc:2804] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:31:24.952490 20122 leader_election.cc:336] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111)
I20260709 12:31:24.952834 20240 raft_consensus.cc:493] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 971f449846cf4a269f817059278b86ea)
I20260709 12:31:24.953248 20240 raft_consensus.cc:3060] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:24.959837 20240 raft_consensus.cc:515] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:24.961802 20240 leader_election.cc:290] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:24.963093 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:24.963696 20078 raft_consensus.cc:3060] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:31:24.965924 20122 leader_election.cc:336] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111)
I20260709 12:31:24.969730 20078 raft_consensus.cc:2468] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 2.
I20260709 12:31:24.970916 20120 leader_election.cc:304] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 971f449846cf4a269f817059278b86ea
I20260709 12:31:24.971715 20240 raft_consensus.cc:2804] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:24.972177 20240 raft_consensus.cc:697] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 2 LEADER]: Becoming Leader. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:24.972983 20240 consensus_queue.cc:237] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:24.981729 19906 catalog_manager.cc:5697] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e reported cstate change: term changed from 1 to 2, leader changed from 971f449846cf4a269f817059278b86ea (127.4.65.1) to 44efc7828ed44d76bb553655566fd86e (127.4.65.3). New cstate: current_term: 2 leader_uuid: "44efc7828ed44d76bb553655566fd86e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:25.102883 20240 raft_consensus.cc:493] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 971f449846cf4a269f817059278b86ea)
I20260709 12:31:25.103542 20240 raft_consensus.cc:515] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:25.106143 20240 leader_election.cc:290] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:25.107329 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1aa8c0c52f34a8c8b4a2aa87fa57d6e" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b" is_pre_election: true
I20260709 12:31:25.108016 20078 raft_consensus.cc:2468] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 1.
I20260709 12:31:25.109220 20120 leader_election.cc:304] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 
I20260709 12:31:25.109948 20240 raft_consensus.cc:2804] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:31:25.110354 20122 leader_election.cc:336] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111)
I20260709 12:31:25.110405 20240 raft_consensus.cc:493] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 971f449846cf4a269f817059278b86ea)
I20260709 12:31:25.111054 20240 raft_consensus.cc:3060] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:25.117340 20240 raft_consensus.cc:515] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:25.118932 20240 leader_election.cc:290] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:25.120463 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1aa8c0c52f34a8c8b4a2aa87fa57d6e" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:25.121300 20078 raft_consensus.cc:3060] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:31:25.123539 20122 leader_election.cc:336] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111)
I20260709 12:31:25.130043 20078 raft_consensus.cc:2468] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 2.
I20260709 12:31:25.135232 20120 leader_election.cc:304] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 971f449846cf4a269f817059278b86ea
I20260709 12:31:25.136062 20240 raft_consensus.cc:2804] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:25.136736 20240 raft_consensus.cc:697] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 2 LEADER]: Becoming Leader. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:25.137636 20240 consensus_queue.cc:237] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:25.143710 20303 raft_consensus.cc:493] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:25.144277 20303 raft_consensus.cc:515] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:25.145447 19906 catalog_manager.cc:5697] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e reported cstate change: term changed from 1 to 2, leader changed from 971f449846cf4a269f817059278b86ea (127.4.65.1) to 44efc7828ed44d76bb553655566fd86e (127.4.65.3). New cstate: current_term: 2 leader_uuid: "44efc7828ed44d76bb553655566fd86e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:25.146396 20303 leader_election.cc:290] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:25.148030 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e" is_pre_election: true
I20260709 12:31:25.148701 20154 raft_consensus.cc:2468] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 2.
I20260709 12:31:25.149709 20047 leader_election.cc:304] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 
I20260709 12:31:25.150425 20303 raft_consensus.cc:2804] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:31:25.150836 20303 raft_consensus.cc:493] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:25.151221 20303 raft_consensus.cc:3060] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:31:25.151353 20048 leader_election.cc:336] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111)
I20260709 12:31:25.157337 20303 raft_consensus.cc:515] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:25.159006 20303 leader_election.cc:290] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 3 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:25.160604 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e2f9bc36a87410f9f14a842ef3785ba" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:25.161489 20154 raft_consensus.cc:3060] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:31:25.163471 20048 leader_election.cc:336] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111)
I20260709 12:31:25.167440 20154 raft_consensus.cc:2468] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 3.
I20260709 12:31:25.168622 20047 leader_election.cc:304] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 971f449846cf4a269f817059278b86ea
I20260709 12:31:25.169400 20303 raft_consensus.cc:2804] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:31:25.172941 20303 raft_consensus.cc:697] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 3 LEADER]: Becoming Leader. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
I20260709 12:31:25.173859 20303 consensus_queue.cc:237] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:25.187891 19906 catalog_manager.cc:5697] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b reported cstate change: term changed from 2 to 3, leader changed from 971f449846cf4a269f817059278b86ea (127.4.65.1) to 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2). New cstate: current_term: 3 leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:25.291774 20154 raft_consensus.cc:1275] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Refusing update from remote peer 4ab3d29ae436426ab6d05fef08b4641b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:25.293342 20303 consensus_queue.cc:1048] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:31:25.310206 20048 consensus_peers.cc:597] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:25.503804 20078 raft_consensus.cc:1275] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Refusing update from remote peer 44efc7828ed44d76bb553655566fd86e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:25.505275 20240 consensus_queue.cc:1048] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:31:25.519021 20122 consensus_peers.cc:597] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:25.570685 20154 raft_consensus.cc:1275] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Refusing update from remote peer 4ab3d29ae436426ab6d05fef08b4641b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch)
I20260709 12:31:25.572184 20303 consensus_queue.cc:1048] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:31:25.586690 20048 consensus_peers.cc:597] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:25.609707 20078 raft_consensus.cc:1275] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Refusing update from remote peer 44efc7828ed44d76bb553655566fd86e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:31:25.610562 20122 consensus_peers.cc:597] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:25.611470 20240 consensus_queue.cc:1048] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:31:25.806398 19954 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:31:26.053052 20122 consensus_peers.cc:597] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:26.266335 20048 consensus_peers.cc:597] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:26.313903 20048 consensus_peers.cc:597] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:26.576870 20048 consensus_peers.cc:597] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:26.723556 20122 consensus_peers.cc:597] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:26.768060 20122 consensus_peers.cc:597] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:26.807454 19954 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:31:27.763105 20048 consensus_peers.cc:597] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:27.808666 19954 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:31:27.929406 20122 consensus_peers.cc:597] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:27.985256 20122 consensus_peers.cc:597] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:28.145342 20048 consensus_peers.cc:597] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:31:28.406502 20122 consensus_peers.cc:597] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:31:28.568276 20048 consensus_peers.cc:597] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:31:28.789558  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:31:28.797374 20325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:28.797796 20326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:28.799026 20328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:28.804575 20122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20260709 12:31:28.805030  4356 server_base.cc:1061] running on GCE node
W20260709 12:31:28.810679 19954 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:31:28.811267  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:31:28.811695  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:31:28.811918  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600288811899 us; error 0 us; skew 500 ppm
I20260709 12:31:28.812733  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:31:28.816541  4356 webserver.cc:533] Webserver started at http://127.4.65.1:36645/ using document root <none> and password file <none>
I20260709 12:31:28.817189  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:31:28.817438  4356 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 12:31:28.822099  4356 fs_manager.cc:714] Time spent opening directory manager: real 0.003s	user 0.005s	sys 0.000s
I20260709 12:31:28.827095 20333 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:28.828208  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:31:28.828648  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-0-root
uuid: "971f449846cf4a269f817059278b86ea"
format_stamp: "Formatted at 2026-07-09 12:31:19 on dist-test-slave-l1c9"
I20260709 12:31:28.829104  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599783304823-4356-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260709 12:31:28.858422 20048 consensus_peers.cc:597] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:31:28.874492  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:31:28.875986  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:31:28.877635  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:31:28.881705 20340 ts_tablet_manager.cc:548] Loading tablet metadata (0/10 complete)
I20260709 12:31:28.933663  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (10 total tablets, 10 live tablets)
I20260709 12:31:28.934018  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.054s	user 0.002s	sys 0.000s
I20260709 12:31:28.934382  4356 ts_tablet_manager.cc:600] Registering tablets (0/10 complete)
I20260709 12:31:28.939316 20340 tablet_bootstrap.cc:492] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:28.953604 20340 tablet_bootstrap.cc:492] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:28.954588 20340 tablet_bootstrap.cc:492] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea: Bootstrap complete.
I20260709 12:31:28.955277 20340 ts_tablet_manager.cc:1403] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.016s	user 0.002s	sys 0.011s
I20260709 12:31:28.957895 20340 raft_consensus.cc:359] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:28.958513 20340 raft_consensus.cc:740] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:28.959473 20340 consensus_queue.cc:260] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:28.962116 20340 ts_tablet_manager.cc:1434] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.001s
I20260709 12:31:28.962893 20340 tablet_bootstrap.cc:492] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:28.976468  4356 ts_tablet_manager.cc:616] Registered 10 tablets
I20260709 12:31:28.976754  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.042s	user 0.038s	sys 0.003s
I20260709 12:31:28.977463 20340 tablet_bootstrap.cc:492] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:28.978366 20340 tablet_bootstrap.cc:492] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea: Bootstrap complete.
I20260709 12:31:28.978947 20340 ts_tablet_manager.cc:1403] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.016s	user 0.010s	sys 0.003s
I20260709 12:31:28.981614 20340 raft_consensus.cc:359] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:28.982278 20340 raft_consensus.cc:740] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:28.982992 20340 consensus_queue.cc:260] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:28.984933 20340 ts_tablet_manager.cc:1434] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.005s
I20260709 12:31:28.985854 20340 tablet_bootstrap.cc:492] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:29.001191 20340 tablet_bootstrap.cc:492] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:29.002118 20340 tablet_bootstrap.cc:492] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea: Bootstrap complete.
I20260709 12:31:29.002877 20340 ts_tablet_manager.cc:1403] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.017s	user 0.010s	sys 0.004s
I20260709 12:31:29.005462 20340 raft_consensus.cc:359] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.005993 20340 raft_consensus.cc:740] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:29.006702 20340 consensus_queue.cc:260] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.008342 20340 ts_tablet_manager.cc:1434] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:31:29.009214 20340 tablet_bootstrap.cc:492] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:29.029548 20340 tablet_bootstrap.cc:492] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:31:29.030514 20340 tablet_bootstrap.cc:492] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: Bootstrap complete.
I20260709 12:31:29.031147 20340 ts_tablet_manager.cc:1403] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.022s	user 0.011s	sys 0.008s
I20260709 12:31:29.033449 20340 raft_consensus.cc:359] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.034204 20340 raft_consensus.cc:740] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:29.034860 20340 consensus_queue.cc:260] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.036731 20340 ts_tablet_manager.cc:1434] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.003s
I20260709 12:31:29.037426 20340 tablet_bootstrap.cc:492] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
W20260709 12:31:29.048831 20122 consensus_peers.cc:597] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:31:29.050474  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.1:44731
I20260709 12:31:29.050603 20402 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.1:44731 every 8 connection(s)
I20260709 12:31:29.052691 20340 tablet_bootstrap.cc:492] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:29.053506 20340 tablet_bootstrap.cc:492] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea: Bootstrap complete.
I20260709 12:31:29.054082 20340 ts_tablet_manager.cc:1403] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.017s	user 0.010s	sys 0.004s
I20260709 12:31:29.055264  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:29.057099 20340 raft_consensus.cc:359] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.057812 20340 raft_consensus.cc:740] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:29.058542 20340 consensus_queue.cc:260] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.061266 20340 ts_tablet_manager.cc:1434] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:31:29.062548 20340 tablet_bootstrap.cc:492] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:29.083992 20340 tablet_bootstrap.cc:492] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:29.085135 20340 tablet_bootstrap.cc:492] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea: Bootstrap complete.
I20260709 12:31:29.085887 20403 heartbeater.cc:344] Connected to a master server at 127.4.65.62:44901
I20260709 12:31:29.086438 20403 heartbeater.cc:461] Registering TS with master...
I20260709 12:31:29.085889 20340 ts_tablet_manager.cc:1403] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.024s	user 0.020s	sys 0.000s
I20260709 12:31:29.087471 20403 heartbeater.cc:507] Master 127.4.65.62:44901 requested a full tablet report, sending...
I20260709 12:31:29.090255 20340 raft_consensus.cc:359] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.091004 20340 raft_consensus.cc:740] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:29.091910 20340 consensus_queue.cc:260] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.092694 20078 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "594477803faa4705a2b7bd7eb621295d"
dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
mode: GRACEFUL
new_leader_uuid: "44efc7828ed44d76bb553655566fd86e"
 from {username='slave'} at 127.0.0.1:45448
I20260709 12:31:29.093533 20078 raft_consensus.cc:606] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Received request to transfer leadership to TS 44efc7828ed44d76bb553655566fd86e
I20260709 12:31:29.094820 20340 ts_tablet_manager.cc:1434] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:31:29.095534 19904 ts_manager.cc:194] Re-registered known tserver with Master: 971f449846cf4a269f817059278b86ea (127.4.65.1:44731)
I20260709 12:31:29.096057 20340 tablet_bootstrap.cc:492] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:29.111176 19904 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48964
I20260709 12:31:29.112950 20153 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c8b4cb1a18524c3d8189e535e8666620"
dest_uuid: "44efc7828ed44d76bb553655566fd86e"
mode: GRACEFUL
new_leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
 from {username='slave'} at 127.0.0.1:48888
I20260709 12:31:29.113611 20340 tablet_bootstrap.cc:492] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:29.113620 20153 raft_consensus.cc:606] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Received request to transfer leadership to TS 4ab3d29ae436426ab6d05fef08b4641b
I20260709 12:31:29.114828 20340 tablet_bootstrap.cc:492] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea: Bootstrap complete.
I20260709 12:31:29.115356 20403 heartbeater.cc:499] Master 127.4.65.62:44901 was elected leader, sending a full tablet report...
I20260709 12:31:29.115535 20340 ts_tablet_manager.cc:1403] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.007s
I20260709 12:31:29.115396  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:31:29.116827  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:29.117870  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:29.118374 20340 raft_consensus.cc:359] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:29.119005 20340 raft_consensus.cc:740] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:29.119712 20340 consensus_queue.cc:260] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:29.121563 20340 ts_tablet_manager.cc:1434] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.005s	user 0.001s	sys 0.003s
I20260709 12:31:29.122470 20340 tablet_bootstrap.cc:492] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:29.138839 20340 tablet_bootstrap.cc:492] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:29.139796 20340 tablet_bootstrap.cc:492] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea: Bootstrap complete.
I20260709 12:31:29.140535 20340 ts_tablet_manager.cc:1403] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.018s	user 0.017s	sys 0.000s
I20260709 12:31:29.143203 20340 raft_consensus.cc:359] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.143713 20340 raft_consensus.cc:740] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:29.144210 20340 consensus_queue.cc:260] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.146234 20340 ts_tablet_manager.cc:1434] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:31:29.147172 20340 tablet_bootstrap.cc:492] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:29.161754 20340 tablet_bootstrap.cc:492] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:29.162592 20340 tablet_bootstrap.cc:492] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea: Bootstrap complete.
I20260709 12:31:29.163190 20340 ts_tablet_manager.cc:1403] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.016s	user 0.013s	sys 0.000s
I20260709 12:31:29.165277 20340 raft_consensus.cc:359] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.165760 20340 raft_consensus.cc:740] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:29.166584 20340 consensus_queue.cc:260] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.168867 20340 ts_tablet_manager.cc:1434] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.004s
I20260709 12:31:29.170055 20340 tablet_bootstrap.cc:492] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea: Bootstrap starting.
I20260709 12:31:29.183938 20340 tablet_bootstrap.cc:492] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:31:29.184831 20340 tablet_bootstrap.cc:492] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea: Bootstrap complete.
I20260709 12:31:29.185456 20340 ts_tablet_manager.cc:1403] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.008s
I20260709 12:31:29.187440 20376 raft_consensus.cc:3060] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:29.188023 20340 raft_consensus.cc:359] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.188864 20340 raft_consensus.cc:740] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 971f449846cf4a269f817059278b86ea, State: Initialized, Role: FOLLOWER
I20260709 12:31:29.189697 20340 consensus_queue.cc:260] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.191895 20340 ts_tablet_manager.cc:1434] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.001s
I20260709 12:31:29.198192 20376 pending_rounds.cc:85] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: Aborting all ops after (but not including) 0
I20260709 12:31:29.199345 20376 pending_rounds.cc:107] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20260709 12:31:29.199657 20376 raft_consensus.cc:2889] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:31:29.289796 20418 raft_consensus.cc:993] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e: : Instructing follower 4ab3d29ae436426ab6d05fef08b4641b to start an election
I20260709 12:31:29.290246 20418 raft_consensus.cc:1081] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Signalling peer 4ab3d29ae436426ab6d05fef08b4641b to start an election
I20260709 12:31:29.292109 20078 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c8b4cb1a18524c3d8189e535e8666620"
dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
 from {username='slave'} at 127.0.0.1:59674
I20260709 12:31:29.292850 20078 raft_consensus.cc:493] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:29.293165 20078 raft_consensus.cc:3060] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:29.300522 20078 raft_consensus.cc:515] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:29.302529 20078 leader_election.cc:290] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 election: Requested vote from peers 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645), 971f449846cf4a269f817059278b86ea (127.4.65.1:44731)
I20260709 12:31:29.303570 20153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8b4cb1a18524c3d8189e535e8666620" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:29.303901 20376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8b4cb1a18524c3d8189e535e8666620" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:29.304397 20153 raft_consensus.cc:3055] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:29.304738 20376 raft_consensus.cc:3060] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:29.305008 20153 raft_consensus.cc:740] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:29.305939 20153 consensus_queue.cc:260] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:29.307011 20153 raft_consensus.cc:3060] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:29.312220 20376 raft_consensus.cc:2468] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 2.
I20260709 12:31:29.313382 20153 raft_consensus.cc:2468] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 2.
I20260709 12:31:29.314003 20048 leader_election.cc:304] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:29.315320 20313 raft_consensus.cc:2804] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:29.316783 20313 raft_consensus.cc:697] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Becoming Leader. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
I20260709 12:31:29.317771 20313 consensus_queue.cc:237] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:29.325156 19904 catalog_manager.cc:5697] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b reported cstate change: term changed from 1 to 2, leader changed from 44efc7828ed44d76bb553655566fd86e (127.4.65.3) to 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:29.332749 20376 raft_consensus.cc:3060] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:29.434814 20419 raft_consensus.cc:993] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b: : Instructing follower 44efc7828ed44d76bb553655566fd86e to start an election
I20260709 12:31:29.435384 20313 raft_consensus.cc:1081] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Signalling peer 44efc7828ed44d76bb553655566fd86e to start an election
I20260709 12:31:29.436996 20153 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "594477803faa4705a2b7bd7eb621295d"
dest_uuid: "44efc7828ed44d76bb553655566fd86e"
 from {username='slave'} at 127.0.0.1:57858
I20260709 12:31:29.437655 20153 raft_consensus.cc:493] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:29.438025 20153 raft_consensus.cc:3060] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:29.445498 20153 raft_consensus.cc:515] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.447830 20153 leader_election.cc:290] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 3 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:29.448755 20376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "594477803faa4705a2b7bd7eb621295d" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:29.449007 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "594477803faa4705a2b7bd7eb621295d" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:29.449498 20376 raft_consensus.cc:3060] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:29.449666 20078 raft_consensus.cc:3055] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:31:29.450206 20078 raft_consensus.cc:740] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
I20260709 12:31:29.450990 20078 consensus_queue.cc:260] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.451965 20078 raft_consensus.cc:3060] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:29.457079 20376 raft_consensus.cc:2468] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 3.
I20260709 12:31:29.457777 20078 raft_consensus.cc:2468] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 3.
I20260709 12:31:29.458582 20122 leader_election.cc:304] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:29.459713 20418 raft_consensus.cc:2804] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:31:29.460264 20418 raft_consensus.cc:697] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 3 LEADER]: Becoming Leader. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:29.461073 20418 consensus_queue.cc:237] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:29.469491 19904 catalog_manager.cc:5697] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e reported cstate change: term changed from 2 to 3, leader changed from 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2) to 44efc7828ed44d76bb553655566fd86e (127.4.65.3). New cstate: current_term: 3 leader_uuid: "44efc7828ed44d76bb553655566fd86e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:29.489621 20376 raft_consensus.cc:3060] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:29.563557 20376 raft_consensus.cc:3060] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:31:29.811743 19954 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:31:29.843801 20153 raft_consensus.cc:1275] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Refusing update from remote peer 4ab3d29ae436426ab6d05fef08b4641b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:29.845872 20313 consensus_queue.cc:1048] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:29.859122 20376 raft_consensus.cc:1275] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Refusing update from remote peer 4ab3d29ae436426ab6d05fef08b4641b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:29.860558 20313 consensus_queue.cc:1048] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:30.023728 20376 raft_consensus.cc:1275] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Refusing update from remote peer 44efc7828ed44d76bb553655566fd86e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:31:30.025269 20319 consensus_queue.cc:1048] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:31:30.034229 20078 raft_consensus.cc:1275] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Refusing update from remote peer 44efc7828ed44d76bb553655566fd86e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:31:30.036247 20430 consensus_queue.cc:1048] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:31:30.128147  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:30.137508 20153 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517"
dest_uuid: "44efc7828ed44d76bb553655566fd86e"
mode: GRACEFUL
new_leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
 from {username='slave'} at 127.0.0.1:48888
I20260709 12:31:30.138070 20153 raft_consensus.cc:606] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Received request to transfer leadership to TS 4ab3d29ae436426ab6d05fef08b4641b
I20260709 12:31:30.140991 20078 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "daaf356d34ab43e399517f8b355739a9"
dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
mode: GRACEFUL
new_leader_uuid: "44efc7828ed44d76bb553655566fd86e"
 from {username='slave'} at 127.0.0.1:45448
I20260709 12:31:30.141521 20078 raft_consensus.cc:606] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 LEADER]: Received request to transfer leadership to TS 44efc7828ed44d76bb553655566fd86e
I20260709 12:31:30.142732  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:31:30.143260  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:30.143942  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:30.290763 20319 raft_consensus.cc:993] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e: : Instructing follower 4ab3d29ae436426ab6d05fef08b4641b to start an election
I20260709 12:31:30.291316 20430 raft_consensus.cc:1081] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Signalling peer 4ab3d29ae436426ab6d05fef08b4641b to start an election
I20260709 12:31:30.293043 20078 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517"
dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
 from {username='slave'} at 127.0.0.1:59674
I20260709 12:31:30.293596 20078 raft_consensus.cc:493] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:30.293892 20078 raft_consensus.cc:3060] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:30.298734 20078 raft_consensus.cc:515] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:30.300137 20078 leader_election.cc:290] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:30.301308 20376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:30.301344 20153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:30.301904 20376 raft_consensus.cc:3060] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:30.302423 20153 raft_consensus.cc:3055] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:30.302744 20153 raft_consensus.cc:740] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:30.303591 20153 consensus_queue.cc:260] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:30.304903 20153 raft_consensus.cc:3060] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:30.307713 20376 raft_consensus.cc:2468] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 2.
I20260709 12:31:30.308761 20048 leader_election.cc:304] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:30.309569 20429 raft_consensus.cc:2804] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:30.310047 20429 raft_consensus.cc:697] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Becoming Leader. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
I20260709 12:31:30.310796 20153 raft_consensus.cc:2468] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 2.
I20260709 12:31:30.311187 20429 consensus_queue.cc:237] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:30.318620 19904 catalog_manager.cc:5697] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b reported cstate change: term changed from 1 to 2, leader changed from 44efc7828ed44d76bb553655566fd86e (127.4.65.3) to 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:30.464260 20419 raft_consensus.cc:993] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b: : Instructing follower 44efc7828ed44d76bb553655566fd86e to start an election
I20260709 12:31:30.464845 20429 raft_consensus.cc:1081] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 LEADER]: Signalling peer 44efc7828ed44d76bb553655566fd86e to start an election
I20260709 12:31:30.466799 20153 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "daaf356d34ab43e399517f8b355739a9"
dest_uuid: "44efc7828ed44d76bb553655566fd86e"
 from {username='slave'} at 127.0.0.1:57858
I20260709 12:31:30.467401 20153 raft_consensus.cc:493] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:30.467705 20153 raft_consensus.cc:3060] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:30.474774 20153 raft_consensus.cc:515] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:30.476766 20153 leader_election.cc:290] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:30.477566 20376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daaf356d34ab43e399517f8b355739a9" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:30.477759 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daaf356d34ab43e399517f8b355739a9" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:30.478229 20376 raft_consensus.cc:3060] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:30.478386 20078 raft_consensus.cc:3055] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:30.478794 20078 raft_consensus.cc:740] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
I20260709 12:31:30.479540 20078 consensus_queue.cc:260] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:30.480584 20078 raft_consensus.cc:3060] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:30.483635 20376 raft_consensus.cc:2468] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 2.
I20260709 12:31:30.484942 20122 leader_election.cc:304] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:30.485491 20078 raft_consensus.cc:2468] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 2.
I20260709 12:31:30.485646 20430 raft_consensus.cc:2804] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:30.486176 20430 raft_consensus.cc:697] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [term 2 LEADER]: Becoming Leader. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:30.487131 20430 consensus_queue.cc:237] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:30.496841 19904 catalog_manager.cc:5697] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e reported cstate change: term changed from 1 to 2, leader changed from 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2) to 44efc7828ed44d76bb553655566fd86e (127.4.65.3). New cstate: current_term: 2 leader_uuid: "44efc7828ed44d76bb553655566fd86e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:30.731664 20376 raft_consensus.cc:1275] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Refusing update from remote peer 4ab3d29ae436426ab6d05fef08b4641b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:30.733024 20429 consensus_queue.cc:1048] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:30.746891 20153 raft_consensus.cc:1275] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Refusing update from remote peer 4ab3d29ae436426ab6d05fef08b4641b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:30.749059 20419 consensus_queue.cc:1048] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:31:30.812846 19954 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:31:30.930644 20078 raft_consensus.cc:1275] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Refusing update from remote peer 44efc7828ed44d76bb553655566fd86e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:30.932101 20441 consensus_queue.cc:1048] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:30.942919 20376 raft_consensus.cc:1275] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Refusing update from remote peer 44efc7828ed44d76bb553655566fd86e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:30.945020 20449 consensus_queue.cc:1048] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:31:31.150322  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:31.157775 20153 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b3177f3e632242e5ba149e10e70ab098"
dest_uuid: "44efc7828ed44d76bb553655566fd86e"
mode: GRACEFUL
new_leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
 from {username='slave'} at 127.0.0.1:48888
I20260709 12:31:31.158438 20153 raft_consensus.cc:606] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Received request to transfer leadership to TS 4ab3d29ae436426ab6d05fef08b4641b
I20260709 12:31:31.161520 20078 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c8b4cb1a18524c3d8189e535e8666620"
dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
mode: GRACEFUL
new_leader_uuid: "44efc7828ed44d76bb553655566fd86e"
 from {username='slave'} at 127.0.0.1:45448
I20260709 12:31:31.162024 20078 raft_consensus.cc:606] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Received request to transfer leadership to TS 44efc7828ed44d76bb553655566fd86e
I20260709 12:31:31.163324  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:31:31.163875  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:31.164507  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:31.335109 20319 raft_consensus.cc:993] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e: : Instructing follower 4ab3d29ae436426ab6d05fef08b4641b to start an election
I20260709 12:31:31.335815 20449 raft_consensus.cc:1081] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Signalling peer 4ab3d29ae436426ab6d05fef08b4641b to start an election
I20260709 12:31:31.338021 20078 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b3177f3e632242e5ba149e10e70ab098"
dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
 from {username='slave'} at 127.0.0.1:59674
I20260709 12:31:31.338636 20078 raft_consensus.cc:493] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:31.339001 20078 raft_consensus.cc:3060] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:31.340683 20419 raft_consensus.cc:993] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b: : Instructing follower 44efc7828ed44d76bb553655566fd86e to start an election
I20260709 12:31:31.341286 20451 raft_consensus.cc:1081] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Signalling peer 44efc7828ed44d76bb553655566fd86e to start an election
I20260709 12:31:31.345044 20153 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c8b4cb1a18524c3d8189e535e8666620"
dest_uuid: "44efc7828ed44d76bb553655566fd86e"
 from {username='slave'} at 127.0.0.1:57858
I20260709 12:31:31.345677 20153 raft_consensus.cc:493] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:31.346040 20153 raft_consensus.cc:3060] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:31.347927 20078 raft_consensus.cc:515] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:31.350099 20078 leader_election.cc:290] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:31.351220 20376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3177f3e632242e5ba149e10e70ab098" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:31.351272 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3177f3e632242e5ba149e10e70ab098" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:31.351912 20376 raft_consensus.cc:3060] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:31.352326 20154 raft_consensus.cc:3055] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:31.352757 20154 raft_consensus.cc:740] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:31.353631 20154 consensus_queue.cc:260] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:31.353912 20153 raft_consensus.cc:515] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:31.355046 20154 raft_consensus.cc:3060] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:31.356608 20153 leader_election.cc:290] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 3 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:31.357189 20375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8b4cb1a18524c3d8189e535e8666620" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:31.357820 20375 raft_consensus.cc:3060] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:31.358085 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8b4cb1a18524c3d8189e535e8666620" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:31.358639 20078 raft_consensus.cc:3055] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:31:31.359084 20078 raft_consensus.cc:740] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
I20260709 12:31:31.359913 20078 consensus_queue.cc:260] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:31.360992 20078 raft_consensus.cc:3060] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:31.362653 20154 raft_consensus.cc:2468] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 2.
I20260709 12:31:31.364185 20047 leader_election.cc:304] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 
I20260709 12:31:31.365655 20451 raft_consensus.cc:2804] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:31.366094 20451 raft_consensus.cc:697] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Becoming Leader. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
I20260709 12:31:31.367657 20376 raft_consensus.cc:2468] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 2.
I20260709 12:31:31.367411 20451 consensus_queue.cc:237] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:31.367875 20078 raft_consensus.cc:2468] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 3.
I20260709 12:31:31.370694 20375 raft_consensus.cc:2468] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 3.
I20260709 12:31:31.372037 20374 raft_consensus.cc:1240] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Rejecting Update request from peer 4ab3d29ae436426ab6d05fef08b4641b for earlier term 2. Current term is 3. Ops: []
I20260709 12:31:31.372264 20120 leader_election.cc:304] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 4ab3d29ae436426ab6d05fef08b4641b; no voters: 
I20260709 12:31:31.373359 20449 raft_consensus.cc:2804] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:31:31.374022 20449 raft_consensus.cc:697] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 3 LEADER]: Becoming Leader. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:31.374903 20449 consensus_queue.cc:237] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } }
I20260709 12:31:31.382293 19904 catalog_manager.cc:5697] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b reported cstate change: term changed from 1 to 2, leader changed from 44efc7828ed44d76bb553655566fd86e (127.4.65.3) to 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2). New cstate: current_term: 2 leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:31.386482 19906 catalog_manager.cc:5697] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e reported cstate change: term changed from 2 to 3, leader changed from 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2) to 44efc7828ed44d76bb553655566fd86e (127.4.65.3). New cstate: current_term: 3 leader_uuid: "44efc7828ed44d76bb553655566fd86e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:31.791967 20153 raft_consensus.cc:1275] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Refusing update from remote peer 4ab3d29ae436426ab6d05fef08b4641b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:31.793452 20451 consensus_queue.cc:1048] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:31.806397 20374 raft_consensus.cc:1275] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Refusing update from remote peer 4ab3d29ae436426ab6d05fef08b4641b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:31.808326 20460 consensus_queue.cc:1048] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:31:31.813903 19954 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:31:31.814116 20375 raft_consensus.cc:1275] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Refusing update from remote peer 44efc7828ed44d76bb553655566fd86e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:31:31.816015 20449 consensus_queue.cc:1048] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:31:31.825481 20078 raft_consensus.cc:1275] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Refusing update from remote peer 44efc7828ed44d76bb553655566fd86e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:31:31.827132 20319 consensus_queue.cc:1048] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:31:32.171295  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:32.179070 20153 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87"
dest_uuid: "44efc7828ed44d76bb553655566fd86e"
mode: GRACEFUL
new_leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
 from {username='slave'} at 127.0.0.1:48888
I20260709 12:31:32.179589 20153 raft_consensus.cc:606] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 2 LEADER]: Received request to transfer leadership to TS 4ab3d29ae436426ab6d05fef08b4641b
I20260709 12:31:32.182726 20078 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517"
dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
mode: GRACEFUL
new_leader_uuid: "44efc7828ed44d76bb553655566fd86e"
 from {username='slave'} at 127.0.0.1:45448
I20260709 12:31:32.183429 20078 raft_consensus.cc:606] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Received request to transfer leadership to TS 44efc7828ed44d76bb553655566fd86e
I20260709 12:31:32.184820  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:31:32.185405  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:32.186010  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:32.201900 20319 raft_consensus.cc:993] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e: : Instructing follower 4ab3d29ae436426ab6d05fef08b4641b to start an election
I20260709 12:31:32.202566 20430 raft_consensus.cc:1081] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 2 LEADER]: Signalling peer 4ab3d29ae436426ab6d05fef08b4641b to start an election
I20260709 12:31:32.204600 20078 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87"
dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
 from {username='slave'} at 127.0.0.1:59674
I20260709 12:31:32.205184 20078 raft_consensus.cc:493] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:32.205523 20078 raft_consensus.cc:3060] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:32.211946 20078 raft_consensus.cc:515] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:32.213519 20078 leader_election.cc:290] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 3 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:32.214735 20374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:32.214819 20153 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:32.215435 20374 raft_consensus.cc:3060] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:32.215507 20153 raft_consensus.cc:3055] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:31:32.215945 20153 raft_consensus.cc:740] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:32.216785 20153 consensus_queue.cc:260] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:32.217970 20153 raft_consensus.cc:3060] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:32.224099 20374 raft_consensus.cc:2468] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 3.
I20260709 12:31:32.225802 20048 leader_election.cc:304] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:32.226236 20153 raft_consensus.cc:2468] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 3.
I20260709 12:31:32.226878 20460 raft_consensus.cc:2804] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:31:32.227612 20460 raft_consensus.cc:697] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 LEADER]: Becoming Leader. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
I20260709 12:31:32.228435 20460 consensus_queue.cc:237] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:32.235569 19904 catalog_manager.cc:5697] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b reported cstate change: term changed from 2 to 3, leader changed from 44efc7828ed44d76bb553655566fd86e (127.4.65.3) to 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2). New cstate: current_term: 3 leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:32.644762 20153 raft_consensus.cc:1275] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Refusing update from remote peer 4ab3d29ae436426ab6d05fef08b4641b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:31:32.646536 20460 consensus_queue.cc:1048] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:31:32.659797 20374 raft_consensus.cc:1275] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Refusing update from remote peer 4ab3d29ae436426ab6d05fef08b4641b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:31:32.661788 20451 consensus_queue.cc:1048] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:31:32.673730 20451 raft_consensus.cc:993] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b: : Instructing follower 44efc7828ed44d76bb553655566fd86e to start an election
I20260709 12:31:32.674278 20460 raft_consensus.cc:1081] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Signalling peer 44efc7828ed44d76bb553655566fd86e to start an election
I20260709 12:31:32.675966 20153 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517"
dest_uuid: "44efc7828ed44d76bb553655566fd86e"
 from {username='slave'} at 127.0.0.1:57858
I20260709 12:31:32.676505 20153 raft_consensus.cc:493] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:32.676792 20153 raft_consensus.cc:3060] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:32.683290 20153 raft_consensus.cc:515] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:32.684844 20153 leader_election.cc:290] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 3 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:32.686110 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:32.686087 20374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:32.687165 20078 raft_consensus.cc:3055] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:31:32.687171 20374 raft_consensus.cc:3060] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:32.687726 20078 raft_consensus.cc:740] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
I20260709 12:31:32.688830 20078 consensus_queue.cc:260] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:32.690196 20078 raft_consensus.cc:3060] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:32.695513 20374 raft_consensus.cc:2468] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 3.
I20260709 12:31:32.697278 20078 raft_consensus.cc:2468] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 3.
I20260709 12:31:32.697381 20122 leader_election.cc:304] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:32.698743 20449 raft_consensus.cc:2804] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:31:32.699353 20449 raft_consensus.cc:697] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 3 LEADER]: Becoming Leader. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:32.700364 20449 consensus_queue.cc:237] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:32.708123 19906 catalog_manager.cc:5697] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e reported cstate change: term changed from 2 to 3, leader changed from 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2) to 44efc7828ed44d76bb553655566fd86e (127.4.65.3). New cstate: current_term: 3 leader_uuid: "44efc7828ed44d76bb553655566fd86e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: HEALTHY } } }
W20260709 12:31:32.815022 19954 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:31:33.191056 20078 raft_consensus.cc:1275] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Refusing update from remote peer 44efc7828ed44d76bb553655566fd86e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:31:33.192622  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:33.192814 20449 consensus_queue.cc:1048] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:31:33.200475 20374 raft_consensus.cc:1275] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Refusing update from remote peer 44efc7828ed44d76bb553655566fd86e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:31:33.206737 20078 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87"
dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
mode: GRACEFUL
new_leader_uuid: "44efc7828ed44d76bb553655566fd86e"
 from {username='slave'} at 127.0.0.1:45448
I20260709 12:31:33.208892 20078 raft_consensus.cc:606] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 LEADER]: Received request to transfer leadership to TS 44efc7828ed44d76bb553655566fd86e
I20260709 12:31:33.208681 20449 consensus_queue.cc:1048] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:31:33.214550 20153 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517"
dest_uuid: "44efc7828ed44d76bb553655566fd86e"
mode: GRACEFUL
new_leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
 from {username='slave'} at 127.0.0.1:48888
I20260709 12:31:33.215320 20153 raft_consensus.cc:606] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 3 LEADER]: Received request to transfer leadership to TS 4ab3d29ae436426ab6d05fef08b4641b
I20260709 12:31:33.216882  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:31:33.217628  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:33.218415  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:33.622076 20484 raft_consensus.cc:993] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e: : Instructing follower 4ab3d29ae436426ab6d05fef08b4641b to start an election
I20260709 12:31:33.623153 20483 raft_consensus.cc:1081] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 3 LEADER]: Signalling peer 4ab3d29ae436426ab6d05fef08b4641b to start an election
I20260709 12:31:33.625875 20078 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517"
dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
 from {username='slave'} at 127.0.0.1:59674
I20260709 12:31:33.627028 20078 raft_consensus.cc:493] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:33.627533 20078 raft_consensus.cc:3060] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:31:33.641036 20451 raft_consensus.cc:993] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b: : Instructing follower 44efc7828ed44d76bb553655566fd86e to start an election
I20260709 12:31:33.641022 20078 raft_consensus.cc:515] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:33.642117 20451 raft_consensus.cc:1081] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 LEADER]: Signalling peer 44efc7828ed44d76bb553655566fd86e to start an election
I20260709 12:31:33.644011 20078 leader_election.cc:290] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 4 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 44efc7828ed44d76bb553655566fd86e (127.4.65.3:44645)
I20260709 12:31:33.644716 20153 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87"
dest_uuid: "44efc7828ed44d76bb553655566fd86e"
 from {username='slave'} at 127.0.0.1:57858
I20260709 12:31:33.645592 20153 raft_consensus.cc:493] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:33.645409 20374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:33.646111 20153 raft_consensus.cc:3060] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:31:33.646447 20154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d224f01fe0364162a1f99b2863aab517" candidate_uuid: "4ab3d29ae436426ab6d05fef08b4641b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "44efc7828ed44d76bb553655566fd86e"
I20260709 12:31:33.646577 20374 raft_consensus.cc:3060] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:31:33.647258 20154 raft_consensus.cc:3055] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:31:33.647688 20154 raft_consensus.cc:740] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:33.648947 20154 consensus_queue.cc:260] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:33.650966 20154 raft_consensus.cc:3060] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:31:33.658215 20153 raft_consensus.cc:515] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:33.659344 20374 raft_consensus.cc:2468] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 4.
I20260709 12:31:33.661131 20048 leader_election.cc:304] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ab3d29ae436426ab6d05fef08b4641b, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:33.661499 20154 raft_consensus.cc:2468] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ab3d29ae436426ab6d05fef08b4641b in term 4.
I20260709 12:31:33.661849 20374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "971f449846cf4a269f817059278b86ea"
I20260709 12:31:33.662770 20451 raft_consensus.cc:2804] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:31:33.662798 20374 raft_consensus.cc:3060] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:31:33.665130 20451 raft_consensus.cc:697] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 4 LEADER]: Becoming Leader. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
I20260709 12:31:33.665486 20153 leader_election.cc:290] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 4 election: Requested vote from peers 971f449846cf4a269f817059278b86ea (127.4.65.1:44731), 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481)
I20260709 12:31:33.666523 20078 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d74a1b173ce4594b4b9090f9b497f87" candidate_uuid: "44efc7828ed44d76bb553655566fd86e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4ab3d29ae436426ab6d05fef08b4641b"
I20260709 12:31:33.667240 20078 raft_consensus.cc:3055] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:31:33.667574 20078 raft_consensus.cc:740] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4ab3d29ae436426ab6d05fef08b4641b, State: Running, Role: LEADER
I20260709 12:31:33.668311 20451 consensus_queue.cc:237] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:33.671886 20078 consensus_queue.cc:260] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:33.673637 20078 raft_consensus.cc:3060] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:31:33.676859 20374 raft_consensus.cc:2468] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 4.
I20260709 12:31:33.678431 20122 leader_election.cc:304] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44efc7828ed44d76bb553655566fd86e, 971f449846cf4a269f817059278b86ea; no voters: 
I20260709 12:31:33.679349 20483 raft_consensus.cc:2804] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:31:33.679869 20483 raft_consensus.cc:697] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 4 LEADER]: Becoming Leader. State: Replica: 44efc7828ed44d76bb553655566fd86e, State: Running, Role: LEADER
I20260709 12:31:33.681082 20483 consensus_queue.cc:237] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } }
I20260709 12:31:33.684092 19906 catalog_manager.cc:5697] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b reported cstate change: term changed from 3 to 4, leader changed from 44efc7828ed44d76bb553655566fd86e (127.4.65.3) to 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2). New cstate: current_term: 4 leader_uuid: "4ab3d29ae436426ab6d05fef08b4641b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:33.686692 20078 raft_consensus.cc:2468] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44efc7828ed44d76bb553655566fd86e in term 4.
I20260709 12:31:33.692233 19904 catalog_manager.cc:5697] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e reported cstate change: term changed from 3 to 4, leader changed from 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2) to 44efc7828ed44d76bb553655566fd86e (127.4.65.3). New cstate: current_term: 4 leader_uuid: "44efc7828ed44d76bb553655566fd86e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 } health_report { overall_health: HEALTHY } } }
W20260709 12:31:33.816371 19954 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:31:34.090444 20375 raft_consensus.cc:1275] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 4 FOLLOWER]: Refusing update from remote peer 44efc7828ed44d76bb553655566fd86e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:31:34.092331 20483 consensus_queue.cc:1048] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:31:34.105859 20078 raft_consensus.cc:1275] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 4 FOLLOWER]: Refusing update from remote peer 44efc7828ed44d76bb553655566fd86e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:31:34.108034 20483 consensus_queue.cc:1048] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ab3d29ae436426ab6d05fef08b4641b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:31:34.186059 20375 raft_consensus.cc:1275] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 4 FOLLOWER]: Refusing update from remote peer 4ab3d29ae436426ab6d05fef08b4641b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:31:34.187762 20451 consensus_queue.cc:1048] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "971f449846cf4a269f817059278b86ea" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 44731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:31:34.198545 20153 raft_consensus.cc:1275] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 4 FOLLOWER]: Refusing update from remote peer 4ab3d29ae436426ab6d05fef08b4641b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:31:34.200202 20451 consensus_queue.cc:1048] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [LEADER]: Connected to new peer: Peer: permanent_uuid: "44efc7828ed44d76bb553655566fd86e" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:31:34.230443  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:34.261483  4356 tablet_server.cc:179] TabletServer@127.4.65.1:44731 shutting down...
I20260709 12:31:34.293983  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:31:34.294847  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:34.296018  4356 raft_consensus.cc:2243] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.296769  4356 raft_consensus.cc:2272] T daaf356d34ab43e399517f8b355739a9 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.300684  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:34.301394  4356 raft_consensus.cc:2243] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.301934  4356 raft_consensus.cc:2272] T d224f01fe0364162a1f99b2863aab517 P 971f449846cf4a269f817059278b86ea [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.305233  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:34.306164  4356 raft_consensus.cc:2243] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.306675  4356 raft_consensus.cc:2272] T aa280773cc224cea8253ae8fa2e32e27 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.309314  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:34.310000  4356 raft_consensus.cc:2243] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.310499  4356 raft_consensus.cc:2272] T b3177f3e632242e5ba149e10e70ab098 P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.313576  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:34.314157  4356 raft_consensus.cc:2243] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.314692  4356 raft_consensus.cc:2272] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 971f449846cf4a269f817059278b86ea [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.317473  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:34.318325  4356 raft_consensus.cc:2243] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.318980  4356 raft_consensus.cc:2272] T 6d74a1b173ce4594b4b9090f9b497f87 P 971f449846cf4a269f817059278b86ea [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.322885  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:34.323798  4356 raft_consensus.cc:2243] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:31:34.323740 20048 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
I20260709 12:31:34.324759  4356 raft_consensus.cc:2272] T 043dc64f584a4a35b66b8c7382e92af4 P 971f449846cf4a269f817059278b86ea [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:34.328327 20048 consensus_peers.cc:597] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:34.329133  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:34.330022  4356 raft_consensus.cc:2243] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.330711  4356 raft_consensus.cc:2272] T 594477803faa4705a2b7bd7eb621295d P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.333736  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:34.334295  4356 raft_consensus.cc:2243] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.335253  4356 raft_consensus.cc:2272] T c8b4cb1a18524c3d8189e535e8666620 P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.337957  4356 tablet_replica.cc:333] stopping tablet replica
I20260709 12:31:34.338515  4356 raft_consensus.cc:2243] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.339025  4356 raft_consensus.cc:2272] T 5e2f9bc36a87410f9f14a842ef3785ba P 971f449846cf4a269f817059278b86ea [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:34.355019 20122 consensus_peers.cc:597] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:34.367230  4356 tablet_server.cc:196] TabletServer@127.4.65.1:44731 shutdown complete.
W20260709 12:31:34.371817 20122 consensus_peers.cc:597] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:34.389837 20048 consensus_peers.cc:597] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:34.406455  4356 tablet_server.cc:179] TabletServer@127.4.65.2:0 shutting down...
I20260709 12:31:34.438884  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:31:34.439711  4356 tablet_replica.cc:333] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b: stopping tablet replica
I20260709 12:31:34.440589  4356 raft_consensus.cc:2243] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.441143  4356 raft_consensus.cc:2272] T 594477803faa4705a2b7bd7eb621295d P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.443662  4356 tablet_replica.cc:333] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b: stopping tablet replica
I20260709 12:31:34.444314  4356 raft_consensus.cc:2243] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:31:34.445227  4356 raft_consensus.cc:2272] T d224f01fe0364162a1f99b2863aab517 P 4ab3d29ae436426ab6d05fef08b4641b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.448091  4356 tablet_replica.cc:333] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b: stopping tablet replica
I20260709 12:31:34.448843  4356 raft_consensus.cc:2243] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.449319  4356 raft_consensus.cc:2272] T daaf356d34ab43e399517f8b355739a9 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.451628  4356 tablet_replica.cc:333] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b: stopping tablet replica
I20260709 12:31:34.452186  4356 raft_consensus.cc:2243] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.452677  4356 raft_consensus.cc:2272] T 6d74a1b173ce4594b4b9090f9b497f87 P 4ab3d29ae436426ab6d05fef08b4641b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.456343  4356 tablet_replica.cc:333] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b: stopping tablet replica
I20260709 12:31:34.457021  4356 raft_consensus.cc:2243] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:34.457945  4356 raft_consensus.cc:2272] T 043dc64f584a4a35b66b8c7382e92af4 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.460989  4356 tablet_replica.cc:333] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b: stopping tablet replica
I20260709 12:31:34.461560  4356 raft_consensus.cc:2243] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.462050  4356 raft_consensus.cc:2272] T c8b4cb1a18524c3d8189e535e8666620 P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.464707  4356 tablet_replica.cc:333] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b: stopping tablet replica
I20260709 12:31:34.465270  4356 raft_consensus.cc:2243] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:31:34.466149  4356 raft_consensus.cc:2272] T 5e2f9bc36a87410f9f14a842ef3785ba P 4ab3d29ae436426ab6d05fef08b4641b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.468727  4356 tablet_replica.cc:333] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b: stopping tablet replica
I20260709 12:31:34.469444  4356 raft_consensus.cc:2243] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:34.470391  4356 raft_consensus.cc:2272] T b3177f3e632242e5ba149e10e70ab098 P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.473145  4356 tablet_replica.cc:333] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b: stopping tablet replica
I20260709 12:31:34.473654  4356 raft_consensus.cc:2243] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.474159  4356 raft_consensus.cc:2272] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 4ab3d29ae436426ab6d05fef08b4641b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.476568  4356 tablet_replica.cc:333] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b: stopping tablet replica
I20260709 12:31:34.477020  4356 raft_consensus.cc:2243] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:34.477898  4356 raft_consensus.cc:2272] T aa280773cc224cea8253ae8fa2e32e27 P 4ab3d29ae436426ab6d05fef08b4641b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.505575  4356 tablet_server.cc:196] TabletServer@127.4.65.2:0 shutdown complete.
W20260709 12:31:34.507972 20122 consensus_peers.cc:597] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:34.530020 20120 consensus_peers.cc:597] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e -> Peer 4ab3d29ae436426ab6d05fef08b4641b (127.4.65.2:38481): Couldn't send request to peer 4ab3d29ae436426ab6d05fef08b4641b. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:38481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:34.538209  4356 tablet_server.cc:179] TabletServer@127.4.65.3:0 shutting down...
W20260709 12:31:34.548218 20122 consensus_peers.cc:597] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e -> Peer 971f449846cf4a269f817059278b86ea (127.4.65.1:44731): Couldn't send request to peer 971f449846cf4a269f817059278b86ea. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:44731: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:34.590721  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:31:34.591653  4356 tablet_replica.cc:333] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e: stopping tablet replica
I20260709 12:31:34.592496  4356 raft_consensus.cc:2243] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.593070  4356 raft_consensus.cc:2272] T 5e2f9bc36a87410f9f14a842ef3785ba P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.595551  4356 tablet_replica.cc:333] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e: stopping tablet replica
I20260709 12:31:34.596220  4356 raft_consensus.cc:2243] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:31:34.597169  4356 raft_consensus.cc:2272] T c8b4cb1a18524c3d8189e535e8666620 P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.599431  4356 tablet_replica.cc:333] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e: stopping tablet replica
I20260709 12:31:34.599926  4356 raft_consensus.cc:2243] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:31:34.600811  4356 raft_consensus.cc:2272] T 6d74a1b173ce4594b4b9090f9b497f87 P 44efc7828ed44d76bb553655566fd86e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.603709  4356 tablet_replica.cc:333] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e: stopping tablet replica
I20260709 12:31:34.604266  4356 raft_consensus.cc:2243] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.604866  4356 raft_consensus.cc:2272] T 043dc64f584a4a35b66b8c7382e92af4 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.607012  4356 tablet_replica.cc:333] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e: stopping tablet replica
I20260709 12:31:34.607676  4356 raft_consensus.cc:2243] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:31:34.608686  4356 raft_consensus.cc:2272] T 594477803faa4705a2b7bd7eb621295d P 44efc7828ed44d76bb553655566fd86e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.611970  4356 tablet_replica.cc:333] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e: stopping tablet replica
I20260709 12:31:34.612941  4356 raft_consensus.cc:2243] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.613590  4356 raft_consensus.cc:2272] T b3177f3e632242e5ba149e10e70ab098 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.616042  4356 tablet_replica.cc:333] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e: stopping tablet replica
I20260709 12:31:34.616703  4356 raft_consensus.cc:2243] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.617122  4356 raft_consensus.cc:2272] T aa280773cc224cea8253ae8fa2e32e27 P 44efc7828ed44d76bb553655566fd86e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.619395  4356 tablet_replica.cc:333] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e: stopping tablet replica
I20260709 12:31:34.619971  4356 raft_consensus.cc:2243] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:34.620918  4356 raft_consensus.cc:2272] T a1aa8c0c52f34a8c8b4a2aa87fa57d6e P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.623256  4356 tablet_replica.cc:333] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e: stopping tablet replica
I20260709 12:31:34.623828  4356 raft_consensus.cc:2243] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:34.624756  4356 raft_consensus.cc:2272] T daaf356d34ab43e399517f8b355739a9 P 44efc7828ed44d76bb553655566fd86e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.626886  4356 tablet_replica.cc:333] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e: stopping tablet replica
I20260709 12:31:34.627388  4356 raft_consensus.cc:2243] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:34.627831  4356 raft_consensus.cc:2272] T d224f01fe0364162a1f99b2863aab517 P 44efc7828ed44d76bb553655566fd86e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.663252  4356 tablet_server.cc:196] TabletServer@127.4.65.3:0 shutdown complete.
I20260709 12:31:34.694008  4356 master.cc:562] Master@127.4.65.62:44901 shutting down...
I20260709 12:31:34.722443  4356 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:34.723024  4356 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:34.723320  4356 tablet_replica.cc:333] T 00000000000000000000000000000000 P 275b9d09083d49e69693e575775f2008: stopping tablet replica
I20260709 12:31:34.745862  4356 master.cc:584] Master@127.4.65.62:44901 shutdown complete.
[       OK ] LeaderRebalancerTest.TestMaintenanceMode (15166 ms)
[ RUN      ] LeaderRebalancerTest.MultiTableLeaderBalance
I20260709 12:31:34.793716  4356 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.65.62:42141
I20260709 12:31:34.794832  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:31:34.800927 20510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:34.801237 20509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:34.801805 20512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:31:34.802183  4356 server_base.cc:1061] running on GCE node
I20260709 12:31:34.803225  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:31:34.803418  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:31:34.803529  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600294803517 us; error 0 us; skew 500 ppm
I20260709 12:31:34.804003  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:31:34.806761  4356 webserver.cc:533] Webserver started at http://127.4.65.62:41841/ using document root <none> and password file <none>
I20260709 12:31:34.807507  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:31:34.807765  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:31:34.808073  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:31:34.809729  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/master-0-root/instance:
uuid: "afb2c4f9d2c44f52a143891a6372308c"
format_stamp: "Formatted at 2026-07-09 12:31:34 on dist-test-slave-l1c9"
I20260709 12:31:34.814375  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.001s
I20260709 12:31:34.818440 20517 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:34.819247  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260709 12:31:34.819635  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/master-0-root
uuid: "afb2c4f9d2c44f52a143891a6372308c"
format_stamp: "Formatted at 2026-07-09 12:31:34 on dist-test-slave-l1c9"
I20260709 12:31:34.819948  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:31:34.839335  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:31:34.840723  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:31:34.885072  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.62:42141
I20260709 12:31:34.885258 20568 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.62:42141 every 8 connection(s)
I20260709 12:31:34.889394 20569 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:34.903175 20569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c: Bootstrap starting.
I20260709 12:31:34.908294 20569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:34.913002 20569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c: No bootstrap required, opened a new log
I20260709 12:31:34.915192 20569 raft_consensus.cc:359] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb2c4f9d2c44f52a143891a6372308c" member_type: VOTER }
I20260709 12:31:34.915583 20569 raft_consensus.cc:385] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:34.915755 20569 raft_consensus.cc:740] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb2c4f9d2c44f52a143891a6372308c, State: Initialized, Role: FOLLOWER
I20260709 12:31:34.916276 20569 consensus_queue.cc:260] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb2c4f9d2c44f52a143891a6372308c" member_type: VOTER }
I20260709 12:31:34.916782 20569 raft_consensus.cc:399] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:31:34.916975 20569 raft_consensus.cc:493] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:31:34.917301 20569 raft_consensus.cc:3060] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:34.922610 20569 raft_consensus.cc:515] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb2c4f9d2c44f52a143891a6372308c" member_type: VOTER }
I20260709 12:31:34.923179 20569 leader_election.cc:304] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afb2c4f9d2c44f52a143891a6372308c; no voters: 
I20260709 12:31:34.924521 20569 leader_election.cc:290] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:31:34.925071 20572 raft_consensus.cc:2804] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:34.926774 20572 raft_consensus.cc:697] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [term 1 LEADER]: Becoming Leader. State: Replica: afb2c4f9d2c44f52a143891a6372308c, State: Running, Role: LEADER
I20260709 12:31:34.927541 20569 sys_catalog.cc:565] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:31:34.927547 20572 consensus_queue.cc:237] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb2c4f9d2c44f52a143891a6372308c" member_type: VOTER }
I20260709 12:31:34.934406 20573 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afb2c4f9d2c44f52a143891a6372308c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb2c4f9d2c44f52a143891a6372308c" member_type: VOTER } }
I20260709 12:31:34.934418 20574 sys_catalog.cc:455] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [sys.catalog]: SysCatalogTable state changed. Reason: New leader afb2c4f9d2c44f52a143891a6372308c. Latest consensus state: current_term: 1 leader_uuid: "afb2c4f9d2c44f52a143891a6372308c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afb2c4f9d2c44f52a143891a6372308c" member_type: VOTER } }
I20260709 12:31:34.935400 20573 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [sys.catalog]: This master's current role is: LEADER
I20260709 12:31:34.935501 20574 sys_catalog.cc:458] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [sys.catalog]: This master's current role is: LEADER
I20260709 12:31:34.937496 20581 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:31:34.945708 20581 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:31:34.946902  4356 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:31:34.956146 20581 catalog_manager.cc:1383] Generated new cluster ID: 5cca2d8f5827440fa0cf77eca7a391ca
I20260709 12:31:34.956494 20581 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:31:34.986608 20581 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:31:34.988230 20581 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:31:35.003759 20581 catalog_manager.cc:6070] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c: Generated new TSK 0
I20260709 12:31:35.004737 20581 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:31:35.016757  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:31:35.023319 20590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:35.023820 20591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:35.026140 20593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:31:35.028007  4356 server_base.cc:1061] running on GCE node
I20260709 12:31:35.029383  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:31:35.029691  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:31:35.029979  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600295029941 us; error 0 us; skew 500 ppm
I20260709 12:31:35.030628  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:31:35.033243  4356 webserver.cc:533] Webserver started at http://127.4.65.1:42067/ using document root <none> and password file <none>
I20260709 12:31:35.033700  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:31:35.033865  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:31:35.034077  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:31:35.035180  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-0-root/instance:
uuid: "d57571da3cac40fc8c10f2ff9e98e9c1"
format_stamp: "Formatted at 2026-07-09 12:31:35 on dist-test-slave-l1c9"
I20260709 12:31:35.040798  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20260709 12:31:35.044561 20598 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:35.045648  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:31:35.045984  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-0-root
uuid: "d57571da3cac40fc8c10f2ff9e98e9c1"
format_stamp: "Formatted at 2026-07-09 12:31:35 on dist-test-slave-l1c9"
I20260709 12:31:35.046301  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:31:35.058493  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:31:35.059762  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:31:35.061548  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:31:35.064244  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:31:35.064508  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20260709 12:31:35.064772  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:31:35.064965  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:35.111594  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.1:43287
I20260709 12:31:35.111713 20660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.1:43287 every 8 connection(s)
I20260709 12:31:35.117296  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:31:35.125046 20665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:35.126928 20666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:35.130272 20668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:31:35.130548 20661 heartbeater.cc:344] Connected to a master server at 127.4.65.62:42141
I20260709 12:31:35.130631  4356 server_base.cc:1061] running on GCE node
I20260709 12:31:35.131413 20661 heartbeater.cc:461] Registering TS with master...
I20260709 12:31:35.132485 20661 heartbeater.cc:507] Master 127.4.65.62:42141 requested a full tablet report, sending...
I20260709 12:31:35.132513  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:31:35.132946  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:31:35.133162  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600295133146 us; error 0 us; skew 500 ppm
I20260709 12:31:35.133920  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:31:35.135160 20534 ts_manager.cc:194] Registered new tserver with Master: d57571da3cac40fc8c10f2ff9e98e9c1 (127.4.65.1:43287)
I20260709 12:31:35.136752  4356 webserver.cc:533] Webserver started at http://127.4.65.2:41395/ using document root <none> and password file <none>
I20260709 12:31:35.137326  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:31:35.137542  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:31:35.137508 20534 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56102
I20260709 12:31:35.138175  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:31:35.139825  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-1-root/instance:
uuid: "80164889ef444f7baf4af798960d957b"
format_stamp: "Formatted at 2026-07-09 12:31:35 on dist-test-slave-l1c9"
I20260709 12:31:35.145169  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.003s
I20260709 12:31:35.149374 20673 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:35.150496  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20260709 12:31:35.150887  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-1-root
uuid: "80164889ef444f7baf4af798960d957b"
format_stamp: "Formatted at 2026-07-09 12:31:35 on dist-test-slave-l1c9"
I20260709 12:31:35.151153  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:31:35.162776  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:31:35.164312  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:31:35.166106  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:31:35.168720  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:31:35.168912  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:35.169184  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:31:35.169409  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:35.214502  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.2:38971
I20260709 12:31:35.214617 20735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.2:38971 every 8 connection(s)
I20260709 12:31:35.219807  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:31:35.227044 20739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:35.229180 20740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:35.230667 20742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:31:35.231343  4356 server_base.cc:1061] running on GCE node
I20260709 12:31:35.232398 20736 heartbeater.cc:344] Connected to a master server at 127.4.65.62:42141
I20260709 12:31:35.232530  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 12:31:35.232825 20736 heartbeater.cc:461] Registering TS with master...
W20260709 12:31:35.232897  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:31:35.233310  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600295233286 us; error 0 us; skew 500 ppm
I20260709 12:31:35.233932  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:31:35.233882 20736 heartbeater.cc:507] Master 127.4.65.62:42141 requested a full tablet report, sending...
I20260709 12:31:35.236840 20534 ts_manager.cc:194] Registered new tserver with Master: 80164889ef444f7baf4af798960d957b (127.4.65.2:38971)
I20260709 12:31:35.237490  4356 webserver.cc:533] Webserver started at http://127.4.65.3:37271/ using document root <none> and password file <none>
I20260709 12:31:35.238188  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:31:35.238425  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:31:35.238750  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:31:35.239053 20534 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56118
I20260709 12:31:35.240322  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-2-root/instance:
uuid: "f9fc9dfee3bd4478a587de49d7bc3789"
format_stamp: "Formatted at 2026-07-09 12:31:35 on dist-test-slave-l1c9"
I20260709 12:31:35.245301  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20260709 12:31:35.249960 20747 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:35.250903  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:31:35.251204  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-2-root
uuid: "f9fc9dfee3bd4478a587de49d7bc3789"
format_stamp: "Formatted at 2026-07-09 12:31:35 on dist-test-slave-l1c9"
I20260709 12:31:35.251464  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599783304823-4356-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:31:35.277550  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:31:35.279032  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:31:35.280696  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:31:35.283089  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:31:35.283296  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:35.283629  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:31:35.283785  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:35.328197  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.3:37457
I20260709 12:31:35.328342 20809 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.3:37457 every 8 connection(s)
I20260709 12:31:35.343888 20810 heartbeater.cc:344] Connected to a master server at 127.4.65.62:42141
I20260709 12:31:35.344477 20810 heartbeater.cc:461] Registering TS with master...
I20260709 12:31:35.345460 20810 heartbeater.cc:507] Master 127.4.65.62:42141 requested a full tablet report, sending...
I20260709 12:31:35.347802 20534 ts_manager.cc:194] Registered new tserver with Master: f9fc9dfee3bd4478a587de49d7bc3789 (127.4.65.3:37457)
I20260709 12:31:35.348265  4356 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016636021s
I20260709 12:31:35.349574 20534 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56120
I20260709 12:31:35.378702 20534 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56126:
name: "multi_table_leader_balance_table1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:31:35.381321 20534 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:31:35.430519 20775 tablet_service.cc:1511] Processing CreateTablet for tablet addee36dfbd94773bd88c7b367e1a3df (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=641ba5c8c1b642868d3b26109126adea]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:31:35.431999 20775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addee36dfbd94773bd88c7b367e1a3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:35.435245 20774 tablet_service.cc:1511] Processing CreateTablet for tablet e69fbefee0a64f58b5b69ac2d087f5fe (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=641ba5c8c1b642868d3b26109126adea]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:31:35.436892 20774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e69fbefee0a64f58b5b69ac2d087f5fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:35.436946 20700 tablet_service.cc:1511] Processing CreateTablet for tablet e69fbefee0a64f58b5b69ac2d087f5fe (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=641ba5c8c1b642868d3b26109126adea]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:31:35.438444 20700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e69fbefee0a64f58b5b69ac2d087f5fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:35.441085 20626 tablet_service.cc:1511] Processing CreateTablet for tablet addee36dfbd94773bd88c7b367e1a3df (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=641ba5c8c1b642868d3b26109126adea]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:31:35.441463 20701 tablet_service.cc:1511] Processing CreateTablet for tablet addee36dfbd94773bd88c7b367e1a3df (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=641ba5c8c1b642868d3b26109126adea]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:31:35.441833 20625 tablet_service.cc:1511] Processing CreateTablet for tablet e69fbefee0a64f58b5b69ac2d087f5fe (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=641ba5c8c1b642868d3b26109126adea]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:31:35.442612 20626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addee36dfbd94773bd88c7b367e1a3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:35.442803 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet addee36dfbd94773bd88c7b367e1a3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:35.445786 20625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e69fbefee0a64f58b5b69ac2d087f5fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:35.459038 20825 tablet_bootstrap.cc:492] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789: Bootstrap starting.
I20260709 12:31:35.468822 20826 tablet_bootstrap.cc:492] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1: Bootstrap starting.
I20260709 12:31:35.467288 20825 tablet_bootstrap.cc:654] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:35.475824 20826 tablet_bootstrap.cc:654] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:35.478981 20828 tablet_bootstrap.cc:492] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b: Bootstrap starting.
I20260709 12:31:35.487397 20828 tablet_bootstrap.cc:654] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:35.494495 20826 tablet_bootstrap.cc:492] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1: No bootstrap required, opened a new log
I20260709 12:31:35.494568 20828 tablet_bootstrap.cc:492] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b: No bootstrap required, opened a new log
I20260709 12:31:35.494805 20825 tablet_bootstrap.cc:492] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789: No bootstrap required, opened a new log
I20260709 12:31:35.495090 20828 ts_tablet_manager.cc:1403] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b: Time spent bootstrapping tablet: real 0.016s	user 0.014s	sys 0.000s
I20260709 12:31:35.495086 20826 ts_tablet_manager.cc:1403] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1: Time spent bootstrapping tablet: real 0.027s	user 0.018s	sys 0.000s
I20260709 12:31:35.495335 20825 ts_tablet_manager.cc:1403] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789: Time spent bootstrapping tablet: real 0.037s	user 0.025s	sys 0.000s
I20260709 12:31:35.498564 20826 raft_consensus.cc:359] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } } peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } }
I20260709 12:31:35.498688 20828 raft_consensus.cc:359] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:35.498859 20825 raft_consensus.cc:359] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } } peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } }
I20260709 12:31:35.499388 20826 raft_consensus.cc:385] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:35.499444 20828 raft_consensus.cc:385] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:35.499662 20825 raft_consensus.cc:385] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:35.499853 20826 raft_consensus.cc:740] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57571da3cac40fc8c10f2ff9e98e9c1, State: Initialized, Role: FOLLOWER
I20260709 12:31:35.499912 20828 raft_consensus.cc:740] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80164889ef444f7baf4af798960d957b, State: Initialized, Role: FOLLOWER
I20260709 12:31:35.500762 20825 raft_consensus.cc:740] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9fc9dfee3bd4478a587de49d7bc3789, State: Initialized, Role: FOLLOWER
I20260709 12:31:35.501758 20825 consensus_queue.cc:260] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } } peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } }
I20260709 12:31:35.501812 20826 consensus_queue.cc:260] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } } peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } }
I20260709 12:31:35.501870 20828 consensus_queue.cc:260] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:35.506896 20825 ts_tablet_manager.cc:1434] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789: Time spent starting tablet: real 0.011s	user 0.001s	sys 0.008s
I20260709 12:31:35.507503 20736 heartbeater.cc:499] Master 127.4.65.62:42141 was elected leader, sending a full tablet report...
I20260709 12:31:35.507759 20828 ts_tablet_manager.cc:1434] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.000s
I20260709 12:31:35.508106 20825 tablet_bootstrap.cc:492] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789: Bootstrap starting.
I20260709 12:31:35.509199 20828 tablet_bootstrap.cc:492] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b: Bootstrap starting.
I20260709 12:31:35.507560 20810 heartbeater.cc:499] Master 127.4.65.62:42141 was elected leader, sending a full tablet report...
I20260709 12:31:35.515934 20661 heartbeater.cc:499] Master 127.4.65.62:42141 was elected leader, sending a full tablet report...
I20260709 12:31:35.517154 20826 ts_tablet_manager.cc:1434] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1: Time spent starting tablet: real 0.021s	user 0.009s	sys 0.005s
I20260709 12:31:35.517648 20828 tablet_bootstrap.cc:654] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:35.518339 20826 tablet_bootstrap.cc:492] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1: Bootstrap starting.
I20260709 12:31:35.516072 20825 tablet_bootstrap.cc:654] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:35.527408 20826 tablet_bootstrap.cc:654] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:35.528831 20828 tablet_bootstrap.cc:492] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b: No bootstrap required, opened a new log
I20260709 12:31:35.529400 20828 ts_tablet_manager.cc:1403] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b: Time spent bootstrapping tablet: real 0.020s	user 0.015s	sys 0.000s
I20260709 12:31:35.532538 20828 raft_consensus.cc:359] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } } peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } }
I20260709 12:31:35.533396 20828 raft_consensus.cc:385] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:35.533701 20828 raft_consensus.cc:740] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80164889ef444f7baf4af798960d957b, State: Initialized, Role: FOLLOWER
I20260709 12:31:35.534775 20826 tablet_bootstrap.cc:492] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1: No bootstrap required, opened a new log
I20260709 12:31:35.534519 20828 consensus_queue.cc:260] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } } peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } }
I20260709 12:31:35.535471 20826 ts_tablet_manager.cc:1403] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1: Time spent bootstrapping tablet: real 0.017s	user 0.009s	sys 0.007s
I20260709 12:31:35.538501 20825 tablet_bootstrap.cc:492] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789: No bootstrap required, opened a new log
I20260709 12:31:35.539093 20825 ts_tablet_manager.cc:1403] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789: Time spent bootstrapping tablet: real 0.031s	user 0.008s	sys 0.004s
I20260709 12:31:35.539412 20826 raft_consensus.cc:359] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:35.540248 20826 raft_consensus.cc:385] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:35.540627 20826 raft_consensus.cc:740] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57571da3cac40fc8c10f2ff9e98e9c1, State: Initialized, Role: FOLLOWER
I20260709 12:31:35.541605 20826 consensus_queue.cc:260] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:35.542361 20825 raft_consensus.cc:359] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:35.543129 20825 raft_consensus.cc:385] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:35.543429 20825 raft_consensus.cc:740] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9fc9dfee3bd4478a587de49d7bc3789, State: Initialized, Role: FOLLOWER
I20260709 12:31:35.544191 20826 ts_tablet_manager.cc:1434] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:31:35.544298 20825 consensus_queue.cc:260] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:35.546406 20825 ts_tablet_manager.cc:1434] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:31:35.548398 20828 ts_tablet_manager.cc:1434] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260709 12:31:35.721743 20832 raft_consensus.cc:493] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:35.722308 20832 raft_consensus.cc:515] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } } peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } }
I20260709 12:31:35.724568 20832 leader_election.cc:290] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9fc9dfee3bd4478a587de49d7bc3789 (127.4.65.3:37457), 80164889ef444f7baf4af798960d957b (127.4.65.2:38971)
I20260709 12:31:35.736141 20785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addee36dfbd94773bd88c7b367e1a3df" candidate_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" is_pre_election: true
I20260709 12:31:35.737035 20785 raft_consensus.cc:2468] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d57571da3cac40fc8c10f2ff9e98e9c1 in term 0.
I20260709 12:31:35.737746 20711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addee36dfbd94773bd88c7b367e1a3df" candidate_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80164889ef444f7baf4af798960d957b" is_pre_election: true
I20260709 12:31:35.738485 20711 raft_consensus.cc:2468] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d57571da3cac40fc8c10f2ff9e98e9c1 in term 0.
I20260709 12:31:35.738396 20602 leader_election.cc:304] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d57571da3cac40fc8c10f2ff9e98e9c1, f9fc9dfee3bd4478a587de49d7bc3789; no voters: 
I20260709 12:31:35.739898 20832 raft_consensus.cc:2804] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:35.740512 20832 raft_consensus.cc:493] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:35.740869 20832 raft_consensus.cc:3060] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:35.746150 20832 raft_consensus.cc:515] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } } peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } }
I20260709 12:31:35.747920 20832 leader_election.cc:290] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [CANDIDATE]: Term 1 election: Requested vote from peers f9fc9dfee3bd4478a587de49d7bc3789 (127.4.65.3:37457), 80164889ef444f7baf4af798960d957b (127.4.65.2:38971)
I20260709 12:31:35.749079 20785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addee36dfbd94773bd88c7b367e1a3df" candidate_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9fc9dfee3bd4478a587de49d7bc3789"
I20260709 12:31:35.749372 20711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "addee36dfbd94773bd88c7b367e1a3df" candidate_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80164889ef444f7baf4af798960d957b"
I20260709 12:31:35.750062 20711 raft_consensus.cc:3060] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:35.750077 20785 raft_consensus.cc:3060] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:35.758538 20711 raft_consensus.cc:2468] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d57571da3cac40fc8c10f2ff9e98e9c1 in term 1.
I20260709 12:31:35.758538 20785 raft_consensus.cc:2468] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d57571da3cac40fc8c10f2ff9e98e9c1 in term 1.
I20260709 12:31:35.760027 20602 leader_election.cc:304] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d57571da3cac40fc8c10f2ff9e98e9c1, f9fc9dfee3bd4478a587de49d7bc3789; no voters: 
I20260709 12:31:35.760783 20832 raft_consensus.cc:2804] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:35.761718 20832 raft_consensus.cc:697] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 LEADER]: Becoming Leader. State: Replica: d57571da3cac40fc8c10f2ff9e98e9c1, State: Running, Role: LEADER
I20260709 12:31:35.762604 20832 consensus_queue.cc:237] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } } peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } }
I20260709 12:31:35.772187 20532 catalog_manager.cc:5697] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d57571da3cac40fc8c10f2ff9e98e9c1 (127.4.65.1). New cstate: current_term: 1 leader_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:35.791566 20833 raft_consensus.cc:493] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:35.792222 20833 raft_consensus.cc:515] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:35.794873 20833 leader_election.cc:290] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d57571da3cac40fc8c10f2ff9e98e9c1 (127.4.65.1:43287), f9fc9dfee3bd4478a587de49d7bc3789 (127.4.65.3:37457)
I20260709 12:31:35.805287 20636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69fbefee0a64f58b5b69ac2d087f5fe" candidate_uuid: "80164889ef444f7baf4af798960d957b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" is_pre_election: true
I20260709 12:31:35.805898 20785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69fbefee0a64f58b5b69ac2d087f5fe" candidate_uuid: "80164889ef444f7baf4af798960d957b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" is_pre_election: true
I20260709 12:31:35.806140 20636 raft_consensus.cc:2468] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80164889ef444f7baf4af798960d957b in term 0.
I20260709 12:31:35.806591 20785 raft_consensus.cc:2468] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80164889ef444f7baf4af798960d957b in term 0.
I20260709 12:31:35.807808 20674 leader_election.cc:304] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80164889ef444f7baf4af798960d957b, d57571da3cac40fc8c10f2ff9e98e9c1; no voters: 
I20260709 12:31:35.808739 20833 raft_consensus.cc:2804] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:35.809214 20833 raft_consensus.cc:493] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:35.809640 20833 raft_consensus.cc:3060] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:35.816996 20833 raft_consensus.cc:515] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:35.819105 20833 leader_election.cc:290] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [CANDIDATE]: Term 1 election: Requested vote from peers d57571da3cac40fc8c10f2ff9e98e9c1 (127.4.65.1:43287), f9fc9dfee3bd4478a587de49d7bc3789 (127.4.65.3:37457)
I20260709 12:31:35.820263 20636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69fbefee0a64f58b5b69ac2d087f5fe" candidate_uuid: "80164889ef444f7baf4af798960d957b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1"
I20260709 12:31:35.820286 20785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e69fbefee0a64f58b5b69ac2d087f5fe" candidate_uuid: "80164889ef444f7baf4af798960d957b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9fc9dfee3bd4478a587de49d7bc3789"
I20260709 12:31:35.820909 20636 raft_consensus.cc:3060] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:35.821135 20785 raft_consensus.cc:3060] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:35.826817 20785 raft_consensus.cc:2468] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80164889ef444f7baf4af798960d957b in term 1.
I20260709 12:31:35.827363 20636 raft_consensus.cc:2468] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80164889ef444f7baf4af798960d957b in term 1.
I20260709 12:31:35.828140 20677 leader_election.cc:304] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80164889ef444f7baf4af798960d957b, f9fc9dfee3bd4478a587de49d7bc3789; no voters: 
I20260709 12:31:35.829238 20833 raft_consensus.cc:2804] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:35.830739 20833 raft_consensus.cc:697] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [term 1 LEADER]: Becoming Leader. State: Replica: 80164889ef444f7baf4af798960d957b, State: Running, Role: LEADER
I20260709 12:31:35.831585 20833 consensus_queue.cc:237] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:35.839581 20532 catalog_manager.cc:5697] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b reported cstate change: term changed from 0 to 1, leader changed from <none> to 80164889ef444f7baf4af798960d957b (127.4.65.2). New cstate: current_term: 1 leader_uuid: "80164889ef444f7baf4af798960d957b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:35.961956 20532 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56130:
name: "multi_table_leader_balance_table2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:31:35.963987 20532 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:31:35.992329 20625 tablet_service.cc:1511] Processing CreateTablet for tablet dd7f0aa14fd649138d3566b7da01786a (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=030a2072492c47a7b382713e616d9174]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:31:35.993211 20701 tablet_service.cc:1511] Processing CreateTablet for tablet dd7f0aa14fd649138d3566b7da01786a (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=030a2072492c47a7b382713e616d9174]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:31:35.993772 20625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7f0aa14fd649138d3566b7da01786a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:35.994629 20701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7f0aa14fd649138d3566b7da01786a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:35.997423 20774 tablet_service.cc:1511] Processing CreateTablet for tablet dd7f0aa14fd649138d3566b7da01786a (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=030a2072492c47a7b382713e616d9174]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:31:35.997968 20700 tablet_service.cc:1511] Processing CreateTablet for tablet 9e510f7834bc49239d921738ee09360e (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=030a2072492c47a7b382713e616d9174]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:31:35.998826 20774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd7f0aa14fd649138d3566b7da01786a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:35.999346 20700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e510f7834bc49239d921738ee09360e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:36.000259 20626 tablet_service.cc:1511] Processing CreateTablet for tablet 9e510f7834bc49239d921738ee09360e (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=030a2072492c47a7b382713e616d9174]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:31:36.001546 20626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e510f7834bc49239d921738ee09360e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:36.010982 20775 tablet_service.cc:1511] Processing CreateTablet for tablet 9e510f7834bc49239d921738ee09360e (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=030a2072492c47a7b382713e616d9174]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:31:36.012393 20775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e510f7834bc49239d921738ee09360e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:36.014094 20826 tablet_bootstrap.cc:492] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1: Bootstrap starting.
I20260709 12:31:36.018121 20828 tablet_bootstrap.cc:492] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b: Bootstrap starting.
I20260709 12:31:36.022828 20826 tablet_bootstrap.cc:654] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:36.025502 20828 tablet_bootstrap.cc:654] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:36.036479 20825 tablet_bootstrap.cc:492] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789: Bootstrap starting.
I20260709 12:31:36.046696 20825 tablet_bootstrap.cc:654] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:36.047412 20826 tablet_bootstrap.cc:492] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1: No bootstrap required, opened a new log
I20260709 12:31:36.047724 20828 tablet_bootstrap.cc:492] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b: No bootstrap required, opened a new log
I20260709 12:31:36.048079 20826 ts_tablet_manager.cc:1403] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1: Time spent bootstrapping tablet: real 0.034s	user 0.012s	sys 0.007s
I20260709 12:31:36.048310 20828 ts_tablet_manager.cc:1403] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b: Time spent bootstrapping tablet: real 0.030s	user 0.011s	sys 0.013s
I20260709 12:31:36.050797 20828 raft_consensus.cc:359] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.051378 20828 raft_consensus.cc:385] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:36.051817 20826 raft_consensus.cc:359] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.052536 20826 raft_consensus.cc:385] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:36.052326 20828 raft_consensus.cc:740] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80164889ef444f7baf4af798960d957b, State: Initialized, Role: FOLLOWER
I20260709 12:31:36.053015 20826 raft_consensus.cc:740] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57571da3cac40fc8c10f2ff9e98e9c1, State: Initialized, Role: FOLLOWER
I20260709 12:31:36.053758 20828 consensus_queue.cc:260] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.054003 20826 consensus_queue.cc:260] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.054584 20825 tablet_bootstrap.cc:492] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789: No bootstrap required, opened a new log
I20260709 12:31:36.055766 20825 ts_tablet_manager.cc:1403] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789: Time spent bootstrapping tablet: real 0.020s	user 0.012s	sys 0.004s
I20260709 12:31:36.056447 20828 ts_tablet_manager.cc:1434] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:31:36.057621 20828 tablet_bootstrap.cc:492] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b: Bootstrap starting.
I20260709 12:31:36.059073 20825 raft_consensus.cc:359] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.059922 20825 raft_consensus.cc:385] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:36.060295 20825 raft_consensus.cc:740] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9fc9dfee3bd4478a587de49d7bc3789, State: Initialized, Role: FOLLOWER
I20260709 12:31:36.061177 20825 consensus_queue.cc:260] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.066422 20828 tablet_bootstrap.cc:654] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:36.067380 20826 ts_tablet_manager.cc:1434] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1: Time spent starting tablet: real 0.019s	user 0.001s	sys 0.006s
I20260709 12:31:36.068181 20825 ts_tablet_manager.cc:1434] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.004s
I20260709 12:31:36.069435 20825 tablet_bootstrap.cc:492] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789: Bootstrap starting.
I20260709 12:31:36.071077 20826 tablet_bootstrap.cc:492] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1: Bootstrap starting.
I20260709 12:31:36.078472 20826 tablet_bootstrap.cc:654] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:36.078533 20825 tablet_bootstrap.cc:654] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:36.093343 20825 tablet_bootstrap.cc:492] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789: No bootstrap required, opened a new log
I20260709 12:31:36.094153 20825 ts_tablet_manager.cc:1403] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789: Time spent bootstrapping tablet: real 0.025s	user 0.009s	sys 0.015s
I20260709 12:31:36.099699 20825 raft_consensus.cc:359] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.100842 20825 raft_consensus.cc:385] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:36.101306 20825 raft_consensus.cc:740] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9fc9dfee3bd4478a587de49d7bc3789, State: Initialized, Role: FOLLOWER
I20260709 12:31:36.102612 20825 consensus_queue.cc:260] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.104915 20826 tablet_bootstrap.cc:492] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1: No bootstrap required, opened a new log
I20260709 12:31:36.105038 20825 ts_tablet_manager.cc:1434] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:31:36.105093 20828 tablet_bootstrap.cc:492] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b: No bootstrap required, opened a new log
I20260709 12:31:36.105708 20826 ts_tablet_manager.cc:1403] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1: Time spent bootstrapping tablet: real 0.035s	user 0.024s	sys 0.010s
I20260709 12:31:36.106081 20828 ts_tablet_manager.cc:1403] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b: Time spent bootstrapping tablet: real 0.049s	user 0.026s	sys 0.019s
I20260709 12:31:36.108198 20826 raft_consensus.cc:359] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.108892 20826 raft_consensus.cc:385] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:36.109157 20826 raft_consensus.cc:740] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57571da3cac40fc8c10f2ff9e98e9c1, State: Initialized, Role: FOLLOWER
I20260709 12:31:36.109139 20828 raft_consensus.cc:359] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.110034 20828 raft_consensus.cc:385] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:36.110355 20828 raft_consensus.cc:740] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80164889ef444f7baf4af798960d957b, State: Initialized, Role: FOLLOWER
I20260709 12:31:36.110386 20826 consensus_queue.cc:260] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.111308 20828 consensus_queue.cc:260] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.112967 20826 ts_tablet_manager.cc:1434] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.002s
I20260709 12:31:36.114684 20828 ts_tablet_manager.cc:1434] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:31:36.190196 20833 raft_consensus.cc:493] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:36.190704 20833 raft_consensus.cc:515] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.192687 20833 leader_election.cc:290] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d57571da3cac40fc8c10f2ff9e98e9c1 (127.4.65.1:43287), f9fc9dfee3bd4478a587de49d7bc3789 (127.4.65.3:37457)
I20260709 12:31:36.193533 20636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd7f0aa14fd649138d3566b7da01786a" candidate_uuid: "80164889ef444f7baf4af798960d957b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" is_pre_election: true
I20260709 12:31:36.193786 20785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd7f0aa14fd649138d3566b7da01786a" candidate_uuid: "80164889ef444f7baf4af798960d957b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" is_pre_election: true
I20260709 12:31:36.194265 20636 raft_consensus.cc:2468] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80164889ef444f7baf4af798960d957b in term 0.
I20260709 12:31:36.194602 20785 raft_consensus.cc:2468] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80164889ef444f7baf4af798960d957b in term 0.
I20260709 12:31:36.195449 20674 leader_election.cc:304] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80164889ef444f7baf4af798960d957b, d57571da3cac40fc8c10f2ff9e98e9c1; no voters: 
I20260709 12:31:36.196382 20833 raft_consensus.cc:2804] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:36.196743 20833 raft_consensus.cc:493] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:36.197036 20833 raft_consensus.cc:3060] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:36.202313 20833 raft_consensus.cc:515] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.204005 20833 leader_election.cc:290] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [CANDIDATE]: Term 1 election: Requested vote from peers d57571da3cac40fc8c10f2ff9e98e9c1 (127.4.65.1:43287), f9fc9dfee3bd4478a587de49d7bc3789 (127.4.65.3:37457)
I20260709 12:31:36.205085 20785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd7f0aa14fd649138d3566b7da01786a" candidate_uuid: "80164889ef444f7baf4af798960d957b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9fc9dfee3bd4478a587de49d7bc3789"
I20260709 12:31:36.205039 20636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd7f0aa14fd649138d3566b7da01786a" candidate_uuid: "80164889ef444f7baf4af798960d957b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1"
I20260709 12:31:36.206007 20785 raft_consensus.cc:3060] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:36.206106 20636 raft_consensus.cc:3060] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:36.211198 20636 raft_consensus.cc:2468] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80164889ef444f7baf4af798960d957b in term 1.
I20260709 12:31:36.211380 20785 raft_consensus.cc:2468] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80164889ef444f7baf4af798960d957b in term 1.
I20260709 12:31:36.212368 20674 leader_election.cc:304] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80164889ef444f7baf4af798960d957b, d57571da3cac40fc8c10f2ff9e98e9c1; no voters: 
I20260709 12:31:36.213165 20833 raft_consensus.cc:2804] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:36.213629 20833 raft_consensus.cc:697] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 1 LEADER]: Becoming Leader. State: Replica: 80164889ef444f7baf4af798960d957b, State: Running, Role: LEADER
I20260709 12:31:36.214366 20833 consensus_queue.cc:237] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.222556 20534 catalog_manager.cc:5697] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b reported cstate change: term changed from 0 to 1, leader changed from <none> to 80164889ef444f7baf4af798960d957b (127.4.65.2). New cstate: current_term: 1 leader_uuid: "80164889ef444f7baf4af798960d957b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:36.280561 20832 consensus_queue.cc:1048] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:36.294724 20832 consensus_queue.cc:1048] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:36.302378 20832 raft_consensus.cc:493] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:36.303005 20832 raft_consensus.cc:515] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.306078 20832 leader_election.cc:290] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80164889ef444f7baf4af798960d957b (127.4.65.2:38971), f9fc9dfee3bd4478a587de49d7bc3789 (127.4.65.3:37457)
I20260709 12:31:36.307135 20711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e510f7834bc49239d921738ee09360e" candidate_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80164889ef444f7baf4af798960d957b" is_pre_election: true
I20260709 12:31:36.307111 20785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e510f7834bc49239d921738ee09360e" candidate_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" is_pre_election: true
I20260709 12:31:36.308115 20711 raft_consensus.cc:2468] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d57571da3cac40fc8c10f2ff9e98e9c1 in term 0.
I20260709 12:31:36.308183 20785 raft_consensus.cc:2468] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d57571da3cac40fc8c10f2ff9e98e9c1 in term 0.
I20260709 12:31:36.309572 20602 leader_election.cc:304] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d57571da3cac40fc8c10f2ff9e98e9c1, f9fc9dfee3bd4478a587de49d7bc3789; no voters: 
I20260709 12:31:36.310369 20832 raft_consensus.cc:2804] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:36.310767 20832 raft_consensus.cc:493] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:36.311158 20832 raft_consensus.cc:3060] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:36.316519 20832 raft_consensus.cc:515] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.318145 20832 leader_election.cc:290] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [CANDIDATE]: Term 1 election: Requested vote from peers 80164889ef444f7baf4af798960d957b (127.4.65.2:38971), f9fc9dfee3bd4478a587de49d7bc3789 (127.4.65.3:37457)
I20260709 12:31:36.319653 20711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e510f7834bc49239d921738ee09360e" candidate_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80164889ef444f7baf4af798960d957b"
I20260709 12:31:36.320065 20785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e510f7834bc49239d921738ee09360e" candidate_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9fc9dfee3bd4478a587de49d7bc3789"
I20260709 12:31:36.320331 20711 raft_consensus.cc:3060] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:36.321029 20785 raft_consensus.cc:3060] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:36.326663 20711 raft_consensus.cc:2468] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d57571da3cac40fc8c10f2ff9e98e9c1 in term 1.
I20260709 12:31:36.327384 20785 raft_consensus.cc:2468] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d57571da3cac40fc8c10f2ff9e98e9c1 in term 1.
I20260709 12:31:36.328099 20602 leader_election.cc:304] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80164889ef444f7baf4af798960d957b, d57571da3cac40fc8c10f2ff9e98e9c1; no voters: 
I20260709 12:31:36.329119 20832 raft_consensus.cc:2804] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:36.329708 20832 raft_consensus.cc:697] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 LEADER]: Becoming Leader. State: Replica: d57571da3cac40fc8c10f2ff9e98e9c1, State: Running, Role: LEADER
I20260709 12:31:36.330757 20832 consensus_queue.cc:237] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.338780 20533 catalog_manager.cc:5697] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d57571da3cac40fc8c10f2ff9e98e9c1 (127.4.65.1). New cstate: current_term: 1 leader_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:36.367792 20833 consensus_queue.cc:1048] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:36.379608 20867 consensus_queue.cc:1048] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:36.437058 20711 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dd7f0aa14fd649138d3566b7da01786a"
dest_uuid: "80164889ef444f7baf4af798960d957b"
mode: GRACEFUL
new_leader_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1"
 from {username='slave'} at 127.0.0.1:58860
I20260709 12:31:36.437783 20711 raft_consensus.cc:606] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 1 LEADER]: Received request to transfer leadership to TS d57571da3cac40fc8c10f2ff9e98e9c1
I20260709 12:31:36.702962 20867 consensus_queue.cc:1048] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:36.713438 20843 consensus_queue.cc:1048] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:36.721979 20867 raft_consensus.cc:993] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b: : Instructing follower d57571da3cac40fc8c10f2ff9e98e9c1 to start an election
I20260709 12:31:36.722538 20843 raft_consensus.cc:1081] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 1 LEADER]: Signalling peer d57571da3cac40fc8c10f2ff9e98e9c1 to start an election
I20260709 12:31:36.724104 20636 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dd7f0aa14fd649138d3566b7da01786a"
dest_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1"
 from {username='slave'} at 127.0.0.1:40588
I20260709 12:31:36.725873 20871 consensus_queue.cc:1048] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:36.726620 20636 raft_consensus.cc:493] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:36.727034 20636 raft_consensus.cc:3060] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:36.734967 20636 raft_consensus.cc:515] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.737655 20636 leader_election.cc:290] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [CANDIDATE]: Term 2 election: Requested vote from peers 80164889ef444f7baf4af798960d957b (127.4.65.2:38971), f9fc9dfee3bd4478a587de49d7bc3789 (127.4.65.3:37457)
I20260709 12:31:36.737900 20710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd7f0aa14fd649138d3566b7da01786a" candidate_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "80164889ef444f7baf4af798960d957b"
I20260709 12:31:36.738664 20710 raft_consensus.cc:3055] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:36.738811 20784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dd7f0aa14fd649138d3566b7da01786a" candidate_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9fc9dfee3bd4478a587de49d7bc3789"
I20260709 12:31:36.739061 20710 raft_consensus.cc:740] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 80164889ef444f7baf4af798960d957b, State: Running, Role: LEADER
I20260709 12:31:36.739501 20784 raft_consensus.cc:3060] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:36.739957 20710 consensus_queue.cc:260] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.741448 20710 raft_consensus.cc:3060] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:36.744493 20832 consensus_queue.cc:1048] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:36.747956 20784 raft_consensus.cc:2468] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d57571da3cac40fc8c10f2ff9e98e9c1 in term 2.
I20260709 12:31:36.749586 20710 raft_consensus.cc:2468] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d57571da3cac40fc8c10f2ff9e98e9c1 in term 2.
I20260709 12:31:36.750833 20602 leader_election.cc:304] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d57571da3cac40fc8c10f2ff9e98e9c1, f9fc9dfee3bd4478a587de49d7bc3789; no voters: 
I20260709 12:31:36.751549 20832 raft_consensus.cc:2804] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:36.752023 20832 raft_consensus.cc:697] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [term 2 LEADER]: Becoming Leader. State: Replica: d57571da3cac40fc8c10f2ff9e98e9c1, State: Running, Role: LEADER
I20260709 12:31:36.753067 20832 consensus_queue.cc:237] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:36.760073 20534 catalog_manager.cc:5697] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 reported cstate change: term changed from 1 to 2, leader changed from 80164889ef444f7baf4af798960d957b (127.4.65.2) to d57571da3cac40fc8c10f2ff9e98e9c1 (127.4.65.1). New cstate: current_term: 2 leader_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:36.778347  4356 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table1
I20260709 12:31:36.780014  4356 auto_leader_rebalancer.cc:440] table: multi_table_leader_balance_table1, leader rebalance finish, leader transfer count: 0
I20260709 12:31:36.780467  4356 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table2
I20260709 12:31:36.794453 20636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9e510f7834bc49239d921738ee09360e"
dest_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1"
mode: GRACEFUL
new_leader_uuid: "f9fc9dfee3bd4478a587de49d7bc3789"
 from {username='slave'} at 127.0.0.1:40598
I20260709 12:31:36.795030 20636 raft_consensus.cc:606] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 LEADER]: Received request to transfer leadership to TS f9fc9dfee3bd4478a587de49d7bc3789
I20260709 12:31:36.796285  4356 auto_leader_rebalancer.cc:440] table: multi_table_leader_balance_table2, leader rebalance finish, leader transfer count: 1
I20260709 12:31:37.153702 20785 raft_consensus.cc:1275] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789 [term 2 FOLLOWER]: Refusing update from remote peer d57571da3cac40fc8c10f2ff9e98e9c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:37.155336 20832 consensus_queue.cc:1048] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:37.171993 20711 raft_consensus.cc:1275] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 2 FOLLOWER]: Refusing update from remote peer d57571da3cac40fc8c10f2ff9e98e9c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:37.174599 20860 consensus_queue.cc:1048] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:31:37.233327 20878 raft_consensus.cc:993] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1: : Instructing follower f9fc9dfee3bd4478a587de49d7bc3789 to start an election
I20260709 12:31:37.233834 20860 raft_consensus.cc:1081] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 LEADER]: Signalling peer f9fc9dfee3bd4478a587de49d7bc3789 to start an election
I20260709 12:31:37.235404 20785 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9e510f7834bc49239d921738ee09360e"
dest_uuid: "f9fc9dfee3bd4478a587de49d7bc3789"
 from {username='slave'} at 127.0.0.1:32880
I20260709 12:31:37.235987 20785 raft_consensus.cc:493] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:37.236398 20785 raft_consensus.cc:3060] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:37.241887 20785 raft_consensus.cc:515] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:37.243966 20785 leader_election.cc:290] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [CANDIDATE]: Term 2 election: Requested vote from peers d57571da3cac40fc8c10f2ff9e98e9c1 (127.4.65.1:43287), 80164889ef444f7baf4af798960d957b (127.4.65.2:38971)
I20260709 12:31:37.257709 20636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e510f7834bc49239d921738ee09360e" candidate_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1"
I20260709 12:31:37.258592 20636 raft_consensus.cc:3055] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:37.258630 20711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e510f7834bc49239d921738ee09360e" candidate_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "80164889ef444f7baf4af798960d957b"
I20260709 12:31:37.259069 20636 raft_consensus.cc:740] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d57571da3cac40fc8c10f2ff9e98e9c1, State: Running, Role: LEADER
I20260709 12:31:37.259308 20711 raft_consensus.cc:3060] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:37.259963 20636 consensus_queue.cc:260] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:37.261293 20636 raft_consensus.cc:3060] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:37.267241 20711 raft_consensus.cc:2468] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9fc9dfee3bd4478a587de49d7bc3789 in term 2.
I20260709 12:31:37.268932 20751 leader_election.cc:304] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80164889ef444f7baf4af798960d957b, f9fc9dfee3bd4478a587de49d7bc3789; no voters: 
I20260709 12:31:37.269028 20636 raft_consensus.cc:2468] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9fc9dfee3bd4478a587de49d7bc3789 in term 2.
I20260709 12:31:37.270099 20854 raft_consensus.cc:2804] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:37.271122 20854 raft_consensus.cc:697] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [term 2 LEADER]: Becoming Leader. State: Replica: f9fc9dfee3bd4478a587de49d7bc3789, State: Running, Role: LEADER
I20260709 12:31:37.271838 20854 consensus_queue.cc:237] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } }
I20260709 12:31:37.278839 20532 catalog_manager.cc:5697] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 reported cstate change: term changed from 1 to 2, leader changed from d57571da3cac40fc8c10f2ff9e98e9c1 (127.4.65.1) to f9fc9dfee3bd4478a587de49d7bc3789 (127.4.65.3). New cstate: current_term: 2 leader_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d57571da3cac40fc8c10f2ff9e98e9c1" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 43287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80164889ef444f7baf4af798960d957b" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 38971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9fc9dfee3bd4478a587de49d7bc3789" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 37457 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:37.440719  4356 tablet_server.cc:179] TabletServer@127.4.65.1:0 shutting down...
I20260709 12:31:37.463179  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:31:37.463822  4356 tablet_replica.cc:333] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1: stopping tablet replica
I20260709 12:31:37.464412  4356 raft_consensus.cc:2243] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:37.464811  4356 raft_consensus.cc:2272] T 9e510f7834bc49239d921738ee09360e P d57571da3cac40fc8c10f2ff9e98e9c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:37.467137  4356 tablet_replica.cc:333] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1: stopping tablet replica
I20260709 12:31:37.467674  4356 raft_consensus.cc:2243] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:37.468585  4356 raft_consensus.cc:2272] T dd7f0aa14fd649138d3566b7da01786a P d57571da3cac40fc8c10f2ff9e98e9c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:37.471359  4356 tablet_replica.cc:333] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1: stopping tablet replica
I20260709 12:31:37.471889  4356 raft_consensus.cc:2243] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:37.472275  4356 raft_consensus.cc:2272] T e69fbefee0a64f58b5b69ac2d087f5fe P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:37.474470  4356 tablet_replica.cc:333] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1: stopping tablet replica
I20260709 12:31:37.474910  4356 raft_consensus.cc:2243] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:37.475680  4356 raft_consensus.cc:2272] T addee36dfbd94773bd88c7b367e1a3df P d57571da3cac40fc8c10f2ff9e98e9c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:37.500840  4356 tablet_server.cc:196] TabletServer@127.4.65.1:0 shutdown complete.
I20260709 12:31:37.515655  4356 tablet_server.cc:179] TabletServer@127.4.65.2:0 shutting down...
I20260709 12:31:37.538803  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:31:37.539489  4356 tablet_replica.cc:333] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b: stopping tablet replica
I20260709 12:31:37.540079  4356 raft_consensus.cc:2243] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:37.540632  4356 raft_consensus.cc:2272] T dd7f0aa14fd649138d3566b7da01786a P 80164889ef444f7baf4af798960d957b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:37.543007  4356 tablet_replica.cc:333] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b: stopping tablet replica
I20260709 12:31:37.543581  4356 raft_consensus.cc:2243] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:37.544020  4356 raft_consensus.cc:2272] T 9e510f7834bc49239d921738ee09360e P 80164889ef444f7baf4af798960d957b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:37.546697  4356 tablet_replica.cc:333] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b: stopping tablet replica
I20260709 12:31:37.547242  4356 raft_consensus.cc:2243] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:37.547691  4356 raft_consensus.cc:2272] T addee36dfbd94773bd88c7b367e1a3df P 80164889ef444f7baf4af798960d957b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:37.549608  4356 tablet_replica.cc:333] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b: stopping tablet replica
I20260709 12:31:37.550115  4356 raft_consensus.cc:2243] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:37.550947  4356 raft_consensus.cc:2272] T e69fbefee0a64f58b5b69ac2d087f5fe P 80164889ef444f7baf4af798960d957b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:37.575263  4356 tablet_server.cc:196] TabletServer@127.4.65.2:0 shutdown complete.
I20260709 12:31:37.588586  4356 tablet_server.cc:179] TabletServer@127.4.65.3:0 shutting down...
I20260709 12:31:37.611579  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:31:37.612197  4356 tablet_replica.cc:333] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789: stopping tablet replica
I20260709 12:31:37.612824  4356 raft_consensus.cc:2243] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:37.613477  4356 pending_rounds.cc:70] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789: Trying to abort 1 pending ops.
I20260709 12:31:37.613696  4356 pending_rounds.cc:77] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7305626817632550912 op_type: NO_OP noop_request { }
I20260709 12:31:37.614008  4356 raft_consensus.cc:2889] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:31:37.614306  4356 raft_consensus.cc:2272] T 9e510f7834bc49239d921738ee09360e P f9fc9dfee3bd4478a587de49d7bc3789 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:37.616734  4356 tablet_replica.cc:333] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789: stopping tablet replica
I20260709 12:31:37.617280  4356 raft_consensus.cc:2243] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:37.617725  4356 raft_consensus.cc:2272] T dd7f0aa14fd649138d3566b7da01786a P f9fc9dfee3bd4478a587de49d7bc3789 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:37.620522  4356 tablet_replica.cc:333] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789: stopping tablet replica
I20260709 12:31:37.621068  4356 raft_consensus.cc:2243] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:37.621515  4356 raft_consensus.cc:2272] T e69fbefee0a64f58b5b69ac2d087f5fe P f9fc9dfee3bd4478a587de49d7bc3789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:37.623311  4356 tablet_replica.cc:333] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789: stopping tablet replica
I20260709 12:31:37.623776  4356 raft_consensus.cc:2243] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:37.624240  4356 raft_consensus.cc:2272] T addee36dfbd94773bd88c7b367e1a3df P f9fc9dfee3bd4478a587de49d7bc3789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:37.647912  4356 tablet_server.cc:196] TabletServer@127.4.65.3:0 shutdown complete.
I20260709 12:31:37.661293  4356 master.cc:562] Master@127.4.65.62:42141 shutting down...
I20260709 12:31:37.681447  4356 raft_consensus.cc:2243] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:37.682147  4356 raft_consensus.cc:2272] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:37.682461  4356 tablet_replica.cc:333] T 00000000000000000000000000000000 P afb2c4f9d2c44f52a143891a6372308c: stopping tablet replica
I20260709 12:31:37.703104  4356 master.cc:584] Master@127.4.65.62:42141 shutdown complete.
[       OK ] LeaderRebalancerTest.MultiTableLeaderBalance (2940 ms)
[----------] 6 tests from LeaderRebalancerTest (514266 ms total)

[----------] 2 tests from FilterSoftDeletedTableTest
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0
I20260709 12:31:37.734202  4356 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.65.62:34691
I20260709 12:31:37.735255  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:31:37.740397 20888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:37.740542 20889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:37.741791 20891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:31:37.743403  4356 server_base.cc:1061] running on GCE node
I20260709 12:31:37.744294  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:31:37.744550  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:31:37.744664  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600297744653 us; error 0 us; skew 500 ppm
I20260709 12:31:37.745160  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:31:37.747514  4356 webserver.cc:533] Webserver started at http://127.4.65.62:37181/ using document root <none> and password file <none>
I20260709 12:31:37.747949  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:31:37.748103  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:31:37.748319  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:31:37.749698  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/master-0-root/instance:
uuid: "4bf755ee6b38462c892f8dffc22963fc"
format_stamp: "Formatted at 2026-07-09 12:31:37 on dist-test-slave-l1c9"
I20260709 12:31:37.755055  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:31:37.758671 20896 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:37.759826  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:31:37.760325  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/master-0-root
uuid: "4bf755ee6b38462c892f8dffc22963fc"
format_stamp: "Formatted at 2026-07-09 12:31:37 on dist-test-slave-l1c9"
I20260709 12:31:37.760705  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:31:37.771741  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:31:37.773240  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:31:37.815641  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.62:34691
I20260709 12:31:37.815738 20947 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.62:34691 every 8 connection(s)
I20260709 12:31:37.819516 20948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:37.831171 20948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc: Bootstrap starting.
I20260709 12:31:37.835845 20948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:37.840101 20948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc: No bootstrap required, opened a new log
I20260709 12:31:37.842515 20948 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf755ee6b38462c892f8dffc22963fc" member_type: VOTER }
I20260709 12:31:37.842914 20948 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:37.843179 20948 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf755ee6b38462c892f8dffc22963fc, State: Initialized, Role: FOLLOWER
I20260709 12:31:37.843885 20948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf755ee6b38462c892f8dffc22963fc" member_type: VOTER }
I20260709 12:31:37.844426 20948 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:31:37.844650 20948 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:31:37.845273 20948 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:37.851377 20948 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf755ee6b38462c892f8dffc22963fc" member_type: VOTER }
I20260709 12:31:37.852197 20948 leader_election.cc:304] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bf755ee6b38462c892f8dffc22963fc; no voters: 
I20260709 12:31:37.853610 20948 leader_election.cc:290] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:31:37.854182 20951 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:37.855618 20951 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [term 1 LEADER]: Becoming Leader. State: Replica: 4bf755ee6b38462c892f8dffc22963fc, State: Running, Role: LEADER
I20260709 12:31:37.856333 20951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf755ee6b38462c892f8dffc22963fc" member_type: VOTER }
I20260709 12:31:37.856637 20948 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:31:37.859468 20953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bf755ee6b38462c892f8dffc22963fc. Latest consensus state: current_term: 1 leader_uuid: "4bf755ee6b38462c892f8dffc22963fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf755ee6b38462c892f8dffc22963fc" member_type: VOTER } }
I20260709 12:31:37.859393 20952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bf755ee6b38462c892f8dffc22963fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bf755ee6b38462c892f8dffc22963fc" member_type: VOTER } }
I20260709 12:31:37.860563 20952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [sys.catalog]: This master's current role is: LEADER
I20260709 12:31:37.860242 20953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [sys.catalog]: This master's current role is: LEADER
I20260709 12:31:37.864121 20958 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:31:37.872263 20958 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:31:37.873315  4356 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:31:37.881538 20958 catalog_manager.cc:1383] Generated new cluster ID: ba9c0a44f4ac41bd9aa662455b1381e7
I20260709 12:31:37.881984 20958 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:31:37.899159 20958 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:31:37.900564 20958 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:31:37.922701 20958 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc: Generated new TSK 0
I20260709 12:31:37.923696 20958 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:31:37.940626  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:31:37.947660 20969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:37.947997 20970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:37.949699 20972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:31:37.950439  4356 server_base.cc:1061] running on GCE node
I20260709 12:31:37.951288  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:31:37.951470  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:31:37.951599  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600297951583 us; error 0 us; skew 500 ppm
I20260709 12:31:37.952073  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:31:37.954623  4356 webserver.cc:533] Webserver started at http://127.4.65.1:32865/ using document root <none> and password file <none>
I20260709 12:31:37.955057  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:31:37.955221  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:31:37.955436  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:31:37.956576  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-0-root/instance:
uuid: "0315789c9afb4bb69a62f2e94c2180ac"
format_stamp: "Formatted at 2026-07-09 12:31:37 on dist-test-slave-l1c9"
I20260709 12:31:37.961589  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:31:37.965526 20977 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:37.966709  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:31:37.967128  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-0-root
uuid: "0315789c9afb4bb69a62f2e94c2180ac"
format_stamp: "Formatted at 2026-07-09 12:31:37 on dist-test-slave-l1c9"
I20260709 12:31:37.967456  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:31:37.985060  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:31:37.986152  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:31:37.987625  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:31:37.990322  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:31:37.990506  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:37.990851  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:31:37.990994  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:38.034719  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.1:42545
I20260709 12:31:38.034861 21039 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.1:42545 every 8 connection(s)
I20260709 12:31:38.039901  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:31:38.046756 21044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:38.047958 21045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:38.050592 21047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:31:38.051088  4356 server_base.cc:1061] running on GCE node
I20260709 12:31:38.052834  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:31:38.053097  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:31:38.053304  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600298053285 us; error 0 us; skew 500 ppm
I20260709 12:31:38.054000  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:31:38.056046 21040 heartbeater.cc:344] Connected to a master server at 127.4.65.62:34691
I20260709 12:31:38.056464 21040 heartbeater.cc:461] Registering TS with master...
I20260709 12:31:38.056927  4356 webserver.cc:533] Webserver started at http://127.4.65.2:39825/ using document root <none> and password file <none>
I20260709 12:31:38.057288 21040 heartbeater.cc:507] Master 127.4.65.62:34691 requested a full tablet report, sending...
I20260709 12:31:38.057476  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:31:38.057804  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:31:38.058084  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:31:38.059401  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-1-root/instance:
uuid: "f9e77ebb794b47949570cbc9f3e55646"
format_stamp: "Formatted at 2026-07-09 12:31:38 on dist-test-slave-l1c9"
I20260709 12:31:38.059612 20913 ts_manager.cc:194] Registered new tserver with Master: 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545)
I20260709 12:31:38.061542 20913 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49194
I20260709 12:31:38.064527  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.005s
I20260709 12:31:38.068944 21052 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:38.070107  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:31:38.070492  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-1-root
uuid: "f9e77ebb794b47949570cbc9f3e55646"
format_stamp: "Formatted at 2026-07-09 12:31:38 on dist-test-slave-l1c9"
I20260709 12:31:38.070813  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:31:38.109274  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:31:38.110498  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:31:38.112213  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:31:38.114637  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:31:38.114880  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:38.115132  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:31:38.115307  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:38.157527  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.2:37027
I20260709 12:31:38.157648 21114 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.2:37027 every 8 connection(s)
I20260709 12:31:38.162477  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:31:38.170295 21118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:38.171883 21119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:31:38.173341 21121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:31:38.175933 21115 heartbeater.cc:344] Connected to a master server at 127.4.65.62:34691
I20260709 12:31:38.176049  4356 server_base.cc:1061] running on GCE node
I20260709 12:31:38.176607 21115 heartbeater.cc:461] Registering TS with master...
I20260709 12:31:38.177313  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:31:38.177637  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:31:38.177572 21115 heartbeater.cc:507] Master 127.4.65.62:34691 requested a full tablet report, sending...
I20260709 12:31:38.178030  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600298178010 us; error 0 us; skew 500 ppm
I20260709 12:31:38.178858  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:31:38.180162 20913 ts_manager.cc:194] Registered new tserver with Master: f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:38.181502  4356 webserver.cc:533] Webserver started at http://127.4.65.3:35091/ using document root <none> and password file <none>
I20260709 12:31:38.181953 20913 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49204
I20260709 12:31:38.182032  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:31:38.182507  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:31:38.182799  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:31:38.184199  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-2-root/instance:
uuid: "2a779b422c3c4bd59a5449144952c73a"
format_stamp: "Formatted at 2026-07-09 12:31:38 on dist-test-slave-l1c9"
I20260709 12:31:38.189076  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:31:38.192772 21126 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:38.193593  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:31:38.193941  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-2-root
uuid: "2a779b422c3c4bd59a5449144952c73a"
format_stamp: "Formatted at 2026-07-09 12:31:38 on dist-test-slave-l1c9"
I20260709 12:31:38.194502  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599783304823-4356-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:31:38.205560  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:31:38.206662  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:31:38.208328  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:31:38.211411  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:31:38.211683  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:38.211882  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:31:38.212013  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:31:38.257176  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.3:33705
I20260709 12:31:38.257289 21188 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.3:33705 every 8 connection(s)
I20260709 12:31:38.272765 21189 heartbeater.cc:344] Connected to a master server at 127.4.65.62:34691
I20260709 12:31:38.273243 21189 heartbeater.cc:461] Registering TS with master...
I20260709 12:31:38.274044 21189 heartbeater.cc:507] Master 127.4.65.62:34691 requested a full tablet report, sending...
I20260709 12:31:38.276244 20913 ts_manager.cc:194] Registered new tserver with Master: 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:38.277328  4356 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016798533s
I20260709 12:31:38.277853 20913 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49216
I20260709 12:31:38.305811 20913 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49220:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:31:38.308658 20913 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:31:38.390622 21154 tablet_service.cc:1511] Processing CreateTablet for tablet 24bbbbdce7934410b6e080a04ab521f7 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:31:38.392904 21154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24bbbbdce7934410b6e080a04ab521f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.400496 21153 tablet_service.cc:1511] Processing CreateTablet for tablet e082efdfaedc435c9b2265aba8e5d211 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:31:38.402113 21153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e082efdfaedc435c9b2265aba8e5d211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.401830 21080 tablet_service.cc:1511] Processing CreateTablet for tablet 24bbbbdce7934410b6e080a04ab521f7 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:31:38.403195 21080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24bbbbdce7934410b6e080a04ab521f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.403949 21076 tablet_service.cc:1511] Processing CreateTablet for tablet c71f4db910844fd28c4a1679dd1477fe (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:31:38.405409 21079 tablet_service.cc:1511] Processing CreateTablet for tablet e082efdfaedc435c9b2265aba8e5d211 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:31:38.407470 21152 tablet_service.cc:1511] Processing CreateTablet for tablet 9d54c78552d24a77a8b1a06013d70728 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:31:38.407848 21075 tablet_service.cc:1511] Processing CreateTablet for tablet 29dfb2ec48164bb0a6e78ff68f55d2d9 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:31:38.408520 21077 tablet_service.cc:1511] Processing CreateTablet for tablet 6aa337c9b7844a03a308ba472bb3bdfd (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:31:38.407470 21078 tablet_service.cc:1511] Processing CreateTablet for tablet 9d54c78552d24a77a8b1a06013d70728 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:31:38.409137 21152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d54c78552d24a77a8b1a06013d70728. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.412909 21076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c71f4db910844fd28c4a1679dd1477fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.413775 21151 tablet_service.cc:1511] Processing CreateTablet for tablet 6aa337c9b7844a03a308ba472bb3bdfd (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:31:38.414909 21075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29dfb2ec48164bb0a6e78ff68f55d2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.415478 21151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6aa337c9b7844a03a308ba472bb3bdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.416741 21079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e082efdfaedc435c9b2265aba8e5d211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.417057 21004 tablet_service.cc:1511] Processing CreateTablet for tablet e082efdfaedc435c9b2265aba8e5d211 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:31:38.417521 21003 tablet_service.cc:1511] Processing CreateTablet for tablet 9d54c78552d24a77a8b1a06013d70728 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:31:38.418406 21078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d54c78552d24a77a8b1a06013d70728. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.419533 21150 tablet_service.cc:1511] Processing CreateTablet for tablet c71f4db910844fd28c4a1679dd1477fe (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:31:38.420065 21077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6aa337c9b7844a03a308ba472bb3bdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.421006 21150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c71f4db910844fd28c4a1679dd1477fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.421468 21001 tablet_service.cc:1511] Processing CreateTablet for tablet c71f4db910844fd28c4a1679dd1477fe (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:31:38.418467 21004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e082efdfaedc435c9b2265aba8e5d211. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.426409 21001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c71f4db910844fd28c4a1679dd1477fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.427176 21003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d54c78552d24a77a8b1a06013d70728. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.426946 21074 tablet_service.cc:1511] Processing CreateTablet for tablet d1aff18dd4734eada226c23cc72e68df (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:31:38.428341 21074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1aff18dd4734eada226c23cc72e68df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.425454 21149 tablet_service.cc:1511] Processing CreateTablet for tablet 29dfb2ec48164bb0a6e78ff68f55d2d9 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:31:38.441314 21149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29dfb2ec48164bb0a6e78ff68f55d2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.430369 21073 tablet_service.cc:1511] Processing CreateTablet for tablet ce50888244a14856b32e824b55bf67c6 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:31:38.445940 21073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce50888244a14856b32e824b55bf67c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.446318 21148 tablet_service.cc:1511] Processing CreateTablet for tablet d1aff18dd4734eada226c23cc72e68df (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:31:38.447803 21148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1aff18dd4734eada226c23cc72e68df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.432339 21005 tablet_service.cc:1511] Processing CreateTablet for tablet 24bbbbdce7934410b6e080a04ab521f7 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:31:38.449851 21005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24bbbbdce7934410b6e080a04ab521f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.430415 21072 tablet_service.cc:1511] Processing CreateTablet for tablet 1b21dacbef9943b4bb80a2c552abf0f6 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:31:38.453902 21072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b21dacbef9943b4bb80a2c552abf0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.459648 21147 tablet_service.cc:1511] Processing CreateTablet for tablet ce50888244a14856b32e824b55bf67c6 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:31:38.461190 21147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce50888244a14856b32e824b55bf67c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.462047 21000 tablet_service.cc:1511] Processing CreateTablet for tablet 29dfb2ec48164bb0a6e78ff68f55d2d9 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:31:38.463446 21000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29dfb2ec48164bb0a6e78ff68f55d2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.431438 21002 tablet_service.cc:1511] Processing CreateTablet for tablet 6aa337c9b7844a03a308ba472bb3bdfd (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:31:38.469341 21002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6aa337c9b7844a03a308ba472bb3bdfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.482554 21146 tablet_service.cc:1511] Processing CreateTablet for tablet 1b21dacbef9943b4bb80a2c552abf0f6 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:31:38.484221 21146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b21dacbef9943b4bb80a2c552abf0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.487635 20999 tablet_service.cc:1511] Processing CreateTablet for tablet d1aff18dd4734eada226c23cc72e68df (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:31:38.489586 20999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1aff18dd4734eada226c23cc72e68df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.504640 21204 tablet_bootstrap.cc:492] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:38.505856 20998 tablet_service.cc:1511] Processing CreateTablet for tablet ce50888244a14856b32e824b55bf67c6 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:31:38.507264 20998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce50888244a14856b32e824b55bf67c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.525420 20997 tablet_service.cc:1511] Processing CreateTablet for tablet 1b21dacbef9943b4bb80a2c552abf0f6 (DEFAULT_TABLE table=test-workload [id=2fe0e6eea6634ae69dc5d12aebbabe5e]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:31:38.527784 20997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b21dacbef9943b4bb80a2c552abf0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:38.588536 21205 tablet_bootstrap.cc:492] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:38.612223 21204 tablet_bootstrap.cc:654] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.627621 21206 tablet_bootstrap.cc:492] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:38.634080 21205 tablet_bootstrap.cc:654] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.634506 21206 tablet_bootstrap.cc:654] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.641824 21206 tablet_bootstrap.cc:492] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:38.642294 21206 ts_tablet_manager.cc:1403] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.015s	user 0.010s	sys 0.000s
I20260709 12:31:38.642355 21204 tablet_bootstrap.cc:492] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:38.643101 21204 ts_tablet_manager.cc:1403] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.142s	user 0.025s	sys 0.000s
I20260709 12:31:38.644912 21206 raft_consensus.cc:359] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.645809 21206 raft_consensus.cc:385] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.646075 21206 raft_consensus.cc:740] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.646763 21206 consensus_queue.cc:260] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.646548 21204 raft_consensus.cc:359] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.647511 21205 tablet_bootstrap.cc:492] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:38.647739 21204 raft_consensus.cc:385] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.648311 21205 ts_tablet_manager.cc:1403] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.060s	user 0.012s	sys 0.004s
I20260709 12:31:38.648260 21204 raft_consensus.cc:740] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.649329 21040 heartbeater.cc:499] Master 127.4.65.62:34691 was elected leader, sending a full tablet report...
I20260709 12:31:38.649499 21204 consensus_queue.cc:260] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.652264 21189 heartbeater.cc:499] Master 127.4.65.62:34691 was elected leader, sending a full tablet report...
I20260709 12:31:38.652809 21204 ts_tablet_manager.cc:1434] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.009s	user 0.012s	sys 0.000s
I20260709 12:31:38.653899 21204 tablet_bootstrap.cc:492] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:38.652156 21205 raft_consensus.cc:359] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.656651 21205 raft_consensus.cc:385] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.657099 21205 raft_consensus.cc:740] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.658190 21205 consensus_queue.cc:260] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.660516 21204 tablet_bootstrap.cc:654] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.663177 21115 heartbeater.cc:499] Master 127.4.65.62:34691 was elected leader, sending a full tablet report...
I20260709 12:31:38.663637 21205 ts_tablet_manager.cc:1434] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.004s
I20260709 12:31:38.664700 21205 tablet_bootstrap.cc:492] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:38.667518 21206 ts_tablet_manager.cc:1434] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.025s	user 0.004s	sys 0.005s
I20260709 12:31:38.669104 21206 tablet_bootstrap.cc:492] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:38.671144 21205 tablet_bootstrap.cc:654] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.675475 21204 tablet_bootstrap.cc:492] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:38.676095 21204 ts_tablet_manager.cc:1403] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.012s
I20260709 12:31:38.677119 21205 tablet_bootstrap.cc:492] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:38.676051 21206 tablet_bootstrap.cc:654] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.677726 21205 ts_tablet_manager.cc:1403] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.013s	user 0.007s	sys 0.004s
I20260709 12:31:38.680915 21205 raft_consensus.cc:359] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.681917 21205 raft_consensus.cc:385] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.681586 21204 raft_consensus.cc:359] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.682353 21205 raft_consensus.cc:740] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.682844 21204 raft_consensus.cc:385] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.683357 21204 raft_consensus.cc:740] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.683764 21205 consensus_queue.cc:260] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.684413 21204 consensus_queue.cc:260] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.686105 21205 ts_tablet_manager.cc:1434] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:31:38.686590 21204 ts_tablet_manager.cc:1434] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:31:38.687222 21205 tablet_bootstrap.cc:492] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:38.687578 21204 tablet_bootstrap.cc:492] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:38.694140 21204 tablet_bootstrap.cc:654] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.695019 21206 tablet_bootstrap.cc:492] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:38.695605 21206 ts_tablet_manager.cc:1403] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.027s	user 0.003s	sys 0.011s
I20260709 12:31:38.699328 21206 raft_consensus.cc:359] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.700042 21206 raft_consensus.cc:385] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.700330 21206 raft_consensus.cc:740] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.701260 21205 tablet_bootstrap.cc:654] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.702001 21204 tablet_bootstrap.cc:492] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:38.702612 21204 ts_tablet_manager.cc:1403] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.015s	user 0.008s	sys 0.006s
I20260709 12:31:38.701295 21206 consensus_queue.cc:260] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.705806 21204 raft_consensus.cc:359] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.706704 21204 raft_consensus.cc:385] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.707105 21204 raft_consensus.cc:740] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.708133 21204 consensus_queue.cc:260] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.711328 21204 ts_tablet_manager.cc:1434] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.001s
I20260709 12:31:38.711396 21206 ts_tablet_manager.cc:1434] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.015s	user 0.003s	sys 0.003s
I20260709 12:31:38.715001 21206 tablet_bootstrap.cc:492] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:38.712604 21204 tablet_bootstrap.cc:492] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:38.721463 21205 tablet_bootstrap.cc:492] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:38.722333 21205 ts_tablet_manager.cc:1403] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.035s	user 0.020s	sys 0.000s
I20260709 12:31:38.723308 21204 tablet_bootstrap.cc:654] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.721551 21206 tablet_bootstrap.cc:654] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.725960 21205 raft_consensus.cc:359] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.727039 21205 raft_consensus.cc:385] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.727514 21205 raft_consensus.cc:740] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.728798 21205 consensus_queue.cc:260] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.734369 21205 ts_tablet_manager.cc:1434] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:31:38.735563 21205 tablet_bootstrap.cc:492] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:38.736027 21204 tablet_bootstrap.cc:492] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:38.736649 21204 ts_tablet_manager.cc:1403] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.024s	user 0.016s	sys 0.003s
I20260709 12:31:38.739508 21212 raft_consensus.cc:493] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:38.739730 21204 raft_consensus.cc:359] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.740618 21204 raft_consensus.cc:385] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.740309 21212 raft_consensus.cc:515] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.740978 21204 raft_consensus.cc:740] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.741742 21205 tablet_bootstrap.cc:654] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.742062 21204 consensus_queue.cc:260] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.746029 21204 ts_tablet_manager.cc:1434] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:31:38.748769 21212 leader_election.cc:290] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:38.748761 21206 tablet_bootstrap.cc:492] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:38.750988 21206 ts_tablet_manager.cc:1403] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.036s	user 0.021s	sys 0.010s
I20260709 12:31:38.755241 21206 raft_consensus.cc:359] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.756444 21206 raft_consensus.cc:385] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.756930 21206 raft_consensus.cc:740] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.758154 21206 consensus_queue.cc:260] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.763226 21204 tablet_bootstrap.cc:492] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:38.768033 21206 ts_tablet_manager.cc:1434] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:31:38.771543 21204 tablet_bootstrap.cc:654] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.778739 21206 tablet_bootstrap.cc:492] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:38.786844 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e082efdfaedc435c9b2265aba8e5d211" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:38.787732 21015 raft_consensus.cc:2468] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 0.
I20260709 12:31:38.788486 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e082efdfaedc435c9b2265aba8e5d211" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:38.789250 21164 raft_consensus.cc:2468] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 0.
I20260709 12:31:38.789410 21055 leader_election.cc:304] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:38.790323 21212 raft_consensus.cc:2804] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:38.790695 21212 raft_consensus.cc:493] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:38.791082 21212 raft_consensus.cc:3060] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:38.792482 21206 tablet_bootstrap.cc:654] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.795215 21204 tablet_bootstrap.cc:492] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:38.796036 21204 ts_tablet_manager.cc:1403] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.033s	user 0.009s	sys 0.005s
I20260709 12:31:38.799331 21204 raft_consensus.cc:359] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.800251 21204 raft_consensus.cc:385] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.800618 21204 raft_consensus.cc:740] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.801128 21212 raft_consensus.cc:515] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.801599 21204 consensus_queue.cc:260] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.804207 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e082efdfaedc435c9b2265aba8e5d211" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:38.804913 21015 raft_consensus.cc:3060] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:38.805567 21206 tablet_bootstrap.cc:492] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:38.805989 21205 tablet_bootstrap.cc:492] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:38.806159 21206 ts_tablet_manager.cc:1403] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.028s	user 0.014s	sys 0.010s
I20260709 12:31:38.806545 21205 ts_tablet_manager.cc:1403] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.071s	user 0.021s	sys 0.044s
I20260709 12:31:38.807365 21204 ts_tablet_manager.cc:1434] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:31:38.808463 21204 tablet_bootstrap.cc:492] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:38.809616 21206 raft_consensus.cc:359] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.810707 21206 raft_consensus.cc:385] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.811026 21206 raft_consensus.cc:740] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.812382 21206 consensus_queue.cc:260] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.813669 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e082efdfaedc435c9b2265aba8e5d211" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:38.814313 21164 raft_consensus.cc:3060] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:38.814709 21206 ts_tablet_manager.cc:1434] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:31:38.809711 21205 raft_consensus.cc:359] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:31:38.815183 21205 raft_consensus.cc:385] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.815506 21205 raft_consensus.cc:740] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.815800 21204 tablet_bootstrap.cc:654] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.816318 21205 consensus_queue.cc:260] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:31:38.811031 21212 leader_election.cc:290] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:38.821219 21206 tablet_bootstrap.cc:492] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:38.824177 21015 raft_consensus.cc:2468] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 1.
I20260709 12:31:38.825702 21055 leader_election.cc:304] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:38.826865 21212 raft_consensus.cc:2804] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:38.827415 21212 raft_consensus.cc:697] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Becoming Leader. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:31:38.828328 21212 consensus_queue.cc:237] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.835454 21206 tablet_bootstrap.cc:654] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.836961 21164 raft_consensus.cc:2468] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 1.
I20260709 12:31:38.839874 21211 raft_consensus.cc:493] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:38.840493 21211 raft_consensus.cc:515] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.841529 21205 ts_tablet_manager.cc:1434] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.035s	user 0.010s	sys 0.009s
I20260709 12:31:38.843175 21205 tablet_bootstrap.cc:492] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:38.844664 20911 catalog_manager.cc:5697] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2). New cstate: current_term: 1 leader_uuid: "f9e77ebb794b47949570cbc9f3e55646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:38.845990 21211 leader_election.cc:290] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:38.851162 21212 raft_consensus.cc:493] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:38.851691 21205 tablet_bootstrap.cc:654] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.851850 21212 raft_consensus.cc:515] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.854202 21212 leader_election.cc:290] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:38.854911 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dfb2ec48164bb0a6e78ff68f55d2d9" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:38.856794 21206 tablet_bootstrap.cc:492] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
W20260709 12:31:38.856837 21055 leader_election.cc:343] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:38.857337 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dfb2ec48164bb0a6e78ff68f55d2d9" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:38.858119 21164 raft_consensus.cc:2468] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 0.
I20260709 12:31:38.858558 21206 ts_tablet_manager.cc:1403] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.037s	user 0.010s	sys 0.004s
I20260709 12:31:38.859958 21055 leader_election.cc:304] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:31:38.861896 21212 raft_consensus.cc:2804] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:38.862594 21212 raft_consensus.cc:493] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:38.862481 21206 raft_consensus.cc:359] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.863175 21212 raft_consensus.cc:3060] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:38.863889 21205 tablet_bootstrap.cc:492] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:38.864436 21206 raft_consensus.cc:385] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.864810 21205 ts_tablet_manager.cc:1403] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.022s	user 0.017s	sys 0.004s
I20260709 12:31:38.868321 21205 raft_consensus.cc:359] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.869115 21205 raft_consensus.cc:385] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.869434 21205 raft_consensus.cc:740] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.870153 21205 consensus_queue.cc:260] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.864792 21206 raft_consensus.cc:740] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.872151 21206 consensus_queue.cc:260] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.877610 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d54c78552d24a77a8b1a06013d70728" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:38.878435 21090 raft_consensus.cc:2468] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 0.
I20260709 12:31:38.879854 21127 leader_election.cc:304] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:38.880900 21211 raft_consensus.cc:2804] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:38.881351 21205 ts_tablet_manager.cc:1434] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.005s
I20260709 12:31:38.881346 21211 raft_consensus.cc:493] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:38.882176 21211 raft_consensus.cc:3060] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:38.882781 21204 tablet_bootstrap.cc:492] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:38.883471 21204 ts_tablet_manager.cc:1403] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.075s	user 0.014s	sys 0.008s
I20260709 12:31:38.883525 21212 raft_consensus.cc:515] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.886433 21212 leader_election.cc:290] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:38.886991 21204 raft_consensus.cc:359] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:31:38.887782 21204 raft_consensus.cc:385] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.888010 21204 raft_consensus.cc:740] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.882858 21205 tablet_bootstrap.cc:492] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:38.881536 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d54c78552d24a77a8b1a06013d70728" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:38.890895 21211 raft_consensus.cc:515] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.889818 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dfb2ec48164bb0a6e78ff68f55d2d9" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
W20260709 12:31:38.896174 21129 leader_election.cc:343] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:38.899173 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d54c78552d24a77a8b1a06013d70728" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:38.889886 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dfb2ec48164bb0a6e78ff68f55d2d9" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:38.880820 21206 ts_tablet_manager.cc:1434] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.022s	user 0.009s	sys 0.000s
I20260709 12:31:38.900821 21164 raft_consensus.cc:3060] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:31:38.901427 21129 leader_election.cc:343] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:31:38.888716 20961 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:31:38.902395 21205 tablet_bootstrap.cc:654] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.904978 21206 tablet_bootstrap.cc:492] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:38.888687 21204 consensus_queue.cc:260] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
W20260709 12:31:38.901443 21055 leader_election.cc:343] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:38.910794 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d54c78552d24a77a8b1a06013d70728" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:38.911667 21090 raft_consensus.cc:3060] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:38.912529 21211 leader_election.cc:290] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:38.917605 21206 tablet_bootstrap.cc:654] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.923410 21090 raft_consensus.cc:2468] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 1.
I20260709 12:31:38.927009 21206 tablet_bootstrap.cc:492] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:38.927269 21164 raft_consensus.cc:2468] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 1.
I20260709 12:31:38.927672 21206 ts_tablet_manager.cc:1403] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.023s	user 0.015s	sys 0.002s
I20260709 12:31:38.928057 21127 leader_election.cc:304] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:31:38.927100 21204 ts_tablet_manager.cc:1434] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.043s	user 0.004s	sys 0.004s
I20260709 12:31:38.929059 21055 leader_election.cc:304] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:31:38.930564 21204 tablet_bootstrap.cc:492] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:38.930795 21212 raft_consensus.cc:2804] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:38.931537 21212 raft_consensus.cc:697] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Becoming Leader. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:31:38.932245 21233 raft_consensus.cc:2804] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:38.932811 21212 consensus_queue.cc:237] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.934120 21206 raft_consensus.cc:359] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.934965 21206 raft_consensus.cc:385] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.935258 21206 raft_consensus.cc:740] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.936183 21233 raft_consensus.cc:697] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:38.937476 21204 tablet_bootstrap.cc:654] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:38.937355 21233 consensus_queue.cc:237] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.939647 21206 consensus_queue.cc:260] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.943161 21205 tablet_bootstrap.cc:492] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:38.943843 21205 ts_tablet_manager.cc:1403] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.061s	user 0.030s	sys 0.020s
I20260709 12:31:38.948123 21205 raft_consensus.cc:359] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.949075 21210 raft_consensus.cc:493] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:38.952847 21210 raft_consensus.cc:515] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.955080 21211 raft_consensus.cc:493] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:38.955865 21211 raft_consensus.cc:515] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.958176 21206 ts_tablet_manager.cc:1434] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.030s	user 0.003s	sys 0.004s
I20260709 12:31:38.959949 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c71f4db910844fd28c4a1679dd1477fe" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:38.960680 21014 raft_consensus.cc:2468] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 0.
I20260709 12:31:38.962018 21210 leader_election.cc:290] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:38.960240 21211 leader_election.cc:290] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:38.962953 21205 raft_consensus.cc:385] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:38.963876 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c71f4db910844fd28c4a1679dd1477fe" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:38.963312 21205 raft_consensus.cc:740] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:38.967489 21129 leader_election.cc:304] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: 
I20260709 12:31:38.970237 21090 raft_consensus.cc:2468] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 0.
I20260709 12:31:38.970293 21206 tablet_bootstrap.cc:492] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:38.970705 21211 raft_consensus.cc:2804] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:38.971170 21211 raft_consensus.cc:493] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:38.971531 21211 raft_consensus.cc:3060] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:38.977288 21205 consensus_queue.cc:260] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.982489 21233 raft_consensus.cc:493] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:38.983518 21233 raft_consensus.cc:515] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:38.987206 21205 ts_tablet_manager.cc:1434] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.043s	user 0.009s	sys 0.000s
I20260709 12:31:38.990859 20910 catalog_manager.cc:5697] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 1 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:38.993443 21205 tablet_bootstrap.cc:492] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:38.992668 20913 catalog_manager.cc:5697] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2). New cstate: current_term: 1 leader_uuid: "f9e77ebb794b47949570cbc9f3e55646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:38.993841 21233 leader_election.cc:290] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:38.995286 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1aff18dd4734eada226c23cc72e68df" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:38.997192 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1aff18dd4734eada226c23cc72e68df" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:38.997599 21090 raft_consensus.cc:2468] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 0.
I20260709 12:31:38.998018 21014 raft_consensus.cc:2468] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 0.
I20260709 12:31:38.999593 21127 leader_election.cc:304] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:39.000670 21233 raft_consensus.cc:2804] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:39.001137 21233 raft_consensus.cc:493] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:39.001588 21233 raft_consensus.cc:3060] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.004330 21206 tablet_bootstrap.cc:654] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:39.012678 21204 tablet_bootstrap.cc:492] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:39.013365 21204 ts_tablet_manager.cc:1403] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.083s	user 0.023s	sys 0.057s
I20260709 12:31:39.013870 21233 raft_consensus.cc:515] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.016688 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1aff18dd4734eada226c23cc72e68df" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:39.016999 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1aff18dd4734eada226c23cc72e68df" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:39.017436 21014 raft_consensus.cc:3060] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.017666 21090 raft_consensus.cc:3060] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.017884 21204 raft_consensus.cc:359] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.018635 21204 raft_consensus.cc:385] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:39.018924 21204 raft_consensus.cc:740] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:39.020079 21233 leader_election.cc:290] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:39.020526 21204 consensus_queue.cc:260] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.025722 21014 raft_consensus.cc:2468] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 1.
I20260709 12:31:39.032006 21129 leader_election.cc:304] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: 
I20260709 12:31:39.026607 21090 raft_consensus.cc:2468] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 1.
I20260709 12:31:39.033543 21240 raft_consensus.cc:2804] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:38.991833 21211 raft_consensus.cc:515] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.034559 21240 raft_consensus.cc:697] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:39.037534 21240 consensus_queue.cc:237] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.041373 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c71f4db910844fd28c4a1679dd1477fe" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:39.040016 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c71f4db910844fd28c4a1679dd1477fe" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:39.028427 21206 tablet_bootstrap.cc:492] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:39.030208 21204 ts_tablet_manager.cc:1434] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.004s
I20260709 12:31:39.042143 21205 tablet_bootstrap.cc:654] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:39.042181 21014 raft_consensus.cc:3060] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.042901 21090 raft_consensus.cc:2468] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 0.
I20260709 12:31:39.045986 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c71f4db910844fd28c4a1679dd1477fe" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:39.043538 21206 ts_tablet_manager.cc:1403] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.074s	user 0.015s	sys 0.004s
I20260709 12:31:39.048411 21164 raft_consensus.cc:2393] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0315789c9afb4bb69a62f2e94c2180ac in current term 1: Already voted for candidate 2a779b422c3c4bd59a5449144952c73a in this term.
I20260709 12:31:39.048640 20978 leader_election.cc:304] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:39.043903 21211 leader_election.cc:290] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:39.045074 21089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c71f4db910844fd28c4a1679dd1477fe" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:39.044768 21204 tablet_bootstrap.cc:492] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:39.050001 21089 raft_consensus.cc:3060] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.058106 21233 raft_consensus.cc:493] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:39.059027 21233 raft_consensus.cc:515] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:31:39.061470 21204 tablet_bootstrap.cc:654] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:39.062670 21089 raft_consensus.cc:2468] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 1.
I20260709 12:31:39.063759 21233 leader_election.cc:290] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545)
I20260709 12:31:39.057945 21014 raft_consensus.cc:2468] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 1.
I20260709 12:31:39.070927 21210 raft_consensus.cc:2764] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:31:39.071507 21127 leader_election.cc:304] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:39.074257 21233 raft_consensus.cc:2804] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:39.074770 21233 raft_consensus.cc:697] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:39.075555 21233 consensus_queue.cc:237] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.077446 21204 tablet_bootstrap.cc:492] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:39.065016 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24bbbbdce7934410b6e080a04ab521f7" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:39.065454 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24bbbbdce7934410b6e080a04ab521f7" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:39.078028 21204 ts_tablet_manager.cc:1403] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.034s	user 0.018s	sys 0.006s
I20260709 12:31:39.067346 21205 tablet_bootstrap.cc:492] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:39.078970 21205 ts_tablet_manager.cc:1403] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.086s	user 0.011s	sys 0.005s
I20260709 12:31:39.081022 21204 raft_consensus.cc:359] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.081800 21204 raft_consensus.cc:385] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:39.082101 21204 raft_consensus.cc:740] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:39.081928 21205 raft_consensus.cc:359] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
W20260709 12:31:39.082125 21129 leader_election.cc:343] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:39.082674 21205 raft_consensus.cc:385] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:39.068504 21206 raft_consensus.cc:359] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.083246 21206 raft_consensus.cc:385] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:39.082958 21204 consensus_queue.cc:260] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.083582 21206 raft_consensus.cc:740] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:39.084403 21206 consensus_queue.cc:260] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.085537 21204 ts_tablet_manager.cc:1434] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.002s
I20260709 12:31:39.086694 21204 tablet_bootstrap.cc:492] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:39.086761 21206 ts_tablet_manager.cc:1434] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.039s	user 0.005s	sys 0.001s
I20260709 12:31:39.083003 21205 raft_consensus.cc:740] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:39.087956 21240 raft_consensus.cc:493] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:39.088342 21205 consensus_queue.cc:260] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.088768 21240 raft_consensus.cc:515] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.090914 21205 ts_tablet_manager.cc:1434] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260709 12:31:39.092624 21240 leader_election.cc:290] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:39.093613 21204 tablet_bootstrap.cc:654] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:39.093411 21089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b21dacbef9943b4bb80a2c552abf0f6" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:39.094121 21206 tablet_bootstrap.cc:492] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:39.094182 21089 raft_consensus.cc:2468] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 0.
I20260709 12:31:39.096030 21127 leader_election.cc:304] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:39.095573 20913 catalog_manager.cc:5697] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 1 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:39.098646 21211 raft_consensus.cc:2804] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:39.099047 21211 raft_consensus.cc:493] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:39.099421 21211 raft_consensus.cc:3060] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.101147 21205 tablet_bootstrap.cc:492] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:39.101478 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b21dacbef9943b4bb80a2c552abf0f6" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:39.102170 21015 raft_consensus.cc:2468] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 0.
I20260709 12:31:39.102860 21090 raft_consensus.cc:2468] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 0.
I20260709 12:31:39.104560 21127 leader_election.cc:304] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:31:39.118273 21240 raft_consensus.cc:493] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:39.119053 21240 raft_consensus.cc:515] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.128523 21233 raft_consensus.cc:2804] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:39.129170 21233 raft_consensus.cc:493] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:39.116151 21206 tablet_bootstrap.cc:654] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:39.129626 21233 raft_consensus.cc:3060] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.130036 21240 leader_election.cc:290] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:39.135836 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce50888244a14856b32e824b55bf67c6" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:39.136135 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce50888244a14856b32e824b55bf67c6" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:39.136724 21204 tablet_bootstrap.cc:492] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:39.136770 21015 raft_consensus.cc:2468] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 0.
I20260709 12:31:39.137625 21204 ts_tablet_manager.cc:1403] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.051s	user 0.023s	sys 0.023s
W20260709 12:31:39.137777 21127 leader_election.cc:343] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:39.139583 21129 leader_election.cc:304] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:39.141487 21204 raft_consensus.cc:359] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.141975 21240 raft_consensus.cc:2804] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:39.142271 21204 raft_consensus.cc:385] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:39.142350 21240 raft_consensus.cc:493] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:39.142573 21204 raft_consensus.cc:740] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:39.142752 21240 raft_consensus.cc:3060] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.143384 21204 consensus_queue.cc:260] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.146397 21204 ts_tablet_manager.cc:1434] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 12:31:39.146165 21233 raft_consensus.cc:515] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:31:39.139591 21205 tablet_bootstrap.cc:654] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:39.132006 21211 raft_consensus.cc:515] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.150414 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24bbbbdce7934410b6e080a04ab521f7" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:39.151340 21090 raft_consensus.cc:3060] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.153431 21211 leader_election.cc:290] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:39.155279 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b21dacbef9943b4bb80a2c552abf0f6" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:39.156193 21015 raft_consensus.cc:3060] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.158119 21089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b21dacbef9943b4bb80a2c552abf0f6" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:39.159047 21089 raft_consensus.cc:3060] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.167940 20910 catalog_manager.cc:5697] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 1 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:39.179131 21089 raft_consensus.cc:2468] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 1.
I20260709 12:31:39.180083 21233 leader_election.cc:290] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Requested vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545)
I20260709 12:31:39.155151 21240 raft_consensus.cc:515] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.180639 21127 leader_election.cc:304] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:39.183926 21233 raft_consensus.cc:2804] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:39.182062 21090 raft_consensus.cc:2468] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 1.
I20260709 12:31:39.185910 21206 tablet_bootstrap.cc:492] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:39.186369 21233 raft_consensus.cc:697] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:39.185102 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24bbbbdce7934410b6e080a04ab521f7" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:39.187199 21210 raft_consensus.cc:493] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:39.187328 21233 consensus_queue.cc:237] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.187796 21210 raft_consensus.cc:515] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.188493 21205 tablet_bootstrap.cc:492] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:39.188977 21205 ts_tablet_manager.cc:1403] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.088s	user 0.023s	sys 0.017s
I20260709 12:31:39.190076 21210 leader_election.cc:290] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:39.186342 21127 leader_election.cc:304] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
W20260709 12:31:39.191395 21129 leader_election.cc:343] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:39.191766 21211 raft_consensus.cc:2804] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:39.192039 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa337c9b7844a03a308ba472bb3bdfd" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:39.192328 21211 raft_consensus.cc:697] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:39.191964 21205 raft_consensus.cc:359] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.192945 21205 raft_consensus.cc:385] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:39.193151 21205 raft_consensus.cc:740] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:39.193001 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa337c9b7844a03a308ba472bb3bdfd" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:39.193349 21211 consensus_queue.cc:237] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:31:39.193780 21164 raft_consensus.cc:2468] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 0.
I20260709 12:31:39.193861 21205 consensus_queue.cc:260] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.195510 20980 leader_election.cc:304] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: 
I20260709 12:31:39.195897 21205 ts_tablet_manager.cc:1434] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:31:39.186349 21206 ts_tablet_manager.cc:1403] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.092s	user 0.019s	sys 0.034s
I20260709 12:31:39.196918 21205 tablet_bootstrap.cc:492] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:39.198513 21240 leader_election.cc:290] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:39.200937 21210 raft_consensus.cc:2804] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:39.201719 21210 raft_consensus.cc:493] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:39.202071 21210 raft_consensus.cc:3060] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.202299 21089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce50888244a14856b32e824b55bf67c6" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:39.202589 21206 raft_consensus.cc:359] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:31:39.203522 21206 raft_consensus.cc:385] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:39.203879 21206 raft_consensus.cc:740] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:39.204953 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce50888244a14856b32e824b55bf67c6" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:39.207986 21206 consensus_queue.cc:260] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:31:39.208601 21014 raft_consensus.cc:3060] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:31:39.204609 21127 leader_election.cc:343] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:39.212502 21205 tablet_bootstrap.cc:654] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:39.216943 21090 raft_consensus.cc:2468] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 0.
I20260709 12:31:39.228309 21205 tablet_bootstrap.cc:492] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:39.229250 21205 ts_tablet_manager.cc:1403] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.033s	user 0.013s	sys 0.008s
I20260709 12:31:39.234866 21206 ts_tablet_manager.cc:1434] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.038s	user 0.009s	sys 0.014s
I20260709 12:31:39.236378 21206 tablet_bootstrap.cc:492] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:39.241230 21210 raft_consensus.cc:515] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.241374 20913 catalog_manager.cc:5697] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 1 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:39.243512 21206 tablet_bootstrap.cc:654] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:39.243896 21210 leader_election.cc:290] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 election: Requested vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:39.247324 21212 consensus_queue.cc:1048] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:39.247766 21015 raft_consensus.cc:2468] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 1.
I20260709 12:31:39.249909 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa337c9b7844a03a308ba472bb3bdfd" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:39.250949 21164 raft_consensus.cc:3060] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.252024 21014 raft_consensus.cc:2468] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 1.
I20260709 12:31:39.253551 21129 leader_election.cc:304] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:39.253185 20913 catalog_manager.cc:5697] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 1 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:39.255146 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa337c9b7844a03a308ba472bb3bdfd" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:39.255894 21090 raft_consensus.cc:3060] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.260774 21233 raft_consensus.cc:2804] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:39.262274 21233 raft_consensus.cc:697] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:39.269398 21233 consensus_queue.cc:237] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.267910 21164 raft_consensus.cc:2468] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 1.
I20260709 12:31:39.267624 21090 raft_consensus.cc:2468] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 1.
I20260709 12:31:39.278824 20980 leader_election.cc:304] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: 
I20260709 12:31:39.281440 21254 raft_consensus.cc:2804] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:39.281980 21254 raft_consensus.cc:697] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Becoming Leader. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:39.282822 21254 consensus_queue.cc:237] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.304234 21225 consensus_queue.cc:1048] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:39.305828 21205 raft_consensus.cc:359] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.307484 21205 raft_consensus.cc:385] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:39.307909 21205 raft_consensus.cc:740] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:39.309303 21205 consensus_queue.cc:260] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.323729 21205 ts_tablet_manager.cc:1434] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.094s	user 0.008s	sys 0.001s
I20260709 12:31:39.335175 20912 catalog_manager.cc:5697] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1). New cstate: current_term: 1 leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:39.340221 21206 tablet_bootstrap.cc:492] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:39.341329 21206 ts_tablet_manager.cc:1403] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.105s	user 0.041s	sys 0.035s
I20260709 12:31:39.345054 21206 raft_consensus.cc:359] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.345870 21206 raft_consensus.cc:385] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:39.346153 21206 raft_consensus.cc:740] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:39.346849 21206 consensus_queue.cc:260] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.349037 21206 ts_tablet_manager.cc:1434] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:31:39.351792 21212 consensus_queue.cc:1048] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:39.354189 20912 catalog_manager.cc:5697] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 1 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:39.365175 21014 raft_consensus.cc:3060] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.377650 21265 consensus_queue.cc:1048] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:39.413565 21254 raft_consensus.cc:493] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:39.414040 21254 raft_consensus.cc:515] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.416638 21254 leader_election.cc:290] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:39.418023 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d54c78552d24a77a8b1a06013d70728" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:39.418087 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d54c78552d24a77a8b1a06013d70728" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:39.419075 21090 raft_consensus.cc:2393] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0315789c9afb4bb69a62f2e94c2180ac in current term 1: Already voted for candidate 2a779b422c3c4bd59a5449144952c73a in this term.
I20260709 12:31:39.420433 20978 leader_election.cc:304] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac; no voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:39.421293 21254 raft_consensus.cc:3060] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.429010 21254 raft_consensus.cc:2749] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:39.473440 21233 consensus_queue.cc:1048] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:39.485404 21211 consensus_queue.cc:1048] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:39.508215 21211 consensus_queue.cc:1048] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:39.522593 21249 consensus_queue.cc:1048] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:39.525784 21164 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "24bbbbdce7934410b6e080a04ab521f7"
dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
mode: GRACEFUL
new_leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
 from {username='slave'} at 127.0.0.1:57198
I20260709 12:31:39.526412 21164 raft_consensus.cc:606] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Received request to transfer leadership to TS 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:31:39.545248 21090 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e082efdfaedc435c9b2265aba8e5d211"
dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
mode: GRACEFUL
new_leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
 from {username='slave'} at 127.0.0.1:50794
I20260709 12:31:39.545831 21090 raft_consensus.cc:606] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Received request to transfer leadership to TS 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:31:39.552331 21164 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9d54c78552d24a77a8b1a06013d70728"
dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
mode: GRACEFUL
new_leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
 from {username='slave'} at 127.0.0.1:57198
I20260709 12:31:39.553032 21164 raft_consensus.cc:606] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Received request to transfer leadership to TS 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:31:39.560981 21249 consensus_queue.cc:1048] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:39.563171 21164 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c71f4db910844fd28c4a1679dd1477fe"
dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
mode: GRACEFUL
new_leader_uuid: "f9e77ebb794b47949570cbc9f3e55646"
 from {username='slave'} at 127.0.0.1:57198
I20260709 12:31:39.563691 21164 raft_consensus.cc:606] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Received request to transfer leadership to TS f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:39.568311 21233 raft_consensus.cc:993] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a: : Instructing follower f9e77ebb794b47949570cbc9f3e55646 to start an election
I20260709 12:31:39.568813 21233 raft_consensus.cc:1081] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Signalling peer f9e77ebb794b47949570cbc9f3e55646 to start an election
I20260709 12:31:39.571894 21089 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c71f4db910844fd28c4a1679dd1477fe"
dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
 from {username='slave'} at 127.0.0.1:50774
I20260709 12:31:39.572731 21089 raft_consensus.cc:493] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:39.573149 21089 raft_consensus.cc:3060] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:39.576186 21164 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d1aff18dd4734eada226c23cc72e68df"
dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
mode: GRACEFUL
new_leader_uuid: "f9e77ebb794b47949570cbc9f3e55646"
 from {username='slave'} at 127.0.0.1:57198
I20260709 12:31:39.576877 21164 raft_consensus.cc:606] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Received request to transfer leadership to TS f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:39.579773 21233 consensus_queue.cc:1048] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:39.583106 21164 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ce50888244a14856b32e824b55bf67c6"
dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
mode: GRACEFUL
new_leader_uuid: "f9e77ebb794b47949570cbc9f3e55646"
 from {username='slave'} at 127.0.0.1:57198
I20260709 12:31:39.583655 21164 raft_consensus.cc:606] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Received request to transfer leadership to TS f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:39.584251 21089 raft_consensus.cc:515] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.585769 21090 raft_consensus.cc:1240] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Rejecting Update request from peer 2a779b422c3c4bd59a5449144952c73a for earlier term 1. Current term is 2. Ops: []
I20260709 12:31:39.588022 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c71f4db910844fd28c4a1679dd1477fe" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:39.590720 21233 consensus_queue.cc:1059] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:39.591156 21089 leader_election.cc:290] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:39.591786 21233 raft_consensus.cc:3055] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:39.592232 21233 raft_consensus.cc:740] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:39.592618 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c71f4db910844fd28c4a1679dd1477fe" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:39.593721 21233 consensus_queue.cc:260] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.595026 21233 raft_consensus.cc:3060] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:39.607314 21164 raft_consensus.cc:2468] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 2.
I20260709 12:31:39.608822 21055 leader_election.cc:304] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:31:39.609558 21265 raft_consensus.cc:2804] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:39.610064 21265 raft_consensus.cc:697] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 2 LEADER]: Becoming Leader. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:31:39.610560 21254 raft_consensus.cc:493] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:39.610968 21265 consensus_queue.cc:237] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.611150 21254 raft_consensus.cc:515] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:31:39.613461 21254 leader_election.cc:290] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:39.614771 21089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24bbbbdce7934410b6e080a04ab521f7" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:39.614987 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24bbbbdce7934410b6e080a04ab521f7" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:39.615661 21089 raft_consensus.cc:2393] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0315789c9afb4bb69a62f2e94c2180ac in current term 1: Already voted for candidate 2a779b422c3c4bd59a5449144952c73a in this term.
I20260709 12:31:39.616842 20980 leader_election.cc:304] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac; no voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:39.617673 21254 raft_consensus.cc:3060] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.621964 20912 catalog_manager.cc:5697] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 reported cstate change: term changed from 1 to 2, leader changed from 2a779b422c3c4bd59a5449144952c73a (127.4.65.3) to f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "f9e77ebb794b47949570cbc9f3e55646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:39.625283 21254 raft_consensus.cc:2749] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:39.625360 21233 consensus_queue.cc:1048] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:39.638563 21248 consensus_queue.cc:1048] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:39.700336 21089 raft_consensus.cc:3060] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:39.707525 21248 consensus_queue.cc:1048] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:39.714844 21248 consensus_queue.cc:1048] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:39.715423 21211 raft_consensus.cc:993] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a: : Instructing follower f9e77ebb794b47949570cbc9f3e55646 to start an election
I20260709 12:31:39.716070 21211 raft_consensus.cc:1081] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Signalling peer f9e77ebb794b47949570cbc9f3e55646 to start an election
I20260709 12:31:39.718699 21088 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ce50888244a14856b32e824b55bf67c6"
dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
 from {username='slave'} at 127.0.0.1:50774
I20260709 12:31:39.722949 21211 consensus_queue.cc:1048] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:39.723282 21088 raft_consensus.cc:493] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:39.723886 21088 raft_consensus.cc:3060] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:39.736565 21254 consensus_queue.cc:1048] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:39.739682 21248 consensus_queue.cc:1048] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:39.744784 21088 raft_consensus.cc:515] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.752800 21088 leader_election.cc:290] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:39.753556 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce50888244a14856b32e824b55bf67c6" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:39.754238 21164 raft_consensus.cc:3055] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:39.754585 21164 raft_consensus.cc:740] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:39.755702 21164 consensus_queue.cc:260] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.757205 21164 raft_consensus.cc:3060] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:39.763000 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce50888244a14856b32e824b55bf67c6" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:39.763876 21210 consensus_queue.cc:1048] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:39.766059 21014 raft_consensus.cc:3060] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:39.766723 21249 raft_consensus.cc:993] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a: : Instructing follower 0315789c9afb4bb69a62f2e94c2180ac to start an election
I20260709 12:31:39.767249 21211 raft_consensus.cc:1081] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Signalling peer 0315789c9afb4bb69a62f2e94c2180ac to start an election
I20260709 12:31:39.771021 21015 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "24bbbbdce7934410b6e080a04ab521f7"
dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
 from {username='slave'} at 127.0.0.1:56916
I20260709 12:31:39.771703 21015 raft_consensus.cc:493] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:39.772117 21015 raft_consensus.cc:3060] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:39.774223 21014 raft_consensus.cc:2468] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 2.
I20260709 12:31:39.775472 21055 leader_election.cc:304] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:39.776376 21212 raft_consensus.cc:2804] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:39.776976 21212 raft_consensus.cc:697] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [term 2 LEADER]: Becoming Leader. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:31:39.778064 21212 consensus_queue.cc:237] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.785275 21015 raft_consensus.cc:515] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:31:39.788398 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24bbbbdce7934410b6e080a04ab521f7" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:39.789049 21163 raft_consensus.cc:3055] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:39.789259 21088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24bbbbdce7934410b6e080a04ab521f7" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:39.789983 21088 raft_consensus.cc:3060] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:39.789361 21163 raft_consensus.cc:740] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:39.794480 21164 raft_consensus.cc:2468] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 2.
I20260709 12:31:39.796069 21088 raft_consensus.cc:2468] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 2.
I20260709 12:31:39.797654 21015 leader_election.cc:290] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 2 election: Requested vote from peers 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:39.797971 20978 leader_election.cc:304] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:39.799204 21210 raft_consensus.cc:2804] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:39.795437 21163 consensus_queue.cc:260] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:31:39.799685 21210 raft_consensus.cc:697] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 LEADER]: Becoming Leader. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:39.800141 21163 raft_consensus.cc:3060] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:39.800624 21210 consensus_queue.cc:237] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:31:39.808136 20912 catalog_manager.cc:5697] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 reported cstate change: term changed from 1 to 2, leader changed from 2a779b422c3c4bd59a5449144952c73a (127.4.65.3) to f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "f9e77ebb794b47949570cbc9f3e55646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:39.809273 21163 raft_consensus.cc:2468] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 2.
I20260709 12:31:39.809657 20913 catalog_manager.cc:5697] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac reported cstate change: term changed from 1 to 2, leader changed from 2a779b422c3c4bd59a5449144952c73a (127.4.65.3) to 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1). New cstate: current_term: 2 leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:39.887802 21265 raft_consensus.cc:993] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646: : Instructing follower 0315789c9afb4bb69a62f2e94c2180ac to start an election
I20260709 12:31:39.888285 21212 raft_consensus.cc:1081] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Signalling peer 0315789c9afb4bb69a62f2e94c2180ac to start an election
I20260709 12:31:39.889922 21015 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e082efdfaedc435c9b2265aba8e5d211"
dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
 from {username='slave'} at 127.0.0.1:56908
I20260709 12:31:39.890698 21015 raft_consensus.cc:493] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:39.891110 21015 raft_consensus.cc:3060] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:39.896834 21015 raft_consensus.cc:515] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.898454 21015 leader_election.cc:290] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 2 election: Requested vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:39.899647 21088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e082efdfaedc435c9b2265aba8e5d211" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:39.899657 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e082efdfaedc435c9b2265aba8e5d211" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:39.900296 21088 raft_consensus.cc:3055] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:39.900388 21163 raft_consensus.cc:3060] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:39.900738 21088 raft_consensus.cc:740] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:31:39.901566 21088 consensus_queue.cc:260] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.902604 21088 raft_consensus.cc:3060] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:39.905931 21163 raft_consensus.cc:2468] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 2.
I20260709 12:31:39.907196 20980 leader_election.cc:304] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: 
I20260709 12:31:39.907907 21210 raft_consensus.cc:2804] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:39.908382 21210 raft_consensus.cc:697] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 LEADER]: Becoming Leader. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:39.909209 21210 consensus_queue.cc:237] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:39.910230 21088 raft_consensus.cc:2468] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 2.
I20260709 12:31:39.917498 20913 catalog_manager.cc:5697] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac reported cstate change: term changed from 1 to 2, leader changed from f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2) to 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1). New cstate: current_term: 2 leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:39.992225 21015 raft_consensus.cc:3060] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:39.992564 21249 raft_consensus.cc:993] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a: : Instructing follower f9e77ebb794b47949570cbc9f3e55646 to start an election
I20260709 12:31:39.993106 21248 raft_consensus.cc:1081] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Signalling peer f9e77ebb794b47949570cbc9f3e55646 to start an election
I20260709 12:31:39.994805 21088 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d1aff18dd4734eada226c23cc72e68df"
dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
 from {username='slave'} at 127.0.0.1:50774
I20260709 12:31:39.995394 21088 raft_consensus.cc:493] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:39.995781 21088 raft_consensus.cc:3060] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:40.001416 21015 raft_consensus.cc:1275] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Refusing update from remote peer f9e77ebb794b47949570cbc9f3e55646: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:40.002486 21088 raft_consensus.cc:515] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:40.003319 21212 consensus_queue.cc:1048] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:40.004733 21088 leader_election.cc:290] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:40.005481 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1aff18dd4734eada226c23cc72e68df" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:40.006075 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1aff18dd4734eada226c23cc72e68df" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:40.006189 21015 raft_consensus.cc:3060] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:40.006763 21163 raft_consensus.cc:3055] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:40.007138 21163 raft_consensus.cc:740] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:40.008116 21163 consensus_queue.cc:260] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:40.009516 21163 raft_consensus.cc:3060] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:40.014424 21164 raft_consensus.cc:1275] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Refusing update from remote peer f9e77ebb794b47949570cbc9f3e55646: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:40.017977 21212 consensus_queue.cc:1048] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:40.021376 21015 raft_consensus.cc:2468] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 2.
I20260709 12:31:40.023869 21055 leader_election.cc:304] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:40.026441 21225 raft_consensus.cc:2804] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:40.026970 21225 raft_consensus.cc:697] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [term 2 LEADER]: Becoming Leader. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:31:40.027846 21225 consensus_queue.cc:237] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:40.028632 21163 raft_consensus.cc:2468] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 2.
I20260709 12:31:40.032804 21248 raft_consensus.cc:993] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a: : Instructing follower 0315789c9afb4bb69a62f2e94c2180ac to start an election
I20260709 12:31:40.034456 21248 raft_consensus.cc:1081] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Signalling peer 0315789c9afb4bb69a62f2e94c2180ac to start an election
I20260709 12:31:40.038067 21015 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9d54c78552d24a77a8b1a06013d70728"
dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
 from {username='slave'} at 127.0.0.1:56916
I20260709 12:31:40.038625 21015 raft_consensus.cc:493] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:40.038956 21015 raft_consensus.cc:3060] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:40.042275 20913 catalog_manager.cc:5697] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 reported cstate change: term changed from 1 to 2, leader changed from 2a779b422c3c4bd59a5449144952c73a (127.4.65.3) to f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "f9e77ebb794b47949570cbc9f3e55646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:40.047636 21015 raft_consensus.cc:515] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:40.050850 21015 leader_election.cc:290] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 2 election: Requested vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:40.051302 21088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d54c78552d24a77a8b1a06013d70728" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:40.051604 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d54c78552d24a77a8b1a06013d70728" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:40.052009 21088 raft_consensus.cc:3060] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:40.052161 21163 raft_consensus.cc:3055] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:40.052639 21163 raft_consensus.cc:740] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:40.053519 21163 consensus_queue.cc:260] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:40.054831 21163 raft_consensus.cc:3060] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:40.060628 21088 raft_consensus.cc:2468] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 2.
I20260709 12:31:40.062301 20978 leader_election.cc:304] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:40.063143 21210 raft_consensus.cc:2804] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:40.063575 21163 raft_consensus.cc:2468] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 2.
I20260709 12:31:40.063704 21210 raft_consensus.cc:697] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 LEADER]: Becoming Leader. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:40.064815 21210 consensus_queue.cc:237] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:40.071695 20913 catalog_manager.cc:5697] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac reported cstate change: term changed from 1 to 2, leader changed from 2a779b422c3c4bd59a5449144952c73a (127.4.65.3) to 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1). New cstate: current_term: 2 leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:40.218357 21015 raft_consensus.cc:1275] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Refusing update from remote peer f9e77ebb794b47949570cbc9f3e55646: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:40.219758 21225 consensus_queue.cc:1048] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:40.228938 21163 raft_consensus.cc:1275] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Refusing update from remote peer f9e77ebb794b47949570cbc9f3e55646: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:40.231032 21265 consensus_queue.cc:1048] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:40.373627 21088 raft_consensus.cc:1275] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Refusing update from remote peer 0315789c9afb4bb69a62f2e94c2180ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:40.375062 21210 consensus_queue.cc:1048] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:40.384709 21163 raft_consensus.cc:1275] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Refusing update from remote peer 0315789c9afb4bb69a62f2e94c2180ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:40.387481 21210 consensus_queue.cc:1048] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:40.395556 21163 raft_consensus.cc:1275] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Refusing update from remote peer 0315789c9afb4bb69a62f2e94c2180ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:40.396878 21210 consensus_queue.cc:1048] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:40.407037 21088 raft_consensus.cc:1275] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Refusing update from remote peer 0315789c9afb4bb69a62f2e94c2180ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:40.408324 21284 consensus_queue.cc:1048] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:40.417577 21015 raft_consensus.cc:1275] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Refusing update from remote peer f9e77ebb794b47949570cbc9f3e55646: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:40.419309 21265 consensus_queue.cc:1048] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:40.428344 21163 raft_consensus.cc:1275] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Refusing update from remote peer f9e77ebb794b47949570cbc9f3e55646: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:40.430051 21286 consensus_queue.cc:1048] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:40.473348 21163 raft_consensus.cc:1275] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Refusing update from remote peer 0315789c9afb4bb69a62f2e94c2180ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:40.474717 21284 consensus_queue.cc:1048] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:40.483876 21088 raft_consensus.cc:1275] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Refusing update from remote peer 0315789c9afb4bb69a62f2e94c2180ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:40.486083 21210 consensus_queue.cc:1048] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:31:41.830101 21111 debug-util.cc:398] Leaking SignalData structure 0x7b08000a0dc0 after lost signal to thread 4359
W20260709 12:31:41.831799 21111 debug-util.cc:398] Leaking SignalData structure 0x7b080003e420 after lost signal to thread 20947
W20260709 12:31:41.833665 21111 debug-util.cc:398] Leaking SignalData structure 0x7b08001c1400 after lost signal to thread 21039
W20260709 12:31:41.834975 21111 debug-util.cc:398] Leaking SignalData structure 0x7b0800133760 after lost signal to thread 21114
W20260709 12:31:41.836047 21111 debug-util.cc:398] Leaking SignalData structure 0x7b0800083140 after lost signal to thread 21188
I20260709 12:31:42.830518 20913 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49232:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:31:42.833241 20913 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:31:42.896818 21000 tablet_service.cc:1511] Processing CreateTablet for tablet 9d8d2fe503cf41af8b8952a9080ed696 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:31:42.898162 21000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d8d2fe503cf41af8b8952a9080ed696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.899549 21073 tablet_service.cc:1511] Processing CreateTablet for tablet 9d8d2fe503cf41af8b8952a9080ed696 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:31:42.901058 21073 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d8d2fe503cf41af8b8952a9080ed696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.906965 21077 tablet_service.cc:1511] Processing CreateTablet for tablet ea1880f9453549a5b74387f60aca8b3a (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:31:42.907372 21151 tablet_service.cc:1511] Processing CreateTablet for tablet 9d8d2fe503cf41af8b8952a9080ed696 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:31:42.908546 21077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1880f9453549a5b74387f60aca8b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.908684 21151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d8d2fe503cf41af8b8952a9080ed696. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.913331 21147 tablet_service.cc:1511] Processing CreateTablet for tablet ea1880f9453549a5b74387f60aca8b3a (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:31:42.914763 21147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1880f9453549a5b74387f60aca8b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.918231 21001 tablet_service.cc:1511] Processing CreateTablet for tablet 4255e87ce188451dab77427dc10cceb5 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:31:42.919631 21001 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4255e87ce188451dab77427dc10cceb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.923643 21005 tablet_service.cc:1511] Processing CreateTablet for tablet ea1880f9453549a5b74387f60aca8b3a (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:31:42.925276 21005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1880f9453549a5b74387f60aca8b3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.926719 21146 tablet_service.cc:1511] Processing CreateTablet for tablet 4255e87ce188451dab77427dc10cceb5 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:31:42.928117 21146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4255e87ce188451dab77427dc10cceb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.929097 21152 tablet_service.cc:1511] Processing CreateTablet for tablet 4a62f5d3a3bd438a896c90b6994dfe46 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:31:42.930755 21152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a62f5d3a3bd438a896c90b6994dfe46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.930469 21072 tablet_service.cc:1511] Processing CreateTablet for tablet 4255e87ce188451dab77427dc10cceb5 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:31:42.931942 21072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4255e87ce188451dab77427dc10cceb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.933751 21148 tablet_service.cc:1511] Processing CreateTablet for tablet 2c9a271be4694580b7755df0975444ae (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:31:42.935885 21148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c9a271be4694580b7755df0975444ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.937842 21154 tablet_service.cc:1511] Processing CreateTablet for tablet 3bc6713b68bd4ccfa12c5a02522a1b9f (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:31:42.939893 21154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bc6713b68bd4ccfa12c5a02522a1b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.940958 21074 tablet_service.cc:1511] Processing CreateTablet for tablet 3bc6713b68bd4ccfa12c5a02522a1b9f (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:31:42.942649 21074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bc6713b68bd4ccfa12c5a02522a1b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.942773 20999 tablet_service.cc:1511] Processing CreateTablet for tablet 2c9a271be4694580b7755df0975444ae (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:31:42.943186 21003 tablet_service.cc:1511] Processing CreateTablet for tablet 3bc6713b68bd4ccfa12c5a02522a1b9f (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:31:42.944285 20999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c9a271be4694580b7755df0975444ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.944898 21003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bc6713b68bd4ccfa12c5a02522a1b9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.945662 20997 tablet_service.cc:1511] Processing CreateTablet for tablet 4a62f5d3a3bd438a896c90b6994dfe46 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:31:42.947230 20997 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a62f5d3a3bd438a896c90b6994dfe46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.951902 21078 tablet_service.cc:1511] Processing CreateTablet for tablet 2c9a271be4694580b7755df0975444ae (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:31:42.953502 21078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c9a271be4694580b7755df0975444ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.956179 21002 tablet_service.cc:1511] Processing CreateTablet for tablet c4f782fea1df4f63aa3da3e6e8a85bd2 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:31:42.956564 20998 tablet_service.cc:1511] Processing CreateTablet for tablet 2b57143f3d8f47b7b11ac682e7327bd6 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:31:42.957710 21002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4f782fea1df4f63aa3da3e6e8a85bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.958503 20998 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b57143f3d8f47b7b11ac682e7327bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.958262 21149 tablet_service.cc:1511] Processing CreateTablet for tablet c4f782fea1df4f63aa3da3e6e8a85bd2 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:31:42.959098 21004 tablet_service.cc:1511] Processing CreateTablet for tablet 400df23b40104970bb996c614dbf7657 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:31:42.959656 21149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4f782fea1df4f63aa3da3e6e8a85bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.960337 21004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 400df23b40104970bb996c614dbf7657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.962997 21150 tablet_service.cc:1511] Processing CreateTablet for tablet 2b57143f3d8f47b7b11ac682e7327bd6 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:31:42.965541 21150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b57143f3d8f47b7b11ac682e7327bd6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:31:42.975286 20961 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:31:42.977901 21153 tablet_service.cc:1511] Processing CreateTablet for tablet 400df23b40104970bb996c614dbf7657 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:31:42.979351 21153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 400df23b40104970bb996c614dbf7657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.984720 21080 tablet_service.cc:1511] Processing CreateTablet for tablet c4f782fea1df4f63aa3da3e6e8a85bd2 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:31:42.985636 21076 tablet_service.cc:1511] Processing CreateTablet for tablet 4a62f5d3a3bd438a896c90b6994dfe46 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:31:42.986620 21080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4f782fea1df4f63aa3da3e6e8a85bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.987464 21076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a62f5d3a3bd438a896c90b6994dfe46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:42.990525 21079 tablet_service.cc:1511] Processing CreateTablet for tablet 2b57143f3d8f47b7b11ac682e7327bd6 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:31:42.992329 21079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b57143f3d8f47b7b11ac682e7327bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:43.012539 21075 tablet_service.cc:1511] Processing CreateTablet for tablet 400df23b40104970bb996c614dbf7657 (DEFAULT_TABLE table=soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:31:43.014034 21075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 400df23b40104970bb996c614dbf7657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:31:43.123890 21301 tablet_bootstrap.cc:492] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:43.164623 21303 tablet_bootstrap.cc:492] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:43.190784 21301 tablet_bootstrap.cc:654] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.194579 21304 tablet_bootstrap.cc:492] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:43.202621 21304 tablet_bootstrap.cc:654] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.193737 21303 tablet_bootstrap.cc:654] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.217120 21304 tablet_bootstrap.cc:492] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:43.217324 21301 tablet_bootstrap.cc:492] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:43.217888 21304 ts_tablet_manager.cc:1403] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.003s
I20260709 12:31:43.218066 21301 ts_tablet_manager.cc:1403] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.095s	user 0.022s	sys 0.006s
I20260709 12:31:43.218504 21303 tablet_bootstrap.cc:492] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:43.219307 21303 ts_tablet_manager.cc:1403] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.055s	user 0.019s	sys 0.000s
I20260709 12:31:43.221454 21304 raft_consensus.cc:359] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.221873 21301 raft_consensus.cc:359] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.222376 21304 raft_consensus.cc:385] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.222757 21301 raft_consensus.cc:385] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.222822 21304 raft_consensus.cc:740] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.222718 21303 raft_consensus.cc:359] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.223520 21301 raft_consensus.cc:740] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.223865 21303 raft_consensus.cc:385] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.224149 21304 consensus_queue.cc:260] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.230386 21303 raft_consensus.cc:740] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.231266 21303 consensus_queue.cc:260] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.224681 21301 consensus_queue.cc:260] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.233987 21303 ts_tablet_manager.cc:1434] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:31:43.235020 21303 tablet_bootstrap.cc:492] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:43.234966 21301 ts_tablet_manager.cc:1434] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.007s
I20260709 12:31:43.237033 21304 ts_tablet_manager.cc:1434] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.019s	user 0.012s	sys 0.004s
I20260709 12:31:43.237497 21301 tablet_bootstrap.cc:492] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:43.243501 21303 tablet_bootstrap.cc:654] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.245132 21301 tablet_bootstrap.cc:654] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.237376 21302 raft_consensus.cc:493] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:43.264964 21302 raft_consensus.cc:515] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.269622 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9a271be4694580b7755df0975444ae" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
W20260709 12:31:43.273938 20978 leader_election.cc:343] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.275910 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9a271be4694580b7755df0975444ae" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:43.277844 21302 leader_election.cc:290] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
W20260709 12:31:43.279215 20980 leader_election.cc:343] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.280434 20980 leader_election.cc:304] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac; no voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:43.303011 21302 raft_consensus.cc:2749] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:43.303733 21304 tablet_bootstrap.cc:492] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:43.323633 21304 tablet_bootstrap.cc:654] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.359767 21301 tablet_bootstrap.cc:492] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:43.359622 21303 tablet_bootstrap.cc:492] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:43.360740 21301 ts_tablet_manager.cc:1403] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.125s	user 0.047s	sys 0.071s
I20260709 12:31:43.360839 21303 ts_tablet_manager.cc:1403] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.126s	user 0.045s	sys 0.065s
I20260709 12:31:43.364331 21301 raft_consensus.cc:359] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.365124 21301 raft_consensus.cc:385] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.364760 21303 raft_consensus.cc:359] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.365474 21301 raft_consensus.cc:740] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.365725 21303 raft_consensus.cc:385] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.366115 21303 raft_consensus.cc:740] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.366328 21301 consensus_queue.cc:260] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.366962 21303 consensus_queue.cc:260] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.368894 21301 ts_tablet_manager.cc:1434] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:31:43.369699 21303 ts_tablet_manager.cc:1434] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.003s
I20260709 12:31:43.370851 21303 tablet_bootstrap.cc:492] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:43.375810 21304 tablet_bootstrap.cc:492] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:43.376797 21304 ts_tablet_manager.cc:1403] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.073s	user 0.018s	sys 0.005s
I20260709 12:31:43.380261 21304 raft_consensus.cc:359] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.381140 21304 raft_consensus.cc:385] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.381456 21304 raft_consensus.cc:740] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.382292 21304 consensus_queue.cc:260] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.383226 21301 tablet_bootstrap.cc:492] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:43.390990 21303 tablet_bootstrap.cc:654] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.392776 21301 tablet_bootstrap.cc:654] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.394150 21304 ts_tablet_manager.cc:1434] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:31:43.395481 21304 tablet_bootstrap.cc:492] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:43.403005 21304 tablet_bootstrap.cc:654] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.412197 21303 tablet_bootstrap.cc:492] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:43.412817 21303 ts_tablet_manager.cc:1403] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.042s	user 0.015s	sys 0.000s
I20260709 12:31:43.416244 21303 raft_consensus.cc:359] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.417100 21303 raft_consensus.cc:385] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.417361 21289 raft_consensus.cc:493] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:43.417423 21303 raft_consensus.cc:740] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.418008 21289 raft_consensus.cc:515] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.420660 21303 consensus_queue.cc:260] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.419989 21304 tablet_bootstrap.cc:492] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:43.422552 21304 ts_tablet_manager.cc:1403] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.027s	user 0.010s	sys 0.005s
I20260709 12:31:43.425689 21289 leader_election.cc:290] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:43.426064 21304 raft_consensus.cc:359] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.426968 21304 raft_consensus.cc:385] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.427294 21304 raft_consensus.cc:740] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.427243 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:43.428542 21304 consensus_queue.cc:260] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
W20260709 12:31:43.429767 21129 leader_election.cc:343] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.431697 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:43.432205 21304 ts_tablet_manager.cc:1434] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:31:43.435437 21304 tablet_bootstrap.cc:492] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
W20260709 12:31:43.438161 21127 leader_election.cc:343] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.438923 21127 leader_election.cc:304] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a; no voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:43.445492 21304 tablet_bootstrap.cc:654] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.464102 21289 raft_consensus.cc:2749] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:43.473853 21300 raft_consensus.cc:493] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:43.475070 21300 raft_consensus.cc:515] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.482074 21300 leader_election.cc:290] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:43.485522 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4255e87ce188451dab77427dc10cceb5" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:43.491423 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4255e87ce188451dab77427dc10cceb5" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
W20260709 12:31:43.494536 21055 leader_election.cc:343] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:31:43.495577 21055 leader_election.cc:343] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.496114 21055 leader_election.cc:304] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a
I20260709 12:31:43.513127 21300 raft_consensus.cc:2749] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:43.549166 21303 ts_tablet_manager.cc:1434] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.136s	user 0.042s	sys 0.026s
I20260709 12:31:43.550586 21303 tablet_bootstrap.cc:492] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:43.553551 21301 tablet_bootstrap.cc:492] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:43.557188 21304 tablet_bootstrap.cc:492] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:43.557749 21304 ts_tablet_manager.cc:1403] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.123s	user 0.016s	sys 0.041s
I20260709 12:31:43.555792 21301 ts_tablet_manager.cc:1403] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.173s	user 0.080s	sys 0.066s
I20260709 12:31:43.558704 21303 tablet_bootstrap.cc:654] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.562407 21301 raft_consensus.cc:359] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.563294 21301 raft_consensus.cc:385] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.564584 21301 raft_consensus.cc:740] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.611548 21289 raft_consensus.cc:493] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:43.620885 21289 raft_consensus.cc:515] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.565455 21301 consensus_queue.cc:260] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.622925 21254 raft_consensus.cc:493] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:43.624325 21254 raft_consensus.cc:515] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.629278 21316 raft_consensus.cc:493] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:43.630401 21316 raft_consensus.cc:515] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.630945 21254 leader_election.cc:290] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:43.632879 21303 tablet_bootstrap.cc:492] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:43.633527 21316 leader_election.cc:290] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:43.629483 21289 leader_election.cc:290] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:43.634120 21303 ts_tablet_manager.cc:1403] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.084s	user 0.007s	sys 0.008s
I20260709 12:31:43.610803 21304 raft_consensus.cc:359] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.636710 21304 raft_consensus.cc:385] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.638037 21304 raft_consensus.cc:740] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.639776 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bc6713b68bd4ccfa12c5a02522a1b9f" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:43.640180 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b57143f3d8f47b7b11ac682e7327bd6" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:43.636574 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bc6713b68bd4ccfa12c5a02522a1b9f" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
W20260709 12:31:43.642045 21055 leader_election.cc:343] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.637014 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:43.629333 21301 ts_tablet_manager.cc:1434] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.070s	user 0.010s	sys 0.002s
W20260709 12:31:43.643594 20980 leader_election.cc:343] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.645066 21301 tablet_bootstrap.cc:492] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
W20260709 12:31:43.648149 21055 leader_election.cc:343] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.648861 21055 leader_election.cc:304] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a
W20260709 12:31:43.649333 21127 leader_election.cc:343] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.651836 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b57143f3d8f47b7b11ac682e7327bd6" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:43.654672 21087 raft_consensus.cc:2468] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 0.
I20260709 12:31:43.654843 21303 raft_consensus.cc:359] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.655751 21303 raft_consensus.cc:385] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.656121 20978 leader_election.cc:304] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 2a779b422c3c4bd59a5449144952c73a
I20260709 12:31:43.653009 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:43.653760 21316 raft_consensus.cc:2749] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:43.659823 21254 raft_consensus.cc:2804] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:43.660084 21301 tablet_bootstrap.cc:654] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.660382 21254 raft_consensus.cc:493] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:31:43.661258 21129 leader_election.cc:343] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.639817 21304 consensus_queue.cc:260] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.660967 21254 raft_consensus.cc:3060] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:43.661840 21129 leader_election.cc:304] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a; no voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:43.656066 21303 raft_consensus.cc:740] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.666910 21303 consensus_queue.cc:260] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.668759 21300 raft_consensus.cc:493] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:43.669575 21303 ts_tablet_manager.cc:1434] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.035s	user 0.010s	sys 0.004s
I20260709 12:31:43.669425 21300 raft_consensus.cc:515] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.670527 21314 raft_consensus.cc:2749] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:43.672848 21302 raft_consensus.cc:493] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:43.673503 21302 raft_consensus.cc:515] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.676010 21303 tablet_bootstrap.cc:492] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:43.672207 21300 leader_election.cc:290] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:43.689651 21302 leader_election.cc:290] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:43.690455 21303 tablet_bootstrap.cc:654] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.691242 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:43.692191 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "400df23b40104970bb996c614dbf7657" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:43.693166 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:43.693123 21164 raft_consensus.cc:2468] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 0.
I20260709 12:31:43.694566 20980 leader_election.cc:304] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: 
I20260709 12:31:43.695413 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "400df23b40104970bb996c614dbf7657" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:43.695806 21302 raft_consensus.cc:2804] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:43.696547 21302 raft_consensus.cc:493] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:43.697046 21302 raft_consensus.cc:3060] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:31:43.697875 21055 leader_election.cc:343] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:31:43.699796 20978 leader_election.cc:343] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.692157 21163 raft_consensus.cc:2468] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 0.
I20260709 12:31:43.701977 21055 leader_election.cc:304] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:31:43.665517 21304 ts_tablet_manager.cc:1434] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.107s	user 0.010s	sys 0.006s
I20260709 12:31:43.704555 21300 raft_consensus.cc:2804] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:43.705209 21300 raft_consensus.cc:493] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:43.705649 21300 raft_consensus.cc:3060] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:43.711601 21301 tablet_bootstrap.cc:492] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:43.712049 21304 tablet_bootstrap.cc:492] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:43.712386 21301 ts_tablet_manager.cc:1403] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.068s	user 0.022s	sys 0.000s
I20260709 12:31:43.718226 21301 raft_consensus.cc:359] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.719708 21301 raft_consensus.cc:385] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.720136 21301 raft_consensus.cc:740] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.721114 21301 consensus_queue.cc:260] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.726022 21254 raft_consensus.cc:515] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.729748 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b57143f3d8f47b7b11ac682e7327bd6" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:43.730718 21087 raft_consensus.cc:3060] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:43.739918 21304 tablet_bootstrap.cc:654] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.740233 21300 raft_consensus.cc:515] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.763765 21303 tablet_bootstrap.cc:492] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:43.764675 21303 ts_tablet_manager.cc:1403] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.089s	user 0.029s	sys 0.020s
I20260709 12:31:43.764402 21302 raft_consensus.cc:515] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.767174 21302 leader_election.cc:290] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 election: Requested vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:43.768414 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "400df23b40104970bb996c614dbf7657" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:43.768460 21088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "400df23b40104970bb996c614dbf7657" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:43.769268 21163 raft_consensus.cc:3060] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:43.768850 21303 raft_consensus.cc:359] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.769809 21303 raft_consensus.cc:385] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.770162 21303 raft_consensus.cc:740] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
W20260709 12:31:43.770577 20978 leader_election.cc:343] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.771056 21303 consensus_queue.cc:260] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.773563 21254 leader_election.cc:290] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 election: Requested vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:43.774176 21303 ts_tablet_manager.cc:1434] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.004s
I20260709 12:31:43.773993 21300 leader_election.cc:290] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:43.780124 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:43.778541 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:43.782122 21164 raft_consensus.cc:3060] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:43.782570 21303 tablet_bootstrap.cc:492] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:43.782742 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b57143f3d8f47b7b11ac682e7327bd6" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:43.783818 21162 raft_consensus.cc:3060] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:43.787582 21087 raft_consensus.cc:2468] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 1.
I20260709 12:31:43.789237 20978 leader_election.cc:304] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:43.790371 21254 raft_consensus.cc:2804] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:31:43.790731 21055 leader_election.cc:343] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.791008 21254 raft_consensus.cc:697] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Becoming Leader. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:43.793659 21254 consensus_queue.cc:237] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.794814 21301 ts_tablet_manager.cc:1434] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.082s	user 0.012s	sys 0.029s
I20260709 12:31:43.796449 21301 tablet_bootstrap.cc:492] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:43.798668 21303 tablet_bootstrap.cc:654] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.807101 21164 raft_consensus.cc:2468] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 1.
I20260709 12:31:43.808509 21055 leader_election.cc:304] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:31:43.809901 21300 raft_consensus.cc:2804] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:43.810746 21300 raft_consensus.cc:697] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Becoming Leader. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:31:43.811614 21163 raft_consensus.cc:2468] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 1.
I20260709 12:31:43.811833 21300 consensus_queue.cc:237] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.813009 20980 leader_election.cc:304] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:43.820814 21322 raft_consensus.cc:2804] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:43.822197 21322 raft_consensus.cc:697] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Becoming Leader. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:43.822825 21304 tablet_bootstrap.cc:492] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:43.823474 21304 ts_tablet_manager.cc:1403] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.112s	user 0.027s	sys 0.016s
I20260709 12:31:43.823354 21322 consensus_queue.cc:237] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.830112 21162 raft_consensus.cc:2468] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 1.
I20260709 12:31:43.830932 21304 raft_consensus.cc:359] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.833097 21304 raft_consensus.cc:385] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.832563 21303 tablet_bootstrap.cc:492] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:43.833346 21301 tablet_bootstrap.cc:654] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.833851 21303 ts_tablet_manager.cc:1403] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.052s	user 0.006s	sys 0.024s
I20260709 12:31:43.833981 20910 catalog_manager.cc:5697] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1). New cstate: current_term: 1 leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:43.833457 21304 raft_consensus.cc:740] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.843797 20912 catalog_manager.cc:5697] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2). New cstate: current_term: 1 leader_uuid: "f9e77ebb794b47949570cbc9f3e55646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:43.844962 21303 raft_consensus.cc:359] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.845997 21303 raft_consensus.cc:385] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.846351 21303 raft_consensus.cc:740] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.847283 21303 consensus_queue.cc:260] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.850329 21303 ts_tablet_manager.cc:1434] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.005s
I20260709 12:31:43.845126 21304 consensus_queue.cc:260] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.854902 21304 ts_tablet_manager.cc:1434] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.031s	user 0.008s	sys 0.004s
I20260709 12:31:43.856599 21304 tablet_bootstrap.cc:492] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:43.859340 21301 tablet_bootstrap.cc:492] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:43.860055 21301 ts_tablet_manager.cc:1403] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.064s	user 0.012s	sys 0.003s
I20260709 12:31:43.863334 21301 raft_consensus.cc:359] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.864176 21303 tablet_bootstrap.cc:492] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:43.869408 21301 raft_consensus.cc:385] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.870028 21301 raft_consensus.cc:740] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.872891 21304 tablet_bootstrap.cc:654] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.873476 21301 consensus_queue.cc:260] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.876610 21301 ts_tablet_manager.cc:1434] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.004s
I20260709 12:31:43.877825 21303 tablet_bootstrap.cc:654] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.878093 21301 tablet_bootstrap.cc:492] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:43.887377 21301 tablet_bootstrap.cc:654] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.890930 21304 tablet_bootstrap.cc:492] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:43.892676 21304 ts_tablet_manager.cc:1403] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.037s	user 0.013s	sys 0.007s
I20260709 12:31:43.897203 21304 raft_consensus.cc:359] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.900794 20910 catalog_manager.cc:5697] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1). New cstate: current_term: 1 leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:43.906777 21304 raft_consensus.cc:385] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.907212 21304 raft_consensus.cc:740] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.908525 21303 tablet_bootstrap.cc:492] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:43.908103 21304 consensus_queue.cc:260] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.909175 21303 ts_tablet_manager.cc:1403] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.052s	user 0.018s	sys 0.000s
I20260709 12:31:43.911087 21304 ts_tablet_manager.cc:1434] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.001s
I20260709 12:31:43.913386 21303 raft_consensus.cc:359] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.914278 21303 raft_consensus.cc:385] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.914646 21303 raft_consensus.cc:740] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.914770 21301 tablet_bootstrap.cc:492] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:43.915325 21289 raft_consensus.cc:493] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:43.915567 21303 consensus_queue.cc:260] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.916082 21289 raft_consensus.cc:515] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.917474 21304 tablet_bootstrap.cc:492] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:43.921080 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9a271be4694580b7755df0975444ae" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:43.922711 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9a271be4694580b7755df0975444ae" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:43.923702 21014 raft_consensus.cc:2468] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 0.
I20260709 12:31:43.925150 21129 leader_election.cc:304] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: 
I20260709 12:31:43.925382 21304 tablet_bootstrap.cc:654] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.926226 21314 raft_consensus.cc:2804] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:31:43.926396 21127 leader_election.cc:343] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.926715 21314 raft_consensus.cc:493] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:43.927121 21314 raft_consensus.cc:3060] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:43.927297 21289 leader_election.cc:290] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:43.950219 21314 raft_consensus.cc:515] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.953359 21301 ts_tablet_manager.cc:1403] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.076s	user 0.014s	sys 0.014s
I20260709 12:31:43.956133 21303 ts_tablet_manager.cc:1434] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.047s	user 0.009s	sys 0.000s
I20260709 12:31:43.958636 21303 tablet_bootstrap.cc:492] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:43.963744 21337 raft_consensus.cc:493] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:43.964821 21337 raft_consensus.cc:515] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.968163 21337 leader_election.cc:290] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:43.968900 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4f782fea1df4f63aa3da3e6e8a85bd2" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:43.969707 21014 raft_consensus.cc:2468] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 0.
I20260709 12:31:43.971067 21129 leader_election.cc:304] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: 
I20260709 12:31:43.972877 21337 raft_consensus.cc:2804] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:43.973497 21337 raft_consensus.cc:493] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:43.974049 21337 raft_consensus.cc:3060] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:43.977414 21314 leader_election.cc:290] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:43.978219 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4f782fea1df4f63aa3da3e6e8a85bd2" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:43.970992 21301 raft_consensus.cc:359] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
W20260709 12:31:43.979179 20961 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:31:43.979245 21087 raft_consensus.cc:2468] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 0.
I20260709 12:31:43.979638 21303 tablet_bootstrap.cc:654] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.979698 21301 raft_consensus.cc:385] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.980367 21088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9a271be4694580b7755df0975444ae" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:43.980315 21301 raft_consensus.cc:740] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.981768 21304 tablet_bootstrap.cc:492] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
W20260709 12:31:43.981920 21127 leader_election.cc:343] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:43.982512 21304 ts_tablet_manager.cc:1403] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.065s	user 0.029s	sys 0.031s
I20260709 12:31:43.982319 21301 consensus_queue.cc:260] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.983155 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9a271be4694580b7755df0975444ae" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:43.983796 21014 raft_consensus.cc:3060] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:43.985992 21304 raft_consensus.cc:359] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.986855 21304 raft_consensus.cc:385] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:43.989526 21301 ts_tablet_manager.cc:1434] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.002s
I20260709 12:31:43.990247 21337 raft_consensus.cc:515] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:43.987197 21304 raft_consensus.cc:740] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:43.991200 21301 tablet_bootstrap.cc:492] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:43.993278 21304 consensus_queue.cc:260] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.003168 21304 ts_tablet_manager.cc:1434] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.020s	user 0.000s	sys 0.005s
I20260709 12:31:43.995473 21337 leader_election.cc:290] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:43.996719 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4f782fea1df4f63aa3da3e6e8a85bd2" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:44.005136 21304 tablet_bootstrap.cc:492] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:43.997053 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4f782fea1df4f63aa3da3e6e8a85bd2" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:44.005945 21087 raft_consensus.cc:3060] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.007907 21301 tablet_bootstrap.cc:654] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:44.005133 21015 raft_consensus.cc:3060] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.011722 21014 raft_consensus.cc:2468] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 1.
I20260709 12:31:44.013140 21129 leader_election.cc:304] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:44.013556 21304 tablet_bootstrap.cc:654] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:43.998219 21303 tablet_bootstrap.cc:492] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:44.015058 21303 ts_tablet_manager.cc:1403] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.057s	user 0.018s	sys 0.001s
I20260709 12:31:44.018152 21337 raft_consensus.cc:2804] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:44.019455 21337 raft_consensus.cc:697] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:44.020532 21337 consensus_queue.cc:237] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.034230 21322 raft_consensus.cc:493] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:44.035072 21322 raft_consensus.cc:515] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.037959 21322 leader_election.cc:290] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.040004 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bc6713b68bd4ccfa12c5a02522a1b9f" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:44.043767 21087 raft_consensus.cc:2468] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 1.
I20260709 12:31:44.045149 21088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bc6713b68bd4ccfa12c5a02522a1b9f" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:44.037328 21015 raft_consensus.cc:2468] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 1.
I20260709 12:31:44.046337 21088 raft_consensus.cc:2468] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 0.
I20260709 12:31:44.040048 21303 raft_consensus.cc:359] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.047420 21127 leader_election.cc:304] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:44.049098 20978 leader_election.cc:304] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:44.046342 21301 tablet_bootstrap.cc:492] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:44.050478 21301 ts_tablet_manager.cc:1403] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.060s	user 0.008s	sys 0.010s
I20260709 12:31:44.054030 21303 raft_consensus.cc:385] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:44.054404 21314 raft_consensus.cc:2804] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:44.056231 21314 raft_consensus.cc:697] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:44.057210 21314 consensus_queue.cc:237] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.055033 21303 raft_consensus.cc:740] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:44.058279 21301 raft_consensus.cc:359] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.059345 21301 raft_consensus.cc:385] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:44.053932 21322 raft_consensus.cc:2804] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:44.059733 21301 raft_consensus.cc:740] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:44.060074 21322 raft_consensus.cc:493] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:44.059540 21303 consensus_queue.cc:260] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.060575 21322 raft_consensus.cc:3060] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.061518 21301 consensus_queue.cc:260] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.064733 21301 ts_tablet_manager.cc:1434] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.014s	user 0.010s	sys 0.000s
I20260709 12:31:44.066455 21301 tablet_bootstrap.cc:492] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:44.072973 21162 raft_consensus.cc:2468] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 0.
I20260709 12:31:44.075289 21303 ts_tablet_manager.cc:1434] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.060s	user 0.011s	sys 0.000s
I20260709 12:31:44.076915 21303 tablet_bootstrap.cc:492] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a: Bootstrap starting.
I20260709 12:31:44.082875 21303 tablet_bootstrap.cc:654] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:44.083742 21322 raft_consensus.cc:515] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.084461 20910 catalog_manager.cc:5697] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 1 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:44.088507 21088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bc6713b68bd4ccfa12c5a02522a1b9f" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:44.089172 21088 raft_consensus.cc:3060] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.091727 21300 raft_consensus.cc:493] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:44.092381 21300 raft_consensus.cc:515] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.094626 21300 leader_election.cc:290] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.097231 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
W20260709 12:31:44.099961 21055 leader_election.cc:343] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:44.099974 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bc6713b68bd4ccfa12c5a02522a1b9f" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:44.101085 21163 raft_consensus.cc:3060] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.101082 21322 leader_election.cc:290] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 election: Requested vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.101238 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:44.102630 21162 raft_consensus.cc:2468] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 0.
I20260709 12:31:44.105309 21055 leader_election.cc:304] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:31:44.105746 21301 tablet_bootstrap.cc:654] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:44.106617 21300 raft_consensus.cc:2804] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:44.107056 21300 raft_consensus.cc:493] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:44.107388 21300 raft_consensus.cc:3060] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.110944 21163 raft_consensus.cc:2468] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 1.
I20260709 12:31:44.113886 20980 leader_election.cc:304] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: 
I20260709 12:31:44.115821 21304 tablet_bootstrap.cc:492] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:44.116483 21304 ts_tablet_manager.cc:1403] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.112s	user 0.013s	sys 0.011s
I20260709 12:31:44.116189 21254 raft_consensus.cc:2804] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:44.117205 21254 raft_consensus.cc:697] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Becoming Leader. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:44.119105 21321 raft_consensus.cc:493] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:44.118713 21254 consensus_queue.cc:237] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.119829 21304 raft_consensus.cc:359] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.120746 21304 raft_consensus.cc:385] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:44.121071 21304 raft_consensus.cc:740] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:44.121870 21304 consensus_queue.cc:260] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.119661 21321 raft_consensus.cc:515] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.124630 21321 leader_election.cc:290] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.126870 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4255e87ce188451dab77427dc10cceb5" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:44.127851 21087 raft_consensus.cc:2468] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 0.
I20260709 12:31:44.128271 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4255e87ce188451dab77427dc10cceb5" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:44.128996 21164 raft_consensus.cc:2468] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 0.
I20260709 12:31:44.129431 20978 leader_election.cc:304] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:44.130472 21321 raft_consensus.cc:2804] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:44.130972 21321 raft_consensus.cc:493] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:44.131399 21321 raft_consensus.cc:3060] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.137339 21300 raft_consensus.cc:515] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.138896 21088 raft_consensus.cc:2468] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 1.
I20260709 12:31:44.141548 21300 leader_election.cc:290] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.142683 21300 raft_consensus.cc:493] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:44.143181 21300 raft_consensus.cc:515] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.144330 21304 ts_tablet_manager.cc:1434] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.000s
I20260709 12:31:44.144582 20910 catalog_manager.cc:5697] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 1 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:44.147936 21301 tablet_bootstrap.cc:492] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:44.148641 21301 ts_tablet_manager.cc:1403] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.083s	user 0.013s	sys 0.012s
I20260709 12:31:44.152191 21301 raft_consensus.cc:359] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.145718 21300 leader_election.cc:290] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.146346 21321 raft_consensus.cc:515] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.153247 21301 raft_consensus.cc:385] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:44.154116 21301 raft_consensus.cc:740] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:44.155315 21301 consensus_queue.cc:260] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.158071 21304 tablet_bootstrap.cc:492] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646: Bootstrap starting.
I20260709 12:31:44.159346 21301 ts_tablet_manager.cc:1434] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:31:44.161209 21301 tablet_bootstrap.cc:492] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac: Bootstrap starting.
I20260709 12:31:44.164379 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:44.166651 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9a271be4694580b7755df0975444ae" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:44.166837 21088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4255e87ce188451dab77427dc10cceb5" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:44.167516 21015 raft_consensus.cc:2393] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9e77ebb794b47949570cbc9f3e55646 in current term 1: Already voted for candidate 2a779b422c3c4bd59a5449144952c73a in this term.
I20260709 12:31:44.167886 21164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:44.167452 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9a271be4694580b7755df0975444ae" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:44.167498 21088 raft_consensus.cc:3060] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.168071 21321 leader_election.cc:290] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 election: Requested vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.170341 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4255e87ce188451dab77427dc10cceb5" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:44.171018 21163 raft_consensus.cc:3060] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.168722 21164 raft_consensus.cc:3060] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.177232 21303 tablet_bootstrap.cc:492] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a: No bootstrap required, opened a new log
I20260709 12:31:44.177970 21303 ts_tablet_manager.cc:1403] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a: Time spent bootstrapping tablet: real 0.101s	user 0.009s	sys 0.009s
W20260709 12:31:44.178323 21055 leader_election.cc:343] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:44.179188 21088 raft_consensus.cc:2468] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 1.
I20260709 12:31:44.180810 21321 raft_consensus.cc:493] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:44.181418 21321 raft_consensus.cc:515] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.182693 20978 leader_election.cc:304] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:44.182802 21055 leader_election.cc:304] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a
I20260709 12:31:44.183828 21322 raft_consensus.cc:2804] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:44.184316 21322 raft_consensus.cc:697] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Becoming Leader. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:44.184803 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea1880f9453549a5b74387f60aca8b3a" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:44.185181 21322 consensus_queue.cc:237] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.185581 21303 raft_consensus.cc:359] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
W20260709 12:31:44.186259 20980 leader_election.cc:343] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:31:44.173655 21301 tablet_bootstrap.cc:654] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:44.173595 21304 tablet_bootstrap.cc:654] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646: Neither blocks nor log segments found. Creating new log.
I20260709 12:31:44.189841 20910 catalog_manager.cc:5697] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1). New cstate: current_term: 1 leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:44.193068 21316 raft_consensus.cc:3060] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.186257 21303 raft_consensus.cc:385] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:44.193631 21303 raft_consensus.cc:740] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Initialized, Role: FOLLOWER
I20260709 12:31:44.194561 21303 consensus_queue.cc:260] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.198460 21303 ts_tablet_manager.cc:1434] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.002s
I20260709 12:31:44.186358 21321 leader_election.cc:290] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.206456 21316 raft_consensus.cc:2749] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:44.204399 21163 raft_consensus.cc:2468] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 1.
I20260709 12:31:44.184808 21088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea1880f9453549a5b74387f60aca8b3a" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:44.217381 20912 catalog_manager.cc:5697] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1). New cstate: current_term: 1 leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:44.217952 21088 raft_consensus.cc:2468] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 0.
I20260709 12:31:44.221805 20978 leader_election.cc:304] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 2a779b422c3c4bd59a5449144952c73a
I20260709 12:31:44.223204 21321 raft_consensus.cc:2804] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:31:44.223562 21321 raft_consensus.cc:493] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:44.223912 21321 raft_consensus.cc:3060] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.182997 21164 raft_consensus.cc:2468] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 1.
I20260709 12:31:44.225476 21055 leader_election.cc:304] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:31:44.227385 21316 raft_consensus.cc:2804] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:44.227890 21316 raft_consensus.cc:697] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Becoming Leader. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:31:44.228803 21316 consensus_queue.cc:237] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.211018 21322 consensus_queue.cc:1048] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:44.214967 21254 consensus_queue.cc:1048] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:44.215777 21301 tablet_bootstrap.cc:492] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac: No bootstrap required, opened a new log
I20260709 12:31:44.233865 21301 ts_tablet_manager.cc:1403] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent bootstrapping tablet: real 0.073s	user 0.012s	sys 0.001s
I20260709 12:31:44.238050 21301 raft_consensus.cc:359] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.238817 21301 raft_consensus.cc:385] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:44.242108 21301 raft_consensus.cc:740] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Initialized, Role: FOLLOWER
I20260709 12:31:44.242887 21301 consensus_queue.cc:260] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.245275 21301 ts_tablet_manager.cc:1434] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.001s
I20260709 12:31:44.216300 21304 tablet_bootstrap.cc:492] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646: No bootstrap required, opened a new log
I20260709 12:31:44.246937 21304 ts_tablet_manager.cc:1403] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646: Time spent bootstrapping tablet: real 0.089s	user 0.007s	sys 0.007s
I20260709 12:31:44.240765 21321 raft_consensus.cc:515] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.256933 21088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea1880f9453549a5b74387f60aca8b3a" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:44.257797 21088 raft_consensus.cc:3060] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.260066 20912 catalog_manager.cc:5697] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2). New cstate: current_term: 1 leader_uuid: "f9e77ebb794b47949570cbc9f3e55646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
W20260709 12:31:44.269083 20978 consensus_peers.cc:597] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac -> Peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Couldn't send request to peer f9e77ebb794b47949570cbc9f3e55646. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:44.272221 21254 raft_consensus.cc:493] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:44.273106 21254 raft_consensus.cc:515] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.274073 21321 leader_election.cc:290] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 election: Requested vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.251027 21304 raft_consensus.cc:359] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.275484 21304 raft_consensus.cc:385] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:31:44.275876 21304 raft_consensus.cc:740] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Initialized, Role: FOLLOWER
I20260709 12:31:44.276439 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:44.276924 21304 consensus_queue.cc:260] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.272490 21088 raft_consensus.cc:2468] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 1.
I20260709 12:31:44.280483 21304 ts_tablet_manager.cc:1434] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646: Time spent starting tablet: real 0.033s	user 0.008s	sys 0.001s
I20260709 12:31:44.289135 20978 leader_election.cc:304] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:44.290257 21302 raft_consensus.cc:2804] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:31:44.291615 21254 leader_election.cc:290] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.293129 21302 raft_consensus.cc:697] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Becoming Leader. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:44.293793 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:44.294745 21162 raft_consensus.cc:2393] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0315789c9afb4bb69a62f2e94c2180ac in current term 1: Already voted for candidate f9e77ebb794b47949570cbc9f3e55646 in this term.
I20260709 12:31:44.294188 21302 consensus_queue.cc:237] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.298492 20980 leader_election.cc:304] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac; no voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:44.296100 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea1880f9453549a5b74387f60aca8b3a" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:44.300446 21163 raft_consensus.cc:3060] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.301265 21347 raft_consensus.cc:3060] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.305337 21322 consensus_queue.cc:1048] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:44.307307 21163 raft_consensus.cc:2468] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 1.
I20260709 12:31:44.315244 21347 raft_consensus.cc:2749] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:44.328929 20913 catalog_manager.cc:5697] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1). New cstate: current_term: 1 leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:44.348665 21316 consensus_queue.cc:1048] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:44.365593 21300 consensus_queue.cc:1048] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:44.462839 21314 consensus_queue.cc:1048] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:44.475423 21289 consensus_queue.cc:1048] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:44.530660 21337 consensus_queue.cc:1048] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:44.551213 21337 consensus_queue.cc:1048] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:44.560247 21302 consensus_queue.cc:1048] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:44.571512 21087 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696"
dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
mode: GRACEFUL
new_leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
 from {username='slave'} at 127.0.0.1:50794
I20260709 12:31:44.572494 21087 raft_consensus.cc:606] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Received request to transfer leadership to TS 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:31:44.595229 21300 raft_consensus.cc:493] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:44.595819 21300 raft_consensus.cc:515] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.598239 21300 leader_election.cc:290] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.599653 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "400df23b40104970bb996c614dbf7657" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:44.602331 21347 consensus_queue.cc:1048] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:44.602437 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "400df23b40104970bb996c614dbf7657" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:44.603948 21055 leader_election.cc:304] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a
I20260709 12:31:44.605127 21300 raft_consensus.cc:2749] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:31:44.610461 21015 raft_consensus.cc:3060] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.620751 21163 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c9a271be4694580b7755df0975444ae"
dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
mode: GRACEFUL
new_leader_uuid: "f9e77ebb794b47949570cbc9f3e55646"
 from {username='slave'} at 127.0.0.1:57198
I20260709 12:31:44.624595 21163 raft_consensus.cc:606] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Received request to transfer leadership to TS f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:44.625366 21346 consensus_queue.cc:1048] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:44.626180 21347 raft_consensus.cc:493] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:44.626718 21347 raft_consensus.cc:515] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.629042 21347 leader_election.cc:290] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.636956 21088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:44.639911 21162 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c4f782fea1df4f63aa3da3e6e8a85bd2"
dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
mode: GRACEFUL
new_leader_uuid: "f9e77ebb794b47949570cbc9f3e55646"
 from {username='slave'} at 127.0.0.1:57198
I20260709 12:31:44.640854 21162 raft_consensus.cc:606] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Received request to transfer leadership to TS f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:44.642185 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:44.642916 21163 raft_consensus.cc:2468] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 1.
I20260709 12:31:44.644023 20980 leader_election.cc:304] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:44.649420 21347 raft_consensus.cc:2804] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:31:44.651364 21347 raft_consensus.cc:493] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Starting leader election (detected failure of leader f9e77ebb794b47949570cbc9f3e55646)
I20260709 12:31:44.651825 21347 raft_consensus.cc:3060] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:44.660951 21316 consensus_queue.cc:1048] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:44.683099 21322 consensus_queue.cc:1048] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:44.679034 21347 raft_consensus.cc:515] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.692848 21015 raft_consensus.cc:1240] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Rejecting Update request from peer f9e77ebb794b47949570cbc9f3e55646 for earlier term 1. Current term is 2. Ops: []
I20260709 12:31:44.697386 21088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:44.698369 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:44.699924 20980 leader_election.cc:304] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac; no voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:44.701249 21322 raft_consensus.cc:2749] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:31:44.701414 21316 consensus_queue.cc:1059] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:44.702337 21316 raft_consensus.cc:3055] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:44.702656 21316 raft_consensus.cc:740] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:31:44.703500 21316 consensus_queue.cc:260] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.705226 21316 raft_consensus.cc:3060] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:44.708901 21254 consensus_queue.cc:1048] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:44.709539 21015 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2b57143f3d8f47b7b11ac682e7327bd6"
dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
mode: GRACEFUL
new_leader_uuid: "f9e77ebb794b47949570cbc9f3e55646"
 from {username='slave'} at 127.0.0.1:37570
I20260709 12:31:44.710186 21015 raft_consensus.cc:606] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Received request to transfer leadership to TS f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:44.712096 21347 leader_election.cc:290] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 2 election: Requested vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.716871 21322 consensus_queue.cc:1048] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:44.723307 21014 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "400df23b40104970bb996c614dbf7657"
dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
mode: GRACEFUL
new_leader_uuid: "2a779b422c3c4bd59a5449144952c73a"
 from {username='slave'} at 127.0.0.1:37570
I20260709 12:31:44.723976 21014 raft_consensus.cc:606] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Received request to transfer leadership to TS 2a779b422c3c4bd59a5449144952c73a
I20260709 12:31:44.727268 21090 raft_consensus.cc:3060] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:31:44.746573 21347 raft_consensus.cc:993] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac: : Instructing follower 2a779b422c3c4bd59a5449144952c73a to start an election
I20260709 12:31:44.747288 21347 raft_consensus.cc:1081] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Signalling peer 2a779b422c3c4bd59a5449144952c73a to start an election
I20260709 12:31:44.749301 21162 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "400df23b40104970bb996c614dbf7657"
dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
 from {username='slave'} at 127.0.0.1:57186
I20260709 12:31:44.749989 21162 raft_consensus.cc:493] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:44.750393 21162 raft_consensus.cc:3060] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:44.767376 21162 raft_consensus.cc:515] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.772140 21162 leader_election.cc:290] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:44.773329 21015 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "400df23b40104970bb996c614dbf7657" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:44.777016 21015 raft_consensus.cc:3055] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:44.774708 21088 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "400df23b40104970bb996c614dbf7657" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:44.777438 21015 raft_consensus.cc:740] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:44.778328 21015 consensus_queue.cc:260] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.779699 21015 raft_consensus.cc:3060] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:44.786357 21347 consensus_queue.cc:1048] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:31:44.788154 21015 raft_consensus.cc:2468] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 2.
I20260709 12:31:44.791884 21129 leader_election.cc:304] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: f9e77ebb794b47949570cbc9f3e55646
I20260709 12:31:44.793217 21337 raft_consensus.cc:2804] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:44.793733 21337 raft_consensus.cc:697] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [term 2 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:44.794602 21337 consensus_queue.cc:237] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.819041 20912 catalog_manager.cc:5697] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 1 to 2, leader changed from 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1) to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 2 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:44.909359 21302 raft_consensus.cc:993] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac: : Instructing follower f9e77ebb794b47949570cbc9f3e55646 to start an election
I20260709 12:31:44.910063 21347 raft_consensus.cc:1081] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Signalling peer f9e77ebb794b47949570cbc9f3e55646 to start an election
I20260709 12:31:44.912317 21087 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2b57143f3d8f47b7b11ac682e7327bd6"
dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
 from {username='slave'} at 127.0.0.1:50780
I20260709 12:31:44.913134 21087 raft_consensus.cc:493] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:44.913671 21087 raft_consensus.cc:3060] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:44.918988 21300 raft_consensus.cc:993] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646: : Instructing follower 0315789c9afb4bb69a62f2e94c2180ac to start an election
I20260709 12:31:44.919653 21316 raft_consensus.cc:1081] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Signalling peer 0315789c9afb4bb69a62f2e94c2180ac to start an election
I20260709 12:31:44.921844 21015 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696"
dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
 from {username='slave'} at 127.0.0.1:56908
I20260709 12:31:44.922394 21015 raft_consensus.cc:493] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:44.922700 21015 raft_consensus.cc:3060] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:44.924994 21087 raft_consensus.cc:515] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.927856 21087 leader_election.cc:290] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.928287 21015 raft_consensus.cc:515] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.929538 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b57143f3d8f47b7b11ac682e7327bd6" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:44.930233 21014 raft_consensus.cc:3055] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:44.930424 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b57143f3d8f47b7b11ac682e7327bd6" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:44.930579 21014 raft_consensus.cc:740] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:44.931170 21163 raft_consensus.cc:3060] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:44.931865 21015 leader_election.cc:290] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 2 election: Requested vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:44.931783 21014 consensus_queue.cc:260] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.933385 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:44.933722 21014 raft_consensus.cc:3060] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:44.934140 21087 raft_consensus.cc:3055] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:44.934746 21087 raft_consensus.cc:740] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:31:44.935838 21087 consensus_queue.cc:260] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.937354 21087 raft_consensus.cc:3060] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:44.938289 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:44.939064 21162 raft_consensus.cc:3060] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:44.945339 21014 raft_consensus.cc:2468] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 2.
I20260709 12:31:44.945911 21087 raft_consensus.cc:2468] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 2.
I20260709 12:31:44.946969 21055 leader_election.cc:304] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:44.947407 20978 leader_election.cc:304] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:44.947907 21316 raft_consensus.cc:2804] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:44.948262 21347 raft_consensus.cc:2804] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:44.948495 21316 raft_consensus.cc:697] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [term 2 LEADER]: Becoming Leader. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:31:44.948864 21347 raft_consensus.cc:697] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 LEADER]: Becoming Leader. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:44.949560 21316 consensus_queue.cc:237] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.949731 21347 consensus_queue.cc:237] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:44.948683 21162 raft_consensus.cc:2468] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 2.
I20260709 12:31:44.957015 21163 raft_consensus.cc:2468] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 2.
I20260709 12:31:44.958333 20912 catalog_manager.cc:5697] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 reported cstate change: term changed from 1 to 2, leader changed from 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1) to f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "f9e77ebb794b47949570cbc9f3e55646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:44.962114 20911 catalog_manager.cc:5697] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac reported cstate change: term changed from 1 to 2, leader changed from f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2) to 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1). New cstate: current_term: 2 leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:45.049814 21289 raft_consensus.cc:993] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a: : Instructing follower f9e77ebb794b47949570cbc9f3e55646 to start an election
I20260709 12:31:45.050402 21289 raft_consensus.cc:1081] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Signalling peer f9e77ebb794b47949570cbc9f3e55646 to start an election
I20260709 12:31:45.052397 21090 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c4f782fea1df4f63aa3da3e6e8a85bd2"
dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
 from {username='slave'} at 127.0.0.1:50774
I20260709 12:31:45.053043 21090 raft_consensus.cc:493] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:45.053750 21090 raft_consensus.cc:3060] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:45.064404 21090 raft_consensus.cc:515] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:45.066754 21090 leader_election.cc:290] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:45.068100 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4f782fea1df4f63aa3da3e6e8a85bd2" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:45.068310 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4f782fea1df4f63aa3da3e6e8a85bd2" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:45.068809 21014 raft_consensus.cc:3060] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:45.069043 21163 raft_consensus.cc:3055] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:45.069478 21163 raft_consensus.cc:740] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:45.070289 21163 consensus_queue.cc:260] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:45.072006 21163 raft_consensus.cc:3060] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:45.079558 21014 raft_consensus.cc:2468] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 2.
I20260709 12:31:45.081934 21163 raft_consensus.cc:2468] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 2.
I20260709 12:31:45.082072 21055 leader_election.cc:304] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:45.083463 21316 raft_consensus.cc:2804] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:45.083931 21316 raft_consensus.cc:697] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [term 2 LEADER]: Becoming Leader. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:31:45.084786 21316 consensus_queue.cc:237] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:45.087424 21337 raft_consensus.cc:993] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a: : Instructing follower f9e77ebb794b47949570cbc9f3e55646 to start an election
I20260709 12:31:45.087976 21289 raft_consensus.cc:1081] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Signalling peer f9e77ebb794b47949570cbc9f3e55646 to start an election
I20260709 12:31:45.089931 21090 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2c9a271be4694580b7755df0975444ae"
dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
 from {username='slave'} at 127.0.0.1:50774
I20260709 12:31:45.090556 21090 raft_consensus.cc:493] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:45.090945 21090 raft_consensus.cc:3060] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:45.093040 20912 catalog_manager.cc:5697] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 reported cstate change: term changed from 1 to 2, leader changed from 2a779b422c3c4bd59a5449144952c73a (127.4.65.3) to f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "f9e77ebb794b47949570cbc9f3e55646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:45.102059 21090 raft_consensus.cc:515] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:45.105218 21090 leader_election.cc:290] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:45.105433 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9a271be4694580b7755df0975444ae" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:45.106194 21014 raft_consensus.cc:3060] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:45.106557 21163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c9a271be4694580b7755df0975444ae" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:45.107141 21163 raft_consensus.cc:3055] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:45.107445 21163 raft_consensus.cc:740] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:45.109064 21163 consensus_queue.cc:260] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:45.110280 21163 raft_consensus.cc:3060] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:45.118192 21014 raft_consensus.cc:2468] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 2.
I20260709 12:31:45.118948 21163 raft_consensus.cc:2468] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 2.
I20260709 12:31:45.119755 21055 leader_election.cc:304] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:45.120728 21316 raft_consensus.cc:2804] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:45.121282 21316 raft_consensus.cc:697] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [term 2 LEADER]: Becoming Leader. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:31:45.122095 21316 consensus_queue.cc:237] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:45.130671 20912 catalog_manager.cc:5697] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 reported cstate change: term changed from 1 to 2, leader changed from 2a779b422c3c4bd59a5449144952c73a (127.4.65.3) to f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "f9e77ebb794b47949570cbc9f3e55646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:45.189328 21014 raft_consensus.cc:1275] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:45.191087 21289 consensus_queue.cc:1048] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:45.207496 21090 raft_consensus.cc:3060] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:45.218364 21090 raft_consensus.cc:1275] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:45.220216 21289 consensus_queue.cc:1048] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:45.338703 21090 raft_consensus.cc:1275] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Refusing update from remote peer 0315789c9afb4bb69a62f2e94c2180ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:45.339986 21347 consensus_queue.cc:1048] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:31:45.359974 21163 raft_consensus.cc:1275] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Refusing update from remote peer 0315789c9afb4bb69a62f2e94c2180ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:45.361657 21376 consensus_queue.cc:1048] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:45.513510 21163 raft_consensus.cc:1275] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Refusing update from remote peer f9e77ebb794b47949570cbc9f3e55646: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:45.515128 21316 consensus_queue.cc:1048] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:45.529650 21015 raft_consensus.cc:1275] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Refusing update from remote peer f9e77ebb794b47949570cbc9f3e55646: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:45.533697 21316 consensus_queue.cc:1048] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:45.547495 21162 raft_consensus.cc:1275] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Refusing update from remote peer f9e77ebb794b47949570cbc9f3e55646: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:45.551481 21300 consensus_queue.cc:1048] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:31:45.576323 21015 raft_consensus.cc:1275] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Refusing update from remote peer f9e77ebb794b47949570cbc9f3e55646: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:45.577680 21163 raft_consensus.cc:1275] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Refusing update from remote peer f9e77ebb794b47949570cbc9f3e55646: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:45.582015 21316 consensus_queue.cc:1048] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:45.582957 21378 consensus_queue.cc:1048] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:45.605333 21014 raft_consensus.cc:1275] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Refusing update from remote peer f9e77ebb794b47949570cbc9f3e55646: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:45.606973 21378 consensus_queue.cc:1048] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:46.165278 21384 raft_consensus.cc:493] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f9e77ebb794b47949570cbc9f3e55646)
I20260709 12:31:46.165908 21384 raft_consensus.cc:515] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:46.168510 21384 leader_election.cc:290] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:46.169207 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:46.169219 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac" is_pre_election: true
I20260709 12:31:46.170020 21090 raft_consensus.cc:2468] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 2.
I20260709 12:31:46.170528 21014 raft_consensus.cc:2393] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2a779b422c3c4bd59a5449144952c73a in current term 2: Already voted for candidate 0315789c9afb4bb69a62f2e94c2180ac in this term.
I20260709 12:31:46.171309 21127 leader_election.cc:304] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:46.172544 21384 raft_consensus.cc:2804] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:31:46.172983 21384 raft_consensus.cc:493] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting leader election (detected failure of leader f9e77ebb794b47949570cbc9f3e55646)
I20260709 12:31:46.173307 21384 raft_consensus.cc:3060] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:46.179750 21384 raft_consensus.cc:515] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:46.183813 21384 leader_election.cc:290] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:46.185505 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:46.186292 21014 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:46.186993 21014 raft_consensus.cc:2393] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2a779b422c3c4bd59a5449144952c73a in current term 2: Already voted for candidate 0315789c9afb4bb69a62f2e94c2180ac in this term.
I20260709 12:31:46.194479 21090 raft_consensus.cc:2468] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 2.
I20260709 12:31:46.196228 21127 leader_election.cc:304] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:31:46.197146 21384 raft_consensus.cc:2804] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:46.197839 21384 raft_consensus.cc:697] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 2 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:46.198704 21384 consensus_queue.cc:237] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:46.207814 20912 catalog_manager.cc:5697] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 1 to 2, leader changed from f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2) to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 2 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:46.504217 21302 raft_consensus.cc:493] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:31:46.504987 21302 raft_consensus.cc:515] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:46.507156 21302 leader_election.cc:290] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:46.508081 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:31:46.508275 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:31:46.508872 21090 raft_consensus.cc:2468] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 2.
I20260709 12:31:46.509941 20978 leader_election.cc:304] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 2a779b422c3c4bd59a5449144952c73a
I20260709 12:31:46.510613 21302 raft_consensus.cc:2804] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:31:46.510962 21302 raft_consensus.cc:493] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:31:46.511195 21302 raft_consensus.cc:3060] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:46.516929 21302 raft_consensus.cc:515] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:46.518735 21302 leader_election.cc:290] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 3 election: Requested vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:31:46.519757 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:46.519865 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "0315789c9afb4bb69a62f2e94c2180ac" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:31:46.520628 21090 raft_consensus.cc:3060] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:46.527592 21090 raft_consensus.cc:2468] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0315789c9afb4bb69a62f2e94c2180ac in term 3.
I20260709 12:31:46.528724 20978 leader_election.cc:304] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646; no voters: 2a779b422c3c4bd59a5449144952c73a
I20260709 12:31:46.529670 21302 raft_consensus.cc:2804] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:31:46.530207 21302 raft_consensus.cc:697] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 3 LEADER]: Becoming Leader. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:46.531250 21302 consensus_queue.cc:237] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:46.539150 20912 catalog_manager.cc:5697] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac reported cstate change: term changed from 2 to 3, leader changed from 2a779b422c3c4bd59a5449144952c73a (127.4.65.3) to 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1). New cstate: current_term: 3 leader_uuid: "0315789c9afb4bb69a62f2e94c2180ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:31:46.632628 21014 raft_consensus.cc:1240] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 3 LEADER]: Rejecting Update request from peer 2a779b422c3c4bd59a5449144952c73a for earlier term 2. Current term is 3. Ops: []
I20260709 12:31:46.633831 21384 consensus_queue.cc:1059] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:31:46.634411 21384 consensus_queue.cc:1243] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Peer 0315789c9afb4bb69a62f2e94c2180ac log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260709 12:31:46.634819 21385 raft_consensus.cc:3055] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:31:46.635144 21385 raft_consensus.cc:740] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:46.635844 21385 consensus_queue.cc:260] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:46.636790 21385 raft_consensus.cc:3060] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:46.916122 21162 raft_consensus.cc:1275] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Refusing update from remote peer 0315789c9afb4bb69a62f2e94c2180ac: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:31:46.916595 21162 pending_rounds.cc:85] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a: Aborting all ops after (but not including) 1
I20260709 12:31:46.916838 21162 pending_rounds.cc:107] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:31:46.917100 21162 raft_consensus.cc:2889] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:31:46.918864 21302 consensus_queue.cc:1048] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:46.931030 21090 raft_consensus.cc:1275] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Refusing update from remote peer 0315789c9afb4bb69a62f2e94c2180ac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:31:46.932567 21302 consensus_queue.cc:1048] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:31:47.729490 20912 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49232:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 12:31:47.751933 20912 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:49232:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 12:31:47.872247  4356 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
W20260709 12:31:47.912879 20980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:33705, user_credentials={real_user=slave}} blocked reactor thread for 47745us
I20260709 12:31:47.922264 21013 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3bc6713b68bd4ccfa12c5a02522a1b9f"
dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
mode: GRACEFUL
new_leader_uuid: "2a779b422c3c4bd59a5449144952c73a"
 from {username='slave'} at 127.0.0.1:37584
I20260709 12:31:47.949620 21437 tablet.cc:1724] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:47.960935 21441 tablet.cc:1724] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:47.964999 21444 tablet.cc:1724] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:47.983235 21440 tablet.cc:1724] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.049705 21444 tablet.cc:1724] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.045544 21441 tablet.cc:1724] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.068421 21453 tablet.cc:1724] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.073361 21454 tablet.cc:1724] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.079510 21461 tablet.cc:1724] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.083947 21452 tablet.cc:1724] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.087729 21456 tablet.cc:1724] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.094646 21459 tablet.cc:1724] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:31:48.103930 21041 tablet_mm_ops.cc:139] Rowset compaction is disabled (check --enable_rowset_compaction and disable_compaction in extra_config for tablet:3bc6713b68bd4ccfa12c5a02522a1b9f)
W20260709 12:31:48.108103 21055 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:33705, user_credentials={real_user=slave}} blocked reactor thread for 119997us
W20260709 12:31:48.109930 21041 tablet_mm_ops.cc:304] Major delta compaction is disabled (check --enable_major_delta_compaction and disable_compaction in extra_config for tablet:3bc6713b68bd4ccfa12c5a02522a1b9f)
W20260709 12:31:48.110864 21041 tablet_mm_ops.cc:220] Minor delta compaction is disabled (check --enable_minor_delta_compaction and disable_compaction in extra_config for tablet:3bc6713b68bd4ccfa12c5a02522a1b9f)
I20260709 12:31:48.099623 21460 tablet.cc:1724] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.135681 21013 raft_consensus.cc:606] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Received request to transfer leadership to TS 2a779b422c3c4bd59a5449144952c73a
I20260709 12:31:48.141286 21013 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696"
dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
mode: GRACEFUL
new_leader_uuid: "2a779b422c3c4bd59a5449144952c73a"
 from {username='slave'} at 127.0.0.1:37584
I20260709 12:31:48.180207 21444 tablet.cc:1724] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.179766 21461 tablet.cc:1724] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.159494 21441 tablet.cc:1724] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.189277 21454 tablet.cc:1724] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.202749 21440 tablet.cc:1724] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.210397 21453 tablet.cc:1724] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.222769 21013 raft_consensus.cc:606] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 LEADER]: Received request to transfer leadership to TS 2a779b422c3c4bd59a5449144952c73a
I20260709 12:31:48.224436 21455 tablet.cc:1724] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.230705 21460 tablet.cc:1724] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.236024 21441 tablet.cc:1724] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.237473  4356 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 2
I20260709 12:31:48.238215  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:48.237602 21461 tablet.cc:1724] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.245476 21459 tablet.cc:1724] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.248695 21437 tablet.cc:1724] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.282143 21455 tablet.cc:1724] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.282279 21440 tablet.cc:1724] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:31:48.304858 21089 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "29dfb2ec48164bb0a6e78ff68f55d2d9"
dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
mode: GRACEFUL
new_leader_uuid: "2a779b422c3c4bd59a5449144952c73a"
 from {username='slave'} at 127.0.0.1:47108
I20260709 12:31:48.305588 21089 raft_consensus.cc:606] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Received request to transfer leadership to TS 2a779b422c3c4bd59a5449144952c73a
I20260709 12:31:48.309361 21013 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6aa337c9b7844a03a308ba472bb3bdfd"
dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
mode: GRACEFUL
new_leader_uuid: "2a779b422c3c4bd59a5449144952c73a"
 from {username='slave'} at 127.0.0.1:37584
I20260709 12:31:48.309968 21013 raft_consensus.cc:606] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Received request to transfer leadership to TS 2a779b422c3c4bd59a5449144952c73a
I20260709 12:31:48.312525  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:31:48.313212  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:48.314528 20912 catalog_manager.cc:6242] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc: soft_deleted_table [id=0a49ce93813f489ca1dfb418317ce2cb] alter complete (version 1)
I20260709 12:31:48.331564 21432 raft_consensus.cc:993] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac: : Instructing follower 2a779b422c3c4bd59a5449144952c73a to start an election
I20260709 12:31:48.332608 21431 raft_consensus.cc:1081] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Signalling peer 2a779b422c3c4bd59a5449144952c73a to start an election
I20260709 12:31:48.334436 21162 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6aa337c9b7844a03a308ba472bb3bdfd"
dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
 from {username='slave'} at 127.0.0.1:57186
I20260709 12:31:48.334970 21162 raft_consensus.cc:493] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:48.335242 21162 raft_consensus.cc:3060] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:48.340561 21162 raft_consensus.cc:515] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:48.342661 21162 leader_election.cc:290] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:48.343847 21089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa337c9b7844a03a308ba472bb3bdfd" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:48.343963 21013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa337c9b7844a03a308ba472bb3bdfd" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:48.344599 21089 raft_consensus.cc:3060] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:48.344861 21013 raft_consensus.cc:3055] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:48.345242 21013 raft_consensus.cc:740] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:48.345944 21013 consensus_queue.cc:260] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:48.347004 21013 raft_consensus.cc:3060] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:48.352593 21089 raft_consensus.cc:2468] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 2.
I20260709 12:31:48.353749 21013 raft_consensus.cc:2468] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 2.
I20260709 12:31:48.354115 21127 leader_election.cc:304] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:48.355043 21468 raft_consensus.cc:2804] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:48.355479 21468 raft_consensus.cc:697] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [term 2 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:48.356428 21468 consensus_queue.cc:237] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:48.360177 21432 raft_consensus.cc:993] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac: : Instructing follower 2a779b422c3c4bd59a5449144952c73a to start an election
I20260709 12:31:48.360754 21431 raft_consensus.cc:1081] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Signalling peer 2a779b422c3c4bd59a5449144952c73a to start an election
I20260709 12:31:48.362555 21162 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3bc6713b68bd4ccfa12c5a02522a1b9f"
dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
 from {username='slave'} at 127.0.0.1:57186
I20260709 12:31:48.363005 21162 raft_consensus.cc:493] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:48.363338 21162 raft_consensus.cc:3060] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:48.366241 20912 catalog_manager.cc:5697] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 1 to 2, leader changed from 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1) to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 2 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:48.370767 21162 raft_consensus.cc:515] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:48.373080 21162 leader_election.cc:290] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:48.374186 21089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bc6713b68bd4ccfa12c5a02522a1b9f" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:48.374568 21013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bc6713b68bd4ccfa12c5a02522a1b9f" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:48.374919 21089 raft_consensus.cc:3060] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:48.375233 21013 raft_consensus.cc:3055] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:48.375628 21013 raft_consensus.cc:740] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:48.376485 21013 consensus_queue.cc:260] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:48.377508 21013 raft_consensus.cc:3060] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:48.380861 21089 raft_consensus.cc:2468] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 2.
I20260709 12:31:48.382006 21127 leader_election.cc:304] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:48.382977 21468 raft_consensus.cc:2804] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:48.383407 21013 raft_consensus.cc:2468] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 2.
I20260709 12:31:48.383580 21468 raft_consensus.cc:697] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [term 2 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:48.384857 21468 consensus_queue.cc:237] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:48.396288 20912 catalog_manager.cc:5697] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 1 to 2, leader changed from 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1) to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 2 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:48.524853 21431 raft_consensus.cc:993] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac: : Instructing follower 2a779b422c3c4bd59a5449144952c73a to start an election
I20260709 12:31:48.525489 21432 raft_consensus.cc:1081] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 LEADER]: Signalling peer 2a779b422c3c4bd59a5449144952c73a to start an election
I20260709 12:31:48.527344 21162 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696"
dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
 from {username='slave'} at 127.0.0.1:57186
I20260709 12:31:48.528052 21162 raft_consensus.cc:493] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:48.528510 21162 raft_consensus.cc:3060] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:48.535127 21162 raft_consensus.cc:515] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:48.536865 21162 leader_election.cc:290] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:48.537612 21013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:48.538307 21086 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d8d2fe503cf41af8b8952a9080ed696" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:48.538705 21013 raft_consensus.cc:3055] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:31:48.538935 21086 raft_consensus.cc:3060] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:48.539150 21013 raft_consensus.cc:740] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0315789c9afb4bb69a62f2e94c2180ac, State: Running, Role: LEADER
I20260709 12:31:48.540866 21386 raft_consensus.cc:993] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646: : Instructing follower 2a779b422c3c4bd59a5449144952c73a to start an election
I20260709 12:31:48.540611 21013 consensus_queue.cc:260] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:48.541525 21465 raft_consensus.cc:1081] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Signalling peer 2a779b422c3c4bd59a5449144952c73a to start an election
I20260709 12:31:48.541774 21013 raft_consensus.cc:3060] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:31:48.543272 21162 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "29dfb2ec48164bb0a6e78ff68f55d2d9"
dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
 from {username='slave'} at 127.0.0.1:57182
I20260709 12:31:48.543856 21162 raft_consensus.cc:493] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:31:48.544202 21162 raft_consensus.cc:3060] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:48.547835 21086 raft_consensus.cc:2468] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 3.
I20260709 12:31:48.548767 21089 raft_consensus.cc:1240] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Rejecting Update request from peer 0315789c9afb4bb69a62f2e94c2180ac for earlier term 2. Current term is 3. Ops: []
I20260709 12:31:48.549393 21013 raft_consensus.cc:2468] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 3.
I20260709 12:31:48.549537 21127 leader_election.cc:304] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:31:48.550716 21468 raft_consensus.cc:2804] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:31:48.551244 21468 raft_consensus.cc:697] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 3 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:48.552193 21468 consensus_queue.cc:237] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:48.552906 21162 raft_consensus.cc:515] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:48.555071 21162 leader_election.cc:290] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:31:48.555985 21013 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dfb2ec48164bb0a6e78ff68f55d2d9" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0315789c9afb4bb69a62f2e94c2180ac"
I20260709 12:31:48.556751 21013 raft_consensus.cc:3060] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:48.558300 21089 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29dfb2ec48164bb0a6e78ff68f55d2d9" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:31:48.558948 21089 raft_consensus.cc:3055] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:31:48.559259 21089 raft_consensus.cc:740] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:31:48.563589 20912 catalog_manager.cc:5697] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 2 to 3, leader changed from 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1) to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 3 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:48.560441 21089 consensus_queue.cc:260] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:48.565775 21013 raft_consensus.cc:2468] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 2.
I20260709 12:31:48.565956 21089 raft_consensus.cc:3060] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:31:48.567157 21129 leader_election.cc:304] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a; no voters: 
I20260709 12:31:48.568082 21468 raft_consensus.cc:2804] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:31:48.568799 21468 raft_consensus.cc:697] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [term 2 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:31:48.569965 21468 consensus_queue.cc:237] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:31:48.574455 21089 raft_consensus.cc:2468] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 2.
I20260709 12:31:48.581308 20912 catalog_manager.cc:5697] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 1 to 2, leader changed from f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2) to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 2 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:31:48.899446 21089 raft_consensus.cc:1275] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:48.900962 21468 consensus_queue.cc:1048] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:48.913082 21013 raft_consensus.cc:1275] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:48.915482 21468 consensus_queue.cc:1048] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:48.938723 21089 raft_consensus.cc:1275] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:31:48.939949 21468 consensus_queue.cc:1048] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:31:48.948902 21013 raft_consensus.cc:1275] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 3 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:31:48.951042 21468 consensus_queue.cc:1048] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:31:48.970223 21013 raft_consensus.cc:1275] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:48.971467 21468 consensus_queue.cc:1048] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:48.982574 21089 raft_consensus.cc:1275] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:31:48.984647 21468 consensus_queue.cc:1048] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:31:48.995450 21089 raft_consensus.cc:1275] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:31:48.996197 21013 raft_consensus.cc:1275] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:31:48.997179 21468 consensus_queue.cc:1048] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:31:48.998131 21477 consensus_queue.cc:1048] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:31:49.314531  4356 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:31:49.319751  4356 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:31:49.320233  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:49.325065  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:31:49.325522  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:50.326941  4356 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:31:50.333424  4356 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:31:50.333936  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:50.338661  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:31:50.339124  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:51.340204  4356 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:31:51.345363  4356 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:31:51.345820  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:51.350953  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:31:51.351619  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:52.353034  4356 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:31:52.358263  4356 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:31:52.358747  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:52.363627  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:31:52.364022  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:53.365170  4356 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:31:53.370414  4356 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:31:53.370843  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:53.375834  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:31:53.376233  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:54.377832  4356 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:31:54.384272  4356 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:31:54.384860  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:54.390108  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:31:54.390559  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:55.392191  4356 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:31:55.398578  4356 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:31:55.399076  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:55.404516  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:31:55.404958  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:31:56.643771 21127 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.2:37027, user_credentials={real_user=slave}} blocked reactor thread for 74965.6us
I20260709 12:31:56.917523  4356 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:31:56.923609  4356 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:31:56.924220  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:56.929185  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:31:56.929708  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:31:57.730962 21036 debug-util.cc:398] Leaking SignalData structure 0x7b0800226fc0 after lost signal to thread 4359
W20260709 12:31:57.732666 21036 debug-util.cc:398] Leaking SignalData structure 0x7b080010a700 after lost signal to thread 20947
W20260709 12:31:57.733938 21036 debug-util.cc:398] Leaking SignalData structure 0x7b08002c8420 after lost signal to thread 21039
W20260709 12:31:57.734987 21036 debug-util.cc:398] Leaking SignalData structure 0x7b0800416840 after lost signal to thread 21114
W20260709 12:31:57.736027 21036 debug-util.cc:398] Leaking SignalData structure 0x7b08003d1c00 after lost signal to thread 21188
I20260709 12:31:57.930925  4356 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:31:57.936766  4356 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:31:57.937245  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:31:57.941992  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:31:57.942416  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:31:59.021934  4356 tablet_server.cc:179] TabletServer@127.4.65.1:0 shutting down...
W20260709 12:31:59.055411 21129 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 7 similar messages]
W20260709 12:31:59.072024 21129 consensus_peers.cc:597] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:59.085282  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:31:59.086864  4356 tablet_replica.cc:333] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.088521  4356 raft_consensus.cc:2243] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:31:59.089877  4356 raft_consensus.cc:2272] T 4a62f5d3a3bd438a896c90b6994dfe46 P 0315789c9afb4bb69a62f2e94c2180ac [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.093353  4356 tablet_replica.cc:333] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.094195  4356 raft_consensus.cc:2243] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:59.095963  4356 raft_consensus.cc:2272] T ea1880f9453549a5b74387f60aca8b3a P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.101241  4356 tablet_replica.cc:333] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.102401  4356 raft_consensus.cc:2243] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:59.103116  4356 raft_consensus.cc:2272] T 9d8d2fe503cf41af8b8952a9080ed696 P 0315789c9afb4bb69a62f2e94c2180ac [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.106819  4356 tablet_replica.cc:333] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.107667  4356 raft_consensus.cc:2243] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:59.108489  4356 raft_consensus.cc:2272] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.111833  4356 tablet_replica.cc:333] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.112656  4356 raft_consensus.cc:2243] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:31:59.113663  4356 raft_consensus.cc:2272] T 4255e87ce188451dab77427dc10cceb5 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.117942  4356 tablet_replica.cc:333] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.119136  4356 raft_consensus.cc:2243] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:59.119907  4356 raft_consensus.cc:2272] T c71f4db910844fd28c4a1679dd1477fe P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:59.121902 21055 consensus_peers.cc:597] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:31:59.124120  4356 tablet_replica.cc:333] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.124975  4356 raft_consensus.cc:2243] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:59.126209  4356 raft_consensus.cc:2272] T 24bbbbdce7934410b6e080a04ab521f7 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.129460  4356 tablet_replica.cc:333] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.130218  4356 raft_consensus.cc:2243] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:59.131345  4356 raft_consensus.cc:2272] T e082efdfaedc435c9b2265aba8e5d211 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.135385  4356 tablet_replica.cc:333] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.136114  4356 raft_consensus.cc:2243] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:31:59.137256  4356 raft_consensus.cc:2272] T 9d54c78552d24a77a8b1a06013d70728 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.140179  4356 tablet_replica.cc:333] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.140872  4356 raft_consensus.cc:2243] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:59.141430  4356 raft_consensus.cc:2272] T d1aff18dd4734eada226c23cc72e68df P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.144613  4356 tablet_replica.cc:333] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.145246  4356 raft_consensus.cc:2243] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:59.145885  4356 raft_consensus.cc:2272] T 400df23b40104970bb996c614dbf7657 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.148639  4356 tablet_replica.cc:333] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.149257  4356 raft_consensus.cc:2243] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:59.149821  4356 raft_consensus.cc:2272] T 1b21dacbef9943b4bb80a2c552abf0f6 P 0315789c9afb4bb69a62f2e94c2180ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.152683  4356 tablet_replica.cc:333] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.153419  4356 raft_consensus.cc:2243] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:59.153954  4356 raft_consensus.cc:2272] T 6aa337c9b7844a03a308ba472bb3bdfd P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.157452  4356 tablet_replica.cc:333] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.158325  4356 raft_consensus.cc:2243] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:59.159126  4356 raft_consensus.cc:2272] T 2c9a271be4694580b7755df0975444ae P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.162744  4356 tablet_replica.cc:333] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.163494  4356 raft_consensus.cc:2243] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:59.164130  4356 raft_consensus.cc:2272] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.167640  4356 tablet_replica.cc:333] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.168432  4356 raft_consensus.cc:2243] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:59.169098  4356 raft_consensus.cc:2272] T ce50888244a14856b32e824b55bf67c6 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.172626  4356 tablet_replica.cc:333] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.173310  4356 raft_consensus.cc:2243] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:59.174055  4356 raft_consensus.cc:2272] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:31:59.177623  4356 tablet_replica.cc:333] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac: stopping tablet replica
I20260709 12:31:59.178457  4356 raft_consensus.cc:2243] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:31:59.179541  4356 raft_consensus.cc:2272] T 2b57143f3d8f47b7b11ac682e7327bd6 P 0315789c9afb4bb69a62f2e94c2180ac [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:31:59.205250 21055 consensus_peers.cc:597] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:59.206264 21055 consensus_peers.cc:597] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:59.212044 21055 consensus_peers.cc:597] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:59.283739 21055 consensus_peers.cc:597] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:59.327101 21055 consensus_peers.cc:597] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:59.384115 21129 consensus_peers.cc:597] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:59.511159 21129 consensus_peers.cc:597] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:59.517243 21129 consensus_peers.cc:597] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:59.549440 21129 consensus_peers.cc:597] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:31:59.562748 21129 consensus_peers.cc:597] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:00.141976 21518 raft_consensus.cc:493] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.142946 21518 raft_consensus.cc:515] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.147616 21518 leader_election.cc:290] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:32:00.150247 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
W20260709 12:32:00.154302 21055 leader_election.cc:336] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.154865 21055 leader_election.cc:304] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a
I20260709 12:32:00.155673 21518 raft_consensus.cc:2749] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:32:00.221110 21518 raft_consensus.cc:493] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.221776 21518 raft_consensus.cc:515] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.224080 21518 leader_election.cc:290] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
W20260709 12:32:00.229296 21055 leader_election.cc:336] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.229732 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d54c78552d24a77a8b1a06013d70728" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:32:00.231642 21055 leader_election.cc:304] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a
I20260709 12:32:00.232542 21518 raft_consensus.cc:2749] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:32:00.248260  4356 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 21036)
I20260709 12:32:00.273378 21521 raft_consensus.cc:493] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.274017 21521 raft_consensus.cc:515] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.276271 21521 leader_election.cc:290] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:32:00.277454 21518 raft_consensus.cc:493] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.277694 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4255e87ce188451dab77427dc10cceb5" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:32:00.278154 21518 raft_consensus.cc:515] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.281127 21518 leader_election.cc:290] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
W20260709 12:32:00.281844 21129 leader_election.cc:336] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.282347 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea1880f9453549a5b74387f60aca8b3a" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:32:00.283219 21129 leader_election.cc:304] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a; no voters: 0315789c9afb4bb69a62f2e94c2180ac, f9e77ebb794b47949570cbc9f3e55646
I20260709 12:32:00.284391 21521 raft_consensus.cc:2749] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:32:00.285362 21055 leader_election.cc:336] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.285902 21055 leader_election.cc:304] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a
I20260709 12:32:00.286963 21518 raft_consensus.cc:2749] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:32:00.291141 21518 raft_consensus.cc:493] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.291733 21518 raft_consensus.cc:515] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:32:00.293587 21518 leader_election.cc:290] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705), 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545)
I20260709 12:32:00.294380 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24bbbbdce7934410b6e080a04ab521f7" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
W20260709 12:32:00.297792 21055 leader_election.cc:336] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.298243 21055 leader_election.cc:304] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a
I20260709 12:32:00.298869 21518 raft_consensus.cc:2749] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:32:00.304901 21521 raft_consensus.cc:493] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.305706 21521 raft_consensus.cc:515] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.308209 21521 leader_election.cc:290] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:32:00.309386 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea1880f9453549a5b74387f60aca8b3a" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:32:00.310462 21087 raft_consensus.cc:2468] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 1.
I20260709 12:32:00.311902 21127 leader_election.cc:304] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
W20260709 12:32:00.312883 21129 leader_election.cc:336] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.313107 21521 raft_consensus.cc:2804] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:32:00.313577 21521 raft_consensus.cc:493] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.313977 21521 raft_consensus.cc:3060] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:00.323470 21521 raft_consensus.cc:515] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.325731 21521 leader_election.cc:290] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:32:00.327190 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea1880f9453549a5b74387f60aca8b3a" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:32:00.328045 21087 raft_consensus.cc:3060] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:00.329731 21521 raft_consensus.cc:493] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
W20260709 12:32:00.330833 21129 leader_election.cc:336] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.330626 21521 raft_consensus.cc:515] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.334507 21521 leader_election.cc:290] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:32:00.335520 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:32:00.336767 21090 raft_consensus.cc:2468] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 3.
W20260709 12:32:00.338976 21129 leader_election.cc:336] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.340924 21127 leader_election.cc:304] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:32:00.341789 21521 raft_consensus.cc:2804] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:32:00.342299 21521 raft_consensus.cc:493] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.342756 21521 raft_consensus.cc:3060] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:32:00.343341 21087 raft_consensus.cc:2468] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 2.
I20260709 12:32:00.344866 21520 raft_consensus.cc:493] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.345554 21520 raft_consensus.cc:515] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.347708 21127 leader_election.cc:304] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:32:00.348042 21520 leader_election.cc:290] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:32:00.348948 21511 raft_consensus.cc:2804] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:00.349560 21511 raft_consensus.cc:697] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 2 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:32:00.350884 21511 consensus_queue.cc:237] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.351861 21518 raft_consensus.cc:493] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.352851 21518 raft_consensus.cc:515] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
W20260709 12:32:00.353348 21129 leader_election.cc:336] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.355928 21518 leader_election.cc:290] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:32:00.360306 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4255e87ce188451dab77427dc10cceb5" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
I20260709 12:32:00.361102 21162 raft_consensus.cc:2468] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 1.
I20260709 12:32:00.362251 21521 raft_consensus.cc:515] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.363648 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d54c78552d24a77a8b1a06013d70728" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:32:00.363994 21055 leader_election.cc:304] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:32:00.364408 21087 raft_consensus.cc:2468] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 2.
I20260709 12:32:00.365700 21512 raft_consensus.cc:2804] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:32:00.366082 21512 raft_consensus.cc:493] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.366441 21512 raft_consensus.cc:3060] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:00.367098 21127 leader_election.cc:304] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:32:00.370537 21521 leader_election.cc:290] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 4 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:32:00.371498 21521 raft_consensus.cc:2804] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:32:00.371109 20911 catalog_manager.cc:5697] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 1 to 2, leader changed from 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1) to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 2 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:32:00.371999 21521 raft_consensus.cc:493] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.372763 21521 raft_consensus.cc:3060] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:32:00.376639 21055 leader_election.cc:336] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
W20260709 12:32:00.377985 21129 leader_election.cc:336] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.381505 21087 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a62f5d3a3bd438a896c90b6994dfe46" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:32:00.382121 21087 raft_consensus.cc:3060] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:32:00.380113 21512 raft_consensus.cc:515] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.382200 21521 raft_consensus.cc:515] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.385178 21512 leader_election.cc:290] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:32:00.386186 21521 leader_election.cc:290] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:32:00.389214 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4255e87ce188451dab77427dc10cceb5" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a"
I20260709 12:32:00.389906 21162 raft_consensus.cc:3060] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:32:00.390412 21129 leader_election.cc:336] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.390810 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9d54c78552d24a77a8b1a06013d70728" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:32:00.391464 21090 raft_consensus.cc:3060] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:32:00.400120 21162 raft_consensus.cc:2468] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9e77ebb794b47949570cbc9f3e55646 in term 2.
I20260709 12:32:00.400625 21087 raft_consensus.cc:2468] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 4.
I20260709 12:32:00.403110 21127 leader_election.cc:304] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:32:00.403053 21090 raft_consensus.cc:2468] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 3.
I20260709 12:32:00.404469 21511 raft_consensus.cc:493] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.404753 21521 raft_consensus.cc:2804] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:32:00.405128 21511 raft_consensus.cc:515] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
W20260709 12:32:00.406291 21055 leader_election.cc:336] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.405673 21521 raft_consensus.cc:697] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 4 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:32:00.408980 21521 consensus_queue.cc:237] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.411979 21511 leader_election.cc:290] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545)
I20260709 12:32:00.407173 21055 leader_election.cc:304] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:32:00.413414 21512 raft_consensus.cc:2804] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:00.413904 21512 raft_consensus.cc:697] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 2 LEADER]: Becoming Leader. State: Replica: f9e77ebb794b47949570cbc9f3e55646, State: Running, Role: LEADER
I20260709 12:32:00.414947 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24bbbbdce7934410b6e080a04ab521f7" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:32:00.414952 21512 consensus_queue.cc:237] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.415803 21090 raft_consensus.cc:2468] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 2.
I20260709 12:32:00.416113 21127 leader_election.cc:304] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
W20260709 12:32:00.417512 21129 leader_election.cc:336] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.418457 21511 raft_consensus.cc:2804] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:32:00.419196 21127 leader_election.cc:304] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:32:00.421100 21520 raft_consensus.cc:2804] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:32:00.421530 21520 raft_consensus.cc:493] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.421914 21520 raft_consensus.cc:3060] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:32:00.429695 21511 raft_consensus.cc:697] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 3 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:32:00.431031 21511 consensus_queue.cc:237] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.440310 20911 catalog_manager.cc:5697] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 reported cstate change: term changed from 1 to 2, leader changed from 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1) to f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "f9e77ebb794b47949570cbc9f3e55646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:00.443339 21520 raft_consensus.cc:515] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:32:00.446527 21520 leader_election.cc:290] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 election: Requested vote from peers f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027), 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545)
I20260709 12:32:00.447314 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24bbbbdce7934410b6e080a04ab521f7" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:32:00.448153 21090 raft_consensus.cc:3060] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:32:00.458515 21090 raft_consensus.cc:2468] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 3.
I20260709 12:32:00.461889 21127 leader_election.cc:304] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:32:00.462837 21511 raft_consensus.cc:2804] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:32:00.463361 21511 raft_consensus.cc:697] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 3 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
W20260709 12:32:00.463888 21129 leader_election.cc:336] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.464386 21511 consensus_queue.cc:237] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } }
I20260709 12:32:00.467432 20911 catalog_manager.cc:5697] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 3 to 4, leader changed from 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1) to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 4 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:32:00.469830 20911 catalog_manager.cc:5697] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 2 to 3, leader changed from 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1) to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 3 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
I20260709 12:32:00.495765 20911 catalog_manager.cc:5697] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 2 to 3, leader changed from 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1) to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 3 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:00.552479 21512 raft_consensus.cc:493] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.553290 21512 raft_consensus.cc:515] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.555721 21512 leader_election.cc:290] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), 2a779b422c3c4bd59a5449144952c73a (127.4.65.3:33705)
I20260709 12:32:00.557879 21162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e082efdfaedc435c9b2265aba8e5d211" candidate_uuid: "f9e77ebb794b47949570cbc9f3e55646" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "2a779b422c3c4bd59a5449144952c73a" is_pre_election: true
W20260709 12:32:00.561884 21055 leader_election.cc:336] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.562260 21055 leader_election.cc:304] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac, 2a779b422c3c4bd59a5449144952c73a
I20260709 12:32:00.562893 21512 raft_consensus.cc:2749] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:32:00.601418 21511 raft_consensus.cc:493] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
I20260709 12:32:00.602205 21511 raft_consensus.cc:515] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.604653 21511 leader_election.cc:290] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:32:00.605808 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e082efdfaedc435c9b2265aba8e5d211" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646" is_pre_election: true
I20260709 12:32:00.606534 21090 raft_consensus.cc:2468] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 2.
I20260709 12:32:00.607478 21127 leader_election.cc:304] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 
I20260709 12:32:00.608176 21511 raft_consensus.cc:2804] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:32:00.608623 21511 raft_consensus.cc:493] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0315789c9afb4bb69a62f2e94c2180ac)
W20260709 12:32:00.608855 21129 leader_election.cc:336] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.609139 21511 raft_consensus.cc:3060] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:32:00.616480 21511 raft_consensus.cc:515] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.618655 21511 leader_election.cc:290] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 election: Requested vote from peers 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545), f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027)
I20260709 12:32:00.620055 21090 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e082efdfaedc435c9b2265aba8e5d211" candidate_uuid: "2a779b422c3c4bd59a5449144952c73a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f9e77ebb794b47949570cbc9f3e55646"
I20260709 12:32:00.620702 21090 raft_consensus.cc:3060] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:32:00.623835 21129 leader_election.cc:336] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111)
I20260709 12:32:00.628180 21090 raft_consensus.cc:2468] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a779b422c3c4bd59a5449144952c73a in term 3.
I20260709 12:32:00.629662 21127 leader_election.cc:304] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a779b422c3c4bd59a5449144952c73a, f9e77ebb794b47949570cbc9f3e55646; no voters: 0315789c9afb4bb69a62f2e94c2180ac
I20260709 12:32:00.630460 21511 raft_consensus.cc:2804] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:32:00.631035 21511 raft_consensus.cc:697] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 3 LEADER]: Becoming Leader. State: Replica: 2a779b422c3c4bd59a5449144952c73a, State: Running, Role: LEADER
I20260709 12:32:00.632036 21511 consensus_queue.cc:237] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } }
I20260709 12:32:00.643790 20911 catalog_manager.cc:5697] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a reported cstate change: term changed from 2 to 3, leader changed from 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1) to 2a779b422c3c4bd59a5449144952c73a (127.4.65.3). New cstate: current_term: 3 leader_uuid: "2a779b422c3c4bd59a5449144952c73a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0315789c9afb4bb69a62f2e94c2180ac" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 42545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 } health_report { overall_health: HEALTHY } } }
W20260709 12:32:00.807054 21055 consensus_peers.cc:597] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:32:00.843755 21129 consensus_peers.cc:597] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:00.846372 21162 raft_consensus.cc:1275] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Refusing update from remote peer f9e77ebb794b47949570cbc9f3e55646: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:32:00.848207 21518 consensus_queue.cc:1048] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a779b422c3c4bd59a5449144952c73a" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 33705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:32:00.862716 21090 raft_consensus.cc:1275] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 4 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:32:00.866302 21511 consensus_queue.cc:1048] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 12:32:00.882249 21129 consensus_peers.cc:597] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:00.887914 21087 raft_consensus.cc:1275] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:32:00.892417 21520 consensus_queue.cc:1048] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:32:00.895852 21129 consensus_peers.cc:597] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:32:00.908535 21129 consensus_peers.cc:597] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:00.970744 21090 raft_consensus.cc:1275] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:32:00.972463 21520 consensus_queue.cc:1048] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:32:01.036625 21090 raft_consensus.cc:1275] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:32:01.037922 21521 consensus_queue.cc:1048] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:32:01.058763 21090 raft_consensus.cc:1275] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Refusing update from remote peer 2a779b422c3c4bd59a5449144952c73a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:32:01.060319 21511 consensus_queue.cc:1048] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9e77ebb794b47949570cbc9f3e55646" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 37027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:32:01.074795 21129 consensus_peers.cc:597] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:32:01.248921  4356 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 21036)
W20260709 12:32:01.583379 21055 consensus_peers.cc:597] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:32:01.665046 21055 consensus_peers.cc:597] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:32:01.667529 21129 consensus_peers.cc:597] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:32:01.714272 21055 consensus_peers.cc:597] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:32:01.740480 21055 consensus_peers.cc:597] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:32:01.775856  4356 tablet_server.cc:196] TabletServer@127.4.65.1:0 shutdown complete.
I20260709 12:32:01.824854  4356 tablet_server.cc:179] TabletServer@127.4.65.2:0 shutting down...
W20260709 12:32:01.833182 21127 consensus_peers.cc:597] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a -> Peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Couldn't send request to peer f9e77ebb794b47949570cbc9f3e55646. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:32:01.853782 21127 consensus_peers.cc:597] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a -> Peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Couldn't send request to peer f9e77ebb794b47949570cbc9f3e55646. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:01.865703  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:32:01.866837  4356 tablet_replica.cc:333] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.868101  4356 raft_consensus.cc:2243] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:01.868796  4356 raft_consensus.cc:2272] T 4a62f5d3a3bd438a896c90b6994dfe46 P f9e77ebb794b47949570cbc9f3e55646 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 12:32:01.869485 21127 consensus_peers.cc:597] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a -> Peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Couldn't send request to peer f9e77ebb794b47949570cbc9f3e55646. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:01.872648  4356 tablet_replica.cc:333] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.873798  4356 raft_consensus.cc:2243] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:01.874800  4356 raft_consensus.cc:2272] T c4f782fea1df4f63aa3da3e6e8a85bd2 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:32:01.877750 21129 consensus_peers.cc:597] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:32:01.879055  4356 tablet_replica.cc:333] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.879896  4356 raft_consensus.cc:2243] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:01.880486  4356 raft_consensus.cc:2272] T ea1880f9453549a5b74387f60aca8b3a P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.883395  4356 tablet_replica.cc:333] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.884037  4356 raft_consensus.cc:2243] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:01.884662  4356 raft_consensus.cc:2272] T 9d8d2fe503cf41af8b8952a9080ed696 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.887060  4356 tablet_replica.cc:333] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.887655  4356 raft_consensus.cc:2243] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:01.888666  4356 raft_consensus.cc:2272] T c71f4db910844fd28c4a1679dd1477fe P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.891155  4356 tablet_replica.cc:333] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.891830  4356 raft_consensus.cc:2243] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:01.892329  4356 raft_consensus.cc:2272] T 24bbbbdce7934410b6e080a04ab521f7 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.895126  4356 tablet_replica.cc:333] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.895810  4356 raft_consensus.cc:2243] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:01.896868  4356 raft_consensus.cc:2272] T 4255e87ce188451dab77427dc10cceb5 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.899219  4356 tablet_replica.cc:333] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.899925  4356 raft_consensus.cc:2243] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:01.900831  4356 raft_consensus.cc:2272] T 2c9a271be4694580b7755df0975444ae P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.903343  4356 tablet_replica.cc:333] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.904089  4356 raft_consensus.cc:2243] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:01.904791  4356 raft_consensus.cc:2272] T 6aa337c9b7844a03a308ba472bb3bdfd P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.907877  4356 tablet_replica.cc:333] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.908694  4356 raft_consensus.cc:2243] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:01.909366  4356 raft_consensus.cc:2272] T 400df23b40104970bb996c614dbf7657 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.912724  4356 tablet_replica.cc:333] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.913621  4356 raft_consensus.cc:2243] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:01.914237  4356 raft_consensus.cc:2272] T 1b21dacbef9943b4bb80a2c552abf0f6 P f9e77ebb794b47949570cbc9f3e55646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.917531  4356 tablet_replica.cc:333] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.918385  4356 raft_consensus.cc:2243] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:01.919592  4356 raft_consensus.cc:2272] T d1aff18dd4734eada226c23cc72e68df P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.923065  4356 tablet_replica.cc:333] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.923877  4356 raft_consensus.cc:2243] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:01.924549  4356 raft_consensus.cc:2272] T 9d54c78552d24a77a8b1a06013d70728 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.928417  4356 tablet_replica.cc:333] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.929229  4356 raft_consensus.cc:2243] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:01.929834  4356 raft_consensus.cc:2272] T e082efdfaedc435c9b2265aba8e5d211 P f9e77ebb794b47949570cbc9f3e55646 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.933611  4356 tablet_replica.cc:333] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.934381  4356 raft_consensus.cc:2243] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:01.935473  4356 raft_consensus.cc:2272] T ce50888244a14856b32e824b55bf67c6 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.938215  4356 tablet_replica.cc:333] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.938838  4356 raft_consensus.cc:2243] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:01.939421  4356 raft_consensus.cc:2272] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.942116  4356 tablet_replica.cc:333] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.942723  4356 raft_consensus.cc:2243] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:01.943233  4356 raft_consensus.cc:2272] T 3bc6713b68bd4ccfa12c5a02522a1b9f P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:01.945904  4356 tablet_replica.cc:333] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646: stopping tablet replica
I20260709 12:32:01.946455  4356 raft_consensus.cc:2243] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:01.947479  4356 raft_consensus.cc:2272] T 2b57143f3d8f47b7b11ac682e7327bd6 P f9e77ebb794b47949570cbc9f3e55646 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:32:01.977890 21129 consensus_peers.cc:597] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:32:02.005697 21127 consensus_peers.cc:597] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a -> Peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Couldn't send request to peer f9e77ebb794b47949570cbc9f3e55646. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:37027: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:02.018316  4356 tablet_server.cc:196] TabletServer@127.4.65.2:0 shutdown complete.
W20260709 12:32:02.041730 21127 consensus_peers.cc:597] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a -> Peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Couldn't send request to peer f9e77ebb794b47949570cbc9f3e55646. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:37027: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:02.063735  4356 tablet_server.cc:179] TabletServer@127.4.65.3:0 shutting down...
W20260709 12:32:02.074860 21129 consensus_peers.cc:597] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:32:02.083361 21127 consensus_peers.cc:597] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a -> Peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Couldn't send request to peer f9e77ebb794b47949570cbc9f3e55646. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:37027: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:32:02.089692 21129 consensus_peers.cc:597] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:32:02.095929 21127 consensus_peers.cc:597] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a -> Peer f9e77ebb794b47949570cbc9f3e55646 (127.4.65.2:37027): Couldn't send request to peer f9e77ebb794b47949570cbc9f3e55646. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.2:37027: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:32:02.097513 21129 consensus_peers.cc:597] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a -> Peer 0315789c9afb4bb69a62f2e94c2180ac (127.4.65.1:42545): Couldn't send request to peer 0315789c9afb4bb69a62f2e94c2180ac. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:42545: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:32:02.107419  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:32:02.108310  4356 tablet_replica.cc:333] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.109447  4356 raft_consensus.cc:2243] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:02.110224  4356 raft_consensus.cc:2272] T c4f782fea1df4f63aa3da3e6e8a85bd2 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.112721  4356 tablet_replica.cc:333] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.113319  4356 raft_consensus.cc:2243] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:02.114168  4356 raft_consensus.cc:2272] T 3bc6713b68bd4ccfa12c5a02522a1b9f P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.116346  4356 tablet_replica.cc:333] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.116952  4356 raft_consensus.cc:2243] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:02.117424  4356 raft_consensus.cc:2272] T 2b57143f3d8f47b7b11ac682e7327bd6 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.119565  4356 tablet_replica.cc:333] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.120088  4356 raft_consensus.cc:2243] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:32:02.121037  4356 raft_consensus.cc:2272] T 4a62f5d3a3bd438a896c90b6994dfe46 P 2a779b422c3c4bd59a5449144952c73a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.123198  4356 tablet_replica.cc:333] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.123725  4356 raft_consensus.cc:2243] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:02.124655  4356 raft_consensus.cc:2272] T ea1880f9453549a5b74387f60aca8b3a P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.126729  4356 tablet_replica.cc:333] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.127241  4356 raft_consensus.cc:2243] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:32:02.127981  4356 raft_consensus.cc:2272] T 9d8d2fe503cf41af8b8952a9080ed696 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.130215  4356 tablet_replica.cc:333] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.130720  4356 raft_consensus.cc:2243] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:02.131212  4356 raft_consensus.cc:2272] T 2c9a271be4694580b7755df0975444ae P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.133224  4356 tablet_replica.cc:333] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.133766  4356 raft_consensus.cc:2243] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:02.134549  4356 raft_consensus.cc:2272] T 6aa337c9b7844a03a308ba472bb3bdfd P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.136689  4356 tablet_replica.cc:333] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.137174  4356 raft_consensus.cc:2243] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:02.137945  4356 raft_consensus.cc:2272] T 400df23b40104970bb996c614dbf7657 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.140015  4356 tablet_replica.cc:333] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.140532  4356 raft_consensus.cc:2243] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:32:02.141340  4356 raft_consensus.cc:2272] T 1b21dacbef9943b4bb80a2c552abf0f6 P 2a779b422c3c4bd59a5449144952c73a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.143509  4356 tablet_replica.cc:333] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.143996  4356 raft_consensus.cc:2243] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:02.144451  4356 raft_consensus.cc:2272] T 4255e87ce188451dab77427dc10cceb5 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.146401  4356 tablet_replica.cc:333] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.147406  4356 raft_consensus.cc:2243] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:02.147840  4356 raft_consensus.cc:2272] T c71f4db910844fd28c4a1679dd1477fe P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.149982  4356 tablet_replica.cc:333] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.150511  4356 raft_consensus.cc:2243] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:32:02.151322  4356 raft_consensus.cc:2272] T 24bbbbdce7934410b6e080a04ab521f7 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.153484  4356 tablet_replica.cc:333] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.153972  4356 raft_consensus.cc:2243] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:32:02.154742  4356 raft_consensus.cc:2272] T e082efdfaedc435c9b2265aba8e5d211 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.156780  4356 tablet_replica.cc:333] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.157255  4356 raft_consensus.cc:2243] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:02.157676  4356 raft_consensus.cc:2272] T d1aff18dd4734eada226c23cc72e68df P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.159823  4356 tablet_replica.cc:333] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.160286  4356 raft_consensus.cc:2243] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:32:02.161085  4356 raft_consensus.cc:2272] T 9d54c78552d24a77a8b1a06013d70728 P 2a779b422c3c4bd59a5449144952c73a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.163075  4356 tablet_replica.cc:333] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.163516  4356 raft_consensus.cc:2243] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:02.163947  4356 raft_consensus.cc:2272] T ce50888244a14856b32e824b55bf67c6 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.166157  4356 tablet_replica.cc:333] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a: stopping tablet replica
I20260709 12:32:02.166704  4356 raft_consensus.cc:2243] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:02.167603  4356 raft_consensus.cc:2272] T 29dfb2ec48164bb0a6e78ff68f55d2d9 P 2a779b422c3c4bd59a5449144952c73a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.212544  4356 tablet_server.cc:196] TabletServer@127.4.65.3:0 shutdown complete.
I20260709 12:32:02.257560  4356 master.cc:562] Master@127.4.65.62:34691 shutting down...
I20260709 12:32:02.288151  4356 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:32:02.288831  4356 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:02.289131  4356 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4bf755ee6b38462c892f8dffc22963fc: stopping tablet replica
I20260709 12:32:02.310752  4356 master.cc:584] Master@127.4.65.62:34691 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0 (24636 ms)
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1
I20260709 12:32:02.370889  4356 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.4.65.62:37877
I20260709 12:32:02.372079  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:32:02.379390 21542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:32:02.379446 21543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:32:02.380190 21545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:32:02.380734  4356 server_base.cc:1061] running on GCE node
I20260709 12:32:02.381875  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:32:02.382102  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:32:02.382277  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600322382260 us; error 0 us; skew 500 ppm
I20260709 12:32:02.382853  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:32:02.386109  4356 webserver.cc:533] Webserver started at http://127.4.65.62:41589/ using document root <none> and password file <none>
I20260709 12:32:02.386579  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:32:02.386740  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:32:02.386972  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:32:02.388083  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/master-0-root/instance:
uuid: "f8c98952f2ef40d4ada68b202df1e026"
format_stamp: "Formatted at 2026-07-09 12:32:02 on dist-test-slave-l1c9"
I20260709 12:32:02.392849  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:32:02.396585 21551 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:32:02.397651  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:32:02.398043  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/master-0-root
uuid: "f8c98952f2ef40d4ada68b202df1e026"
format_stamp: "Formatted at 2026-07-09 12:32:02 on dist-test-slave-l1c9"
I20260709 12:32:02.398447  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:32:02.426971  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:32:02.428298  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:32:02.472681  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.62:37877
I20260709 12:32:02.472781 21602 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.62:37877 every 8 connection(s)
I20260709 12:32:02.477085 21603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:02.490195 21603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026: Bootstrap starting.
I20260709 12:32:02.496076 21603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:02.500944 21603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026: No bootstrap required, opened a new log
I20260709 12:32:02.503212 21603 raft_consensus.cc:359] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c98952f2ef40d4ada68b202df1e026" member_type: VOTER }
I20260709 12:32:02.503602 21603 raft_consensus.cc:385] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:02.503873 21603 raft_consensus.cc:740] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c98952f2ef40d4ada68b202df1e026, State: Initialized, Role: FOLLOWER
I20260709 12:32:02.504752 21603 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c98952f2ef40d4ada68b202df1e026" member_type: VOTER }
I20260709 12:32:02.505453 21603 raft_consensus.cc:399] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:32:02.505704 21603 raft_consensus.cc:493] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:32:02.506014 21603 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:02.511087 21603 raft_consensus.cc:515] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c98952f2ef40d4ada68b202df1e026" member_type: VOTER }
I20260709 12:32:02.511657 21603 leader_election.cc:304] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8c98952f2ef40d4ada68b202df1e026; no voters: 
I20260709 12:32:02.512863 21603 leader_election.cc:290] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:32:02.513221 21606 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:02.514762 21606 raft_consensus.cc:697] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [term 1 LEADER]: Becoming Leader. State: Replica: f8c98952f2ef40d4ada68b202df1e026, State: Running, Role: LEADER
I20260709 12:32:02.515425 21606 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c98952f2ef40d4ada68b202df1e026" member_type: VOTER }
I20260709 12:32:02.516479 21603 sys_catalog.cc:565] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:32:02.520378 21608 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8c98952f2ef40d4ada68b202df1e026. Latest consensus state: current_term: 1 leader_uuid: "f8c98952f2ef40d4ada68b202df1e026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c98952f2ef40d4ada68b202df1e026" member_type: VOTER } }
I20260709 12:32:02.521126 21608 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [sys.catalog]: This master's current role is: LEADER
I20260709 12:32:02.525058 21613 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:32:02.520310 21607 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8c98952f2ef40d4ada68b202df1e026" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c98952f2ef40d4ada68b202df1e026" member_type: VOTER } }
I20260709 12:32:02.528999 21607 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [sys.catalog]: This master's current role is: LEADER
I20260709 12:32:02.534924 21613 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:32:02.536031  4356 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:32:02.545028 21613 catalog_manager.cc:1383] Generated new cluster ID: b718b3263dc14aa7864ffd645af3e26e
I20260709 12:32:02.545382 21613 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:32:02.575647 21613 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:32:02.577088 21613 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:32:02.596246 21613 catalog_manager.cc:6070] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026: Generated new TSK 0
I20260709 12:32:02.596956 21613 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:32:02.603798  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:32:02.609858 21624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:32:02.611071 21625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:32:02.612636 21627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:32:02.612952  4356 server_base.cc:1061] running on GCE node
I20260709 12:32:02.614008  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:32:02.614245  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:32:02.614420  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600322614398 us; error 0 us; skew 500 ppm
I20260709 12:32:02.614923  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:32:02.617509  4356 webserver.cc:533] Webserver started at http://127.4.65.1:40971/ using document root <none> and password file <none>
I20260709 12:32:02.617981  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:32:02.618192  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:32:02.618499  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:32:02.619623  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-0-root/instance:
uuid: "634b4ccc587741a1b468e0a06fe92dd3"
format_stamp: "Formatted at 2026-07-09 12:32:02 on dist-test-slave-l1c9"
I20260709 12:32:02.624612  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.004s
I20260709 12:32:02.628608 21632 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:32:02.629788  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:32:02.630179  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-0-root
uuid: "634b4ccc587741a1b468e0a06fe92dd3"
format_stamp: "Formatted at 2026-07-09 12:32:02 on dist-test-slave-l1c9"
I20260709 12:32:02.630540  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:32:02.644992  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:32:02.646489  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:32:02.648239  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:32:02.650849  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:32:02.651091  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:32:02.651348  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:32:02.651556  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:32:02.698619  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.1:37865
I20260709 12:32:02.698740 21694 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.1:37865 every 8 connection(s)
I20260709 12:32:02.704027  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:32:02.712898 21699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:32:02.714164 21700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:32:02.716791 21702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:32:02.717739  4356 server_base.cc:1061] running on GCE node
I20260709 12:32:02.719194  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:32:02.719386  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:32:02.719610  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600322719573 us; error 0 us; skew 500 ppm
I20260709 12:32:02.720176  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:32:02.723176  4356 webserver.cc:533] Webserver started at http://127.4.65.2:37155/ using document root <none> and password file <none>
I20260709 12:32:02.723716  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:32:02.723919  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:32:02.723932 21695 heartbeater.cc:344] Connected to a master server at 127.4.65.62:37877
I20260709 12:32:02.724241  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:32:02.724510 21695 heartbeater.cc:461] Registering TS with master...
I20260709 12:32:02.725322 21695 heartbeater.cc:507] Master 127.4.65.62:37877 requested a full tablet report, sending...
I20260709 12:32:02.725585  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-1-root/instance:
uuid: "a6e0e4824e8447c49a6cd89a1a208801"
format_stamp: "Formatted at 2026-07-09 12:32:02 on dist-test-slave-l1c9"
I20260709 12:32:02.728032 21568 ts_manager.cc:194] Registered new tserver with Master: 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865)
I20260709 12:32:02.730607 21568 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52026
I20260709 12:32:02.732290  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:32:02.736435 21707 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:32:02.737449  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:32:02.738003  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-1-root
uuid: "a6e0e4824e8447c49a6cd89a1a208801"
format_stamp: "Formatted at 2026-07-09 12:32:02 on dist-test-slave-l1c9"
I20260709 12:32:02.738343  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:32:02.753643  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:32:02.754932  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:32:02.756601  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:32:02.758854  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:32:02.759085  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:32:02.759351  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:32:02.759526  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:32:02.807236  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.2:42355
I20260709 12:32:02.807399 21769 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.2:42355 every 8 connection(s)
I20260709 12:32:02.812915  4356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:32:02.820494 21773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:32:02.822598 21774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:32:02.824160 21776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:32:02.824375  4356 server_base.cc:1061] running on GCE node
I20260709 12:32:02.825438  4356 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:32:02.825726  4356 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:32:02.825977  4356 hybrid_clock.cc:648] HybridClock initialized: now 1783600322825957 us; error 0 us; skew 500 ppm
I20260709 12:32:02.826730  4356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:32:02.827121 21770 heartbeater.cc:344] Connected to a master server at 127.4.65.62:37877
I20260709 12:32:02.827637 21770 heartbeater.cc:461] Registering TS with master...
I20260709 12:32:02.828918 21770 heartbeater.cc:507] Master 127.4.65.62:37877 requested a full tablet report, sending...
I20260709 12:32:02.830338  4356 webserver.cc:533] Webserver started at http://127.4.65.3:46163/ using document root <none> and password file <none>
I20260709 12:32:02.830983  4356 fs_manager.cc:362] Metadata directory not provided
I20260709 12:32:02.831202  4356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:32:02.831524  4356 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:32:02.831696 21568 ts_manager.cc:194] Registered new tserver with Master: a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:02.833501  4356 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-2-root/instance:
uuid: "e2f153c452a54ca9af8d83e3d977beed"
format_stamp: "Formatted at 2026-07-09 12:32:02 on dist-test-slave-l1c9"
I20260709 12:32:02.833657 21568 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52034
I20260709 12:32:02.839848  4356 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.005s
I20260709 12:32:02.843693 21781 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:32:02.844704  4356 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:32:02.845041  4356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-2-root
uuid: "e2f153c452a54ca9af8d83e3d977beed"
format_stamp: "Formatted at 2026-07-09 12:32:02 on dist-test-slave-l1c9"
I20260709 12:32:02.845316  4356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task3zAVL8/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599783304823-4356-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:32:02.859742  4356 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:32:02.861460  4356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:32:02.863106  4356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:32:02.865437  4356 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:32:02.865694  4356 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:32:02.865962  4356 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:32:02.866140  4356 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:32:02.910892  4356 rpc_server.cc:307] RPC server started. Bound to: 127.4.65.3:35391
I20260709 12:32:02.911163 21843 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.65.3:35391 every 8 connection(s)
I20260709 12:32:02.927032 21844 heartbeater.cc:344] Connected to a master server at 127.4.65.62:37877
I20260709 12:32:02.927562 21844 heartbeater.cc:461] Registering TS with master...
I20260709 12:32:02.928313 21844 heartbeater.cc:507] Master 127.4.65.62:37877 requested a full tablet report, sending...
I20260709 12:32:02.930605 21568 ts_manager.cc:194] Registered new tserver with Master: e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:02.931336  4356 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016740094s
I20260709 12:32:02.932245 21568 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52050
I20260709 12:32:02.961192 21568 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52054:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:32:02.963985 21568 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:32:03.041648 21660 tablet_service.cc:1511] Processing CreateTablet for tablet 9ae6b3a3385a420ca2a71c45a1746834 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:32:03.042079 21659 tablet_service.cc:1511] Processing CreateTablet for tablet a09e03b0d2f7447fb6af987f95aa4660 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:32:03.043207 21660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ae6b3a3385a420ca2a71c45a1746834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.043854 21659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a09e03b0d2f7447fb6af987f95aa4660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.051832 21809 tablet_service.cc:1511] Processing CreateTablet for tablet 9ae6b3a3385a420ca2a71c45a1746834 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:32:03.052176 21808 tablet_service.cc:1511] Processing CreateTablet for tablet a09e03b0d2f7447fb6af987f95aa4660 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:32:03.053548 21809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ae6b3a3385a420ca2a71c45a1746834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.054112 21807 tablet_service.cc:1511] Processing CreateTablet for tablet d82dd69565514cf498933ccc35fbedf2 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:32:03.059755 21802 tablet_service.cc:1511] Processing CreateTablet for tablet 27cd681d2bdd4f55913b3ea3ba7fbb4c (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:32:03.059755 21803 tablet_service.cc:1511] Processing CreateTablet for tablet cdb21adbf3984dc0a5e3371e5151334f (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:32:03.057673 21804 tablet_service.cc:1511] Processing CreateTablet for tablet 26f565ff0b8943889da95d998b274773 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:32:03.060920 21805 tablet_service.cc:1511] Processing CreateTablet for tablet c2fe8ccb5201424abf468b013c79119e (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:32:03.057631 21806 tablet_service.cc:1511] Processing CreateTablet for tablet 47ee249250654b38bef5ffa786146d29 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:32:03.063601 21807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d82dd69565514cf498933ccc35fbedf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.068681 21801 tablet_service.cc:1511] Processing CreateTablet for tablet 97f6d0a18b194f388f889318024f8420 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:32:03.069631 21859 tablet_bootstrap.cc:492] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:03.070645 21805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2fe8ccb5201424abf468b013c79119e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.071516 21806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47ee249250654b38bef5ffa786146d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.072930 21802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27cd681d2bdd4f55913b3ea3ba7fbb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.076768 21808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a09e03b0d2f7447fb6af987f95aa4660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.080756 21804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26f565ff0b8943889da95d998b274773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.081663 21803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb21adbf3984dc0a5e3371e5151334f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.080754 21859 tablet_bootstrap.cc:654] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.084523 21801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97f6d0a18b194f388f889318024f8420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.130714 21659 tablet_service.cc:1511] Processing CreateTablet for tablet d82dd69565514cf498933ccc35fbedf2 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:32:03.132967 21659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d82dd69565514cf498933ccc35fbedf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.143801 21660 tablet_service.cc:1511] Processing CreateTablet for tablet 47ee249250654b38bef5ffa786146d29 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:32:03.146744 21660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47ee249250654b38bef5ffa786146d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.160858 21658 tablet_service.cc:1511] Processing CreateTablet for tablet c2fe8ccb5201424abf468b013c79119e (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:32:03.162932 21658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2fe8ccb5201424abf468b013c79119e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.176313 21657 tablet_service.cc:1511] Processing CreateTablet for tablet 26f565ff0b8943889da95d998b274773 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:32:03.179308 21657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26f565ff0b8943889da95d998b274773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.198200 21859 tablet_bootstrap.cc:492] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:03.198894 21859 ts_tablet_manager.cc:1403] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.130s	user 0.027s	sys 0.040s
I20260709 12:32:03.200577 21860 tablet_bootstrap.cc:492] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:03.202224 21859 raft_consensus.cc:359] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } }
I20260709 12:32:03.203543 21859 raft_consensus.cc:385] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.203945 21859 raft_consensus.cc:740] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.204998 21859 consensus_queue.cc:260] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } }
I20260709 12:32:03.214632 21859 ts_tablet_manager.cc:1434] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.004s
I20260709 12:32:03.215474 21695 heartbeater.cc:499] Master 127.4.65.62:37877 was elected leader, sending a full tablet report...
I20260709 12:32:03.216110 21860 tablet_bootstrap.cc:654] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.230309 21657 tablet_service.cc:1511] Processing CreateTablet for tablet cdb21adbf3984dc0a5e3371e5151334f (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:32:03.233747 21657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb21adbf3984dc0a5e3371e5151334f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.235033 21859 tablet_bootstrap.cc:492] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:03.237259 21659 tablet_service.cc:1511] Processing CreateTablet for tablet 97f6d0a18b194f388f889318024f8420 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:32:03.238659 21659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97f6d0a18b194f388f889318024f8420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.242369 21732 tablet_service.cc:1511] Processing CreateTablet for tablet 47ee249250654b38bef5ffa786146d29 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:32:03.243861 21732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47ee249250654b38bef5ffa786146d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.243634 21728 tablet_service.cc:1511] Processing CreateTablet for tablet 27cd681d2bdd4f55913b3ea3ba7fbb4c (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:32:03.244030 21735 tablet_service.cc:1511] Processing CreateTablet for tablet 9ae6b3a3385a420ca2a71c45a1746834 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:32:03.245486 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ae6b3a3385a420ca2a71c45a1746834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.245523 21727 tablet_service.cc:1511] Processing CreateTablet for tablet 97f6d0a18b194f388f889318024f8420 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:32:03.247452 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97f6d0a18b194f388f889318024f8420. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.248678 21731 tablet_service.cc:1511] Processing CreateTablet for tablet c2fe8ccb5201424abf468b013c79119e (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:32:03.249430 21728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27cd681d2bdd4f55913b3ea3ba7fbb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.250353 21731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2fe8ccb5201424abf468b013c79119e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.250695 21734 tablet_service.cc:1511] Processing CreateTablet for tablet a09e03b0d2f7447fb6af987f95aa4660 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:32:03.252230 21734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a09e03b0d2f7447fb6af987f95aa4660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.253726 21730 tablet_service.cc:1511] Processing CreateTablet for tablet 26f565ff0b8943889da95d998b274773 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:32:03.255321 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26f565ff0b8943889da95d998b274773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.257853 21729 tablet_service.cc:1511] Processing CreateTablet for tablet cdb21adbf3984dc0a5e3371e5151334f (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:32:03.248893 21733 tablet_service.cc:1511] Processing CreateTablet for tablet d82dd69565514cf498933ccc35fbedf2 (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:32:03.265770 21859 tablet_bootstrap.cc:654] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.237259 21658 tablet_service.cc:1511] Processing CreateTablet for tablet 27cd681d2bdd4f55913b3ea3ba7fbb4c (DEFAULT_TABLE table=test-workload [id=238b3c7c7b114f9c9f7dc2502e475864]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:32:03.267700 21729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb21adbf3984dc0a5e3371e5151334f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.268953 21658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27cd681d2bdd4f55913b3ea3ba7fbb4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.269265 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d82dd69565514cf498933ccc35fbedf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:03.278594 21864 tablet_bootstrap.cc:492] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:03.294888 21864 tablet_bootstrap.cc:654] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.313031 21860 tablet_bootstrap.cc:492] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:03.313583 21860 ts_tablet_manager.cc:1403] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.113s	user 0.018s	sys 0.015s
I20260709 12:32:03.314718 21864 tablet_bootstrap.cc:492] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:03.315291 21864 ts_tablet_manager.cc:1403] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.037s	user 0.020s	sys 0.013s
I20260709 12:32:03.318341 21864 raft_consensus.cc:359] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } }
I20260709 12:32:03.319079 21864 raft_consensus.cc:385] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.319388 21864 raft_consensus.cc:740] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.320107 21864 consensus_queue.cc:260] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } }
I20260709 12:32:03.327869 21770 heartbeater.cc:499] Master 127.4.65.62:37877 was elected leader, sending a full tablet report...
I20260709 12:32:03.328434 21864 ts_tablet_manager.cc:1434] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.013s	user 0.011s	sys 0.003s
I20260709 12:32:03.328452 21860 raft_consensus.cc:359] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } }
I20260709 12:32:03.329602 21860 raft_consensus.cc:385] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.329885 21860 raft_consensus.cc:740] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.330961 21859 tablet_bootstrap.cc:492] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:03.330799 21860 consensus_queue.cc:260] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } }
I20260709 12:32:03.331656 21859 ts_tablet_manager.cc:1403] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.097s	user 0.014s	sys 0.000s
I20260709 12:32:03.334468 21864 tablet_bootstrap.cc:492] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:03.342432 21859 raft_consensus.cc:359] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.345465 21862 raft_consensus.cc:493] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:03.346026 21862 raft_consensus.cc:515] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } }
I20260709 12:32:03.347323 21859 raft_consensus.cc:385] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.347713 21859 raft_consensus.cc:740] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.352530 21864 tablet_bootstrap.cc:654] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.350135 21862 leader_election.cc:290] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:03.357404 21860 ts_tablet_manager.cc:1434] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.044s	user 0.015s	sys 0.007s
I20260709 12:32:03.358227 21844 heartbeater.cc:499] Master 127.4.65.62:37877 was elected leader, sending a full tablet report...
I20260709 12:32:03.362684 21860 tablet_bootstrap.cc:492] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:03.368568 21859 consensus_queue.cc:260] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.377398 21860 tablet_bootstrap.cc:654] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.378852 21859 ts_tablet_manager.cc:1434] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.047s	user 0.006s	sys 0.003s
I20260709 12:32:03.380012 21859 tablet_bootstrap.cc:492] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:03.385913 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ae6b3a3385a420ca2a71c45a1746834" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:03.386723 21819 raft_consensus.cc:2468] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 0.
I20260709 12:32:03.388379 21634 leader_election.cc:304] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:03.389221 21862 raft_consensus.cc:2804] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:03.389595 21862 raft_consensus.cc:493] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:03.389941 21862 raft_consensus.cc:3060] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.391023 21859 tablet_bootstrap.cc:654] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.399022 21862 raft_consensus.cc:515] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } }
I20260709 12:32:03.402311 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ae6b3a3385a420ca2a71c45a1746834" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:03.402925 21819 raft_consensus.cc:3060] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.406718 21867 raft_consensus.cc:493] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:03.406914 21860 tablet_bootstrap.cc:492] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:03.407804 21860 ts_tablet_manager.cc:1403] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.045s	user 0.011s	sys 0.009s
I20260709 12:32:03.407588 21867 raft_consensus.cc:515] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } }
I20260709 12:32:03.410135 21745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ae6b3a3385a420ca2a71c45a1746834" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:03.411065 21745 raft_consensus.cc:2468] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 0.
I20260709 12:32:03.411731 21860 raft_consensus.cc:359] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.412719 21860 raft_consensus.cc:385] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.413030 21860 raft_consensus.cc:740] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.413887 21860 consensus_queue.cc:260] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
W20260709 12:32:03.415678 21845 tablet.cc:2406] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:32:03.418011 21860 ts_tablet_manager.cc:1434] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.004s
I20260709 12:32:03.419140 21860 tablet_bootstrap.cc:492] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:03.420149 21745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ae6b3a3385a420ca2a71c45a1746834" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:03.420840 21745 raft_consensus.cc:3060] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.425827 21860 tablet_bootstrap.cc:654] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.430153 21745 raft_consensus.cc:2468] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 1.
I20260709 12:32:03.431130 21819 raft_consensus.cc:2468] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 1.
I20260709 12:32:03.433104 21634 leader_election.cc:304] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:03.435901 21862 leader_election.cc:290] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 election: Requested vote from peers e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:03.436208 21877 raft_consensus.cc:2804] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:03.436807 21877 raft_consensus.cc:697] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:03.437705 21877 consensus_queue.cc:237] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } }
I20260709 12:32:03.440325 21867 leader_election.cc:290] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:03.442566 21860 tablet_bootstrap.cc:492] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:03.443142 21860 ts_tablet_manager.cc:1403] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.024s	user 0.012s	sys 0.004s
I20260709 12:32:03.444905 21859 tablet_bootstrap.cc:492] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:03.445482 21859 ts_tablet_manager.cc:1403] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.066s	user 0.025s	sys 0.023s
I20260709 12:32:03.444945 21864 tablet_bootstrap.cc:492] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:03.446245 21860 raft_consensus.cc:359] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.447331 21860 raft_consensus.cc:385] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.447525 21864 ts_tablet_manager.cc:1403] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.113s	user 0.006s	sys 0.013s
I20260709 12:32:03.452428 21860 raft_consensus.cc:740] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.453063 21859 raft_consensus.cc:359] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.454056 21859 raft_consensus.cc:385] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.454588 21859 raft_consensus.cc:740] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.453804 21860 consensus_queue.cc:260] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.457340 21860 ts_tablet_manager.cc:1434] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.014s	user 0.010s	sys 0.005s
I20260709 12:32:03.457410 21859 consensus_queue.cc:260] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.458350 21860 tablet_bootstrap.cc:492] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:03.459385 21859 ts_tablet_manager.cc:1434] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.003s
I20260709 12:32:03.460558 21859 tablet_bootstrap.cc:492] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:03.463675 21864 raft_consensus.cc:359] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.464561 21864 raft_consensus.cc:385] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.464872 21864 raft_consensus.cc:740] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.465734 21860 tablet_bootstrap.cc:654] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.465600 21864 consensus_queue.cc:260] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.468103 21864 ts_tablet_manager.cc:1434] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.015s	user 0.004s	sys 0.000s
I20260709 12:32:03.468729 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ae6b3a3385a420ca2a71c45a1746834" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:03.469261 21864 tablet_bootstrap.cc:492] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:03.471378 21859 tablet_bootstrap.cc:654] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.473156 21566 catalog_manager.cc:5697] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1). New cstate: current_term: 1 leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:03.482174 21864 tablet_bootstrap.cc:654] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.536127 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ae6b3a3385a420ca2a71c45a1746834" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:03.537367 21819 raft_consensus.cc:2393] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a6e0e4824e8447c49a6cd89a1a208801 in current term 1: Already voted for candidate 634b4ccc587741a1b468e0a06fe92dd3 in this term.
I20260709 12:32:03.538930 21709 leader_election.cc:304] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801; no voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed
I20260709 12:32:03.540241 21867 raft_consensus.cc:2749] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:32:03.552996 21867 raft_consensus.cc:493] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:03.554118 21867 raft_consensus.cc:515] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.567237 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a09e03b0d2f7447fb6af987f95aa4660" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:03.568380 21670 raft_consensus.cc:2468] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 0.
I20260709 12:32:03.570741 21708 leader_election.cc:304] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:03.573176 21867 leader_election.cc:290] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:03.573309 21884 raft_consensus.cc:2804] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:03.574133 21884 raft_consensus.cc:493] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:03.574574 21884 raft_consensus.cc:3060] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.574599 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a09e03b0d2f7447fb6af987f95aa4660" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
W20260709 12:32:03.576900 21709 leader_election.cc:343] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:32:03.575091 21616 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:32:03.585394 21860 tablet_bootstrap.cc:492] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:03.586164 21860 ts_tablet_manager.cc:1403] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.128s	user 0.053s	sys 0.066s
I20260709 12:32:03.586175 21859 tablet_bootstrap.cc:492] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:03.586227 21864 tablet_bootstrap.cc:492] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:03.587105 21859 ts_tablet_manager.cc:1403] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.127s	user 0.060s	sys 0.052s
I20260709 12:32:03.587173 21864 ts_tablet_manager.cc:1403] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.118s	user 0.051s	sys 0.044s
I20260709 12:32:03.589092 21884 raft_consensus.cc:515] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.589519 21860 raft_consensus.cc:359] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.590241 21860 raft_consensus.cc:385] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.590270 21859 raft_consensus.cc:359] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.590704 21860 raft_consensus.cc:740] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.591223 21859 raft_consensus.cc:385] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.590725 21864 raft_consensus.cc:359] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.591594 21859 raft_consensus.cc:740] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.591934 21864 raft_consensus.cc:385] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.591833 21860 consensus_queue.cc:260] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.592327 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a09e03b0d2f7447fb6af987f95aa4660" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:03.592545 21864 raft_consensus.cc:740] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.593076 21670 raft_consensus.cc:3060] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.592571 21859 consensus_queue.cc:260] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.593900 21864 consensus_queue.cc:260] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.594635 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a09e03b0d2f7447fb6af987f95aa4660" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:03.595366 21884 leader_election.cc:290] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:03.596506 21819 raft_consensus.cc:3060] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.601042 21864 ts_tablet_manager.cc:1434] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:32:03.602324 21864 tablet_bootstrap.cc:492] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:03.606782 21819 raft_consensus.cc:2468] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 1.
I20260709 12:32:03.608914 21864 tablet_bootstrap.cc:654] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.610067 21859 ts_tablet_manager.cc:1434] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.023s	user 0.004s	sys 0.004s
I20260709 12:32:03.610581 21670 raft_consensus.cc:2468] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 1.
I20260709 12:32:03.611255 21859 tablet_bootstrap.cc:492] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:03.611048 21709 leader_election.cc:304] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:03.616286 21884 raft_consensus.cc:2804] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:03.617048 21884 raft_consensus.cc:697] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Becoming Leader. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:03.617950 21868 raft_consensus.cc:493] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:03.618445 21859 tablet_bootstrap.cc:654] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.618480 21868 raft_consensus.cc:515] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.621722 21868 leader_election.cc:290] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:03.622423 21884 consensus_queue.cc:237] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.629643 21860 ts_tablet_manager.cc:1434] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.043s	user 0.011s	sys 0.000s
I20260709 12:32:03.631147 21860 tablet_bootstrap.cc:492] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:03.639513 21864 tablet_bootstrap.cc:492] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:03.640614 21864 ts_tablet_manager.cc:1403] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.039s	user 0.019s	sys 0.018s
I20260709 12:32:03.641705 21860 tablet_bootstrap.cc:654] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.644320 21859 tablet_bootstrap.cc:492] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:03.645007 21859 ts_tablet_manager.cc:1403] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.034s	user 0.008s	sys 0.007s
I20260709 12:32:03.644783 21864 raft_consensus.cc:359] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.646085 21864 raft_consensus.cc:385] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.646608 21864 raft_consensus.cc:740] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.648002 21864 consensus_queue.cc:260] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.651460 21864 ts_tablet_manager.cc:1434] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:32:03.652927 21864 tablet_bootstrap.cc:492] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:03.655254 21860 tablet_bootstrap.cc:492] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:03.655239 21868 raft_consensus.cc:493] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:03.656236 21868 raft_consensus.cc:515] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.659416 21868 leader_election.cc:290] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:03.656244 21860 ts_tablet_manager.cc:1403] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.025s	user 0.008s	sys 0.013s
I20260709 12:32:03.660367 21859 raft_consensus.cc:359] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.661181 21859 raft_consensus.cc:385] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.661500 21859 raft_consensus.cc:740] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.661996 21864 tablet_bootstrap.cc:654] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.662367 21859 consensus_queue.cc:260] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.663275 21860 raft_consensus.cc:359] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.664008 21860 raft_consensus.cc:385] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.664312 21860 raft_consensus.cc:740] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.665390 21859 ts_tablet_manager.cc:1434] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
I20260709 12:32:03.665139 21860 consensus_queue.cc:260] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.669450 21859 tablet_bootstrap.cc:492] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:03.671751 21567 catalog_manager.cc:5697] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2). New cstate: current_term: 1 leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:03.680225 21860 ts_tablet_manager.cc:1434] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.000s
I20260709 12:32:03.681540 21860 tablet_bootstrap.cc:492] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:03.683508 21859 tablet_bootstrap.cc:654] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.688308 21860 tablet_bootstrap.cc:654] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.689313 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26f565ff0b8943889da95d998b274773" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:03.690083 21670 raft_consensus.cc:2468] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 0.
I20260709 12:32:03.691378 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d82dd69565514cf498933ccc35fbedf2" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:03.693205 21669 raft_consensus.cc:2468] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 0.
I20260709 12:32:03.695552 21782 leader_election.cc:304] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:03.697093 21868 raft_consensus.cc:2804] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:03.697692 21868 raft_consensus.cc:493] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:03.698155 21868 raft_consensus.cc:3060] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.704380 21864 tablet_bootstrap.cc:492] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:03.704748 21782 leader_election.cc:304] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:03.705380 21864 ts_tablet_manager.cc:1403] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.053s	user 0.013s	sys 0.008s
I20260709 12:32:03.709481 21864 raft_consensus.cc:359] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.713866 21864 raft_consensus.cc:385] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.714376 21864 raft_consensus.cc:740] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.715502 21859 tablet_bootstrap.cc:492] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:03.716766 21859 ts_tablet_manager.cc:1403] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.050s	user 0.017s	sys 0.008s
I20260709 12:32:03.715540 21864 consensus_queue.cc:260] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.719213 21864 ts_tablet_manager.cc:1434] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.002s
I20260709 12:32:03.721292 21864 tablet_bootstrap.cc:492] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:03.720894 21859 raft_consensus.cc:359] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.721958 21859 raft_consensus.cc:385] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.722299 21859 raft_consensus.cc:740] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.723222 21859 consensus_queue.cc:260] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.728844 21859 ts_tablet_manager.cc:1434] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 12:32:03.732570 21859 tablet_bootstrap.cc:492] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:03.724074 21868 raft_consensus.cc:515] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.733299 21895 raft_consensus.cc:2804] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:03.737190 21895 raft_consensus.cc:493] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:03.737627 21895 raft_consensus.cc:3060] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.781412 21745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26f565ff0b8943889da95d998b274773" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:03.781749 21744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d82dd69565514cf498933ccc35fbedf2" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:03.783594 21877 raft_consensus.cc:493] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:03.785491 21877 raft_consensus.cc:515] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.736399 21868 leader_election.cc:290] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:03.789680 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26f565ff0b8943889da95d998b274773" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:03.789803 21877 leader_election.cc:290] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:03.791428 21669 raft_consensus.cc:3060] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.791534 21743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2fe8ccb5201424abf468b013c79119e" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:03.793529 21743 raft_consensus.cc:2468] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 0.
I20260709 12:32:03.737313 21864 tablet_bootstrap.cc:654] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.795646 21860 tablet_bootstrap.cc:492] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:03.796427 21633 leader_election.cc:304] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:03.798731 21868 raft_consensus.cc:493] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:03.798897 21744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26f565ff0b8943889da95d998b274773" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:03.799562 21859 tablet_bootstrap.cc:654] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.797063 21860 ts_tablet_manager.cc:1403] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.116s	user 0.014s	sys 0.028s
I20260709 12:32:03.799783 21868 raft_consensus.cc:515] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
W20260709 12:32:03.801457 21782 leader_election.cc:343] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:03.800606 21877 raft_consensus.cc:2804] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:03.804226 21868 leader_election.cc:290] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:03.804797 21877 raft_consensus.cc:493] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:03.805307 21877 raft_consensus.cc:3060] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.806368 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27cd681d2bdd4f55913b3ea3ba7fbb4c" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:03.807276 21670 raft_consensus.cc:2468] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 0.
I20260709 12:32:03.816461 21745 raft_consensus.cc:2468] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 0.
I20260709 12:32:03.808485 21860 raft_consensus.cc:359] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.822258 21745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27cd681d2bdd4f55913b3ea3ba7fbb4c" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:03.823441 21745 raft_consensus.cc:2468] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 0.
I20260709 12:32:03.819232 21860 raft_consensus.cc:385] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.825160 21860 raft_consensus.cc:740] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.826455 21860 consensus_queue.cc:260] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.842549 21860 ts_tablet_manager.cc:1434] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.042s	user 0.008s	sys 0.008s
I20260709 12:32:03.844324 21860 tablet_bootstrap.cc:492] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:03.849452 21895 raft_consensus.cc:515] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.859333 21895 leader_election.cc:290] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:03.854637 21860 tablet_bootstrap.cc:654] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.856132 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2fe8ccb5201424abf468b013c79119e" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:03.856842 21782 leader_election.cc:304] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:03.853713 21669 raft_consensus.cc:2468] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 1.
I20260709 12:32:03.862607 21864 tablet_bootstrap.cc:492] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:03.863518 21864 ts_tablet_manager.cc:1403] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.143s	user 0.017s	sys 0.013s
I20260709 12:32:03.868840 21868 raft_consensus.cc:2804] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:03.869354 21868 raft_consensus.cc:493] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:03.869797 21868 raft_consensus.cc:3060] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.869686 21782 leader_election.cc:304] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:03.871410 21859 tablet_bootstrap.cc:492] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:03.871657 21895 raft_consensus.cc:2804] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:03.872383 21859 ts_tablet_manager.cc:1403] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.141s	user 0.025s	sys 0.031s
I20260709 12:32:03.873023 21877 raft_consensus.cc:515] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.872285 21895 raft_consensus.cc:697] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Leader. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:03.876286 21859 raft_consensus.cc:359] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.867729 21864 raft_consensus.cc:359] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.896540 21743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d82dd69565514cf498933ccc35fbedf2" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:03.897019 21864 raft_consensus.cc:385] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.897372 21864 raft_consensus.cc:740] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.897787 21859 raft_consensus.cc:385] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.898211 21859 raft_consensus.cc:740] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
W20260709 12:32:03.898775 21634 leader_election.cc:343] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:03.876293 21745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2fe8ccb5201424abf468b013c79119e" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:03.899176 21859 consensus_queue.cc:260] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.900189 21745 raft_consensus.cc:3060] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.895310 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d82dd69565514cf498933ccc35fbedf2" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
W20260709 12:32:03.900815 21782 leader_election.cc:343] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:03.901225 21669 raft_consensus.cc:3060] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.881883 21868 raft_consensus.cc:515] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.898326 21864 consensus_queue.cc:260] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.910353 21864 ts_tablet_manager.cc:1434] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.046s	user 0.008s	sys 0.010s
I20260709 12:32:03.910748 21868 leader_election.cc:290] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:03.887019 21895 consensus_queue.cc:237] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.893690 21860 tablet_bootstrap.cc:492] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:03.912336 21860 ts_tablet_manager.cc:1403] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.068s	user 0.011s	sys 0.008s
I20260709 12:32:03.916527 21860 raft_consensus.cc:359] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.917387 21860 raft_consensus.cc:385] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:03.917683 21860 raft_consensus.cc:740] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:03.919214 21860 consensus_queue.cc:260] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.898586 21862 consensus_queue.cc:1048] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:03.921603 21860 ts_tablet_manager.cc:1434] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.002s
I20260709 12:32:03.922816 21862 consensus_queue.cc:1048] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:03.927714 21864 tablet_bootstrap.cc:492] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:03.931074 21669 raft_consensus.cc:2468] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 1.
I20260709 12:32:03.933413 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27cd681d2bdd4f55913b3ea3ba7fbb4c" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:03.933763 21860 tablet_bootstrap.cc:492] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:03.934165 21670 raft_consensus.cc:3060] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.936864 21744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27cd681d2bdd4f55913b3ea3ba7fbb4c" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:03.937737 21744 raft_consensus.cc:3060] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:03.938122 21868 raft_consensus.cc:493] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:03.938886 21868 raft_consensus.cc:515] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.943130 21868 leader_election.cc:290] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:03.945863 21860 tablet_bootstrap.cc:654] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.946362 21859 ts_tablet_manager.cc:1434] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.073s	user 0.030s	sys 0.031s
I20260709 12:32:03.956131 21900 raft_consensus.cc:493] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:03.957324 21859 tablet_bootstrap.cc:492] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:03.957024 21900 raft_consensus.cc:515] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:03.959606 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97f6d0a18b194f388f889318024f8420" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:03.962394 21900 leader_election.cc:290] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:03.963201 21567 catalog_manager.cc:5697] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed reported cstate change: term changed from 0 to 1, leader changed from <none> to e2f153c452a54ca9af8d83e3d977beed (127.4.65.3). New cstate: current_term: 1 leader_uuid: "e2f153c452a54ca9af8d83e3d977beed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: HEALTHY } } }
I20260709 12:32:03.964717 21741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47ee249250654b38bef5ffa786146d29" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:03.967622 21782 leader_election.cc:304] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:03.975669 21745 raft_consensus.cc:2468] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 1.
I20260709 12:32:03.977152 21895 raft_consensus.cc:2804] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:03.977967 21895 raft_consensus.cc:697] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Leader. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:03.979117 21895 consensus_queue.cc:237] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
W20260709 12:32:03.980528 21782 leader_election.cc:343] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:03.966560 21859 tablet_bootstrap.cc:654] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:03.964721 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97f6d0a18b194f388f889318024f8420" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
W20260709 12:32:03.968626 21633 leader_election.cc:343] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:03.986377 21633 leader_election.cc:304] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:03.974925 21860 tablet_bootstrap.cc:492] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:03.989662 21860 ts_tablet_manager.cc:1403] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.056s	user 0.022s	sys 0.002s
I20260709 12:32:03.945910 21877 leader_election.cc:290] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 election: Requested vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:03.986430 21742 raft_consensus.cc:2468] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 0.
I20260709 12:32:03.982519 21670 raft_consensus.cc:2468] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 1.
I20260709 12:32:03.984668 21744 raft_consensus.cc:2468] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 1.
I20260709 12:32:03.994236 21900 raft_consensus.cc:2804] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:03.999423 21900 raft_consensus.cc:697] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:04.001562 21900 consensus_queue.cc:237] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
W20260709 12:32:03.997663 21634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:35391, user_credentials={real_user=slave}} blocked reactor thread for 95254.8us
I20260709 12:32:04.005275 21782 leader_election.cc:304] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:04.006264 21868 raft_consensus.cc:2804] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:04.006747 21868 raft_consensus.cc:697] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Leader. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:04.007037 21860 raft_consensus.cc:359] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.008003 21860 raft_consensus.cc:385] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:04.008420 21860 raft_consensus.cc:740] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:04.008049 21868 consensus_queue.cc:237] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.010089 21860 consensus_queue.cc:260] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.013319 21860 ts_tablet_manager.cc:1434] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.002s
I20260709 12:32:04.014070 21898 raft_consensus.cc:493] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:04.014683 21898 raft_consensus.cc:515] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.015331 21860 tablet_bootstrap.cc:492] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:04.017236 21898 leader_election.cc:290] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:04.024509 21859 tablet_bootstrap.cc:492] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:04.025162 21859 ts_tablet_manager.cc:1403] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.068s	user 0.020s	sys 0.011s
I20260709 12:32:04.027415 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2fe8ccb5201424abf468b013c79119e" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:04.028200 21819 raft_consensus.cc:3060] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:04.030030 21860 tablet_bootstrap.cc:654] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:04.038651 21818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47ee249250654b38bef5ffa786146d29" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:04.039798 21818 raft_consensus.cc:2468] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 0.
I20260709 12:32:04.042649 21859 raft_consensus.cc:359] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.044080 21859 raft_consensus.cc:385] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:04.044798 21859 raft_consensus.cc:740] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:04.043939 21567 catalog_manager.cc:5697] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed reported cstate change: term changed from 0 to 1, leader changed from <none> to e2f153c452a54ca9af8d83e3d977beed (127.4.65.3). New cstate: current_term: 1 leader_uuid: "e2f153c452a54ca9af8d83e3d977beed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: HEALTHY } } }
I20260709 12:32:04.046005 21864 tablet_bootstrap.cc:654] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:04.046811 21859 consensus_queue.cc:260] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.053435 21859 ts_tablet_manager.cc:1434] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.001s
I20260709 12:32:04.055246 21859 tablet_bootstrap.cc:492] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:04.027347 21782 leader_election.cc:304] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed; no voters: 634b4ccc587741a1b468e0a06fe92dd3
I20260709 12:32:04.067948 21868 raft_consensus.cc:2804] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:04.068506 21868 raft_consensus.cc:493] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:04.068954 21868 raft_consensus.cc:3060] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:04.075131 21634 leader_election.cc:304] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:04.075665 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2fe8ccb5201424abf468b013c79119e" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:04.077072 21900 raft_consensus.cc:2804] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:04.077593 21900 raft_consensus.cc:493] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:04.080848 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2fe8ccb5201424abf468b013c79119e" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:04.081847 21742 raft_consensus.cc:2393] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2f153c452a54ca9af8d83e3d977beed in current term 1: Already voted for candidate 634b4ccc587741a1b468e0a06fe92dd3 in this term.
I20260709 12:32:04.078032 21900 raft_consensus.cc:3060] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:04.093958 21782 leader_election.cc:304] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2f153c452a54ca9af8d83e3d977beed; no voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:04.096654 21859 tablet_bootstrap.cc:654] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:04.095435 21568 catalog_manager.cc:5697] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1). New cstate: current_term: 1 leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:04.101631 21884 consensus_queue.cc:1048] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:04.103832 21864 tablet_bootstrap.cc:492] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:04.104511 21864 ts_tablet_manager.cc:1403] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.177s	user 0.018s	sys 0.007s
I20260709 12:32:04.105518 21819 raft_consensus.cc:2468] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 1.
I20260709 12:32:04.113265 21895 raft_consensus.cc:2749] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:32:04.109516 21860 tablet_bootstrap.cc:492] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:04.110922 21864 raft_consensus.cc:359] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.116716 21864 raft_consensus.cc:385] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:04.117281 21864 raft_consensus.cc:740] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:04.116780 21860 ts_tablet_manager.cc:1403] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.102s	user 0.022s	sys 0.003s
I20260709 12:32:04.111845 21868 raft_consensus.cc:515] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.120599 21900 raft_consensus.cc:515] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.121711 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97f6d0a18b194f388f889318024f8420" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:04.128402 21669 raft_consensus.cc:3060] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:04.131526 21744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47ee249250654b38bef5ffa786146d29" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:04.132542 21900 leader_election.cc:290] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 election: Requested vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:04.135373 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47ee249250654b38bef5ffa786146d29" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:04.136518 21819 raft_consensus.cc:3060] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:04.118250 21864 consensus_queue.cc:260] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.124491 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97f6d0a18b194f388f889318024f8420" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:04.127247 21859 tablet_bootstrap.cc:492] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:04.127151 21860 raft_consensus.cc:359] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.138619 21742 raft_consensus.cc:3060] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:04.139012 21860 raft_consensus.cc:385] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:04.139415 21860 raft_consensus.cc:740] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:04.140199 21860 consensus_queue.cc:260] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.141953 21860 ts_tablet_manager.cc:1434] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.024s	user 0.005s	sys 0.000s
W20260709 12:32:04.133926 21633 leader_election.cc:343] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:04.133211 21568 catalog_manager.cc:5697] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed reported cstate change: term changed from 0 to 1, leader changed from <none> to e2f153c452a54ca9af8d83e3d977beed (127.4.65.3). New cstate: current_term: 1 leader_uuid: "e2f153c452a54ca9af8d83e3d977beed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: HEALTHY } } }
I20260709 12:32:04.137333 21867 consensus_queue.cc:1048] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:04.143267 21859 ts_tablet_manager.cc:1403] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.089s	user 0.012s	sys 0.004s
I20260709 12:32:04.158403 21859 raft_consensus.cc:359] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.159148 21859 raft_consensus.cc:385] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:04.159420 21859 raft_consensus.cc:740] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:04.160257 21859 consensus_queue.cc:260] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.160512 21669 raft_consensus.cc:2468] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 1.
I20260709 12:32:04.164316 21859 ts_tablet_manager.cc:1434] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.001s
I20260709 12:32:04.168186 21864 ts_tablet_manager.cc:1434] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.063s	user 0.006s	sys 0.000s
I20260709 12:32:04.169595 21864 tablet_bootstrap.cc:492] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:04.172346 21742 raft_consensus.cc:2468] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 1.
I20260709 12:32:04.123255 21868 leader_election.cc:290] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:04.173998 21782 leader_election.cc:304] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:04.175061 21868 raft_consensus.cc:2804] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:04.175853 21868 raft_consensus.cc:697] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Leader. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:04.164011 21819 raft_consensus.cc:2468] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 1.
I20260709 12:32:04.177367 21864 tablet_bootstrap.cc:654] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:04.177855 21868 consensus_queue.cc:237] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.187106 21634 leader_election.cc:304] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:04.188730 21864 tablet_bootstrap.cc:492] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:04.189553 21864 ts_tablet_manager.cc:1403] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.004s
I20260709 12:32:04.189445 21900 raft_consensus.cc:2804] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:04.191470 21900 raft_consensus.cc:697] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:04.193061 21900 consensus_queue.cc:237] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.195565 21864 raft_consensus.cc:359] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.196738 21864 raft_consensus.cc:385] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:04.197124 21864 raft_consensus.cc:740] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:04.198293 21864 consensus_queue.cc:260] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.200820 21864 ts_tablet_manager.cc:1434] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 12:32:04.201985 21864 tablet_bootstrap.cc:492] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:04.206988 21864 tablet_bootstrap.cc:654] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:04.206663 21567 catalog_manager.cc:5697] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed reported cstate change: term changed from 0 to 1, leader changed from <none> to e2f153c452a54ca9af8d83e3d977beed (127.4.65.3). New cstate: current_term: 1 leader_uuid: "e2f153c452a54ca9af8d83e3d977beed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: HEALTHY } } }
I20260709 12:32:04.220440 21864 tablet_bootstrap.cc:492] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:04.220971 21864 ts_tablet_manager.cc:1403] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.019s	user 0.011s	sys 0.004s
I20260709 12:32:04.224018 21864 raft_consensus.cc:359] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.224895 21864 raft_consensus.cc:385] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:04.223898 21567 catalog_manager.cc:5697] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1). New cstate: current_term: 1 leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:04.225199 21864 raft_consensus.cc:740] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:04.225991 21864 consensus_queue.cc:260] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.228164 21864 ts_tablet_manager.cc:1434] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:32:04.302917 21868 consensus_queue.cc:1048] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:04.304303 21895 raft_consensus.cc:493] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:04.304873 21895 raft_consensus.cc:515] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.306981 21895 leader_election.cc:290] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:04.307689 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdb21adbf3984dc0a5e3371e5151334f" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:04.308108 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdb21adbf3984dc0a5e3371e5151334f" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:04.308390 21670 raft_consensus.cc:2468] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 0.
I20260709 12:32:04.308910 21742 raft_consensus.cc:2468] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 0.
I20260709 12:32:04.309473 21782 leader_election.cc:304] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:04.310326 21895 raft_consensus.cc:2804] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:04.310709 21895 raft_consensus.cc:493] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:04.311051 21895 raft_consensus.cc:3060] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:04.316720 21742 raft_consensus.cc:3060] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:04.319979 21895 raft_consensus.cc:515] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.323566 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdb21adbf3984dc0a5e3371e5151334f" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:04.323865 21744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdb21adbf3984dc0a5e3371e5151334f" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:04.324497 21895 leader_election.cc:290] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:04.324644 21744 raft_consensus.cc:3060] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:04.325304 21669 raft_consensus.cc:3060] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:04.327979 21895 consensus_queue.cc:1048] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:04.332662 21744 raft_consensus.cc:2468] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 1.
I20260709 12:32:04.333554 21669 raft_consensus.cc:2468] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 1.
I20260709 12:32:04.334389 21782 leader_election.cc:304] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:04.335592 21895 raft_consensus.cc:2804] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:04.336140 21895 raft_consensus.cc:697] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Leader. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:04.337152 21895 consensus_queue.cc:237] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.346503 21568 catalog_manager.cc:5697] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed reported cstate change: term changed from 0 to 1, leader changed from <none> to e2f153c452a54ca9af8d83e3d977beed (127.4.65.3). New cstate: current_term: 1 leader_uuid: "e2f153c452a54ca9af8d83e3d977beed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: HEALTHY } } }
I20260709 12:32:04.392683 21742 raft_consensus.cc:3060] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:04.402369 21895 consensus_queue.cc:1048] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:04.413192 21868 consensus_queue.cc:1048] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:04.418043 21925 consensus_queue.cc:1048] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:04.434528 21925 consensus_queue.cc:1048] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:04.489847 21918 raft_consensus.cc:493] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:04.490340 21918 raft_consensus.cc:515] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.492332 21918 leader_election.cc:290] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:04.493634 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47ee249250654b38bef5ffa786146d29" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:04.493824 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47ee249250654b38bef5ffa786146d29" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:04.494587 21819 raft_consensus.cc:2393] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a6e0e4824e8447c49a6cd89a1a208801 in current term 1: Already voted for candidate 634b4ccc587741a1b468e0a06fe92dd3 in this term.
I20260709 12:32:04.496636 21709 leader_election.cc:304] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801; no voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed
I20260709 12:32:04.497665 21918 raft_consensus.cc:3060] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:04.504745 21918 raft_consensus.cc:2749] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:32:04.512708 21900 consensus_queue.cc:1048] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:04.527685 21877 consensus_queue.cc:1048] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:04.561020 21742 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a09e03b0d2f7447fb6af987f95aa4660"
dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
mode: GRACEFUL
new_leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
 from {username='slave'} at 127.0.0.1:60444
I20260709 12:32:04.561964 21742 raft_consensus.cc:606] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Received request to transfer leadership to TS 634b4ccc587741a1b468e0a06fe92dd3
I20260709 12:32:04.579625 21819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d82dd69565514cf498933ccc35fbedf2"
dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
mode: GRACEFUL
new_leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
 from {username='slave'} at 127.0.0.1:48634
I20260709 12:32:04.580426 21819 raft_consensus.cc:606] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Received request to transfer leadership to TS 634b4ccc587741a1b468e0a06fe92dd3
I20260709 12:32:04.595854 21918 raft_consensus.cc:993] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801: : Instructing follower 634b4ccc587741a1b468e0a06fe92dd3 to start an election
I20260709 12:32:04.596441 21884 raft_consensus.cc:1081] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Signalling peer 634b4ccc587741a1b468e0a06fe92dd3 to start an election
I20260709 12:32:04.598462 21670 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a09e03b0d2f7447fb6af987f95aa4660"
dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
 from {username='slave'} at 127.0.0.1:52658
I20260709 12:32:04.599305 21670 raft_consensus.cc:493] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:04.599828 21670 raft_consensus.cc:3060] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:04.610217 21670 raft_consensus.cc:515] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.612995 21670 leader_election.cc:290] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 election: Requested vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:04.613198 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a09e03b0d2f7447fb6af987f95aa4660" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:04.613957 21742 raft_consensus.cc:3055] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:04.614338 21742 raft_consensus.cc:740] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:04.614634 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a09e03b0d2f7447fb6af987f95aa4660" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:04.615253 21819 raft_consensus.cc:3060] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:04.615311 21742 consensus_queue.cc:260] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.616434 21670 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c2fe8ccb5201424abf468b013c79119e"
dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
mode: GRACEFUL
new_leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:52668
I20260709 12:32:04.616894 21742 raft_consensus.cc:3060] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:04.616997 21670 raft_consensus.cc:606] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Received request to transfer leadership to TS a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:04.623375 21817 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "26f565ff0b8943889da95d998b274773"
dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
mode: GRACEFUL
new_leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:48634
I20260709 12:32:04.624109 21819 raft_consensus.cc:2468] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 2.
I20260709 12:32:04.624246 21817 raft_consensus.cc:606] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Received request to transfer leadership to TS a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:04.626006 21742 raft_consensus.cc:2468] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 2.
I20260709 12:32:04.626303 21634 leader_election.cc:304] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:04.627354 21877 raft_consensus.cc:2804] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:04.627866 21877 raft_consensus.cc:697] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Becoming Leader. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:04.628860 21877 consensus_queue.cc:237] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.629374 21907 consensus_queue.cc:1048] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:04.634100 21819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cdb21adbf3984dc0a5e3371e5151334f"
dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
mode: GRACEFUL
new_leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:48634
I20260709 12:32:04.634768 21819 raft_consensus.cc:606] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Received request to transfer leadership to TS a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:04.638098 21567 catalog_manager.cc:5697] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 reported cstate change: term changed from 1 to 2, leader changed from a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2) to 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1). New cstate: current_term: 2 leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:04.644869 21900 consensus_queue.cc:1048] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:04.646404 21819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27cd681d2bdd4f55913b3ea3ba7fbb4c"
dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
mode: GRACEFUL
new_leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:48634
I20260709 12:32:04.649114 21925 consensus_queue.cc:1048] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:32:04.647188 21819 raft_consensus.cc:606] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Received request to transfer leadership to TS a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:04.668861 21868 consensus_queue.cc:1048] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:04.752288 21868 consensus_queue.cc:1048] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:04.763692 21868 consensus_queue.cc:1048] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:04.772028 21898 raft_consensus.cc:993] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed: : Instructing follower a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:04.772668 21868 raft_consensus.cc:1081] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Signalling peer a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:04.774714 21744 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cdb21adbf3984dc0a5e3371e5151334f"
dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:60440
I20260709 12:32:04.775270 21744 raft_consensus.cc:493] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:04.775599 21744 raft_consensus.cc:3060] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:04.781605 21744 raft_consensus.cc:515] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.783350 21744 leader_election.cc:290] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:04.784477 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdb21adbf3984dc0a5e3371e5151334f" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:04.784832 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cdb21adbf3984dc0a5e3371e5151334f" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:04.787711 21670 raft_consensus.cc:3060] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:04.788050 21819 raft_consensus.cc:3055] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:04.788475 21819 raft_consensus.cc:740] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:04.789175 21819 consensus_queue.cc:260] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.790293 21819 raft_consensus.cc:3060] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:04.794009 21670 raft_consensus.cc:2468] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:04.795617 21708 leader_election.cc:304] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:04.796425 21884 raft_consensus.cc:2804] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:04.796890 21884 raft_consensus.cc:697] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Becoming Leader. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:04.797081 21819 raft_consensus.cc:2468] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:04.797771 21884 consensus_queue.cc:237] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.801929 21898 raft_consensus.cc:993] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed: : Instructing follower a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:04.802516 21868 raft_consensus.cc:1081] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Signalling peer a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:04.804313 21744 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "26f565ff0b8943889da95d998b274773"
dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:60440
I20260709 12:32:04.804942 21744 raft_consensus.cc:493] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:04.805347 21744 raft_consensus.cc:3060] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:04.808032 21565 catalog_manager.cc:5697] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 reported cstate change: term changed from 1 to 2, leader changed from e2f153c452a54ca9af8d83e3d977beed (127.4.65.3) to a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:04.813657 21744 raft_consensus.cc:515] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.816771 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26f565ff0b8943889da95d998b274773" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:04.817354 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26f565ff0b8943889da95d998b274773" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:04.817567 21670 raft_consensus.cc:3060] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:04.818104 21819 raft_consensus.cc:3055] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:04.816782 21744 leader_election.cc:290] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:04.818451 21819 raft_consensus.cc:740] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:04.819360 21819 consensus_queue.cc:260] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.820720 21819 raft_consensus.cc:3060] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:04.826535 21670 raft_consensus.cc:2468] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:04.828078 21708 leader_election.cc:304] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:04.828258 21819 raft_consensus.cc:2468] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:04.829162 21884 raft_consensus.cc:2804] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:04.829797 21884 raft_consensus.cc:697] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Becoming Leader. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:04.831154 21884 consensus_queue.cc:237] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:04.838901 21565 catalog_manager.cc:5697] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 reported cstate change: term changed from 1 to 2, leader changed from e2f153c452a54ca9af8d83e3d977beed (127.4.65.3) to a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:05.006495 21907 raft_consensus.cc:993] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3: : Instructing follower a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:05.006999 21900 raft_consensus.cc:1081] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Signalling peer a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:05.008630 21744 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c2fe8ccb5201424abf468b013c79119e"
dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:60426
I20260709 12:32:05.009234 21744 raft_consensus.cc:493] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:05.009647 21744 raft_consensus.cc:3060] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:05.013705 21925 raft_consensus.cc:993] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed: : Instructing follower 634b4ccc587741a1b468e0a06fe92dd3 to start an election
I20260709 12:32:05.014431 21925 raft_consensus.cc:1081] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Signalling peer 634b4ccc587741a1b468e0a06fe92dd3 to start an election
I20260709 12:32:05.016121 21670 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d82dd69565514cf498933ccc35fbedf2"
dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
 from {username='slave'} at 127.0.0.1:52666
I20260709 12:32:05.016662 21670 raft_consensus.cc:493] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:05.016994 21670 raft_consensus.cc:3060] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:05.017163 21744 raft_consensus.cc:515] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:05.019291 21744 leader_election.cc:290] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:05.019981 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2fe8ccb5201424abf468b013c79119e" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:05.020694 21669 raft_consensus.cc:3055] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:05.021028 21669 raft_consensus.cc:740] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:05.021562 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2fe8ccb5201424abf468b013c79119e" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:05.022068 21819 raft_consensus.cc:3060] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:05.021854 21669 consensus_queue.cc:260] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:05.023290 21669 raft_consensus.cc:3060] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:05.024557 21670 raft_consensus.cc:515] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:05.026971 21819 raft_consensus.cc:2468] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:05.027609 21744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d82dd69565514cf498933ccc35fbedf2" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:05.027664 21670 leader_election.cc:290] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 election: Requested vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:05.027870 21817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d82dd69565514cf498933ccc35fbedf2" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:05.028828 21817 raft_consensus.cc:3055] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:05.029134 21817 raft_consensus.cc:740] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:05.029275 21709 leader_election.cc:304] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:05.028312 21744 raft_consensus.cc:3060] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:05.030129 21884 raft_consensus.cc:2804] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:05.030002 21817 consensus_queue.cc:260] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:05.031029 21669 raft_consensus.cc:2468] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:05.031368 21817 raft_consensus.cc:3060] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:05.031486 21884 raft_consensus.cc:697] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Becoming Leader. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:05.033056 21884 consensus_queue.cc:237] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:05.037956 21744 raft_consensus.cc:2468] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 2.
I20260709 12:32:05.039206 21633 leader_election.cc:304] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:05.040055 21900 raft_consensus.cc:2804] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:05.040556 21817 raft_consensus.cc:2468] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 2.
I20260709 12:32:05.040657 21900 raft_consensus.cc:697] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Becoming Leader. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:05.040220 21565 catalog_manager.cc:5697] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 reported cstate change: term changed from 1 to 2, leader changed from 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1) to a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:05.041807 21900 consensus_queue.cc:237] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:05.050670 21568 catalog_manager.cc:5697] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 reported cstate change: term changed from 1 to 2, leader changed from e2f153c452a54ca9af8d83e3d977beed (127.4.65.3) to 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1). New cstate: current_term: 2 leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:05.052975 21898 raft_consensus.cc:993] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed: : Instructing follower a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:05.053586 21868 raft_consensus.cc:1081] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Signalling peer a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:05.055394 21744 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "27cd681d2bdd4f55913b3ea3ba7fbb4c"
dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:60440
I20260709 12:32:05.055934 21744 raft_consensus.cc:493] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:05.056253 21744 raft_consensus.cc:3060] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:05.062350 21744 raft_consensus.cc:515] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:05.064010 21744 leader_election.cc:290] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:05.064955 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27cd681d2bdd4f55913b3ea3ba7fbb4c" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:05.065150 21817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27cd681d2bdd4f55913b3ea3ba7fbb4c" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:05.065673 21669 raft_consensus.cc:3060] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:05.065755 21817 raft_consensus.cc:3055] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:05.066187 21817 raft_consensus.cc:740] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:05.066859 21817 consensus_queue.cc:260] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:05.067822 21817 raft_consensus.cc:3060] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:05.072289 21669 raft_consensus.cc:2468] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:05.073058 21817 raft_consensus.cc:2468] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:05.073836 21708 leader_election.cc:304] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:05.074720 21884 raft_consensus.cc:2804] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:05.075271 21884 raft_consensus.cc:697] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Becoming Leader. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:05.076148 21884 consensus_queue.cc:237] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:05.083664 21568 catalog_manager.cc:5697] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 reported cstate change: term changed from 1 to 2, leader changed from e2f153c452a54ca9af8d83e3d977beed (127.4.65.3) to a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:05.084601 21742 raft_consensus.cc:1275] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Refusing update from remote peer 634b4ccc587741a1b468e0a06fe92dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:05.086311 21900 consensus_queue.cc:1048] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:32:05.095887 21817 raft_consensus.cc:1275] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Refusing update from remote peer 634b4ccc587741a1b468e0a06fe92dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:05.097393 21900 consensus_queue.cc:1048] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:05.207821 21669 raft_consensus.cc:1275] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:05.209586 21884 consensus_queue.cc:1048] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:05.217974 21819 raft_consensus.cc:1275] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:05.220170 21944 consensus_queue.cc:1048] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:05.261878 21669 raft_consensus.cc:1275] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:05.263295 21944 consensus_queue.cc:1048] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:05.272292 21819 raft_consensus.cc:1275] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:05.273998 21944 consensus_queue.cc:1048] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:05.430810 21819 raft_consensus.cc:1275] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Refusing update from remote peer 634b4ccc587741a1b468e0a06fe92dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:05.432865 21900 consensus_queue.cc:1048] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:05.442800 21742 raft_consensus.cc:1275] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Refusing update from remote peer 634b4ccc587741a1b468e0a06fe92dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:05.444433 21907 consensus_queue.cc:1048] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:05.508013 21819 raft_consensus.cc:1275] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:05.509789 21944 consensus_queue.cc:1048] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:05.518195 21669 raft_consensus.cc:1275] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:05.519869 21944 consensus_queue.cc:1048] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:05.558051 21819 raft_consensus.cc:1275] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:05.559410 21944 consensus_queue.cc:1048] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:05.569639 21669 raft_consensus.cc:1275] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:05.571877 21918 consensus_queue.cc:1048] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:07.695384 21567 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34256:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:32:07.698256 21567 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:32:07.754436 21657 tablet_service.cc:1511] Processing CreateTablet for tablet f00bda93a32d4b7097c987570f9bd200 (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:32:07.755425 21732 tablet_service.cc:1511] Processing CreateTablet for tablet f00bda93a32d4b7097c987570f9bd200 (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:32:07.756126 21657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f00bda93a32d4b7097c987570f9bd200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.757063 21732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f00bda93a32d4b7097c987570f9bd200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.757135 21803 tablet_service.cc:1511] Processing CreateTablet for tablet f00bda93a32d4b7097c987570f9bd200 (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:32:07.758811 21803 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f00bda93a32d4b7097c987570f9bd200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.768044 21801 tablet_service.cc:1511] Processing CreateTablet for tablet 3bcdf6089fe44597974ffc8c1a242fae (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:32:07.769701 21733 tablet_service.cc:1511] Processing CreateTablet for tablet 3bcdf6089fe44597974ffc8c1a242fae (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:32:07.771291 21733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bcdf6089fe44597974ffc8c1a242fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.771013 21729 tablet_service.cc:1511] Processing CreateTablet for tablet 51ddca609eb944c6b5d2d3db607e6a5b (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:32:07.773697 21727 tablet_service.cc:1511] Processing CreateTablet for tablet 2200cddad7904f43b989bf2aa66d463e (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:32:07.775261 21805 tablet_service.cc:1511] Processing CreateTablet for tablet 51ddca609eb944c6b5d2d3db607e6a5b (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:32:07.777138 21729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51ddca609eb944c6b5d2d3db607e6a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.778913 21658 tablet_service.cc:1511] Processing CreateTablet for tablet 2200cddad7904f43b989bf2aa66d463e (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:32:07.780182 21660 tablet_service.cc:1511] Processing CreateTablet for tablet 3bcdf6089fe44597974ffc8c1a242fae (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:32:07.780722 21727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2200cddad7904f43b989bf2aa66d463e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.781358 21658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2200cddad7904f43b989bf2aa66d463e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.778908 21659 tablet_service.cc:1511] Processing CreateTablet for tablet 51ddca609eb944c6b5d2d3db607e6a5b (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:32:07.785044 21659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51ddca609eb944c6b5d2d3db607e6a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.792665 21806 tablet_service.cc:1511] Processing CreateTablet for tablet 823c1ea9255c4ea9babb0445a43d721b (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:32:07.792968 21802 tablet_service.cc:1511] Processing CreateTablet for tablet 8d5befb6b1e942dd997193ab1c266e24 (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:32:07.794309 21809 tablet_service.cc:1511] Processing CreateTablet for tablet 2200cddad7904f43b989bf2aa66d463e (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:32:07.795068 21807 tablet_service.cc:1511] Processing CreateTablet for tablet 7aa93a18f572472ab61d61ae8b464c83 (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:32:07.769793 21801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bcdf6089fe44597974ffc8c1a242fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.796470 21805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51ddca609eb944c6b5d2d3db607e6a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.797430 21804 tablet_service.cc:1511] Processing CreateTablet for tablet c5895ee5d036404fbe0eb3227d060196 (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:32:07.795370 21808 tablet_service.cc:1511] Processing CreateTablet for tablet 86f4fd3c8b08460aa53f4f5a7d3e20cf (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:32:07.799228 21809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2200cddad7904f43b989bf2aa66d463e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.802410 21730 tablet_service.cc:1511] Processing CreateTablet for tablet 823c1ea9255c4ea9babb0445a43d721b (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:32:07.803515 21807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa93a18f572472ab61d61ae8b464c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.803860 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 823c1ea9255c4ea9babb0445a43d721b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.805660 21734 tablet_service.cc:1511] Processing CreateTablet for tablet 8d5befb6b1e942dd997193ab1c266e24 (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:32:07.807057 21734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5befb6b1e942dd997193ab1c266e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.807176 21804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5895ee5d036404fbe0eb3227d060196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.810066 21731 tablet_service.cc:1511] Processing CreateTablet for tablet 86f4fd3c8b08460aa53f4f5a7d3e20cf (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:32:07.813386 21653 tablet_service.cc:1511] Processing CreateTablet for tablet 7aa93a18f572472ab61d61ae8b464c83 (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:32:07.816598 21728 tablet_service.cc:1511] Processing CreateTablet for tablet 7aa93a18f572472ab61d61ae8b464c83 (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:32:07.795648 21660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bcdf6089fe44597974ffc8c1a242fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.819396 21735 tablet_service.cc:1511] Processing CreateTablet for tablet c5895ee5d036404fbe0eb3227d060196 (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:32:07.832774 21806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 823c1ea9255c4ea9babb0445a43d721b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.820734 21653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa93a18f572472ab61d61ae8b464c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.821667 21655 tablet_service.cc:1511] Processing CreateTablet for tablet 8d5befb6b1e942dd997193ab1c266e24 (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:32:07.823417 21652 tablet_service.cc:1511] Processing CreateTablet for tablet c5895ee5d036404fbe0eb3227d060196 (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:32:07.825255 21654 tablet_service.cc:1511] Processing CreateTablet for tablet 86f4fd3c8b08460aa53f4f5a7d3e20cf (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:32:07.827170 21656 tablet_service.cc:1511] Processing CreateTablet for tablet 823c1ea9255c4ea9babb0445a43d721b (DEFAULT_TABLE table=soft_deleted_table [id=87baefb7c688482abebc93c4f2784827]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:32:07.837322 21808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86f4fd3c8b08460aa53f4f5a7d3e20cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.840838 21802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5befb6b1e942dd997193ab1c266e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.846419 21731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86f4fd3c8b08460aa53f4f5a7d3e20cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.853878 21963 tablet_bootstrap.cc:492] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:07.855942 21652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5895ee5d036404fbe0eb3227d060196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.856703 21655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5befb6b1e942dd997193ab1c266e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.857386 21656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 823c1ea9255c4ea9babb0445a43d721b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.857920 21735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5895ee5d036404fbe0eb3227d060196. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.864740 21654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86f4fd3c8b08460aa53f4f5a7d3e20cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.896492 21728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa93a18f572472ab61d61ae8b464c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:32:07.925359 21964 tablet_bootstrap.cc:492] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:07.933883 21965 tablet_bootstrap.cc:492] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:07.947395 21963 tablet_bootstrap.cc:654] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:07.967123 21964 tablet_bootstrap.cc:654] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:07.975214 21965 tablet_bootstrap.cc:654] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:07.981581 21963 tablet_bootstrap.cc:492] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:07.982246 21963 ts_tablet_manager.cc:1403] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.129s	user 0.012s	sys 0.008s
I20260709 12:32:07.985491 21963 raft_consensus.cc:359] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:07.986361 21963 raft_consensus.cc:385] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:07.986682 21963 raft_consensus.cc:740] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:07.987471 21963 consensus_queue.cc:260] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:07.991920 21964 tablet_bootstrap.cc:492] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:07.992694 21964 ts_tablet_manager.cc:1403] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.068s	user 0.024s	sys 0.022s
I20260709 12:32:07.993938 21963 ts_tablet_manager.cc:1434] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.003s
I20260709 12:32:07.995157 21963 tablet_bootstrap.cc:492] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:07.995755 21965 tablet_bootstrap.cc:492] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:07.996402 21965 ts_tablet_manager.cc:1403] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.063s	user 0.010s	sys 0.009s
I20260709 12:32:07.996328 21964 raft_consensus.cc:359] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:07.997397 21964 raft_consensus.cc:385] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:07.997730 21964 raft_consensus.cc:740] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:07.998634 21964 consensus_queue.cc:260] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.000324 21965 raft_consensus.cc:359] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.001255 21965 raft_consensus.cc:385] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.001538 21964 ts_tablet_manager.cc:1434] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.002s
I20260709 12:32:08.001623 21965 raft_consensus.cc:740] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.002804 21964 tablet_bootstrap.cc:492] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:08.002789 21965 consensus_queue.cc:260] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.005540 21965 ts_tablet_manager.cc:1434] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:32:08.006683 21965 tablet_bootstrap.cc:492] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:08.010566 21964 tablet_bootstrap.cc:654] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.002887 21963 tablet_bootstrap.cc:654] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.013797 21965 tablet_bootstrap.cc:654] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.025116 21963 tablet_bootstrap.cc:492] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:08.025674 21963 ts_tablet_manager.cc:1403] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.031s	user 0.008s	sys 0.009s
I20260709 12:32:08.029353 21963 raft_consensus.cc:359] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.030202 21963 raft_consensus.cc:385] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.030500 21963 raft_consensus.cc:740] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.031311 21963 consensus_queue.cc:260] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.054211 21868 raft_consensus.cc:493] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:08.055097 21868 raft_consensus.cc:515] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.058223 21868 leader_election.cc:290] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:08.060746 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51ddca609eb944c6b5d2d3db607e6a5b" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:08.061873 21742 raft_consensus.cc:2468] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 0.
I20260709 12:32:08.064091 21782 leader_election.cc:304] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:08.065449 21868 raft_consensus.cc:2804] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:08.065970 21868 raft_consensus.cc:493] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:08.066394 21868 raft_consensus.cc:3060] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.068893 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51ddca609eb944c6b5d2d3db607e6a5b" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
W20260709 12:32:08.070866 21782 leader_election.cc:343] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:08.080906 21868 raft_consensus.cc:515] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.084333 21868 leader_election.cc:290] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:08.086148 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51ddca609eb944c6b5d2d3db607e6a5b" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:08.087153 21742 raft_consensus.cc:3060] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.086146 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51ddca609eb944c6b5d2d3db607e6a5b" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:08.089028 21963 ts_tablet_manager.cc:1434] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.063s	user 0.016s	sys 0.043s
W20260709 12:32:08.089743 21782 leader_election.cc:343] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:08.090196 21963 tablet_bootstrap.cc:492] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:08.091324 21965 tablet_bootstrap.cc:492] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:08.095000 21742 raft_consensus.cc:2468] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 1.
I20260709 12:32:08.096423 21782 leader_election.cc:304] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed; no voters: 634b4ccc587741a1b468e0a06fe92dd3
I20260709 12:32:08.097052 21963 tablet_bootstrap.cc:654] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.097043 21964 tablet_bootstrap.cc:492] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:08.097286 21868 raft_consensus.cc:2804] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:08.097906 21964 ts_tablet_manager.cc:1403] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.095s	user 0.037s	sys 0.039s
I20260709 12:32:08.098897 21868 raft_consensus.cc:697] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Leader. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:08.099995 21868 consensus_queue.cc:237] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.100827 21964 raft_consensus.cc:359] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.101481 21964 raft_consensus.cc:385] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.101750 21964 raft_consensus.cc:740] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.102468 21964 consensus_queue.cc:260] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.097038 21965 ts_tablet_manager.cc:1403] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.091s	user 0.022s	sys 0.049s
I20260709 12:32:08.107657 21965 raft_consensus.cc:359] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.108412 21965 raft_consensus.cc:385] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.108703 21965 raft_consensus.cc:740] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.109820 21963 tablet_bootstrap.cc:492] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:08.109529 21965 consensus_queue.cc:260] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.110402 21963 ts_tablet_manager.cc:1403] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.020s	user 0.014s	sys 0.003s
I20260709 12:32:08.111910 21965 ts_tablet_manager.cc:1434] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20260709 12:32:08.112777 21964 ts_tablet_manager.cc:1434] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:32:08.113057 21965 tablet_bootstrap.cc:492] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:08.113721 21963 raft_consensus.cc:359] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.114637 21963 raft_consensus.cc:385] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.114917 21963 raft_consensus.cc:740] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.118402 21963 consensus_queue.cc:260] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.119920 21964 tablet_bootstrap.cc:492] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:08.121270 21963 ts_tablet_manager.cc:1434] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 12:32:08.124337 21868 raft_consensus.cc:493] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:08.124974 21868 raft_consensus.cc:515] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.126775 21965 tablet_bootstrap.cc:654] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.127276 21964 tablet_bootstrap.cc:654] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.127911 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7aa93a18f572472ab61d61ae8b464c83" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
W20260709 12:32:08.130121 21782 leader_election.cc:343] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:08.130302 21568 catalog_manager.cc:5697] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed reported cstate change: term changed from 0 to 1, leader changed from <none> to e2f153c452a54ca9af8d83e3d977beed (127.4.65.3). New cstate: current_term: 1 leader_uuid: "e2f153c452a54ca9af8d83e3d977beed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: HEALTHY } } }
I20260709 12:32:08.140213 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7aa93a18f572472ab61d61ae8b464c83" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:08.141630 21963 tablet_bootstrap.cc:492] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:08.143038 21868 leader_election.cc:290] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
W20260709 12:32:08.143142 21782 leader_election.cc:343] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:08.144022 21782 leader_election.cc:304] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2f153c452a54ca9af8d83e3d977beed; no voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:08.145781 21868 raft_consensus.cc:2749] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:32:08.148195 21964 tablet_bootstrap.cc:492] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:08.148789 21964 ts_tablet_manager.cc:1403] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.029s	user 0.009s	sys 0.013s
I20260709 12:32:08.148801 21965 tablet_bootstrap.cc:492] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:08.149586 21965 ts_tablet_manager.cc:1403] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.037s	user 0.012s	sys 0.012s
I20260709 12:32:08.150252 21963 tablet_bootstrap.cc:654] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.151889 21964 raft_consensus.cc:359] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.152709 21964 raft_consensus.cc:385] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.153009 21964 raft_consensus.cc:740] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.153878 21964 consensus_queue.cc:260] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.155014 21965 raft_consensus.cc:359] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.156076 21965 raft_consensus.cc:385] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.156603 21965 raft_consensus.cc:740] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.158164 21964 ts_tablet_manager.cc:1434] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260709 12:32:08.157657 21965 consensus_queue.cc:260] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.159379 21964 tablet_bootstrap.cc:492] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:08.160797 21965 ts_tablet_manager.cc:1434] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 12:32:08.161918 21965 tablet_bootstrap.cc:492] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:08.163053 21963 tablet_bootstrap.cc:492] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:08.163615 21963 ts_tablet_manager.cc:1403] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.022s	user 0.018s	sys 0.000s
I20260709 12:32:08.167261 21964 tablet_bootstrap.cc:654] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.166882 21963 raft_consensus.cc:359] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.167788 21963 raft_consensus.cc:385] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.168113 21963 raft_consensus.cc:740] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.168948 21965 tablet_bootstrap.cc:654] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.169023 21963 consensus_queue.cc:260] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.172176 21963 ts_tablet_manager.cc:1434] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.008s	user 0.010s	sys 0.000s
I20260709 12:32:08.183969 21963 tablet_bootstrap.cc:492] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:08.190587 21964 tablet_bootstrap.cc:492] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:08.191862 21964 ts_tablet_manager.cc:1403] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.033s	user 0.012s	sys 0.009s
I20260709 12:32:08.194410 21963 tablet_bootstrap.cc:654] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.194978 21964 raft_consensus.cc:359] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.195590 21964 raft_consensus.cc:385] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.195844 21964 raft_consensus.cc:740] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.194454 21965 tablet_bootstrap.cc:492] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:08.197086 21965 ts_tablet_manager.cc:1403] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.035s	user 0.018s	sys 0.007s
I20260709 12:32:08.196518 21964 consensus_queue.cc:260] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.197598 21951 raft_consensus.cc:493] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:08.198218 21951 raft_consensus.cc:515] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.201646 21965 raft_consensus.cc:359] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.202813 21965 raft_consensus.cc:385] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.203203 21965 raft_consensus.cc:740] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.204290 21965 consensus_queue.cc:260] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.205559 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5895ee5d036404fbe0eb3227d060196" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:08.206563 21669 raft_consensus.cc:2468] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 0.
I20260709 12:32:08.208794 21965 ts_tablet_manager.cc:1434] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 12:32:08.212258 21964 ts_tablet_manager.cc:1434] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.011s
I20260709 12:32:08.213071 21965 tablet_bootstrap.cc:492] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:08.213842 21964 tablet_bootstrap.cc:492] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:08.218165 21963 tablet_bootstrap.cc:492] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:08.218755 21963 ts_tablet_manager.cc:1403] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.045s	user 0.020s	sys 0.002s
I20260709 12:32:08.221539 21964 tablet_bootstrap.cc:654] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.222400 21965 tablet_bootstrap.cc:654] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.222196 21708 leader_election.cc:304] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:08.223337 21982 raft_consensus.cc:2804] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:08.223744 21982 raft_consensus.cc:493] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:08.224068 21982 raft_consensus.cc:3060] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.222052 21963 raft_consensus.cc:359] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.227051 21963 raft_consensus.cc:385] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.227372 21963 raft_consensus.cc:740] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.229311 21963 consensus_queue.cc:260] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.231755 21963 ts_tablet_manager.cc:1434] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.005s
I20260709 12:32:08.237227 21963 tablet_bootstrap.cc:492] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:08.238865 21982 raft_consensus.cc:515] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.242038 21982 leader_election.cc:290] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.245276 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5895ee5d036404fbe0eb3227d060196" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:08.246126 21669 raft_consensus.cc:3060] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.261014 21951 leader_election.cc:290] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.263700 21963 tablet_bootstrap.cc:654] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.268864 21965 tablet_bootstrap.cc:492] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:08.269393 21965 ts_tablet_manager.cc:1403] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.057s	user 0.022s	sys 0.009s
I20260709 12:32:08.269701 21964 tablet_bootstrap.cc:492] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:08.269666 21817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5895ee5d036404fbe0eb3227d060196" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:08.270393 21964 ts_tablet_manager.cc:1403] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.057s	user 0.013s	sys 0.021s
I20260709 12:32:08.270500 21817 raft_consensus.cc:3060] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.270888 21669 raft_consensus.cc:2468] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 1.
I20260709 12:32:08.272648 21708 leader_election.cc:304] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:08.272801 21965 raft_consensus.cc:359] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.273811 21965 raft_consensus.cc:385] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.273550 21964 raft_consensus.cc:359] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.274062 21965 raft_consensus.cc:740] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.274315 21964 raft_consensus.cc:385] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.274642 21964 raft_consensus.cc:740] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.274839 21965 consensus_queue.cc:260] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.275370 21964 consensus_queue.cc:260] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.277136 21965 ts_tablet_manager.cc:1434] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:32:08.277714 21964 ts_tablet_manager.cc:1434] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:32:08.280800 21951 raft_consensus.cc:2804] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:08.281387 21951 raft_consensus.cc:697] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Becoming Leader. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:08.278733 21964 tablet_bootstrap.cc:492] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:08.282390 21951 consensus_queue.cc:237] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.278653 21965 tablet_bootstrap.cc:492] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:08.286211 21963 tablet_bootstrap.cc:492] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:08.286942 21963 ts_tablet_manager.cc:1403] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.050s	user 0.010s	sys 0.009s
I20260709 12:32:08.289623 21964 tablet_bootstrap.cc:654] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.290244 21963 raft_consensus.cc:359] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.291051 21963 raft_consensus.cc:385] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.291766 21963 raft_consensus.cc:740] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.292527 21963 consensus_queue.cc:260] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.294544 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5895ee5d036404fbe0eb3227d060196" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:08.297479 21963 ts_tablet_manager.cc:1434] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.003s
I20260709 12:32:08.298724 21972 raft_consensus.cc:493] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:08.299278 21972 raft_consensus.cc:515] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.300513 21963 tablet_bootstrap.cc:492] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:08.303357 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51ddca609eb944c6b5d2d3db607e6a5b" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:08.304474 21972 leader_election.cc:290] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.304584 21862 raft_consensus.cc:493] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:08.305109 21862 raft_consensus.cc:515] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.304284 21742 raft_consensus.cc:2393] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 634b4ccc587741a1b468e0a06fe92dd3 in current term 1: Already voted for candidate e2f153c452a54ca9af8d83e3d977beed in this term.
I20260709 12:32:08.307379 21862 leader_election.cc:290] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.307478 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51ddca609eb944c6b5d2d3db607e6a5b" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:08.311024 21633 leader_election.cc:304] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3; no voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed
I20260709 12:32:08.313189 21868 raft_consensus.cc:493] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:08.313766 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d5befb6b1e942dd997193ab1c266e24" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:08.314967 21982 raft_consensus.cc:493] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:08.315483 21982 raft_consensus.cc:515] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.311868 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d5befb6b1e942dd997193ab1c266e24" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:08.317615 21982 leader_election.cc:290] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
W20260709 12:32:08.317735 21634 leader_election.cc:343] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:08.318169 21982 raft_consensus.cc:493] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:08.313838 21868 raft_consensus.cc:515] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.320209 21982 raft_consensus.cc:515] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.315523 21965 tablet_bootstrap.cc:654] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.322271 21868 leader_election.cc:290] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:08.323100 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2200cddad7904f43b989bf2aa66d463e" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:08.323541 21964 tablet_bootstrap.cc:492] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:08.323841 21670 raft_consensus.cc:2468] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 0.
I20260709 12:32:08.324545 21964 ts_tablet_manager.cc:1403] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.046s	user 0.020s	sys 0.019s
I20260709 12:32:08.325374 21708 leader_election.cc:304] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:08.326318 21862 raft_consensus.cc:3060] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.314570 21742 raft_consensus.cc:2468] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 0.
I20260709 12:32:08.328990 21633 leader_election.cc:304] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: e2f153c452a54ca9af8d83e3d977beed
I20260709 12:32:08.319542 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00bda93a32d4b7097c987570f9bd200" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:08.330624 21669 raft_consensus.cc:2468] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 0.
I20260709 12:32:08.331229 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00bda93a32d4b7097c987570f9bd200" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:08.331979 21708 leader_election.cc:304] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:08.331955 21670 raft_consensus.cc:2468] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 0.
I20260709 12:32:08.335229 21782 leader_election.cc:304] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:08.334824 21964 raft_consensus.cc:359] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.336012 21964 raft_consensus.cc:385] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.336207 21972 raft_consensus.cc:2804] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:08.336504 21964 raft_consensus.cc:740] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.336839 21868 raft_consensus.cc:2804] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:08.337313 21868 raft_consensus.cc:493] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:08.337809 21868 raft_consensus.cc:3060] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.337859 21964 consensus_queue.cc:260] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.339812 21972 raft_consensus.cc:493] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:08.340245 21972 raft_consensus.cc:3060] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.340711 21982 leader_election.cc:290] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.341463 21982 raft_consensus.cc:2804] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:08.341846 21982 raft_consensus.cc:493] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:08.342193 21982 raft_consensus.cc:3060] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.349291 21983 raft_consensus.cc:2804] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:08.349828 21983 raft_consensus.cc:493] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:08.350162 21983 raft_consensus.cc:3060] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.362910 21963 tablet_bootstrap.cc:654] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.366521 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00bda93a32d4b7097c987570f9bd200" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:08.367528 21862 raft_consensus.cc:2749] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:32:08.368407 21818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2200cddad7904f43b989bf2aa66d463e" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:08.370678 21982 raft_consensus.cc:515] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.373045 21982 leader_election.cc:290] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.374046 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2200cddad7904f43b989bf2aa66d463e" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:08.374748 21670 raft_consensus.cc:3060] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.377310 21964 ts_tablet_manager.cc:1434] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.052s	user 0.023s	sys 0.026s
I20260709 12:32:08.381222 21964 tablet_bootstrap.cc:492] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:08.385483 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00bda93a32d4b7097c987570f9bd200" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:08.390455 21964 tablet_bootstrap.cc:654] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.392664 21862 raft_consensus.cc:493] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:08.393258 21862 raft_consensus.cc:515] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.395478 21862 leader_election.cc:290] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.397264 21818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86f4fd3c8b08460aa53f4f5a7d3e20cf" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
W20260709 12:32:08.399113 21634 leader_election.cc:343] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:08.400367 21744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86f4fd3c8b08460aa53f4f5a7d3e20cf" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
W20260709 12:32:08.402134 21633 leader_election.cc:343] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:08.402606 21633 leader_election.cc:304] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3; no voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed
I20260709 12:32:08.403753 21862 raft_consensus.cc:2749] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:32:08.396481 21817 raft_consensus.cc:2468] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 1.
I20260709 12:32:08.404441 21964 tablet_bootstrap.cc:492] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:08.405187 21964 ts_tablet_manager.cc:1403] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.024s	user 0.016s	sys 0.006s
I20260709 12:32:08.405719 21972 raft_consensus.cc:515] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.409000 21964 raft_consensus.cc:359] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.409992 21964 raft_consensus.cc:385] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.410362 21964 raft_consensus.cc:740] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.411314 21964 consensus_queue.cc:260] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
W20260709 12:32:08.412123 21709 leader_election.cc:343] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:08.414736 21964 ts_tablet_manager.cc:1434] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:32:08.431447 21964 tablet_bootstrap.cc:492] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:08.431936 21963 tablet_bootstrap.cc:492] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:08.432652 21963 ts_tablet_manager.cc:1403] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.134s	user 0.021s	sys 0.033s
I20260709 12:32:08.434299 21972 leader_election.cc:290] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 election: Requested vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.419049 21670 raft_consensus.cc:2468] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 1.
I20260709 12:32:08.435052 21744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d5befb6b1e942dd997193ab1c266e24" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:08.403940 21868 raft_consensus.cc:515] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.435714 21744 raft_consensus.cc:3060] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.436252 21819 raft_consensus.cc:2393] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a6e0e4824e8447c49a6cd89a1a208801 in current term 1: Already voted for candidate e2f153c452a54ca9af8d83e3d977beed in this term.
I20260709 12:32:08.436323 21983 raft_consensus.cc:515] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.437814 21742 raft_consensus.cc:2393] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2f153c452a54ca9af8d83e3d977beed in current term 1: Already voted for candidate a6e0e4824e8447c49a6cd89a1a208801 in this term.
I20260709 12:32:08.440227 21817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d5befb6b1e942dd997193ab1c266e24" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:08.440722 21868 leader_election.cc:290] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:08.427548 21567 catalog_manager.cc:5697] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2). New cstate: current_term: 1 leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:08.442951 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00bda93a32d4b7097c987570f9bd200" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
W20260709 12:32:08.443270 21634 leader_election.cc:343] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:08.443727 21742 raft_consensus.cc:2393] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e2f153c452a54ca9af8d83e3d977beed in current term 1: Already voted for candidate a6e0e4824e8447c49a6cd89a1a208801 in this term.
I20260709 12:32:08.444545 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00bda93a32d4b7097c987570f9bd200" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:08.444794 21964 tablet_bootstrap.cc:654] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.444708 21708 leader_election.cc:304] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:08.445166 21669 raft_consensus.cc:3060] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.445649 21951 raft_consensus.cc:2804] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:08.446143 21951 raft_consensus.cc:697] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Becoming Leader. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:08.446967 21951 consensus_queue.cc:237] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.447341 21817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2200cddad7904f43b989bf2aa66d463e" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:08.452451 21965 tablet_bootstrap.cc:492] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:08.454200 21965 ts_tablet_manager.cc:1403] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.176s	user 0.020s	sys 0.054s
I20260709 12:32:08.457494 21983 leader_election.cc:290] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.460887 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00bda93a32d4b7097c987570f9bd200" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
W20260709 12:32:08.462080 21709 leader_election.cc:343] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:32:08.462498 21782 leader_election.cc:304] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2f153c452a54ca9af8d83e3d977beed; no voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:08.463501 21973 raft_consensus.cc:2749] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:32:08.459756 21965 raft_consensus.cc:359] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.461020 21744 raft_consensus.cc:2468] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 1.
I20260709 12:32:08.455106 21963 raft_consensus.cc:359] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.464288 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00bda93a32d4b7097c987570f9bd200" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:08.465724 21963 raft_consensus.cc:385] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.466212 21963 raft_consensus.cc:740] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.466423 21819 raft_consensus.cc:2393] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a6e0e4824e8447c49a6cd89a1a208801 in current term 1: Already voted for candidate e2f153c452a54ca9af8d83e3d977beed in this term.
I20260709 12:32:08.466696 21633 leader_election.cc:304] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: e2f153c452a54ca9af8d83e3d977beed
I20260709 12:32:08.467096 21963 consensus_queue.cc:260] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.468703 21972 raft_consensus.cc:2804] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:08.464807 21965 raft_consensus.cc:385] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.470268 21669 raft_consensus.cc:2468] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 1.
I20260709 12:32:08.469357 21965 raft_consensus.cc:740] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.474267 21708 leader_election.cc:304] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: e2f153c452a54ca9af8d83e3d977beed
I20260709 12:32:08.476006 21983 raft_consensus.cc:2804] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:08.475890 21965 consensus_queue.cc:260] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.476785 21983 raft_consensus.cc:697] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Becoming Leader. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:08.477002 21963 ts_tablet_manager.cc:1434] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.044s	user 0.004s	sys 0.004s
I20260709 12:32:08.477674 21983 consensus_queue.cc:237] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.478165 21963 tablet_bootstrap.cc:492] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:08.473556 21972 raft_consensus.cc:697] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:08.483098 21972 consensus_queue.cc:237] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.484617 21965 ts_tablet_manager.cc:1434] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.001s
I20260709 12:32:08.484311 21567 catalog_manager.cc:5697] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2). New cstate: current_term: 1 leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:08.486462 21964 tablet_bootstrap.cc:492] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:08.492110 21964 ts_tablet_manager.cc:1403] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.061s	user 0.010s	sys 0.017s
I20260709 12:32:08.492383 21965 tablet_bootstrap.cc:492] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:08.497337 21963 tablet_bootstrap.cc:654] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.496929 21964 raft_consensus.cc:359] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.497843 21964 raft_consensus.cc:385] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.498136 21964 raft_consensus.cc:740] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.498865 21964 consensus_queue.cc:260] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.502964 21964 ts_tablet_manager.cc:1434] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:32:08.505033 21964 tablet_bootstrap.cc:492] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801: Bootstrap starting.
I20260709 12:32:08.505579 21963 tablet_bootstrap.cc:492] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:08.506091 21963 ts_tablet_manager.cc:1403] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.028s	user 0.010s	sys 0.003s
I20260709 12:32:08.506302 21965 tablet_bootstrap.cc:654] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.509263 21963 raft_consensus.cc:359] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.510033 21963 raft_consensus.cc:385] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.510303 21963 raft_consensus.cc:740] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.511493 21964 tablet_bootstrap.cc:654] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.513140 21568 catalog_manager.cc:5697] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1). New cstate: current_term: 1 leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:08.514935 21963 consensus_queue.cc:260] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.517678 21963 ts_tablet_manager.cc:1434] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.003s
I20260709 12:32:08.520043 21963 tablet_bootstrap.cc:492] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3: Bootstrap starting.
I20260709 12:32:08.526522 21963 tablet_bootstrap.cc:654] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.528504 21964 tablet_bootstrap.cc:492] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801: No bootstrap required, opened a new log
I20260709 12:32:08.529177 21964 ts_tablet_manager.cc:1403] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801: Time spent bootstrapping tablet: real 0.024s	user 0.013s	sys 0.000s
I20260709 12:32:08.533160 21964 raft_consensus.cc:359] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.533900 21964 raft_consensus.cc:385] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.534209 21964 raft_consensus.cc:740] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.534931 21964 consensus_queue.cc:260] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.537365 21964 ts_tablet_manager.cc:1434] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:32:08.542778 21568 catalog_manager.cc:5697] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2). New cstate: current_term: 1 leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:08.545475 21963 tablet_bootstrap.cc:492] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3: No bootstrap required, opened a new log
I20260709 12:32:08.545979 21963 ts_tablet_manager.cc:1403] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3: Time spent bootstrapping tablet: real 0.026s	user 0.015s	sys 0.001s
I20260709 12:32:08.546705 21965 tablet_bootstrap.cc:492] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:08.547250 21965 ts_tablet_manager.cc:1403] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.055s	user 0.015s	sys 0.014s
I20260709 12:32:08.548604 21963 raft_consensus.cc:359] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.549388 21963 raft_consensus.cc:385] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.549680 21963 raft_consensus.cc:740] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.550048 21965 raft_consensus.cc:359] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.550431 21963 consensus_queue.cc:260] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.550797 21965 raft_consensus.cc:385] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.551131 21965 raft_consensus.cc:740] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.551901 21965 consensus_queue.cc:260] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.553012 21963 ts_tablet_manager.cc:1434] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:32:08.554194 21965 ts_tablet_manager.cc:1434] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 12:32:08.555492 21965 tablet_bootstrap.cc:492] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:08.562062 21965 tablet_bootstrap.cc:654] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.562203 21996 raft_consensus.cc:493] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:08.563066 21973 raft_consensus.cc:493] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:08.562784 21996 raft_consensus.cc:515] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.563609 21973 raft_consensus.cc:515] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.565333 21996 leader_election.cc:290] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.566020 21973 leader_election.cc:290] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:08.567687 21744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7aa93a18f572472ab61d61ae8b464c83" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:08.568465 21744 raft_consensus.cc:2468] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 0.
I20260709 12:32:08.568830 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "823c1ea9255c4ea9babb0445a43d721b" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:08.569481 21742 raft_consensus.cc:2468] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 0.
I20260709 12:32:08.569671 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "823c1ea9255c4ea9babb0445a43d721b" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:08.569887 21633 leader_election.cc:304] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:08.570451 21669 raft_consensus.cc:2468] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 0.
I20260709 12:32:08.571041 21972 raft_consensus.cc:2804] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:08.571538 21972 raft_consensus.cc:493] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:08.571736 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7aa93a18f572472ab61d61ae8b464c83" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:08.572132 21782 leader_election.cc:304] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:08.572417 21819 raft_consensus.cc:2468] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 0.
I20260709 12:32:08.571903 21972 raft_consensus.cc:3060] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.573374 21868 raft_consensus.cc:2804] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:08.573748 21868 raft_consensus.cc:493] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:08.574084 21868 raft_consensus.cc:3060] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.587939 21973 consensus_queue.cc:1048] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:08.588080 21972 raft_consensus.cc:515] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.592171 21817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7aa93a18f572472ab61d61ae8b464c83" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:08.592568 21868 raft_consensus.cc:515] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.592989 21817 raft_consensus.cc:3060] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.591418 21972 leader_election.cc:290] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 election: Requested vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.594959 21868 leader_election.cc:290] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:08.595784 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7aa93a18f572472ab61d61ae8b464c83" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:08.596467 21742 raft_consensus.cc:3060] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.602561 21965 tablet_bootstrap.cc:492] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:08.602330 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "823c1ea9255c4ea9babb0445a43d721b" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:08.603091 21965 ts_tablet_manager.cc:1403] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.048s	user 0.011s	sys 0.003s
I20260709 12:32:08.603051 21669 raft_consensus.cc:3060] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.603922 21742 raft_consensus.cc:2468] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 1.
I20260709 12:32:08.605095 21633 leader_election.cc:304] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:08.606416 21965 raft_consensus.cc:359] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.607199 21965 raft_consensus.cc:385] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.607511 21965 raft_consensus.cc:740] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.608551 21972 raft_consensus.cc:2804] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:08.608232 21965 consensus_queue.cc:260] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.609176 21972 raft_consensus.cc:697] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:08.610177 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "823c1ea9255c4ea9babb0445a43d721b" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:08.610173 21972 consensus_queue.cc:237] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.610846 21965 ts_tablet_manager.cc:1434] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:32:08.610992 21742 raft_consensus.cc:3060] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.612212 21965 tablet_bootstrap.cc:492] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed: Bootstrap starting.
I20260709 12:32:08.619858 21742 raft_consensus.cc:2468] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 1.
I20260709 12:32:08.620263 21669 raft_consensus.cc:2468] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 1.
I20260709 12:32:08.621861 21782 leader_election.cc:304] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:08.622368 21965 tablet_bootstrap.cc:654] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed: Neither blocks nor log segments found. Creating new log.
I20260709 12:32:08.623586 21868 raft_consensus.cc:2804] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:08.624187 21868 raft_consensus.cc:697] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Leader. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:08.625243 21868 consensus_queue.cc:237] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.627449 21817 raft_consensus.cc:2468] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 1.
I20260709 12:32:08.632788 21965 tablet_bootstrap.cc:492] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed: No bootstrap required, opened a new log
I20260709 12:32:08.633823 21965 ts_tablet_manager.cc:1403] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed: Time spent bootstrapping tablet: real 0.022s	user 0.018s	sys 0.000s
I20260709 12:32:08.636824 21965 raft_consensus.cc:359] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.637578 21965 raft_consensus.cc:385] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:32:08.637882 21965 raft_consensus.cc:740] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Initialized, Role: FOLLOWER
I20260709 12:32:08.638556 21965 consensus_queue.cc:260] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.639074 21568 catalog_manager.cc:5697] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1). New cstate: current_term: 1 leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
W20260709 12:32:08.655604 21616 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:32:08.656283 21868 consensus_queue.cc:1048] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:08.657384 21983 raft_consensus.cc:493] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:08.657961 21983 raft_consensus.cc:515] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.658560 21567 catalog_manager.cc:5697] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed reported cstate change: term changed from 0 to 1, leader changed from <none> to e2f153c452a54ca9af8d83e3d977beed (127.4.65.3). New cstate: current_term: 1 leader_uuid: "e2f153c452a54ca9af8d83e3d977beed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: HEALTHY } } }
I20260709 12:32:08.660636 21965 ts_tablet_manager.cc:1434] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed: Time spent starting tablet: real 0.027s	user 0.007s	sys 0.009s
I20260709 12:32:08.662475 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bcdf6089fe44597974ffc8c1a242fae" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:08.662797 21983 leader_election.cc:290] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.663264 21670 raft_consensus.cc:2468] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 0.
I20260709 12:32:08.664330 21817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bcdf6089fe44597974ffc8c1a242fae" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:08.664752 21708 leader_election.cc:304] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:08.665307 21817 raft_consensus.cc:2468] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 0.
I20260709 12:32:08.665756 21983 raft_consensus.cc:2804] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:08.666234 21983 raft_consensus.cc:493] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:08.667336 21983 raft_consensus.cc:3060] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.675585 21983 raft_consensus.cc:515] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.678575 22010 raft_consensus.cc:493] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:08.679136 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bcdf6089fe44597974ffc8c1a242fae" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:08.679948 21669 raft_consensus.cc:3060] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.679176 22010 raft_consensus.cc:515] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.680640 21817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bcdf6089fe44597974ffc8c1a242fae" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:08.681406 21817 raft_consensus.cc:3060] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.682380 22010 leader_election.cc:290] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:08.684583 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86f4fd3c8b08460aa53f4f5a7d3e20cf" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:08.685259 21670 raft_consensus.cc:2468] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 0.
I20260709 12:32:08.686959 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86f4fd3c8b08460aa53f4f5a7d3e20cf" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:08.687659 21742 raft_consensus.cc:2468] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 0.
I20260709 12:32:08.688221 21669 raft_consensus.cc:2468] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 1.
I20260709 12:32:08.689136 21983 leader_election.cc:290] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.689545 21817 raft_consensus.cc:2468] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 1.
I20260709 12:32:08.689625 21708 leader_election.cc:304] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: 
I20260709 12:32:08.690775 21983 raft_consensus.cc:2804] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:08.691227 21983 raft_consensus.cc:697] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Becoming Leader. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:08.692036 21983 consensus_queue.cc:237] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.692580 21782 leader_election.cc:304] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:08.693531 21868 raft_consensus.cc:2804] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:32:08.693913 21868 raft_consensus.cc:493] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:32:08.694257 21868 raft_consensus.cc:3060] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.700210 21868 raft_consensus.cc:515] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.701751 21868 leader_election.cc:290] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:08.701170 21567 catalog_manager.cc:5697] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2). New cstate: current_term: 1 leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:08.703058 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86f4fd3c8b08460aa53f4f5a7d3e20cf" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:08.703426 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86f4fd3c8b08460aa53f4f5a7d3e20cf" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:08.704521 21669 raft_consensus.cc:3060] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.711584 21669 raft_consensus.cc:2468] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 1.
I20260709 12:32:08.703682 21742 raft_consensus.cc:3060] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.713325 21782 leader_election.cc:304] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:08.714159 21868 raft_consensus.cc:2804] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:32:08.714568 21868 raft_consensus.cc:697] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Leader. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:08.715222 21868 consensus_queue.cc:237] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.718940 21742 raft_consensus.cc:2468] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 1.
I20260709 12:32:08.723047 21567 catalog_manager.cc:5697] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed reported cstate change: term changed from 0 to 1, leader changed from <none> to e2f153c452a54ca9af8d83e3d977beed (127.4.65.3). New cstate: current_term: 1 leader_uuid: "e2f153c452a54ca9af8d83e3d977beed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: HEALTHY } } }
I20260709 12:32:08.735334 21983 consensus_queue.cc:1048] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:08.747721 21951 consensus_queue.cc:1048] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:08.753535 21742 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f00bda93a32d4b7097c987570f9bd200"
dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
mode: GRACEFUL
new_leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
 from {username='slave'} at 127.0.0.1:60444
I20260709 12:32:08.754344 21742 raft_consensus.cc:606] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Received request to transfer leadership to TS 634b4ccc587741a1b468e0a06fe92dd3
I20260709 12:32:08.766703 21817 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "51ddca609eb944c6b5d2d3db607e6a5b"
dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
mode: GRACEFUL
new_leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
 from {username='slave'} at 127.0.0.1:48634
I20260709 12:32:08.767730 21817 raft_consensus.cc:606] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Received request to transfer leadership to TS 634b4ccc587741a1b468e0a06fe92dd3
I20260709 12:32:08.775879 21742 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3bcdf6089fe44597974ffc8c1a242fae"
dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
mode: GRACEFUL
new_leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
 from {username='slave'} at 127.0.0.1:60444
I20260709 12:32:08.776503 21742 raft_consensus.cc:606] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Received request to transfer leadership to TS 634b4ccc587741a1b468e0a06fe92dd3
I20260709 12:32:08.782012 21742 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2200cddad7904f43b989bf2aa66d463e"
dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
mode: GRACEFUL
new_leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
 from {username='slave'} at 127.0.0.1:60444
I20260709 12:32:08.782523 21742 raft_consensus.cc:606] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Received request to transfer leadership to TS 634b4ccc587741a1b468e0a06fe92dd3
I20260709 12:32:08.788681 21817 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "823c1ea9255c4ea9babb0445a43d721b"
dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
mode: GRACEFUL
new_leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:48634
I20260709 12:32:08.789382 21817 raft_consensus.cc:606] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Received request to transfer leadership to TS a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:08.795878 21669 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8d5befb6b1e942dd997193ab1c266e24"
dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
mode: GRACEFUL
new_leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:52668
I20260709 12:32:08.796554 21669 raft_consensus.cc:606] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Received request to transfer leadership to TS a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:08.802275 21819 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "86f4fd3c8b08460aa53f4f5a7d3e20cf"
dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
mode: GRACEFUL
new_leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:48634
I20260709 12:32:08.802866 21819 raft_consensus.cc:606] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Received request to transfer leadership to TS a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:08.808218 21669 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7aa93a18f572472ab61d61ae8b464c83"
dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
mode: GRACEFUL
new_leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:52668
I20260709 12:32:08.808841 21669 raft_consensus.cc:606] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Received request to transfer leadership to TS a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:08.815246 21742 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c5895ee5d036404fbe0eb3227d060196"
dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
mode: GRACEFUL
new_leader_uuid: "e2f153c452a54ca9af8d83e3d977beed"
 from {username='slave'} at 127.0.0.1:60444
I20260709 12:32:08.815750 21742 raft_consensus.cc:606] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Received request to transfer leadership to TS e2f153c452a54ca9af8d83e3d977beed
I20260709 12:32:08.830542 21868 raft_consensus.cc:493] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:32:08.831233 21868 raft_consensus.cc:515] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.833218 21868 leader_election.cc:290] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:08.834328 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d5befb6b1e942dd997193ab1c266e24" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3" is_pre_election: true
I20260709 12:32:08.834478 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d5befb6b1e942dd997193ab1c266e24" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:08.835454 21742 raft_consensus.cc:2393] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e2f153c452a54ca9af8d83e3d977beed in current term 1: Already voted for candidate 634b4ccc587741a1b468e0a06fe92dd3 in this term.
I20260709 12:32:08.836654 21782 leader_election.cc:304] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e2f153c452a54ca9af8d83e3d977beed; no voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801
I20260709 12:32:08.837540 21868 raft_consensus.cc:3060] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:08.842607 21868 raft_consensus.cc:2749] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:32:08.887192 21972 consensus_queue.cc:1048] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:08.898908 21862 consensus_queue.cc:1048] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:08.906947 21862 raft_consensus.cc:993] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3: : Instructing follower a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:08.907403 21862 raft_consensus.cc:1081] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Signalling peer a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:08.911173 21742 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8d5befb6b1e942dd997193ab1c266e24"
dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:60426
I20260709 12:32:08.912026 21742 raft_consensus.cc:493] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:08.912519 21742 raft_consensus.cc:3060] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:08.919188 21742 raft_consensus.cc:515] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.921000 21742 leader_election.cc:290] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.922144 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d5befb6b1e942dd997193ab1c266e24" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:08.922811 21819 raft_consensus.cc:3060] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:08.922923 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d5befb6b1e942dd997193ab1c266e24" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:08.923703 21669 raft_consensus.cc:3055] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:08.923993 21669 raft_consensus.cc:740] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:08.924696 21669 consensus_queue.cc:260] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.925871 21669 raft_consensus.cc:3060] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:08.929299 21819 raft_consensus.cc:2468] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:08.930725 21709 leader_election.cc:304] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:08.931407 21951 raft_consensus.cc:2804] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:08.931668 21669 raft_consensus.cc:2468] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:08.931914 21951 raft_consensus.cc:697] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Becoming Leader. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:08.932870 21951 consensus_queue.cc:237] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.933634 21983 consensus_queue.cc:1048] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:08.940846 21565 catalog_manager.cc:5697] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 reported cstate change: term changed from 1 to 2, leader changed from 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1) to a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:08.944279 21951 consensus_queue.cc:1048] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:08.954792 21991 raft_consensus.cc:993] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801: : Instructing follower 634b4ccc587741a1b468e0a06fe92dd3 to start an election
I20260709 12:32:08.955277 21991 raft_consensus.cc:1081] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Signalling peer 634b4ccc587741a1b468e0a06fe92dd3 to start an election
I20260709 12:32:08.957258 21669 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f00bda93a32d4b7097c987570f9bd200"
dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
 from {username='slave'} at 127.0.0.1:52658
I20260709 12:32:08.957753 21669 raft_consensus.cc:493] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:08.958056 21669 raft_consensus.cc:3060] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:08.964581 21669 raft_consensus.cc:515] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.966555 21669 leader_election.cc:290] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 election: Requested vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:08.968055 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00bda93a32d4b7097c987570f9bd200" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:08.968789 21819 raft_consensus.cc:3060] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:08.968735 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f00bda93a32d4b7097c987570f9bd200" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:08.969568 21742 raft_consensus.cc:3055] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:08.969923 21742 raft_consensus.cc:740] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:08.970768 21742 consensus_queue.cc:260] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.971812 21742 raft_consensus.cc:3060] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:08.974431 21819 raft_consensus.cc:2468] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 2.
I20260709 12:32:08.975509 21634 leader_election.cc:304] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:08.976222 21996 raft_consensus.cc:2804] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:08.976704 21996 raft_consensus.cc:697] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Becoming Leader. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:08.977501 21996 consensus_queue.cc:237] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:08.979094 21742 raft_consensus.cc:2468] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 2.
I20260709 12:32:08.985034 21565 catalog_manager.cc:5697] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 reported cstate change: term changed from 1 to 2, leader changed from a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2) to 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1). New cstate: current_term: 2 leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:09.005067 21951 consensus_queue.cc:1048] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:09.011915 21983 raft_consensus.cc:993] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801: : Instructing follower 634b4ccc587741a1b468e0a06fe92dd3 to start an election
I20260709 12:32:09.012475 21983 raft_consensus.cc:1081] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Signalling peer 634b4ccc587741a1b468e0a06fe92dd3 to start an election
I20260709 12:32:09.015403 21670 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2200cddad7904f43b989bf2aa66d463e"
dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
 from {username='slave'} at 127.0.0.1:52658
I20260709 12:32:09.016474 21670 raft_consensus.cc:493] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:09.017772 21670 raft_consensus.cc:3060] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.015631 21819 raft_consensus.cc:3060] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:32:09.021595 21996 consensus_queue.cc:1048] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:09.026518 21670 raft_consensus.cc:515] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.029803 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2200cddad7904f43b989bf2aa66d463e" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:09.029843 21951 consensus_queue.cc:1048] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:09.030959 21670 leader_election.cc:290] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 election: Requested vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:09.033972 21868 consensus_queue.cc:1048] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:09.034600 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2200cddad7904f43b989bf2aa66d463e" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:09.035271 21819 raft_consensus.cc:3060] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.035558 21742 raft_consensus.cc:3055] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:09.035925 21742 raft_consensus.cc:740] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:09.040936 21862 consensus_queue.cc:1048] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:09.042687 21819 raft_consensus.cc:2468] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 2.
I20260709 12:32:09.036984 21742 consensus_queue.cc:260] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.050891 21742 raft_consensus.cc:3060] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.051018 21868 consensus_queue.cc:1048] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:32:09.043663 21817 raft_consensus.cc:1240] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Rejecting Update request from peer a6e0e4824e8447c49a6cd89a1a208801 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 12:32:09.054186 21634 leader_election.cc:304] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:09.055234 21862 raft_consensus.cc:2804] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:09.055706 21862 raft_consensus.cc:697] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Becoming Leader. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:09.056571 21862 consensus_queue.cc:237] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.070813 21568 catalog_manager.cc:5697] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 reported cstate change: term changed from 1 to 2, leader changed from a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2) to 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1). New cstate: current_term: 2 leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:09.076511 22007 raft_consensus.cc:993] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed: : Instructing follower a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:09.076884 21972 raft_consensus.cc:993] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3: : Instructing follower a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:09.077054 22007 raft_consensus.cc:1081] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Signalling peer a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:09.077351 21972 raft_consensus.cc:1081] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Signalling peer a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:09.080789 21742 raft_consensus.cc:2468] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 2.
I20260709 12:32:09.080853 21743 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7aa93a18f572472ab61d61ae8b464c83"
dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:60426
I20260709 12:32:09.081593 21743 raft_consensus.cc:493] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:09.081961 21745 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "823c1ea9255c4ea9babb0445a43d721b"
dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:60440
I20260709 12:32:09.082566 21745 raft_consensus.cc:493] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:09.082968 21745 raft_consensus.cc:3060] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.082005 21743 raft_consensus.cc:3060] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.090143 21745 raft_consensus.cc:515] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.092993 21745 leader_election.cc:290] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:09.092744 21743 raft_consensus.cc:515] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.093168 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "823c1ea9255c4ea9babb0445a43d721b" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:09.093575 21817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "823c1ea9255c4ea9babb0445a43d721b" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:09.093796 21669 raft_consensus.cc:3060] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.094187 21817 raft_consensus.cc:3055] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:09.094527 21817 raft_consensus.cc:740] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:09.095387 21817 consensus_queue.cc:260] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.095682 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7aa93a18f572472ab61d61ae8b464c83" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:09.096256 21743 leader_election.cc:290] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:09.096316 21670 raft_consensus.cc:3055] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:09.097270 21670 raft_consensus.cc:740] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:09.097760 21817 raft_consensus.cc:3060] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.098084 21670 consensus_queue.cc:260] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.099431 21670 raft_consensus.cc:3060] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.104175 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7aa93a18f572472ab61d61ae8b464c83" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:09.104838 21819 raft_consensus.cc:3060] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.105960 21817 raft_consensus.cc:2468] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:09.107208 21669 raft_consensus.cc:2468] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:09.107324 21709 leader_election.cc:304] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:09.108520 21951 raft_consensus.cc:2804] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:09.109581 21951 raft_consensus.cc:697] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Becoming Leader. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:09.110414 21951 consensus_queue.cc:237] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.112879 21819 raft_consensus.cc:2468] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:09.114382 21709 leader_election.cc:304] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:09.115304 21991 raft_consensus.cc:2804] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:09.115818 21991 raft_consensus.cc:697] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Becoming Leader. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:09.116860 21991 consensus_queue.cc:237] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.117551 22007 raft_consensus.cc:993] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed: : Instructing follower 634b4ccc587741a1b468e0a06fe92dd3 to start an election
I20260709 12:32:09.118052 22007 raft_consensus.cc:1081] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Signalling peer 634b4ccc587741a1b468e0a06fe92dd3 to start an election
I20260709 12:32:09.119328 21990 consensus_queue.cc:1048] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:09.121102 21668 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "51ddca609eb944c6b5d2d3db607e6a5b"
dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
 from {username='slave'} at 127.0.0.1:52666
I20260709 12:32:09.121745 21668 raft_consensus.cc:493] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:09.122123 21668 raft_consensus.cc:3060] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.123008 21568 catalog_manager.cc:5697] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 reported cstate change: term changed from 1 to 2, leader changed from e2f153c452a54ca9af8d83e3d977beed (127.4.65.3) to a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:09.130343 21670 raft_consensus.cc:2468] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:09.132516 21668 raft_consensus.cc:515] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.136089 21743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51ddca609eb944c6b5d2d3db607e6a5b" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:09.136705 21743 raft_consensus.cc:3060] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.138178 21990 raft_consensus.cc:993] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801: : Instructing follower 634b4ccc587741a1b468e0a06fe92dd3 to start an election
I20260709 12:32:09.138687 21990 raft_consensus.cc:1081] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Signalling peer 634b4ccc587741a1b468e0a06fe92dd3 to start an election
I20260709 12:32:09.142606 21817 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "51ddca609eb944c6b5d2d3db607e6a5b" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:09.143258 21817 raft_consensus.cc:3055] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:09.143573 21817 raft_consensus.cc:740] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:09.144501 21817 consensus_queue.cc:260] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.146163 21817 raft_consensus.cc:3060] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.139132 21668 leader_election.cc:290] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 election: Requested vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:09.147307 21669 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3bcdf6089fe44597974ffc8c1a242fae"
dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
 from {username='slave'} at 127.0.0.1:52658
I20260709 12:32:09.147444 21983 consensus_queue.cc:1048] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:09.147941 21669 raft_consensus.cc:493] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:09.148310 21669 raft_consensus.cc:3060] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.149080 21568 catalog_manager.cc:5697] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 reported cstate change: term changed from 1 to 2, leader changed from 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1) to a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:09.153530 21817 raft_consensus.cc:2468] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 2.
I20260709 12:32:09.154819 21634 leader_election.cc:304] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:09.155694 21862 raft_consensus.cc:2804] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:09.156149 21743 raft_consensus.cc:2468] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 2.
I20260709 12:32:09.156155 21862 raft_consensus.cc:697] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Becoming Leader. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:09.157356 21862 consensus_queue.cc:237] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.156193 21669 raft_consensus.cc:515] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.164278 21669 leader_election.cc:290] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 election: Requested vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:09.164597 21818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bcdf6089fe44597974ffc8c1a242fae" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:09.167095 21743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3bcdf6089fe44597974ffc8c1a242fae" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:09.167712 21743 raft_consensus.cc:3055] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:09.168025 21743 raft_consensus.cc:740] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:09.169332 21743 consensus_queue.cc:260] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.170023 21868 consensus_queue.cc:1048] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:09.171026 21743 raft_consensus.cc:3060] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.179353 21743 raft_consensus.cc:2468] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 634b4ccc587741a1b468e0a06fe92dd3 in term 2.
I20260709 12:32:09.180723 21633 leader_election.cc:304] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, a6e0e4824e8447c49a6cd89a1a208801; no voters: e2f153c452a54ca9af8d83e3d977beed
I20260709 12:32:09.181973 21862 raft_consensus.cc:2804] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:09.182528 21862 raft_consensus.cc:697] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Becoming Leader. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:09.182366 21568 catalog_manager.cc:5697] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 reported cstate change: term changed from 1 to 2, leader changed from e2f153c452a54ca9af8d83e3d977beed (127.4.65.3) to 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1). New cstate: current_term: 2 leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:09.183508 21862 consensus_queue.cc:237] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.186368 22010 raft_consensus.cc:993] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed: : Instructing follower a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:09.186897 22010 raft_consensus.cc:1081] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Signalling peer a6e0e4824e8447c49a6cd89a1a208801 to start an election
I20260709 12:32:09.188802 21743 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "86f4fd3c8b08460aa53f4f5a7d3e20cf"
dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
 from {username='slave'} at 127.0.0.1:60440
I20260709 12:32:09.189409 21743 raft_consensus.cc:493] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:09.189802 21743 raft_consensus.cc:3060] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.193380 22010 consensus_queue.cc:1048] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:09.198649 21743 raft_consensus.cc:515] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.200074 21745 raft_consensus.cc:1240] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Rejecting Update request from peer e2f153c452a54ca9af8d83e3d977beed for earlier term 1. Current term is 2. Ops: []
I20260709 12:32:09.200898 21567 catalog_manager.cc:5697] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 reported cstate change: term changed from 1 to 2, leader changed from a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2) to 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1). New cstate: current_term: 2 leader_uuid: "634b4ccc587741a1b468e0a06fe92dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:09.202298 21668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86f4fd3c8b08460aa53f4f5a7d3e20cf" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:09.204092 21868 consensus_queue.cc:1059] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:09.204957 21868 raft_consensus.cc:3055] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:09.205225 21868 raft_consensus.cc:740] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:09.205947 21868 consensus_queue.cc:260] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.206317 21819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86f4fd3c8b08460aa53f4f5a7d3e20cf" candidate_uuid: "a6e0e4824e8447c49a6cd89a1a208801" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
I20260709 12:32:09.206815 21868 raft_consensus.cc:3060] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.206343 21743 leader_election.cc:290] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:09.222760 21819 raft_consensus.cc:2468] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e0e4824e8447c49a6cd89a1a208801 in term 2.
I20260709 12:32:09.224675 21709 leader_election.cc:304] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed; no voters: 634b4ccc587741a1b468e0a06fe92dd3
I20260709 12:32:09.225747 21990 raft_consensus.cc:2804] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:09.226358 21990 raft_consensus.cc:697] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Becoming Leader. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:09.227293 21990 consensus_queue.cc:237] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.235862 21567 catalog_manager.cc:5697] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 reported cstate change: term changed from 1 to 2, leader changed from e2f153c452a54ca9af8d83e3d977beed (127.4.65.3) to a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2). New cstate: current_term: 2 leader_uuid: "a6e0e4824e8447c49a6cd89a1a208801" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: UNKNOWN } } }
I20260709 12:32:09.326201 21819 raft_consensus.cc:1275] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:09.327977 21990 consensus_queue.cc:1048] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:09.337111 21669 raft_consensus.cc:1275] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:09.339205 21983 consensus_queue.cc:1048] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:32:09.347672 21991 raft_consensus.cc:993] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801: : Instructing follower e2f153c452a54ca9af8d83e3d977beed to start an election
I20260709 12:32:09.348242 21951 raft_consensus.cc:1081] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Signalling peer e2f153c452a54ca9af8d83e3d977beed to start an election
I20260709 12:32:09.349984 21819 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c5895ee5d036404fbe0eb3227d060196"
dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
 from {username='slave'} at 127.0.0.1:48626
I20260709 12:32:09.350512 21819 raft_consensus.cc:493] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:09.350834 21819 raft_consensus.cc:3060] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.358422 21819 raft_consensus.cc:515] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.360123 21819 leader_election.cc:290] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 2 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:09.361333 21743 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5895ee5d036404fbe0eb3227d060196" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:09.361796 21669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5895ee5d036404fbe0eb3227d060196" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:09.362102 21743 raft_consensus.cc:3055] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:32:09.362375 21669 raft_consensus.cc:3060] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.362485 21743 raft_consensus.cc:740] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:09.363201 21743 consensus_queue.cc:260] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.364151 21743 raft_consensus.cc:3060] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.367511 21669 raft_consensus.cc:2468] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 2.
I20260709 12:32:09.368698 21782 leader_election.cc:304] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:09.369536 21868 raft_consensus.cc:2804] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:32:09.369833 21743 raft_consensus.cc:2468] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 2.
I20260709 12:32:09.370302 21868 raft_consensus.cc:697] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [term 2 LEADER]: Becoming Leader. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:09.371151 21868 consensus_queue.cc:237] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:09.377933 21568 catalog_manager.cc:5697] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed reported cstate change: term changed from 1 to 2, leader changed from a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2) to e2f153c452a54ca9af8d83e3d977beed (127.4.65.3). New cstate: current_term: 2 leader_uuid: "e2f153c452a54ca9af8d83e3d977beed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: HEALTHY } } }
I20260709 12:32:09.438937 21819 raft_consensus.cc:1275] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Refusing update from remote peer 634b4ccc587741a1b468e0a06fe92dd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:09.440657 21862 consensus_queue.cc:1048] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:32:09.451810 21743 raft_consensus.cc:1275] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Refusing update from remote peer 634b4ccc587741a1b468e0a06fe92dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:09.453459 21996 consensus_queue.cc:1048] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:32:09.470162 21743 raft_consensus.cc:1275] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Refusing update from remote peer 634b4ccc587741a1b468e0a06fe92dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:09.471873 21996 consensus_queue.cc:1048] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:09.475790 21819 raft_consensus.cc:1275] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Refusing update from remote peer 634b4ccc587741a1b468e0a06fe92dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:09.477495 21996 consensus_queue.cc:1048] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:32:09.540683 21669 raft_consensus.cc:1275] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:09.541698 21819 raft_consensus.cc:1275] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:09.542033 21951 consensus_queue.cc:1048] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:09.543064 21991 consensus_queue.cc:1048] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:09.555577 21669 raft_consensus.cc:1275] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:09.557024 21983 consensus_queue.cc:1048] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:09.557331 21819 raft_consensus.cc:1275] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:09.558439 21743 raft_consensus.cc:1275] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Refusing update from remote peer 634b4ccc587741a1b468e0a06fe92dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:09.559388 21983 consensus_queue.cc:1048] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:09.560711 21862 consensus_queue.cc:1048] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:09.572628 21819 raft_consensus.cc:1275] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Refusing update from remote peer 634b4ccc587741a1b468e0a06fe92dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:09.574156 21862 consensus_queue.cc:1048] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:09.585036 21743 raft_consensus.cc:1275] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Refusing update from remote peer 634b4ccc587741a1b468e0a06fe92dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:09.586264 22032 consensus_queue.cc:1048] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:09.596966 21818 raft_consensus.cc:3060] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:09.608214 21818 raft_consensus.cc:1275] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Refusing update from remote peer 634b4ccc587741a1b468e0a06fe92dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:09.609841 21996 consensus_queue.cc:1048] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:32:10.768702 21972 raft_consensus.cc:493] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2f153c452a54ca9af8d83e3d977beed)
I20260709 12:32:10.770416 22037 raft_consensus.cc:493] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a6e0e4824e8447c49a6cd89a1a208801)
I20260709 12:32:10.785333 22037 raft_consensus.cc:515] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:10.787786 22037 leader_election.cc:290] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:10.788589 21738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26f565ff0b8943889da95d998b274773" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:10.812213 21669 raft_consensus.cc:1275] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Refusing update from remote peer e2f153c452a54ca9af8d83e3d977beed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:10.818466 22039 consensus_queue.cc:1048] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:10.819639 21568 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52034 (request call id 91) took 1204 ms. Trace:
I20260709 12:32:10.785140 21972 raft_consensus.cc:515] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:10.820427 21818 raft_consensus.cc:1275] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:10.820155 21568 rpcz_store.cc:276] 0709 12:32:09.615605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:32:09.615768 (+   163us) service_pool.cc:224] Handling call
0709 12:32:10.819555 (+1203787us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1214,"reported_tablets":1}
I20260709 12:32:10.826264 21742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86f4fd3c8b08460aa53f4f5a7d3e20cf" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:10.821785 21972 leader_election.cc:290] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:10.837477 22037 raft_consensus.cc:493] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e2f153c452a54ca9af8d83e3d977beed)
I20260709 12:32:10.838485 22037 raft_consensus.cc:515] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:10.842104 21739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97f6d0a18b194f388f889318024f8420" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801" is_pre_election: true
I20260709 12:32:10.842250 22041 consensus_queue.cc:1048] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:32:10.846817 22037 leader_election.cc:290] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355), e2f153c452a54ca9af8d83e3d977beed (127.4.65.3:35391)
I20260709 12:32:10.865844 21668 raft_consensus.cc:3060] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:32:10.867586 21739 raft_consensus.cc:1275] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Refusing update from remote peer e2f153c452a54ca9af8d83e3d977beed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:10.869518 22039 consensus_queue.cc:1048] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:32:10.874073 21668 raft_consensus.cc:1275] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Refusing update from remote peer a6e0e4824e8447c49a6cd89a1a208801: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:32:10.877769 21818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26f565ff0b8943889da95d998b274773" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:10.880419 21818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86f4fd3c8b08460aa53f4f5a7d3e20cf" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:10.892944 22041 consensus_queue.cc:1048] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:32:10.897435 21818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97f6d0a18b194f388f889318024f8420" candidate_uuid: "634b4ccc587741a1b468e0a06fe92dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e2f153c452a54ca9af8d83e3d977beed" is_pre_election: true
I20260709 12:32:10.910408 21634 leader_election.cc:304] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3; no voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed
I20260709 12:32:10.924626 22044 raft_consensus.cc:2749] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:32:10.925987 21634 leader_election.cc:304] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3; no voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed
I20260709 12:32:10.926779 22044 raft_consensus.cc:2749] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:32:10.927227 21634 leader_election.cc:304] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3; no voters: a6e0e4824e8447c49a6cd89a1a208801, e2f153c452a54ca9af8d83e3d977beed
I20260709 12:32:10.928104 22044 raft_consensus.cc:2749] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:32:11.316193 21691 debug-util.cc:398] Leaking SignalData structure 0x7b080018eb20 after lost signal to thread 4359
W20260709 12:32:11.317919 21691 debug-util.cc:398] Leaking SignalData structure 0x7b08003cd600 after lost signal to thread 21602
W20260709 12:32:11.319432 21691 debug-util.cc:398] Leaking SignalData structure 0x7b080015aa20 after lost signal to thread 21694
W20260709 12:32:11.320657 21691 debug-util.cc:398] Leaking SignalData structure 0x7b0800228700 after lost signal to thread 21769
W20260709 12:32:11.321985 21691 debug-util.cc:398] Leaking SignalData structure 0x7b0800267d60 after lost signal to thread 21843
I20260709 12:32:11.894399 21568 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34256:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 12:32:11.926532 21568 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34256:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 12:32:12.141492  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:32:12.179612 22065 log.cc:927] Time spent T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3: Append to log took a long time: real 0.134s	user 0.005s	sys 0.000s
I20260709 12:32:12.252638 22094 tablet.cc:1724] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:32:12.274039 22071 log.cc:927] Time spent T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3: Append to log took a long time: real 0.227s	user 0.004s	sys 0.000s
I20260709 12:32:12.353150 22109 tablet.cc:1724] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.366945 21742 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "27cd681d2bdd4f55913b3ea3ba7fbb4c"
dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
mode: GRACEFUL
new_leader_uuid: "e2f153c452a54ca9af8d83e3d977beed"
 from {username='slave'} at 127.0.0.1:52974
I20260709 12:32:12.368017 21742 raft_consensus.cc:606] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Received request to transfer leadership to TS e2f153c452a54ca9af8d83e3d977beed
W20260709 12:32:12.374603 22097 log.cc:927] Time spent T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
W20260709 12:32:12.368602 22089 log.cc:927] Time spent T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:32:12.364521 22102 tablet.cc:1724] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:32:12.389803 22100 log.cc:927] Time spent T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:32:12.374377 22106 tablet.cc:1724] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:32:12.417410 22098 log.cc:927] Time spent T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801: Append to log took a long time: real 0.132s	user 0.004s	sys 0.002s
I20260709 12:32:12.430756 22107 tablet.cc:1724] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.403752 22108 tablet.cc:1724] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.484246 22118 tablet.cc:1724] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.488435 22115 tablet.cc:1724] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:32:12.491947 21634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:35391, user_credentials={real_user=slave}} blocked reactor thread for 94206.3us
I20260709 12:32:12.497236 22114 tablet.cc:1724] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.545068 22123 tablet.cc:1724] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.565881 22094 tablet.cc:1724] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.570125 21668 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d82dd69565514cf498933ccc35fbedf2"
dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
mode: GRACEFUL
new_leader_uuid: "e2f153c452a54ca9af8d83e3d977beed"
 from {username='slave'} at 127.0.0.1:55886
I20260709 12:32:12.570832 21668 raft_consensus.cc:606] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Received request to transfer leadership to TS e2f153c452a54ca9af8d83e3d977beed
I20260709 12:32:12.572511  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:32:12.573216  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:32:12.594508 22094 tablet.cc:1724] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.510283 22107 tablet.cc:1724] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:32:12.549953 21709 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.65.3:35391, user_credentials={real_user=slave}} blocked reactor thread for 152176us
I20260709 12:32:12.605891 22109 tablet.cc:1724] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.619515 22118 tablet.cc:1724] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.621048 22108 tablet.cc:1724] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.615952 22094 tablet.cc:1724] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.642656 22119 raft_consensus.cc:993] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3: : Instructing follower e2f153c452a54ca9af8d83e3d977beed to start an election
I20260709 12:32:12.644572 22119 raft_consensus.cc:1081] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Signalling peer e2f153c452a54ca9af8d83e3d977beed to start an election
I20260709 12:32:12.545037 22122 tablet.cc:1724] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.646277 21815 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d82dd69565514cf498933ccc35fbedf2"
dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
 from {username='slave'} at 127.0.0.1:48616
I20260709 12:32:12.646864 21815 raft_consensus.cc:493] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:12.647228 21815 raft_consensus.cc:3060] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:32:12.649252 22104 raft_consensus.cc:993] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801: : Instructing follower e2f153c452a54ca9af8d83e3d977beed to start an election
I20260709 12:32:12.650048 22104 raft_consensus.cc:1081] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Signalling peer e2f153c452a54ca9af8d83e3d977beed to start an election
I20260709 12:32:12.657415 22115 tablet.cc:1724] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.662788 21814 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "27cd681d2bdd4f55913b3ea3ba7fbb4c"
dest_uuid: "e2f153c452a54ca9af8d83e3d977beed"
 from {username='slave'} at 127.0.0.1:48626
I20260709 12:32:12.663471 21814 raft_consensus.cc:493] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:32:12.664216 21814 raft_consensus.cc:3060] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:32:12.663372 22106 tablet.cc:1724] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.674283 22125 tablet.cc:1724] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.661022 21815 raft_consensus.cc:515] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:12.689548 21670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d82dd69565514cf498933ccc35fbedf2" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:12.690285 21670 raft_consensus.cc:3055] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:32:12.690575 21670 raft_consensus.cc:740] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 634b4ccc587741a1b468e0a06fe92dd3, State: Running, Role: LEADER
I20260709 12:32:12.691366 21670 consensus_queue.cc:260] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:12.693423 22107 tablet.cc:1724] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.696461 21815 leader_election.cc:290] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 3 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:12.698535 21670 raft_consensus.cc:3060] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:32:12.663995 22109 tablet.cc:1724] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.705423 22123 tablet.cc:1724] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.709015 22122 tablet.cc:1724] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.714290 22125 tablet.cc:1724] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.711066 22118 tablet.cc:1724] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:32:12.721325 21739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d82dd69565514cf498933ccc35fbedf2" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:12.722064 21739 raft_consensus.cc:3060] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:32:12.699304 21814 raft_consensus.cc:515] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:12.737377 21670 raft_consensus.cc:2468] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 3.
I20260709 12:32:12.737578 21668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27cd681d2bdd4f55913b3ea3ba7fbb4c" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "634b4ccc587741a1b468e0a06fe92dd3"
I20260709 12:32:12.738422 21668 raft_consensus.cc:3060] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:32:12.739035 21782 leader_election.cc:304] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:12.740253 22110 raft_consensus.cc:2804] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:32:12.740813 22110 raft_consensus.cc:697] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 3 LEADER]: Becoming Leader. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:12.741693 22110 consensus_queue.cc:237] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:12.753578 21745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27cd681d2bdd4f55913b3ea3ba7fbb4c" candidate_uuid: "e2f153c452a54ca9af8d83e3d977beed" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a6e0e4824e8447c49a6cd89a1a208801"
I20260709 12:32:12.754412 21745 raft_consensus.cc:3055] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:32:12.754738 21745 raft_consensus.cc:740] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a6e0e4824e8447c49a6cd89a1a208801, State: Running, Role: LEADER
I20260709 12:32:12.754825 21814 leader_election.cc:290] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 3 election: Requested vote from peers 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865), a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2:42355)
I20260709 12:32:12.755683 21745 consensus_queue.cc:260] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:12.757740 21745 raft_consensus.cc:3060] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:32:12.768407 21567 catalog_manager.cc:5697] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed reported cstate change: term changed from 2 to 3, leader changed from 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1) to e2f153c452a54ca9af8d83e3d977beed (127.4.65.3). New cstate: current_term: 3 leader_uuid: "e2f153c452a54ca9af8d83e3d977beed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: HEALTHY } } }
I20260709 12:32:12.770540 21739 raft_consensus.cc:2468] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 3.
I20260709 12:32:12.753695 21668 raft_consensus.cc:2468] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 3.
I20260709 12:32:12.790104 21782 leader_election.cc:304] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 634b4ccc587741a1b468e0a06fe92dd3, e2f153c452a54ca9af8d83e3d977beed; no voters: 
I20260709 12:32:12.790993 22110 raft_consensus.cc:2804] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:32:12.791505 22110 raft_consensus.cc:697] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 3 LEADER]: Becoming Leader. State: Replica: e2f153c452a54ca9af8d83e3d977beed, State: Running, Role: LEADER
I20260709 12:32:12.792191 22110 consensus_queue.cc:237] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } }
I20260709 12:32:12.805573 21745 raft_consensus.cc:2468] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2f153c452a54ca9af8d83e3d977beed in term 3.
I20260709 12:32:12.819697 21565 catalog_manager.cc:6242] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026: soft_deleted_table [id=87baefb7c688482abebc93c4f2784827] alter complete (version 1)
I20260709 12:32:12.827289 21565 catalog_manager.cc:5697] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed reported cstate change: term changed from 2 to 3, leader changed from a6e0e4824e8447c49a6cd89a1a208801 (127.4.65.2) to e2f153c452a54ca9af8d83e3d977beed (127.4.65.3). New cstate: current_term: 3 leader_uuid: "e2f153c452a54ca9af8d83e3d977beed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2f153c452a54ca9af8d83e3d977beed" member_type: VOTER last_known_addr { host: "127.4.65.3" port: 35391 } health_report { overall_health: HEALTHY } } }
I20260709 12:32:13.206945 21745 raft_consensus.cc:1275] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801 [term 3 FOLLOWER]: Refusing update from remote peer e2f153c452a54ca9af8d83e3d977beed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:32:13.208370 22110 consensus_queue.cc:1048] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:32:13.231716 21668 raft_consensus.cc:1275] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 3 FOLLOWER]: Refusing update from remote peer e2f153c452a54ca9af8d83e3d977beed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:32:13.233978 22110 consensus_queue.cc:1048] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:32:13.351096 21745 raft_consensus.cc:1275] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 3 FOLLOWER]: Refusing update from remote peer e2f153c452a54ca9af8d83e3d977beed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:32:13.352751 22137 consensus_queue.cc:1048] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e0e4824e8447c49a6cd89a1a208801" member_type: VOTER last_known_addr { host: "127.4.65.2" port: 42355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:32:13.363140 21670 raft_consensus.cc:1275] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [term 3 FOLLOWER]: Refusing update from remote peer e2f153c452a54ca9af8d83e3d977beed: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:32:13.365633 22110 consensus_queue.cc:1048] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [LEADER]: Connected to new peer: Peer: permanent_uuid: "634b4ccc587741a1b468e0a06fe92dd3" member_type: VOTER last_known_addr { host: "127.4.65.1" port: 37865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:32:13.574582  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:32:13.579770  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:32:13.580253  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:32:14.581624  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:32:14.588099  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:32:14.588670  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:32:15.590065  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:32:15.596030  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:32:15.596622  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:32:16.597631  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:32:16.603691  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:32:16.604163  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:32:17.605255  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:32:17.610787  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:32:17.611281  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:32:18.612507  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:32:18.618641  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:32:18.619270  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:32:19.825624  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:32:19.832490  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:32:19.833158  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:32:20.091148 21691 debug-util.cc:398] Leaking SignalData structure 0x7b080018b280 after lost signal to thread 4359
W20260709 12:32:20.092962 21691 debug-util.cc:398] Leaking SignalData structure 0x7b080029a840 after lost signal to thread 21602
W20260709 12:32:20.094419 21691 debug-util.cc:398] Leaking SignalData structure 0x7b0800204d40 after lost signal to thread 21694
W20260709 12:32:20.095960 21691 debug-util.cc:398] Leaking SignalData structure 0x7b08001c6ea0 after lost signal to thread 21769
W20260709 12:32:20.097182 21691 debug-util.cc:398] Leaking SignalData structure 0x7b08000492c0 after lost signal to thread 21843
I20260709 12:32:20.834208  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:32:20.839458  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:32:20.839998  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:32:21.841032  4356 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:32:21.846624  4356 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:32:21.847133  4356 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:32:22.915074  4356 tablet_server.cc:179] TabletServer@127.4.65.1:0 shutting down...
I20260709 12:32:22.957496  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:32:22.958480  4356 tablet_replica.cc:333] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:22.959519  4356 raft_consensus.cc:2243] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:22.960480  4356 raft_consensus.cc:2272] T 823c1ea9255c4ea9babb0445a43d721b P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:32:22.962925 21782 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 111 similar messages]
I20260709 12:32:22.964123  4356 tablet_replica.cc:333] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:22.965130  4356 raft_consensus.cc:2243] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:22.965951  4356 raft_consensus.cc:2272] T c5895ee5d036404fbe0eb3227d060196 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:32:22.966966 21782 consensus_peers.cc:597] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed -> Peer 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865): Couldn't send request to peer 634b4ccc587741a1b468e0a06fe92dd3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:22.969480  4356 tablet_replica.cc:333] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:22.970234  4356 raft_consensus.cc:2243] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:22.970798  4356 raft_consensus.cc:2272] T 8d5befb6b1e942dd997193ab1c266e24 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:22.974594  4356 tablet_replica.cc:333] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:22.975301  4356 raft_consensus.cc:2243] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:22.976269  4356 raft_consensus.cc:2272] T 3bcdf6089fe44597974ffc8c1a242fae P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:22.978953  4356 tablet_replica.cc:333] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:22.979529  4356 raft_consensus.cc:2243] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:22.979990  4356 raft_consensus.cc:2272] T 97f6d0a18b194f388f889318024f8420 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:22.982714  4356 tablet_replica.cc:333] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:22.983532  4356 raft_consensus.cc:2243] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:22.984591  4356 raft_consensus.cc:2272] T f00bda93a32d4b7097c987570f9bd200 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:22.987337  4356 tablet_replica.cc:333] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:22.987978  4356 raft_consensus.cc:2243] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:22.989024  4356 raft_consensus.cc:2272] T 51ddca609eb944c6b5d2d3db607e6a5b P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:22.991647  4356 tablet_replica.cc:333] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:22.992430  4356 raft_consensus.cc:2243] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:22.993044  4356 raft_consensus.cc:2272] T cdb21adbf3984dc0a5e3371e5151334f P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:22.995573  4356 tablet_replica.cc:333] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:22.996192  4356 raft_consensus.cc:2243] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:22.996820  4356 raft_consensus.cc:2272] T 7aa93a18f572472ab61d61ae8b464c83 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:22.999383  4356 tablet_replica.cc:333] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:23.000008  4356 raft_consensus.cc:2243] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.001250  4356 raft_consensus.cc:2272] T 47ee249250654b38bef5ffa786146d29 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.004113  4356 tablet_replica.cc:333] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:23.004792  4356 raft_consensus.cc:2243] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.005393  4356 raft_consensus.cc:2272] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P 634b4ccc587741a1b468e0a06fe92dd3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.008378  4356 tablet_replica.cc:333] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:23.009621  4356 raft_consensus.cc:2243] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.010488  4356 raft_consensus.cc:2272] T 26f565ff0b8943889da95d998b274773 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:32:23.013975 21708 consensus_peers.cc:597] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 -> Peer 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865): Couldn't send request to peer 634b4ccc587741a1b468e0a06fe92dd3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:23.014672  4356 tablet_replica.cc:333] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:23.015686  4356 raft_consensus.cc:2243] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.016750  4356 raft_consensus.cc:2272] T d82dd69565514cf498933ccc35fbedf2 P 634b4ccc587741a1b468e0a06fe92dd3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.020318  4356 tablet_replica.cc:333] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:23.021119  4356 raft_consensus.cc:2243] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Raft consensus shutting down.
W20260709 12:32:23.021133 21782 consensus_peers.cc:597] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed -> Peer 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865): Couldn't send request to peer 634b4ccc587741a1b468e0a06fe92dd3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:23.022183  4356 raft_consensus.cc:2272] T a09e03b0d2f7447fb6af987f95aa4660 P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.024945  4356 tablet_replica.cc:333] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:23.025581  4356 raft_consensus.cc:2243] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.026229  4356 raft_consensus.cc:2272] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.029256  4356 tablet_replica.cc:333] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:23.029909  4356 raft_consensus.cc:2243] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.030579  4356 raft_consensus.cc:2272] T c2fe8ccb5201424abf468b013c79119e P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:32:23.033159 21708 consensus_peers.cc:597] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 -> Peer 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865): Couldn't send request to peer 634b4ccc587741a1b468e0a06fe92dd3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:23.034528  4356 tablet_replica.cc:333] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:23.035241  4356 raft_consensus.cc:2243] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.036465  4356 raft_consensus.cc:2272] T 2200cddad7904f43b989bf2aa66d463e P 634b4ccc587741a1b468e0a06fe92dd3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.039268  4356 tablet_replica.cc:333] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3: stopping tablet replica
I20260709 12:32:23.039837  4356 raft_consensus.cc:2243] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.040846  4356 raft_consensus.cc:2272] T 9ae6b3a3385a420ca2a71c45a1746834 P 634b4ccc587741a1b468e0a06fe92dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:32:23.083738 21708 consensus_peers.cc:597] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 -> Peer 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865): Couldn't send request to peer 634b4ccc587741a1b468e0a06fe92dd3. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:37865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:23.111706  4356 tablet_server.cc:196] TabletServer@127.4.65.1:0 shutdown complete.
W20260709 12:32:23.155818 21782 consensus_peers.cc:597] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed -> Peer 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865): Couldn't send request to peer 634b4ccc587741a1b468e0a06fe92dd3. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:37865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:23.173553  4356 tablet_server.cc:179] TabletServer@127.4.65.2:0 shutting down...
W20260709 12:32:23.182742 21708 consensus_peers.cc:597] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 -> Peer 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865): Couldn't send request to peer 634b4ccc587741a1b468e0a06fe92dd3. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:37865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:32:23.212869 21782 consensus_peers.cc:597] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed -> Peer 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865): Couldn't send request to peer 634b4ccc587741a1b468e0a06fe92dd3. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:37865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:32:23.214088 21708 consensus_peers.cc:597] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 -> Peer 634b4ccc587741a1b468e0a06fe92dd3 (127.4.65.1:37865): Couldn't send request to peer 634b4ccc587741a1b468e0a06fe92dd3. Status: Network error: Client connection negotiation failed: client connection to 127.4.65.1:37865: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:32:23.216508  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:32:23.217330  4356 tablet_replica.cc:333] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.218272  4356 raft_consensus.cc:2243] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.219237  4356 raft_consensus.cc:2272] T 823c1ea9255c4ea9babb0445a43d721b P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.221859  4356 tablet_replica.cc:333] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.222494  4356 raft_consensus.cc:2243] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.223392  4356 raft_consensus.cc:2272] T 8d5befb6b1e942dd997193ab1c266e24 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.225893  4356 tablet_replica.cc:333] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.226640  4356 raft_consensus.cc:2243] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.227330  4356 raft_consensus.cc:2272] T c5895ee5d036404fbe0eb3227d060196 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.229964  4356 tablet_replica.cc:333] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.230609  4356 raft_consensus.cc:2243] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.231130  4356 raft_consensus.cc:2272] T 2200cddad7904f43b989bf2aa66d463e P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.233693  4356 tablet_replica.cc:333] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.234331  4356 raft_consensus.cc:2243] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.234899  4356 raft_consensus.cc:2272] T 3bcdf6089fe44597974ffc8c1a242fae P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.237257  4356 tablet_replica.cc:333] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.237864  4356 raft_consensus.cc:2243] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.238366  4356 raft_consensus.cc:2272] T 97f6d0a18b194f388f889318024f8420 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.240717  4356 tablet_replica.cc:333] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.241324  4356 raft_consensus.cc:2243] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.241897  4356 raft_consensus.cc:2272] T f00bda93a32d4b7097c987570f9bd200 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.244143  4356 tablet_replica.cc:333] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.244774  4356 raft_consensus.cc:2243] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.245694  4356 raft_consensus.cc:2272] T cdb21adbf3984dc0a5e3371e5151334f P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.247956  4356 tablet_replica.cc:333] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.248625  4356 raft_consensus.cc:2243] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.249151  4356 raft_consensus.cc:2272] T 51ddca609eb944c6b5d2d3db607e6a5b P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.251353  4356 tablet_replica.cc:333] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.251924  4356 raft_consensus.cc:2243] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.253172  4356 raft_consensus.cc:2272] T 7aa93a18f572472ab61d61ae8b464c83 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.256572  4356 tablet_replica.cc:333] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.257333  4356 raft_consensus.cc:2243] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.257835  4356 raft_consensus.cc:2272] T 47ee249250654b38bef5ffa786146d29 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.260170  4356 tablet_replica.cc:333] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.260769  4356 raft_consensus.cc:2243] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.261601  4356 raft_consensus.cc:2272] T 26f565ff0b8943889da95d998b274773 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.263877  4356 tablet_replica.cc:333] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.264483  4356 raft_consensus.cc:2243] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.264945  4356 raft_consensus.cc:2272] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P a6e0e4824e8447c49a6cd89a1a208801 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.267298  4356 tablet_replica.cc:333] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.267874  4356 raft_consensus.cc:2243] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.268502  4356 raft_consensus.cc:2272] T d82dd69565514cf498933ccc35fbedf2 P a6e0e4824e8447c49a6cd89a1a208801 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.271062  4356 tablet_replica.cc:333] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.271667  4356 raft_consensus.cc:2243] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.272234  4356 raft_consensus.cc:2272] T a09e03b0d2f7447fb6af987f95aa4660 P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.275002  4356 tablet_replica.cc:333] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.275554  4356 raft_consensus.cc:2243] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.276536  4356 raft_consensus.cc:2272] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.279062  4356 tablet_replica.cc:333] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.279615  4356 raft_consensus.cc:2243] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.280581  4356 raft_consensus.cc:2272] T c2fe8ccb5201424abf468b013c79119e P a6e0e4824e8447c49a6cd89a1a208801 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.283095  4356 tablet_replica.cc:333] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801: stopping tablet replica
I20260709 12:32:23.283619  4356 raft_consensus.cc:2243] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.284101  4356 raft_consensus.cc:2272] T 9ae6b3a3385a420ca2a71c45a1746834 P a6e0e4824e8447c49a6cd89a1a208801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.331039  4356 tablet_server.cc:196] TabletServer@127.4.65.2:0 shutdown complete.
I20260709 12:32:23.384227  4356 tablet_server.cc:179] TabletServer@127.4.65.3:0 shutting down...
I20260709 12:32:23.421005  4356 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:32:23.421980  4356 tablet_replica.cc:333] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.423156  4356 raft_consensus.cc:2243] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.424106  4356 raft_consensus.cc:2272] T 2200cddad7904f43b989bf2aa66d463e P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.427603  4356 tablet_replica.cc:333] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.428601  4356 raft_consensus.cc:2243] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.429128  4356 raft_consensus.cc:2272] T 8d5befb6b1e942dd997193ab1c266e24 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.431433  4356 tablet_replica.cc:333] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.431970  4356 raft_consensus.cc:2243] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.432559  4356 raft_consensus.cc:2272] T 823c1ea9255c4ea9babb0445a43d721b P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.435021  4356 tablet_replica.cc:333] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.435604  4356 raft_consensus.cc:2243] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.436556  4356 raft_consensus.cc:2272] T c5895ee5d036404fbe0eb3227d060196 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.439059  4356 tablet_replica.cc:333] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.439590  4356 raft_consensus.cc:2243] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.440058  4356 raft_consensus.cc:2272] T 51ddca609eb944c6b5d2d3db607e6a5b P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.442360  4356 tablet_replica.cc:333] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.442884  4356 raft_consensus.cc:2243] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.443337  4356 raft_consensus.cc:2272] T cdb21adbf3984dc0a5e3371e5151334f P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.445638  4356 tablet_replica.cc:333] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.446156  4356 raft_consensus.cc:2243] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.446556  4356 raft_consensus.cc:2272] T 47ee249250654b38bef5ffa786146d29 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.449095  4356 tablet_replica.cc:333] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.449621  4356 raft_consensus.cc:2243] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.450073  4356 raft_consensus.cc:2272] T 7aa93a18f572472ab61d61ae8b464c83 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.452170  4356 tablet_replica.cc:333] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.452913  4356 raft_consensus.cc:2243] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.453470  4356 raft_consensus.cc:2272] T 86f4fd3c8b08460aa53f4f5a7d3e20cf P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.455695  4356 tablet_replica.cc:333] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.456267  4356 raft_consensus.cc:2243] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.456816  4356 raft_consensus.cc:2272] T c2fe8ccb5201424abf468b013c79119e P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.459280  4356 tablet_replica.cc:333] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.459897  4356 raft_consensus.cc:2243] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.460459  4356 raft_consensus.cc:2272] T 3bcdf6089fe44597974ffc8c1a242fae P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.462711  4356 tablet_replica.cc:333] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.463307  4356 raft_consensus.cc:2243] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.463806  4356 raft_consensus.cc:2272] T f00bda93a32d4b7097c987570f9bd200 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.466146  4356 tablet_replica.cc:333] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.466797  4356 raft_consensus.cc:2243] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.467710  4356 raft_consensus.cc:2272] T 97f6d0a18b194f388f889318024f8420 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.470067  4356 tablet_replica.cc:333] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.470643  4356 raft_consensus.cc:2243] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.471149  4356 raft_consensus.cc:2272] T 26f565ff0b8943889da95d998b274773 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.473502  4356 tablet_replica.cc:333] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.474071  4356 raft_consensus.cc:2243] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.474949  4356 raft_consensus.cc:2272] T d82dd69565514cf498933ccc35fbedf2 P e2f153c452a54ca9af8d83e3d977beed [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.477209  4356 tablet_replica.cc:333] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.477802  4356 raft_consensus.cc:2243] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.478711  4356 raft_consensus.cc:2272] T 27cd681d2bdd4f55913b3ea3ba7fbb4c P e2f153c452a54ca9af8d83e3d977beed [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.481128  4356 tablet_replica.cc:333] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.481781  4356 raft_consensus.cc:2243] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.482333  4356 raft_consensus.cc:2272] T a09e03b0d2f7447fb6af987f95aa4660 P e2f153c452a54ca9af8d83e3d977beed [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.484629  4356 tablet_replica.cc:333] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed: stopping tablet replica
I20260709 12:32:23.485167  4356 raft_consensus.cc:2243] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:32:23.485668  4356 raft_consensus.cc:2272] T 9ae6b3a3385a420ca2a71c45a1746834 P e2f153c452a54ca9af8d83e3d977beed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.531560  4356 tablet_server.cc:196] TabletServer@127.4.65.3:0 shutdown complete.
I20260709 12:32:23.574141  4356 master.cc:562] Master@127.4.65.62:37877 shutting down...
I20260709 12:32:23.599159  4356 raft_consensus.cc:2243] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:32:23.599841  4356 raft_consensus.cc:2272] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:32:23.600198  4356 tablet_replica.cc:333] T 00000000000000000000000000000000 P f8c98952f2ef40d4ada68b202df1e026: stopping tablet replica
I20260709 12:32:23.622680  4356 master.cc:584] Master@127.4.65.62:37877 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1 (21315 ms)
[----------] 2 tests from FilterSoftDeletedTableTest (45951 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 2 test suites ran. (560218 ms total)
[  PASSED  ] 7 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LeaderRebalancerTest.AddTserver

 1 FAILED TEST
I20260709 12:32:23.689738  4356 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:220: suppressed but not reported on 2457 messages since previous log ~35 seconds ago
I20260709 12:32:23.690006  4356 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:304: suppressed but not reported on 2457 messages since previous log ~35 seconds ago
I20260709 12:32:23.690207  4356 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:139: suppressed but not reported on 2452 messages since previous log ~35 seconds ago
I20260709 12:32:23.690420  4356 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 9 messages since previous log ~54 seconds ago
I20260709 12:32:23.690616  4356 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:548: suppressed but not reported on 9 messages since previous log ~54 seconds ago
I20260709 12:32:23.690833  4356 logging.cc:436] LogThrottler Service Queue Overflow: suppressed but not reported on 9 messages since previous log ~420 seconds ago
I20260709 12:32:23.691032  4356 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/rpc/service_pool.cc:136: suppressed but not reported on 1 messages since previous log ~420 seconds ago
I20260709 12:32:23.691236  4356 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 4 messages since previous log ~165 seconds ago
I20260709 12:32:23.691512  4356 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 9 messages since previous log ~0 seconds ago
I20260709 12:32:23.691843  4356 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet.cc:2406: suppressed but not reported on 53 messages since previous log ~20 seconds ago
